吴正龙’s large-language-models Bookmarks

The reliable performance of on-policy training with the cost-efficiency of a dense reward signal.

LoRA may offer advantages in the cost and speed of post-training, and there are also a few operational reasons to prefer it to full fine-tuning.

The industry rocking paper from DeepSeek that sent shock waves around the world. DeepSeek-R1 incorporates multi-stage training and cold-start data before RL.

To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures.

A practical guide to building successful LLM products, covering the tactical, operational, and strategic.

MoEs are pretrained much faster versus dense models, have faster inference compared to a model with the same number of parameters, and require high VRAM as all experts are loaded in memory.

Practical patterns for integrating large language models (LLMs) into real systems and products. Overview of seven key patterns: evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.

Different angle on attention mechanism to help build further intuition. Intended for people who have read the "Attention is All You Need" paper and have a basic understanding of how attention works.