#training
1 post · all tags
-
· Deepseek, an overview and quick notes
When DeepSeek-V3 came out I first wrote these notes as a bullet list of its headline claims. The numbers are striking, but what interested me more were the architectural tricks that make a 671B-parameter model cheap to train and cheap to serve. So I went back to the technical report and worked through the main ideas properly: how Multi-head Latent Attention shrinks the KV cache, how the MoE layers are routed, the auxiliary-loss-free load balancing rule (with a small simulation you can play with), multi-token prediction, and a short note on the FP8 and pipeline-parallel engineering.