What is Mixtral 8x7B?

Large language models are machine learning systems trained on vast amounts of text so they can generate, summarize, translate and analyze language. A hard part of this field is that bigger models tend to be more capable but cost more to run. Mixtral 8x7B is a language model from Mistral AI that tackles this with a design called a sparse mixture of experts: each layer contains 8 separate feedforward blocks (the "experts"), and for every token (a word or word fragment) a router network picks two of them to process it. The idea is that the model holds many parameters in total while using only a part of them for each token. The authors released a paper describing it in January 2024.

Written from primary sources and reference pages, and checked against them by an independent AI fact-checker. Drafted with AI assistance.