What is AI model distillation?
It is why there are models almost as good as the best ones, but much cheaper.
Distillation is a technique in which a small AI model is trained to imitate a large one, passing on a good part of its knowledge.
The idea comes from a 2015 scientific paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean 1: the knowledge of a large model, or of a group of models, can be compressed into a single model that is much easier to use in production 1.
How it works, in one picture
Think of an apprentice learning by watching an experienced craftsperson. The apprentice doesn't have to repeat all the master's years of experience: they copy the master's choices, one after another, until they make very similar ones. The large model is the master, the small one is the apprentice.
Why it matters to you
A small model needs less computing for each answer. Less computing means faster answers, lower prices and, in the best cases, the option of running the model on a computer in your business instead of in the cloud.
When a vendor launches a "mini", "light" or mid-range version that comes close to its flagship model at a fraction of the price, this technique is often part of the reason.
What to do with it
For repetitive tasks, such as answering the same customer questions or sorting incoming requests, try a small model first. If it does the job, you save money. If it doesn't, move up a size. Starting with the biggest one "to be safe" is the most common way of overpaying for AI.
Frequently asked questions
Is a distilled model worse?
Usually a little less capable than the model it learns from, especially on difficult tasks. On repetitive ones the difference often doesn't show, but you need to test it.
Can I distil a model for my company?
Technically yes, but it rarely makes sense for an SMB: it is simpler to pick a small model that is already available and connect it to your own data.
Read next
What a frontier model is, and when it's not worth paying
They are the most powerful models on the market. For many jobs in an SMB, they are also the most wasteful.
GlossaryWhat's an LLM? The model behind ChatGPT, Claude and Gemini
Almost every AI tool you are offered uses one. Often it is the same one, resold at different prices.
News that mattersOllama: route requests with AI locally, at no cost
Decision models don't write text: they pick a label and tell you how confident they are. For sorting emails and tickets, that is what you need.
Want to know whether a small model is enough for your case? Let's talk.
Sources
- Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network (2015), https://arxiv.org/abs/1503.02531
Text created with the support of AI. Method and content are AIR's.