Ollama: route requests with AI locally, at no cost

Much of the "intelligent" work in an office is sorting. Is this email a complaint or an order? Does it go to administration or to support? Is it urgent? Until recently, to get an AI to do this, you had to send the text to an external service and pay as you go.

Since 29 September, Ollama, a free programme for running AI models on your own computer, has supported decision models ¹.

What they are, in practice

A decision model doesn't write. You ask it a closed question about a text and it answers with a choice, a yes or no, or a score, and for each option it gives you a probability ¹. Ollama has offered them since version 0.35 ².

The probability is the useful part. If the model is almost sure an email is for administration, you route it automatically. If it is unsure, you pass it to a person. That way the automation stops by itself where it doesn't know.

A concrete example

Ollama's example is a support ticket: "you charged me twice, refund me" ¹. With a single call the model answers three questions: which department (billing, technical, other), whether it is a refund request, and how urgent it is ¹.

Apply it to your inbox: every incoming message gets a department, a type and an urgency before anyone opens it.

Which models, and what machine you need

There are three models available ¹:

• nimble, with 9 billion parameters, an Apache 2.0 licence and a download of about 9 GB ³;
• tev1, in a 4 billion version (about 4.5 GB) and a 0.8 billion version (about 800 MB), experimental ⁴.

Ollama states 91 milliseconds per decision for nimble on a high-end Mac ¹. It doesn't publish requirements for an ordinary computer or one without a graphics card: before you decide, test on the machine you would really use. The smallest model accepts short texts, about 2,000 tokens ⁴, so for long emails you need to use nimble or send only the beginning.

Why it matters for an SMB

The first reason is privacy. Your customers' emails stay on your computer: no text goes to an external provider, and there is no data processing contract to negotiate for this task.

The second is cost. Running locally, you don't pay per request ¹, so the cost doesn't grow with the number of emails.

The third is the limit, and it is only fair to say so: Ollama is a tool, not a finished product. Someone has to connect it to the mailbox or the management software and decide the probability thresholds.

What to do now

• Write down the three or four questions you would ask yourself about every incoming email. Those are the questions to put to the model.
• Collect around a hundred real emails already sorted by hand: you will use them to see how often the model gets it wrong.
• Decide the threshold below which the email goes to a person. It is better to start high and lower it.