Which local models can tell when a message should become a task? For Folio, I had six model profiles evaluated on synthetic cases. Gemma matched the reference most often. Kev found every reference task, but also turned many other cases into tasks. An additional context experiment improved the structure of task proposals while producing more false tasks.
This compares specific local configurations, not general model quality. The reference and content assessment came from the same AI author; there was no independent human accuracy assessment.
The practical question
An invoice, a changed appointment or a contract update can require action. An informational message should not automatically become another item on a task list. Sometimes there is not enough context to decide.
Folio is being built to help make sense of these inputs and connect them with existing project knowledge. A valid response format is only part of that job. I also want to know which tasks a model misses, which it invents and whether a proposed task preserves the relevant source information.
The immediate reason for this comparison was the arrival of Aleph Alpha's Kolibri and Jared Palmer's Kev. Kev is an open Jev-like decision model that returns probabilities over supplied options. Jev itself was not tested. Existing Gemma and Qwen profiles provided comparisons.
What 6,000 attempts actually means
The classification test contained 300 synthetic cases from 75 dossiers: 100 tasks, 100 unclear cases and 100 non-tasks. Six profiles answered the same cases in three planned rounds, giving 5,400 requests.
A separate workflow test examined proposals from prepared mail imports. It contained 40 cases from 40 dossiers, including 20 reference-task dossiers. Five configurations, including the additional context variant, ran for three rounds: another 600 requests. This did not test the preceding mail preparation or the whole Folio application.
6,000 requests are not 6,000 independent cases. Cases from the same dossier are related, and repeated requests do not create new independent evidence. The tables below use the first round, chosen in advance as the primary round. Parameters were not changed in response to main-test results.
Finding tasks and avoiding false tasks
| Local profile | Reference matches out of 300 | Tasks found out of 100 | False tasks out of 200 |
|---|---|---|---|
| Gemma 4 31B / QAT Q4_0 | 202 | 65 | 35 |
| Qwen3.8 27B / MLX | 164 | 8 | 1 |
| Qwen3 30B A3B / Thinking / MLX | 163 | 78 | 64 |
| Kolibri 1 / Q3_K_S / CPU + Accelerate | 162 | 74 | 95 |
| Qwen3.6 35B A3B / MLX | 158 | 30 | 42 |
| Kev 4B / bf16 / MLX | 130 | 100 | 170 |
Reference matches measure agreement with the AI-authored three-class reference. The last denominator combines 100 unclear cases and 100 non-tasks. A false task means one of those cases was classified as a task; no real action was executed as a result.
Gemma matched 202 of 300 references, or 67.3%. It was the strongest starting point in this mixture, but still missed 35 of the 100 tasks. Qwen3.8 produced almost no false tasks while identifying only eight actual tasks. Caution alone is not enough.
In the setup described on the website, Qwen3.8 is the primary voice in the multi-model mail process and powers the local companion. The 8/100 result comes from the separate task-detection test and its own instructions; the quality of the full mail process requires its own assessment.
Kev illustrates the opposite trade-off. Finding all 100 tasks sounds excellent until the 170 false tasks are included. This profile would be unsuitable as an unchecked source of tasks in this test mixture. Its filter policy had already been disabled before the main run and remained disabled. The Kev-plus-Gemma workflow therefore called only Gemma. It provides no measured benefit from a Kev prefilter.
Kolibri nearly matched Qwen3 30B A3B Thinking on overall reference agreement, but generated more false tasks. The paired dossier comparison does not establish a classification-quality advantage for Kolibri. This is not a verdict on other Kolibri configurations or applications.
Better context improved parts of the proposals
The workflow test also included a new prompt and context package for Gemma. It was developed using separate development cases and is an exploratory addition, not a replacement for the original baseline.
| First-round measure | Baseline | Context variant |
|---|---|---|
| Correct class | 27/40 | 30/40 |
| Output rules satisfied | 26/40 | 39/40 |
| Appropriate chapter or topic for tasks | 9/20 | 20/20 |
| Appropriate future completion for tasks | 5/20 | 19/20 |
| False tasks | 6/20 | 9/20 |
The proposals more often respected the expected fields and their rules, and fit their project context. This assessed the original output, including its text-length limits. False tasks also increased. The three-case classification improvement is uncertain: its paired 95% dossier interval ranges from minus five to plus twenty percentage points.
Several rules changed together, so the experiment does not isolate a single cause. For Folio, this is a candidate for further development, not a reason to create tasks without review.
A strict test of completeness
One content criterion, W2, required every essential, predefined fact to appear in the proposal itself. A general reference back to the source material did not satisfy that requirement.
None of the 73 distinct assessed task answers passed this criterion in full. Those answers came from the 20 reference-task dossiers across configurations and rounds; they are not 73 independent cases. Identical answer content received the same assessment.
That result remains part of the report. It does not mean every concise proposal linked to its source is useless for Folio. The rubric demands more. A later assessment accepting a sufficient source reference would be an additional analysis and would need to remain visibly separate. Reporting individual measures is therefore more useful than compressing everything into a single success rate.
Limits of this local comparison
The profiles ran locally on an Apple M5 Pro with 48 GiB of memory. Kolibri used a community Q3_K_S quantization through an experimental CPU/Accelerate port, not Aleph Alpha's official FP8 configuration. Other profiles used different quantizations, context limits and, in some cases, native thinking. These runs do not establish a general speed winner.
Qwen3 30B A3B Thinking produced 22 technical classification errors in round one: output stopped at the fixed token budget. These were not timeouts. The first Kolibri workflow round contained one 90-second timeout. Failed answers were neither repaired nor retried.
The reference and semantic assessment came from the same AI author. Model names, variants, rounds and timings were hidden during content assessment, but style and known development findings could still reveal identity. The assessment was neither independent nor fully blind. No private production inbox was measured.
What changes for Folio
Gemma remains the strongest classification starting point among these measured profiles. Kev will not be enabled as a filter. The context variant deserves further work on multiple requests, source grounding and false tasks. Valid structure and correct meaning remain separate requirements. This test does not automatically switch any product model.
The refreshed Aion Lumen website shows this work through mail, sources and recurring costs. The wave provides the visual story; the evaluation shows some of the engineering underneath. The surfboard article explains how the project developed.
The aggregate results and definitions support checking the reported counts. They contain no private mail and are not a complete inference reproduction package. I would be interested in the neutral edge cases other developers use when testing their own task systems.
Measurement date: 7 October 2026. AI-authored reference and content assessment; article prepared with AI assistance.