The Navalia AI agent is a mid-sized open model that, according to its authors, beat GPT-5.4 and Claude Opus 4.6 on a tough benchmark of financial documents. The trick isn’t a bigger model. It’s a factory that turns real company reports into thousands of hard practice problems, each with an answer that has been checked twice.
Key takeaways
- Navalia is a training framework from the AllSpark Team, described in a paper submitted to arXiv on 13 October 2026.
- It builds multistep finance tasks from corporate reports, then verifies every answer with a separate model and with code.
- Starting from Qwen3.6-35B-A3B, Navalia improved on all five benchmarks tested, by 13 to 23 points.
- On OfficeQA Pro it scored 62.66, ahead of the 50.40 the paper reports for GPT-5.4.
- Weights and code are promised but not yet public. The GitHub page says they are “undergoing approval for public release”.
What is the Navalia AI agent?
Navalia is a recipe for training what the authors call a “Cowork Agent”. That’s an LLM that doesn’t just chat. It plans, calls tools, opens files and works through a long task on its own, the way a junior analyst might.
The Navalia paper lists ten authors, led by Hengqi Liu and Ziyuan Liu, with Yang Qin as project lead. It appeared in arXiv’s AI feed on 21 October 2026. The authors chose finance as the test case because the work is unforgiving: you have to pull numbers from several documents, line up the right periods and units, and chain calculations without a single slip.
Why is training finance agents so hard?
Agents learn from tasks with known answers. In finance, good tasks are expensive. Vals AI’s Finance Agent Benchmark needed experts from banks, hedge funds and private equity firms to write 537 questions.
The cheap shortcut is to ask an LLM to make up questions. The Navalia authors say that tends to produce either easy tasks or hard ones with answers you can’t trust. Training on wrong answers is worse than not training at all.
How Navalia builds tasks it can trust
Think of it as a test-writing desk with a strict fact-checker sitting next to it. The pipeline has five stages:
- Two views of each report. Every financial report becomes readable Markdown plus a structured database of facts. Facts missing a number, a date or a unit are thrown out.
- Pick an operation. The system samples an analytical operation, such as a growth rate or a ratio, then picks the companies, periods and metrics. It also adds distractor documents: other years from the same company and reports from its peers.
- Choose the evidence. An LLM reads each document’s facts and selects the ones the task needs. If the definitions or scopes don’t match, the task is dropped.
- Build the calculation chain. Complex steps run as deterministic code, not LLM arithmetic. Each step has to feed the next, so a task can’t be solved by skipping ahead.
- Check it twice. A separate reviewer model works out its own answer without seeing the generator’s reasoning. Then code checks that every intermediate result is used and the periods match. Only tasks that pass both checks survive.
The resulting tasks are heavy. On average, each one comes with 5.85 relevant documents buried among 208.69 distractors, and up to 994 in the worst case. A typical solved attempt took about 59 tool-use turns.
How was the model trained?
Training ran in two phases. First came supervised fine-tuning (SFT), which means imitating worked examples. A stronger teacher model solved the tasks, and only correct attempts with valid tool calls were kept. The paper reports roughly 6,000 to 6,800 of these long trajectories, depending on how they were sliced.
Then came reinforcement learning (RL), where the model practises and gets rewarded for correct final answers. Tasks that every attempt solved, or every attempt failed, were dropped because they teach nothing. Attempts could run up to 200 turns.
Navalia benchmark results compared
Here is how Navalia-35B-A3B compares with its base model and two rivals, using scores from the paper’s own table. A dash means the paper gives no score.
| Benchmark | Qwen3.6-35B-A3B (base) | Navalia-35B-A3B | GPT-5.4 | Kimi-K3 |
|---|---|---|---|---|
| OfficeQA Pro | 39.6 | 62.66 | 50.40 | 67.92 |
| FinSearchComp T2&T3 | 41.7 | 55.80 | – | 68.8 |
| Finance Agent (Public-77) | 43.19 | 57.10 | – | 65.02 |
| APEX-Agents-AA Pass@1 | 12.24 | 25.44 | 33.26 | 37.13 |
| Workspace-Bench-Lite | 46.2 | 59.27 | 52.9 | 60.56 |
Two things stand out. The jump over the base model is large and consistent, from +13.07 to +23.06 points. And the gains spread beyond finance to general office tasks such as Workspace-Bench-Lite, which suggests the model learned habits, not just finance trivia.
Context helps here. Databricks’ OfficeQA Pro paper found that frontier agents averaged just 34.1% when given its corpus of U.S. Treasury Bulletins. A score in the 60s from a model with roughly 3 billion active parameters per token is a big deal.
My take: be careful with the headline win over GPT-5.4. Navalia still trails GPT-5.4 on APEX-Agents, and larger open models like Kimi-K3 beat it on every benchmark. These are the authors’ own numbers, and harness choices change scores a lot. If you build agents, our AI agent evaluation checklist explains why.
What are the limits?
The authors list them plainly. Navalia uses one agent in a simple ReAct loop, a pattern where the model alternates between thinking and acting. It has no memory beyond a sliding window, so early steps fall out of view on long tasks. The RL reward also only scores the final answer, so it can’t tell which step earned the credit.
Those gaps line up with the higher rungs of our maturity model for production-grade AI agents: memory, orchestration and finer-grained feedback.
What happens next
The paper says the team plans to release model weights and open-source key parts of its task-building, training and evaluation pipelines. As of 22 October 2026, the Navalia GitHub page holds only a README saying the code and weights are awaiting approval.
If the release happens, the task factory may matter more than the model. A pipeline that makes verified, multistep tasks from public filings could be reused for tax, audit or insurance work. Independent reruns of the benchmarks will show whether the scores hold up.
FAQ
Is the Navalia AI agent available to download?
Not yet. The authors plan to release weights and key code, but as of 22 October 2026 the GitHub page says both are still awaiting approval.
What model is Navalia based on?
It starts from Qwen3.6-35B-A3B, a mixture-of-experts model. Only a small slice of its 35 billion parameters is active for each token, which keeps it cheap to run.
Does Navalia really beat GPT-5.4?
On OfficeQA Pro and Workspace-Bench-Lite it does, according to the paper’s own tests. On APEX-Agents it scores lower, and the results have not been independently verified.
Building your own agent? Steal Navalia’s best idea first: never train on an answer that hasn’t been checked by a second model and a piece of code.
Leave a comment