Vals AI publishes an Inference Provider Index that compares how the same open-weight model behaves when called through different APIs. The benchmark page is marked Updated 10/20/2026; Vals AI’s Updates list dated 10/22/2026 announces the index. The run covers DeepSeek V4.1 Flash, Kimi K3, and MiniMax M3 on each model’s first-party API and on third-party providers that served those weights at the time—Together AI, Fireworks, and Baseten (MiniMax M3 was not available on Baseten’s Model APIs during the evaluation). Related reading on how Vals scores agent work is in our Vals Index v2.2 roundup.
What Vals measured
Each model–provider pair ran the same 165 multi-step agent tasks: 66 from Finance Agent v2, 33 from Vibe Code Bench, and 66 from Legal Research. Vals AI says every provider used the same agent and task definitions, with sampling and reasoning settings aligned within each model. Overall accuracy averages the three benchmarks with equal weight. Cost per task averages recorded successful request costs. Timing metrics include time to first token, time to first answer (excluding reasoning), and generation speed in tokens per second. Vals AI says it plans to rerun the index about every two weeks on the same tasks and settings, with a fresh first-party baseline each time.
Overall accuracy stayed within error
On the headline overall score, every third-party provider landed within statistical error of the model creator’s API. Vals AI does not frame the run as proof that third-party hosts broadly “dumb down” these models. Gaps still show up on single benchmarks. On Legal Research, Kimi K3 scored 60.6% through Baseten against 75.8% on Moonshot’s API. On Coding, the same model scored 87.7% through Fireworks against 80.1% on Moonshot. Tool-call failure rates stayed within about two percentage points across providers for each model and benchmark; failure rates tracked the benchmark more than the host—roughly 1–6% on Finance and Coding, and 6–12% on Legal, according to Vals AI.
Cost can move more than the overall score
Vals AI reports that the same model cost up to about 2.3× more per task depending on the provider. For all three models in this run, the creator’s own API was neither the cheapest nor the fastest option.
Kimi K3 is the clearest cost outlier: about $12.65 per task on Baseten, more than twice the about $5.43–$5.97 Vals AI records on Together AI, Fireworks, and Moonshot, while writing about the same amount. Vals AI also reports a lower prompt-cache hit rate on Baseten for Kimi K3—72% of input tokens from cache against 94% on Together AI and Fireworks, and 70% against 98% on the Coding subset.
MiniMax M3 cost about half as much on Together AI (about $1.01) and Fireworks (about $1.07) as on MiniMax’s own API (about $2.18).
On DeepSeek V4.1 Flash, Vals AI’s summary table shows Fireworks at about $0.36 per task (about 34% below DeepSeek’s first-party about $0.55), Together AI at about $0.48, and Baseten at about $0.59. Official DeepSeek list prices by model and peak hours are covered separately in DeepSeek API Pricing (October 2026); how Flash 1017 sits on other public boards is in our DeepSeek V4.1 Flash 1017 benchmark note.
Speed and first-token wait also flip by host
Generation speed and time-to-first-token change with the host even when the weights share a name. For Kimi K3, Vals AI reports a median about 229 tokens per second on Baseten—about five times the about 45 tokens per second on Moonshot’s API—while Moonshot’s API was the slowest to start: a median about 7.2 seconds to the first token and about 12.6 seconds to the first answer, against about 1.8 seconds and 5.5 seconds on Together AI.
For DeepSeek V4.1 Flash in the same summary table, Together AI leads generation speed at a median about 398 tokens per second (about 1.4× DeepSeek’s first-party about 286), Baseten about 316, and Fireworks about 103 (about 0.4× first party)—even though Fireworks posts the highest overall accuracy in that DeepSeek slice.
Snapshot: accuracy, cost, and speed by model
Numbers below are Vals AI’s chart and summary figures for this run (page Updated 10/20/2026). Accuracy includes ±1 standard error where Vals AI published it.
DeepSeek V4.1 Flash
| Provider | Overall accuracy | Cost / task (USD) | Median tokens/sec |
|---|---|---|---|
| Fireworks | 73.0% ± 3.0 pp | ~$0.36 | 103 |
| DeepSeek (first party) | 70.3% ± 3.1 pp | ~$0.55 | 286 |
| Baseten | 68.9% ± 3.3 pp | ~$0.59 | 316 |
| Together AI | 66.4% ± 3.3 pp | ~$0.48 | 398 |
Kimi K3
| Provider | Overall accuracy | Cost / task (USD) | Median tokens/sec |
|---|---|---|---|
| Fireworks | 73.9% ± 2.7 pp | ~$5.98 | 48 |
| Together AI | 73.2% ± 3.0 pp | ~$5.43 | 70 |
| Moonshot (first party) | 71.8% ± 2.9 pp | ~$5.81 | 45 |
| Baseten | 67.3% ± 3.1 pp | ~$12.65 | 229 |
This index run is on Kimi K3, not Kimi K3.1. List prices and product notes for K3.1 Pro are in Kimi K3.1 Pro API Pricing (October 2026); K3 provider gaps are not K3.1 results.
MiniMax M3
| Provider | Overall accuracy | Cost / task (USD) | Median tokens/sec |
|---|---|---|---|
| MiniMax (first party) | 42.9% ± 3.3 pp | ~$2.18 | 157 |
| Fireworks | 39.3% ± 3.4 pp | ~$1.07 | 262 |
| Together AI | 37.6% ± 3.4 pp | ~$1.01 | 116 |
How to read this for routing choices
When picking an open-weight model by name and then shopping hosts on OpenRouter or a direct third-party API, Vals AI’s framing is that hardware, caching, and tool-call parsing can move cost and latency even when overall accuracy stays inside the error bars. A cheaper or faster host in this table is not a claim about a full production bill, every task type, or a later model version. Provider stacks also change over time; Vals AI notes OpenRouter has reported quality regressions as providers adjusted serving, and says it will rerun the index on a fixed task set so gaps can be checked again.
Whether a given provider still matches these October 2026 figures on a given prompt and tool stack is for reference only. Feedback can go through the site-owner contact link in the footer.
