This report turns our preregistered Kimi K3 protocol into a real API run. The official Global Kimi API completed 15 of 15 planned single-pass calls using the verified model ID kimi-k3: eight synthetic tool-selection cases, one code repair, one visual extraction and five long-context retrieval bands.
API PILOT COMPLETED — SINGLE PASS. Every preregistered exact check passed in this run, including retrieval from a request with 950,149 prompt tokens. This is encouraging evidence for these fixtures, not a reliability benchmark. Each case or context band ran once, on one account and date, with no corrective retries.
The direct answer
Kimi K3 produced the exact expected result in every bounded task in this pilot:
| Track | Single-pass result | What was independently verified |
|---|---|---|
| Synthetic tool calls | 8/8 cases passed | Exact tool, valid arguments, exact values and types, no extra arguments |
| Code repair | 5/5 tests passed | One applicable unified diff; only the permitted implementation file changed |
| Vision extraction | 6/6 fields correct | Five extracted values plus the requested computed total |
| Long-context retrieval | 20/20 facts correct | Four fixed facts at each of five actual prompt sizes from 32,154 to 950,149 tokens |
| API completion | 15/15 calls completed | Every scored response reported kimi-k3; no HTTP or transport failure |
| Usage-derived token cost | $5.687397 | 1,885,885 prompt tokens, 3,840 cached tokens and 2,674 completion tokens at the dated official rates |
| Later balance deduction | $5.68739 | Observed 13 minutes 41 seconds after completion; $0.000007 below the usage-derived calculation |
The result does not establish a universal K3 score, consistent performance across repeated runs, perfect recall for arbitrary 1M-token documents or general coding quality. It shows what happened once under a frozen, public method.
What we tested
The four tracks were fixed before the first paid response:
- Repair one deliberately broken JavaScript invoice function against five visible tests.
- Select among six non-operative synthetic tools across eight prompts and return exact JSON arguments.
- Read an original synthetic support dashboard and return six numeric fields.
- Retrieve four target facts from deterministic corpora at five approximate length bands.
We excluded web search, official tools, third-party gateways and cross-model comparisons. No production file, personal record or private business data was submitted.
Frozen environment and controls
| Control | Recorded value |
|---|---|
| Provider | Official Global Kimi Open Platform |
| Endpoint | https://api.moonshot.ai/v1/chat/completions |
| Model | kimi-k3, verified in GET /v1/models and every scored response |
| Returned context length | 1,048,576 tokens |
| Reasoning effort | high for all comparable requests |
| Streaming | Enabled with usage requested in the final stream event |
| Concurrency | 1 |
| Corrective retries | 0 |
| Approved hard cap | $7.50 with a $0.10 operational reserve |
| Preflight worst case | $6.21057 for all 15 calls |
| Usage-derived token cost | $5.687397 |
| Later balance deduction | $5.68739 |
| Run completion | August 6, 2026 UTC — August 5 in the operating time zone |
| Network region | Not independently verified |
K3 fixes temperature, top_p, n and the frequency and presence penalties. We omitted those fields as the current K3 quickstart instructs. max_completion_tokens was explicitly capped per track so the provider’s much larger default could not create an uncontrolled charge.
Why the protocol changed from v1.0 to v1.1
Version 1.0 remains unchanged as the original preregistration record. Once the owner approved a $7.50 ceiling, its repetition rules and maximum outputs produced a theoretical total slightly above that cap before the smaller input costs were counted.
Before any paid response, we froze v1.1 as a single-pass pilot. It kept every fixture and oracle unchanged, reduced output caps, defined the client-side cost gate and made one run valid for Code and Vision as a pilot. The full reliability test still requires the repetitions specified in v1.0.
Track 1: exact tool selection and arguments
The model received all six complete JSON Schemas on each request. tool_choice: "required" required a tool call but did not force a particular function. The selected function and arguments therefore remained the measured output.
| Case | Expected call | Observed result |
|---|---|---|
| TC01 | inventory_lookup({"sku":"ORB-17"}) | Exact pass |
| TC02 | calculate_shipping with 02139, 2.5, false | Exact pass |
| TC03 | schedule_review with ALTA-42, 2032-09-17 | Exact pass |
| TC04 | find_document({"filename":"field-notes-v3.md"}) | Exact pass |
| TC05 | add_line_item with NOVA-8, integer 4 | Exact pass |
| TC06 | convert_currency with 125.5, USD, EUR | Exact pass |
| TC07 | inventory_lookup({"sku":"teal-9x"}) | Exact case-preserving pass |
| TC08 | calculate_shipping with SW1A 1AA, 0.75, true | Exact pass |
All eight responses contained one valid call, the expected function, the exact required argument types and values, and no invented fields. The calls were synthetic and did not execute any action.
Tool-call usage
The eight calls used 5,001 prompt tokens, including 2,560 cached tokens, and 915 completion tokens. Their combined calculated cost was $0.021816. Individual wall times ranged from 3.426 to 5.698 seconds. These are eight different tasks, so the timing range is descriptive rather than a controlled latency comparison.
Track 2: code repair with executable tests
The unmodified fixture passed 2 of 5 tests and failed the three quantity-dependent cases as designed. K3 returned an applicable unified diff that made one functional change:
- (sum, line) => sum + line.unitPriceCents,
+ (sum, line) => sum + line.unitPriceCents * line.quantity,The harness rejected edits outside src/calculate-invoice.js, applied the patch in a clean copy and ran the unchanged suite.
| Code check | Result |
|---|---|
| Patch parsed and applied | Pass |
| Files changed | src/calculate-invoice.js only |
| Tests passed | 5/5 |
| Tests failed | 0 |
| Prompt tokens | 767 |
| Completion tokens | 222 |
| Wall time | 5.704 seconds |
| Calculated cost | $0.005631 |
The model saw the implementation and tests. This measures a bounded repair with visible tests, not general repository-level software engineering or hidden-test performance.
Track 3: visual extraction and arithmetic
We rendered the frozen 1200 × 700 SVG dashboard deterministically to PNG, retained both SHA-256 hashes and submitted the PNG as base64. A strict JSON Schema permitted exactly six numeric fields.
| Requested field | Expected | Observed |
|---|---|---|
| Tickets opened | 48 | 48 |
| Resolved | 37 | 37 |
| Escalated | 5 | 5 |
| Average first response | 12.4 | 12.4 |
| Response target | 15.0 | 15 |
| Resolved plus escalated | 42 | 42 |
The single response scored 6/6. It used 1,359 prompt tokens and 145 completion tokens, took 6.595 seconds wall time and had a calculated cost of $0.006252. Numeric JSON represents 15.0 as 15; they are the same number under the frozen oracle.
Track 4: retrieval from 32K to 950K prompt tokens
Each deterministic corpus contained four authoritative target facts at approximately 10%, 33%, 67% and 90%, plus close records explicitly marked as obsolete drafts. One strict-schema request asked for all four facts.
| Band label | API-reported prompt tokens | Correct facts | Completion tokens | Wall time | Calculated cost |
|---|---|---|---|---|---|
| 32K | 32,154 | 4/4 | 201 | 8.747 s | $0.099477 |
| 128K | 128,145 | 4/4 | 292 | 11.453 s | $0.388815 |
| 256K | 256,160 | 4/4 | 211 | 26.740 s | $0.771645 |
| 512K | 512,150 | 4/4 | 420 | 61.073 s | $1.539294 |
| 950K | 950,149 | 4/4 | 268 | 76.219 s | $2.854467 |
All twenty field checks matched the fixed answers exactly. The 950K request plus its 2,048-token output allowance stayed below the model’s returned 1,048,576-token context length.
This result demonstrates successful retrieval for four deliberately signposted facts in these generated corpora. It does not demonstrate equivalent recall in dense legal files, source code, noisy OCR, conversations or adversarial documents. A request being accepted near the limit is also separate from answering it correctly; we measured both.
Token use and calculated cost
The formula separates cache misses, cache hits and output:
((prompt_tokens - cached_tokens) × $3
+ cached_tokens × $0.30
+ completion_tokens × $15) / 1,000,000| Track | Prompt tokens | Cached tokens | Completion tokens | Calculated cost |
|---|---|---|---|---|
| Tool calls | 5,001 | 2,560 | 915 | $0.021816 |
| Code repair | 767 | 0 | 222 | $0.005631 |
| Vision | 1,359 | 0 | 145 | $0.006252 |
| Long context | 1,878,758 | 1,280 | 1,392 | $5.653698 |
| Total | 1,885,885 | 3,840 | 2,674 | $5.687397 |
The rates were checked immediately before execution: $3.00 per million uncached input tokens, $0.30 per million cached input tokens and $15.00 per million output tokens. Provider prices exclude applicable taxes. The API balance endpoint showed no deduction immediately after completion. A read-only check 13 minutes 41 seconds later showed a $5.68739 balance deduction, only $0.000007 below the $5.687397 usage-derived calculation. That match is a balance observation, not a tax invoice.
Download the reproducibility pack
The public ZIP contains 83 files, includes a SHA-256 file manifest and passed ZIP-integrity and API-key-pattern checks before upload. It excludes the credential, authorization headers, unrelated account data and all returned reasoning_content. Each response records only that reasoning was redacted and its character count.
The archive retains:
- protocol v1.0 and the frozen v1.1 Pilot amendment;
- complete tool schemas and eight oracles;
- source SVG, rendered PNG and image hashes;
- the broken code fixture, model diff, patched file and test output;
- five generated context corpora with deterministic seed and hashes;
- redacted request records and sanitized streaming events;
- model, usage, timing and cost ledgers;
- the sanitized delayed balance-deduction observation;
- independent audit output and a file-level SHA-256 manifest.
What passed — and what remains untested
Supported by this pilot
- The authenticated
/v1/modelsresponse listedkimi-k3with a 1,048,576-token context length. - All 15 frozen requests completed without a recorded API, transport or rate-limit error.
- The exact tool, code, vision and context checks all passed once.
- The runner’s calculated total remained under the approved hard cap.
- The later API balance deduction matched the usage-derived total within $0.000007.
Not supported by this pilot
- Reliability across repeated stochastic runs.
- A comparative claim against ChatGPT, Claude, Gemini or another Kimi model.
- General coding, visual understanding or long-document accuracy.
- A guaranteed latency level; the network region was not independently verified.
- A provider tax invoice or accounting statement; the evidence is a balance-endpoint observation.
- Web search performance or official-tool safety.
Method limitations
The sample is deliberately small. Tool prompts are synthetic, the code defect is simple and visible tests disclose expected behavior. The dashboard is clean and high contrast. Context targets are short and labelled as authoritative while distractors are labelled obsolete. Those choices make the run reproducible, but they are easier to audit than many real workloads.
Kimi’s benchmarking guidance recommends far larger samples for claims about variance and benchmark performance. We therefore call this a real-world reproducible pilot, not a leaderboard benchmark. Our broader rules for preregistration, retained evidence, corrections and constrained conclusions are in the testing methodology and Sources and Corrections policy.
Frequently asked questions
Did KI AI Team really call Kimi K3?
Yes. Fifteen billable Chat Completions calls returned the exact model field kimi-k3. The retained response, usage and request records were scanned for secrets before the public bundle was prepared.
Did Kimi K3 pass every test?
It passed every exact check in this single pass: 8/8 tool cases, 5/5 code tests, 6/6 visual fields and 4/4 facts in each of five context bands. That is not the same as proving reliability over repetitions or broader tasks.
Was the 950K request actually close to one million tokens?
The official estimator predicted 950,148 prompt tokens and the paid response reported 950,149. With a 2,048-token completion allowance, it remained below the returned 1,048,576-token context length.
Did automatic context caching reduce the cost?
Only 3,840 prompt tokens were reported as cached across all calls. Most of the 1.886M prompt tokens were therefore treated as cache misses in the usage-derived calculation. This run was designed around fresh independent requests, not cache optimization.
Did the calculated cost match the account balance?
Yes, within the endpoint’s displayed precision. Returned usage produced a $5.687397 calculation. The balance endpoint showed a $5.68739 deduction 13 minutes 41 seconds after completion, a $0.000007 difference. We report this as a balance observation, not a tax invoice.
Why did you remove the reasoning text from the evidence pack?
The evaluation scores final answers, tool arguments and executable effects; it does not need private chain-of-thought. We removed reasoning_content, kept a redaction marker and retained the final answer, timing, model and usage fields.
Does this replace the Kimi K3 model profile?
No. The Kimi K3 model profile owns specifications, architecture, access and vendor benchmarks. This URL owns the independent protocol, observations, raw evidence and limits. Developers can also use our Kimi API guide, API pricing guide and cost calculator.
Official sources and update policy
Run-day product controls were checked against Kimi’s K3 quickstart, Chat Completions reference, model-list endpoint, token estimator, balance endpoint, Tool Choice guide, Vision guide and context-caching guide.
We will preserve this single-pass result and disclose later repetitions as a new dated phase rather than rewriting the original record. Any correction to a fixture, score, rate or billing observation will be logged with the reason and affected evidence files.