Tokens Worker Checking collector status

Architecture

One Worker, three bindings. Writes are events — submissions, plus one reported snapshot per subscription — so they drive everything downstream and there is no cron. Reads never queue behind writes, and never touch D1 at all in the steady state: the dashboard payload is precomposed into KV and served with an ETag so browsers revalidate instead of guessing.

Storage

Usage is stored at maximum granularity: one row per (device, date, client, model, provider) carrying input, output, cacheRead, cacheWrite, reasoning, messages, cost and the parser revision that produced it. /api/site is an aggregation of this matrix, and raw ids are kept here, so nothing canonicalization does is lossy. Sidecar tables: daily_activity (per-day active time), devices (name, CLI version, time metrics, MCP servers) and submissions (audit log, 30-day rolling window). All dates are calendar days in Asia/Shanghai.

Write path

The merge engine ports the official server’s semantics, so a self-hosted history behaves exactly like a hosted one:

Per-day, per-client replace

A submission replaces each (device, day, client) bucket it mentions; clients or days it does not mention are left untouched.

Regression guard

Within the same parser revision, a resubmit that would shrink a client’s day is ignored with a warning — local log cleanup can never erase stored history.

Parser revision floors

A client reporting with a schema version older than any revision already stored for it is rejected wholesale (stale-parser protection).

Authoritative coverage

tokens submit --replace sends full-coverage windows that replace a client authoritatively — including tombstoning days the new scan no longer contains.

Validation

The official mathematical-consistency checks run on every submission (day totals vs client sums, future dates, duplicates…). Failures return 400 with details.

Atomic set-based writes

Changed rows travel as JSON parameters expanded server-side (json_each), so even a full-history first upload is a handful of statements in one D1 batch — one transaction, no partial-write states, safely below D1’s per-invocation query limits.

Read API

Public, no auth — and deliberately one endpoint. A filterable aggregation API (stats, timeseries, breakdown, graph, meta, devices, submissions) used to sit here and nothing ever called it: the CLI computes those views locally, and the dashboard reads the precomposed payload. The matrix behind them is untouched in D1 at full fidelity — reachable through wrangler d1 execute, the R2 raw archives and the daily exports — so what went away is a public surface, not the data or any question you could ask it.

/api/site serves Cache-Control: no-cache with a strong ETag — browsers revalidate every load and get a ~0-byte 304 while the data is unchanged. Every breakdown row carries the full metric set: the five token categories, their sum, messages and cost. Model and provider ids are canonicalized before aggregation (claude-fable-5-thinking-max → claude-fable-5, while Qwen's real -plus/-max product tiers stay distinct; openai-codex → openai), which is what makes the per-day model slices agree with the model breakdown. Providers mean model vendors: a provider id that isn't a vendor claim is re-attributed by model name (qwen3.8-max → alibaba) — either the client's own gateway (zed.dev, opencode, opencode-go, opencode_go) or anything reported by a client whose models the user configures, where openai names an OpenAI-compatible endpoint rather than the vendor. Models the rules can't place stay under the reported id. D1 keeps the raw spellings, so none of it is lossy.

1. GET /api/site open ↗

   The one-request dashboard view: totals and breakdowns for five ranges (today, 7/30/90 days, all time), each client and provider row split into the models behind it, the daily series sliced by provider, client and model, and the device inventory. The body carries a schemaVersion the consumer validates before rendering. No filters — served from KV with its precomputed ETag, never waits on D1.

2. GET /api/health open ↗

   Liveness check.

Examples

# the whole dashboard, one request
curl "https://tokens.lkwplus.com/api/site"

# revalidate: ~0 bytes back while the data is unchanged
curl -H 'If-None-Match: "<etag>"' -i "https://tokens.lkwplus.com/api/site"

# ad-hoc questions go to the matrix directly, not to an endpoint
npx wrangler d1 execute tokens-usage --remote \
  --command "select model, sum(cost) c from daily_usage group by 1 order by c desc limit 10"

CLI endpoints

The write side implements the official server’s wire contract, so a stock CLI works with TOKENS_API_URL=https://tokens.lkwplus.com. Every CLI endpoint requires Authorization: Bearer $TOKENS_API_TOKEN. The browser OAuth device flow is not implemented — log in with tokens login --token instead.

1. POST /api/submit auth

   Full submission payload from tokens submit / serve / autosubmit. Responds with {success, submissionId, metrics, mode, warnings?}.

2. POST /api/quota/:plan auth

   One subscription’s rate-limit snapshot: codex takes the verbatim output of tokens codex status --json, claude takes Anthropic’s /api/oauth/usage response. Each is narrowed into the payload’s own shape — windows, reset times and unspent reset credits — and overwrites that plan’s key alone. Responds with {success, provider, capturedAt}.

3. DEL /api/settings/submitted-data auth

   Wipes all stored data across all three stores (tokens delete-submitted-data): the D1 tables, every R2 archive and backup, every reported quota plan, and the KV site payload — recomposed before the response, so success means the public dashboard is already clean.

4. GET /api/auth/token auth

   Token validation for tokens login --token. Responds with {user: {username}}.

Operations

• Deploys are Git-driven: every push to main builds and deploys through Workers Builds, whose build command runs the full gate — type checks plus the vitest suite (unit tests for the merge engine, validation and canonicalization; integration tests over real D1/KV/R2 bindings) — so a red check blocks the deploy. CI runs the same gate on every push. Manual wrangler deploy is an emergency escape hatch only.
• Backups are layered — D1 Time Travel covers the last 30 days, the daily R2 export covers forever, and the per-device raw payload can replay-rebuild the database after a schema change.
• Browser reads carry a static Access-Control-Allow-Origin: * (src/http.ts). Responses are cacheable, so headers must never depend on the request's Origin — a per-origin variant would poison the HTTP cache.
• Verify locally with npm run check (types, typecheck, vitest — the same gate CI and the deploy build run); exercise it against a real CLI with TOKENS_API_URL=http://localhost:8787 tokens submit.