Module tokens 

Deterministic approximate tokenizer.

Used for budgets, token-velocity tracking, and compression ratios. This is an approximation (documented, deliberate): exact counts differ per provider/model and arrive with responses; we record those separately when present. Properties guaranteed (and property-tested):

• deterministic;
• monotone: appending text never lowers the count;
• Unicode-safe (multi-byte chars never split or panic);
• zero for the empty string.

Heuristic: ASCII words contribute ceil(len/4) tokens (the ~4 chars/token BPE rule of thumb), each punctuation/symbol char is one token, and each CJK or other non-ASCII alphabetic char is one token.

Functions

approx_tokens

Approximate token count for text.

approx_tokens_json

Approximate token count for a serialized JSON value.