All posts
Why the knowledge base has a hard character cap
The cap and prompt caching are the same decision. Chunking would vary the prompt on every message and cost us the cache, which at these sizes is worth more than retrieval.
Draft
Back to all posts
The full post is being written. The summary above is the argument in short.