Jul 27, 2026 23:00:00

What is the mechanism behind 'Prompt Cash,' which can reduce the price of AI tokens to one-tenth of their original value?

Large-scale language models (LLMs) such as OpenAI's GPT series and Anthropic's Claude employ a ' prompt cache ' that reuses input token processing when the same prompt is used repeatedly, and can sometimes be used at about one-tenth the cost of regular input tokens. ngrok , a developer service provider, explains the prompt cache mechanism in detail.

Prompt caching: 10x cheaper LLM tokens, but how? | ngrok blog
https://ngrok.com/blog/prompt-caching

Anthropic explains that prompt caching 'can reduce latency by up to 85% for long prompts.' The following graph compares the response speed of ngrok with and without prompt caching, showing that in both cases—OpenAI's GPT-5 (left) and Anthropic's Sonnet 4.5 (right)—using prompt caching significantly reduced the 'time to first token acquisition.'

Anthropic doesn't specifically explain what these 'cached tokens' are. Intuitively, one might think they're like a web browser cache, 'remembering a question you asked once and being able to quickly reuse it the next time,' but since different responses are often returned even when the same prompt is sent repeatedly, it's not actually 'storing and retrieving past conversations.'

Therefore, ngrok is thoroughly investigating the mechanisms of LLM to uncover what specific caches data providers are using, what their purposes are, and how this enables the use of LLM faster and cheaper.

First, LLM doesn't understand the input text directly; instead, it divides the text into units called 'tokens.' Then, it converts each token into a numerical vector, and a deep learning model called

Transformer calculates how each word relates to the others. ngrok explains the 'huge graph consisting of billions of meticulously placed operations' that converts input numbers into output numbers by dividing it into four parts: 'Tokenizer (token generation),' ' Embedding ,' 'Transformer,' and 'Output.'

The tokenizer mechanism differs for each AI model, but the basic mechanism involves splitting a prompt like 'Check out ngrok.ai' into 'Check/out/ng/rok/.ai' and assigning an integer ID called a token to each chunk. Because it doesn't understand words directly but rather breaks them down into multiple chunks for understanding, it sometimes faces a problem specific to LLMs, such as being unable to answer simple questions like 'How many 'r' letters are in the word 'strawberry'?'

What is the 'strawberry problem' plaguing large-scale language models like GPT-4 and Claude? - GIGAZINE

The mechanism for understanding tokenized prompts is called 'embedding.' Embedding is a representation learning technique that maps complex high-dimensional data to a low-dimensional vector space of numerical vectors. The more dimensions you give to the embedding, the more dimensions you need to compare to correctly understand the sentence. ngrok illustrates this with the following diagram. The left side shows eight shapes placed in a one-dimensional space, which is cluttered and difficult to understand. On the other hand, by increasing the space to two or three dimensions, the eight shapes can be clearly distinguished.

The LLM's mechanism involves using a Transformer to calculate the weights to assign to each input token after correctly understanding them, and then presenting a more appropriate answer based on the text's comprehension.

Each node can be thought of as a function that takes some input and produces some output. The input is fed to the LLM in a loop until a specific output value signals a stop. The prompt cache is a mechanism for storing and reusing the results of this computation.

For example, chatbots often receive long, recurring inputs such as system prompts, terms of service, and tool definitions. Normally, these would be calculated from scratch for each request, but prompt caching saves the results of a previous calculation, allowing subsequent requests to be made without recalculating the same parts. This reduces the time it takes to start a response and significantly lowers API costs.

The following is a table showing examples of action weights for the prompt 'Mary had a little (the beginning of Mary Had a Little Lamb).' Since there is only one subject, 'Mary,' 'Mary' contributes 100% to the verb 'had.' We can see that 'Mary' contributes 79% to the article 'a' and 'had' contributes 21%, and so on. In either case, LLM considers 'Mary' to be the most important word in this sentence and infers the following word 'lamb' from it.

If we extend the sentence 'Mary had a little' by adding words like 'lamb,' 'whose,' and 'flee,' the calculation shows that 'Mary' contributes significantly to 'lamb,' which follows the lyrics of 'Mary Had a Little,' but not to 'whose' and 'flee,' which deviate from the lyrics. In this case, even though the first part, 'Mary had a little,' remains unchanged, the calculation is recalculated every time a word is added. This is how the 'inference loop' of LLM works. By shortening this part with a prompt cache, tokens can be saved significantly.

While a web browser cache stores the completed web page, a prompt cache stores the intermediate state calculated within the LLM. Because only the common intermediate state is cached, rather than the generated answer, sending the same prompt can potentially produce different answers. The prompt cache is used for requests that include the same prompt's beginning; in English, this means the calculation of prefixes like 'What is...' is omitted, and only the newly added part needs to be processed.

The prompt cache stores internal data called 'Key' and 'Value' that the Transformer generates for each token. The Transformer's 'Self-Attention' performs extensive calculations using the Key, Value, and Query, but once calculated, the Key and Value can be reused as long as the input remains unchanged. Therefore, the prompt cache is sometimes called the 'KV cache.'

According to ngrok, OpenAI and Anthropic employ completely different caching approaches. OpenAI manages its cache entirely automatically, utilizing cached data whenever possible. In ngrok's experiments, the cache utilization rate (hit rate) when sending the same request again immediately afterward was only about 50%, suggesting that this approach could lead to inconsistent response times. Anthropic, on the other hand, gives users more control, allowing developers to specify when to create caches and for how long to retain them. While this feature incurs an additional fee, ngrok's experiments showed that caching was utilized 100% of the time when specified, resulting in stable response times even when dealing with long contexts. Therefore, OpenAI might be more suitable for those who prefer automation, while Anthropic's approach might be better suited for those who require more granular control.