Context, output limits and memory: what your local AI actually needs

From our desk: a model’s context window, an application’s response limit and a project’s saved memory solve different problems. Confusing them can make a local assistant appear to forget, stop too soon or run out of space even when its model advertises a large window.

The space shared by a request

Context is the material the model can use during a request: instructions, conversation history, tool results and the answer being generated. The exact accounting depends on the model’s tokenizer and runtime. A server may allocate less context per request or slot than the model’s advertised maximum, and the available memory determines what you can actually run.

An output limit is a separate control over how much the model may generate in one response. An application can impose a lower limit than the model or server allows. Raising it may help a long coding answer, but leaves less room for the input and can increase latency or memory pressure. See the llama.cpp server documentation for its current runtime options.

Memory and cache are different

Prefix or KV caching can avoid recalculating reusable input under supported conditions. It is an optimization, not a permanent memory of every conversation. A saved project note can preserve selected facts across sessions, but it should be reviewed: stale or incorrect notes can mislead future work.

When a conversation grows, a client can summarize or start a new session with carefully selected context. That can help preserve headroom, but summaries may omit important details. If an answer stops because it reached a length limit, an explicit continuation can help; tool calls and structured JSON need special handling because stitching outputs together can corrupt them.

Our advice: use realistic output budgets, preserve important decisions deliberately, and make context resets visible enough to understand. Test with your actual project rather than assuming a bigger advertised number means reliable recall. Ai2’s StateBench is one example of research into this distinction.