AI Memory & Agent Glossary

Context engineering is the discipline of selecting, structuring, ordering, and updating the information an LLM receives while completing a task. It applies when model behavior depends on more than one instruction, including retrieved documents, tool results, conversation state, examples, and system constraints that must fit within a finite context window.

Agentic engineering is the software engineering discipline of building systems in which a language model can choose actions, use tools, inspect results, and continue toward a bounded goal. The term applies when model output changes program state or controls a multi-step workflow, not when an application merely sends a prompt and displays the response.

Flow engineering is the practice of decomposing an LLM task into an explicit sequence of generation, execution, checking, and revision steps. It applies when intermediate artifacts can be tested or inspected and a controlled workflow is more reliable than one model call; it does not require the model to choose its own route.

Test-time compute is the computation spent after a trained model receives an input in order to improve or select its answer. It applies when an inference system can spend extra tokens, samples, search steps, verification passes, or tool calls on harder requests; it is distinct from the compute used to train or fine-tune model weights.

Model Context Protocol, or MCP, is an open protocol for connecting AI applications to servers that expose tools, resources, and reusable prompts. It applies when a host needs a standard way to discover and invoke capabilities across process or network boundaries; MCP does not grant trust, authorization, or safe execution by itself.

Semantic caching is an application cache that reuses a stored result when a new request is sufficiently similar in meaning to an earlier request. It applies when semantically equivalent inputs should produce the same valid response; it is unsafe when small differences in identity, time, permissions, or wording can change the correct answer.

Matryoshka Representation Learning is a training method that makes prefixes of one embedding vector useful at several dimensionalities. It applies when a model was trained with nested representation objectives and a system needs flexible storage or multi-stage retrieval; arbitrary truncation of a conventional embedding is not Matryoshka representation learning.

Cross-encoder reranking is a search stage in which one model reads a query and a candidate document together and assigns that pair a relevance score. It applies after a faster retriever has produced a manageable candidate set, especially when relevance depends on exact wording, negation, or relationships that independent embeddings may miss.

Late chunking is an embedding method that encodes a long document before pooling token representations into chunk vectors. It applies when chunk meanings depend on context elsewhere in the same document and the embedding model can process the required span; it differs from embedding each isolated chunk after splitting the text.

Tool calling is an LLM application pattern in which a model selects a host-provided operation and returns structured arguments for that operation. It applies when the model must request data or an action outside its generated text; the application, not the model, remains responsible for validating the arguments, executing the tool, and returning the result.

PagedAttention is a KV-cache memory-management technique introduced by the vLLM project that stores cache blocks in non-contiguous physical memory through an indirection table. It applies to batched autoregressive model serving where variable-length sequences cause fragmentation and duplicated cache data; it does not change the model’s attention semantics or make the context window larger.

KV-cache eviction is the removal of selected key-value states retained from earlier tokens during autoregressive inference. It applies when a serving system cannot keep the full cache for a long sequence or many concurrent requests; eviction trades memory savings for a risk that later tokens can no longer attend to information the policy removed.

Semantic chunking is a document-segmentation method that places boundaries where meaning or topic changes rather than at a fixed character or token interval. It applies when coherent sections vary in length and retrieval quality depends on keeping related statements together; it is unnecessary when the source already provides reliable, task-sized structural units.

Hypothetical Document Embeddings, or HyDE, is a zero-shot retrieval method that asks a language model to draft a hypothetical answer-like document, embeds that draft, and uses its vector to retrieve real documents. It applies when a short query and relevant passages occupy different language or detail levels; the generated document is a search representation, not evidence.

Self-querying retrieval is a search pattern in which a model converts a natural-language request into a semantic query plus structured filters for a retrieval system. It applies when the collection has trustworthy metadata and users express constraints such as date, product, author, or region in ordinary language; it does not create metadata the index lacks.

A bi-encoder is a retrieval architecture that encodes a query and each candidate document independently into vectors that can be compared with a distance or similarity function. It applies when a system must search many pre-encoded candidates quickly; it is less suitable when relevance depends on fine-grained interaction between every query token and document token.

Product quantization is a lossy vector-compression method that splits a vector into sub-vectors and represents each sub-vector by the identifier of a nearby learned centroid. It applies when storing or scanning full-precision embeddings is too expensive and approximate distance estimates are acceptable; the codebooks must be trained on representative vectors.

Hierarchical Navigable Small World, or HNSW, is a graph-based approximate nearest-neighbor index for finding vectors close to a query vector. It applies when exact comparison against every stored vector is too costly and the system can trade some recall for faster search, additional index memory, and slower index construction.

ReAct prompting is an agent pattern that interleaves reasoning steps with actions and observations from an external environment. It applies when a task cannot be completed from the initial prompt alone and the model must use new evidence to choose later steps; it does not describe every tool-using assistant or every hidden reasoning process.

Plan-and-Solve is a prompting framework that asks a language model to create a multi-step plan before carrying out the plan. It applies to tasks where omitted intermediate steps are a major source of error and the work can be decomposed in advance; it is weaker when new observations should continually change the route.

Speculative decoding is an inference technique in which a faster draft process proposes several tokens and the target model verifies those proposals in parallel while preserving the target model’s output distribution. It applies when draft tokens are accepted often enough to offset proposal and verification overhead; it changes generation speed, not the model’s knowledge or reasoning ability.

An agentic router is a decision layer that directs an incoming request to a model, agent, tool, workflow, or human path. It applies when destinations have meaningfully different capabilities, costs, permissions, or failure modes; it adds needless complexity when one destination can handle the full request set under the same policy.

JSON mode is a model setting intended to produce syntactically valid JSON, while structured outputs constrain a response to a supplied schema. The distinction applies when an application needs machine-readable model output: JSON mode addresses parsing, whereas structured outputs also address required fields, types, enums, and nesting defined by that schema.