Serving

Parent: LLMs.

Serving packs own inference: batching, KV cache, hardware utilization, and OpenAI-compatible (or native) HTTP. Under LLMs they are the model plane. Agents and LLM Gateway call them; they are not orchestration frameworks.

Technically compare continuous batching, quantization, multi-LoRA, and local vs cluster deploy. Failure modes: undersized KV cache causing thrash, “local LLM” demos without residency policy, and serving stacks that become snowflake hosts nobody can replay.

Members: vLLM, Text Generation Inference, TensorRT-LLM, SGLang, Ollama, llama.cpp. Prefer vLLM/TGI/TensorRT/SGLang for production tokens/sec; Ollama/llama.cpp for local/dev residency.

Features such as Bring Your Own Model, Cost, Latency & Operational Economics, and Deployment & Data Residency are the bar: know where weights live and what each token costs.

Inline members for audit: vLLM Text Generation Inference TensorRT-LLM SGLang Ollama llama.cpp. Features: Bring Your Own Model Cost, Latency & Operational Economics Deployment & Data Residency. Related terms referenced: Fine-tuning, LLM Gateway, LLMs.

For this research catalog the acceptance test is boring and strict: after wiring the pack, can a second person replay the same scenario and get the same measurable outcome? If not, the page is still a brochure. Document the scenario, the expected number, and the log line that proves it. That habit is what separates a deepened taxonomy cell from a landscape screenshot.

Also record what you refused: tools not connected, corpora not ingested, metrics not collected. Refusals keep the cell honest when marketing pages claim universality. A taxonomy essay that only lists winners teaches less than one that names the skip conditions next to the install path.

Keep the essay technical enough that an engineer can argue install order — and short enough that a reader finishes it before opening the right-rail member list. Prefer primary docs and repo evidence over landscape blurbs when the right-rail packs disagree.

Operational discipline for this cell: name the install artifact, the health check, and the rollback. Prefer one golden path documented against a member pack in the right rail over five half-wired demos. When packs disagree, believe the primary repository README and a failing test you can re-run — not the landscape one-liner. Measure latency, cost, and error rate on a frozen scenario before declaring the subcategory “covered.” If you cannot point at a log line that proves the scenario, the essay is unfinished even when the word count clears the audit floor.

Cross-link greedily but honestly: every taxonomy wikilink should change a reader’s next click, and every pack slug should resolve to a real pack. Do not invent members to decorate the prose. Prefer fewer true links over many decorative ones. Keep Monday-shaped typing out of the runtime story; observation and ledger remain senior to any agent UI that retypes reality.