DatattentionQuants and context windows, written down
Pick a quant and a context length, then record the numbers you measured.
A quant is a compression choice. Smaller quants fit on smaller GPUs and usually lose some accuracy. The right one is the smallest that still passes the checks you care about, not the one a model card calls “recommended” for a different machine.
Context length is also a choice. The architecture may allow 128k tokens. Your latency and memory budget may allow 8k. Publish the number you tested. Callers who assume the advertised maximum will time out and blame the model.
Measure with a prompt of a known size and a fixed max-token setting. Record tokens per second, peak memory, and whether the answer still followed the format you asked for. One table beats a paragraph of impressions.
When you change drivers, quants, or context, run the same table again. Keep the old row. Regressions in local inference are usually boring: a new wheel, a new flag, a context length copied from a README.
Write the final pair, quant and context, next to the model file name. Future you will not remember which folder was the good one.
On this host
The Ollama-style process on this host accepts the model string you send. Port 11434, no API key. Put the name in the JSON. The same idea works on the OpenAI-compatible ports if the client speaks that shape instead.
curl http://datattention.com:11434/api/generate \
-H 'Content-Type: application/json' \
-d '{"model":"llama3","prompt":"Hello","stream":false}'The full list of ports and paths is on the endpoints page.