DatattentionA small local LLM setup for a team
One machine, one model, one proxy, and the notes that keep the setup from drifting.
A team that wants a local model rarely needs a platform. One GPU machine, one daemon, and a short page that says who can call it will outlast a weekend of orchestration. Pick the model for the job you actually have, usually a mid-size instruct model with a context window you have measured.
Decide the access path before you pull weights. People on the same desk can use SSH tunnels. People in another office need a hostname and TLS. Write the hostname in DNS you control, and point it only at the proxy, not at the model process. The model process should listen on localhost or on a private interface.
Keep the operational facts on one card: disk path of the weights, the quant, the context length you tested, the port, and the person who restarts it. When the card is missing, the next person will start a second copy on a second port and you will have two slightly different services.
Measure a prompt you care about and save the tokens per second and the memory reading. Those two numbers tell you whether a later driver update helped. A demo sentence does not.
Review the setup when someone new joins. The usual drift is an open port left over from a demo, or a token pasted into a shared document. Close the port or rotate the token the same day.
On this host
The card for this host is short. The hostname is datattention.com (194.238.25.127). Chat clients should use port 8000 at /v1/chat/completions. Nothing on that port checks an API key. The same JSON works on port 8080 if a client is already pointed there.
import json, urllib.request
req = urllib.request.Request(
"http://datattention.com:8000/v1/chat/completions",
data=json.dumps({
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello"}],
}).encode(),
headers={"Content-Type": "application/json"},
)
print(urllib.request.urlopen(req, timeout=30).read().decode())The full list of ports and paths is on the endpoints page.