DatattentionTensorFlow Serving REST, when you already have a saved model
The REST port is for predictions. Keep the model server off the public internet unless a proxy sits in front.
TensorFlow Serving is a model server for saved models. The REST port people remember is 8501. A client posts JSON to a path shaped like /v1/models/<name>:predict and gets a predictions array back. There is also a gRPC port, often 8500, which is easy to forget because browsers never call it.
Use it when you already have a SavedModel and you want a stable process, version labels, and a health check. Do not use it as a general chat API. Chat wants a different request shape. Forcing chat through a predict route makes both sides harder to operate.
Bind the serving process to a private interface. Put a proxy on the hostname humans use, with a request size limit and a timeout. A predict call with a huge tensor is a fine way to exhaust memory, and the default limits are not a policy.
Version the model in the path or in the model config, and keep the previous version loadable until the new one has answered a real request. “Available” in the status route means the process loaded the files. It does not mean the answers are acceptable.
Log the model name, the version, the status, and the latency. Log the raw tensor only when you have a reason and a retention limit. Prediction payloads can be as sensitive as the documents they came from.
On this host
TensorFlow Serving REST on this host is port 8501, with no API key. GET /v1/models/demo reports the model available, and a predict call is POST /v1/models/demo:predict. The gRPC port 8500 is not published here.
curl http://datattention.com:8501/v1/models/demo:predict \
-H 'Content-Type: application/json' \
-d '{"instances":[[1.0, 2.0]]}'The full list of ports and paths is on the endpoints page.