JotaOdiceu
f568e43362
Covers building, installing, running models with llama-server, and complete uninstallation of llama.cpp.
1.3 KiB
1.3 KiB
Run Models on llama.cpp
This recipe runs a model with llama-server inside a persistent tmux session, downloading it directly from Hugging Face.
Install tmux
sudo apt install -y tmux
Start a tmux Session
tmux new -s llama
Run the Model Server
Replace:
<REPO>with the Hugging Face model repo (e.g.deepreinforce-ai/Ornith-1.0-9B-GGUF)<HOST>by default0.0.0.0hear all network interfaces<PORT>with the port to serve on (e.g.8000)<CONTEXT_SIZE>with the context window size (e.g.262144)
llama-server -hf <REPO> --host <HOST> --port <PORT> -c <CONTEXT_SIZE>
Detach from the Session
Press:
Ctrl+B, D
List tmux Sessions
tmux ls
Expected output:
llama: 1 windows (created ...)
Reattach to the Session
tmux attach -t llama
Verify Server is Running
Replace <PORT> with the configured port.
curl http://localhost:<PORT>/health
Expected output:
{"status":"ok"}
Troubleshooting
Port already in use
Verify that no other process is bound to the port.
sudo ss -tulpn | grep <PORT>
Session not found
List active sessions and verify the name.
tmux ls
Server not responding
Reattach to the session and check the server logs.
tmux attach -t llama