JotaOdiceu
f568e43362
Covers building, installing, running models with llama-server, and complete uninstallation of llama.cpp.
95 lines
1.3 KiB
Markdown
95 lines
1.3 KiB
Markdown
# Run Models on llama.cpp
|
|
|
|
This recipe runs a model with `llama-server` inside a persistent `tmux` session, downloading it directly from Hugging Face.
|
|
|
|
## Install tmux
|
|
|
|
```bash
|
|
sudo apt install -y tmux
|
|
```
|
|
|
|
## Start a tmux Session
|
|
|
|
```bash
|
|
tmux new -s llama
|
|
```
|
|
|
|
## Run the Model Server
|
|
|
|
Replace:
|
|
|
|
- `<REPO>` with the Hugging Face model repo (e.g. `deepreinforce-ai/Ornith-1.0-9B-GGUF`)
|
|
- `<HOST>` by default `0.0.0.0` hear all network interfaces
|
|
- `<PORT>` with the port to serve on (e.g. `8000`)
|
|
- `<CONTEXT_SIZE>` with the context window size (e.g. `262144`)
|
|
|
|
```bash
|
|
llama-server -hf <REPO> --host <HOST> --port <PORT> -c <CONTEXT_SIZE>
|
|
```
|
|
|
|
## Detach from the Session
|
|
|
|
Press:
|
|
|
|
```text
|
|
Ctrl+B, D
|
|
```
|
|
|
|
## List tmux Sessions
|
|
|
|
```bash
|
|
tmux ls
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```text
|
|
llama: 1 windows (created ...)
|
|
```
|
|
|
|
## Reattach to the Session
|
|
|
|
```bash
|
|
tmux attach -t llama
|
|
```
|
|
|
|
## Verify Server is Running
|
|
|
|
Replace `<PORT>` with the configured port.
|
|
|
|
```bash
|
|
curl http://localhost:<PORT>/health
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```json
|
|
{"status":"ok"}
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Port already in use
|
|
|
|
Verify that no other process is bound to the port.
|
|
|
|
```bash
|
|
sudo ss -tulpn | grep <PORT>
|
|
```
|
|
|
|
### Session not found
|
|
|
|
List active sessions and verify the name.
|
|
|
|
```bash
|
|
tmux ls
|
|
```
|
|
|
|
### Server not responding
|
|
|
|
Reattach to the session and check the server logs.
|
|
|
|
```bash
|
|
tmux attach -t llama
|
|
```
|