feat(llama.cpp): add llama.cpp comprehensive recipes
Covers building, installing, running models with llama-server, and complete uninstallation of llama.cpp.
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
# Run Models on llama.cpp
|
||||
|
||||
This recipe runs a model with `llama-server` inside a persistent `tmux` session, downloading it directly from Hugging Face.
|
||||
|
||||
## Install tmux
|
||||
|
||||
```bash
|
||||
sudo apt install -y tmux
|
||||
```
|
||||
|
||||
## Start a tmux Session
|
||||
|
||||
```bash
|
||||
tmux new -s llama
|
||||
```
|
||||
|
||||
## Run the Model Server
|
||||
|
||||
Replace:
|
||||
|
||||
- `<REPO>` with the Hugging Face model repo (e.g. `deepreinforce-ai/Ornith-1.0-9B-GGUF`)
|
||||
- `<HOST>` by default `0.0.0.0` hear all network interfaces
|
||||
- `<PORT>` with the port to serve on (e.g. `8000`)
|
||||
- `<CONTEXT_SIZE>` with the context window size (e.g. `262144`)
|
||||
|
||||
```bash
|
||||
llama-server -hf <REPO> --host <HOST> --port <PORT> -c <CONTEXT_SIZE>
|
||||
```
|
||||
|
||||
## Detach from the Session
|
||||
|
||||
Press:
|
||||
|
||||
```text
|
||||
Ctrl+B, D
|
||||
```
|
||||
|
||||
## List tmux Sessions
|
||||
|
||||
```bash
|
||||
tmux ls
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```text
|
||||
llama: 1 windows (created ...)
|
||||
```
|
||||
|
||||
## Reattach to the Session
|
||||
|
||||
```bash
|
||||
tmux attach -t llama
|
||||
```
|
||||
|
||||
## Verify Server is Running
|
||||
|
||||
Replace `<PORT>` with the configured port.
|
||||
|
||||
```bash
|
||||
curl http://localhost:<PORT>/health
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{"status":"ok"}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port already in use
|
||||
|
||||
Verify that no other process is bound to the port.
|
||||
|
||||
```bash
|
||||
sudo ss -tulpn | grep <PORT>
|
||||
```
|
||||
|
||||
### Session not found
|
||||
|
||||
List active sessions and verify the name.
|
||||
|
||||
```bash
|
||||
tmux ls
|
||||
```
|
||||
|
||||
### Server not responding
|
||||
|
||||
Reattach to the session and check the server logs.
|
||||
|
||||
```bash
|
||||
tmux attach -t llama
|
||||
```
|
||||
Reference in New Issue
Block a user