Language Model (LLM) Setup
The LLM powers Aegis AI's ability to chat, analyze events, generate alerts, and respond to your commands. You can use a built-in local model or connect to an API provider.
Open the LLM configuration from ⚙️ Settings → Persona → LLM.
Built-in (Local Models)
The Built-in tab runs models entirely on your machine using the bundled llama-server engine. No API key or internet connection is needed.
Downloaded Tab
Lists every GGUF model currently on disk. Each card shows:
| Field | Description |
|---|---|
| Name | HuggingFace repo ID (e.g. bartowski/Llama-3.1-8B-GGUF) |
| Quantization | GGUF quant level — Q4_K_M, Q5_K_S, Q8_0, etc. |
| Size | File size in GB |
| Type | Text (language-only) or Vision + Text (multimodal with mmproj) |
Actions per model:
- Load — starts
llama-serverwith this model. The model becomes the active LLM for all Aegis AI features. Only one model can be loaded at a time. - Eject — stops the running model and frees memory.
- Delete — removes the GGUF file from disk. You are asked to confirm before deletion.
When a model is loaded, a green "Active" indicator appears in the header showing the model ID.
Search Tab
Search HuggingFace for GGUF-format language models.
Filters:
| Filter | Options |
|---|---|
| Sort | Trending, Most Downloads, Most Likes, Recently Updated |
| Size preset | ≤1B, ≤3B, ≤7B, ≤13B, ≤32B, All |
Search results show model cards with download count, likes, and parameter count. Click a card to view available GGUF files in the repo, then click Download on a specific quantization.
Downloads are tracked with a progress bar showing percentage, speed (MB/s), and bytes transferred. When a model download includes a matching mmproj file (for vision support), both files are downloaded together.
Results load with infinite scroll — scroll to the bottom to load the next page of 20 results.
API Providers
Switch to the API Providers tab to use a cloud or third-party LLM instead of a local model.
OpenAI
Connect to OpenAI's API (GPT-5.3, GPT-5.2 Pro, Codex, etc.).
- Enter your API Key in the key field. The key auto-saves when you click away.
- Optionally change the Base URL (default:
https://api.openai.com/v1). - Select a Model from the dropdown (populated from the API).
- Click Select to activate and validate the connection.
The system will test the connection and show:
- Connected (green) if the key and endpoint work
- Failed (red) with an error message if the connection fails
OpenAI-Compatible
Connect to any server that exposes an OpenAI-compatible /v1/chat/completions endpoint. Compatible with:
- LM Studio (default endpoint:
http://localhost:1234/v1) - vLLM
- LocalAI
- Ollama (with OpenAI compatibility layer)
- Any other OpenAI-compatible server
- Enter the Base URL (e.g.
http://localhost:1234/v1). - Click Load Models to fetch the list of available models from the server.
- Select a model from the dropdown.
- Click Select to activate.
No API key is needed for local servers.
How to Choose
| Scenario | Recommendation |
|---|---|
| No internet, full privacy | Built-in with a 3–7B model |
| Fast responses, best quality | OpenAI (GPT-5.x) |
| Already running LM Studio/vLLM | OpenAI-Compatible pointed at your server |
| Low-spec hardware (≤8GB RAM) | Built-in with ≤3B model, or use OpenAI API |
Troubleshooting
| Issue | Solution |
|---|---|
| "Failed to start LLM server" | Model file may be corrupted. Delete and re-download. |
| Model loads but responses are slow | Try a smaller quantization (Q4 instead of Q8) or a smaller model. |
| OpenAI key rejected | Verify the key at platform.openai.com. Check billing status. |
| Load Models returns empty list | Verify the server is running and the Base URL is correct (include /v1). |
| "No models found" in Downloaded tab | Switch to the Search tab and download a model first. |