Vision Model (VLM) Setup
The VLM is what lets Aegis AI see and understand camera frames. Every time a clip is recorded, the VLM analyzes it and generates a text description. This powers alerts, timeline search, and the agent's situational awareness.
Open VLM configuration from ⚙️ Settings → Persona → VLM.
Built-in (Local Models)
Runs vision models locally using the bundled llama-server. No API key or internet needed after the model is downloaded.
Staff Picks
Curated models tested with Aegis AI, sorted by size:
| Model | Size | Video Support | Notes |
|---|---|---|---|
| SmolVLM2 256M Video | 0.4 GB | ✓ | Smallest video model, runs on almost any hardware |
| SmolVLM2 500M Video | 0.7 GB | ✓ | Lightweight, good for quick testing |
| nanoLLaVA | 1.3 GB | — | Ultra-compact 430M parameter model |
| InternVL3 1B | 1.5 GB | ✓ | Fast and efficient with excellent vision |
| LFM2.5-VL 1.6B | 1.6 GB | ✓ | Top pick — Liquid Foundation Model |
| SmolVLM2 2.2B | 1.7 GB | ✓ | Compact, great for resource-constrained setups |
| Qwen3-VL 2B Instruct | 1.8 GB | — | Strong reasoning and instruction-following |
| Qwen2.5-VL 3B | 2.1 GB | — | Balanced size and performance |
| LLaVA Phi-3 Mini | 2.7 GB | ✓ | Microsoft Phi-3 backbone |
| MiniCPM-V 2.6 | 2.8 GB | ✓ | Strong OCR, excellent for security cameras |
| Moondream2 | 3.5 GB | ✓ | Latest Moondream with improved vision |
| Gemma 3 4B | 3.7 GB | ✓ | Google Gemma 3, great general-purpose |
| LLaVA 1.6 Mistral 7B | 5.0 GB | — | Mistral 7B backbone, high quality |
Click Download on any staff pick card. The download progress bar shows percentage and speed.
Downloaded Tab
Lists all downloaded GGUF vision models with:
- Model name, quantization, size
- Load / Eject / Delete buttons (same behavior as LLM)
Search Tab
Search HuggingFace for additional vision-language models. Same filters as the LLM Search tab (sort, size presets).
System Tab
Shows llama-server status:
- Running / Stopped indicator with PID
- Currently loaded model ID
- Server health check (polled every 3 seconds)
You can start/stop the server and select which downloaded model to serve.
Authentication Tab
Manage your HuggingFace token for accessing gated model repositories.
API Providers
Cloud Providers
Connect to a cloud vision API. Supported providers:
| Provider | Models | API Key Source |
|---|---|---|
| OpenAI | gpt-4o-mini, gpt-4o, gpt-4-turbo | platform.openai.com |
| Anthropic | claude-3.5-sonnet, claude-3-haiku | console.anthropic.com |
gemini-1.5-flash, gemini-1.5-pro | aistudio.google.com |
Setup:
- Select a provider tab.
- Enter your API Key. Click Save Key.
- Select a model from the dropdown.
- Click Select to activate and validate the connection.
Cost Estimation
When using a cloud provider, Aegis AI displays a real-time cost estimate:
| Metric | Description |
|---|---|
| Per Clip | Cost to analyze a single clip (e.g. $0.0043) |
| Monthly | Projected monthly spend based on your usage pattern |
| Clips/Day | Average clips recorded per day |
| Total Clips | Total clips analyzed in the last 30 days |
| Projected Monthly | Estimated clips next month |
If you have fewer than a few days of data, Aegis shows a default estimate based on 50 clips/day.
An expandable Usage Trend chart shows daily clip counts and cost over time.
External (OpenAI-Compatible)
Point Aegis to any server with an OpenAI-compatible /v1/chat/completions endpoint that supports vision.
- Enter the Endpoint URL (default:
http://localhost:1234/v1). - Click Load Models to fetch the model list.
- Select a model.
- Click Select — the connection is auto-validated, and latency is displayed (e.g.
Connected (142ms)).
Compatible with: LM Studio, vLLM, LocalAI, text-generation-webui, and similar.
How the VLM Is Used
- A camera records a clip (triggered by motion, schedule, or a skill).
- Aegis extracts a frame or set of frames from the clip.
- The frame(s) are sent to the active VLM with the configured analysis prompt.
- The VLM returns a text description (e.g. "Person walking on driveway carrying a package").
- This description is stored with the clip and used for alerts, timeline search, and agent context.
Troubleshooting
| Issue | Solution |
|---|---|
| "LLaMA Server: Offline" in System tab | Click Start, or check that a model is downloaded and selected. |
| Cloud API returns errors | Verify the API key. Check the provider's status page. |
| Analysis too slow with local model | Use a smaller model (SmolVLM2 256M or 500M). Reduce the frame rate in Video Analysis settings. |
| External endpoint returns empty model list | Verify the server is running and the URL includes /v1. |
| Cost estimate shows "$0.00" | Wait for at least 1 day of clip data, or select a cloud provider to see the default estimate. |