What Are Skills
Skills are modular AI capabilities that extend what Aegis AI can do. Each skill is a self-contained package — typically a Python script plus a machine learning model — that adds a specific ability like detecting objects, estimating depth, or recognizing faces.
Think of skills like apps on your phone: Aegis AI is the operating system, and skills are the apps you install to do specific things.
How Skills Work
Each skill runs as an independent process alongside the main application. When a skill is running:
- It receives camera frames from Aegis AI
- Processes them using its own ML model
- Returns structured results (detections, classifications, measurements)
- Those results feed into alerts, timeline events, and the agent's understanding of your environment
Skills communicate with the main app through a standardized protocol, so they can be installed, started, stopped, and removed independently without affecting other skills or the core application.
What This Means in Practice
- Skills don't slow down the core app — they run as separate processes with their own memory
- You can mix and match — run object detection and depth estimation simultaneously
- Skills can be hot-swapped — stop one, start another, without restarting Aegis
- New skills can be added over time — as the community develops new capabilities, you install them like updating an app
Available Skills
Aegis AI's marketplace includes the following skills:
YOLO 2026 Object Detection
Category: Detection
State-of-the-art real-time object detection with 80+ COCO classes. Displays colored bounding boxes with class labels and confidence scores directly on camera feeds. Supports multiple model sizes (nano, small, medium, large) so you can balance speed and accuracy for your hardware.
Unlocks: Detection Studio view in the sidebar — a dedicated interface for visualizing detections across all cameras, adjusting confidence thresholds, and selecting model sizes.
Requirements: Python 3.9+
Depth Estimation (Privacy)
Category: Privacy / Transformation
Privacy-first depth map transforms using Depth Anything v2. Transforms camera feeds with colorized depth maps — near objects appear warm, far objects appear cool. When used in "depth only" mode, the scene is fully anonymized while preserving spatial layout and activity, enabling security monitoring without revealing identities.
Display modes:
- Depth Only — full anonymization, shows only the depth map
- Overlay — depth map blended over the original image at adjustable opacity
- Side by Side — original and depth map shown adjacent for comparison
Colormaps: Inferno, Viridis, Plasma, Magma, Jet, Turbo, Hot, Cool
Unlocks: 3D Depth Vision view in the sidebar
Requirements: Python 3.9+. Uses CoreML on macOS (Apple Neural Engine) and CUDA or CPU on Windows/Linux.
SAM2 Segmentation
Category: Segmentation
Interactive click-to-segment using Meta's Segment Anything 2. Click any point or draw a box on a camera frame to get pixel-perfect segmentation masks. Supports video tracking — select an object once and SAM2 tracks it across frames.
Unlocks: Segmentation Studio view in the sidebar
Requirements: Python 3.9+
Annotation Data
Category: Annotation
Dataset annotation management for building custom training datasets. Import camera frames, label objects with COCO-format annotations, organize into sequences, and export for model training or upload to Kaggle.
Unlocks: Annotation Studio view in the sidebar
Requirements: Python 3.9+
Model Training
Category: Training
Agent-driven YOLO fine-tuning workflow. Use your annotated camera data to fine-tune a custom YOLO model that's specialized for your specific environment. The agent guides you through the training process and automatically exports optimized models for your hardware (TensorRT for NVIDIA, CoreML for Apple Silicon, OpenVINO for Intel).
Requirements: Python 3.9+
Home Security AI Benchmark
Category: Analysis
Evaluation suite for testing LLM and VLM performance on real home security tasks. Tests deduplication accuracy, scene classification, tool usage, and description quality. Generates detailed benchmark reports to help you choose the right models for your setup.
Requirements: Node.js 18+
Camera Claw
Category: Integrations
Security camera for your AI agent — integrates with OpenClaw robotic arm systems. Sandbox, record, and monitor OpenClaw activity with full AI observation.
Unlocks: OpenClaw view in the sidebar
Requirements: Docker
The Skills Panel
Open the Skills panel by clicking the puzzle piece icon in the sidebar. The panel has a layout inspired by extension managers:
→ Search bar — filters both Installed and Marketplace sections simultaneously → Installed section — skills you've installed, with lifecycle controls (Start, Stop, Restart) → Marketplace section — available skills from configured repositories, with Install buttons → Console — click the terminal icon to see live output from running skills (stdout, stderr, images) → Source browser — view the source code of any installed skill for transparency and debugging → Repository config — manage which skill repositories the marketplace pulls from
AI-Assisted Installation
When you install a skill, the AI agent assists with the process. It:
→ Checks system compatibility (OS, GPU availability, Python version) → Downloads the skill package from the configured repository → Sets up the Python environment and installs dependencies → Downloads required ML models (which can be several gigabytes) → Validates that everything is working before marking the skill as ready
If a skill requires a specific model that isn't yet downloaded, the agent will ask for confirmation before downloading.
Installation progress is shown in real-time with a step-by-step progress bar.
Skill Lifecycle
Every installed skill has a lifecycle:
| State | Meaning |
|---|---|
| Stopped | Installed but not running. Not using any resources. |
| Starting | Launching the skill process and loading the model. |
| Running | Actively processing frames. Green indicator pulsing. |
| Stopping | Gracefully shutting down. |
| Restarting | Stopping and immediately re-starting. |
| Error | Crashed or failed to start. Check the console for details. |
Console Output
Each running skill produces console output — exactly like a terminal. Click the terminal icon in the Skills panel header to see it. The console supports:
→ Multiple tabs — one per running skill, with colored status dots → Color-coded streams — stdout (white), stderr (red), system messages (purple) → Inline images — skills like VLM test display images directly in the console → Input capability — type commands and send them to the running skill → Auto-scroll — follows new output, with a manual scroll-override indicator
Skill Categories
Skills are organized into categories that describe their purpose:
| Category | Purpose | Examples |
|---|---|---|
| Detection | Identify and locate objects in camera frames | YOLO Detection (people, vehicles, animals, packages) |
| Analysis | VLM scene understanding and evaluation | Home Security Benchmark |
| Segmentation | Pixel-level object isolation | SAM2 Segmentation |
| Privacy | Anonymization and privacy-preserving transforms | Depth Estimation |
| Annotation | Dataset labeling and export | Annotation Data |
| Training | Model fine-tuning and deployment | Model Training |
| Integrations | Smart home and IoT platform connections | Camera Claw |
| Streaming | RTSP/WebRTC live video proxying | Stream services |
| Channels | Messaging platform integrations | Telegram, Discord, Slack |
| Automation | MQTT, webhooks, Home Assistant triggers | Event-driven automations |