Details of Project

Deputy is a local-first AI copilot designed to assist users during live meetings, interviews, and technical discussions. Unlike traditional note-taking tools, it operates in real time by capturing system and microphone audio, generating live transcripts, and providing contextual AI suggestions directly inside a stealth overlay interface.

The system solves the challenge of thinking, listening, and responding simultaneously by converting live conversation context into actionable intelligence such as suggested answers, follow-up questions, clarifications, and summaries.

It supports dual-channel audio processing, real-time speech-to-text streaming, screenshot-based context understanding, and multi-mode AI reasoning (Answer, Clarify, Follow-up, Code Hint, Brainstorm, Recap). All conversation data is stored locally, enabling privacy-first usage with optional cloud enhancement through BYOK (Bring Your Own Key).

Project Features

• Real-time dual-channel audio capture (system audio + microphone separation)
• Live transcription using multi-provider STT (Google, Deepgram, OpenAI, Azure, Soniox, ElevenLabs)
• AI-powered overlay assistant for instant interview and meeting support
• Multiple AI modes: Answer, Clarify, Follow-up, Code Hint, Brainstorm, Recap
• Screenshot understanding for context-aware reasoning during discussions
• Local-first memory system with SQLite + vector embeddings (RAG-based retrieval)
• Multi-provider LLM routing (OpenAI, Gemini, Claude, Groq, Ollama)
• Stealth overlay UI with always-on-top and pass-through interaction mode
• Secure credential handling with in-memory scrubbing and OS key storage
• Meeting history dashboard with searchable transcript and export support

Technical Implementation

The system is architected as a multi-layer desktop intelligence engine. The UI layer is built in Electron with React, while the performance-critical audio pipeline is implemented in Rust using N-API bindings for low-latency capture and DSP processing. A central orchestration engine manages transcription streams, AI routing, and context memory generation.

Incoming audio is processed through a dual-stream architecture separating system audio and microphone input. These streams are transcribed in real time using pluggable STT providers with fallback chains and retry logic. The resulting text is fed into a rolling context engine that maintains conversation state and prevents repetition.

A local RAG pipeline chunks and embeds meeting data into SQLite vector storage, enabling semantic retrieval across both active and historical sessions. AI responses are generated through a mode-based intent classifier that dynamically selects response strategies depending on user actions such as answering, clarifying, or summarizing.

The system also includes reliability engineering features such as provider failover, streaming cancellation control, epoch-based summarization for long sessions, and crash-safe queue recovery to ensure uninterrupted real-time performance.