Email, Calendar & Outlook Integration Plan
Date: 2026-04-01 Milestones: v0.20.0 (MCP + browser fallback), v0.23.0 (autonomous agents) Priority: P0 — Email triage is the highest-demand use case for local AI agents. Always-on + sensitive data = strongest case for local processing.
1. Why Email & Calendar
Email management is the top use case for AI agent users: overnight triage, morning briefs, draft responses. These workloads are always-on and involve sensitive data — exactly where local processing has the strongest advantage over cloud APIs. All email content is processed locally. No cloud APIs touch your inbox.2. Existing GitHub Issues
3. Integration Approaches
3.1 Browser Automation (Fallback)
Approach: Use Playwright MCP (or BrowserToolsMixin #458) to interact with Gmail, Outlook Web, Google Calendar through the browser. Advantages:- Works with ANY web email/calendar — no per-provider API integration
- No API keys or OAuth setup — user just logs in once
- Sandboxed via Playwright — safe, cross-platform
- Enables demos immediately — “check my email” works day one
- Auth persistence — Playwright browser context saves login sessions
- Slower than direct API (browser rendering overhead)
- Fragile to UI changes in Gmail/Outlook
- Requires Node.js (for Playwright MCP)
- Limited to what’s visible in the web UI
- Gmail (mail.google.com)
- Outlook Web (outlook.office.com / outlook.live.com)
- Google Calendar (calendar.google.com)
- Outlook Calendar (outlook.office.com/calendar)
- Yahoo Mail, ProtonMail, Fastmail, etc.
3.2 MCP Servers (Phase 1 — Primary Path)
Approach: Community MCP servers for Gmail and Google Calendar, installable via MCP Settings UI.
Advantages:
- Faster than browser automation (direct API)
- More reliable (structured API, not scraping)
- Works in headless/background mode
- Requires OAuth setup (user must create Google Cloud project or use provided credentials)
- Each provider needs its own MCP server
- Dependency on community-maintained npm packages
3.3 Direct IMAP/OAuth (Phase 2 — v0.23.0)
Approach: Native Python email agent using IMAP (read) and SMTP (send) with OAuth2 authentication for Gmail/Outlook. Advantages:- Fastest — direct protocol, no browser or MCP overhead
- Most reliable — standard protocols, no third-party dependencies
- Works offline after initial auth (cached credentials)
- Full control over email parsing and threading
- More implementation effort (~2-3 weeks)
- Need to handle MIME parsing, threading, attachments
- OAuth2 flow requires redirect URI handling
imaplib (stdlib), email (stdlib), aiohttp (OAuth2 flows),
or exchangelib for Exchange/Outlook
3.4 Recommended Phased Approach
Key decision: Gmail/Calendar are too important for browser automation as the primary
path. MCP servers provide structured, reliable API access. Browser automation is a
fallback for edge cases (unsupported providers, demo scenarios without OAuth setup).
4. Email Triage Agent
GitHub issue: #645 (v0.23.0)4.1 Architecture
4.2 Triage Categories (AI-learned)
Categories are learned from user behavior:
- User responds quickly → sender becomes “urgent”
- User ignores → sender becomes “low priority”
- User corrects categorization → agent updates preferences
4.3 Email Providers
4.4 Privacy Guarantees
- All email content processed locally — never sent to cloud APIs
- Credentials stored in local credential vault (~/.gaia/credentials.db)
- Email content not stored permanently — only summaries in memory
- User can audit what the agent “remembers” about their email patterns
- No email forwarding or external relay
5. Calendar Agent
GitHub issue: #662 (v0.23.0)5.1 Capabilities
5.2 Calendar Providers
6. Daily Briefs
GitHub issue: #663 (v0.20.0)6.1 Brief Contents (personalized over time)
6.2 Delivery Channels
6.3 Personalization via Memory
The brief adapts over time:- Learns which email senders are important to you
- Learns your meeting prep preferences
- Learns which news topics you engage with
- Adapts format (detailed vs concise) based on feedback
- “Stop including weather” → agent remembers permanently
7. Outlook-Specific Considerations
Outlook is important because:- AMD’s primary consumer platform is Windows, and Outlook is the default email client
- Enterprise users (Tier 3 target) overwhelmingly use Outlook/Exchange
- Outlook has both web (outlook.office.com) and desktop (COM API) versions
7.1 Outlook Integration Paths
Recommendation: Start with Outlook Web (Playwright) for demos. Add MS Graph API
for production quality in Phase 2. Skip COM — too fragile and Windows-only.
7.2 MS Graph API Integration (Phase 2)
8. Security Considerations
Critical policy: The agent can READ email autonomously but NEVER SEND without
explicit user confirmation. Draft responses are shown in the UI for approval.
9. Dependencies
10. Implementation Phases
Phase 1: MCP Server Integration (v0.20.0, primary path)
Deliverable: “Check my email” and “What’s on my calendar today?” work via MCP servers.
Phase 1 Fallback: Browser Automation (v0.20.0, for providers without MCP)
Deliverable: Fallback path for providers without MCP servers.
Phase 2: Dedicated Email Triage Agent (v0.23.0, 2-3 weeks)
Deliverable: Autonomous overnight email triage + morning calendar brief.
Phase 3: Personalization (v0.23.0+, ongoing)
11. Meeting Notes Capture & Analysis
GitHub issues: #700 (Meeting notes with diarization), #389 (Voice Agent demo) — v0.21.0 Meeting transcription is a natural extension of the email + calendar ecosystem. After checking your calendar for today’s meetings, the agent can record, transcribe, and summarize each meeting — then include highlights in tomorrow’s daily brief.11.1 Capabilities
11.2 Speaker Diarization
Current Whisper ASR has no speaker diarization. Two approaches:
Recommendation: Use
pyannote.audio — best accuracy, runs locally (AMD GPU),
well-maintained. Install as optional: pip install gaia[meeting-notes].
11.3 Pipeline
11.4 Output Format
11.5 CLI Commands
11.6 Dependencies
- #389 (Voice Agent demo) — base agent with ASR + DatabaseMixin
- #372 (streaming ASR via Lemonade WebSocket)
- #386 (TalkSDK auto-detection)
- #375 (reranking for search)
pyannote.audio— speaker diarization (optional dependency)- Calendar agent (#662) — for meeting context
- Daily briefs (#663) — for including meeting highlights
11.7 Privacy
- All audio processed locally — never sent to cloud
- Speaker models stored locally (no voice fingerprint exfiltration)
- Recordings can be auto-deleted after transcription (configurable)
- Meeting content indexed locally in RAG, never shared