Skip to main content

Email, Calendar & Outlook Integration Plan

Date: 2026-04-01 Milestones: v0.20.0 (MCP + browser fallback), v0.23.0 (autonomous agents) Priority: P0 — Email triage is the highest-demand use case for local AI agents. Always-on + sensitive data = strongest case for local processing.

1. Why Email & Calendar

Email management is the top use case for AI agent users: overnight triage, morning briefs, draft responses. These workloads are always-on and involve sensitive data — exactly where local processing has the strongest advantage over cloud APIs. All email content is processed locally. No cloud APIs touch your inbox.

2. Existing GitHub Issues


3. Integration Approaches

3.1 Browser Automation (Fallback)

Approach: Use Playwright MCP (or BrowserToolsMixin #458) to interact with Gmail, Outlook Web, Google Calendar through the browser. Advantages:
  • Works with ANY web email/calendar — no per-provider API integration
  • No API keys or OAuth setup — user just logs in once
  • Sandboxed via Playwright — safe, cross-platform
  • Enables demos immediately — “check my email” works day one
  • Auth persistence — Playwright browser context saves login sessions
Disadvantages:
  • Slower than direct API (browser rendering overhead)
  • Fragile to UI changes in Gmail/Outlook
  • Requires Node.js (for Playwright MCP)
  • Limited to what’s visible in the web UI
Supported providers (via browser):
  • Gmail (mail.google.com)
  • Outlook Web (outlook.office.com / outlook.live.com)
  • Google Calendar (calendar.google.com)
  • Outlook Calendar (outlook.office.com/calendar)
  • Yahoo Mail, ProtonMail, Fastmail, etc.
Dependency: #458 (BrowserToolsMixin with Playwright)

3.2 MCP Servers (Phase 1 — Primary Path)

Approach: Community MCP servers for Gmail and Google Calendar, installable via MCP Settings UI. Advantages:
  • Faster than browser automation (direct API)
  • More reliable (structured API, not scraping)
  • Works in headless/background mode
Disadvantages:
  • Requires OAuth setup (user must create Google Cloud project or use provided credentials)
  • Each provider needs its own MCP server
  • Dependency on community-maintained npm packages

3.3 Direct IMAP/OAuth (Phase 2 — v0.23.0)

Approach: Native Python email agent using IMAP (read) and SMTP (send) with OAuth2 authentication for Gmail/Outlook. Advantages:
  • Fastest — direct protocol, no browser or MCP overhead
  • Most reliable — standard protocols, no third-party dependencies
  • Works offline after initial auth (cached credentials)
  • Full control over email parsing and threading
Disadvantages:
  • More implementation effort (~2-3 weeks)
  • Need to handle MIME parsing, threading, attachments
  • OAuth2 flow requires redirect URI handling
Libraries: imaplib (stdlib), email (stdlib), aiohttp (OAuth2 flows), or exchangelib for Exchange/Outlook Key decision: Gmail/Calendar are too important for browser automation as the primary path. MCP servers provide structured, reliable API access. Browser automation is a fallback for edge cases (unsupported providers, demo scenarios without OAuth setup).

4. Email Triage Agent

GitHub issue: #645 (v0.23.0)

4.1 Architecture

4.2 Triage Categories (AI-learned)

Categories are learned from user behavior:
  • User responds quickly → sender becomes “urgent”
  • User ignores → sender becomes “low priority”
  • User corrects categorization → agent updates preferences

4.3 Email Providers

4.4 Privacy Guarantees

  • All email content processed locally — never sent to cloud APIs
  • Credentials stored in local credential vault (~/.gaia/credentials.db)
  • Email content not stored permanently — only summaries in memory
  • User can audit what the agent “remembers” about their email patterns
  • No email forwarding or external relay

5. Calendar Agent

GitHub issue: #662 (v0.23.0)

5.1 Capabilities

5.2 Calendar Providers


6. Daily Briefs

GitHub issue: #663 (v0.20.0)

6.1 Brief Contents (personalized over time)

6.2 Delivery Channels

6.3 Personalization via Memory

The brief adapts over time:
  • Learns which email senders are important to you
  • Learns your meeting prep preferences
  • Learns which news topics you engage with
  • Adapts format (detailed vs concise) based on feedback
  • “Stop including weather” → agent remembers permanently

7. Outlook-Specific Considerations

Outlook is important because:
  1. AMD’s primary consumer platform is Windows, and Outlook is the default email client
  2. Enterprise users (Tier 3 target) overwhelmingly use Outlook/Exchange
  3. Outlook has both web (outlook.office.com) and desktop (COM API) versions

7.1 Outlook Integration Paths

Recommendation: Start with Outlook Web (Playwright) for demos. Add MS Graph API for production quality in Phase 2. Skip COM — too fragile and Windows-only.

7.2 MS Graph API Integration (Phase 2)

Auth: OAuth2 with MSAL (Microsoft Authentication Library). User authenticates once via browser popup, token refreshed automatically.

8. Security Considerations

Critical policy: The agent can READ email autonomously but NEVER SEND without explicit user confirmation. Draft responses are shown in the UI for approval.

9. Dependencies


10. Implementation Phases

Phase 1: MCP Server Integration (v0.20.0, primary path)

Deliverable: “Check my email” and “What’s on my calendar today?” work via MCP servers.

Phase 1 Fallback: Browser Automation (v0.20.0, for providers without MCP)

Deliverable: Fallback path for providers without MCP servers.

Phase 2: Dedicated Email Triage Agent (v0.23.0, 2-3 weeks)

Deliverable: Autonomous overnight email triage + morning calendar brief.

Phase 3: Personalization (v0.23.0+, ongoing)


11. Meeting Notes Capture & Analysis

GitHub issues: #700 (Meeting notes with diarization), #389 (Voice Agent demo) — v0.21.0 Meeting transcription is a natural extension of the email + calendar ecosystem. After checking your calendar for today’s meetings, the agent can record, transcribe, and summarize each meeting — then include highlights in tomorrow’s daily brief.

11.1 Capabilities

11.2 Speaker Diarization

Current Whisper ASR has no speaker diarization. Two approaches: Recommendation: Use pyannote.audio — best accuracy, runs locally (AMD GPU), well-maintained. Install as optional: pip install gaia[meeting-notes].

11.3 Pipeline

11.4 Output Format

11.5 CLI Commands

11.6 Dependencies

  • #389 (Voice Agent demo) — base agent with ASR + DatabaseMixin
  • #372 (streaming ASR via Lemonade WebSocket)
  • #386 (TalkSDK auto-detection)
  • #375 (reranking for search)
  • pyannote.audio — speaker diarization (optional dependency)
  • Calendar agent (#662) — for meeting context
  • Daily briefs (#663) — for including meeting highlights

11.7 Privacy

  • All audio processed locally — never sent to cloud
  • Speaker models stored locally (no voice fingerprint exfiltration)
  • Recordings can be auto-deleted after transcription (configurable)
  • Meeting content indexed locally in RAG, never shared

12. Success Metrics


13. Open Questions