+-------------------------------------------------------------+
|                    GAIA File System Agent                     |
+--------------+------------------+----------------------------+
|  Layer 1     |  Layer 2         |  Layer 3                   |
|  NAVIGATOR   |  SEARCH ENGINE   |  KNOWLEDGE BASE            |
|              |                  |                            |
|  * Tree view |  * Name search   |  * Semantic index (RAG)    |
|  * Browse    |  * Content grep  |  * File system map         |
|  * Bookmarks |  * Metadata      |  * Usage patterns          |
|              |    queries       |  * Persistent memory       |
|              |  * Glob patterns |  * Category tagging        |
+--------------+------------------+----------------------------+
|             File System Index (SQLite + WAL mode)            |
|  * File metadata cache    * Metadata-based change detection  |
|  * Directory structure    * Last-seen timestamps             |
|  * User bookmarks         * Category tags                    |
+--------------------------------------------------------------+
|          Existing GAIA Infrastructure                        |
|  * FileSearchToolsMixin   * RAGSDK (FAISS + embeddings)      |
|  * ShellToolsMixin        * FileWatcher (watchdog)           |
|  * PathValidator          * compute_file_hash()              |
|  * DatabaseMixin          * FileChangeHandler                |
+--------------------------------------------------------------+

ChatAgent (enhanced)
  |
  +-- FileSystemToolsMixin (NEW - Layer 1 & 2, shared location)
  |     +-- browse_directory()         # NEW tool
  |     +-- tree()                     # NEW tool
  |     +-- file_info()                # NEW tool
  |     +-- find_files()               # REPLACES search_file + search_directory
  |     +-- bookmark()                 # NEW tool
  |     +-- read_file()                # ENHANCED existing tool (more formats)
  |
  +-- FileSystemIndexService (NEW - Layer 3 backend)
  |     Inherits: DatabaseMixin
  |     +-- scan_directory()
  |     +-- build_map()
  |     +-- update_incremental()
  |     +-- query_index()
  |     +-- get_statistics()
  |
  +-- RAGToolsMixin (EXISTING - enhanced)
  |     +-- index_document()           # add DOCX/PPTX/XLSX support
  |     +-- query_documents()          # integrate with file system map
  |     +-- index_directory()          # incremental with metadata check
  |
  +-- ShellToolsMixin (EXISTING - no changes)
  |
  +-- FileSearchToolsMixin (DEPRECATED - replaced by FileSystemToolsMixin)
        search_file()                  # -> merged into find_files()
        search_directory()             # -> merged into find_files()
        read_file()                    # -> moved to FileSystemToolsMixin (enhanced)
        search_file_content()          # -> enhanced and moved

@tool(atomic=True)
def browse_directory(
    path: str = "~",           # Directory to browse (default: home)
    show_hidden: bool = False,  # Include hidden files/dirs
    sort_by: str = "name",      # name | size | modified | type
    filter_type: str = None,    # Filter by extension (e.g., "pdf", "py")
    max_items: int = 50,        # Limit results
) -> str:
    """Browse a directory and list its contents with metadata.

    Returns files and subdirectories with size, modification date, and type info.
    Use this to explore what's inside a folder.
    """

C:\Users\John\Documents (23 items, 4.2 GB total)

  Type  Name                     Size      Modified
  ----  ----                     ----      --------
  [DIR] Projects/                1.2 GB    2026-03-08 14:30
  [DIR] Tax Returns/             340 MB    2026-02-15 09:12
  [DIR] Photos/                  2.1 GB    2026-03-07 18:45
  [FIL] resume.pdf               2.1 MB    2026-01-20 11:00
  [FIL] budget-2026.xlsx         145 KB    2026-03-01 16:22
  [FIL] notes.md                 12 KB     2026-03-09 08:15
  ...

@tool(atomic=True)
def tree(
    path: str = ".",
    max_depth: int = 3,
    show_sizes: bool = False,
    include_pattern: str = None,   # Only show matching files
    exclude_pattern: str = None,   # Hide matching files/dirs
    dirs_only: bool = False,       # Only show directories
) -> str:
    """Show a tree visualization of a directory structure.

    Useful for understanding project layouts and folder hierarchies.
    Shows nested directories and files with optional size info.
    """

C:\Users\John\Projects\my-app
+-- src/
|   +-- components/
|   |   +-- Header.tsx (4.2 KB)
|   |   +-- Footer.tsx (2.1 KB)
|   |   +-- Sidebar.tsx (3.8 KB)
|   +-- pages/
|   |   +-- index.tsx (1.5 KB)
|   |   +-- about.tsx (980 B)
|   +-- utils/
|       +-- helpers.ts (2.3 KB)
+-- package.json (1.2 KB)
+-- tsconfig.json (450 B)
+-- README.md (3.4 KB)

3 directories, 8 files, 20.0 KB total

@tool(atomic=True)
def file_info(path: str) -> str:
    """Get comprehensive information about a file or directory.

    Returns size, dates, type, MIME type, encoding, and format-specific
    metadata (line count for text, dimensions for images, page count for PDFs).
    For directories: item count, total size, file type breakdown.
    """

@tool(atomic=True)
def read_file(
    file_path: str,
    lines: int = 100,          # Number of lines to show (0 = all)
    encoding: str = "auto",    # Auto-detect encoding
    mode: str = "full",        # full | preview | metadata
) -> str:
    """Read and display a file's contents with intelligent type-based analysis.

    For text/code: shows content with line numbers.
    For CSV/TSV: shows tabular format with column headers.
    For JSON/YAML: pretty-printed with truncation for large objects.
    For images: dimensions, format, EXIF metadata.
    For PDF: page count, title, text preview.
    For DOCX/XLSX: structure overview and text content.
    For binary: hex dump header and file type detection.
    Use mode='preview' for a quick summary, mode='metadata' for info only.
    """

@tool(atomic=True)
def bookmark(
    action: str = "list",      # add | remove | list
    path: str = None,
    label: str = None,         # Human-friendly name
) -> str:
    """Save, list, or remove bookmarks for frequently accessed files and directories.

    Bookmarks persist across sessions in the file system index.
    Use 'add' with a path and optional label to save a bookmark.
    Use 'remove' with a path to delete a bookmark.
    Use 'list' to see all saved bookmarks.
    """

@tool(atomic=True)
def find_files(
    query: str,                     # Search query (name, content, or natural language)
    search_type: str = "auto",      # auto | name | content | metadata
    scope: str = "smart",           # smart | home | cwd | everywhere | <specific path>
    file_types: str = None,         # Comma-separated extensions: "pdf,docx,txt"
    size_range: str = None,         # e.g., ">10MB", "<1KB", "1MB-100MB"
    date_range: str = None,         # e.g., "today", "this-week", "2026-01", ">2026-01-01"
    max_results: int = 25,
    sort_by: str = "relevance",     # relevance | name | size | modified
) -> str:
    """Search for files by name, content, or metadata.

    This is the primary file search tool. Replaces search_file and search_directory.
    When index is available, searches the index first (<100ms).
    Falls back to filesystem glob when index is unavailable (<10sec).

    Search types:
    - auto: intelligently picks the best strategy based on query
    - name: search by file/directory name pattern (glob)
    - content: search inside file contents (grep-like)
    - metadata: filter by size, date, type

    Scope 'smart' searches: CWD first, then home common locations,
    then indexed directories. Use 'everywhere' for full drive search (slow).
    """

-- Schema version tracking for migrations
CREATE TABLE schema_version (
    version INTEGER PRIMARY KEY,
    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    description TEXT
);
INSERT INTO schema_version (version, description) VALUES (1, 'Initial schema');

-- Enable WAL mode for concurrent read/write access
PRAGMA journal_mode=WAL;

-- Core file metadata index
CREATE TABLE files (
    id INTEGER PRIMARY KEY,
    path TEXT UNIQUE NOT NULL,
    name TEXT NOT NULL,
    extension TEXT,
    mime_type TEXT,
    size INTEGER,
    created_at TIMESTAMP,
    modified_at TIMESTAMP,
    -- Change detection: size + mtime is the PRIMARY method (fast, no I/O)
    -- Content hash is OPTIONAL and computed only on user request (Phase 4)
    content_hash TEXT DEFAULT NULL,
    parent_dir TEXT NOT NULL,
    depth INTEGER,                -- Depth from scan root
    is_directory BOOLEAN DEFAULT FALSE,
    indexed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    metadata_json TEXT            -- Extra metadata (dimensions, page count, etc.)
);

-- Full-text search on file names and paths
CREATE VIRTUAL TABLE files_fts USING fts5(
    name, path, extension,
    content='files',
    content_rowid='id'
);

-- Directory statistics cache
CREATE TABLE directory_stats (
    path TEXT PRIMARY KEY,
    total_size INTEGER,
    file_count INTEGER,
    dir_count INTEGER,
    deepest_depth INTEGER,
    common_extensions TEXT,       -- JSON array of top extensions
    last_scanned TIMESTAMP
);

-- User bookmarks (persist across sessions)
CREATE TABLE bookmarks (
    id INTEGER PRIMARY KEY,
    path TEXT NOT NULL UNIQUE,
    label TEXT,
    category TEXT,               -- "project", "documents", "media", etc.
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Scan history for incremental updates
CREATE TABLE scan_log (
    id INTEGER PRIMARY KEY,
    directory TEXT NOT NULL,
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    files_scanned INTEGER,
    files_added INTEGER,
    files_updated INTEGER,
    files_removed INTEGER,
    duration_ms INTEGER
);

-- File categories (auto-tagged by extension)
CREATE TABLE file_categories (
    file_id INTEGER,
    category TEXT,               -- "code", "document", "image", "video", "data", etc.
    subcategory TEXT,            -- "python", "pdf", "jpeg", "csv", etc.
    FOREIGN KEY (file_id) REFERENCES files(id) ON DELETE CASCADE
);

-- Indexes for fast queries
CREATE INDEX idx_files_parent ON files(parent_dir);
CREATE INDEX idx_files_ext ON files(extension);
CREATE INDEX idx_files_modified ON files(modified_at);
CREATE INDEX idx_files_size ON files(size);
CREATE INDEX idx_files_hash ON files(content_hash) WHERE content_hash IS NOT NULL;
CREATE INDEX idx_categories ON file_categories(category, subcategory);
CREATE INDEX idx_bookmarks_path ON bookmarks(path);

MIGRATIONS = {
    1: "Initial schema (see above)",
    # Future migrations:
    # 2: "ALTER TABLE files ADD COLUMN ...",
}

def migrate(self):
    """Apply pending schema migrations.

    On startup, checks schema_version and applies any missing migrations.
    If database is corrupted or schema is unrecognizable, drops and rebuilds.
    """
    current = self._get_schema_version()
    for version in sorted(MIGRATIONS.keys()):
        if version > current:
            self._apply_migration(version)

def _check_integrity(self) -> bool:
    """Run PRAGMA integrity_check on startup.

    If corrupted, log warning, delete database, and rebuild from scratch.
    The index is fully reconstructable from the filesystem.
    """

from gaia.database.mixin import DatabaseMixin

class FileSystemIndexService(DatabaseMixin):
    """Persistent file system index backed by SQLite.

    Inherits from DatabaseMixin for all database operations (init_db, query,
    insert, update, delete, transaction, table_exists, execute).

    Inspired by Everything's speed philosophy but with content awareness.
    Uses SQLite FTS5 for fast name/path search and incremental scanning
    with metadata-based change detection (size + mtime).

    Content hashing is OPT-IN and only computed during Phase 2 background
    analysis or on explicit user request.
    """

    DB_PATH = "~/.gaia/file_index.db"

    def __init__(self):
        self.init_db(str(Path(self.DB_PATH).expanduser()))
        self._ensure_schema()
        self._check_integrity()

    def _ensure_schema(self):
        """Create tables if they don't exist, run migrations if needed."""
        if not self.table_exists("schema_version"):
            self.execute(SCHEMA_SQL)
        else:
            self.migrate()

    def scan_directory(
        self,
        path: str,
        max_depth: int = 10,
        exclude_patterns: list = None,
        incremental: bool = True,
    ) -> ScanResult:
        """Scan a directory tree and populate the index.

        Phase 1 (quick): Metadata only — names, sizes, mtime.
        Uses size + mtime comparison for incremental change detection.
        Does NOT read file contents or compute hashes.

        Args:
            path: Directory to scan
            max_depth: Maximum recursion depth (default: 10)
            exclude_patterns: Directory names to skip (merged with defaults)
            incremental: If True, skip files where size+mtime unchanged
        """

    def query_files(
        self,
        name: str = None,        # FTS5 search on name/path
        extension: str = None,
        min_size: int = None,
        max_size: int = None,
        modified_after: str = None,
        modified_before: str = None,
        parent_dir: str = None,
        category: str = None,
        limit: int = 25,
    ) -> list[dict]:
        """Query the file index. Uses DatabaseMixin.query() internally."""

    def get_directory_stats(self, path: str) -> dict:
        """Get cached directory statistics."""

    def get_file_system_map(
        self,
        root: str = "~",
        depth: int = 2,
    ) -> "FileSystemMap":
        """Returns a structured summary of the file system for LLM context."""

    def auto_categorize(self, file_path: str) -> tuple:
        """Returns (category, subcategory) based on extension.

        Categories: code, document, image, video, audio, data, archive, config, other
        """

    def get_statistics(self) -> dict:
        """Total files indexed, breakdown by type, storage used, etc."""

    def cleanup_stale(self, max_age_days: int = 30) -> int:
        """Remove entries for files that no longer exist on disk."""

    # Bookmark operations (use DatabaseMixin.insert/query/delete)
    def add_bookmark(self, path: str, label: str = None, category: str = None) -> int
    def remove_bookmark(self, path: str) -> bool
    def list_bookmarks(self) -> list[dict]

@dataclass
class FileSystemMap:
    """A compact 'mental model' of the user's file system.

    Injected into the LLM system prompt ON DEMAND (not always-on)
    when the user's query involves file operations.

    Decision: On-demand injection, not always-on.
    Rationale: Saves ~500-1000 tokens per non-file query. The agent
    can request it via a tool call when needed. Small local LLMs
    (Qwen3-0.6B) have limited context and cannot afford the overhead.
    """
    home_dir: str
    total_indexed: int
    last_scan: datetime

    # Top-level directory summary
    key_directories: list   # Documents, Projects, Downloads, etc.

    # Bookmarked locations
    bookmarks: list

    # Recent activity
    recently_modified: list  # Last 10 files modified

    # File type distribution
    type_breakdown: dict     # {"pdf": 234, "py": 1502, ...}

    def to_context_string(self, max_tokens: int = 800) -> str:
        """Render as a compact string for LLM system prompt injection.

        Token budget reduced from 2000 to 800 to accommodate smaller
        local LLMs. Prioritizes bookmarks and recent files.
        """

## Your File System (indexed 2026-03-09)
Home: C:\Users\John (45.2 GB, 23,456 files)

Key Directories:
  Documents/ (12.3 GB) - PDFs, DOCX, spreadsheets
  Projects/ (8.1 GB) - Code repos: gaia, my-app, data-pipeline
  Downloads/ (6.2 GB) - Recent: installer.exe, report.pdf
  Desktop/ (1.1 GB) - Shortcuts, quick notes

Bookmarks:
  "GAIA Project" -> C:\Users\John\Work\gaia5
  "Tax Docs" -> C:\Users\John\Documents\Tax Returns\2025

Recently Modified:
  notes.md (8 min ago), budget.xlsx (2 hrs ago), app.py (yesterday)

File Types: 1,502 Python | 234 PDF | 189 Markdown | 156 JSON | ...

# In FileSystemToolsMixin initialization:
from gaia.utils.file_watcher import FileWatcher

def _start_watching(self, directories: list[str]):
    """Watch bookmarked/indexed directories for changes.

    IMPORTANT: Only watches explicitly bookmarked or user-scanned
    directories. Does NOT watch the entire home directory.
    Rationale: Watching too many directories exhausts OS watch handles
    (especially on Windows with ReadDirectoryChangesW buffer limits).
    """
    for directory in directories:
        watcher = FileWatcher(
            directory=directory,
            on_created=self._on_file_created,
            on_modified=self._on_file_modified,
            on_deleted=self._on_file_deleted,
            extensions=None,  # Watch all file types
        )
        watcher.start()
        self._active_watchers.append(watcher)

def _on_file_created(self, path: str):
    """Add new file to index (metadata only, no content read)."""

def _on_file_modified(self, path: str):
    """Update index entry with new size/mtime."""

def _on_file_deleted(self, path: str):
    """Remove file from index."""

Phase 1: Quick Structure Scan (~5 seconds for typical home dir)
  - Walk directory tree using pathlib (names, sizes, mtime only)
  - NO file content reading, NO hashing
  - Build directory_stats entries
  - Populate files table with metadata
  - Build FTS5 index for name/path search
  - Change detection: compare size + mtime against existing index entries

Phase 2: Content Analysis (background, progressive, OPT-IN)
  - Only runs if user explicitly requests deeper indexing
  - Hash files for duplicate detection (user-facing dirs first)
  - Extract metadata from rich files (PDFs, images, DOCX)
  - Auto-categorize files
  - Update index progressively

Phase 3: Ongoing Maintenance
  - FileWatcher on bookmarked/scanned directories only
  - Periodic re-scan (configurable, default: weekly) to catch missed changes
  - Stale entry cleanup (files that no longer exist)

class SmartChunker:
    """Content-aware document chunking.

    Uses Python stdlib for chunking — NO tree-sitter dependency.
    AST-based code chunking uses Python's built-in ast module for .py files,
    and regex-based function/class detection for other languages.

    Tree-sitter integration is DEFERRED to a future phase.
    """

    def chunk_markdown(self, content: str) -> list:
        """Split by headers, preserving section boundaries."""

    def chunk_prose(self, content: str) -> list:
        """Split by paragraphs with semantic boundary detection."""

    def chunk_tabular(self, content: str) -> list:
        """Split tables preserving header context with each chunk."""

    def chunk_python(self, content: str) -> list:
        """Split Python code by functions/classes using stdlib ast module."""

def index_directory_incremental(self, directory: str) -> dict:
    """Index a directory, skipping files that haven't changed.

    Uses size + mtime from FileSystemIndexService for change detection.
    Only re-chunks and re-embeds files where size or mtime differs.
    Content hashing is NOT used for change detection (too slow).
    """

Document Pipeline:
                                                    +------------------+
  PDF/DOCX/CSV  -->  RAG Extractor  -->  LLM  -->  | SQLite Scratchpad |
  (raw file)        (text/tables)      (parse     | +-- transactions  |
                                        to struct) | +-- categories    |
                                                    | +-- summaries    |
                                                    +--------+---------+
                                                             |
                                          SQL Query  <-------+
                                             |
                                          Results  -->  LLM  -->  Natural Language
                                                       (interpret      Summary
                                                        & present)

@tool(atomic=True)
def create_table(
    table_name: str,
    columns: str,
) -> str:
    """Create a table in the scratchpad database for storing extracted data.

    Use this to set up structured storage before processing documents.
    Column definitions follow SQLite syntax.

    Example: create_table("transactions",
        "date TEXT, description TEXT, amount REAL, category TEXT, source_file TEXT")
    """

@tool(atomic=True)
def insert_data(
    table_name: str,
    data: str,
) -> str:
    """Insert rows into a scratchpad table.

    Data is a JSON array of objects matching the table columns.
    Use this after extracting structured data from a document.

    Example: insert_data("transactions", '[
        {"date": "2026-01-05", "description": "NETFLIX", "amount": 15.99,
         "category": "subscription", "source_file": "jan-statement.pdf"},
        {"date": "2026-01-07", "description": "WHOLE FOODS", "amount": 87.32,
         "category": "groceries", "source_file": "jan-statement.pdf"}
    ]')
    """

@tool(atomic=True)
def query_data(
    sql: str,
) -> str:
    """Run a SQL query against the scratchpad database.

    Use SELECT queries to analyze accumulated data. Supports all SQLite
    functions: SUM, AVG, COUNT, GROUP BY, ORDER BY, JOINs, subqueries, etc.

    Examples:
        "SELECT category, SUM(amount) as total FROM transactions GROUP BY category ORDER BY total DESC"
        "SELECT description, COUNT(*) as freq, SUM(amount) as total FROM transactions GROUP BY description HAVING freq > 1 ORDER BY freq DESC"
        "SELECT strftime('%Y-%m', date) as month, SUM(amount) FROM transactions GROUP BY month"
    """

@tool(atomic=True)
def list_tables() -> str:
    """List all tables in the scratchpad database with their schemas and row counts.

    Use this to see what data has been accumulated so far.
    """

@tool(atomic=True)
def drop_table(table_name: str) -> str:
    """Remove a scratchpad table when analysis is complete.

    Use this to clean up after a task is done.
    """

from gaia.database.mixin import DatabaseMixin

class ScratchpadService(DatabaseMixin):
    """SQLite-backed working memory for multi-document data analysis.

    Inherits from DatabaseMixin for all database operations.
    Uses the same database file as FileSystemIndexService but with
    a 'scratch_' prefix on all table names to avoid collisions.

    Tables are user-created via tools and can persist across sessions
    or be cleaned up after analysis.
    """

    TABLE_PREFIX = "scratch_"

    def __init__(self, db_path: str = "~/.gaia/file_index.db"):
        self.init_db(str(Path(db_path).expanduser()))

    def create_table(self, name: str, columns: str) -> str:
        """Create a prefixed table. Returns confirmation."""
        safe_name = self._sanitize_name(name)
        self.execute(f"CREATE TABLE IF NOT EXISTS {self.TABLE_PREFIX}{safe_name} ({columns})")
        return f"Table '{safe_name}' created."

    def insert_rows(self, table: str, data: list[dict]) -> int:
        """Bulk insert rows. Returns count inserted."""
        safe_name = f"{self.TABLE_PREFIX}{self._sanitize_name(table)}"
        count = 0
        with self.transaction():
            for row in data:
                self.insert(safe_name, row)
                count += 1
        return count

    def query_data(self, sql: str) -> list[dict]:
        """Execute a SELECT query. Only allows SELECT statements.

        Security: Rejects INSERT/UPDATE/DELETE/DROP/ALTER in this method.
        Those operations have their own dedicated methods.
        """
        normalized = sql.strip().upper()
        if not normalized.startswith("SELECT"):
            raise ValueError("Only SELECT queries allowed via query_data(). "
                           "Use insert_data() or drop_table() for mutations.")
        return self.query(sql)

    def list_tables(self) -> list[dict]:
        """List all scratchpad tables with schema and row count."""
        tables = self.query(
            "SELECT name FROM sqlite_master WHERE type='table' AND name LIKE :prefix",
            {"prefix": f"{self.TABLE_PREFIX}%"}
        )
        result = []
        for t in tables:
            display_name = t["name"].replace(self.TABLE_PREFIX, "", 1)
            schema = self.query(f"PRAGMA table_info({t['name']})")
            count = self.query(f"SELECT COUNT(*) as count FROM {t['name']}", one=True)
            result.append({
                "name": display_name,
                "columns": [{"name": c["name"], "type": c["type"]} for c in schema],
                "rows": count["count"],
            })
        return result

    def drop_table(self, name: str) -> str:
        """Drop a scratchpad table."""
        safe_name = f"{self.TABLE_PREFIX}{self._sanitize_name(name)}"
        self.execute(f"DROP TABLE IF EXISTS {safe_name}")
        return f"Table '{name}' dropped."

    def _sanitize_name(self, name: str) -> str:
        """Sanitize table/column names to prevent SQL injection."""
        import re
        clean = re.sub(r'[^a-zA-Z0-9_]', '_', name)
        if not clean or clean[0].isdigit():
            clean = f"t_{clean}"
        return clean

Step 1: DISCOVER    find_files("credit card statement", file_types="pdf")
                    -> Found 12 PDF files in Documents/Statements/

Step 2: CREATE      create_table("transactions",
                      "date TEXT, description TEXT, amount REAL,
                       category TEXT, source_file TEXT")

Step 3: EXTRACT     For each PDF:
          (loop)      read_file(statement.pdf)
                      -> LLM extracts transactions from text
                      insert_data("transactions", [...extracted rows...])

Step 4: ANALYZE     query_data("SELECT category, SUM(amount), COUNT(*)
                      FROM transactions GROUP BY category
                      ORDER BY SUM(amount) DESC")

Step 5: INSIGHT     query_data("SELECT description, COUNT(*) as months,
                      SUM(amount) as total FROM transactions
                      GROUP BY description HAVING months >= 3
                      ORDER BY total DESC")
                    -> LLM interprets: "Hidden subscriptions detected..."

Step 6: REPORT      LLM synthesizes all query results into a natural
                    language report with actionable recommendations

User: analyze my credit card spending for the past year

Agent Step 1 (find_files):
  Searching for credit card statements...
  Found 12 PDFs in C:\Users\John\Documents\Statements\Chase\
    jan-2025.pdf through dec-2025.pdf (total: 4.2 MB)

Agent Step 2 (create_table):
  Creating analysis workspace...
  Table 'transactions' ready (date, description, amount, category, source_file)

Agent Steps 3-6 (read_file + insert_data, batched 3 per step):
  Processing jan-2025.pdf... extracted 47 transactions
  Processing feb-2025.pdf... extracted 52 transactions
  Processing mar-2025.pdf... extracted 61 transactions
  ... (12 files, ~600 transactions total)

Agent Step 7 (query_data - category breakdown):
  SELECT category, SUM(amount) as total, COUNT(*) as count
  FROM transactions GROUP BY category ORDER BY total DESC

  Category        Total      Txns
  --------------- ---------- ----
  Housing         $24,000    12
  Groceries       $8,432     156
  Dining Out      $4,891     89
  Transportation  $3,200     48
  Subscriptions   $2,147     36
  Shopping        $1,893     23
  Entertainment   $1,456     18
  ...

Agent Step 8 (query_data - subscription detection):
  SELECT description, COUNT(*) as months, ROUND(AVG(amount),2) as avg
  FROM transactions GROUP BY description
  HAVING months >= 3 AND avg > 5 ORDER BY avg DESC

  Description          Months  Avg/Mo
  -------------------- ------  ------
  SPOTIFY PREMIUM      12      $15.99
  NETFLIX              12      $22.99
  NYT DIGITAL          12      $17.00
  ADOBE CREATIVE       12      $54.99
  CHATGPT PLUS         10      $20.00
  DROPBOX PLUS          8      $11.99
  HULU                  6      $17.99
  CRUNCHYROLL           4       $7.99
  ──────────────────────────────────────
  Total subscriptions:           $168.95/month ($2,027/year)

Agent Step 9 (query_data - month-over-month trend):
  SELECT strftime('%Y-%m', date) as month,
         ROUND(SUM(amount),2) as total
  FROM transactions GROUP BY month ORDER BY month

  Month      Total
  -----      -----
  2025-01    $4,120
  2025-02    $3,890
  2025-03    $5,210  << spike
  2025-04    $4,450
  ...
  2025-12    $6,780  << holiday spending

Agent Step 10 (final synthesis):

  # Your 2025 Spending Analysis

  **Total Spent:** $52,341 across 612 transactions

  ## Key Findings

  1. **Subscriptions are bleeding you dry**
     You're paying $168.95/month ($2,027/year) across 8 subscriptions.
     - Hulu + Netflix + Crunchyroll = 3 streaming services ($48.97/mo)
     - You haven't used Dropbox since August (consider canceling: saves $144/yr)

  2. **Dining out is your biggest discretionary expense**
     $4,891/year on restaurants and delivery. That's $407/month.
     March was the worst month ($890 on dining alone).

  3. **Holiday spending spike**
     December was 64% above your monthly average ($6,780 vs $4,362 avg).

  4. **Grocery spending is steady and reasonable**
     $702/month average with low variance. Well managed.

  ## Recommendations
  - Cancel Dropbox ($144/yr saved) and consolidate streaming ($215/yr saved)
  - Set a dining budget alert at $300/month
  - Start a holiday fund: $200/month to avoid December spikes

User: help me prepare my 2025 taxes

Agent:
  Searching for tax-relevant documents...

  Found 23 documents across your PC:
  Income:
    W-2_2025_AMD.pdf — Documents/Tax/
    1099-INT_Chase.pdf — Documents/Tax/
    1099-DIV_Fidelity.pdf — Downloads/ (not filed yet!)

  Deductions:
    mortgage-interest-2025.pdf — Documents/Tax/
    property-tax-receipt.pdf — Documents/Tax/
    charitable-donations/ (4 receipts) — Documents/Tax/Donations/

  Medical:
    medical-expenses-2025.xlsx — Documents/Tax/

  Missing (expected but not found):
    - No 1099-NEC found (did you do any freelance work?)
    - No HSA contribution statement
    - No student loan interest form (1098-E)

  I've extracted and organized all the numbers:

  INCOME
    W-2 Wages:           $125,000
    Interest (1099-INT):  $1,234
    Dividends (1099-DIV): $3,456
    ─────────────────────────────
    Total Income:        $129,690

  DEDUCTIONS (Itemized)
    Mortgage Interest:    $14,200
    Property Tax:          $6,800
    Charitable:            $2,350
    Medical (above 7.5%):    $890
    ─────────────────────────────
    Total Deductions:     $24,240

  ESTIMATE
    Taxable Income:      $105,450
    Estimated Federal Tax: ~$18,200
    Already Withheld (W-2): $19,500
    ─────────────────────────────────
    Estimated Refund:      ~$1,300

  Action Items:
  1. Move 1099-DIV from Downloads/ to Documents/Tax/ (want me to suggest this?)
  2. Check if you have an HSA — you might be missing a deduction
  3. Review charitable donations — did you make any cash donations not receipted?

User: analyze all the ML papers in my research folder

Agent:
  Found 34 PDFs in Documents/Research/ML/
  Processing and indexing all papers...

  Created analysis tables:
  - papers (title, authors, year, venue, abstract, key_findings)
  - citations (paper_id, cited_paper_id, context)
  - topics (paper_id, topic, relevance_score)

  Extracted data from 34 papers (287 pages total).

  # Research Collection Summary

  ## By Topic
  | Topic                  | Papers | Key Authors        |
  |------------------------|--------|--------------------|
  | Attention Mechanisms   | 12     | Vaswani, Dao       |
  | Model Compression      | 8      | Hinton, Zhu        |
  | Multi-Modal Learning   | 7      | Radford, Li        |
  | Efficient Inference    | 5      | Leviathan, Pope    |
  | RLHF / Alignment       | 2      | Ouyang, Bai        |

  ## Timeline
  2023: 8 papers (mostly attention + compression)
  2024: 14 papers (shift toward efficiency + multi-modal)
  2025: 12 papers (inference optimization dominates)

  ## Gap Analysis
  Your collection is strong on attention and compression but MISSING:
  - Mixture of Experts (MoE) — no papers despite being a 2024-25 trend
  - Speculative decoding — only 1 brief mention across all papers
  - On-device inference (relevant to AMD NPU work!)

  ## Cross-Paper Insights
  Papers #4, #12, and #28 all cite the same Flash Attention result
  but reach different conclusions about optimal head dimensions.
  This might be worth investigating further.

  Want me to generate a literature review draft from these?

User: find all my contracts and tell me about upcoming deadlines

Agent:
  Searching for contracts, leases, and agreements...

  Found 8 documents:
  - apartment-lease-2025.pdf — Documents/Housing/
  - car-lease-agreement.pdf — Documents/Auto/
  - employment-agreement-amd.pdf — Documents/Work/
  - freelance-contract-acme.pdf — Documents/Work/
  - gym-membership.pdf — Documents/
  - insurance-policy-home.pdf — Documents/Insurance/
  - insurance-policy-auto.pdf — Documents/Insurance/
  - phone-plan-tmobile.pdf — Downloads/

  Extracting key terms and dates...

  # Active Contracts & Deadlines

  | Contract           | Expires     | Monthly Cost | Auto-Renew | Action Needed |
  |--------------------|-------------|-------------|------------|---------------|
  | Apartment Lease    | 2026-08-31  | $2,100      | No         | 60-day notice required (by Jul 1) |
  | Car Lease          | 2026-11-15  | $389        | No         | Return or buyout ($18K residual) |
  | Gym Membership     | 2026-04-01  | $49         | YES        | Cancel by Mar 15 to avoid renewal |
  | T-Mobile Plan      | Month-to-month | $85      | YES        | No action needed |
  | Home Insurance     | 2026-06-01  | $142        | YES        | Review coverage before renewal |
  | Auto Insurance     | 2026-09-15  | $118        | YES        | Shop rates before renewal |

  ## Urgent (Next 30 Days)
  - GYM MEMBERSHIP auto-renews April 1 — cancel by March 15 if you don't want it
  - Home insurance renewal June 1 — good time to compare rates

  ## Coming Up (Next 90 Days)
  - Apartment lease notice deadline: July 1 (decide if staying or moving)
  - Home insurance renewal: June 1

  Want me to set up reminders for these dates?

User: my disk is almost full, help me clean up

Agent:
  Scanning your drives...

  Drive C: 237 GB used of 256 GB (92% full!)

  # Space Analysis

  ## Biggest Offenders
  | Location                    | Size    | What's There |
  |-----------------------------|---------|-------------|
  | Users/John/AppData/         | 45.2 GB | App caches, browser data |
  | Users/John/Downloads/       | 23.1 GB | 847 files, oldest from 2023 |
  | Users/John/Videos/          | 18.4 GB | 23 video files |
  | Users/John/.conda/          | 12.8 GB | 4 conda environments |
  | Users/John/node_modules/    |  8.3 GB | 12 projects with node_modules |

  ## Safe to Delete (potential savings: 38.4 GB)

  1. **Downloads older than 6 months** — 14.2 GB
     327 files you probably don't need (installers, ZIPs, temp files)

  2. **Duplicate files** — 3.8 GB
     Found 89 duplicate files across your Documents and Downloads

  3. **Browser caches** — 6.1 GB
     Chrome: 3.2 GB, Firefox: 1.8 GB, Edge: 1.1 GB

  4. **Old conda environments** — 8.9 GB
     'env_old', 'test-env', 'backup-env' haven't been activated in 6+ months

  5. **Orphaned node_modules** — 5.4 GB
     6 projects with node_modules/ but no recent git activity

  That's 38.4 GB — enough to get you down to 77% disk usage.
  Want me to show the specific files for any of these categories?

User: I just cloned github.com/some/project into Projects/new-project.
      Help me understand it.

Agent:
  Analyzing C:\Users\John\Projects\new-project...

  # Project Overview

  **Type:** Python web application (FastAPI + React)
  **Size:** 12,847 files, 234 MB (4,521 source files, rest is node_modules)
  **Languages:** Python 62%, TypeScript 31%, CSS 5%, Other 2%

  ## Architecture
  +-- backend/          FastAPI app (47 endpoints)
  |   +-- api/          REST routes
  |   +-- models/       SQLAlchemy ORM (12 tables)
  |   +-- services/     Business logic
  |   +-- tasks/        Celery background workers
  +-- frontend/         React 18 + Vite
  |   +-- components/   42 React components
  |   +-- pages/        8 page views
  |   +-- hooks/        Custom hooks (auth, data fetching)
  +-- docker/           Docker Compose (3 services: app, db, redis)
  +-- tests/            pytest (68% coverage)

  ## Key Files to Start With
  1. backend/api/routes.py — All API endpoints defined here
  2. frontend/src/App.tsx — Main routing and layout
  3. docker/docker-compose.yml — How to run everything
  4. README.md — Setup instructions (looks well-maintained)

  ## Dependencies of Note
  - PostgreSQL 15 (main database)
  - Redis (caching + Celery broker)
  - Stripe SDK (payment processing — handle with care)
  - SendGrid (email — check for API keys in .env)

  ## Potential Issues Found
  - .env.example exists but no .env — you'll need to create one
  - requirements.txt has 3 packages with known CVEs (run safety check)
  - No pre-commit hooks configured

  Want me to index the full codebase for RAG so you can ask questions about it?

+------------------------------------------------------------------+
|  GAIA Chat Agent                                    [Settings] [?] |
+------------------+-----------------------------------------------+
|                  |                                                 |
|  SIDEBAR         |  CHAT AREA                                      |
|                  |                                                 |
|  [Chat]          |  User: analyze my credit card spending          |
|  [Dashboard]  <- |                                                 |
|  [Scratchpad] <- |  Agent: Searching for statements...             |
|  [File Index] <- |  [Step 1/10] find_files: Found 12 PDFs          |
|  [Documents]     |  [Step 2/10] create_table: "transactions"       |
|                  |  [Step 3/10] read_file: jan-2025.pdf             |
|  BOOKMARKS       |    -> Extracted 47 transactions                  |
|  * GAIA Project  |  ...                                            |
|  * Tax Docs      |                                                 |
|  * Statements    |  [SCRATCHPAD PREVIEW]                            |
|                  |  +------------------------------------------+   |
|  RECENT FILES    |  | transactions (612 rows)                  |   |
|  * notes.md      |  | date  | description  | amount | category|   |
|  * budget.xlsx   |  | 01-05 | NETFLIX      | 15.99  | sub     |   |
|  * app.py        |  | 01-07 | WHOLE FOODS  | 87.32  | grocery |   |
|                  |  | ...   | ...          | ...    | ...     |   |
|  INDEX STATUS    |  +------------------------------------------+   |
|  23,456 files    |                                                 |
|  Last: 2 min ago |  Final Answer: Your 2025 Spending Analysis...   |
|                  |                                                 |
+------------------+-----------------------------------------------+

+------------------------------------------------------------------+
|  Agent Dashboard                                                   |
+------------------------------------------------------------------+
|                                                                    |
|  AGENT STATUS                          SYSTEM INFO                 |
|  +----------------------------+        +------------------------+  |
|  | State: Idle                |        | Model: Qwen3-Coder-30B |  |
|  | Session: 12 messages       |        | Backend: Lemonade      |  |
|  | Steps used: 0/20          |        | Max Steps: 20          |  |
|  | Tools registered: 16      |        | RAG: Active (5 docs)   |  |
|  +----------------------------+        +------------------------+  |
|                                                                    |
|  FILE SYSTEM INDEX                                                 |
|  +--------------------------------------------------------------+ |
|  | Status: Active | Files: 23,456 | Size: 12 MB | Last: 2m ago  | |
|  |                                                                | |
|  | Top Directories:                                               | |
|  | Documents/ ........... 12.3 GB  [======####] 27%               | |
|  | AppData/ ............. 10.1 GB  [=====###] 22%                 | |
|  | Downloads/ ............ 8.7 GB  [====###] 19%                  | |
|  |                                                                | |
|  | File Types: 1,502 .py | 234 .pdf | 189 .md | 156 .json       | |
|  |                                                                | |
|  | [Scan Now]  [Clear Index]  [View Full Index]                   | |
|  +--------------------------------------------------------------+ |
|                                                                    |
|  SCRATCHPAD                                                        |
|  +--------------------------------------------------------------+ |
|  | Tables: 2 | Total Rows: 724 | Size: 1.2 MB                   | |
|  |                                                                | |
|  | transactions .... 612 rows  (date, desc, amount, category)     | |
|  | tax_documents ... 112 rows  (type, source, amount, status)     | |
|  |                                                                | |
|  | [View Tables]  [Clear Scratchpad]  [Export CSV]                 | |
|  +--------------------------------------------------------------+ |
|                                                                    |
|  BOOKMARKS                                                         |
|  +--------------------------------------------------------------+ |
|  | GAIA Project -> C:\Users\John\Work\gaia5          [Remove]     | |
|  | Tax Docs     -> C:\Users\John\Documents\Tax       [Remove]     | |
|  | Statements   -> C:\Users\John\Documents\Statements [Remove]    | |
|  | [+ Add Bookmark]                                               | |
|  +--------------------------------------------------------------+ |
|                                                                    |
|  ACTIVE WATCHERS                                                   |
|  +--------------------------------------------------------------+ |
|  | Watching 3 directories for changes:                            | |
|  | C:\Users\John\Work\gaia5\             (142 events today)       | |
|  | C:\Users\John\Documents\Tax\          (0 events today)         | |
|  | C:\Users\John\Documents\Statements\   (2 events today)         | |
|  +--------------------------------------------------------------+ |
+------------------------------------------------------------------+

+------------------------------------------------------------------+
|  Scratchpad Explorer                                               |
+------------------+-----------------------------------------------+
|  TABLES          |  TABLE: transactions (612 rows)                 |
|                  |                                                 |
|  > transactions  |  [SQL Query Bar]                                |
|    612 rows      |  SELECT * FROM transactions LIMIT 100           |
|                  |  [Run Query]                                    |
|  > tax_documents |                                                 |
|    112 rows      |  +---+--------+-------------+--------+--------+|
|                  |  | # | date   | description | amount | categ  ||
|  > summaries     |  +---+--------+-------------+--------+--------+|
|    5 rows        |  | 1 | 01-05  | NETFLIX     | 15.99  | sub    ||
|                  |  | 2 | 01-07  | WHOLE FOODS | 87.32  | groc   ||
|                  |  | 3 | 01-09  | SHELL GAS   | 45.00  | trans  ||
|                  |  | 4 | 01-12  | AMAZON      | 129.99 | shop   ||
|                  |  | ...                                         ||
|  [+ New Table]   |  +---+--------+-------------+--------+--------+|
|  [Clear All]     |                                                 |
|                  |  QUICK STATS                                     |
|                  |  Total: $52,341 | Avg/mo: $4,362 | Rows: 612   |
|                  |                                                 |
|                  |  [Export CSV]  [Export JSON]  [Drop Table]       |
+------------------+-----------------------------------------------+

+------------------------------------------------------------------+
|  File System Index                                                 |
+------------------------------------------------------------------+
|  [Search: ________________________] [Type: All v] [Sort: Name v]  |
|                                                                    |
|  PATH BROWSER                                                      |
|  C:\Users\John\                                                    |
|  +-- Documents/ (12.3 GB, 4,521 files)                             |
|  |   +-- Tax/ (890 MB, 23 files)                                   |
|  |   +-- Statements/ (340 MB, 48 files)                            |
|  |   +-- Projects/ (8.1 GB, 12,340 files)                          |
|  +-- Downloads/ (8.7 GB, 847 files)                                |
|  +-- Desktop/ (1.1 GB, 34 files)                                   |
|                                                                    |
|  SCAN HISTORY                                                      |
|  2026-03-09 14:30  Home directory  23,456 files  4.2s              |
|  2026-03-08 09:15  Documents/Tax   23 files      0.3s              |
|                                                                    |
|  [Scan Directory]  [Refresh]  [Clear Index]                        |
+------------------------------------------------------------------+

# In MessageBubble.tsx, detect scratchpad data markers in agent response:

# Agent response contains embedded data:
# <!--SCRATCHPAD_TABLE:transactions:SELECT * FROM transactions LIMIT 5-->

# Frontend renders this as an interactive table widget instead of markdown text.
# The widget supports:
# - Sortable column headers
# - Row count indicator
# - "Show more" / "View in Scratchpad" link
# - Expandable to full scratchpad tab

GET  /v1/dashboard/status           Agent state, model info, step count
GET  /v1/dashboard/index/stats      File index statistics
GET  /v1/dashboard/index/tree       Directory tree from index
GET  /v1/dashboard/scratchpad       List scratchpad tables
GET  /v1/dashboard/scratchpad/:table  Query a scratchpad table (paginated)
POST /v1/dashboard/scratchpad/query   Run a SELECT query
GET  /v1/dashboard/bookmarks        List bookmarks
POST /v1/dashboard/scan             Trigger a directory scan
DELETE /v1/dashboard/scratchpad     Clear all scratchpad tables
DELETE /v1/dashboard/index          Reset file index

from gaia.agents.base.tools import tool
from gaia.security import PathValidator

class FileSystemToolsMixin:
    """File system navigation, search, and management tools.

    Provides browse, tree, search, file info, bookmarks, and read capabilities.
    All path parameters are validated through PathValidator before access.

    Available to: ChatAgent, CodeAgent, or any agent needing file system access.

    Tool registration follows GAIA pattern: register_filesystem_tools() method
    with @tool decorator using docstrings for descriptions.
    """

    _fs_index: "FileSystemIndexService" = None
    _path_validator: PathValidator = None
    _active_watchers: list = []

    def _validate_path(self, path: str) -> Path:
        """Validate and resolve a path. Raises ValueError if blocked.

        All tools call this before any filesystem access.
        """
        resolved = Path(path).expanduser().resolve()
        if self._path_validator and not self._path_validator.is_path_allowed(str(resolved)):
            raise ValueError(f"Access denied: {resolved}")
        return resolved

    def register_filesystem_tools(self):
        """Register all file system tools. Called during agent init."""

        # Phase 1 Core Tools (6 tools):
        @tool(atomic=True)
        def browse_directory(...): ...

        @tool(atomic=True)
        def tree(...): ...

        @tool(atomic=True)
        def file_info(...): ...

        @tool(atomic=True)
        def find_files(...): ...

        @tool(atomic=True)
        def read_file(...): ...

        @tool(atomic=True)
        def bookmark(...): ...

        # Phase 3 Tools (added later):
        # disk_usage, recent_files

        # Phase 4 Tools (added later):
        # compare_files, find_duplicates

class ScratchpadToolsMixin:
    """SQLite scratchpad tools for structured data analysis.

    Gives the agent working memory to accumulate, transform, and query
    data extracted from documents. Enables multi-document analysis
    workflows like financial analysis, tax preparation, research reviews.

    Tool registration follows GAIA pattern: register_scratchpad_tools() method.
    """

    _scratchpad: "ScratchpadService" = None

    def register_scratchpad_tools(self):
        """Register scratchpad tools. Called during agent init."""

        @tool(atomic=True)
        def create_table(...): ...

        @tool(atomic=True)
        def insert_data(...): ...

        @tool(atomic=True)
        def query_data(...): ...

        @tool(atomic=True)
        def list_tables(...): ...

        @tool(atomic=True)
        def drop_table(...): ...

# src/gaia/agents/chat/agent.py

class ChatAgent(
    Agent,
    RAGToolsMixin,
    FileToolsMixin,        # Chat-specific file tools (add_watch_directory)
    ShellToolsMixin,
    FileSystemToolsMixin,  # NEW: replaces FileSearchToolsMixin
    ScratchpadToolsMixin,  # NEW: structured data analysis
):
    """Chat Agent with RAG, file system navigation, data analysis,
    and shell capabilities."""

src/gaia/filesystem/
+-- __init__.py
+-- index.py              # FileSystemIndexService (inherits DatabaseMixin)
+-- map.py                # FileSystemMap dataclass + context rendering
+-- categorizer.py        # Auto-categorization by extension
+-- extractors/
|   +-- __init__.py
|   +-- text.py           # Plain text, code files
|   +-- office.py         # DOCX, PPTX, XLSX (optional deps)
|   +-- pdf.py            # PDF text extraction (wraps existing rag/pdf_utils)
|   +-- pdf_tables.py     # PDF table extraction (pdfplumber extract_tables)
|   +-- image.py          # Image metadata (PIL if available)
+-- chunkers/
    +-- __init__.py
    +-- markdown_chunker.py   # Header/section-aware chunking
    +-- prose_chunker.py      # Paragraph-boundary chunking
    +-- python_chunker.py     # ast module-based Python chunking
    +-- table_chunker.py      # Header-preserving table chunking

src/gaia/scratchpad/
+-- __init__.py
+-- service.py            # ScratchpadService (inherits DatabaseMixin)

@dataclass
class ChatAgentConfig:
    """Configuration for ChatAgent."""

    # ... existing fields ...

    # File System settings (NEW)
    enable_filesystem_index: bool = True        # Enable persistent file index
    filesystem_index_path: str = "~/.gaia/file_index.db"
    filesystem_auto_scan: bool = True           # Quick-scan home on first use
    filesystem_scan_depth: int = 3              # Default scan depth (conservative)
    filesystem_exclude_patterns: List[str] = field(default_factory=list)  # Extra exclusions
    filesystem_content_hashing: bool = False    # Opt-in content hashing for duplicates
    filesystem_watch_bookmarks: bool = True     # Watch bookmarked dirs for changes
    filesystem_map_max_tokens: int = 800        # Token budget for FS map in prompt

gaia fs scan [PATH]         Scan a directory and add to index
  --depth N                 Maximum depth (default: 3)
  --full                    Full scan with content hashing

gaia fs status              Show index statistics
  --verbose                 Show per-directory breakdown

gaia fs search QUERY        Search the file index
  --type EXT                Filter by extension
  --size RANGE              Filter by size (e.g., ">10MB")
  --date RANGE              Filter by date (e.g., "this-week")

gaia fs bookmarks           List saved bookmarks
  --add PATH [--label NAME] Add a bookmark
  --remove PATH             Remove a bookmark

gaia fs tree [PATH]         Show directory tree
  --depth N                 Maximum depth (default: 3)

gaia fs cleanup             Remove stale entries from index
  --days N                  Remove entries older than N days (default: 30)

gaia fs reset               Delete and rebuild the index from scratch

def add_fs_parser(subparsers):
    """Add 'gaia fs' CLI subcommand."""
    fs_parser = subparsers.add_parser("fs", help="File system index management")
    fs_sub = fs_parser.add_subparsers(dest="fs_command")

    # gaia fs scan
    scan = fs_sub.add_parser("scan", help="Scan a directory")
    scan.add_argument("path", nargs="?", default="~")
    scan.add_argument("--depth", type=int, default=3)
    scan.add_argument("--full", action="store_true")

    # gaia fs status
    fs_sub.add_parser("status", help="Show index statistics")

    # ... etc

import platform

# Cross-platform exclusions
EXCLUDE_ALWAYS = [
    ".git", "node_modules", "__pycache__", ".venv", "venv",
    ".cache", ".tmp", "tmp",
]

# Windows-only exclusions
EXCLUDE_WINDOWS = [
    "AppData/Local/Temp",
    "AppData/Local/Microsoft",
    "$Recycle.Bin",
    "System Volume Information",
    "Windows",
    "Program Files",
    "Program Files (x86)",
    "ProgramData",
]

# macOS-only exclusions
EXCLUDE_MACOS = [
    ".Trash",
    "Library/Caches",
    "Library/Application Support",
]

# Linux-only exclusions
EXCLUDE_LINUX = [
    "/proc", "/sys", "/dev", "/tmp",
    ".local/share/Trash",
]

def get_default_exclusions() -> list:
    """Return platform-appropriate exclusion patterns."""
    exclusions = list(EXCLUDE_ALWAYS)
    system = platform.system()
    if system == "Windows":
        exclusions.extend(EXCLUDE_WINDOWS)
    elif system == "Darwin":
        exclusions.extend(EXCLUDE_MACOS)
    elif system == "Linux":
        exclusions.extend(EXCLUDE_LINUX)
    return exclusions

# In pyproject.toml or setup.cfg:
[project.optional-dependencies]
filesystem = [
    "charset-normalizer>=3.0",
    "python-docx>=1.0",
    "python-pptx>=1.0",
    "openpyxl>=3.1",
    "beautifulsoup4>=4.12",
]

tests/
+-- unit/
|   +-- test_filesystem_tools.py      # Tool unit tests
|   +-- test_filesystem_index.py      # Index service unit tests
|   +-- test_filesystem_extractors.py # Extractor unit tests
|   +-- test_filesystem_chunkers.py   # Chunker unit tests
+-- integration/
|   +-- test_filesystem_integration.py  # End-to-end with real FS
|   +-- test_filesystem_cli.py          # CLI command tests
+-- fixtures/
    +-- filesystem/
        +-- sample.pdf
        +-- sample.docx
        +-- sample.xlsx
        +-- sample.csv
        +-- sample.py
        +-- sample.md

# tests/benchmarks/test_filesystem_perf.py

def test_scan_50k_files(tmp_path):
    """Create 50K files and verify scan completes in < 60 seconds."""

def test_fts5_search_latency(populated_index):
    """Verify FTS5 search returns in < 100ms on 50K file index."""

def test_memory_usage_during_scan():
    """Verify memory stays under 100MB during scan of 50K files."""

@tool(atomic=True)
def disk_usage(path: str = "~", depth: int = 2, top_n: int = 15) -> str:
    """Analyze disk usage for a directory.

    Shows which folders and file types are consuming the most space.
    Uses index data when available for fast results.
    """

@tool(atomic=True)
def compare_files(path1: str, path2: str, context_lines: int = 3) -> str:
    """Compare two files or directories.

    For text files, shows a unified diff.
    For directories, shows structural differences (files added/removed/changed).
    """

@tool(atomic=True)
def find_duplicates(
    directory: str = "~", method: str = "hash", min_size: str = "1KB"
) -> str:
    """Find duplicate files by comparing content hashes, names, or sizes.

    Requires content hashing to be enabled (--full scan or config flag).
    Uses size-based pre-filtering to avoid hashing small files.
    """