Skip to main content

SD Agent

Status: Partially shipped — a Stable Diffusion agent already ships today (gaia sd, SD guide; source in hub/agents/python/sd/). This document describes the aspirational superset (VLM-driven iterative refinement, gallery UI, task queue) and its PromptAgent/src/gaia/agents/sd/ references are design-time names, not the shipped layout. Priority: Medium Vote with 👍 on GitHub Issue #272

Overview

An AI agent that helps users generate better Stable Diffusion images through intelligent optimization of both prompts and generation parameters. The agent analyzes user intent, enhances text descriptions, and recommends optimal settings (model, size, steps, cfg_scale) for high-quality results. Triple Optimization Approach:
  1. Prompt Enhancement: Transform simple text (“a cat”) into detailed, effective SD prompts
  2. Parameter Optimization: Recommend model selection, dimensions, inference steps, and guidance scale
  3. VLM-Powered Iteration: Analyze generated images with Vision LLM, score quality across categories, and automatically iterate until quality threshold is met
Key Features:
  • LLM-powered prompt analysis and enhancement (AMD NPU-accelerated)
  • Intelligent parameter recommendation (model, size, steps, cfg_scale)
  • VLM-powered image evaluation (composition, lighting, prompt adherence, style, technical quality)
  • Autonomous iteration loop - generate → evaluate → refine → regenerate until quality threshold met
  • Template library with proven patterns
  • A/B testing and strategy comparison
  • Terminal image display for immediate visual feedback
  • SQLite database for generation history
  • Agent-powered search and filtering (natural language queries)
  • Web gallery UI with task-based interface for browsing, annotating, and rating
  • Version control and reproducibility
Goal: Make professional-quality Stable Diffusion image generation accessible through intelligent optimization of all generation parameters, with a searchable database and gallery for managing your creations. The agent learns from your ratings to personalize future recommendations based on your unique aesthetic preferences.
SD Agent Gallery UI mockup showing task queue, generation progress, and image gallery with ratings
View the interactive mockup for a full preview of the gallery UI.

System Architecture

High-Level Overview

Data Flow

Generation Workflow (with VLM Iteration Loop):
  1. User Request → CLI/Task Interface submits generation task
  2. Prompt Enhancement → Agent sends original prompt to LLM
  3. Parameter Optimization → Agent recommends SD parameters based on prompt + user preferences
  4. Image Generation → Agent calls Lemonade SD endpoint
  5. VLM Evaluation → Image Evaluator analyzes output using Qwen3-VL-4B
    • Scores across categories: composition, lighting, prompt adherence, style consistency, technical quality
    • Returns overall score (1-10) + category breakdown + improvement suggestions
  6. Iteration Decision → If score < threshold (default 7/10) AND iterations < max:
    • VLM feedback refines prompt and/or parameters
    • Return to step 4 (regenerate with improvements)
  7. Storage → Agent saves final image + all iterations to SQLite + file system
  8. Display → Final image shown in terminal or Gallery UI with quality report
  9. Learning → Successful patterns stored for future preference learning
Search Workflow:
  1. User Query (natural language) → “show me all cyberpunk cities”
  2. LLM Translation → Agent converts to SQL query
  3. Database Query → Execute against SQLite
  4. Results → Return matching generations with images
VLM Evaluation Workflow:
  1. Image Input → Generated image passed to VLM (Qwen3-VL-4B)
  2. Multi-Category Scoring → VLM evaluates:
    • Composition (1-10): Rule of thirds, balance, focal point
    • Lighting (1-10): Consistency, mood, shadows/highlights
    • Prompt Adherence (1-10): How well image matches the prompt
    • Style Consistency (1-10): Coherent artistic style throughout
    • Technical Quality (1-10): Sharpness, artifacts, resolution
  3. Overall Score → Weighted average of categories
  4. Improvement Suggestions → VLM provides specific feedback for refinement
  5. Iteration Trigger → If below threshold, suggestions feed back into enhancement loop
Personalization Workflow:
  1. User Rates Images → 1-5 stars stored in database
  2. Pattern Analysis → Agent analyzes high-rated generations
  3. Preference Learning → Identify preferred styles, models, parameters
  4. Future Enhancements → Bias recommendations toward learned preferences

Technical Decisions


Architecture

Component Structure

Database Schema

Class Hierarchy


User Experience

Mode 1: Prompt Enhancement Only

Use Case: Optimize prompts before generating
Terminal Output:

Mode 2: Full Generation Pipeline

Use Case: Generate with automatic quality iteration
Terminal Output:
With Iteration (below threshold):

Mode 3: Task Queue

Use Case: Batch processing with autonomous execution
Terminal Output:
View task details:

Mode 4: Strategy Comparison

Use Case: A/B test different styles with VLM scoring
Terminal Output:

Mode 5: Download Images

Use Case: Export images from gallery to Downloads folder
Terminal Output:
Gallery UI: Each image card displays a download button. Clicking opens a format picker (PNG/JPEG) and saves to the user’s Downloads folder.

Core Features

1. LLM-Powered Prompt Enhancement Engine

Problem: Users struggle to write effective prompts that produce desired results from AI systems (SD, LLMs, etc.). Prompt engineering is a specialized skill. Solution: Use GAIA’s AMD NPU-accelerated LLM to analyze user intent and generate optimized prompts using domain-specific best practices. Core Capabilities:

A) Intent Analysis

B) Domain-Specific Enhancement

Domain-Specific Strategies: Enhancement Techniques:
User Control:
  • --auto (default): Fully automated enhancement
  • --suggest: Show suggestions, let user pick
  • --interactive: Collaborative refinement
  • --no-enhance: Use prompt as-is
  • --style <style>: Force specific artistic style

2. Prompt Quality Scoring & Analysis

Goal: Provide objective feedback on prompt quality before generation. Scoring Criteria (for Stable Diffusion): Scoring Implementation:
Visual Scoring Display:

3. Terminal Image Display & Verification

Goal: Provide immediate visual feedback on prompt quality by generating test images in-terminal. Use Case: After enhancing a prompt, quickly verify it produces desired results without leaving the CLI. Terminal Support: Implementation:
Fallback Strategy:
  1. Try native terminal image protocol
  2. Fall back to Unicode block art (better than ASCII)
  3. Finally, open image in default viewer
Rich Integration:

4. Prompt Template Library

Goal: Codify and reuse successful prompt patterns. Template Structure:
Template Commands:
Built-in Template Categories:

5. Iterative Refinement Workflow

Goal: Progressively improve prompts through multiple enhancement cycles. Workflow:
Example Refinement Session:

6. Prompt Comparison & A/B Testing

Goal: Empirically determine which prompt strategies work best.
Comparison Output:

7. Prompt Version Control & History

Goal: Track prompt evolution and enable rollback. Version Structure:
Version Commands:

8. Image Storage & Verification Cache

Cache Structure:
Prompt Metadata Format (prompt.json):
Benefits:
  • Reproducible generations (same seed = same image)
  • Easy prompt history tracking
  • Performance benchmarking data
  • Version control friendly (JSON diff)
Cache Management:

9. Prompt Testing with Visual Variations

Goal: Test prompt robustness by generating multiple images with different seeds. Use Case: A good prompt should produce consistently good results across different seeds.
Consistency Test Output:

Agent Implementation

PromptAgent Class

System Prompt


CLI Integration

Command Structure

Command Handlers


Lemonade Client Extension

Add Image Generation Method


Dependencies

New Libraries Required

Optional Dependencies

Notes:
  • Pillow: For image manipulation, format conversion
  • term-image: Cross-platform terminal image rendering (supports sixel, iTerm2, Kitty)
  • Both are lightweight and well-maintained

Testing Strategy

Unit Tests

Integration Tests

CLI Tests


Documentation Requirements

User Guide

Create docs/guides/image.mdx:
  • Getting started with image generation
  • Prompt writing best practices
  • Model selection guide
  • Parameter tuning tips
  • Examples gallery

SDK Reference

Create docs/sdk/agents/image-agent.mdx:
  • ImageAgent API reference
  • Tool specifications
  • Code examples for programmatic use

CLI Reference

Update docs/reference/cli.mdx:
  • Add gaia image command documentation
  • All flags and options
  • Usage examples

Implementation Plan

Phased Approach: Terminal CLI → Web Gallery UI

Phase 1: Core Optimization Engine (Week 1)

Goal: SD prompt enhancement + parameter optimization working in CLI
  • Create SDAgent class skeleton (Agent + DatabaseMixin)
  • Initialize SQLite database with schema (generations, templates, prompt_versions, evaluations, task_queue)
  • Implement PromptEnhancer class with LLM backend (AMD NPU)
  • Implement PromptAnalyzer class for quality scoring (1-10)
  • Implement ParamOptimizer class for SD parameter recommendations
  • Add analyze_prompt tool
  • Add enhance_prompt tool
  • Add optimize_parameters tool (model, size, steps, cfg_scale)
  • Add basic CLI commands (gaia sd analyze, gaia sd enhance)
  • Unit tests for enhancer, analyzer, and optimizer
Deliverable: gaia sd enhance "a mountain" produces enhanced prompt + recommended parameters with quality score

Phase 2: Image Generation + VLM Evaluation (Week 2)

Goal: Full generation pipeline with VLM-powered quality iteration
  • Extend LemonadeClient with generate_image() method
  • Create ImageGenerator wrapper class
  • Implement ImageEvaluator class (VLM-powered using Qwen3-VL-4B)
    • Score across 5 categories: composition, lighting, prompt adherence, style, technical
    • Return overall score + improvement suggestions
  • Implement IterationController (generate → evaluate → refine loop)
    • Configurable quality threshold (default 7/10)
    • Max iterations limit (default 3)
    • Tracks all iterations in database
  • Implement generate_image tool (full pipeline: enhance → optimize → generate → evaluate → iterate)
  • Add evaluate_image and iterate_until_quality tools
  • Save all generations + evaluations to database
  • Implement image file storage (.gaia/cache/sd/images/)
  • Create TerminalDisplay class for in-terminal image preview
    • Sixel support (Windows Terminal)
    • iTerm2/Kitty support
    • Fallback to external viewer
  • Add gaia sd generate command (with --quality-threshold and --max-iterations flags)
  • Add gaia sd history command (list recent generations from DB)
  • Integration tests with Lemonade Server (LLM + VLM + SD)
Deliverable: gaia sd generate "mountain" enhances, generates, evaluates with VLM, iterates if needed, displays final result with quality report

Phase 3: Templates & Search (Week 3)

Goal: Template library and natural language search
  • Implement TemplateLibrary class (DB-backed)
  • Build starter template set (10+ templates) with prompt+parameter combos
    • Photography styles (portrait, landscape, macro)
    • Artistic styles (photorealistic, anime, oil-painting, watercolor)
    • Genre templates (cyberpunk, fantasy, sci-fi, horror)
  • Add template tools: list_templates, use_template, save_as_template
  • Implement natural language search tools:
    • search_generations (e.g., “show me all cyberpunk images”)
    • filter_by_params (e.g., “find images generated with SDXL-Turbo”)
    • get_favorites, get_top_rated
  • Add CLI commands: gaia sd templates, gaia sd use, gaia sd search
  • LLM-powered query translation (natural language → SQL)
  • Unit tests
Deliverable: Users can browse templates, use proven patterns, and search generation history with natural language Goal: Standalone web UI for task-based image creation and gallery management UI Components:
  • Gallery Server (Flask/FastAPI)
    • REST API for CRUD operations on generations
    • WebSocket for real-time generation progress updates
    • Static file serving for images
  • Task Submission Interface
    • Natural language input: “a cyberpunk city at night, neon lights”
    • Optional parameter locks: hardwire model, size, steps, seed (override agent recommendations)
    • Submit task → agent processes autonomously → returns result
    • Live progress indicator (enhancing → generating → evaluating → iterating)
    • Quality score display with category breakdown
    • Iteration history (show all attempts if multiple iterations)
  • Gallery View
    • Grid/list view of all generations
    • Filter controls (model, size, date range, rating)
    • Natural language search box
    • Sort by date, rating, favorites
  • Image Detail View
    • Full-size image display
    • Prompt and parameters display
    • Rating system (1-5 stars)
    • Notes/annotations text area
    • Tags editor
    • Favorite toggle
    • Actions: regenerate, refine, save as template
  • Task Queue System
    • Implement TaskQueue class with SQLite persistence
    • Submit multiple tasks to queue (natural language + optional parameter locks)
    • Agent processes tasks sequentially (or parallel if resources allow)
    • Queue status display (pending, in-progress, completed, failed)
    • Priority ordering (urgent tasks jump queue)
    • Cancel/pause/resume individual tasks
    • Batch submission (“generate 5 variations of this prompt”)
    • WebSocket notifications when tasks complete
  • Reference-Based Generation
    • Agent can retrieve top-rated images
    • Use high-rated prompts/parameters as inspiration
    • “Generate something similar to my favorite landscapes”
  • Template Browser
    • Browse available templates
    • Preview example images
    • Quick-apply to new generation
Technical Stack:
  • Backend: FastAPI + SQLite (via DatabaseMixin)
  • Task Queue: In-memory queue with SQLite persistence for recovery
  • Frontend: React/Vue + Tailwind CSS
  • Communication: REST API + WebSockets for live updates
  • Packaging: Electron wrapper for desktop app
Web UI Mockup:
Deliverable: Standalone UI at http://localhost:5000 with task-based image creation, queue management, and searchable gallery

Phase 5: Advanced Features & Polish (Week 5)

Goal: Production-ready with full feature set CLI Enhancements:
  • Interactive mode (gaia sd with no args → task submission interface)
  • Comparison mode (gaia sd compare "dragon" --strategies photorealistic,anime)
  • Batch generation (gaia sd batch "prompt" --count 10 --vary-params)
  • Queue management (gaia sd queue status, gaia sd queue cancel <id>)
  • Export tools (JSON, CSV, ZIP with images)
Gallery UI Enhancements:
  • Keyboard shortcuts
  • Bulk operations (tag multiple, export selection, delete)
  • Advanced filters (tag combinations, parameter ranges)
  • Gallery statistics (total images, by model, avg rating)
  • Settings page (default parameters, UI preferences)
Quality & Performance:
  • Performance optimization (query caching, lazy loading)
  • Error handling and user-friendly messages
  • Loading states and progress indicators
  • Image thumbnails for faster gallery loading
  • Database optimization (indexes, cleanup old entries)
Documentation:
  • User guide (docs/guides/sd-agent.mdx)
  • SDK reference (docs/sdk/agents/sd-agent.mdx)
  • Update CLI reference (docs/reference/cli.mdx)
  • Gallery UI guide
  • Prompt engineering best practices
  • Example gallery showcase
Testing:
  • Full test coverage (unit, integration, E2E)
  • Performance benchmarks
  • UI testing (Playwright/Cypress)
Deliverable: Production-ready SD optimization agent with CLI + Gallery UI, fully documented and tested

Future Enhancements

Advanced Prompt Engineering Features

  1. Multi-Domain Expansion
    • LLM Prompts: Chain-of-thought, few-shot, role-playing optimization
    • Code Generation: Language-specific patterns, framework templates
    • Vision Models: VLM-specific prompt engineering (Qwen2-VL, etc.)
    • Audio Models: TTS/ASR prompt optimization (Whisper, Kokoro)
  2. Collaborative Prompt Engineering
    • Team Templates: Shared prompt libraries across organization
    • Version Control Integration: Git-style branching for prompts
    • Feedback Loop: Track which prompts perform best over time
    • Prompt Marketplace: Share/discover templates from community
  3. Advanced Analysis
    • Semantic Similarity: Find similar successful prompts
    • Performance Tracking: Which styles/keywords correlate with quality
    • Automated A/B Testing: Run overnight experiments
    • CLIP Score Integration: Objective image-prompt alignment scoring
  4. MCP Integration
    • Expose prompt enhancement as MCP tool
    • Integration with VSCode, Claude Desktop, etc.
    • Real-time prompt suggestions in external editors

Image Generation Enhancements

  1. Advanced SD Features
    • Negative Prompts: Specify what NOT to include
    • Prompt Weights: Control emphasis on different elements
    • ControlNet Support: Pose, depth, edge guidance
    • LoRA Integration: Custom model fine-tuning
    • Image-to-Image: Style transfer, variations from reference
    • Inpainting/Outpainting: Edit specific regions
  2. Multi-Model Support
    • Support for multiple SD checkpoints
    • FLUX, Midjourney-style prompts
    • Cross-model prompt translation
    • Model recommendation based on use case
  3. Batch Operations
    • Grid Search: Systematically test parameter combinations
    • Style Exploration: Generate matrix of style variations
    • Parameter Optimization: Find best steps/cfg_scale for prompt
    • Scheduled Generation: Queue overnight batch jobs

Integration & Collaboration

  1. Agent Ecosystem Integration
    • PromptAgent + BlenderAgent: Generate texture prompts for 3D scenes
    • PromptAgent + CodeAgent: Generate documentation with illustrations
    • PromptAgent + ChatAgent: Enhance chat responses with visuals
    • PromptAgent + JiraAgent: Create visual mockups from issue descriptions
  2. UI/UX Enhancements
    • Web UI: Browser-based prompt engineering workspace
    • Electron App: Desktop app with drag-drop, galleries
    • Mobile Companion: Review generations, rate prompts on mobile
    • Browser Extension: Enhance prompts for SD web UIs (AUTOMATIC1111, ComfyUI)
  3. Export & Publishing
    • Prompt Cards: Beautiful shareable images of prompt+result
    • Portfolio Export: Generate HTML galleries
    • API Access: Programmatic prompt enhancement API
    • Webhook Integration: Notify on completion, feed to other systems

Research & Experimental Features

  1. LLM-as-Judge
    • Use LLM to rate generated images
    • Automated quality assessment
    • Suggest prompt improvements based on output
  2. Reinforcement Learning
    • Learn from user preferences over time
    • Personalized prompt enhancement
    • Adapt to individual artistic style
  3. Cross-Modal Prompt Engineering
    • Text → Image → Text (caption generated images)
    • Video prompts (SD animation)
    • 3D prompt engineering (for 3D generative models)
  4. Educational Features
    • Prompt Engineering Tutor: Interactive lessons
    • Challenge Mode: Daily prompt challenges
    • Skill Progression: Track improvement over time

Success Metrics

Performance Targets

Quality Metrics

User Experience Goals

Adoption Metrics


Open Questions

Technical Decisions Needed

  1. Prompt Enhancement Philosophy:
    • How aggressive should auto-enhancement be?
    • Always show before/after diff, or hide unless --show-enhancement?
    • Should enhancement preserve exact user phrasing or fully rewrite?
    • Support for multiple enhancement styles (conservative, creative, etc.)?
  2. Scoring Algorithm:
    • Use LLM-as-judge or rule-based scoring?
    • How to weigh different criteria (clarity vs. detail vs. style)?
    • Should scores be domain-specific or universal?
    • Calibrate scores against human ratings?
  3. Template System Design:
    • JSON vs. Jinja2 templates vs. custom DSL?
    • How much flexibility vs. simplicity?
    • Support for nested/composed templates?
    • Template versioning and updates?
  4. Cache Management:
    • Max cache size before cleanup warnings?
    • LRU eviction vs. user-controlled deletion?
    • Compress old metadata or keep full history?
    • Export/import cache across machines?
  5. SD Integration Scope:
    • Support only Lemonade Server or also AUTOMATIC1111, ComfyUI?
    • Implement image-to-image or MVP text-to-image only?
    • Support for custom checkpoints/LoRAs?
  6. AMD Hardware Optimization:
    • Auto-detect NPU and adjust LLM model selection?
    • Warn if running on CPU-only?
    • Benchmark mode to showcase AMD performance?

Product Decisions

  1. Target Audience Priority:
    • Beginners (teach prompt engineering) or experts (power tools)?
    • Both? How to balance?
  2. Domain Expansion:
    • Launch with SD-only or include LLM/code from start?
    • Which domain after SD? (LLM, code, VLM, audio?)
  3. UI Strategy:
    • CLI-only MVP or invest in web UI early?
    • Electron app before or after web UI?
    • Terminal UI (TUI) with rich interactive widgets?
  4. Community Features:
    • Public template marketplace?
    • Share prompts anonymously or with attribution?
    • Rating/review system for templates?
    • Moderation approach?
  5. Integration Priority:
    • Which GAIA agent integration first?
      • BlenderAgent (texture prompts)
      • ChatAgent (illustrated responses)
      • CodeAgent (UI mockups)
    • External integration (VSCode, Claude Desktop, etc.)?
  6. Monetization/Sustainability:
    • Open source all features or premium tier?
    • Cloud service for prompt analysis (privacy concerns)?
    • Commercial template packs?
  7. Documentation Approach:
    • Auto-generate templates from successful prompts?
    • Interactive tutorials vs. static docs?
    • Video content priority?

References

External Documentation

Prompt Engineering: Stable Diffusion: Terminal Display:

Internal GAIA Specs

Core Framework: Related Agents:

Academic & Research

Similar Implementations

Within GAIA:
  • BlenderAgent: Domain-specific tool enhancement (3D scene generation)
  • SummarizerAgent: Multi-file processing, caching patterns
  • ChatAgent: Conversational refinement, RAG integration
  • RoutingAgent: Agent selection based on analysis
External Tools:
  • Midjourney: /imagine command with prompt engineering
  • DALL-E: Prompt suggestions and variations
  • ChatGPT: System prompts and role optimization

Approval Checklist

Planning & Scope

  • Problem statement clear (prompt engineering accessibility)
  • User experience defined (multiple workflows)
  • Primary use case identified (Stable Diffusion)
  • Secondary use cases documented (LLM, code, future)
  • User personas considered (beginner to expert)

Technical Design

  • Architecture designed (PromptAgent + components)
  • Technical decisions documented
  • Integration points identified (LLM, SD, cache)
  • Dependencies listed (Pillow, term-image)
  • AMD NPU optimization strategy defined

Implementation

  • Implementation plan with 5 phases
  • Clear milestones and deliverables
  • Testing strategy defined (unit, integration, CLI)
  • Performance targets established
  • Error handling approach outlined

Documentation & Quality

  • Documentation requirements listed
  • Success metrics established
  • Quality benchmarks defined
  • User adoption goals set
  • Open questions documented

Approval & Next Steps

  • AMD stakeholder review
  • Product team review
  • Engineering team capacity confirmed
  • Timeline approved
  • Ready for Phase 1 implementation

Document Version: 1.0 Last Updated: 2026-01-26 Author: Claude Sonnet 4.5 (with kalin) Status: Awaiting approval Next Steps:
  1. Review with AMD team
  2. Finalize open questions
  3. Confirm resource allocation
  4. Begin Phase 1: Core Prompt Analysis & Enhancement