Blender Agent - GAIA SDK

Source Code: hub/agents/python/blender/gaia_agent_blender/agent.py · src/gaia/mcp/blender_mcp_server.py

GAIA provides a powerful Blender agent that enables natural language interaction with Blender for 3D scene creation and modification. The agent can create objects, apply materials, manipulate transformations, and manage scenes through conversational commands.

First time here? Complete the Setup guide first to install GAIA and its dependencies.

Overview

The GAIA Blender agent bridges the gap between natural language and 3D modeling by:

Natural Language Processing: Understanding complex 3D scene creation requests
Automated Planning: Breaking down complex tasks into manageable steps
Real-time Execution: Direct communication with Blender through MCP (Model Context Protocol)
Interactive Workflows: Supporting both example-based learning and custom queries

Key Features

Scene Management

Clear scenes and remove objects
Get scene information and object listings
Manage scene hierarchy and organization

Object Creation

Primitive Objects: Cubes, spheres, cylinders, cones, and torus objects
Positioning: Precise placement using 3D coordinates
Scaling: Custom sizing and proportions
Multiple Objects: Create and arrange multiple objects in complex scenes

Material System

Color Assignment: Apply RGBA colors to objects
Material Properties: Set material characteristics and appearance
Visual Consistency: Maintain material standards across objects

Interactive Planning

Multi-step Execution: Break complex requests into logical steps
Automatic Planning: AI-driven task decomposition
Progress Tracking: Monitor execution through each step
Error Handling: Graceful handling of invalid operations

Installation & Setup

Prerequisites

Blender Installation: Blender version 4.3+ recommended
GAIA Installation: Core GAIA system must be installed
Lemonade Server: Must be running for AI processing

MCP Server Setup

The Blender agent requires the MCP (Model Context Protocol) server to communicate with Blender:

Step-by-Step Setup:

Open Blender (version 4.3 or newer recommended)
Access Add-ons Menu:
- Go to Edit > Preferences > Add-ons
Install the MCP Server:
- Click the down arrow button, then Install...
- Navigate to: src/gaia/mcp/blender_mcp_server.py
- Select and install the file
Enable the Add-on:
- Find Simple Blender MCP in the add-ons list
- Check the box to enable it
Configure the Server:
- Open the 3D viewport sidebar (press N key if not visible)
- Find the Blender MCP panel in the sidebar
- Set port to 9876 (default) or customize as needed
- Click Start Server
Verify Connection:
- The server status should show as running
- GAIA CLI will validate the connection when starting Blender commands

Visual Setup Guide

For detailed setup instructions with screenshots, see: workshop/blender.ipynb

Command Reference

Basic Command Structure

gaia blender [OPTIONS]

Available Options

Option	Type	Default	Description
`--model`	string	auto-selected by agent	Language model ID for AI processing (not a Blender 3D model). When omitted, the Blender agent picks a model based on your installed profile — typically a Qwen3 variant. Pass e.g. `--model Qwen3.5-35B-A3B-GGUF` to pin the 35B model.
`--example`	int (1-6)	None	Run a specific example, if not specified run interactive mode
`--steps`	int	global limit (50)	Maximum number of steps per query. Defaults to the global agent step limit (50, or `$GAIA_AGENT_MAX_STEPS` if set).
`--output-dir`	string	”output”	Directory to save output files
`--stream`	flag	False	Enable streaming mode for LLM responses
`--stats`	flag	False	Enable statistics collection (stats saved to output files, not displayed)
`--query`	string	None	Custom query to run instead of examples
`--interactive`	flag	False	Enable interactive mode for continuous queries
`--debug-prompts`	flag	False	Enable debug prompts for development
`--print-result`	flag	False	Enable result printing (output depends on query type)
`--mcp-port`	int	9876	Port for the Blender MCP server

Usage Examples

Running Built-in Examples

# Run all Blender examples in sequence
gaia blender

# Run a specific example (1-5 available)
gaia blender --example 2

# Run example with custom model
gaia blender --example 3 --model "custom-model"

Interactive Mode

# Start interactive Blender mode for custom 3D scene creation
gaia blender --interactive

# Interactive mode with debug information
gaia blender --interactive --debug-prompts --output-dir ./blender_results

Custom Queries

# Single custom query to create specific 3D objects
gaia blender --query "Create a red cube and blue sphere arranged in a line"

# Complex scene setup
gaia blender --query "Clear the scene, then create a green cylinder at (0,0,0) and a yellow cone 3 units above it"

# Advanced scene with multiple operations
gaia blender --query "Create a complex 3D scene with multiple colored objects arranged in a circle"

Advanced Configuration

# Use different model with streaming enabled
gaia blender --model "custom-model" --stream --query "Create a sunset scene with mountains"

# Custom MCP port and output directory
gaia blender --mcp-port 9877 --output-dir ./my_scenes --query "Create a modern office space"

# Performance monitoring
gaia blender --stats --steps 10 --query "Build a complete house structure"

Built-in Examples

The Blender agent includes several built-in examples to demonstrate capabilities:

Example 1: Clearing the Scene

Purpose: Remove all objects from the scene Command: Clear the scene to start fresh Learning: Basic scene management and cleanup

Example 2: Creating a Basic Cube

Purpose: Create a red cube at the center Command: Create a red cube at the center of the scene with red material Learning: Object creation and material assignment

Example 3: Creating a Sphere with Properties

Purpose: Blue sphere with custom position and scale Command: Create a blue sphere at position (3, 0, 0) with scale (2, 2, 2) Learning: Positioning, scaling, and color properties

Example 4: Multiple Objects

Purpose: Green cube and red sphere arrangement Command: Create a green cube at (0, 0, 0) and a red sphere 3 units above it Learning: Multi-object scenes and spatial relationships

Positioning Note: Relative positioning (e.g., “3 units above”) may not always position objects as expected. For precise placement, use explicit 3D coordinates like (0, 0, 3) instead.

Example 5: Object Modification

Purpose: Create and then modify a blue cylinder Command: Create a blue cylinder, then make it taller and move it up 2 units Learning: Object modification and transformation workflows Note: Examples 1-5 are currently implemented. Example 6 is planned for future releases.

Agent Capabilities

Scene Management

Clear Scenes: Remove all objects to start fresh
Object Inventory: List and identify existing objects
Scene Information: Get comprehensive scene details
Hierarchy Management: Organize complex scenes

Object Creation & Manipulation

Primitive Objects:
- Cubes with customizable dimensions
- Spheres with radius control
- Cylinders with height and radius settings
- Cones with base and tip configuration
- Torus objects with major and minor radius
Positioning: Precise 3D coordinate placement
Rotation: Object orientation control
Scaling: Non-uniform scaling on X, Y, Z axes

Material & Appearance

Color Assignment: Full RGBA color control
Material Properties: Basic material characteristics
Visual Consistency: Standardized material application
Material Library: Reusable material definitions

Advanced Features

Multi-step Planning: Complex task breakdown
Dependency Resolution: Handle object relationships
Error Recovery: Graceful handling of invalid operations
Progress Monitoring: Step-by-step execution tracking

Interactive Mode

Interactive mode provides a continuous interface for 3D scene creation:

Starting Interactive Mode

gaia blender --interactive

Interactive Commands

Scene Creation: Describe the scene you want to create
Object Modification: Request changes to existing objects
Scene Queries: Ask questions about the current scene
Control Commands:
- Type exit, quit, or q to exit
- Use Ctrl+C for immediate termination

Example Interactive Session

Enter Blender query: Create a red cube at the origin
[Agent processes and creates cube]

Enter Blender query: Add a blue sphere 2 units above the cube
[Agent adds sphere with proper positioning]

Enter Blender query: Make the cube twice as large
[Agent modifies cube scale]

Enter Blender query: exit
Exiting Blender interactive mode.

Requirements & Dependencies

System Requirements

Blender: Version 4.3+ (4.2 may work but not fully tested)
Python: Compatible with GAIA’s Python environment
Memory: Sufficient RAM for both GAIA and Blender (8GB+ recommended)
Storage: Space for 3D scene files and outputs

Software Dependencies

GAIA Core: Full GAIA installation required
Lemonade Server: Must be running for AI processing
Blender MCP Server: Must be installed and running in Blender

Network Requirements

Local Communication: MCP server runs on localhost
Port Availability: Default port 9876 (customizable)
Firewall: Allow local connections on MCP port

Troubleshooting

Lemonade Server Not Running

Error: Connection errors or “server not accessible” messagesSolution:

lemonade-server serve

Blender MCP Server Issues

Error: “Blender MCP server is not running or not accessible”Solutions:

Verify Blender is open with the MCP add-on installed
Check the MCP server is started in Blender’s sidebar panel
Confirm port 9876 is available (or use custom port with --mcp-port)
Restart Blender if the server appears unresponsive

Installation Problems

Error: MCP server add-on not found or won’t installSolutions:

Verify you’re navigating to the correct file: src/gaia/mcp/blender_mcp_server.py
Check Blender version compatibility (4.3+ recommended)
Ensure GAIA is properly installed
Try restarting Blender after installation

Performance Issues

Error: Slow response times or timeoutsSolutions:

Reduce --steps parameter for simpler operations
Use smaller models for faster processing
Ensure adequate system resources
Check for memory constraints in both GAIA and Blender

Port Conflicts

Error: MCP server won’t start due to port conflictsSolutions:

Use --mcp-port to specify an alternative port
Check for other applications using port 9876
Kill conflicting processes: gaia kill --port 9876

Debug Mode

Enable debug mode for detailed troubleshooting:

gaia blender --debug-prompts --interactive

This provides:

Detailed prompt information
Step-by-step execution logs
Error stack traces
Communication details between GAIA and Blender

Getting Help

For additional support:

Check the workshop notebook: workshop/blender.ipynb
Review the Development Guide
Consult the FAQ for common solutions
Enable debug mode to gather detailed information

Advanced Usage

Custom Models

The Blender agent supports different AI models for varying performance characteristics:

# Use a more powerful model for complex scenes
gaia blender --model "Llama-3.2-8B-Instruct" --query "Create a detailed cityscape"

# Use a smaller model for simple operations
gaia blender --model "Llama-3.2-1B-Instruct" --query "Create a red cube"

Batch Operations

Process multiple scenes or operations efficiently:

# Save results to organized output directory
gaia blender --output-dir ./batch_scenes --query "Create scene 1: office setup"

# Use streaming for real-time feedback
gaia blender --stream --query "Build a complex architectural structure"

Output Directory Behavior: The --output-dir creates the specified directory, but output files depend on the specific operations performed. Some queries may not generate output files even with this flag set.

Integration with Other Tools

The Blender agent can be integrated into larger workflows:

Automated Content Creation: Script multiple scene generations
Educational Tools: Demonstrate 3D concepts through natural language
Rapid Prototyping: Quick visualization of 3D ideas
AI-Assisted Modeling: Accelerate traditional 3D modeling workflows

Best Practices

Query Construction

Be Specific: Include precise positions, colors, and sizes
Use Clear Language: Avoid ambiguous terms
Break Complex Requests: Split large scenes into manageable parts
Specify Relationships: Clearly describe object positioning relative to others

Scene Organization

Start Clean: Begin with Clear the scene for predictable results
Logical Progression: Build scenes step by step
Test Incrementally: Verify each step before adding complexity
Use Consistent Naming: Reference objects clearly in follow-up commands

Performance Optimization

Appropriate Models: Match model size to task complexity
Step Limits: Set reasonable --steps values
Resource Monitoring: Use --stats to track performance
Batch Similar Operations: Group related tasks together

Future Enhancements

Planned improvements to the Blender agent include:

Extended Object Library: More primitive types and complex shapes
Advanced Materials: PBR materials, textures, and lighting
Animation Support: Keyframe animation and motion paths
Scene Templates: Pre-built scene configurations
Import/Export: Integration with external 3D file formats
Collaborative Features: Multi-user scene editing capabilities

​Overview

​Key Features

​Scene Management

​Object Creation

​Material System

​Interactive Planning

​Installation & Setup

​Prerequisites

​MCP Server Setup

​Step-by-Step Setup:

​Visual Setup Guide

​Command Reference

​Basic Command Structure

​Available Options

​Usage Examples

​Running Built-in Examples

​Interactive Mode

​Custom Queries

​Advanced Configuration

​Built-in Examples

​Example 1: Clearing the Scene

​Example 2: Creating a Basic Cube

​Example 3: Creating a Sphere with Properties

​Example 4: Multiple Objects

​Example 5: Object Modification

​Agent Capabilities

​Scene Management

​Object Creation & Manipulation

​Material & Appearance

​Advanced Features

​Interactive Mode

​Starting Interactive Mode

​Interactive Commands

​Example Interactive Session

​Requirements & Dependencies

​System Requirements

​Software Dependencies

​Network Requirements

​Troubleshooting

​Advanced Usage

​Custom Models

​Batch Operations

​Integration with Other Tools

​Best Practices

​Query Construction

​Scene Organization

​Performance Optimization

​Future Enhancements

Overview

Key Features

Scene Management

Object Creation

Material System

Interactive Planning

Installation & Setup

Prerequisites

MCP Server Setup

Step-by-Step Setup:

Visual Setup Guide

Command Reference

Basic Command Structure

Available Options

Usage Examples

Running Built-in Examples

Interactive Mode

Custom Queries

Advanced Configuration

Built-in Examples

Example 1: Clearing the Scene

Example 2: Creating a Basic Cube

Example 3: Creating a Sphere with Properties

Example 4: Multiple Objects

Example 5: Object Modification

Agent Capabilities

Scene Management

Object Creation & Manipulation

Material & Appearance

Advanced Features

Interactive Mode

Starting Interactive Mode

Interactive Commands

Example Interactive Session

Requirements & Dependencies

System Requirements

Software Dependencies

Network Requirements

Troubleshooting

Advanced Usage

Custom Models

Batch Operations

Integration with Other Tools

Best Practices

Query Construction

Scene Organization

Performance Optimization

Future Enhancements