DiffuGen - Advanced Local Image Generator with MCP Integration

**New**: Now includes OpenAPI server support and OpenWebUI OpenAPI Tools (OWUI Version 0.60.0 Required) integration for seamless image generation and display in chat interfaces! The OpenAPI is seperate from the MCP server and allowss for initigrations into your own projects!

Table of Contents

• [Introduction](#-introduction)

• [Understanding MCP and DiffuGen](#-understanding-mcp-and-diffugen)

• [Features](#-features)

• [System Requirements](#-system-requirements)

• [Installation](#-installation)

• [IDE Setup Instructions](#-ide-setup-instructions)

• [Usage](#-usage) - [OpenAPI Server Usage](#openapi-server-usage) - [Default Parameters by Model](#default-parameters-by-model) - [Asking a LLM to Generate Images](#asking-a-llm-to-generate-images) - [Parameter Reference](#parameter-reference) - [Model-Specific Parameter Recommendations](#model-specific-parameter-recommendations) - [Default Parameter Changes](#default-parameter-changes) - [Command Line Usage Notes](#command-line-usage-notes)

• [Configuration](# -configuration) - [Configuration Approach](#configuration-approach) - [Environment Variable Overrides](#environment-variable-overrides) - [Setting IDE-Specific Configurations](#setting-ide-specific-configurations) - [Key Configuration Elements](#key-configuration-elements) - [IDE-Specific Options](#ide-specific-options) - [Customizing Default Parameters](#customizing-default-parameters) - [Updating Configuration Files](#updating-configuration-files)

• [Advanced Usage](#-advanced-usage) - [Using the OpenAPI Server](#using-the-openapi-server)

• [License](#-license)

• [Acknowledgments](#-acknowledgments)

• [Contact](#-contact)

Introduction

Understanding MCP and DiffuGen

What is MCP?

DiffuGen as an MCP Server

1. `generate_stable_diffusion_image`: Generate with Stable Diffusion models

2. `generate_flux_image`: Generate with Flux models

Technical Architecture

• **setup-diffugen.sh**: The complete install utility and model downloader and manager

• **diffugen.py**: The core Python script that implements the MCP server functionality and defines the generation tools

• **diffugen.sh**: A shell script launcher that sets up the environment and launches the Python server

• **diffugen.json**: Template configuration file for MCP integration with various IDEs (to be copied into IDE's MCP configuration)

• **stable-diffusion.cpp**: The optimized C++ implementation of Stable Diffusion used for actual image generation

1. Receiving prompt and parameter data from an MCP client

2. Processing the request through the Python server

3. Calling the stable-diffusion.cpp binary with appropriate parameters

4. Saving the generated image to a configured output directory

5. Returning the path and metadata of the generated image to the client

About stable-diffusion.cpp

• Significantly faster inference speed (up to 3-4x faster)

• Lower memory usage (works on GPUs with as little as 4GB VRAM)

• Optimized CUDA kernels for NVIDIA GPUs

• Support for various sampling methods and model formats

• Support for model quantization for better performance

• No Python dependencies for the core generation process

Features

• **Multiple Model Support**: Generate images using various models including Flux Schnell, Flux Dev, SDXL, SD3, and SD1.5

• **MCP Integration**: Seamlessly integrates with IDEs that support MCP (Cursor, Windsurf, Roo Code, Cline, etc.)

• **OpenAPI Server**: Additional REST API interface for direct HTTP access to image generation capabilities

• **Cross-Platform**: Works on Linux, macOS, and Windows (via native or WSL)

• **Parameter Control**: Fine-tune your generations with controls for: - Image dimensions (width/height) - Sampling steps - CFG scale - Seed values - Negative prompts (for SD models only, Flux does not support negative prompts.) - Sampling methods

• **CUDA Acceleration**: Utilizes GPU acceleration for faster image generation

• **Natural Language Interface**: Generate images using simple natural language commands

• **Smart Error Recovery**: Robust error handling with operation-aware recovery procedures

• **User-Friendly Setup**: Interactive setup script with improved interrupt handling

• **Resource Tracking**: Session-aware resource management for efficient cleanup

• **Customizable Interface**: Support for custom ANSI art logos and visual enhancements

System Requirements

Minimum Requirements:

• **CPU**: 4-core processor (Intel i5/AMD Ryzen 5 or equivalent)

• **RAM**: 8GB system memory

• **Storage**: 5GB free disk space (SSD preferred for faster model loading)

• **Python**: 3.8 or newer

• **GPU**: Integrated graphics or entry-level dedicated GPU (optional)

• **Network**: Broadband connection for model downloads (5+ Mbps)

Recommended Requirements:

• **CPU**: 8+ core processor (Intel i7/i9 or AMD Ryzen 7/9)

• **RAM**: 16GB+ system memory

• **GPU**: NVIDIA GPU with 6GB+ VRAM (RTX 2060 or better for optimal performance)

• **Storage**: 20GB+ free SSD space

• **Python**: 3.10 or newer (3.11 offers best performance)

• **Network**: High-speed connection (20+ Mbps) for efficient model downloads

Installation

Automatic Installation (Recommended)

• Install necessary dependencies

• Clone and build stable-diffusion.cpp

• Set up a Python virtual environment

• Download selected models (Note: Some models require Clip\VAE Models as well)

• Configure file paths for your system

Manual Installation

1. Clone the repositories:

1. Build stable-diffusion.cpp:

1. Create and activate a Python virtual environment:

1. Download required models (structure shown below):

1. Update file paths in configuration:

1. Edit `diffugen.json` in the DiffuGen root directory with your desired settings

2. Run option 5 in `setup_diffugen.sh` to automatically update paths in this file

3. Copy the content of `diffugen.json` to your IDE's MCP configuration file

• File paths (command, SD_CPP_PATH, models_dir, output_dir)

• Default model parameters (steps, cfg_scale, sampling_method)

• VRAM usage settings

• Metadata for IDE integration

IDE Setup Instructions

Setting up with Cursor

1. Download and install [Cursor](https://cursor.sh)

2. Go to Cursor Settings > MCP and click "Add new global MCP server"

3. **Copy the contents of your DiffuGen's `diffugen.json` file** and paste it into `~/.cursor/mcp.json`

4. Refresh MCP Servers in Settings > MCP

5. Use DiffuGen by opening the AI chat panel (Ctrl+K or Cmd+K) and requesting image generation

Setting up with Windsurf

1. Download and install [Windsurf](https://codeium.com/windsurf)

2. Navigate to Windsurf > Settings > Advanced Settings or Command Palette > Open Windsurf Settings Page

3. Scroll down to the Cascade section and click "Add Server" > "Add custom server +"

4. **Copy the contents of your DiffuGen's `diffugen.json` file** and paste into `~/.codeium/windsurf/mcp_config.json`

5. Use DiffuGen through the Cascade chat interface

Setting up with Roo Code

1. Download and install [Roo Code](https://roo.ai)

2. Locate the MCP configuration file for Roo Code

3. **Copy the contents of your DiffuGen's `diffugen.json` file** into Roo Code's MCP configuration

4. Use DiffuGen through the AI assistant feature

Setting up with Cline

1. Download and install [Cline](https://cline.live)

2. **Copy the contents of your DiffuGen's `diffugen.json` file** into Cline's MCP settings

3. Use DiffuGen through the AI chat or command interface

Setting up with Claude in Anthropic Console

Usage

OpenAPI Server Usage

• Direct HTTP API access

• Integration with other tools that don't support MCP

• Custom applications that need programmatic access

• Host: 0.0.0.0

• Port: 8080

OpenWebUI Integration

1. Open OpenWebUI Settings (gear icon)

2. Navigate to the "Tools" section

3. Click the "+" button to add a new tool server

4. Enter the following details: - URL: http://0.0.0.0:5199 - API Key: (leave empty)

5. Click "Save"

• `generate_stable_image_generate_stable_post`: Generate with Stable Diffusion

• `generate_flux_image_endpoint_generate_flux_post`: Generate with Flux Models

• `list_models_models_get`: List Available Models

Default Parameters by Model

**Note**: For model-specific command line examples and recommendations, see [Model-Specific Parameter Recommendations](#model-specific-parameter-recommendations) section.

Asking a LLM to Generate Images

Basic Requests:

With Model Specification:

With Advanced Parameters:

Parameter Reference

• **In MCP/AI Assistant**: `parameter=value` (e.g., `model=sdxl, width=768, height=512`)

• **In Command Line**: `--parameter value` (e.g., `--model sdxl --width 768 --height 512`)

Model-Specific Parameter Recommendations

**Note**: These recommendations build on the [Default Parameters by Model](#default-parameters-by-model) section and provide practical examples.

Flux Models (flux-schnell, flux-dev)

Standard SD Models (sdxl, sd3, sd15)

Default Parameter Changes

• Default Model: If not specified, function-appropriate models are used (flux-schnell for Flux functions, sd15 for SD functions)

• Default Sampling Method: `euler` (best for Flux models)

• Default CFG Scale: `1.0` for Flux models, `7.0` for standard SD models

• Default Steps: `8` for flux-schnell, `20` for other models

• Default Dimensions: 512x512 pixels

Command Line Usage Notes

• Generated images are saved to the configured output directory with filenames based on timestamp and parameters

• You can generate multiple images in sequence by running the command multiple times

• For batch processing, consider creating a shell script that calls DiffuGen with different parameters

• To see all available command-line options, run `./diffugen.sh --help`

• The same engine powers both the MCP interface and command-line tool, so quality and capabilities are identical

Configuration

Configuration Approach

1. **Primary Configuration File**: `diffugen.json` in the DiffuGen root directory is the single source of truth for all settings

2. **IDE Integration**: Copy the contents of `diffugen.json` to your IDE's MCP configuration file

3. **Environment Variables**: For advanced usage, you can override settings with environment variables

Environment Variable Overrides

• `SD_CPP_PATH`: Override the path to stable-diffusion.cpp

• `DIFFUGEN_OUTPUT_DIR`: Override the output directory

• `DIFFUGEN_DEFAULT_MODEL`: Override the default model

• `DIFFUGEN_VRAM_USAGE`: Override VRAM usage settings

• `CUDA_VISIBLE_DEVICES`: Control which GPUs are used for generation

Setting IDE-Specific Configurations

1. Environment variables (highest priority)

2. Settings from local `diffugen.json` file (base configuration)

Key Configuration Elements

Command and Arguments

• **command**: Full path to the `diffugen.sh` script (must be absolute path)

• **args**: Additional command-line arguments to pass to the script (usually left empty)

Environment Variables

• **CUDA_VISIBLE_DEVICES**: Controls which GPUs are used for generation - `"0"`: Use only the first GPU - `"1"`: Use only the second GPU - `"0,1"`: Use both first and second GPUs - `"-1"`: Disable CUDA and use CPU only

• **SD_CPP_PATH**: Path to the stable-diffusion.cpp installation directory - This is used to locate the stable-diffusion.cpp binary and models

• **default_model**: The default model to use when none is specified

Resource Configuration

• **models_dir**: Directory containing the model files - Should point to the `models` directory inside your stable-diffusion.cpp installation

• **output_dir**: Directory where generated images will be saved - Must be writable by the user running DiffuGen

• **vram_usage**: Controls VRAM usage strategy - `"adaptive"`: Automatically adjust memory usage based on available VRAM - `"minimal"`: Use minimal VRAM at the cost of speed - `"balanced"`: Balance memory usage and speed (default) - `"maximum"`: Use maximum available VRAM for best performance

IDE-Specific Options

Cursor Options

Windsurf Options

Customizing Default Parameters

Updating Configuration Files

1. Run option 5 in `setup_diffugen.sh` to update paths in `diffugen.json`

2. Copy the entire contents of the generated `diffugen.json` file

3. Paste it into your IDE's MCP configuration file (e.g., `~/.cursor/mcp.json`)

4. Restart your IDE to apply changes

Advanced Usage

Using the OpenAPI Server

Troubleshooting

Common Issues and Solutions

1. **Missing models or incorrect paths** - Ensure all model files are downloaded and placed in the correct directories - Check that paths in the configuration file are correctly set - Verify file permissions allow read access to model files

2. **CUDA/GPU issues** - Make sure your NVIDIA drivers are up-to-date - Set `CUDA_VISIBLE_DEVICES` to target a specific GPU - If running out of VRAM, try using a smaller model or reducing dimensions

3. **Image quality issues** - Increase steps for better quality (at the cost of generation time) - Adjust CFG scale: higher for more prompt adherence, lower for creativity - Try different sampling methods (dpm++2m often provides good results) - Use more detailed prompts with specific style descriptions

4. **File permission errors** - Ensure the output directory is writable by the user running DiffuGen - Check that all scripts have execution permissions (`chmod +x diffugen.sh`)

Getting Help

• Check the GitHub repository for issues and solutions

• Run with debug logging enabled: `DEBUG=1 ./diffugen.sh "your prompt"`

• Contact the developers via GitHub issues

Contributing

1. Fork the repository

2. Create a feature branch

3. Make your changes

4. Submit a pull request

License

• All models are licensed under their respective distribution and are not in any way licensed or provided by CLOUDWERX.DEV

• HuggingFace.co is used to download models and is not affiliated in any way with CLOUDWERX.DEV

Acknowledgments

• [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) for the optimized C++ implementation

• [Stability AI](https://stability.ai/) for Stable Diffusion models

• [Black Forest Labs](https://blackforestlabs.ai/) for their Flux Models

• [Hugging Face](https://huggingface.co/) for the download links

• All contributors to the MCP protocol

Contact

• GitHub: [CLOUDWERX-DEV](https://github.com/CLOUDWERX-DEV)

• Website: [cloudwerx.dev](http://cloudwerx.dev)

• Mail: [sysop@cloudwerx.dev](mailto:sysop@cloudwerx.dev)

• Discord: [Join our server](https://discord.gg/SvZFuufNTQ)