Multimodal

• Vision input -- send images (URLs, files, bytes) to any vision-capable LLM
• Audio input -- send audio files to speech-capable models
• Image generation -- DALL-E, Flux, Stable Diffusion via LiteLLM and fal.ai
• Speech generation -- TTS via OpenAI, fal.ai
• Video generation — text-to-video and image-to-video via fal.ai and OpenRouter (all SDKs)
• Music generation — instrumental tracks via Google Lyria 3 Pro (Python)
• Unified response — MultimodalResponse / MediaResponse with .text, .images, .audio, .videos, .files

The Python SDK provides convenience functions for building multimodal prompts.

from agentfield.multimodal import (
    text, image_from_url, image_from_file,
    audio_from_file, file_from_path,
)

# Mix text and images
response = await app.ai(
    text("Compare the architectural styles:"),
    image_from_file("./building_a.jpg"),
    image_from_file("./building_b.jpg"),
    text("Which one is Art Deco?"),
)

# Audio transcription
response = await app.ai(
    text("Transcribe this recording:"),
    audio_from_file("./meeting.wav"),
)

When you pass positional arguments to app.ai(), the SDK auto-detects the content type.

# These are equivalent:
await app.ai("Describe this:", "https://example.com/photo.jpg")
await app.ai(text("Describe this:"), image_from_url("https://example.com/photo.jpg"))

Media generation uses a pluggable provider system. All three SDKs support image, audio, and video generation. Python additionally supports music generation and transcription. See the Media Generation page for full details.

Quick Examples

# Image generation — OpenRouter (Gemini)
result = await app.ai_generate_image(
    "A watercolor painting of a mountain lake",
    model="openrouter/google/gemini-3.1-flash-image-preview",
)
result.images[0].save("lake.webp")

# Video generation — async polling
result = await app.ai_generate_video(
    "Camera slowly pans across a mountain landscape",
    model="openrouter/kling-video/v2.0/master",
    duration=10,
)
if result.videos:
    result.videos[0].save("mountain_video.mp4")

# Audio generation — SSE streaming
result = await app.ai_generate_audio(
    "Welcome to AgentField.",
    model="openrouter/openai/tts-1",
    voice="alloy",
)
result.audio.save("welcome.wav")

# Music generation — Google Lyria 3 Pro
result = await app.ai_generate_music(
    "Calm ambient background music",
    model="openrouter/google/lyria-3-pro",
    duration=30,
)
result.audio.save("bgm.wav")

All media generation methods return a MultimodalResponse.

Save All Content

result = await app.ai_generate_image("A sunset over the ocean", num_images=3)

# Save everything to a directory
saved = result.save_all("./output", prefix="sunset")
# Returns: {"image_0": "./output/sunset_image_0.webp", ...}

ImageOutput Methods

AudioOutput Methods

Go provides functional options for attaching images to requests.

// From URL
resp, _ := client.Complete(ctx, "What breed is this dog?",
    ai.WithImageURL("https://example.com/dog.jpg"),
)

// From local file (auto base64 + MIME detection)
resp, _ := client.Complete(ctx, "Analyze this architecture diagram.",
    ai.WithImageFile("/path/to/diagram.webp"),
)

// From raw bytes
resp, _ := client.Complete(ctx, "Describe this image.",
    ai.WithImageBytes(pngData, "image/png"),
)

All three SDKs use a pluggable provider system for media generation. A MediaRouter dispatches to the right provider based on model prefix.

See the Media Generation page for provider configuration, custom providers, and the MediaRouter API.