Media generation

• Unified generation API — ai_generate_image(), ai_generate_video(), ai_generate_audio(), ai_generate_music() with consistent signatures across all three SDKs.
• Pluggable providers — FalProvider (fal.ai), LiteLLMProvider (DALL-E, OpenAI TTS), OpenRouterProvider (Gemini, Kling, GPT Audio, Lyria). Register custom providers.
• MediaRouter — Automatic provider dispatch based on model prefix. "openrouter/..." routes to OpenRouter, "fal-ai/..." routes to fal.ai.
• Rich output types — MultimodalResponse / MediaResponse with .images, .audio, .videos, .files, all with .save() and .get_bytes() methods.
• VideoOutput — Dedicated video type with duration, resolution, aspect_ratio, has_audio, and cost_usd metadata.
• Music generation — Generate instrumental tracks with Google Lyria 3 Pro (Python).
• Model override per call — Use different models for different media types without reconfiguring.

The MediaRouter automatically selects the right provider based on the model name prefix. This means you never manually pick a provider — just pass the full model name.

# Automatic routing based on model prefix — no manual provider selection
await app.ai_generate_video(
    "A cat playing with yarn",
    model="openrouter/kling-video/v2.0/master",    # → OpenRouterProvider
)
await app.ai_generate_video(
    "A cat playing with yarn",
    model="fal-ai/minimax-video/image-to-video",   # → FalProvider
)
await app.ai_generate_image(
    "A sunset over mountains",
    model="dall-e-3",                                # → LiteLLMProvider (default)
)

Routing rules

• Longest prefix wins — "openrouter/google/" is preferred over "openrouter/" if both are registered.
• Capability check — The router verifies the matched provider supports the requested modality (image, audio, video, music).
• No match = error — A ValueError (Python) / MediaProviderError (TypeScript) / error (Go) is raised if no provider matches.
• Prefix stripped — The matched prefix is removed before passing the model name to the provider, so providers see clean model names.

AgentField uses a pluggable provider system for media generation. Each provider wraps an external API and returns standardized output types.

Built-in Providers

OpenRouter audio routing — /audio/speech vs chat-completions

OpenRouter exposes two audio APIs. The SDK chooses automatically per model so you never have to:

• POST /audio/speech (OpenAI-compatible TTS) — used for models whose output_modalities is ["speech"]. Covers every dedicated TTS model on OpenRouter, e.g. hexgrad/kokoro-82m, unrealspeech/*, etc.
• POST /chat/completions with modalities=["text","audio"] SSE streaming — used for chat-audio models whose output_modalities contains "audio", i.e. the openai/gpt-audio* family.

The SDK fetches /api/v1/models/{id}/endpoints once per model and caches the result. When the upstream returns raw PCM, the SDK wraps it in a WAV header on the client so format="wav" always returns something playable.

Configuration

from agentfield import Agent, AIConfig

app = Agent(
    node_id="media-agent",
    ai_config=AIConfig(
        fal_api_key="...",           # or FAL_KEY env var
        openrouter_api_key="...",    # or OPENROUTER_API_KEY env var
    ),
)

# OpenRouter image generation (Gemini)
image = await app.ai_generate_image(
    prompt="A sunset over mountains",
    model="openrouter/google/gemini-3.1-flash-image-preview",
    image_config={"aspect_ratio": "16:9"},
)

# Fal.ai image generation (Flux)
image = await app.ai_generate_image(
    prompt="A sunset over mountains",
    model="fal-ai/flux/dev",
    size="landscape_16_9",
)

# DALL-E image generation (default provider)
image = await app.ai_generate_image(
    prompt="A sunset over mountains",
    model="dall-e-3",
    size="1792x1024",
    quality="hd",
)

Custom Providers

from agentfield.media_providers import MediaProvider, register_provider

class MyProvider(MediaProvider):
    @property
    def name(self) -> str:
        return "my-provider"

    @property
    def supported_modalities(self):
        return ["image"]

    async def generate_image(self, prompt, model=None, **kwargs):
        result = await my_api.generate(prompt, **kwargs)
        return MultimodalResponse(
            text=prompt,
            images=[ImageOutput(url=result["url"])],
        )

    async def generate_audio(self, text, **kwargs):
        raise NotImplementedError("Audio not supported")

register_provider("my-provider", MyProvider)

All generation methods return typed output objects with convenience methods.

ImageOutput

AudioOutput

VideoOutput

Video generation returns dedicated VideoOutput objects with rich metadata — not generic file outputs.

Transcription (Python)

Transcription returns a MultimodalResponse — use .text for the transcribed text and .raw_response for provider-specific details like segments.

Image-to-Video Pipeline

Generate an image, then animate it into a video.

@app.reasoner()
async def image_to_video(prompt: str) -> dict:
    # Generate a hero image
    image = await app.ai_generate_image(
        prompt=f"Cinematic still frame: {prompt}",
        model="openrouter/google/gemini-3.1-flash-image-preview",
    )

    # Animate the image into a video
    video = await app.ai_generate_video(
        prompt=f"Smooth camera pan across: {prompt}",
        model="openrouter/kling-video/v2.0/master",
        image_url=image.images[0].url,
        duration=10,
    )

    return {
        "image": image.images[0].url,
        "video": video.videos[0].url if video.videos else None,
    }

Multi-Format Content Generation

Generate a complete content package — blog post, hero image, audio narration — from a single topic.

@app.reasoner()
async def generate_blog_assets(topic: str) -> dict:
    # Generate the blog post text
    post = await app.ai(
        system="Write an engaging technical blog post.",
        user=topic,
    )

    # Generate hero image via OpenRouter
    image = await app.ai_generate_image(
        prompt=f"Blog hero image for: {topic}. Modern, clean, technical.",
        model="openrouter/google/gemini-3.1-flash-image-preview",
    )

    # Generate audio version for podcast feed
    audio = await app.ai_generate_audio(
        text=post,
        model="openrouter/openai/tts-1",
        voice="nova",
    )

    # Generate background music for the audio version
    music = await app.ai_generate_music(
        prompt=f"Calm background music for a tech podcast about {topic}",
        duration=60,
    )

    return {
        "post": post,
        "image_url": image.images[0].url if image.images else None,
        "audio_url": audio.audio.url if audio.audio else None,
    }

Transcription Pipeline (Python)

Process uploaded audio files and extract structured data.

@app.reasoner()
async def process_meeting(audio_path: str) -> dict:
    transcript = await app.ai_transcribe_audio(
        audio_url=audio_path,
        model="fal-ai/whisper",
        language="en",
    )

    from pydantic import BaseModel

    class MeetingNotes(BaseModel):
        summary: str
        action_items: list[str]
        decisions: list[str]

    notes = await app.ai(
        system="Extract structured meeting notes from this transcript.",
        user=transcript.text,
        schema=MeetingNotes,
    )

    return notes.model_dump()

Generation Methods

Provider Management

Video generation uses an async submit-poll-download lifecycle. You submit a job, poll for completion, and the SDK downloads the result automatically.

How it works

1. Submit — POST to /api/v1/videos with prompt, model, and parameters.
2. Poll — GET /api/v1/videos/{jobId} every poll_interval seconds until status is "complete".
3. Download — Fetch the video from the returned URL and encode as base64.

Parameters

Image-to-video — first / last frame guidance

Pass the start frame, end frame, or both to frame_images. The SDK serialises items as-is; URLs can be https://… or data:image/jpeg;base64,….

video = await app.ai_generate_video(
    prompt="A red panda climbs from the base of a bamboo grove to the top.",
    model="openrouter/google/veo-3.1-lite",
    duration=4,
    frame_images=[
        {"type": "image_url",
         "image_url": {"url": first_frame_data_url},
         "frame_type": "first_frame"},
        {"type": "image_url",
         "image_url": {"url": last_frame_data_url},
         "frame_type": "last_frame"},
    ],
    extra={"seed": 42},  # any model-specific passthrough
)

Example models

The OpenRouter provider routes by querying each model's output_modalities metadata, so any OpenRouter video model works without an SDK change — the list below is just a sample of what people actually use.

Generate instrumental music tracks using Google Lyria 3 Pro via OpenRouter.

# Simple music generation
music = await app.ai_generate_music(
    prompt="Upbeat electronic track for a product demo",
    model="openrouter/google/lyria-3-pro",
    duration=30,
)
music.audio.save("demo_music.wav")

# Access raw audio bytes
audio_bytes = music.audio.get_bytes()
print(f"Generated {len(audio_bytes)} bytes of audio")

Parameters