Models registry — every model Ralphy actually calls

How to use this file

1. Before any model call, open the matching section. The top pick has the reason it's the top pick.
2. For video, also run ralphy models show <id>. It returns the live supported_durations, supported_resolutions, supported_aspect_ratios, supported_frame_images from OR. Don't hand-pick parameters that aren't in those arrays; the submit will fail validation (ralphy generate video runs the check pre-flight; bypass with --no-validate).
3. For image, the --size flag is a prompt-level hint, not an enforced constraint. Gemini and gpt image models round to their internal natural sizes (1024², 768×1376, …). If you need an exact dimension, post-process with ralphy video extract-segment or ffmpeg after generation.
4. For cost preview, every video gen accepts --dry-run. It prints the resolved request body and cost estimate without spending credits.
5. If the task is new (not in this file) — DO NOT invent a provider. Tell the user the task is out of scope or needs a model-list extension.

---

Image generation

Endpoint: POST /api/v1/chat/completions with modalities: ["image","text"]. Output bytes arrive on choices[0].message.images[0].image_url.url as a data: URI or http URL. cli/lib/providers/media.ts → generateImage() decodes both. Don't make direct fetches to fal.ai or openai.com.

Reference images: --ref accepts URL, local path, or data: URI. Local paths are auto-converted to data: URI in-process; there's no upload step. Both gpt-5.4-image-2 and gemini-3-pro-image-preview accept image inputs; gemini is much better at multi-ref consistency (2+ refs).

Size / aspect ratio: ralphy generate image maps --size WxH to OpenRouter's structured image_config.aspect_ratio (nearest of 1:1 2:3 3:2 3:4 4:3 4:5 5:4 9:16 16:9 21:9) and forwards it alongside the in-prompt hint (openrouter.ts → sizeToAspectRatio). This is the only reliable lever for non-square output: openai/gpt-5.4-image-2 ignores the in-prompt size hint and defaults to 1024² unless image_config is sent — with it, --size 1080x1920 returns a real 9:16 (720×1280 native bucket), validated on loud-kids-poster-001 (2026-05-27). The result still lands on the model's nearest native bucket (1024² for 1:1, ~768×1376 / 720×1280 for 9:16, ~1280×720 for 16:9), not pixel-exact WxH. Downstream HyperFrames / ffmpeg compositions handle the scale-to-cover. image_size (1K/2K/4K) is available in the OR API but not yet wired to a flag.

Prompt cookbook: mode-by-mode masters in docs/prompts/image/ (product-shot, lifestyle-scene, closeup-with-person, macro-detail, flat-lay, virtual-model-tryout, hero-banner, conceptual-product, iteration-edit). The agent fills slots from a user request, then calls ralphy generate image --prompt "<filled>". There's no new CLI flag for this; it's a curated library, not a feature.

Avoid:

• Any model more than a year old (stable-diffusion-xl, flux/schnell, dall-e-3). Quality is below the current top picks at the same price.
• gpt-image-1: legacy line. gpt-5.4-image-2 is the current stable OpenAI image model (not to be confused with the "gpt-image-2" naming some external docs use).
• Hard-coded fal.ai endpoints. We left fal.ai behind in Sprint 2.

Failure modes (per-model quick reference)

Append-only. Add a row when a new quirk costs >5 min of debugging or a re-roll.

---

Video generation (text-to-video + image-to-video)

Endpoint: async-job pattern at POST /api/v1/videos. Submit returns { id, status, polling_url }; poll until completed; download via GET /api/v1/videos/{id}/content?index=0 (auth required). The legacy /api/v1/videos/generations returns 404. cli/lib/providers/media.ts → generateVideo() handles the full job lifecycle (15s × 80-poll = 20 min budget; tunable).

Per-model matrix (live from /api/v1/videos/models, snapshot 2026-05-08)

Always recheck via ralphy models list. These arrays change.

Pricing reality check (2026-05-11): OpenRouter bills video generation per-clip flat. The duration parameter sets the clip length and the billed cost ≈ rate × duration. A ✓ in the rate column means we verified the rate against actual OR billing on 2026-05-11 (see docs/render-test-2026-05-11.md §1.1). Earlier docs claimed half-price std and per-second steps that didn't match observation; those have been corrected here and in cli/lib/providers/media.ts:VIDEO_PRICE_PER_SEC. Models without ✓ are ballparks from the OR catalog. Verify on first use and add ✓ once confirmed.

When to pick which

Lessons from this session (2026-05-08)

1. kwaivgi/kling-v3.0-pro rotates "wide" prompts inside the 9:16 container. Phrases like "wide overhead cityscape", "massive crowd in town square", "dancers under starlit sky" bias the model toward landscape composition; OR returns a 1080×1920 file but the content is laid out for 16:9. Fix: anchor with --first-frame <portrait-image> and rewrite the prompt with explicit vertical wording ("tall vertical portrait shot, low camera angle looking up, narrow alley framing, subjects centered vertically, half-timbered houses tower vertically on both sides"). The first-frame image overrides the model's compositional bias.

2. --resolution 720p is silently upgraded to 1080p by kwaivgi/kling-v3.0-pro even though the catalog only lists 720p. The output dimension is whatever the model decides; treat resolution as a soft hint and let downstream HyperFrames / ffmpeg crop+scale to the composition's exact frame.

3. OR's per-clip billing is fixed (e.g., a 5s kling-pro clip is ~$0.70 regardless of "duration in body" precision). The per-second figures above are therefore ballparks; pre-flight --dry-run to see the estimate before submitting.

4. generate_audio: true is unsafe outside English. Confirmed for kwaivgi/kling-v3.0-pro, bytedance/seedance-2.0, and google/veo-3.1 on Russian — accent slips, voice age drifts, text gets cut. Default is false; only enable for EN with --audio.

5. kwaivgi/kling-v3.0-pro --last-frame historically returned 400 "File is not in a valid base64 format" even with a clean PNG anchor. Root cause was C2PA / EXIF metadata that the provider parsed too eagerly; the 2026-05-19 resolveImageRef() strip helped on single-frame but the multi-frame (first+last together) path kept failing across flipper / glitter-cream / playdate / venom regardless of source. As of 2026-05-30 (#008), generateVideo() preflight-rejects the kling-v3.0-pro multi-frame combination as TerminalProviderError with a pointer to bytedance/seedance-2.0 — which honors --last-frame natively across all aspects. No more wasted round-trips. Postmortems flagging this: playdate / flipper / venom / glitter-cream.

6. bytedance/seedance-2.0 privacy filter rejects photoreal-human anchors with InputImageSensitiveContentDetected.PrivacyInformation, even when the human was itself AI-generated. Reserve seedance for cartoons / non-human anchors / landscapes / hands / abstract motion. For photoreal humans, default to kwaivgi/kling-v3.0-pro (single-frame or multi-frame after #5 lands). Postmortems: tokyo / noski / venom.

7. kwaivgi/kling-v3.0-pro 2500-char prompt cap. OR returns 400 after a round-trip if you exceed it. As of 2026-05-30 (#008), generateVideo() preflight-rejects prompts >2500 chars as TerminalProviderError before any submit — no more 4× wasted round-trips per session. The cap lives in a per-model MAX_PROMPT_CHARS map in cli/lib/providers/openrouter.ts; add a row when another model documents a hard cap. Trim atmosphere/setting paragraphs first — voice-tag, no-music, on-camera-EN clauses are load-bearing and should never be cut. Postmortem: glitter-cream.

8. ElevenLabs Music 2-concurrent cap per subscription. Three+ parallel calls return 429 concurrent_limit_exceeded and pollute generations.jsonl with error rows. Serialize music gen or stay ≤2 in-flight. Postmortem: tokyo.

9. Per-endpoint concurrent-call caps are now self-throttled in-process (cli/lib/providers/concurrency.ts, #007). The semaphore wraps every network round-trip so the CLI never round-trips an over-cap call; the retry helper sits OUTSIDE the semaphore so backoff sleeps don't pin a slot. Current caps:

   • elevenlabs/tts — 3 concurrent (choose-your-guide-001: 9 parallel → 6 hard-failed 429).
   • elevenlabs/music_v1 — 2 concurrent (tokyo-y2k-001: 3 parallel → 1 hard-failed 429).
   • openrouter:openai/gpt-5.4-image-2 — 2 concurrent (validated 2026-05-29; appstore-takeaminute-001 hit 73/73 403 on uncapped fan-out).
   • openrouter:google/gemini-3-pro-image-preview — 2 concurrent (catalog tolerates ≥4, kept at 2 for shared-key safety).
   • openrouter:bytedance/seedance-2.0 — 1 concurrent (queue depth + multi-block extend is inherently sequential).
   • openrouter:kwaivgi/kling-v3.0-pro — 2 concurrent.
   • LLM chat-completions default — 4 concurrent.
   • Default fallback for unknown endpoints — 2 concurrent. When the per-key OR cap still trips despite the semaphore (e.g. two ralphy processes sharing one key), shared.ts → rewriteUpstreamError() now surfaces the 403 as "concurrent-call limit on <model>; try --concurrency 1 or switch model. (This is NOT a $ balance issue — check 'ralphy doctor' for credits.)" — no more credits-misread. Follow-up: the in-process semaphore does NOT span processes. The queue daemon's own worker count gates cross-process for now; a file-locked queue is a future enhancement. Postmortems: appstore / analog-horror / tokyo / choose-your-guide.

10. google/gemini-3-pro-image-preview image-safety filter is materially stricter than openai/gpt-5.4-image-2 on body-horror / cryptid / skinwalker / werewolf register. Gemini returns native_finish_reason: IMAGE_SAFETY with an empty content body even when the prompt uses softened surreal-anatomy language (concentric maws lined with tongue-like protrusions, biological apertures with internal teeth on a recognizable creature in a real-world setting). The reasoning trace literally describes the requested transformation before the filter refuses. openai/gpt-5.4-image-2 accepts the same prompts and delivers — validated on voidstomper-test-001's skinwalker BBQ frame after three gemini refusals. Route rule: for voidstomper-adjacent / Cronenberg / The Thing / mid-warp body-horror anchor images, start at --model openai/gpt-5.4-image-2. Accept the 1024×1024 default (gpt-image ignores arbitrary --size); use --ref <gemini-9:16-scene> to carry scene + character identity, then post-process to 9:16 with ffmpeg pad if your downstream i2v requires matching dimensions. Postmortem: voidstomper-test-001 (2026-05-25).

11. google/veo-3.1 body-horror filter rejects both the prompt AND the input frame independently. Sanitizing the prompt (stripping skinwalker / werewolf / vertebrae / Cronenberg words) does NOT unblock the path — Google's filter ALSO scans the first/last-frame anchor and refuses when the anchor itself is clearly body-horror. Combined with seedance's photoreal-human rejection (cap #6), this leaves kwaivgi/kling-v3.0-pro as the only viable i2v provider for the photoreal-human + body-horror combination that voidstomper-style content requires. Skip seedance and veo round-trips on those jobs; go straight to Kling. Postmortem: voidstomper-test-001 (2026-05-25).

12. kwaivgi/kling-v3.0-pro and bytedance/seedance-2.0 overshoot --duration by ~1s. Both models return clips ~1 second longer than the requested --duration — silent, billed against the requested duration, not the delivered one. tokyo-y2k-001 measured 5s/4s/9s storyboard → 6.04/5.04/10.04 actual; planned 75s of clips landed as 90.7s of raw mp4. Editor recipe (pre-shorten at art-director stage, or budget a vision-trim pass) lives in docs/playbooks/editor/render-pipeline.md → Source-clip duration overshoot. Filed long-form: notes/issues/done/042-clip-duration-overshoot-undocumented.md. Postmortem: tokyo-y2k-001 (workflow-fixes #3).

Failure modes (per-model quick reference)

Append-only. Each row is a quirk that has already cost ≥1 re-roll across sessions.

Routing rules (which video model for what)

Apply these before drafting a ralphy generate video prompt. They short-circuit re-rolls.

• Hyper-motion (explosions, runway-sprint, coin-arcs, particle bursts, sports collisions, parkour, falling) → bytedance/seedance-2.0. Kling reads these as narrative beats and softens the physics. Caveat: seedance's privacy filter rejects photoreal-human i2v anchors — only use seedance hyper-motion on cartoon / non-human / landscape / hands / abstract subjects.
• Talking-head, photoreal humans, slow narrative, lip-sync → kwaivgi/kling-v3.0-pro. Multi-frame works post-2026-05-19. --audio for EN VO only.
• 4K hero piece → google/veo-3.1-fast (the only catalog entry with 4K).
• 3:4 / 4:3 portrait magazine aspect → alibaba/wan-2.7 (only model with these in stock).
• Cinema 21:9 → bytedance/seedance-2.0 (only model with 21:9 in stock).
• Photoreal humans + body-horror anchor → kwaivgi/kling-v3.0-pro only. Seedance rejects the anchor, veo rejects both prompt and anchor (caps #6 + #11).
• Cheapest acceptable batch → google/veo-3.1-lite (~$0.09/s) for non-photoreal-human work; otherwise kwaivgi/kling-v3.0-std (same rate as pro but worse keyframe holds).

--audio policy: SPEECH vs AMBIENT/DIEGETIC

Two distinct audio jobs hide behind the single --audio flag — different rules apply.

• SPEECH (on-camera VO / dialogue / lip-sync). Stakes are high — voice age, accent, and word-cut are visible.
  • Default: google/veo-3.1 for English on-camera dialogue.
  • kwaivgi/kling-v3.0-pro --audio works for EN only. Slips on RU / UA / non-English (memory feedback_kling_no_ru_audio). For RU, render silent + post-mix ElevenLabs eleven_multilingual_v2 in the editor stage.
  • bytedance/seedance-2.0 --audio is not validated for SPEECH — treat as off.
• AMBIENT / DIEGETIC (background noise, footsteps, crowd murmur, wind, traffic, room tone, prop SFX). Stakes are low — language doesn't apply.
  • Any of kling / seedance / veo --audio: true are fine. Generate alongside the visuals; cheaper than a separate SFX pass.
  • If the brief explicitly bans music (most UGC beds do), keep the no music — only diegetic ambient + sparse SFX clause in the prompt regardless of which model.

When in doubt: SPEECH = veo (EN) / kling-EN-only / silent+ElevenLabs (everything else). AMBIENT = whichever video model is already in the shot list.

Tried-and-dropped (postmortem cross-reference)

Avoid:

• kling-video/v1.6 or v2.x: outdated.
• luma-dream-machine: worse than kling at the same price; out of OR catalog now.
• fal-ai/* endpoints: the stack moved to OpenRouter in Sprint 2.

---

Voiceover (TTS)

Voice settings (deadpan young Russian):

{ "model_id": "eleven_multilingual_v2", "voice_settings": { "stability": 0.55, "similarity_boost": 0.8, "style": 0.25, "use_speaker_boost": true }, "output_format": "mp3_44100_128" }

Voice settings (English brainrot dramatic narrator):

{ "model_id": "eleven_v3", "voice_settings": { "stability": 0.30, "similarity_boost": 0.75, "style": 0.40, "use_speaker_boost": true } }

Voice picks: Adam (pNInz6obpgDQGcFmaJgB) for dramatic narrator, Brian (nPczCjzI2devNBz1zQrb) for Reddit-monotone, Daniel (onwK4e9ZLuTAKqWW03F9) for British-sarcastic.

Failure modes:

• Default UA on Node 20+ → Cloudflare 403. Send User-Agent: Mozilla/5.0 (...).
• Free/starter cap is 3 concurrent → 429. Run sequentially, not in parallel.
• Default library voices (clyde-warvet, daniel-deep, etc.): too theatrical for RU. A custom clone is mandatory for RU production.
• VO drift: dramatic narration on eleven_v3 consistently runs ~15-25% longer than scripted word-count would suggest (Strasbourg 45s scenario rendered at 54.6s). Time-budget compositions to actual VO duration via ralphy project transcribe, not scenario text length.

Avoid:

• OpenAI tts-1-hd on Russian — flat American accent.
• ElevenLabs eleven_v3 on Russian in production — unstable.

---

Music generation

Endpoint contract (for cli/lib/providers/media.ts → generateMusic()):

POST https://api.elevenlabs.io/v1/music
Headers: xi-api-key: $ELEVENLABS_API_KEY, Content-Type: application/json
Body: { "prompt": "...", "music_length_ms": 30000, "force_instrumental": true, "output_format": "mp3_44100_128", "model_id": "music_v1" }
Response: 200 → binary mp3 (application/octet-stream)
         400 → JSON ToS rejection (`bad_prompt`) — see "Prompt content policy"
         422 → JSON validation error

Prompt content policy (#006). ElevenLabs Music ToS rejects prompts that name specific artists / producers / copyrighted tracks (rappers, named producers, song titles, scored themes) with HTTP 400 bad_prompt. The 400 envelope carries a detail.data.prompt_suggestion field containing a provider-sanitized rewrite the CLI can resubmit verbatim. The connector surfaces this on TerminalProviderError.promptSuggestion; ralphy generate music --auto-retry-on-tos-rejection will log the original failure and resubmit once using the rewrite. Use genre + tempo + instrumentation + mood framing instead of named entities (e.g. "trap beat, 140 BPM, 808 sub-bass, dark minor-key piano stab"). The CLI runs a soft pre-submit linter (cli/lib/music-prompt-lint.ts) against a known artist/track regex set and warns to stderr before submit — non-blocking, false positives are cheaper than missed catches.

Trend-music rule: if a template references a specific trend track (assets/trend-*.mp3), copy the file from the ralphy-assets companion repo. Don't generate a substitute. Track recognition is half of what makes a trend video a trend.

Fallback (if ElevenLabs Music quality regresses): temporarily route to fal-ai/lyria2 via FAL_KEY as a documented exception. As of 2026-05-08 the fallback is not active; ElevenLabs Music is the only path.

Avoid:

• Suno (not on OpenRouter).
• lyria2 directly via FAL_KEY while ElevenLabs Music works: don't multiply optional keys.

---

Audio transcription / Captions

CLI: ralphy generate captions --project <id> --audio <vo.mp3> --language en (writes captions.json next to the project).

Avoid:

• Local whisper.cpp: large binary, no real benefit over the cloud at our volumes.
• Direct OpenAI API: we route through OpenRouter so one key covers vision + scoring + transcription.

---

LLM (for skills and analytics)

Provider routing. All LLM / vision calls go through cli/lib/providers/llm.ts → callLLM(). The only provider is OpenRouter.

Avoid:

• Direct fetch("https://openrouter.ai/...") calls in new scripts. Go through callLLM() so users can switch providers via ralphy setup.
• Hard-coded anthropic.com or openai.com URLs; everything goes through OpenRouter.

---

Out-of-scope / dropped

These models / families were removed during OpenRouter consolidation (Sprint 1.3 / 2). Don't bring them back without an explicit plan upgrade:

• fal-ai/nano-banana-pro/edit — replaced with google/gemini-3-pro-image-preview via OR.
• fal-ai/flux-pro/v1.1-ultra, fal-ai/flux/dev/inpainting, fal-ai/flux-pro/v1.1-ultra/redux — out.
• fal-ai/luma-dream-machine/image-to-video — out (worse than kling).
• fal-ai/wan-25 — lipsync stage dropped entirely in v2 (no FAL_KEY pipeline).
• fal-ai/sync-lipsync, fal-ai/veed/lipsync — out.
• fal-ai/lyria2 — out (fallback reserved in the Music section).
• fal-ai/seedream — out.
• Replicate wav2lip — out (no token, no stage).
• openai/gpt-image-1, dall-e-3, stable-diffusion-xl, flux/schnell — outdated.
• Apify — replaced with a Playwright scraper in /researcher (deferred to v2).
• Higgsfield Soul, Fireworks Whisper — require separate keys, not in the stack.
• Vercel AI Gateway, direct OpenAI API — single-provider OpenRouter in v2.

---

When to update this file

• On the first session in a new chat: check Last reviewed, refresh if stale.
• After every failure mode on a new model: add it to "Avoid" / "Lessons" with the reason.
• When you change a default in a skill or script: sync it here.
• When you add a verb to ralphy generate or a flag: sync the price / param notes here.
• At least once a month, even if nothing broke.