Most of the AI video models that made headlines this year — Veo, Sora, Kling — are closed. You send a prompt to an API, you get back a clip, and you have no idea what's happening in between beyond what the provider's blog post tells you. MiniMax H3 breaks that pattern in a specific way: it's a 33-billion-parameter omni-modal video-and-audio generation model released with open weights, a published architecture, and inference recipes for SGLang, vLLM, diffusers, and ComfyUI. You can read exactly how it works, and if you have the GPUs, you can run it yourself.
That distinction — open weights on a model in the same capability class as the frontier closed labs — is the actual news here, more than any single benchmark number. This piece explains what H3 does, how its three-stage architecture is put together, and where it fits if you're deciding between a hosted API and something you control end to end.
What MiniMax H3 Actually Does
H3 takes text, images, video clips, and audio as input and produces video with native stereo audio as output — up to 2K resolution, 4 to 15 seconds long, at 24 FPS, in a wide range of aspect ratios (21:9 down to 9:16). It supports 11 languages for dialogue generation and handles a broad set of input combinations:
| Mode | What you give it | What comes out |
|---|---|---|
| Text-to-video | A text prompt, nothing else | Generated video + audio from scratch |
| First/last-frame | Text plus one or two reference images | Video that starts and/or ends on those exact frames |
| Omni-reference | Up to 9 images, 3 video clips, and 3 audio clips, mixed | Video that draws style, subjects, or sound from all of them |
If you're new to the category, our explainers on multimodal AI and diffusion models cover the underlying ideas H3 builds on.
That last mode is the one that separates H3 from a typical text-to-video tool. Most generators take a single prompt and maybe one reference image. H3's omni-reference mode can be handed a product photo, a clip of camera motion you like, and a piece of reference audio, and asked to synthesize a new video that's consistent with all three — which is closer to how a creative brief actually gets handed to a human video editor than how most AI video tools are used today.
The Three-Module Pipeline
H3 isn't one monolithic model — it's three components that run in sequence, and understanding the split explains both its capabilities and what's not open-sourced yet.
H3-Context-IR: Turning a Messy Prompt Into Structured Instructions
Before any pixels get generated, H3-Context-IR parses the input — text, images, video, audio, in whatever combination was supplied — and works out how the pieces relate to each other and to the intended output. It handles instruction parsing, cross-modal association (which image goes with which part of the prompt), temporal understanding, and fills in underspecified details without drifting from what the user actually asked for. The output is a structured "Context Intermediate Representation" that the generation model consumes directly.
This module is not part of the open-source release — MiniMax runs it as a hosted API, because it depends on a multi-stage workflow across several internal models. What is open is the prompting guidance that documents how to replicate its behavior, so teams that want a fully self-hosted pipeline can build an equivalent preprocessing layer rather than depending on the hosted service.
H3-Base: The Model You Actually Download
This is the open-weight core: the H3-Omni-Transformer, a 33B-parameter dense transformer (roughly 13B of those parameters sit in AdaLN-related branches, which can be precomputed and cached for inference-only deployment). It's a single-stream design — the attention and feedforward layers carry no modality-specific structure at all. Text, image, and audio each get their own encoder or VAE, get packed into one unified sequence using RoPE-based positional encoding across time and space, and then flow through the same transformer blocks. Modality only re-enters through the AdaLN branches and the input/output layers.
That single-stream choice matters architecturally. A lot of multimodal systems bolt separate expert pathways onto a shared backbone per modality; H3 instead keeps the diffusion transformer itself modality-agnostic and pushes all the modality-specific work to the edges. It's a bet on generalization: the same core reasoning capacity gets applied to visual and audio generation instead of being split across specialized subnetworks.
Two separate VAEs handle compression:
- H3-VisualVAE — a temporally causal video autoencoder with 16× spatial and 4× temporal compression, 24 latent channels. Visual latents get patchified further before hitting the transformer, for an effective 32× spatial / 4× temporal downsample overall.
- H3-AudioVAE — compresses 32kHz stereo audio into latent tokens at 40Hz per channel, processing left and right independently with a shared encoder/decoder before recombining them into stereo output.
The transformer jointly predicts video and audio latents in the same forward pass, which is what produces synchronized sound rather than sound bolted on afterward — the same native-audio principle behind Veo 3-class models, applied here inside an openly published architecture.
H3-Regenerate-2K: Upscaling by Re-Generating, Not Interpolating
Rather than running a conventional super-resolution model over the 768p output, H3 feeds the low-res result back into itself, alongside the original multimodal context, and regenerates at 2K in-context. The reasoning: a traditional upscaler has to guess fine detail — small text, precise textures — from pixels alone. Feeding the original context back in lets the model recover detail it already had access to the first time, rather than hallucinating it from a blurry input. Like H3-Context-IR, this module isn't open-sourced yet; MiniMax provides an API to validate 2K output while it's still being prepared for release.
Put together, the practical picture is: the generation core is open and self-hostable, the orchestration layers around it currently are not. That's a meaningfully different position than a fully closed model, and a meaningfully different position than a fully open one — worth being precise about if you're evaluating it for a real deployment.
How to Actually Run It
H3 ships as two task-specific checkpoints rather than one model that does everything:
| Checkpoint | Tasks | Input |
|---|---|---|
| H3 Base FL2VA | Text-to-video, first/last-frame-to-video | Text, plus optional first/last frame images |
| H3 Base Ref2VA | Omni-reference generation | Text plus mixed image/video/audio references |
Both are distributed as self-contained repositories (model_index.json, processor, tokenizer, text encoder, transformer, visual VAE, audio VAE) and can be pulled with a scoped hf download, loaded directly through diffusers' ModularPipeline.from_pretrained, or served with SGLang or vLLM. MiniMax's own reference deployment for either checkpoint targets 4 GPUs with tensor/sequence parallelism (--num-gpus 4 --ulysses-degree 4) — a meaningful but not exotic hardware bar, closer to what a mid-sized team already running open-weight LLM inference would have on hand than to a hyperscaler-only requirement.
For teams that want the full 2K pipeline end to end without standing up H3-Context-IR themselves, MiniMax exposes it alongside the base model as a combined API workflow — locally hosted H3-Base for generation, hosted Context-IR and Regenerate-2K for the preprocessing and upscaling steps. That hybrid setup is arguably the most realistic way most teams will run this in the near term: self-host the expensive, capability-defining part; call an API for the two pieces that aren't open yet.
Benefits of an Open-Weight Video Model Like MiniMax H3
Native-audio video generation at usable resolution has mostly been the domain of closed frontier labs over the past year, largely because the training and serving cost is high enough that only a handful of organizations have built it. H3 doesn't change the training cost — building a model like this from scratch still isn't something most teams will do — but it changes who can run one at this capability tier. That matters for a few concrete reasons.
No Per-Call Pricing Risk
A hosted API's per-second video pricing is fine for prototyping and painful at production video volume. (The same trade-off shows up with text models; our breakdown of self-hosting LLM costs walks through how to compare the two.) Self-hosting shifts the cost structure to fixed GPU spend, which is a better fit once usage is predictable and high. It also removes exposure to a provider changing prices or rate limits after your product depends on them.
Data Doesn't Leave Your Infrastructure
Reference images, brand assets, or proprietary footage used in the omni-reference mode never have to be sent to a third-party API if the generation step runs on hardware you control. For agencies handling unreleased products, or companies with strict rules about where customer media can go, that can be the deciding factor rather than a nice extra.
You Can Fine-Tune It
MiniMax released full model weights specifically to support further development, not just inference — a closed API gives you prompt engineering as your only lever; open weights give you fine-tuning. A team with enough curated footage can adapt the model toward a house visual style, a recurring character or a specific domain instead of fighting the base model's defaults through ever-longer prompts.
It's Inspectable
When output looks wrong, you can trace it back through a documented architecture instead of guessing at a black box, which matters for any team that needs to explain or debug production behavior. Knowing that video and audio latents are predicted jointly, or that 2K detail comes from in-context regeneration, helps engineers reason about failure modes rather than retrying prompts at random.
A Stable Base You Control
Closed models can change behaviour silently when a provider ships an update. With downloaded weights, you decide when to upgrade, so a product's output stays consistent until you have tested the next version. For teams building workflows on top of generated video, that predictability is easy to underrate.
None of that makes H3 strictly better than the closed alternatives — Veo and Sora-class models still have the advantage of being fully managed, with the orchestration layer (H3's equivalent of Context-IR) baked in rather than partially external. The honest framing is that H3 opens up a third option between "call a closed API" and "train your own multimodal video model from scratch," and that option didn't really exist at this capability level a year ago.
MiniMax H3 Use Cases
The model is new, so most deployments are still experiments. These are the applications its input modes and open weights are best suited to.
High-Volume Product and Marketing Video
Retailers and marketplaces that need short clips for many products face API bills that grow with every listing. Self-hosting H3-Base turns that into fixed GPU spend, and first/last-frame mode can anchor each clip on approved product shots. The result is consistent, on-brand clips at a cost that stays predictable as the catalogue grows, with a human still reviewing output before it goes live.
Brand-Consistent Creative From Multiple References
Creative teams rarely brief with a single sentence. The omni-reference mode can combine a product photo, a clip showing the camera movement they want and a reference audio track into one generation. That is closer to how a brief reaches a human editor, and it suits agencies producing variations of a campaign that must stay visually and sonically consistent.
Multilingual Dialogue and Localised Clips
With dialogue generation in 11 languages and native stereo audio, H3 can produce short clips with synchronised speech for several markets from a shared concept. Localisation teams can test regional variants quickly. Anything customer-facing still needs native-speaker review, since generated dialogue can be fluent but subtly wrong.
Storyboarding and Pre-Visualisation
Film, advertising and game teams use generated video to explore shots before committing to production. First/last-frame generation is useful here: give the opening and closing frames of a scene and see how the model fills the motion in between. These drafts are for internal decisions rather than final output, so the 768p self-hosted tier is often enough. Directors get something closer to moving footage than a static storyboard, early enough to change course cheaply.
Domain-Specific Fine-Tuning
Organisations with a distinctive visual style, such as an animation studio or a brand with strict art direction, can fine-tune the open weights on their own footage. That is a substantial project, but it is the only route to a model that defaults to your look rather than a generic one, and it gets more valuable the more often you generate. Closed APIs offer no equivalent.
Where the Rough Edges Are
- The full 2K pipeline isn't fully self-hostable yet. H3-Context-IR and H3-Regenerate-2K remain hosted-only for now, so a genuinely offline deployment currently tops out at the 768p H3-Base output, or requires building an equivalent preprocessing system from the published prompting guidance.
- Hardware requirements are real. A 33B-parameter multimodal transformer serving both video and audio latents needs multi-GPU deployment for practical throughput — this isn't something you run on a single consumer card.
- Sparse attention isn't out yet. MiniMax notes the model was trained with native sparse-attention support for longer sequences, but the initial open-source release only ships full-attention inference; the more efficient path is coming in a future update.
- Safety filtering sits in the hosted layer. Automated content moderation on inputs and enhanced prompts is described as part of the hosted Context-IR workflow — teams building a fully self-hosted pipeline will need to think through their own moderation layer rather than inheriting one automatically.
Common Mistakes When Adopting MiniMax H3
Assuming the Whole Pipeline Is Self-Hosted
It is easy to read "open weights" and plan a fully offline 2K deployment. Today only H3-Base is open; Context-IR and Regenerate-2K are hosted. Teams that discover this late either accept 768p output, rush to build their own preprocessing, or quietly route data through hosted services they had promised stakeholders would not be used. Map which stage runs where before making commitments.
Believing Data Stays Local in a Hybrid Setup
In the hybrid pattern, generation runs on your GPUs but prompts and references may still pass through the hosted Context-IR stage. If the reason for self-hosting was keeping brand assets or client footage off third-party infrastructure, that assumption may not hold. Check exactly which inputs each hosted call receives, and whether that is acceptable for every client whose material you plan to use.
Self-Hosting Too Early
Running a 33B multimodal model on multiple GPUs carries fixed cost and operational effort. For a few clips a week, that is far more expensive than an API. Teams that self-host before they have real volume data often end up with idle hardware and an infrastructure project nobody wanted, while the API they skipped would have answered the product question in weeks.
Forgetting Moderation
Content filtering on inputs and enhanced prompts sits in the hosted layer. A fully self-hosted deployment has none by default. Shipping a user-facing generation feature without your own moderation exposes the product to misuse and brand risk that a managed API would have partly absorbed.
Treating the Initial Release as Final
The open release ships full-attention inference only, with sparse attention promised later, and the 2K module is still being prepared. Architectures that hard-code today's limits, or benchmarks that assume today's speed, will need revisiting. Building with an eye on the roadmap avoids rework when those pieces arrive.
MiniMax H3 Best Practices
- Prototype on the hosted workflow first. Use the combined API workflow to test whether H3's output suits your use case before buying or renting GPUs. Measure quality, latency and how many clips you actually generate, and keep a set of representative prompts so later comparisons are like for like.
- Decide self-hosting with real numbers. Compare your measured monthly volume against GPU cost, including idle time and engineering effort. Self-host when volume is high and predictable, or when data residency requires it. Rented cloud GPUs are a sensible middle step before committing to owned hardware.
- Map data flows stage by stage. Document which inputs go to Context-IR, which stay on your hardware and which go to Regenerate-2K, and confirm that matches your privacy and client commitments. Share that map with legal or client teams before production use, not after.
- Build or adopt a moderation layer. For any self-hosted path, add input and output filtering before exposing generation to users, rather than assuming the model is safe by default.
- Start from the reference deployment. Use MiniMax's four-GPU recipe with sequence parallelism as a baseline, then tune for your throughput needs instead of designing a serving stack from scratch.
- Pick the right checkpoint for the job. Use FL2VA for text and first/last-frame generation and Ref2VA for omni-reference work, rather than forcing one checkpoint to cover everything.
- Keep humans in the review loop. Check dialogue, on-screen text and brand details before anything ships, especially in languages your team doesn't speak, and log reviewer rejections so recurring problems can feed into prompts or fine-tuning.
- Pin versions and track releases. Lock to a specific checkpoint for production and plan an evaluation pass when sparse attention or the open 2K module lands. Rerun your saved prompt set against each new release so you can see what actually changed before switching.
Practical Takeaway
If you're building a product that needs occasional AI video generation — marketing clips, prototyping, low-volume content — a closed API is still the path of least resistance; you don't want to run multi-GPU inference for a handful of clips a week. If video generation is becoming a real part of your product's cost structure, or you need to keep reference assets off third-party infrastructure, or you want to fine-tune generation behavior for a specific brand or use case, H3 is the first model in its capability class where self-hosting that decision is actually on the table rather than theoretical.
Teams evaluating whether to self-host an open-weight generative model like H3 versus building against a hosted AI video generation API can get hands-on architecture and deployment help from Woyce Technologies.
FAQ
What is MiniMax H3?
MiniMax H3 is an open-weight, omni-modal generative AI system that produces video with synchronized stereo audio from text, image, video, and audio inputs, at resolutions up to 2K and durations up to 15 seconds. Its core generation model, H3-Base, is a 33-billion-parameter single-stream transformer released with open weights, which means teams can download it, inspect how it works and run it on their own GPUs instead of relying only on a closed video API.
Is MiniMax H3 fully open source?
The core generation model, H3-Base (the H3-Omni-Transformer plus its VAEs), is released with open weights and can be self-hosted. Two supporting modules — H3-Context-IR for input preprocessing and H3-Regenerate-2K for upscaling — are currently hosted-only via API and not yet open-sourced. So H3 is open-weight at its core rather than fully open source end to end. Teams that need a completely self-hosted pipeline have to build their own preprocessing and upscaling replacements for those two stages.
What hardware do you need to run MiniMax H3?
MiniMax's reference deployment uses 4 GPUs with sequence parallelism for either checkpoint. It's a multi-GPU workload, not something that runs practically on a single consumer graphics card. In practice, plan for data-centre class GPUs, either rented from a cloud provider or owned, plus enough storage for the checkpoints. Teams already serving open-weight LLMs on multi-GPU nodes will find the setup familiar; teams starting from zero should budget time for infrastructure work as well as the model itself.
How is MiniMax H3 different from Veo or Sora?
The underlying techniques — diffusion transformers with native joint audio-video generation — are similar in spirit. The practical difference is access: Veo and Sora are closed APIs, while H3's core model can be downloaded, inspected, fine-tuned, and run on your own infrastructure. The trade-off is convenience: closed models handle orchestration, upscaling and moderation for you, whereas a self-hosted H3 deployment currently depends on hosted modules for some of those steps or on your own replacements.
What is H3-Context-IR?
It's the preprocessing stage that interprets and structures multimodal input — text, images, video, audio — before generation, resolving how the different inputs relate to each other and to the intended output. It runs as a hosted API rather than as part of the open-source release. MiniMax has published prompting guidance describing its behaviour, so teams that need a fully self-hosted pipeline can build an equivalent preprocessing layer, though that is real engineering work rather than a configuration switch.
Can MiniMax H3 generate video from a reference image and audio clip?
Yes — its omni-reference mode (the Ref2VA checkpoint) accepts up to 9 images, 3 video clips, and 3 audio clips in combination, and generates new video that draws on all of them, which goes well beyond a typical single-image-reference workflow. A single reference image paired with one audio clip is simply the smallest version of that mode. Keep in mind that multimodal input is interpreted by the hosted H3-Context-IR stage, so a fully self-hosted setup needs its own equivalent.
Can you fine-tune MiniMax H3?
Yes. Because the full weights of the core model are released, teams can fine-tune it for a particular visual style, brand or domain, which is not possible with a closed video API where prompting is the only lever. Fine-tuning a 33B-parameter multimodal model is still a substantial project, requiring curated video and audio data, multi-GPU training capacity and careful evaluation, so it makes most sense once video generation is a core, high-volume part of your product.
Should a small team self-host MiniMax H3 or use a hosted API?
For most small teams, a hosted API is the better starting point. Self-hosting only pays off when video generation volume is high and predictable, when reference assets must stay on your own infrastructure, or when you need fine-tuning. Start by prototyping with an API, measure real usage and cost, and revisit self-hosting once the numbers justify running multi-GPU inference and maintaining your own moderation and orchestration layers.
Conclusion
Until recently, video generation with synchronized audio at usable resolution was mostly available through closed APIs, which meant per-clip pricing, assets leaving your infrastructure, and no way to inspect or adapt the model. MiniMax H3 changes that for the core generation step by releasing open weights for a 33B-parameter, single-stream transformer that jointly produces video and stereo audio.
The architecture is worth understanding before you adopt it. Context-IR structures the multimodal prompt, H3-Base generates video and audio latents together, and Regenerate-2K re-generates detail at higher resolution instead of guessing it. Only the middle piece is open today, so the realistic deployment for now is hybrid: self-host generation, call hosted services for preprocessing and 2K output, or build your own replacements.
Be clear about the costs. Multi-GPU hardware is required, sparse attention hasn't shipped in the open release, and moderation that the hosted layer provides becomes your responsibility in a fully self-hosted setup. For low-volume use, a closed API remains simpler.
If video generation is becoming part of your product and you're weighing self-hosting against an API, talk to our LLM integration team about the architecture and cost trade-offs.