How AI Video Generation Works

The short answer

AI video generators like Gemini Omni are latent diffusion models. They are trained to add noise to videos, then learn to reverse that process generating video by progressively removing noise, guided by your text prompt. The result is new video that never existed, generated frame-by-frame with physics-plausible motion.

Step 1: Latent space compression

Video is expensive to process directly at the pixel level a single 4K frame contains ~8.3 million pixels, and a 10-second video at 24fps is 240 frames. That's 2 billion pixel values to process per generation.

To make this tractable, video diffusion models use a Variational Autoencoder (VAE) to compress frames into a lower-dimensional "latent space." A 4K frame that takes megabytes of raw pixel data is compressed to a latent representation hundreds of times smaller, while preserving the essential structure (shapes, colors, textures).

The model does all its generation work in this compressed latent space, then uses the VAE decoder to reconstruct high-resolution frames at the end.

Step 2: Controlled noise removal

During training, the model sees millions of videos with noise added at different levels. It learns to predict and remove that noise. At generation time, you start with pure random noise in latent space and the model runs through 20–50 "denoising steps," removing noise iteratively until a coherent video emerges.

Your text prompt guides each denoising step via a technique called Classifier-Free Guidance (CFG). The model simultaneously predicts a noise-free version conditioned on your prompt and an unconditioned version, then moves in the direction that increases agreement with your prompt. Higher CFG scale = stricter adherence to the prompt.

Step 3: Temporal consistency

The hardest problem in AI video is making objects stay stable across frames not flickering, morphing, or disappearing. This is called temporal consistency.

Modern video diffusion models address this with 3D attention: instead of only attending to spatial relationships within a single frame, the model's attention mechanism spans across the time dimension too. Every frame can "see" every other frame when computing what it should look like.

This is why Gemini Omni clips look smooth and physically plausible the temporal attention layers are what prevent the frame-to-frame incoherence of early AI video systems.

Character consistency a different challenge

Standard text-to-video generates a new random character every time, even from identical prompts. Character consistency requires reference image conditioning: the model encodes a reference photo and uses it to anchor the character's identity features (face, proportions, style) across all subsequent generations.

In Gemini Omni's character model, the reference image is encoded into the same embedding space as the text prompt, and both guide the denoising process together. The result is the same character placed in any scene you describe.

Lip sync phoneme mapping

Lip-sync works by extracting phonemes (individual sound units) from the audio track, mapping each phoneme to a viseme (the mouth shape for that sound), and then generating frames where the character's mouth matches the audio at each moment in time.

Modern AI lip-sync also generates secondary motion head nods, blinks, micro-expressions because isolated mouth movement without any other animation looks unnatural. The temporal consistency mechanism ensures these secondary motions flow smoothly across the clip.