Видео по референсам: консистентные персонажи и локации в нейросети

Расшифровка английского закадрового текста.

0:04Text alone can't hold a face. You can describe Amelie in perfect detail and two generations will still give you two different women. References are what turn a description into a specific person, a specific car, a specific photograph. In this lecture, you'll learn the four ways to drive a video model with images and how Screenweaver's two pipelines use them.

0:30Four modes, from loosest to tightest. Text to video. Words only, maximum freedom, minimum consistency. Image to video. One start image and the model animates it. Reference to video. Several images that define who and what appears while the model stages the shot itself. And first and last frame. You give the start and the end and the model finds the way between them. The more images you give, the more you control and the less the model invents.

1:02What should you give as references? For each character in the shot, their production sheet. For the location, one clean image of the set. For key props, a sheet of the object alone. And the art direction image for the overall look. Keep it to what the shot needs. Every extra reference competes for the model's attention.

1:23Quality matters more than quantity. A reference should be sharp, evenly lit and show the subject clearly with nothing that competes. No busy background, no second person, no text. A blurry or cluttered reference gives you a blurry, cluttered character.

1:41Then bind each reference to its job in the prompt. Our real prompt starts with a line of asset bindings. Thomas's car is image one. The photograph is image two. Then the text says, the vehicle interior and ambient lighting of image one and image two governing the prop's physical structure and the print on it. Name it and give it a job.

2:05In Screenweaver, the first pipeline is called reference to video. It takes the action of each shot. A prompt director writes one brief for the whole scene and a scene video node generates one clip. The characters, places and objects you mention in your shot actions are resolved into their production sheets and bound to the prompt automatically. Note what it doesn't use, your storyboard panels. The model stages the scene from your words and your references.

2:35The second pipeline is storyboard panels to video. It assembles your storyboard panels into a single grid image, up to 14 panels and sends it to the scene video node together with your art direction text and image and the grid tells the model the order and the composition of the shots. Never the look. The look comes from the art direction and the sheets.

3:00So which one should you use? Reference to video when you want the model to stage freely from your actions. Storyboard panels to video when your framing has to be respected shot by shot. Here's the photo generated both ways. On the left, reference to video. On the right, storyboard panels to video. Same story, same actors. Look at how the framing follows the previous on the right.

3:29In practice, I often start with reference to video to see how the model stages the scene on its own. If the staging works, I keep it. If the framing drifts away from what I designed, I switch to storyboard panels to video and the grid pulls the camera back to my previous.

3:45First and last frames are your tool for continuity between clips. Take the last frame of one clip and use it as the first frame of the next. The second clip starts exactly where the first one ended. Same pose, same light, same position. It's the cleanest way to continue an action across a cut or to extend a moment that needs more time than one generation allows.

4:09Start frames also let you control exactly where an action begins. Give the model a first frame where Thomas is already turned toward Emily. Then it won't spend two seconds of your clip turning.

4:21Watch out for four traps. A reference that contains a set, like our photograph shot on the back seat, which moved Emily into the back. Two references that contradict each other, like a sheet with a beige coat and a mood board image with a red one. Too many references for one shot and photorealistic faces on models that filter them. When a reference -based shot fails, check the references before you touch the prompt.

4:50Your exercise. For one scene, list the references each shot needs and write the bindings. Which image governs what? Then generate the scene with both pipelines and compare. In the next lecture, we'll deal with the shots that still go wrong and how to fix them without starting over.