Some great news to lead with: over the past few months I’ve had the pleasure of directing an AI-assisted animated short film for Amazon and BuzzFeed Studios, while beta testing Amazon’s newly announced AI creators’ platform, Project NARA.
Amazon broke the news this week: our film is going to series. I’ll be directing and executive producing with Peter Grumbine (BuzzFeed Studio’s VP of Video & AI).
The news broke this week at AI on the Lot, where I had the pleasure of sitting on a panel with Pete and Kartik Garg (Amazon’s Head of Product and Tech, AI Studios), in front of an audience of journalists, executives, technologists, and creatives all trying to figure out what the next chapter of storytelling looks like.
It was a wonderful event, and I’ll have more to say about it next week. But one big takeaway is that by and large the community is approaching generative AI the right way. And Amazon’s strong “creators first” approach should be music to everyone’s ears. They’re leading the charge, and leading it responsibly. That’s not the sound of a creative blowing smoke at his corporate overlords. It’s a fact.
Now, the hard part: what comes next as we build a team of talented humans to create a full scripted series using AI as a tool.
Almost everything I’ve made with AI up to this point has been experimental. Trailers. A trippy claymation short. Projects deliberately designed to test the medium, find its edges, and figure out where the cheat codes live. They were short, mostly solo, and built around playing to whatever the models happened to do well that week.
The Amazon pilot was a different animal entirely. It’s an animal I don’t think you could have made even five or six months ago. We were working from a real script. It was dialogue-driven. Heavy on lip sync. The characters have non-traditional bodies and non-traditional faces, which the video models have strong opinions about and are not afraid to share.
So the pilot was already a big jump. And now the jump gets bigger again: from one six-and-a-half-minute proof of concept, made by a very small team; to eight episodes of broadcast-quality content built by a full production team. That’s a different operation entirely. And it’s the puzzle I’ve been thinking about every waking minute since the pickup.
Before I get to the team question, though, I want to talk about how the pilot actually worked. Because the principles that got us through the short may not be the principles that get us through the series.
Every instinct, on every new AI project, screams open the timeline, build a thing, make me feel something. The power of the tools urges you to leap before looking. It’s seductive. And I have been seduced. I always remind myself that the single most valuable thing you can do at the start of an AI project is sit with the tools, in low-stakes prep mode, and figure out the cheat codes.
What does this model do well, with these characters? What does it refuse to do? Which tool is right for which task? What prompt structure consistently produces usable output for this specific art style? Where are the points of failure that are going to bite you in week six if you don’t surface them in week one? And, most importantly, what projects should be abandoned before they get too far?
Lately I’ve jettisoned a bunch of good work on a photorealistic film because the uncanny valley is still very much a thing. And on a pre-teen adventure because the models don’t like to generate children (which, ok, fair play).
Both abandonments hurt, badly. But not nearly as badly as struggling with inadequate tools would have hurt. Storytelling is supposed to be fun. Wrestling with inadequate tools for the job is not fun at all.
The honest truth is that on this project we finished the first two weeks of an eight-week schedule with almost nothing tangible to show for it. No cut footage. No assembled scenes. Just a growing pile of tests, notes, and a slowly-building confidence that we knew how to execute. That confidence turned out to be the most valuable asset we produced in that time. And when we started building shots in earnest, we weren’t guessing anymore. We were running a play we knew the shape of. And that allowed us to truly run.
The R&D phase isn’t fun. It’s not sexy. But it’s where the film gets made.
There are different ways of doing lip sync with AI. I can’t promise that we found the best solution, but we found the one that worked for this project. It’s the method that let us breathe and feel confident.
Lip sync in AI animation is notoriously brutal. You can’t just point the model at a recorded performance and expect it to match. The audio comes out close, or it comes out exact for half a sentence and then drifts, or it comes out nothing like what you asked for. One option we discussed was to fix this by hiring an After Effects artist to manually adjust mouth shapes frame by frame. I’m all for creating jobs, but for this it seemed counter-intuitive.
What we discovered – and this was the actual unlock – is that simple 2D animation is incredibly forgiving in the edit timeline. Unlike high-end CGI, where every frame is unique and any tampering shows immediately, simple 2D animation is full of repeated frames. A character holds a freeze for six frames. Her eyes flick left, then right, then she smiles. Those beats are loopable. Cuttable. Stretchable. You can double time a two-second “character looks around” beat into one second and nobody notices. You can slow a beat down by 300% and still get away with it. It’s enormously forgiving, and allows for an enormous amount of control and finesse.
You can tweak the performance of the voice actors AND the performance of the AI.
That’s the kind of control I’m always looking for.
The surgery was possible. It was repeatable. And once we had it down, lip sync stopped being the existential threat to the project that it looked like in week two.
That’s the kind of cheat code that changes what’s possible. Not because it’s clever, but because it converts something terrifying into something laborious.
Laborious is fine. Laborious is just work. Terrifying is what kills projects.
Here’s where the real puzzle starts. Up to now, almost all of my AI work has been a one-man-band operation: me, the model, the timeline. A couple of gentle-handed collaborators tops. Lean is good because the work is improvisational by necessity: the model’s output reshapes the plan in real time. Shot A comes back with a surprise; and because you love the surprise, shot B has to change to accommodate it; so by the time you’re working on shot C, the storyboard you started with bears almost no resemblance to where the sequence actually went.
Before anyone accuses me of yielding the creative reins to a machine, that’s not what I’m describing. Collaboration is the same with humans: you can put twenty writers on the same problem and they’ll give you twenty different paths to the same destination. Working with humans, your job as the showrunner is to hold the destination tightly, but hold the path loosely. AI is the same. Some of the model’s ideas are bad. Some are offensively wrong. Some are better than mine. Some are just different in a way that opens a door I wouldn’t have found on my own. The job is recognizing which is which and steering accordingly. That’s how humans collaborate with each other. It’s also how humans collaborate with machines.
That’s the job.
The trouble is that this kind of improvisational, shot-by-shot rebuilding doesn’t scale logically to a team. I know a lot of AI filmmaking teams who have struggled to work as a team. I have too. It’s hard to collaborate when one of your collaborators is fundamentally unreliable.
So how do you do it?
I went looking for inspiration from the greats. A colleague pointed me to The Illusion of Life: Disney Animation by Frank Thomas and Ollie Johnston. It’s one of the foundational books on the medium. (To the animators out there: yes, I know. It’s a classic. Allow me my late-arriving wonder.)
And what I found is that a hundred years ago, when Disney was first inventing animation as a medium, they were wrestling with our exact problems. How do you take a drawing – an inert line on a page – and make it feel? How do you make a viewer care about something that demonstrably isn’t alive? Those were genuinely open questions then, the same way “how do you direct an AI film” is a genuinely open question now.
The workflow Disney settled on was elegant. The director – or the lead animator – drew the keyframes. The big poses. The major moments. Then those keyframes got handed off to a team of animators who drew all the in-between frames that connected them. One person owned the vision. Other people owned the bridges.
The closest thing we have to keyframes in current AI parlance is “start frame” and “end frame.” But really, that’s not close at all. The start and end frames of a shot are often fundamentally the least interesting frames. They aren’t where the emotion live. They aren’t where the story is told. As the AI models are maturing there will be a shift towards models that are built to reference start, end, and key frames.
(Currently the work-around is to explain to the model what a key frame is and where it should happen in every prompt. It’s tiresome. It’s annoying. But it’s gonna get better.)
I think that’s the answer for us, too.
As we’re writing the scripts for the series, I’ll do all the keyframe work in parallel. Build the locations. Lock the character designs. Set the cameras. Block the scenes. Stage the visual gags. Set the style. Create the entire show in keyframes: a rigorous, fully-realized blueprint (possibly even a fully scored and edited animatic) before anyone generates a single moving shot.
Then the team takes those keyframes and makes them move. Lip sync. The micro-edits. The fills between A and B. And of course we leave plenty of room for them to improvise, to push past what I “drew,” to bring their own ideas. It’s a collaboration, not a paint-by-numbers. But the spine is locked, the destination is shared, and nobody has to rebuild shot three because shot two surprised everybody.
My gut tells me this is going to work, and that it’s going to be fast.
Of course, anyone who’s spent serious time with AI will tell you that when your gut tells you AI is going to be fast at something, there’s a fifty-fifty chance you’re about to get a swift kick in the ass.
So we’ll grasp onto this plan the same way we hold every other plan in this medium – holding our intention firmly, and our execution loosely. Stay flexible. Stay adaptable. Stay ready to throw the whole workflow out the window if the technology surprises us in either direction over the multi-month production.
Which it will. It always does.
And that, honestly, is the part I’m looking forward to most. Every challenge in this medium is an invitation to creativity. We’re inventing this process together, in real time, and I’m lucky enough to get to spend the next several months figuring out whether a hundred-year-old workflow holds up against the most advanced storytelling tools that have ever existed.
As much as I can, I’ll be sharing what we learn – the wins, the disasters, the cheat codes – as we go. Thanks for joining me on this wild ride.
If this is your first time here: welcome. Thanks for reading and sharing. The comment section is open!
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.