用 Claude Code 以代码生成音乐视频：从 STORYBOARD 到逐帧渲染的工程复盘

This video (the script, the voice timings, the source code, the storyboard, the briefs the subagents got, the grade a reviewer gave an earlier draft) is the real artifact from making this video. Claude Opus 5.5 built it in Claude Code, starting from an open-source project, and this post walks through the innards. The seed: a video is a function of time It started from JohnHeibel/PDoomVideo, the source of a music video Opus 5.5 made entirely in code: p5.js, a studio page, a render.mjs that drives headless Chrome, and a STORYBOARD.md written before any chapter code. The core idea is one rule: every frame is a pure function of t. window.renderAt = async (t, type = 'image/png', q = .92) => { T = t; await redraw(); composite(t); return outC.toDataURL(type, q); }; Enter fullscreen mode Exit fullscreen mode No Math.random(), no clock, no state carried between frames. Randomness comes from a seeded hash, the old GLSL one-liner: const hash = n => { const s = Math.sin(n * 127.1 + 311.7) * 43758.5453; return s - Math.floor(s); }; Enter fullscreen mode Exit fullscreen mode That one constraint buys everything else: frames can render out of order, on several browsers at once, resumably. Any single frame can be regenerated on demand, which is what lets the video show itself at frame 1,365 while it's on frame 2,000-something. The first real change was a portability fix. The repo launched Chrome with --use-angle=d3d11 (Direct3D, Windows only). On a Mac that fails with "Error creating webgl context"; --use-angle=metal plus the macOS Chrome path fixed it. Script → voice → timeline The narration is script.md: 15 numbered lines. Words in {braces} are cue words, where a demonstration fires. Each line goes to ElevenLabs' with-timestamps endpoint, which returns the audio plus a start and end time for every character: { "characters": ["T","h","o","s","e"," ", ...], "character_start_times_seconds": [0.0, 0.046, 0.081, ...] } Enter fullscreen mode Exit fullscreen mode tools/voice.py turns characters into words, finds each line's {cue} word, stitches the 15 clips at exact offsets, and writes timeline.js. The flash in the video fires on the frame where the word "now" is spoken: const cueT = n => line(n).cue ? line(n).cue.a : line(n).a; // when line n's {cue} word is spoken Enter fullscreen mode Exit fullscreen mode For music, the same slot is filled differently. Suno's .m4a exports carry a hidden mov_text subtitle stream with every lyric line timestamped (ffmpeg -map 0:s:0 subs.srt); Whisper large-v3-turbo adds word times; difflib.SequenceMatcher aligns what Whisper heard to the real lyrics; and a comb search over the onset envelope finds the beat grid. Storyboard before code, then subagents Before any chapter code, Opus wrote STORYBOARD.md (one row per script line: the shot and the demonstration on the cue word) and ANIMATION_GUIDE.md, the contract every builder follows: the API, the "every example is itself" rule, legibility, and the hand-off frames between chapters. Then it handed three chapters to three subagents in parallel, each with a short brief. The brief for chapter B is shown in chapter B. Each subagent verified its own work with contact sheets and reported back; those reports are in the repo too. The critic A reviewer agent graded draft 1. The specific frame line 11 talks about got 4/10: "the side being graded is a placeholder, so the frame promises a review it can't show yet." It also caught that the HUD said 76.033 s when the target was 76.008 s. Those notes went back to the builders, and the final video shows the real draft-1 frame beside the fixed one, annotated with those exact notes. The blind spot Partway through this project, the model couldn't see images at all: every screenshot came back (media removed — rejected by API). So it checked its own frames two other ways: Pixel math: a small script reports each frame's mean colour, bright-pixel share and the bounding box of bright content. This catches blank frames, blown-out flashes and content in the wrong place. Other agents: reviewer subagents, who could see, described frames and scored them. The video shows both, using its own frames. Rendering render.mjs opens four headless Chrome workers. Each pulls the next frame index from a shared queue, calls renderAt(i / 30), and writes a JPEG. ffmpeg then stitches the frames to the narration with libx264. One real bug from the music-video build: with a subtitle stream as an input, ffmpeg -shortest never finishes. The encode sat at 0% CPU for 20 minutes. The fix is an explicit -t . The ending The last shot is a DEV page with a player inside it, and on the word "inside" the player shows the video recursively. That isn't faked: the renderer draws its own output canvas back into the player, so it's a real Droste loop. One honest wrinkle: the video was rendered before this post existed, so that page is labelled "DRAFT · the real post comes later." You're reading the real one now. Related reading on DEV: HTML/CSS Animation to Video (MP4): the Headless, Deterministic Way · I asked an agent to make a product video. It wrote HTML and rendered an MP4 · How to make AI videos: production diary of a one-minute film The video above is draft 2: 3,224 frames at 30 fps, 107.5 s. The 4/10 was the reviewer's grade on draft 1.