Prompt → real song, then edit the words without moving the band.
A staff-engineer brief: Suno-class generate on open weights, then lyric lock, then licensed voices. Stages, goals, kill criteria, and who does the GPU work. The canonical file is BRIEF.md in the repo — that is what a remote Claude session can actually read.
Stage goals
Ship a Suno-class product loop first (text to full song with vocals, lyrics, library, player). Then add the thing Suno does not actually do: new lyrics on an existing mix, instrumental frozen, timing locked. Then a menu of licensed voices — never an unlicensed famous throat. Stage 2 does not start until Stage 0 writes a go/no-go. Stage 3 does not start until you own a voice.
Viability bakeoff
Prove or kill lyric-lock with open models on a known 8-bar clip before writing a training script.
- Claude
- Done on CPU: packer, G2P, eval harness, A/B/C lyrics, synth fixture, GPU runbook. Remaining: go/no-go memo after GPU audio + ears.
- GPU
- Demucs, WhisperX/MFA, ACE-Step repaint, RVC same-lyric control, optional naive vocode+crossfade.
- Not this stage
- No Next.js scope creep. No foundation-model train. No celebrity voice.
Definition of done
- Alignment exists
WhisperX or MFA produces usable phone/word times on the dry vocal. If alignment is garbage, stop — later stages will lie.
- Same-lyric RVC is clean
Convert the original vocal to another licensed timbre, mix back, drums untouched. This is the YouTube-cover ceiling. If this fails, the stems are too wet to edit.
- Feasible lyric can infill
Same phoneme-count line: new words, original instrumental, no drum movement, consonants on the old onsets ±30 ms on the dry verse.
- Infeasible lyrics are rejected
The +25% and half-syllable lines are either rewritten by the packer or explicitly failed. Never silently chew vowels and call it locked.
- Wet chorus is measured, not wished
Repeat the feasible line on a stacked/wet chorus. Record whether bleed/reverb kills the splice. That number decides Stage 2 training, not vibes.
Kill criteria
- Band or drums move on a supposed lock (it is a cover — label it, do not ship as lock).
- Non-edit region regenerates or loudness-matches into a new noise floor.
- Alignment cannot be trusted on the dry verse.
- Wet chorus collapse with no salvage path (then lock is verse-only, or kill the feature).
Deliverables
- A frozen test pack: one dry-ish mix, 8 bars, isolated vocal + instrumental, alignment JSON, original lyrics.
- Three lyric conditions generated by Claude: same phoneme count, +25% syllables, half syllables.
- Baseline audio for each condition, plus a same-lyric RVC control.
- A one-page memo with metrics, kill-criteria hits, and a Stage 2 go / no-go / pivot.
Exit. A written go/no-go. Go = Stage 2 infill is the gap (train $25k–$120k GPU band) or glue is enough. No-go = wrap generate-only (Stage 1) and drop lock, or restrict lock to dry stems the user uploads.
Suno functionality, staged
The product loop people mean by “Suno”: prompt, optional lyrics, vocals, library, player. Then their editor: extend, stems, cover. Lyric lock is the feature they think already exists. Unlicensed famous voices are not a feature.
| Feature | Suno | We ship | Notes |
|---|---|---|---|
| Custom mode (style + lyrics) | Ships | Stage 1 | Primary generate path. Lyrics are a request to the music model, not a lock to an existing mix. |
| Simple / describe a song | Ships | Stage 1 | Claude (or any LLM) writes titled lyrics + [Verse]/[Chorus] tags, then the music model sings them. |
| Instrumental toggle | Ships | Stage 1 | Same worker, no vocal stem. Easy. |
| Full song with vocals | Chirp v5 | Stage 1 | ACE-Step 1.5 or YuE. Expect open-weight quality, not v5. That is the honest 2026 clone. |
| Library + player + download | Ships | Stage 1 | Object storage + CDN later. Local files + Postgres/SQLite is fine for the first GPU slice. |
| Credits / queue UI | Ships | Stage 1 | Mock credits are fine. Real billing is Stage 4. The queue must survive a 30s–2min GPU job. |
| Extend | Ships | Stage 1b | Continue from the last N seconds. ACE-Step-class models already have this hook. Band is allowed to evolve. |
| Cover / restyle | Ships | Stage 1b | New performance of the same song idea. Drums may move. Do not advertise this as lyric lock. |
| Get stems | Ships | Stage 1b | Demucs/UVR on the master. Needed anyway for lyric lock. Ship as a user feature once the worker exists. |
| Add vocals / add instrumental | Ships | Stage 1b | Latent edit / second pass. Useful. Still a regenerate, not a surgical lyric swap. |
| Replace a section | Studio | Stage 2 | Suno’s replace often re-performs the band. Our replace must freeze non-edit audio bit-identically. |
| Lyric lock (new words, frozen mix) | Does not ship this | Stage 2 | The differentiator. Duration packer + infill + stem mix. Stage 0 decides if open weights are enough. |
| Persona / custom voice | Ships | Stage 3 | Licensed or user-consented banks only. A menu of 5–20 singers beats one stolen star. |
| Feed / social / Copilot / iOS | Ships | Stage 4 | Do not build a social network before the songs are real. Web first. |
| Unlicensed famous likeness on a hit | Legal war | Never | WMG settlement was reported ~$500M. Not a sprint. Not a Claude task. |
Stage 0 bakeoff
One dry-ish commercial-style track you have rights to (or an original). First 8 bars of a verse, not a wet chorus. Export: mix, instrumental, vocal, 44.1 kHz WAV. Do this before any training script. Claude can write the harness without a GPU; the go/no-go needs one.
- 01Demucs or UVR → vocal + instrumental. Keep the original mix as ground truth.
- 02RVC/Seed-VC on the original vocal, same lyrics, mix onto the instrumental (control: YouTube cover).
- 03WhisperX or MFA → word/phone times. Hand-correct the 8 bars if needed. This file is gold.
- 04Claude packer + G2P assigns phonemes into the original span (MeloDRP-style ratios that sum to the old duration).
- 05ACE-Step lyric-edit / repaint constrained to the vocal region if the API allows; else score-path SVS over the original instrumental.
- 06Naive vocode + crossfade last: non-edit samples must be bit-identical. If this baseline wins, do not train.
- 07Repeat A on a wet stacked chorus as a stress test, not as the first test.
Hard rules
- Do not train a new foundation music model. ACE-Step already paid that bill (~32k A100-hours published). Wrap it.
- Do not treat the Tone.js toy studio as the music model. It is a seeded synth that proves the product loop.
- Do not clone living artists or scrape celebrity likeness. Licensed voices only. Famous names are a legal stage, not an ML stage.
- Do not start Stage 2 training until Stage 0 (bakeoff) writes a go/no-go memo with audio and metrics.
- Claude writes glue, lyrics, duration packing, eval, and product code. A GPU does Demucs / WhisperX / ACE-Step / RVC. Claude cannot be the singer and cannot hear mix artifacts.
- Non-edit regions of a lyric-lock job must stay bit-identical to the source mix. If the drums move, it is a cover, not an edit. Fail the job.
- If asked for both a lyric-lock and an unlicensed star voice, ship the lock on a licensed or user-owned voice. Refuse the star.
Who does what
What Claude does this week
- 1. Do not train anything
CPU Stage 0 is implemented. If you start a foundation-model run, you have failed the assignment.
- 2. Rent one GPU for inference
Packer, eval, synth fixture, and GPU.md are in scripts/bakeoff/. The gate is Demucs + WhisperX + ACE-Step + RVC on a 4090/A100. No go/no-go without those WAVs and human ears.
- 3. Keep the studio as the generate UX
src/lib/generate.ts is the swap. When a worker exists, return { renderer: 'gpu', jobId, wavUrl }. Until then, do not pretend the synth is Chirp.
- 4. Packer is the product surface
Locked / Crammed / Broken plus rewrite. Python G2P in the worker later; the TypeScript heuristic is what Stage 0 uses now.