Type a sentence like "melancholic synthwave with a driving bassline and a female vocal" into an AI music tool and thirty seconds later you have a finished track — verse, chorus, mixed vocals, mastered levels, all of it. No MIDI files, no sample libraries, no musicians. That output is the end product of a pipeline that has almost nothing in common with how music software worked five years ago. It's not sequencing pre-recorded loops. It's generating a continuous audio waveform, sample by sample or token by token, the same way a language model generates text one word at a time.
Understanding that pipeline matters beyond curiosity. The mechanics determine what these tools can and can't do reliably, what data they were trained on, and — increasingly — who gets paid when a generated song sounds like it belongs on the radio. That last question just became a live commercial issue: Suno and Udio, the two companies that pushed AI music generation into the mainstream, are in the process of rebuilding their platforms on licensed catalogs rather than the scraped training data that got them there. How the underlying models work is directly why that rebuild is hard.
What AI music generation actually produces
There are two fundamentally different things people mean when they say "AI-generated music," and conflating them causes most of the confusion around this space.
Symbolic generation produces a score: MIDI notes, chord symbols, or sheet music. A model predicts which notes come next, at what pitch, duration, and velocity. This is the older approach — it dates back to algorithmic composition experiments in the 1990s and 2000s — and it still requires a separate synthesis or performance step to become audio you can listen to. It's good at capturing musical structure (harmony, rhythm, melody) but the resulting audio only sounds as good as whatever instrument or synthesizer renders it.
Audio (waveform) generation produces the actual sound pressure signal — the thing that comes out of your speakers — directly. This is what Suno, Udio, Google's MusicLM/Lyria, Meta's MusicGen, and Stability AI's Stable Audio do. There's no intermediate score; the model has learned statistical patterns in raw or compressed audio and generates a new sequence that fits those patterns and matches a prompt. This is dramatically harder computationally — a 3-minute song at CD quality is roughly 8 million samples per channel — but it's what produces the fully mixed, vocals-included, radio-ready output that made tools like Suno viral in 2023 and 2024.
Modern commercial tools are almost entirely in the second category. The rest of this piece focuses on how waveform generation works, because that's the technology actually generating "songs" rather than sheet music.
Audio tokenization: compressing sound into a learnable vocabulary
Raw audio is a brutal format to generate directly. A single second of CD-quality audio is 44,100 samples per channel. A three-minute song is close to 16 million numbers. No transformer architecture generates sequences that long efficiently — attention costs scale quadratically with sequence length, so this needs to shrink drastically before any generative model can touch it.
The fix is the same one that made large language models tractable for text: tokenization. Instead of predicting raw audio samples, the system first compresses audio into a much shorter sequence of discrete tokens, then trains a generative model to predict those tokens, then decodes tokens back into audio at the end.
This compression step is handled by a neural audio codec — models like SoundStream, EnCodec, or DAC (Descript Audio Codec). Here's the general process:
- Encode: A neural network compresses a chunk of raw audio into a much shorter sequence of continuous vectors, similar to how a JPEG encoder compresses pixel data.
- Quantize: Those continuous vectors get mapped to a finite set of discrete codes — "audio tokens" — using a technique called residual vector quantization (RVQ), which uses several small codebooks stacked on top of each other to capture both coarse and fine acoustic detail.
- Generate: A sequence model (usually a transformer, sometimes a diffusion model) is trained to predict the next token or denoise a full sequence of tokens, conditioned on a text prompt, melody reference, or lyrics.
- Decode: A decoder network converts the generated token sequence back into a continuous audio waveform.
The effect is that a model can generate roughly 50-100 tokens per second of audio instead of 44,100 raw samples per second — a compression ratio steep enough that transformer-style sequence modeling becomes computationally feasible, while the codec is trained to preserve enough acoustic detail that decoding back to audio doesn't sound degraded.
Why this borrowed idea from language models
Audio tokenization exists specifically so that the enormous amount of engineering progress made on text-generating transformers — efficient attention, scaling laws, instruction-following, prompt conditioning — could be redirected at audio. Once sound is a sequence of discrete tokens, "generate a song" becomes structurally the same problem as "generate a paragraph": predict the next token given everything before it and a conditioning signal (the text prompt). That's the core insight behind Google's AudioLM and MusicLM, and it's the lineage most current commercial tools descend from.
The two dominant model architectures
Once audio is tokenized, there are two broad families of models used to actually generate the token sequence.
| Approach | How it generates | Strengths | Weaknesses |
|---|---|---|---|
| Autoregressive transformer | Predicts tokens one at a time, left to right, conditioned on prior tokens and the prompt | Strong long-range coherence, good at following structure and lyrics timing | Slow to sample (sequential generation), can drift or repeat over very long sequences |
| Diffusion model | Starts from noise and iteratively denoises an entire audio (or token) sequence at once | Fast parallel sampling, strong fidelity and texture, good for shorter fixed-length clips | Historically weaker at long-range structure like verse/chorus arrangement without extra conditioning |
Most production systems today are hybrids rather than a pure implementation of either. A common pattern: use a transformer to plan coarse structure (which section, what key, what tempo, roughly what should happen when) and use a diffusion or cascaded model to fill in the fine acoustic detail — the timbre, texture, and mix. This mirrors how image generation evolved too — coarse-to-fine generation tends to outperform single-pass approaches for anything longer than a few seconds.
Full songs also need explicit handling for things that plain audio generation doesn't naturally produce:
- Structure (intro, verse, chorus, bridge, outro) is usually controlled by conditioning tokens or a separate structure-planning step, not left to emerge from raw generation.
- Lyrics and vocal alignment require either a text-to-speech-style alignment model or joint training on paired audio and time-stamped lyrics, so sung words land on the right beats.
- Instrumentation and mix are influenced by prompt conditioning (genre, instrument, mood tags) learned from how those tags correlated with training audio.
Why it matters now: the licensing rebuild
For the first two years of commercial AI music generation, the industry ran on an unresolved legal assumption: that training a model on recorded music scraped from the internet, without licenses, was defensible the way training an LLM on web text was being argued to be. Major labels didn't accept that framing. Universal Music Group, Sony Music, and Warner Music sued both Suno and Udio in mid-2024 for copyright infringement over training data.
What's happening now is the practical resolution of that fight, and it's a direct consequence of how these models are built. Because waveform generation models learn statistical patterns from the actual audio they're trained on — not from a licensed database of composable elements — the labels' leverage was real: the quality and style range of Suno's and Udio's output was a direct product of what recordings went into training. That's what makes the current shift significant. Suno and Udio are rebuilding on licensed catalogs, which means the raw material that shapes what the model can generate is being renegotiated at the source, not patched after the fact with output filters.
This has a few concrete implications for anyone watching the space or building on it:
- Training data provenance becomes a product feature, not a footnote. A model trained on licensed catalogs can advertise clean rights and revenue-sharing with rightsholders — a real differentiator for commercial and enterprise use where indemnification matters.
- Style range may narrow before it widens again. Licensed catalogs are a smaller, more controlled dataset than "everything scraped from the internet." Expect near-term shifts in what genres and eras a model handles well as training data composition changes.
- Attribution and payout infrastructure has to be built into the generation pipeline itself. If a model can trace which licensed recordings influenced a given output's style, that's the technical hook needed for royalty distribution — a nontrivial addition to a system originally built to just generate plausible audio.
- The rest of the industry is watching the precedent. How labels, platforms, and courts settle this shapes the ground rules for every company doing generative audio, image, or video work trained on copyrighted creative material.
Practical implications for businesses and builders
If you're evaluating AI music tools for a product — background music for video content, jingle generation, game audio, an app feature — the architecture explains what you should actually expect.
What these systems are reliably good at:
- Producing full, mixed, radio-quality-sounding audio from a short text prompt in under a minute.
- Matching broad style, mood, tempo, and genre descriptors.
- Generating usable instrumental beds and background music at low marginal cost.
- Iterating fast — generating dozens of variations to pick from is cheap compared to hiring a composer for dozens of drafts.
What they're still inconsistent at:
- Precise structural control (exact bar counts, exact section lengths, hitting a specific sync point in a video).
- Faithful, intelligible lyrics over long songs — words can slur, repeat, or drift out of alignment, especially past the two-minute mark.
- Matching a specific existing artist's voice or style closely enough for professional broadcast use without raising rights questions.
- Consistent audio quality across genres the training data underrepresents.
For a business decision, the practical questions to ask a vendor now look less like "can it make a good song" and more like:
- What catalog is the model trained or fine-tuned on, and is that licensing verified or disclosed?
- What's the commercial usage license on generated output — can it be monetized, used in ads, used in a product you sell?
- Is there an indemnification clause if a generated track is later challenged as derivative of a specific copyrighted recording?
- What controls exist over structure, length, and lyric accuracy for production use, versus casual generation?
These questions didn't matter much when the tools were novelties. They matter now that companies are shipping AI-generated audio inside paid products.
Real limitations and open questions
A few things the current generation of models genuinely does not solve well yet:
Long-form coherence. Generating a coherent 3-4 minute song with a recognizable verse-chorus-verse arc, a bridge that actually feels like a bridge, and a satisfying ending is still harder than generating a compelling 15-second loop. Structure conditioning helps, but songs can still feel like they're drifting or repeating rather than developing.
Evaluation is subjective and unresolved. There's no equivalent of a BLEU score or accuracy metric that reliably captures "is this a good song." Human listening tests are expensive and slow, and automated proxies (spectral similarity, prompt-adherence classifiers) correlate only loosely with what people actually enjoy.
Style attribution is technically fuzzy. Even with licensed training data, pinning down "how much did any single recording influence this specific output" is an open research problem, similar to attribution challenges in image and text generation. That fuzziness is exactly what the ongoing litigation and licensing negotiations are trying to force into a workable commercial framework.
Voice cloning and likeness remain contentious regardless of licensing. Training on a licensed catalog addresses recording copyright, but a generated vocal that sounds like a specific, identifiable performer raises separate right-of-publicity and voice-likeness questions that catalog licensing alone doesn't resolve.
Compute and latency tradeoffs are real. Autoregressive generation of a full song, token by token, is slow relative to a diffusion model's parallel denoising — but diffusion approaches have historically traded off some long-range structural coherence to get that speed. Production systems are still actively tuning this tradeoff rather than having settled it.
What to watch next
A few threads worth tracking if this space affects your work:
- How licensing terms shape output quality. Watch whether Suno's and Udio's rebuilt, licensed-catalog versions maintain the stylistic range that made them popular, or whether output narrows initially while catalogs scale up.
- Label-platform revenue-share models. Whether royalties get paid per-generation, per-subscription, or via some blended pool will set precedent for every other generative-audio company negotiating similar deals.
- Structure and lyric-accuracy improvements. These are the most commercially relevant weak points; expect vendors to compete hard on getting full songs to hold together over 3+ minutes with clean, intelligible vocals.
- Convergence with symbolic tools. Expect more hybrid workflows where AI-generated audio gets exported into traditional DAWs (Ableton, Logic, Pro Tools) as stems for human editing, rather than treated as a finished, unmodifiable product.
- Enterprise and B2B licensing products. As catalog licensing matures, expect purpose-built tiers aimed at advertising, game studios, and media companies that need indemnified, commercially safe generated audio at scale, distinct from consumer-facing creative tools.
FAQ
How does AI music generation actually create sound?
Modern systems compress raw audio into a short sequence of discrete tokens using a neural audio codec, train a generative model (usually a transformer, diffusion model, or hybrid) to predict a new token sequence conditioned on a text prompt, then decode that sequence back into an audible waveform. It's directly analogous to how large language models generate text token by token, applied to compressed audio instead of words.
What's the difference between AI music generation and MIDI-based algorithmic composition?
Algorithmic composition and symbolic AI models generate a score — notes, timing, velocity — that still needs to be performed or synthesized to produce audio. Waveform generation models like Suno and Udio produce the actual finished audio signal directly, including vocals, mix, and mastering, with no separate performance step required.
Why are Suno and Udio rebuilding on licensed catalogs?
Both companies were sued by major record labels in 2024 over training on copyrighted recordings without licenses. Because these models' output quality and style range are a direct function of their training data, resolving that dispute means renegotiating what audio the models are trained on at the source, rather than filtering outputs after generation.
Can AI-generated music sound like a specific artist?
It can approximate a style or genre closely, and in some documented cases has produced outputs resembling identifiable performers' voices, which is a separate legal issue (right of publicity, voice likeness) from recording copyright. Licensed training data addresses copyright in the source recordings but doesn't automatically resolve voice-likeness concerns.
Is AI-generated music copyrightable?
This varies by jurisdiction and is still being actively litigated and clarified through policy guidance. In the US, purely AI-generated work with no meaningful human authorship has generally not qualified for copyright protection under current Copyright Office guidance, though human-edited or human-directed hybrid works occupy a grayer area.
Why do AI-generated songs sometimes have garbled or repeated lyrics?
Vocal-lyric alignment over a full song is one of the harder open problems in the field — the model has to keep sung words synchronized with generated melody and rhythm over a much longer span than short audio clips typically used in training. This tends to degrade past the two-minute mark, which is why shorter generated clips are usually more reliable than full-length songs.
What businesses are actually using AI music generation for today?
Common commercial uses include background music for video content and social media, game and app audio, jingle and ad-scoring drafts, and rapid prototyping for composers before human refinement. Full replacement of professional composition for high-stakes broadcast or film work remains limited by the structural and licensing issues discussed above.
Teams evaluating AI music generation for a product — from licensing questions to integration architecture — can get hands-on help from Woyce Technologies.