Create Music Variations with Machines

Abstract

This thesis establishes the methodological foundations and key components required for creating music variations with machines, starting from music in audio form. We explore symbolic representations as a unifying abstraction for content-aware music modeling, motivated by their interpretability, controllability, and modularity compared to end-to-end audio-based approaches. Such properties make symbolic modeling particularly suitable for sophisticated creative tasks, including lyric composition, instrumental adaptation, and long-range musical structuring. Building on this perspective, the thesis develops a two-stage analysis--generation framework that first extracts structured symbolic information from raw audio and subsequently performs conditional lyric or music generation based on that content. To investigate how music variation can be realized in this manner, we decompose the overall pipeline and systematically address key analytical and generative challenges involving symbolic representations.

In the analysis stage, we study three complementary aspects of musical content: lyrics, note events, and instrumental roles. (1) For lyric transcription, we demonstrate that transfer learning from self-supervised speech models enables accurate recognition of singing voice audio under limited supervision, particularly when combined with hybrid CTC/attention training. (2) For music transcription, we examine the strengths and limitations of encoder-only Transformers within a multi-task framework, showing that frame-level architectures benefit the detection of specific note attributes, while token-level prediction with encoder-decoder models offers greater overall performance. (3) Beyond notes and text, we further investigate lead instrument detection as a representative case study for instrument-role analysis, illustrating how track-wise attention mechanisms can effectively capture melodic prominence and functional organization among instruments in multitrack music.

On the generation side, the thesis covers four tasks spanning lyric and music generation. (1) For singable lyric translation, we introduce a controllable generation framework that incorporates constraints such as rhyme, rhythm, and word boundaries, demonstrating that domain-specific musical requirements can be enforced within general-purpose language models. (2) For melody-conditioned lyric generation, we propose decoupling task learning from controllability learning, augmented by auxiliary supervision based on syllable-level properties, leading to improved structural control and melody--lyric compatibility. (3) For music arrangement, we design a task-agnostic fine-tuning objective and a compact symbolic music tokenization scheme, enabling a generatively pre-trained model to outperform task-specific baselines across multiple arrangement scenarios, including piano reduction, band arrangement, and drum arrangement. (4) Finally, to support long-range musical structure, we propose a new paradigm for full-song multitrack symbolic music modeling that operates on musically meaningful units rather than fine-grained note-attribute tokens. In this framework, phrase-level note sequences are compressed into compact latent representations, substantially reducing effective context length, which in turn enables efficient and high-quality full-song generation.

Taken together, these investigations demonstrate the feasibility and versatility of symbolic music modeling as a foundation for content-level music variation. By bridging raw audio with symbolic analysis and manipulation, this thesis presents a modular, interpretable, and controllable pathway toward music AI systems that support creative workflows across languages, instruments, and musical contexts.