Ncog.link devlog 4: Continued work on muse

Archived Projects ncog.id
1

For now, I’m categorizing work on muse with ncog.link, at least as long as it falls under general audio synthesis or playback. If it pertains to a project that is built with muse but isn’t related in some way to ncog, I’ll put it in another category.

This past week has been a rollercoaster for me. On my Twitch channel, I got lucky with several hosts, and I reached affiliate. I then got a little under the weather on Friday and spent the last couple days recovering a bit. Honestly, I felt mostly ok yesterday, I just wasn’t feeling in the mood to code.

Musings on Muse

To recap this past week, my work been entirely on muse. The first major project was to refactor how Envelopes worked so that Envelopes are independent of Oscillators. I needed to be able to have more than one Envelope exist to drive different Parameters such as Detune or Pan, not just an Oscillators amplitude.

The next project is part of a goal of making a file format to represent music compositions that can be loaded and saved to disk through serde. For the Instrument format, I thought up this format:

Instrument(
    name: "Synth",
    envelopes: {
        "main-volume": (
            attack: Milliseconds(50),
            sustain: Sustain(0.6),
            release: Milliseconds(50),
        ),
    },
    nodes: {
        "output": Amplify(
            value: NoteVelocity,
            input: "unison",
        ),
        "unison": Unison(
            detune: Value(0.25),
            input: "oscillators",
            quantity: 5,
        ),
        "oscillators": Multiply(
            inputs: [ "sine", "triangle" ],
        ),
        "sine": Oscillator(
            function: Sine,
            frequency: NoteHertz,
            amplitude: Envelope("main-volume"),
        ),
        "triangle": Oscillator(
            function: Triangle,
            frequency: NoteHertz,
            amplitude: Envelope("main-volume"),
        ),
    },
)

The format for this file is ron. I think that this file format is serviceable, but I’m confident anyone who wants to produce sounds will desire a graphical interface. I’m still trying to noodle out how to approach that within Kludgine.

My first approach had me deserializing into an intermediate format, and then directly instantiating the various samplers. Unfortunately, the only way to do this meant converting each time you played the note because you couldn’t clone a PreparedSampler. A very helpful Twitch streamer and crate author museun helped me figure out how to make them Clonable despite me thinking that the Rust type system was preventing it. Unfortunately, as I was finishing up, I realized that Clone didn’t make sense, because what does cloning an Envelope mean? What if an Envelope was in the middle of execution? Do you clone without the execution state? But that doesn’t fit the spirit of what std::clone::Clone means in Rust.

I spent the rest of the day off-stream adding an intermediary format. Here’s the current nomenclature:

  • muse::instrument::serialization::Instrument is what is serialized and deserialized with Serde. This step is mostly just a syntax check – the format must be able to match the types defined in this module.
  • muse::instrument::LoadedInstrument<T> represents an in-memory “template” of an instrument that can be instantiated into a PreparedSampler for a given Note. Converting from serialization::Instrument into LoadedInstrument will report any structural errors, such as broken node references or cyclical references. Creating a PreparedSampler is a process that cannot generate an error.
  • muse::node: This namespace holds all of the nodes that can be turned into PreparedSamplers. These are the types that LoadedInstrument uses.
  • PreparedSampler: The output node in the LoadedInstrument is instantiated, passing in any parameters necessary and instantiating those as needed. Each time a Node is passed into another Node, it is boxed into a PreparedSampler. The interface allows fn sample(&FrameInfo) -> Option<Sample> to be invoked.
  • Sample: A Sample currently always is a stereo sample, with a left and right value.
  • FrameInfo: Contains information about sample rate, current global clock, and the note being synthesized.

This refactoring took a while, and I continued working on it into the next day. In the end, I am overall pleased with how this works now, and it even supports consumers of muse to be able to implement their custom Node type and associated Samplers by just implementing a few traits.

Unfortunately, when I got a bit under the weather on Friday, I started having a few performance doubts on the project, so I wanted to approach multithreading the code. I had just implemented a Unison sampler, which converted a sound using two oscillators into ten oscillators when using a Unison quantity of five. The performance overhead of each note playing in polyphony was potentially going to be an issue.

When I finally decided to start coding some last night, I further demoralized myself by having many failed attempts at introducing multithreading. The primary win that I achieved was adding a buffer between my sampler thread and the cpal thread, so that the cpal thread could always fill its buffer, and I could peg a single thread at 100% generating samples if necessary.

It just wasn’t enough. I think the only approach to parallelization will be to hand-write my own thread-pool task system. I would use primitives from crossbeam to make it easier. My attempts at using rayon and also just manually creating threads created more overhead from the context switching than the benefits of multithreading allowed. I didn’t try the approach of one thread per playing note, because in my head, it seemed that at that point, I might as well try to optimize for the number of threads against the number of CPUs to try to minimize context switching.

I will probably attempt that approach this week because I want to have at least 64-note polyphony… but I’m not sure what value of Unison I want to shoot for. However, if I can’t utilize multiple threads to generate the audio, my audio generation thread may cause a disproportionate power draw for my game engine. I want my game engine to be reasonably efficient to try not to be super hungry on battery draw for laptop or mobile users. I know running a game will always eat up the battery, but I also know that in general, splitting loads across multiple cores vs. pegging a single core will use lower power. The issue I was running into was that while I could achieve more polyphony on my beefy Ryzen 3800x, the total CPU usage was significantly over the single-core CPU usage.

In the end, I have a good mental image of how I want to approach that last performance refactor. This morning, however, I broke out cargo flamegraph and started profiling my code. I quickly remembered museun mentioning RwLock’s performance wasn’t that great, and I switched my ControlHandles from using a RwLock to using crossbeam’s AtomicCell. That produces a significant increase in performance, because Envelopes during each sample in many phases whether it should stop. This means at 44.1KHz sampling rate, the function was being called 44,100 times per envelope, and my basic example has two Envelope instances.

The next performance improvement was to optimize Unison not to use heap allocations (a Vec in this case). It was a simple refactor.

With these two optimizations, I can now lay both of my arms across my midi keyboard depressing pretty close to my goal of 64 notes (I think?) when running in Release mode. As you can tell, I take a very scientific approach to testing. :slight_smile:

What’s Next?

Automatic Accompaniment

For Muse/Amuse, I haven’t had a chance to share my piano story. I wrote it up thinking I’d put it in last week’s devlog, and then I quickly realized how long it was. I’m going to reveal the project that is inspired by a practical desire of mine. I’m currently learning a few Game of Thrones songs arranged by an incredibly talented YouTuber NDG. Specifically, I’m pushing the limits of my playing abilities (and getting better each week for it), and attempting to play both Light of the Seven and The Night King.

NDG does an incredible job of translating the soundtrack audio to the piano. If you were a Game of Thrones fan, The Light of the Seven is from the opening sequence where the Sept of Baelor was blown up from the underground wildfire stores and The Night King is from the final fight sequence culminating in the sadly slightly anticlimactic way he’s killed (not a fan but the music is top-notch). While he has an incredible ability to condense an entire orchestra into a single piano arrangement, some textures and feelings are lost when you don’t have some strings or some choir “ahhhs” or some organ sounds for these pieces.

This desire was the other inspiration for Muse. I’m hoping to start this week working on notation-style playback for the project. Once I have that working, I’ll be starting a separate project that will use the library to attempt to allow me to create a track that I expect to hear from my midi keyboard, and an associated accompaniment score for one or more synthesized instruments. The algorithm of trying to recover from a mistake that I make while I playing and resume orchestration will be the challenge I’m looking forward to the most. If I can get it working reasonably well, the final goal will be to try to analyze audio to recognize audio playback instead of MIDI playback so that it can accompany on my acoustic piano – because let’s face it, there isn’t a keyboard out there that competes with a real piano.

ncog.link

I’m not sure where I’m most interested in working this week. Part of me wants to work on the automatic accompaniment. Still, the other part of me wants to dive back into Kludgine and tackle either vector drawing (for UI purposes) or tackle my idea for adding an Elm-like component system to the render engine.

In taking a break from Kludgine and working with yew again, it has made me think about how that approach could fit into Kludgine. One thing I didn’t like about my previous attempt at a UI layer was that it seemed like some of the things I would want to do in a UI framework I would also want to do if I was just writing a Sprite-based game. That’s what lead me to the conclusion of trying to unify my idea of Sprites and Components into one system.

The web app is ready to just run with my ideas of architecture. I have some reluctance to execute on my current ideas of architecture. I’m not sure I’ve fully reflected on that hesitance, but I do know some of the uncertainty stems from what responsibilities I’m expecting to be handled client-side versus server-side. Settling on a design of how Kludgine will work for UIs will help me shape this idea a bit more firmly in my mind.

Community

I have a few posts over at ecton.dev, but I don’t really like maintaining static websites. My current plan is to create a new category on here and just move my “blog” topics there. There are a lot of topics I am exploring as I come to my design of ncog that I find fascinating and want to share or hopefully discuss. The piano backstory is an example of a post that I think is better suited for the “extra section,” so to speak, not the one that I anticipate most people subscribing to.

I hope people are interested enough in me that they might dig through there, but my primary goal here is to find people interested in ncog. Despite that being my goal, I know that several people who have started following me are following me partially out of the enjoyment of seeing someone put themselves out there and attempt something big. It’s that latter half that I think the audience of the new category will be.

I’m trying to decide on some overall theme ideas for my Twitch stream, which will probably carry over into here and Discord in some fashion. I’m also still trying to figure out how much variety I want on my Twitch stream. Eventually, I will hopefully split my time between new feature development (rust programming) and working on games within my engine (a bit more casual game dev of an audience). Until I get to that stage, however, I’m trying to think of ways to add more casual game-dev content before then.

Wrap-up

I think I’ve written way more than I should have for this week’s update. I hope the headings allow you to focus on what areas you’re interested in most. I’d love to hear from you on Twitch, Twitter, Discord or here if you’re following along and have any questions or feedback.

Have a great week, everyone, and stay safe out there.