Building NotateAI: An AI-Powered Music Notation Assistant

The Problem with Music Notation Software

If you've ever written music on a computer before, you know how tedious it can be. You spend more time manually fixing typos and copy-pasting than actually letting your creative juices flow.

As a composer and arranger in my spare time, I've decided to take action into my own hands, and start developing NotateAI, the first AI-powered music notation software. The key selling feature is an AI assistant that handles the tedious work for you - so you can spend more time writing actual music. If you've ever used Cursor before, it's basically that but for writing music.

How Does It Work?

For the editor itself, I chose to create a fork of MuseScore, a pre-existing music editor. I chose it since it was open source, so I didn't have to re-create a lot of basic functionality. In fact, in my opinion, MuseScore is the best music editor out there.

In Musescore, all music files are stored as a special version of .xml files, which is just a plain-text file formatted in a way that's easy for machines to understand. That makes it very easy to feed the existing music data to an AI, and then have it understand what's currently there. With every request, the user can choose to send in the full .xml file (to save tokens when it's not needed).

This pipeline was surprisingly good at getting information across to the AI. It could easily return information like key signature and style, while also performing more detailed analysis. This part here would be very useful when someone is stuck and needing some inspiration, or checking to make sure that they're following a chord progression properly.

Reverse Engineering the MuseScore API

The next part is getting the AI to directly edit the music score. To accomplish that, I first tried to research the MuseScore API, which ended up being completely useless. The existing documentation was sparse and contained almost nothing useful. In fact, the MuseScore4 documentation just said "MuseScore 4 doesn't support plugins. Good luck".

I had to dig even further into the source code, and had to reverse engineer the API and create documentation myself (perhaps one day I will publish my findings and make a guide on creating MuseScore mods). After figuring out what functions did what, I described each one as well as created examples.

Then, in the beginning of each AI conversation, I included the documentation as well as some basic information telling the agent what it was. If the AI was called to do something, it would generate some code using this documentation and send it back to the client. An unexpected side effect I found very interesting is that the AI would find shortcuts to make the code more efficient, such as by defining variables and functions in the code to simplify repetitive music. The code would be displayed for the user, and then they could choose whether or not to execute the code, which would bring the AI's changes directly to the score.

Performance and Limitations

So, how good is it as an assistant? I tested it, and found it was better for ideation. Giving you boilerplate notes to start you off, or choosing a rhythm. However, my original purpose, replacing tedious tasks, seemed too hard for the AI to pull off.

In the future, one of my plans is to train an AI model specifically on music files, so it can understand them better. I am currently using Gemini, a generic LLM, just because they had free API credits I could use in development.

Try It Yourself

Right now, you can actually test out NotateAI for yourself. I've released the very first version, beta v0.1, which is the barebones agent. A lot of quality-of-life features are still in the works: version control, conversation management, and improved music generation.

Check it out at notateai.vercel.app!