MuseScore+Arduino+LEDs Tutorial

Week-end project - building a DIY piano learning & tutoring system.

This article explains how to turn your MIDI-capable keyboard or digital piano into a full piano learning & tutoring system, i.e., an interactive lighted keyboard/piano, based on (a modification of) the open-source MuseScore sheet music editor & player, a common Arduino board and a LED stripe. The system will let you practice a piano piece on the keyboard without any need for reading sheet music: just load any MIDI file on MuseScore, play it activating the Piano Tutor function, and then follow the LEDs lighting up on the stripe deployed along your keyboard, repeating as many times as needed.

You can see it in action in the video below:

Why - motivations & history

Real piano players are used to the sheet music notation, to the point that they can read and play it back in almost real-time, even seeing a non-trivial piece with chords written down for the first time. However, I've been playing piano basically as a hobby since high-school. At that time, I had incidentally at home an old Bontempi for kids, not even 61-keys in size, and without any MIDI capability. As I was already programming in Basic and Assembler on an Amiga500, and my brother was already liking to build custom electronics such as home-made audio sampling devices, a 1200 bps modem and can't remember what else, he ended up building a parallel port interface to that old Bontempi, which I happily used as input device for my own 100% 68000-Assembly code SoundTracker clone, so that I could expand in software the synthesizing capabilities of that little keyboard, with the full power of the 4 hardware audio channels in the fantastic stereo output of that old Amiga.

Later, during my university studies, I've got a Roland digital piano, which I still have nowadays, and with which I like to study a few easy pieces from time to time, typically in batches of intensive dedication of a few weeks, followed by months of total inactivity. As a late and occasional piano player, reading sheet music has never been easy nor straightforward, nor especially fast for me, and I've often got the impression (or illusion) that having a system to show the music straight on the keyboard would have been such a relief for my hobby sessions. To the point that, along with my brother, we had built this ugly stripe with simple red LEDs, shift registers chained one after another, and can't remember how many wires, all soldered on a piece of cardboard with the length of a full 88-keys piano keyboard (I wish I had a picture to show it to you, but I couldn't find it anywhere, however it looked a lot like this, just not so nice). I have to confess that the cabling killed that project, and we never finished it!

Last summer, I happened to have at reach an Arduino Uno board, as well as a Neopixel RGB LED stripe, and I decided to give a second shot to that old dream, perfectly aware that the toughest part would have been this time to modify an open-source music editor and player such as MuseScore, something I knew was very possible, due to some prior experience with its code a few years ago, when I realized a patch for automated repeats detection therein (check this video if you're curious). Indeed, while the Arduino side of the project you can find described below was done in a couple of hours, the MuseScore modifications for the very first working prototype required the rest of a full couple of days, ..., plus plenty of additional time later to get to the status of the patch as it is now, with a nice dockable dialog with customizable options, and reasonable serial communications with Arduino that doesn't disrupt the real-time performance of the MuseScore synthesizer during playback, and the MuseScore real-time playback and MIDI handling threads avoiding to tip-toe on each other while trying to uselessly talk to a deaf Arduino board, as being busy in the update of the RGB colors on 144 LEDs!

Today I'm aware many people love to learn & play through Synthesia or Piano Master (or similar app) on a tablet, and you can buy a Casio LK-280 lighted keyboard for a relatively affordable price, or equivalently a Yamaha EZ-220, or even an expensive digital piano with integrated a full lighted keyboard, a tablet and learning software. There's also the excellent ONE Piano Hi-Lite add-on for who has a real acoustic piano!

However, I've always been liking since my childhood, and I still like nowadays, to build my own stuff, no matter what, if time and resources allow, with software having evolved over years into adapting existing open-source software to your own needs, rather than writing it from scratch, which makes things just so faster. This takes surely part of your time, and sometimes it's even more expensive than something you can just go and buy elsewhere, but it still gives you the satisfaction to see something you did yourself, that works as you thought it would, that does exactly what you need and how you need it, and that you can always tweak and fine-tune tomorrow. Isn't this priceless ?

If you're still reading, and share my viewpoint, then I'm sure you'll have some fun in going on with this tutorial. Happy hacking then :-)!

What you need

• PC or Laptop with Linux
• Arduino Uno board
• NeoPixel 144 LEDs stripe
• MIDI-capable keyboard or digital piano, possibly along with a MIDI-to-USB converter

Step-by-step tutorial

Hardware set-up

Let's talk about the hardware first. The NeoPixel (or any compatible) individually addressable RGB LEDs stripe is really simple to connect and use: it has 3 wires on each side, the black and red ones are for GND and +5V, respectively, and the third wire (usually of a different color, like green or white), normally coming out from the middle of the stripe, is the data. Connections can be seen from this image on the Adafruit website:

An external power source is advisable in case you plan to turn on many LEDs at full brightness, as a 144 LEDs stripe can draw as much as 7A at 5V, for a total of ~35W. However, in this project we're going to see just a few LEDs up at the same time, and at a very minimal brightness level, so you can pick both the GND and the +5V directly from the corresponding Arduino PINs. Under these conditions, this works for me pretty well even without the capacitor nor the resistance (albeit the specs warn you about possible damages to the initial LEDs on the stripe):

On one side of the stripe there is the data IN, on the other side the data OUT. Which one is which is understood by the little arrows depicted on the stripe, showing the direction of the data as it flows from the data IN side, towards the data OUT side, as visible on the Adafruit page:

For this project, you need to connect the Arduino pin no.7 to the data IN wire of the LEDs stripe.

Note: in case you have two (or more) stripes, you can chain them with one another, just connecting the data OUT of the first stripe with the data IN of the second one, in addition to the GND and +5V wires.

The last and easiest step of the hardware-setup phase is to plug your MIDI capable keyboard or digital piano into your PC. If your device is already USB capable, then it is sufficient to plug it into any USB port of your laptop, using a USB cable. If, on the other hand, your keyboard/piano has a MIDI OUT connector, then you'll need to plug it into a MIDI-to-USB converter, using a MIDI cable, then your MIDI-to-USB converter into your PC, using a USB cable. This is my own case, and this is how this comes out:

Software

First, clone the piano-tutor branch of my modified MuseScore git repository from github, i.e., open a terminal and type:

1. git clone https://github.com/tomcucinotta/MuseScore.git
2. cd MuseScore
3. git checkout piano-tutor

Arduino set-up

1. Install the Arduino IDE on your PC/laptop, in case you don't have it already: sudo apt-get install arduino;
2. From the Arduino IDE: 2.a) install the Adafruit NeoPixel library, e.g., as explained here; 2.b) open within the above checked out git repository the .ino file: miditools/PianoTutor/PianoTutor.ino;
3. Plug in the USB cable from your Arduino board;
4. Ensure the IDE is correctly configured in terms of USB port/device (menu Tools->Port), as well as exact Arduino board model (menu Tools->Board);
5. Compile the program pushing the top-left Verify button;
6. Upload the program on Arduino pushing the Upload button.

The best thing at this point is to make a quick test of the Arduino-only part:

1. In the Arduino IDE, pick Tools->Serial Monitor (Ctrl+Shift+M);
2. Select 115200 as serial port speed;
3. You should see a message from Arduino like "Piano Tutor v0.x is ready!". If you can't see such a message, try closing the Serial Monitor and opening it back, double-check the USB cable and repeat the Upload step above.
4. Type this in the Serial Monitor, followed by Enter/Return: h10202020
5. You should see the 17th LED (10 hex is 16 decimal) lighting up with a dim white
6. Type this in the Serial Monitor: h10000000 [Enter]
7. You should see the 17th LED turning off
8. Close the Serial Monitor, as well as the Arduino IDE

Note: access to the Arduino serial device can be exclusively claimed by both the Arduino IDE Serial Monitor, and the modified MuseScore software being used in this project: ensure at least the Serial Monitor window is closed, so that it releases the serial device, before starting MuseScore, or the latter won't be able to access the serial device in what follows in this guide.

Laptop/PC set-up

Ensure you have all MuseScore build dependencies:

1. sudo apt-get build-dep musescore
2. sudo apt-get install cmake-qt-gui

Finally, compile and install the program with a custom install prefix and program suffix, in order to avoid confusion with your local version of the program, if you have it already installed:

1. make PREFIX=/usr/local/mscore-git SUFFIX=-git release
2. sudo make install
3. /usr/local/mscore-git/bin/mscore-git /path/to/file.mid

Sound set-up on Linux

Additional pieces of software that are very handy, are aconnect from alsa-tools and qjackctl, both of which allow you to check logical connections of MIDI devices within the Linux ALSA sound subsystem:

1. sudo apt-get install alsa-tools
2. sudo apt-get install qjackctl

Various sound set-up options are possible on Linux, where the general advice is to get rid of PulseAudio in the audio production pipeline, in order to experience an acceptable interactive playback latency. For the quickest and easiest set-up, it is sufficient that you:

• configure MuseScore using ALSA: Edit->Preferences...->I/O pane, and check "ALSA audio" with a period size of no more than 512-1024 samples; try increasing this value if audio skips, or decreasing it if you hear too much latency (see also the section "Real-time and glitch-free playback" below);

• enable MIDI input in MuseScore: Edit->Preferences...->Note Input pane, and check "Enable MIDI input";

• connect logically your MIDI-to-USB converter (or USB keyboard/piano) to the MuseScore MIDI input, visually using qjackctl (or from a terminal through aconnect): just launch qjackctl, do NOT launch jackd if asked, and/or stop jackd in case it is auto-started; push the Connect button to pop-up the connections dialog, go to the ALSA pane, and connect your MIDI/USB device (visible on the left) to the MuseScore input port (visible on the right).

Note: you can actually use jackd and take advantage of the full JACK low-latency audio infrastructure along with the plethora of client applications: just please refer to some other guide for how to set that up.

Tip: you can check whether MIDI input is correctly working by switching MuseScore to edit mode (type 'n'), then pressing any key on the piano/keyboard: you should see your notes being added to the current score (and being played as well).

Features at a glance

The modified MuseScore software can play the piece waiting for you to type all the keys in each sequence/chord (sensed through MIDI), before moving on to the next sequence/chord. Also, it can play the piece slower or faster using its "Tempo" playback function, and it can loop over a few measures as many times as needed using its "Loop" playback function, for repetitive practicing. Also, you can reduce the number of times you need to switch among the piano keyboard and the laptop by enabling the MuseScore "Remote Control" feature, allowing you to control various actions of the software via customizable key presses on your keyboard, e.g., you can configure a specific piano key to toggle play/pause, another one to rewind playback from the beginning, etc.

Using the MuseScore PianoTutor

Launch your just compiled MuseScore variant, open a MIDI file, then press 't' on the laptop keyboard, or mouse-click on the rightmost button on the top toolbar, to open the Piano Tutor panel:

The meaning of the widgets on this pane is as follows:

• Enable tutor: check this in order to enable all other Piano Tutor functions, first and foremost showing on the LEDs stripe all notes being played during a playback;
• Tutor waits for user: check this in order to enable the MIDI-based back channel, so that MuseScore can wait for you to press each keys' sequence/chord, before moving to the next one, during playback;
• Tutor lookahead: check this in order to let MuseScore show on the LEDs not only the next keys sequence/chord, but also the very next one, in a very dimmer light.

The last checkbox also allows you to press the next sequence ahead of time, compared to when MuseScore would play it according to the MIDI own timing, synchronizing with your speed of execution: this way, you can go faster on parts of the score you already remember, and slower on more difficult or newer parts, where MuseScore waits for you before moving on (second checkbox checked).

The LEDs stripe features section of the pane allows you to configure what exact LEDs stripe you have, and how you exactly laid it down along your piano keyboard. Defaults are OK for a 144 LEDs / 1 meter stripe, with the middle point of the stripe falling just between the B-3 and the C-4 keys, resulting in the C-4 key corresponding to LED no.71 for a right-to-left (reverse) deployed stripe, or to LED no.72 for a left-to-right deployment. In case you're lost with these calculations, just press the Wizard... button and follow the on-screen instructions. These will be filled automatically in by the software.

Finally, the two left-hand and right-hand colored buttons allow you to customize the LEDs colors to be used for the left-hand vs right-hand on the stripe. In a piano score, it is commonplace to find the right-hand as the first score, and the left-hand as the second one, on paper. In this patched MuseScore variant, you need to ensure that the first one is mapped to channel 1, whilst the second one to channel 2. Actually, anything mapped to an odd channel would be displayed on the LEDs stripe using the configured left-hand color, and anything mapped to an even channel would be displayed using the configured right-hand color.

With a classical two-staves score, after ensuring left-hand and right-hand are mapped to an odd and even channel, you can play and practice separately for either of the two hands, by muting the channel you'd like to ignore, from the Mixer MuseScore pane.

A typical session showing also separate hands practicing can be seen in the video below, where the graphical interface of the Piano Tutor was still at its early-stage (the two buttons visible in the video correspond roughly to the first two check-boxes of the Piano Tutor pane explained above):

Tip: Personally, I like to use the Piano Tutor in three major modes:

• slow playback mode, to have a quick glance at what keys will have to be pressed with what tempo, and how the part sounds: check only the first one out of the 3 checkboxes presented above, keep your PC/laptop volume up (and ensure you have a nice piano soundfont configured in the MuseScore Preferences), and let MuseScore play a segment of a piece slowing down the playback decreasing its Tempo from the MuseScore Play pane;
• full tutor mode, to practice and learn/memorize a piece: check all of the 3 checkboxes, but mute your PC/laptop volume while keeping your keyboard/piano volume up; this way, MuseScore will only light up LEDs without playing any sound, and when you press the keys you'll get the sound out of the keyboard/piano, not MuseScore;
• full tutor with soft-synth sound, if you want to practice a piece, but you also like to use a synthesizer with specific soundfonts on your PC/laptop instead of your keyboard/piano own sound: still check all of the 3 checkboxes, but this time mute only MuseScore own volume (from the Play pane), leaving your laptop/PC overall volume up; then, using qjackctl (or aconnect) mentioned above, connect your MIDI USB device to a different synthesizer program running on your PC/laptop (in addition to the MuseScore MIDI input, still required for letting MuseScore wait for & follow you during tutored playback); such an independent synthesizer could be, for example, QSynth (using underneath the same fluidsynth library/API as used by MuseScore) or Timidity++, both of which can easily be configured to use ALSA MIDI input.

Devil's in the details

The above should be enough for you to get started, but there is a number of additional details that is good to know about this project, to help with troubleshooting, or fine-tune its usability and experience.

About the keyboard and LEDs stripe type

The stripe I had at reach has 144 individually addressable RGB LEDs in 1m, and it fits sufficiently well (albeit non-perfectly) a standard digital piano keyboard, when exploiting 1 LED every 2 (i.e., the stripe has roughly twice the LEDs that would be needed). For LEDs at the edge, the 1-LED-every-2 rule tends to need correction over a long 88-keys keyboard, so in my dockable Piano Tutor panel there's a real number (which is 2.0 by default) that can be tuned exactly for this reason, and having more LEDs than strictly needed helps in this regard. This kind of stripe can be cut at any length, for a 61-keys keyboard I've cut it, but for my full 88-keys piano it cannot reach out to cover edge keys: I think for reasonable music pieces the stripe covers a sufficient region of the keyboard so that this is not really a problem (and I can easily shift the stripe left/right if needed, there's the "C-4 LED number" tunable in the Piano Tutor pane for that, and even a wizard button that makes all the auto-tuning interactively). Alternatively, this kind of stripes can be composed one after another, so it is possible to connect the three wires to the next segment, and obtain a prologued stripe that covers the full 88-keys span (at the 2/1 rate, it would need overall 176 LEDs).

About the Operating System

MuseScore is cross-platform, and it can be compiled on Linux, Mac OS-X and Windows. However, the modifications to the software in my github repository mentioned above are made specifically for Linux. With some more little effort, it should be straightforward to port the changes to Mac OS-X and Microsoft operating systems as well.

In detail, on Linux I'm accessing the serial device via the standard syscalls open(), close(), write(), read(), tcgetattr(), tcsetattr(), tcdrain(). This is all confined within mscore/tutor.cpp, so a few platform-specific changes to that file and you should be able to see this working on a different OS.

Contributions along these lines, after some testing on Mac OS-X and/or Windows, are very welcome of course!

About the Arduino serial device

The Arduino serial device PC/laptop side shows up as /dev/ttyACM0 or /dev/ttyACM1 in my case. Therefore, the modified MuseScore published in the above mentioned github repository tries these two devices, which are hard-coded in the software (mscore/tutor.cpp). In case your USB MIDI becomes visible with a different device name on Linux, just modify those names as needed. You can check what device shows up from the last few lines coming out of the command dmesg (typed in a terminal) right after having plugged your Arduino USB cable.

Real-time and glitch-free playback

Albeit not being a crucial part of this project, if you're going to use your PC/laptop as a soft-synthesizer for extended periods of time, it will be useful to fine-tune its OS behavior for a real-time, glitch-free audio processing pipeline with an acceptably low playback latency. Indeed, a reasonable real-time playback performance is achieved with an audio processing latency, i.e., the time between a key press on your MIDI device, and the instant you start hearing sound out of your PC/laptop speakers, not exceeding roughly 5ms-20ms. For example, at a sampling frequency of 48 kHz, a ~10ms latency corresponds to configuring audio buffers as small as 512 samples. However, doing so requires a tight schedule of the various processes involved in the sound production pipeline, increasing chances to experience glitches during the playback, due to other interfering user-space processes, kernel threads or interrupt drivers.

Linux has undergone over years a remarkable and relentless effort by several developers to improve its real-time performance, e.g., adding full preemptability, high-resolution nanosecond-precision timers, advanced fine-grain RCU locks, up to the PREEMPT RT rework of its kernel internals, and to an experimental deadline-based scheduler. However, tuning the kernel for low and predictable latency is traditionally detrimental to the average throughput of the platform, thus you may need to manually and explicitly enable a set of real-time features, to prevent glitches during sound playback, if using short buffers as needed for live playback. The following is a simple and non-exhaustive list of steps you may want to try out, in case you're experiencing glitches during sound playback:

• set the CPU frequency governor to maximum performance, e.g., using a convenient applet in your desktop or by typing in a terminal:

for c in $(grep processor /proc/cpuinfo | sed -e 's/.*: //'); do sudo cpufreq-set -c $c -g performance; done

• close any application you don't need during playback
• enable real-time priority and in-memory locking for processes/threads involved in the audio processing pipeline, e.g., as explained in the JACK real-time howto (and you may want to disable page swapping to disk altogether)
• switch to a low-latency kernel, e.g., with something like

sudo apt-get install linux-image-lowlatency

(and a reboot with the new kernel)

• switch to a PREEMPT_RT kernel, e.g., as explained in this howto (normally requires to manually reconfigure, recompile, install a custom kernel and reboot)