Imagine a world where musicians no longer have to lug around heavy tomes of sheet music or struggle with bulky tablets. This dream is becoming a reality thanks to innovative technology that integrates sheet music directly into smart glasses. In a recent project spearheaded by a passionate engineer and musician, the concept of augmented reality for musicians has taken a giant leap forward. This article dives into the fascinating journey of creating smart glasses capable of displaying sheet music seamlessly, enhancing the performance experience for musicians everywhere.
The vision for this project has been years in the making. The creator, who has an engineering background, expressed their long-standing desire to combine engineering with music in a unique way. The main driving force behind this innovation is portability. Traditional sheet music can be cumbersome, making it impractical for musicians on the go. Although tablets and phones provide some solutions, they often lead to uncomfortable playing positions, detracting from the performance experience.
The primary goal was to design a pair of glasses that could hold sheet music, acting as both eyewear and a display device. This would allow musicians to perform without the audience realizing they were reading music—the impression would be that they had it all memorized, perhaps due to poor eyesight.
The Journey Begins
The journey to create these smart glasses began with the technological challenge of making them functional and user-friendly. With an engineering degree in hand, the creator was determined to utilize modern tools like ChatGPT to bring this concept to life. However, they faced a significant hurdle: investing in expensive hardware without a clear plan of making it work.
To overcome this, the stars aligned when a hackathon was announced, promising participants smart glasses if they could build a compelling application. This event took place in the Bay Area, where the creator lived, making it the perfect opportunity to kickstart the project. After a whirlwind of events, including dropping out of MIT and quitting a job, the creator secured funding from Y Combinator and was ready to showcase their vision at the hackathon.
Building the System
Upon arriving at the hackathon, the creator received third-party glasses running a custom operating system called Augment OS, developed by Menra, the startup hosting the event. The glasses connected via Bluetooth to a smartphone app that communicated with Menra’s servers. This setup allowed the laptops to receive input from the glasses’ microphone and output text or bitmap images to the glasses.
The main technical challenge was to bridge the gap between sheet music and bitmap images. The creator utilized the MusicXML format, which describes the structure of sheet music in a way similar to how HTML describes web pages. This format provided rich information and flexibility, crucial for rendering high-quality music visuals.
The Rendering Process
To convert the MusicXML into images, the creator employed the Music21 Python library. The process involved:
- Rendering chunks of music into images.
- Downscaling these images and converting them into bitmaps using the Pillow library.
- Optimizing the bitmap file sizes with ImageMagick.
Despite the initial excitement, the creator faced numerous bugs during the conversion process, highlighting the often frustrating nature of software engineering. After a few hours of meticulous work, the first bar of sheet music appeared in the glasses, drawing positive reactions from fellow participants.
Tackling Challenges
On the second day of the hackathon, another problem arose: the downscaling of sheet music caused the stems of notes to disappear. The creator needed to find a way to enhance these stems without enlarging the entire image. This led to a crucial learning moment involving an image processing technique called dilation.
Dilation allows for the resizing of certain elements in an image by using a kernel—a small area that slides across the original image. By adjusting the kernel to emphasize only vertical lines, the creator was able to bold the stems effectively, ensuring they remained visible in the final display.
Innovation in Action
With the technical challenges addressed, the creator and their teammate worked collaboratively, producing several applications in just two days. The hackathon culminated in a presentation session where participants showcased their projects. As the judges deliberated, the anticipation grew, with the creator nervously awaiting the results.
When the results were announced, the team was ecstatic to learn that they had won first place. This victory not only provided them with the smart glasses but also validated their hard work and creativity. The experience solidified the notion that innovation is possible, even in a short timeframe.
Looking to the Future
While the current version of the smart glasses is just the beginning, the creator is excited about the future of this technology. They acknowledge that this is the worst version of the product they will ever produce, with ongoing efforts by Menra and other tech companies to enhance smart glasses in terms of reliability, speed, resolution, and latency.
As the creator reflects on this journey, they invite others to consider what technology they wish existed and how they might take steps to create it. In today’s world, where access to AI and global communication is unprecedented, there is no need to sit idly by while waiting for advancements; one can actively contribute to the future of technology.
Conclusion
The creation of smart glasses that display sheet music is a remarkable intersection of music and technology, promising to transform the way musicians perform. This project showcases the potential of innovation, driven by passion and determination. As technology continues to evolve, the possibilities for enhancing the musical experience seem limitless. The future is indeed bright, and for musicians, it may soon be clearer than ever.
Credit: Linguine on YouTube
