The Challenges of Deploying a Generative AI Singing Synthesis Model on iOS using Onnx and Unity

Do you love singing along to your favorite tunes?
Do you wish you could customise them with your own lyrics? Yeah, me too.

I’m detailing my attempt to create a unique singing experience with reactive dynamic musical singing voices. I will run through building a proof of concept app that can generate singing content and the outcomes from investigating the current ecosystem of deploying pytorch models, along with integrating it inside Unity to create an iOS app. I'll summarise the challenges and pitfalls I faced along the way, such as optimizing the model performance, running the inference on a background thread, and attempting to make use of CoreML technology. This post can help anyone interested in deploying a generative AI model into a client side production environment.

I have always been fascinated by the power of music and sound to create immersive and emotional experiences. As a developer and occasional musician, I wanted to explore the possibilities of using a generative AI model to personalise audio content on mobile platforms.

Specifically, I wanted to create a reactive audio soundtrack that could be customized with user-specific singing content. To do this I would need to create a generative ai model to run on mobile. To test out the current state of the art models I planned to create a working proof of concept combining a singing synthesis model running inside a Unity app on iOS.

Text to Singing Synthesis vs Text to Speech vs Voice Cloning

For the casual reader; text to singing synthesis uses similar techniques as text to speech synthesis but includes musical and emotional analysis and also takes in some musical input and then attempts to generate vocals with timbre, prosody, intonation and realistic vibrato. This is harder to achieve in polysyllabic languages like English, than in languages with words that are mono or dual syllabic, like Japanese or Chinese.

You may have also heard about the “Drake” voice cloning, a voice to voice method which takes the audio data of one singer and modifies the timbre, tone, and style of the voice to match a different target singer, that is a very different process, here I'm attempting to generate singing from just text (words and note) inputs.

A summary of the recent work on this can be found here Awesome Singing Voice Synthesis and Singing Voice Conversion.

Training an English singing model

My starting point was an open source project by Deutsche Telekom MMS which culminated in an AI Opera last year.

A lot of the underlying code comes from projects from the Far East and aren’t great with polysyllabic languages, the source vocal singing data needs all the syllables annotating and breath aspirations marked up, I found that the quality of annotation had a huge impact (no surprise really!).

I made multiple revisions, removing certain audio samples which skewed the output to being too shouty or powerful, and also attempted to remove slurring issues, to prevent words merging into each other too much in the generated output this had to be tackled with adjustments in training and inference.

Deployment Hurdles

To deploy the trained model onto iOS, I wrote exporters to export the pytorch model to the onnx format. This looked like a sensible solution, onnx models can run on most platforms, so should allow the solution to be platform agnostic. From the onnx documentation it shows support for running onnx models on iOS and suggest it can make use of Apple’s CoreML technology

Unity’s C# support is .NET 41, and required a lot of reworking of the onnx runtime wrapper. I attempted to use the published nuget packages but after a litany of errors, I moved over to using the main source code from Microsoft. It needed a few compatibility updates to run in Unity’s iOS build system and started working after resolving the various compiler error messages.

Next I rewrote the original python inference code to work with the C# onnx runtime, this allowed the singing model to run anywhere that Unity could target (iOS, Android, Mac and PC) which had a matching Onnx plugin.

Great, now I could begin testing on the iOS platform. I built the Unity app and deployed it onto my iPhone 11. Ran the inference. It locked up the UI while running the inference (which took around 2 minutes) but singing did get generated.
Break out the champagne. Oh, not quite yet.

Lockups and threading

Next I needed to prevent the system lockups while running inference. Unity is not really set up for this type of thing, its very single core centric (although burst and jobs is helping move away from this) I implemented UniTask to run the inference on a background thread, which worked well in terms of allowing the user interface and Unity systems to continue unaffected while the models were running.

I assumed there would be some configuration to divert the model to run using CoreML, a leap too far unfortunately. Only models with compatible “nodes” can take advantage of CoreML. Running a profiler and logging out the inference code, my models had 580 nodes of which none are compatible with CoreML.

The least I thought I could do was to optimise the model to float 16, simple to do in the onnx ecosystem (they provide essentially a “one click” optimiser). The model ran but then the output was un-parsable, the float 16 (half type in C#) is not a native method in Unity’s .NET4, it was added in .NET5.

To get the outcome I was hoping for would require re-authoring the training code to only use operations that are CoreML compatible, or create a CoreML specific exporter for the pytorch model.

Both are possible solutions, and are good learning outcomes, the documentation for implementation certainly assumes a level of knowledge above which I had while scoping out the project at the beginning. There are not many resources on deploying generative models in Unity and C#, hopefully this is a useful overview, if you’re interested in any of the fine details of the implementation do reach out to me.

It does feel like we're in a (another!) mild format war, this one is very organic, we have the ubiquitous torch for training and then a multitude of deployment options; PyScript, TensorFlow Lite, Onnx, CoreML, NNAPI and a few others.

The takeaway from this is to be hyper aware of your final deployment target; which platform (mobile / desktop ) and language your technology (C++, Java, Node etc..) is planning to use as the answers need to be fed back right to the start directing modifications to your model training code.

I did attempt to publish a live copy of the app but Apple rejected it with the following report;

Guideline 4.0 - Design
We noticed an issue in your app that contributes to a lower-quality user experience than App Store users expect:
- Portions of your app, or resources needed to use your app, loaded, refreshed, ran or responded very slowly. For example, when we we tapped on “Generate, it took more than 5 minutes to load.
Since App Store users expect apps to be simple, refined, and easy to use, we want to call your attention to this design issue so you can make the appropriate changes.

My Singing Prototype is available on TestFlight, be warned you’re in for a wait while the audio is generated but it is working and is just a first stage in a much longer roadmap, enjoy the Jazz.

Get Jazz Flow on TestFlight