Real-Time Inference of Neural Networks: A Guide for DSP Engineers – Valentin Ackva & Fares Schulz

https://audio.dev/ -- @audiodevcon​

Real-time Inference of Neural Networks: A Practical Approach for DSP Engineers - Valentin Ackva & Fares Schulz - ADC 2023

In upcoming audio processing innovations the intersection of neural networks and real-time environments is set to play a decisive role. Our recent experience of implementing neural timbre transfer technology in a real-time setting has presented us with diverse challenges. Overcoming them has provided us with significant insights into the practicalities of inferencing neural networks inside an audio plugin.

This talk presents a pragmatic approach: Starting with a trained model, we guide you through the necessary steps for inferencing the model in a real-time environment. On our way we delve into the critical aspect of maintaining real-time safety, share proven strategies to ensure a seamless and uninterrupted signal flow. Moreover, we address the delicate balance between latency, performance, and stability. For this we utilize three different inference engines: libtorch, tensorflow-lite and onnxruntime. While the in-house solutions for the popular machine learning frameworks PyTorch and TensorFlow, seem obvious choices, sometimes other engines may be better suited for certain use cases. By contrasting the characteristics of the engines, we hope to simplify your decision-making process.
_

Valentin Ackva

I am an audio programmer and electronic musician based in Berlin. With a background in computer science, I'm currently working towards my master's degree in Audio Communication and Technology at the Technische Universität Berlin. My passion lies at the intersection of music, programming, and technology, especially where artistry meets innovation. For the last 3 years, I have been working as an audio software developer at a speech processing startup in Leipzig. At my position there, I am responsible for the development of audio effects for speech enhancement. This role includes research into the real-time implementation of state-of-the-art neural networks for tasks such as denoising, audio super-resolution, and dereverberation. This year, I have co-founded a collective that combines the fields of DSP and AI, bringing together a group of audio programmers, machine learning engineers, and artists based in Berlin. In March, we released our first software, "Scyclone", an audio plugin that utilizes neural timbre transfer technology, introducing a new approach to automatic layering. Scyclone's innovative design and interaction of DSP and AI led to it winning the Audio Plugin Competition organised by the Audio Programmer.
_

Fares Schulz

Hello! I am a student assistant at the Electronic Studio of Technische Universität Berlin, currently pursuing a master's degree in Audio Communication and Technology. My educational background includes two bachelor's degrees in physics and audio engineering. During this time, my passion for audio software gradually led me from theoretical mathematical equations and abstract artistic concepts to their development as DSP algorithms in Python and their implementation as real-time audio applications in C++. Currently, I am particularly interested in exploring novel applications of neural networks for digital signal processing. Together with like-minded people, I recently developed the open source project Scyclone, which won the Neural Audio Plugin Competition organized by Audio Programmer. In addition to my academic and open source endeavors, I am actively involved in the development of noise reduction algorithms in Python and C++ for medical devices at Miethke.
_

Streamed & Edited by Digital Medium Ltd: https://online.digital-medium.co.uk
_

Organized and produced by JUCE: https://juce.com/
_

Special thanks to the ADC23 Team:

Sophie Carus
Derek Heimlich
Andrew Kirk
Bobby Lombardi
Tom Poole
Ralph Richbourg
Jim Roper
Jonathan Roper
Prashant Mishra

#adc #audiodev #dsp #audio