Audio signal processing effects (FX) are used to manipulate sound characteristics across a variety of media. Many FX, however, can be difficult or tedious to use, particularly for novice users. In our work, we aim to simplify how audio FX are used by training a machine to use FX directly and perform automatic audio production tasks. By using familiar and existing tools for processing and suggesting control parameters, we can create a unique paradigm that blends the power of AI with human creative control to empower creators. For a quick demonstration, please see our demo video:
To combine deep learning and audio plugins together, we have developed a new method to incorporate third-party, audio signal processing effects (FX) plugins as layers within deep neural networks. We then use a deep encoder to analyze sounds and learn to control audio FX that themselves performs signal manipulation. To train our network with non-differentiable FX layers, we compute FX layer gradients via a fast, parallel stochastic approximation scheme within a standard automatic differentiation graph, enabling efficient end-to-end backpropagation for deep learning training.
For technical details of the work, please see our ArXiv Paper: https://arxiv.org/abs/2105.04752.
“Differentiable Signal Processing with Black-Box Audio Effects.” Marco A. Martínez Ramírez, Oliver Wang, Paris Smaragdis, and Nicholas J. Bryan. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2021.
@inproceedings{martinez2021deepafx,
title={Differentiable Signal Processing with Black-Box Audio Effects},
author={Mart'{i}nez Ram'{i}rez, Marco A. and Wang, Oliver and Smaragdis, Paris and Bryan, Nicholas J.},
booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
month={June},
year={2021},
publisher={IEEE}
}
All code and models are available at https://github.com/adobe-research/DeepAFx.
The video is avaliable here and the poster is available here.
All listening test examples are available here.