Processing AVPlayer’s audio with MTAudioProcessingTap

The MTAudioProcessingTap that was introduced with iOS 6 is incredibly powerful – it sits at a point in the audio chain of AVPlayer (or any of the other AVFoundation classes that utilize AVAssets) which greatly simplifies the amount of work required to roll your own signal processing/analysis in an app.

By attaching the simple tap object to a track of your asset, all of the existing powerful AVFoundation functionality is available, which lets the system handle demuxing of your media container, decoding of the audio as well as its rendering, handling audio interruptions and endpoint changes and so on – letting the app developer focus on what is important to them – working directly with PCM audio.

There are a bunch of applications for the audio processing tap, including:

• Displaying a graphical spectrum analyser for a song/movie playing
• Providing some user-controllable graphical EQ
• Implementing some preset filters (a base booster, karaoke mix removing the vocal track, etc)
• Dynamic range compression, limiting, etc
• Pitch shifting, time dilation, etc

The most important feature of all, however, is as follows: All of the above works on any local file AVAsset, including tracks from the user’s iPod library which applications cannot obtain a file handle to and thus we cannot use our own mp4 demuxer/aac decoders (even Apple’s own AudioFileOpenURL doesn’t work with a user’s iPod library, and use of this library would still require us to set up our own AudioQueue which is not rocket science, but again, more work that we’d like to avoid). This has the side-effect of allowing our tap code to be distributed as an add-on module for any code which already utilises AVFoundation for its media playback. Rather than asking an app-developer to rewrite their entire playback infrastructure to use lower-level Audio Queues or even Audio Units, they can add on our processing to their existing app with ease. Update: Ryan McGrath found a way to get the MTAudioProcessingTap to work with remote streams. He has a nice writeup here: Recording live audio streams on iOS.

When coupled with the AVAssetReader/AVAssetWriter group of classes, we don’t have to throw away the audio that was processed – it can be written back to disk if we were inclined to write some sort of DJ’ing app.

The code

We will begin with an AVFoundation base which just sets up an AVPlayer to play us a song that we have loaded into our application’s bundle.

NSURL *assetURL = [[NSBundle mainBundle] URLForResource:@"skyfall" withExtension:@"m4a"];
assert(assetURL);

// Create the AVAsset
AVAsset *asset = [AVAsset assetWithURL:assetURL];
assert(asset);

// Create the AVPlayerItem
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];
assert(playerItem);

assert([asset tracks]);
assert([[asset tracks] count]);

self.player = [AVPlayer playerWithPlayerItem:playerItem];
assert(self.player);

[self.player play];

Next, we pull out the audio track from the asset, and attach to its audioMix the inputParameters that we are going to create. The inputParameters are what contain the audio processing tap that will be used by the AVPlayer when it is ready to obtain audio for rendering in its internal Audio Queue. To create the tap, we provide it with a collection of callback functions – they are where all the heavy-lifting of audio is done, and allow us to prepare our system for processing as well as tear it down when iOS decides we’re not needed any more (e.g. when we’re done with the AVPlayer and it is released).

// Continuing on from where we created the AVAsset...
AVAssetTrack *audioTrack = [[asset tracks] objectAtIndex:0];
AVMutableAudioMixInputParameters *inputParams = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:audioTrack];

// Create a processing tap for the input parameters
MTAudioProcessingTapCallbacks callbacks;
callbacks.version = kMTAudioProcessingTapCallbacksVersion_0;
callbacks.clientInfo = (__bridge void *)(self);
callbacks.init = init;
callbacks.prepare = prepare;
callbacks.process = process;
callbacks.unprepare = unprepare;
callbacks.finalize = finalize;

MTAudioProcessingTapRef tap;
// The create function makes a copy of our callbacks struct
OSStatus err = MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks,
 kMTAudioProcessingTapCreationFlag_PostEffects, &tap);
if (err || !tap) {
    NSLog(@"Unable to create the Audio Processing Tap");
    return;
}
assert(tap);

// Assign the tap to the input parameters
inputParams.audioTapProcessor = tap;

// Create a new AVAudioMix and assign it to our AVPlayerItem
AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
audioMix.inputParameters = @[inputParams];
playerItem.audioMix = audioMix;

// And then we create the AVPlayer with the playerItem, and send it the play message...

None of this makes any sense without the callbacks for the processing tap. Here are the first four:

 void init(MTAudioProcessingTapRef tap, void *clientInfo, void **tapStorageOut)
{
    NSLog(@"Initialising the Audio Tap Processor");
    *tapStorageOut = clientInfo;
}

void finalize(MTAudioProcessingTapRef tap)
{
    NSLog(@"Finalizing the Audio Tap Processor");
}

void prepare(MTAudioProcessingTapRef tap, CMItemCount maxFrames, const AudioStreamBasicDescription *processingFormat)
{
    NSLog(@"Preparing the Audio Tap Processor");
}

void unprepare(MTAudioProcessingTapRef tap)
{
    NSLog(@"Unpreparing the Audio Tap Processor");
}

These functions give us a chance to prepare our processing before the real process() function is invoked.

The init() callback helps us by passing in the client data that we provided when initially setting up the collection of callbacks that are passed through when creating the tap. We can assign this into our tap storage, which allows us to easily obtain it again in any of the other callbacks with a call to MTAudioProcessingTapGetStorage(), passing in our tap. The client data can be used if we need some storage (say, if we’re implementing some simple time-domain FIR filter and we need to keep frames equal to the order of the filter from the end of the previous block, or vDSP’s setup data, or any user-defined parameter that could help us retain some state between blocks, etc). In our example, I just pass through a reference to the UIViewController subclass that contains all of this code. This is hardly an ideal design, but we’re just focusing on the minimum amount of code possible to get processing happening. Perhaps one could design a processing class or struct that keeps all of this state concisely packed together, and pass that in as the client data.

The prepare() callback is useful in that it will give us hints as to the maximum block size that it will feed us during each process callback. It also lets us know the format of the audio coming in (is it interleaved? floating or fixed point? what bit depth is it?). These parameters could be useful if we were going to be performing FFTs on the data, as Apple’s vDSP framework ensures good performance on its transforms by requiring the user to perform its memory allocation/setup functions ahead of time (and they require the block size). This kind of setup code is perfect for the prepare callback.

The process() callback is where the real work is done. In this case, our “processing” simply scales the samples down by some fraction defined by a slider in our view. This gives us a simple volume control. It is scaling linearly, which isn’t particularly intuitive/pleasing for the ears – a logarithmic volume slider would be better. We cheat and use the vDSP framework to multiply the signal efficiently.

#define LAKE_LEFT_CHANNEL (0)
#define LAKE_RIGHT_CHANNEL (1)

void process(MTAudioProcessingTapRef tap, CMItemCount numberFrames,
 MTAudioProcessingTapFlags flags, AudioBufferList *bufferListInOut,
 CMItemCount *numberFramesOut, MTAudioProcessingTapFlags *flagsOut)
{
    OSStatus err = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut,
                   flagsOut, NULL, numberFramesOut);
    if (err) NSLog(@"Error from GetSourceAudio: %ld", err);

    LAKEViewController *self = (__bridge LAKEViewController *) MTAudioProcessingTapGetStorage(tap);

    float scalar = self.slider.value;

    vDSP_vsmul(bufferListInOut->mBuffers[LAKE_RIGHT_CHANNEL].mData, 1, &scalar, bufferListInOut->mBuffers[LAKE_RIGHT_CHANNEL].mData, 1, bufferListInOut->mBuffers[LAKE_RIGHT_CHANNEL].mDataByteSize / sizeof(float));
    vDSP_vsmul(bufferListInOut->mBuffers[LAKE_LEFT_CHANNEL].mData, 1, &scalar, bufferListInOut->mBuffers[LAKE_LEFT_CHANNEL].mData, 1, bufferListInOut->mBuffers[LAKE_LEFT_CHANNEL].mDataByteSize / sizeof(float));
}

To grab audio to process, the processing tap provides the function MTAudioProcessingTapGetSourceAudio which we use to place the output audio into the bufferList that was passed in. We then do processing on this audio in-place. vDSP_vsmul is a function where, given a “vector” (in this case, the audio provided to us by the tap) it will skip along it one float at a time (hence the “1” we passed in) and multiply it by a scalar that we provide.

We utilise the processing tap’s storage to keep a reference to the view controller, where we pull out the value from the slider which we use when processing.

In another post I will show code implementing some simple filters using the MTAudioProcessingTap.