Audio Pipeline

The audio pipeline is the main abstraction. It puts into relation audio endpoints and audio processing stages.

Figure 47: Audio Pipeline

As you can see in the above picture, an audio pipeline has an audio producing endpoint, an arbitrary number of processing stages and an audio consuming endpoint. In most cases, if an audio device records and playbacks audio, it will need two audio pipelines. In some specific use cases, an audio pipeline can have more than one consumer and more than one producer.

Audio Endpoint

An audio endpoint is something, either hardware or software, that deals with audio samples. It can either produce them or consume them. For example, an audio endpoint could be:

• A hardware audio codec (e.g., MAX98091)

• A network stack (e.g., SPARK Wireless Core)

• An USB stack with the USB-Audio class

• A dummy audio codec which generates a pre-recorded sine wave

• An I2S interface used to create an audio pipe

To be considered a valid audio endpoint, one must adhere to a specific endpoint interface. The Audio Core includes two endpoints that a user can use for his audio pipelines. The user can also create his own endpoints to suit his application needs.

Here is the list of what makes an endpoint:

• An instance specific to the endpoint. This instance enables the creation of multiple endpoints of the same type, useful when the same endpoint is used for producing samples in a pipeline, and consuming samples in another. The instance can also contain configuration parameters.

• A name to describe the endpoint.

• An endpoint interface which is the standard set of functions the endpoint must implement.

An endpoint is either used as a producer or as a consumer, but never both. If an audio device uses a single hardware codec to playback and record audio, for example, two endpoints need to be instantiated for each direction.

An endpoint has memory allocated to it to queue up audio packets. A free queue contains nodes to be used. The main queue will be enqueued nodes either after the produce action is finished or after the pipeline processing. Multiple nodes can share the same free queue. This allows a node to be enqueued on multiple endpoints and the memory space to be shared.

The endpoint interface is a set of three functions that each endpoint must implement to be compatible with the Audio Core. These functions are:

• Action: Used by the Audio Core when it needs the endpoint to produce audio samples (e.g., recording of samples for an audio codec) if the endpoint is a producer, or when it needs the endpoint to consume audio samples (e.g., playback of samples for an audio codec) if the endpoint is a consumer.

• Start: Used by the Audio Core to start the production or consumption of audio samples of the endpoint. For some endpoints (e.g., SPARK Wireless Core), this can do nothing.

• Stop: Used by the Audio Core to stop the production or consumption of audio samples of the endpoint. For some endpoints (e.g., SPARK Wireless Core), this can do nothing.

Audio Processing Stage

An audio processing stage represents operations that will be applied on audio samples after they are created but before they are consumed. Any number of processing stages can be chained together. The output of one processing stage is the input of the next one, which means that the order used to add processing stages to a pipeline matters. Each processing stage has requirements for their inputs. Some will expect to process raw audio samples, some could expect the audio to have gone through an encoder and others could expect to process non-audio information. For example, when receiving compressed audio, the decompression processing stage needs to be added first, then the digital volume control.

Here is a list of included processing stage that be used with the SDK:

• Clock Drift Compensation
• Audio Compression
• Audio Fallback
• Mute on Underflow
• Mute Packet
• Audio Packing
• Sampling Rate Converter
• Digital Volume Control
• Audio Sample Accumulator

To be considered a valid audio processing stage, one must adhere to a specific audio processing interface. The Audio Core includes three processing stages that a user can add to his audio pipelines. The user can also create his own processing stages to suit his application needs.

Here is the list of what makes a processing stage:

• An instance specific to the processing stage. This instance enables the creation of multiple processing stages of the same type.

• A name to describes the processing stage.

• A processing stage interface which is the standard set of functions the processing stage has to implement.

The processing stage interface is a set of four functions that each processing stage must implement to be compatible with the Audio Core. These functions are:

• Init: Used by the Audio Core to initialize the processing stage. This can be left empty if no specific initialization steps are needed.

• Ctrl: Used by the application to interact with the processing stage. This can be used to get information from (e.g., get statistics) or modify its behavior (e.g., raise the volume).

• Process: Used by the Audio Core to modify audio samples.

• Gate: Used by the Audio Core to know if the processing stage should be run or skipped. If the gate function returns true, the processing stage will execute. If the gate function returns false, the processing stage will be skipped. If the gate function is set to NULL, the processing stage will always execute. The gate function is mostly used when a processing stage needs to be activated/deactivated on the fly. A common use case is Fallback mode.

When the Audio Core processes a pipeline, it will cycle through every registered processing stage and call their Process function (only if their Gate function returns true or if it is set to NULL), specifying the size of the chunk and passing the samples through data_in. Once the processing is done, the Process function will return the processed samples through data_out and return the number of bytes processed through its return value. If Process has not done anything on the samples, it will return 0. The Audio Core will then know that there is nothing valid from data_out. In this case, data_in will be passed to the next stage instead of data_out.

The audio header is passed to the Process function since it may contain necessary information that the processing stage needs.

Audio Header

The Audio Core encapsulates the audio samples with a header before they are consumed by the Wireless Core endpoint. Its fields are:

Field	Length	Description
TX queue level high	1 bit	Used by the CDC mechanism. This bit is set when the TX audio buffer load reaches a certain level indicating that the wireless link is bad.
Fallback	3 bit	This indicates to the receiving audio device that the audio payload is using a audio fallback mode. The payload should be decoded according to the fallback configuration.
CRC4	4 bits	This is the 4-bit CRC used to validate the integrity of the audio header.
Payload size	8 bits	The size, in bytes, of the audio samples carried in this audio packet. This size does not take into account the audio header.

Audio Mixer

The Audio Mixer module merges two or more audio streams into a single stream using an audio mixing algorithm. To enable audio mixing, at least three Audio Pipelines using Audio Mixer Endpoints are required.

Audio Mixer Pipeline

The Audio Mixer Pipeline is different from a regular Audio Pipeline as it is split in two stages. The first stage is the Input Mixer Pipeline, and the second stage is the Output Mixer Pipeline.

The number of Input Mixer Pipelines matches the number of audio input streams that require mixing and only one Output Mixer Pipeline is present.

An example of an Audio Mixer Pipeline with two audio stream inputs is shown below:

Figure 50: Audio Mixer Pipeline

The Input Mixer Pipeline Producer Endpoint generates an audio stream, either from the SPARK Wireless Core or an audio codec like any regular audio pipeline. Audio packets are then sent into the processing stages before being stored in the Mixer Consumer Endpoint’s queue.

The Output Mixer Pipeline Producer Endpoint fetches the packets from the two endpoints that precede it. The packets are mixed and sent to the next processing stage.

Mixer Consumer Endpoint

This endpoint is part of the Mixer Input Stage and acts as a buffer; it does not have any “consume” action. Its queue is linked to the Mixer Producer Endpoint in the Output Mixer Pipeline. The length of the queue determines the audio latency.

Mixer Producer Endpoint

This endpoint is instantiated once per input stream (Input Mixer Pipeline). Its queue is linked to the respective consumer queue from the previous stage. This type of endpoint does not have any “produce” action.

Mixer Process

When a Mixer Producer Endpoint is used, a Mixer Process is performed on the audio samples before sending to other downstream processes. The Mixer Process will call the Audio Mixer Module to mix the audio samples.

Audio Mixer Module

The Audio Mixer Module is used by the application to configure audio mixing. It can be configured with the following parameters:

• Number of inputs: This parameter is the number of inputs to be mixed. The currently supported number of inputs is 2 and 3.

• Payload size: This parameter configures the audio payload size that must match the Output Mixer Pipeline’s consuming endpoint.

• Bit depth: This parameter corresponds to the bit depth of the samples in the payload.

The configuration parameters are applied to the Audio Mixer Module upon initialization. The instance also includes the following parameters:

• Input Samples Queue: This queue stores the incoming samples. There are as many Input Sample Queues as there are audio inputs.

• Output Packet Buffer: This buffer stores the mixed samples.

Audio Mixer Module Algorithm

The Audio Mixer Module Algorithm is used by the Audio Mixer Module to mix the audio packets together. The process involves summing all the channel input values and then dividing the result by the number of input channels. To prevent clipping, this operation is done with 32-bit registers. After the division, the output result is converted back to 16 bits.

The following diagram illustrates the mixing algorithm:

Figure 51: Audio mixing algorithm