Audio Core

Introduction

The SPARK Audio Core is the piece of software used by all SPARK-based audio applications. Its role is to manage audio samples flow from their recording to their playback, including audio processing and buffering. The Audio Core will mostly be used in conjunction with the SPARK Wireless Core, but it has been built in such a way that it is not dependant on it. A few abstractions like audio pipelines, audio endpoints and audio processing stages makes it flexible and decoupled from other libraries and hardware.

Most of the Audio Core features are demonstrated using application examples found in the SPARK SDK. Those are the best place to start learning about how to use the Audio Core while reading through this documentation. Sections in the documentation have been ordered from a top-down perspective, which means it starts by speaking about application requirements, then the Audio API and finally the Audio Core itself.

End-User Application

The end-user’s application must take care of a few things before the Audio Core can be operational. First, it must initialize and configure the endpoints it wishes to use. In the case of endpoints that rely on hardware, such as a hardware audio codec, the application would need to:

• Initialize MCU’s peripherals that interface with it like the I2S and I2C controller

• Configure it to have, for example, the proper sampling rate and bias voltage

• Manage its interrupt service requests or the ones of the MCU’s peripherals

In most cases, the application will also use the SPARK Wireless Core as an endpoint for sending or receiving audio samples over the air. If so, the Wireless Core will need to be configured by the application using the Wireless Core API.

Once the endpoints dependencies are taken care of, the application will need to make sure it can make available a chunk of RAM big enough for the Audio Core to initialize properly. The Audio Core will then be able to dynamically allocate memory from this pool to create the audio buffers amongst other things. The chunk of RAM is usually just a statically allocated bytes array.

Finally, Audio Core objects can be created and configured. Those are the endpoints themselves, the processing stages and the pipelines.

A few audio application examples are available in the SPARK SDK that the user can use as a starting point.

Audio Core Components

The Audio Core is composed of several interacting components that will be described in this section.

Audio Pipeline

The audio pipeline is the main abstraction. It puts into relation audio endpoints and audio processing stages.

Figure 28: Audio pipeline diagram.

As you can see in the above picture, an audio pipeline has an audio producing endpoint, an arbitrary number of processing stages and an audio consuming endpoint. In most cases, if an audio device records and playbacks audio, it will need two audio pipelines. In some specific use cases, an audio pipeline can have more than one consumer and more than one producer.

Audio Endpoint

An audio endpoint is something, either hardware or software, that deals with audio samples. It can either produce them or consume them. For example, an audio endpoint could be:

• A hardware audio codec (e.g., MAX98091)

• A network stack (e.g., SPARK Wireless Core)

• An USB stack with the USB-Audio class

• A dummy audio codec which generates a pre-recorded sin wave

• An I2S interface used to create an audio pipe

To be considered a valid audio endpoint, one must adhere to a specific endpoint interface. The Audio Core includes two endpoints that a user can use for his audio pipelines. The user can also create his own endpoints to suit his application needs.

Here is the list of what makes an endpoint:

• An instance specific to the endpoint. This instance enables the creation of multiple endpoints of the same type, useful when the same endpoint is used for producing samples in a pipeline, and consuming samples in another. The instance can also contain configuration parameters.

• A name to describe the endpoint.

• An endpoint interface which is the standard set of functions the endpoint has to implement.

The endpoint is either used as a producer or as a consumer, not both. If an audio device uses a single hardware codec to playback and record audio, for example, two endpoints need to be instantiated.

The endpoint interface is a set of three functions that each endpoint must implement in order to be compatible with the Audio Core. These functions are:

• Action: Used by the Audio Core when it needs the endpoint to produce audio samples (e.g., recording of samples for an audio codec) if the endpoint is a producer, or when it needs the endpoint to consume audio samples (e.g., playback of samples for an audio codec) if the endpoint is a consumer.

• Start: Used by the Audio Core to start the production or consumption of audio samples of the endpoint. For some endpoints (e.g., SPARK Wireless Core), this can do nothing.

• Stop: Used by the Audio Core to stop the production or consumption of audio samples of the endpoint. For some endpoints (e.g., SPARK Wireless Core), this can do nothing.

Audio Processing Stage

An audio processing stage represents operations that will be applied on audio samples after they are created but before they are consumed. Any number of processing stages can be chained together. The output of a processing stage is the input of the next one, which means that the order used to add processing stages to a pipeline matters. For example, when receiving compressed audio, the decompression processing stage needs to be added first, then the digital volume control.

To be considered a valid audio processing stage, one must adhere to a specific audio processing interface. The Audio Core includes three processing stages that a user can add to his audio pipelines. The user can also create his own processing stages to suit his application needs.

Here is the list of what makes a processing stage:

• An instance specific to the processing stage. This instance enables the creation of multiple processing stages of the same type.

• A name to describes the processing stage.

• An processing stage interface which is the standard set of functions the processing stage has to implement.

The processing stage interface is a set of four functions that each processing stage must implement in order to be compatible with the Audio Core. These functions are:

• Init: Used by the Audio Core to initialize the processing stage. This can be left empty if no specific initialization steps are needed before it is being used.

• De-init: Used by the Audio Core to deinitialize the processing stage. This can be left empty if no specific de-initialization steps are needed.

• Ctrl: Used by the application to interact with the processing stage. This can be used to get information from (e.g., get statistics) or modify its behavior (e.g., raise the volume).

• Process: Used by the Audio Core to modify audio samples. This is called on every audio packet produced in the pipeline the processing stage is attached to.

• Gate: Used by the Audio Core to know if the processing stage should be run or skipped. If the gate function returns true, the processing stage will be run. If the gate function returns false, the processing stage will be skipped. If the gate function is set to NULL, the processing stage will always be run. The gate function is mostly used when a processing stage needs to be activated/deactivated on the fly. A common use case is when using Fallback mode.

When the Audio Core processes a pipeline, it will cycle through every registered processing stage and call their Process function (only if their Gate function returns true or if it is set to NULL), passing the samples that need to be processed through data_in as well as the size of the chunk. Once the processing is done, the Process function will return the processed samples through data_out and return the number of bytes processed through its return value. If Process has not done anything on the samples, it will return 0. The Audio Core will then know that there is nothing valid from data_out. In this case, data_in will be passed to the next stage instead of data_out.

The audio header is passed to the Process function since it may contain necessary information that the processing stage needs.

Audio Compression

This processing stage can compress and decompress audio samples using Adaptive differential pulse-code modulation (ADPCM). ADPCM was initially developed for speech coding, but often offers acceptable performance for music coding. It can be used to save a significant amount of bandwidth as it shrinks 16-bit samples to 4-bit ones.

In order to use it, a compression instance must be created and added to an audio recording pipeline. On the playback audio device, a decompression instance must be created and added to the audio playback pipeline. This processing stage works for mono and stereo streams, but it must be specified when configuring the instance.

Digital Volume Control

This processing stage will digitally modify the amplitude of the audio samples. Each time the control command is called to increase or decrease the volume, the amplitude will vary gradually until it reaches the target. The maximum amplitude corresponds to the source level, which means this processing stage cannot increase it beyond that. This also means that there is no processing involved when the volume level is set to 100%.

User Data

This processing stage enables a user to add 1 byte of trailing data to an audio payload. This is an efficient way to have a small data channel without creating a dedicated Wireless Core connection.

To use it, a TX instance of this processing stage must be created and added to the pipeline that has the Wireless Core as a consuming endpoint. Then, whenever the control command to send a byte is called, the audio header bit User data is valid is set and the byte is appended to the audio payload. On the receiving audio device, an RX instance of this processing stage must be created and added to the pipeline that has the Wireless Core as a producing endpoint. Then, whenever an audio packet is received with the audio header bit User data is valid is set, the processing stage will call the callback function passed by the user to retrieve the byte. The callback function is not called when the actual audio payload is played back, but when it is received. Hence, the RX audio buffer latency won’t affect this data.

Sampling Rate Converter

This processing stage can modify audio samples in such way that they will look as if they were initially sampled at a different rate. This is useful when an audio device must handle audio streams with different sampling rates as most hardware audio codecs support a single sampling rate for their record and playback path. A common use case is a gaming headset, where the main audio stream could be sampled at 48 kHz and the microphone stream at 16 kHz.

Note

The current implementation only supports sampling rate conversions on mono audio streams. It is also using the CMSIS DSP Software Library for its interpolation and decimation functions making it incompatible with non-Arm Cortex-M processor based devices.

Audio Header

The Audio Core encapsulates the audio samples with a header before they are consumed by endpoints with encapsulation enabled such as the Wireless Core. Its fields are:

Field	Length	Description
TX queue level high	1 bit	Used by the CDC mechanism. This bit is set when the TX audio buffer load reaches a certain level indicating that the wireless link is bad.
User data is valid	1 bit	This indicates to the receiving audio device that the last byte of the audio packet is a valid data byte that can be retrieved and passed to the application.
Fallback	1 bit	This indicates to the receiving audio device that the audio payload has been modified on the fly (e.g. compressed when it was originally uncompressed).
Reserved	1 bits	Reserved for future use.
CRC4	4 bits	This is the 4-bit CRC used to validate the integrity of the audio header.
Payload size	8 bits	The size, in bytes, of the audio samples carried in this audio packet. This size does not take into account the audio header nor the user data.

Audio Module

Clock Drift Compensation

Clock Drift Compensation (CDC) is necessary because the audio clocks from the record and playback audio devices will eventually drift. A relevant example is when streaming audio through a SPARK RF link. One board will record the samples and the other one will play them back. The audio clock is asynchronous between both boards. This means that even if both audio master clocks are theoretically the same and a sampling rate of 48 kHz is configured, the effective sampling rate on both boards will be slightly different due to crystal tolerance. This could lead to a device having a sampling rate of 47.980 kHz while the other has 48.030 kHz for example. The clock drift needs to be compensated or some audio glitches will happen.

The CDC mechanism used in the Audio Core makes use of a SPARK-made resampling library. The strategy used is that as soon as a drift is detected through the RX audio buffer monitoring, the module will smoothly start resampling the configured amount of samples (e.g., 1440 samples). At the end of the resampling period, a new sample will have been created or dropped with minimal impact on the audio quality. A sample is dropped to compensate for an audio clock on the playback device that is too slow compared to the recording device, and a sample is added to compensate for an audio clock on the playback device that is too fast compared to the recording device. CDC must be enabled only on the playback device and not on the recording device. Also, if the bit TX queue level high is set in the audio header, the playback device won’t start compensating the clock drift since its RX audio buffer load variation is due to bad RF link conditions and not clock drifting.