Simple Multicore Core to Core Communication Using FreeRTOS Message Buffers - FreeRTOS (2024)

by Richard Barry on 18 Feb 2020

[The STM32H745I demo in the FreeRTOSdownload provides a worked example of the control buffer scheme described below.]

In this post I describe how to implement a basic and light weight core to core communicationscheme using FreeRTOS Message Buffers,which are lockless circular buffers thatcan pass data packets of varying sizes from a single sender to a single receiver.Message buffers just provide the transport for the data - they do not impose anyformatting or higher level protocol to which the data must conform.

In the use case described below the sending and receiving tasks are on differentcores of a multicore microcontroller (MCU) in an Asymmetric Multi-Processor (AMP)configuration - which means each core runs its own instance of FreeRTOS. The onlyhardware requirements (other than there being more than one core) are the abilityfor one core to generate an interrupt in the other core, and for there to be anarea of memory that is accessible to both cores (shared memory). The message buffersare placed in the shared memory at an address known to the application runningon each core. See figure #1.Ideally there will also be a memory protection unit (MPU) to ensure the Message Buffer can onlybe accessed through the kernel's MessageBuffer API, and preferably mark the shared memory as non cacheable.

Figure 1: Hardware topology. Click to enlarge.

The following two pseudo code listings show the structure of the API functionsused to send to and receive from a message buffer. It can be seen that, in bothcases, the calling task can optionally enter the blocked state (so not consumingany CPU cycles) to wait until the operation can complete.

xMessageBufferSend(){ /* If a time out is specified and there isn't enough space in the message buffer to send the data, then enter the blocked state to wait for more space. */ if( time out != 0 ) { while( there is insufficient space in the buffer && not timed out waiting ) { Enter the blocked state to wait for space in the buffer } } if( there is enough space in the buffer ) { write data to buffer sbSEND_COMPLETED() }}

xMessageBufferReceive(){ /* If a time out is specified and the buffer doesn't contain any data that can be read, then enter the blocked state to wait for the buffer to contain data. */ if( time out != 0 ) { while( there is no data in the buffer && not timed out waiting ) { Enter the blocked state to wait for data } } if( there is data in the buffer ) { read data from buffer sbRECEIVE_COMPLETED() }}

If a task entered the blocked state in xMessageBufferReceive() to wait for the bufferto contain data then sending data to the buffer must unblock the task so it cancomplete its operation. The task gets unblocked when xMessageBufferSend() callssbSEND_COMPLETED(), which is a preprocessor macro.

The default sbSEND_COMPLETED implementation assumes the sending task(or interrupt) and the receiving task are under the control of the sameinstance of the FreeRTOS kernel and run on the same MCU core. In this AMPexample the sending task and the receiving task are under the control of twodifferent instances of the FreeRTOS kernel, and run on different MCU cores,so the default sbSEND_COMPLETED implementation won't work (each FreeRTOSkernel instance only knows about the tasks under its control). AMPscenarios therefore require the sbSEND_COMPLETED macro (and potentially thesbRECEIVE_COMPLETED macro, see below) to be overridden, which is done bysimply providing your own implementation in the project's FreeRTOSConfig.hfile. The re-implemented sbSEND_COMPLETED() macro can simply trigger an interruptin the other MCU core. The interrupt's handler (the ISR that was triggered byone core but executed in another core) must then do the job that would otherwisebe done by the default implementation of sbSEND_COMPLETE - namely unblock a taskif the task was waiting to receive data from the message buffer that now contains data.The ISR unblocks the task by passing the message buffer's handle as a parameterto the xMessageBufferSendCompletedFromISR() function. This sequence is shown bythe numbered arrows in figure 2, where the sending and receiving tasks are on different MCUcores:

1. The receiving task attempts to read from an empty message buffer andenters the blocked state to wait for data to arrive.
2. The sending task writes data to the message buffer.
3. sbSEND_COMPLETED() triggers an interrupt in the core on which thereceiving task is executing.
4. The interrupt service routine calls xMessageBufferSendCompletedFromISR()to unblock the receiving task, which can now read from the buffer as thebuffer is no longer empty.

Figure 2: The numbered arrows correspond to the numbered list above, which describes the transfer of one data item through the message buffer. Click to enlarge.

It is easy to pass the handle of the message buffer into xMessageBufferSendCompletedFromISR()when there is only one message buffer, but consider the case where there are twoor more message buffers - then the ISR must first determine which of the messagebuffers contains data. There are several ways this can be done if the number ofmessage buffers is small. For example:

• If the hardware allows then each message buffer can use a different interrupt line,which keeps the one to one mapping between the interrupt service routine and themessage buffer.
• The interrupt service routine could simply query each message buffer tosee if it contains data.
• Multiple message buffers could be replaced by a single message buffer thatpasses both metadata (what the message is, what its intended recipient is,etc.) as well as the actual data.

However these techniques are inefficient if there are a large or unknown number ofmessage buffers - in which cases a scalable solution is to introduce a separate controlmessage buffer. As demonstrated by the code below, sbSEND_COMPLETED() uses thecontrol message buffer to pass the handle of the message buffer that contains datainto the interrupt service routine.

/* Added to FreeRTOSConfig.h to override the default implementation. */#define sbSEND_COMPLETED( pxStreamBuffer ) vGenerateCoreToCoreInterrupt( pxStreamBuffer )/* Implemented in a C file. */void vGenerateCoreToCoreInterrupt( MessageBufferHandle_t xUpdatedBuffer ){size_t BytesWritten; /* Called by the implementation of sbSEND_COMPLETED() in FreeRTOSConfig.h. If this function was called because data was written to any message buffer other than the control message buffer then write the handle of the message buffer that contains data to the control message buffer, then raise an interrupt in the other core. If this function was called because data was written to the control message buffer then do nothing. */ if( xUpdatedBuffer != xControlMessageBuffer ) { BytesWritten = xMessageBufferSend( xControlMessageBuffer, &xUpdatedBuffer, sizeof( xUpdatedBuffer ), 0 ); /* If the bytes could not be written then the control message buffer is too small! */ configASSERT( BytesWritten == sizeof( xUpdatedBuffer ); /* Generate interrupt in the other core (pseudocode). */ GenerateInterrupt(); }}

The ISR then reads the control message buffer to obtain the handle, then passesthe handle as a parameter into xMessageBufferSendCompletedFromISR(). See thecode listing below.

void InterruptServiceRoutine( void ){MessageBufferHandle_t xUpdatedMessageBuffer;BaseType_t xHigherPriorityTaskWoken = pdFALSE; /* Receive the handle of the message buffer that contains data from the control message buffer. Ensure to drain the buffer before returning. */ while( xMessageBufferReceiveFromISR( xControlMessageBuffer, &xUpdatedMessageBuffer, sizeof( xUpdatedMessageBuffer ), &xHigherPriorityTaskWoken ) == sizeof( xUpdatedMessageBuffer ) ) { /* Call the API function that sends a notification to any task that is blocked on the xUpdatedMessageBuffer message buffer waiting for data to arrive. */ xMessageBufferSendCompletedFromISR( xUpdatedMessageBuffer, &xHigherPriorityTaskWoken ); } /* Normal FreeRTOS "yield from interrupt" semantics, where xHigherPriorityTaskWoken is initialised to pdFALSE and will then get set to pdTRUE if the interrupt unblocks a task that has a priority above that of the currently executing task. */ portYIELD_FROM_ISR( xHigherPriorityTaskWoken );} 

Figure 3 shows the sequence when a control message buffer is used. Again thenumbered items related to the numbered arrows in the diagram:

1. The receiving task attempts to read from an empty message buffer andenters the blocked state to wait for data to arrive.
2. The sending task writes data to the message buffer.
3. sbSEND_COMPLETED() sends the handle of the message buffer that nowcontains data to the control message buffer.
4. sbSEND_COMPLETED() triggers an interrupt in the core on which thereceiving task is executing.
5. The interrupt service routine reads the handle of the message buffer thatcontains data from the control message buffer, then passes the handleinto the xMessageBufferSendCompletedFromISR() API function to unblockthe receiving task, which can now read from the buffer as the buffer isno longer empty.

Figure 3: The numbered arrows correspond to the numbered list above,which describes the transfer of one data item through one of manymessage buffers using a control message buffer to allow the ISRto know which message buffer contains data. Click to enlarge.

So far we have only considered the cases where the sending task must unblock thereceiving task. If it is possible for a message buffer used for core to corecommunication to get full, causing the sending task to block, then it is alsonecessary to consider how the receiving task unblocks the sending task. That canbe done by overriding the default implementation of the sbRECEIVE_COMPLETED() inexactly the same way as already described for sbSEND_COMPLETED().

In all cases it is good defensive programming practice to ensure a task never blocks indefinitely on amessage queue, in case an interrupt is missed, and always drains a message queuecompletely, rather than assuming there is one message per interrupt.