Mailbox - 3.3 English

Versal ACAP CPM DMA and Bridge Mode for PCI Express Product Guide (PG347)

Document ID
PG347
Release Date
2022-11-02
Version
3.3 English

In a virtualized environment, the driver attached to a PF has enough privilege to program and access QDMA registers. For all the lesser privileged functions, certain PFs and all VFs must communicate with privileged drivers using the mailbox mechanism. The communication API must be defined by the driver. The QDMA IP does not define it.

Each function (both PF and VF) has an inbox and an outbox that can fit a message size of 128B. A VF accesses its own mailbox, and a PF accesses its own mailbox and all the functions (PF or VF) associated with that PF.

Note: pcie_qdma_mailbox IP supports up to 4PFs and 240VFs. You can build a mailbox system in the PL to support more number of PFs and VFs (CPM5 limit is 16PFs and 2KVFs). Adding pcie_qdma_mailbox IP increases PL utilization.

The QDMA mailbox allows the following access:

  • From a VF to the associated PF.
  • From a PF to any VF belonging to its own virtual function group (VFG).
  • From a PF (typically a driver that does not have access to QDMA registers) to another PF.
Figure 1. Mailbox

VF To PF Messaging

A VF is allowed to post one message to a target PF mailbox until the target function (PF) accepts it. Before posting the message the source function should make sure its o_msg_status is cleared, then the VF can write the message to its Outgoing Message Registers. After finishing message writing, the VF driver sends msg_send command through write 0x1 at the control/status register (CSR) address 0x5004. The mailbox hardware then informs the PF driver by asserting i_msg_status field.

The function driver should enable the periodic polling of the i_msg_status to check the availability of incoming messages. At a PF side, i_msg_status = 0x1 indicates one or more message is pending for the PF driver to pick up. The cur_src_fn in the Mailbox Status Register gives the function ID of the first pending message. The PF driver should then set the Mailbox Target Function Register to the source function ID of the first pending message. Then access to a PF’s Incoming Message Registers is indirectly, which means the mailbox hardware will always return the corresponding message bytes sent by the Target function. Upon finishing the message reading, the PF driver should also send msg_rcv command through write 0x2 at the CSR address. The hardware will deassert the o_msg_status at the source function side. The following figure illustrates the messaging flow from a VF to a PF at both the source and destination sides.

Figure 2. VF to PF Messaging Flow

PF To VF Messaging

The messaging flow from a PF to the VFs that belong to its VFG is slightly different than the VF to PF flow because:

A PF can send messages to multiple destination functions, therefore, it may receives multiple acknowledgments at the moment when checking the status. As illustrated in the following figure, a PF driver must set Mailbox Target Function Register to the destination function ID before doing any message operation; for example, checking the incoming message status, write message, or send the command. At the VF side (receiving side), whenever a VF driver get the i_msg_status = 0x1, the VF driver should read its Incoming Message Registers to pick up the message. Depending on the application, the VF driver can send the msg_rcv immediately after reading the message or after the corresponding message being processed.

To avoid one-by-one polling of the status of outgoing messages, the mailbox hardware provides a set of Acknowledge Status Registers (ASR) for each PF. Upon the mailbox receiving the msg_rcv command from a VF, it deasserts the o_msg_status field of the source PF and it also sets the corresponding bit in the Acknowledge Status Registers. For a given VF with function ID <N>, acknowledge status is at:

  • Acknowledge Status Register address: <N> / 32 + <0x22420 Register Address>
  • Acknowledge Status bit location: <N> / 32

The mailbox hardware asserts the ack_status filed in the Status Register (0x22400) when there is any bit was asserted in the Acknowledge Status Register (ASR). The PF driver can poll the ack_status before actually reading out the Acknowledge status registers. The PF driver may detect multiple completions through one register access. After being processed, the PF driver should also write the value back to the same register address to clear the status.

Figure 3. PF to VF Messaging Flow

Mailbox Interrupts

The mailbox module supports interrupt as the alternative event notification mechanism. Each mailbox has an Interrupt Control Register (at the offset 0x22410 for a PF, or at the offset 0x5010 for a VF). Set 1 to this register to enable the interrupt. Once the interrupt is enabled, the mailbox will send the interrupt to the QDMA given there is any pending event for the mailbox to process, namely, any incoming message pending or any acknowledgment for the outgoing messages. Configure the interrupt vector through the Function Interrupt Vector Register (0x22408 for a FP, or 0x5008 for a VF) according to the driver configuration.

Enabling the interrupt does not change the event logging mechanism, which means the user must check the pending events through reading the Function Status Registers. The first step to respond to an interrupt request is disabling the interrupt. It is possible that the actual number of the pending events is more than the number of the events at the moment when the mailbox is sent the interrupt.

The mailbox will check its event status at the time the interrupt control change from disabled to enabled. If there is any new events that arrived the mailbox between reading the interrupt status and the re-enabling the interrupt, the mailbox will generate a new interrupt request immediately.