Mixed Adapters Sharing a Broadcast Domain

Onload User Guide (UG1586)

Document ID
Release Date
1.2 English

Onload should not be used when Solarflare and non-Solarflare interfaces in the same network server are configured in the same broadcast domain, as depicted by the following figure.

Note: A Broadcast domain can be a local network segment or VLAN.
Figure 1. Mixed Adapters Sharing a Broadcast Domain

When an originating server (S1) sends an ARP request to a remote server (S2) having more than one interface within the same broadcast domain, ARP responses from S2 will be generated from all interfaces and it is non-deterministic which response the originator uses. When Onload detects this situation, it prompts a message identifying 'duplicate claim of ip address' to appear in the (S1) host syslog as a warning of potential problems.

Problem 1

Traffic from S1 to S2 can be delivered through either of the interfaces on S2, irrespective of the IP address used. This means that if one interface is accelerated by Onload and the other is not, you might or might not get acceleration.

To resolve the situation (for the current session) issue the following command:

echo 1 >/proc/sys/net/ipv4/conf/all/arp_ignore

or to resolve it permanently add the following line to the /etc/sysctl.conf file:

net.ipv4.conf.all.arp_ignore = 1

and run the sysctl command for this be effective.

sysctl -p

These commands ensure that an interface will only respond to an ARP request when the IP address matches its own. Refer to the Linux documentation Linux/Documentation/networking/ip-sysctl.txt for further details.

Problem 2

A more serious problem arises if one interface on S2 carries Onload accelerated TCP connections and another interface on the same host and same broadcast domain is non-Solarflare:

A TCP packet received on the non-Solarflare interface can result in accelerated TCP connections being reset by the kernel stack and therefore appear to the application as if TCP connections are being dropped/terminated at random.

To prevent this situation the Solarflare and non-Solarflare interfaces should not be configured in the same broadcast domain. The solution described for Problem 1 above can reduce the frequency of Problem 2, but does not eliminate it.

TCP packets can be directed to the wrong interface because:

  • the originator S1 needs to refresh its ARP table for the destination IP address - so sends an ARP request and subsequently directs TCP packets to the non-Solarflare interface
  • a switch within the broadcast domain broadcasts the TCP packets to all interfaces.