Jun 01, 2020 - Pitfalls and Traps in Linux Multicast

Pitfalls and Traps in Linux Multicast

IPv4 multicast seems like a wonderful idea: you send one packet, your network processes it and sends a copy of it to each host that needs it. Over the years, wiser network engineers than myself have learned that multicast on a routed network can be difficult to configure and may not be worthwhile, but multicast has found adoption on local networks as an alternative to broadcast, and can be efficient if the majority of network switches support IGMP snooping. There are, however, many pitfalls on the path to using multicast on a local network.

If you want to use multicast on a Linux host with multiple network interface, in addition to the usual pitfalls and traps, there are several well-hidden tar pits where you can get stuck for days or weeks.

Background on IGMP and Multicast Group Membership

IGMP and Membership

The IGMP protcol is used by hosts to inform the routers and switches on their local network that they wish to receive multicast traffic. End devices send IGMP membership advertisement, and routers and IGMP-aware switches listen for these IGMP membership advertisements and ensure that multicast traffic is forwarded to those end devices. The IGMP protocol is defined by RFC-1112 (IGMP v1), RFC-2236 (IGMP v2) and RFC-3376 (IGMP v3).

IPv4 Multicast to Ethernet Layer 2 address mapping

To allow switches to handle IPv4 multicast traffic with minimal knowledge of IPv4, each IPv4 multicast address is mapped to an Ethernet MAC address by Section 6.4 of RFC-1112. The lower 23 bits of the IPv4 multicast address are placed in the lower 23 bits of the 01:00:5E:00:00:00 MAC address to create the Ethernet multicast address. (This results in multiple IPv4 multicast addresses mapping to the same Ethernet address).

Multicast vs IGMP

When working with multicast, it is important to recognize that IPv4 multicast, Ethernet multicast, and IGMP are related but separate protocols. IGMP is used to dynamically define and manage multicast groups on a network, but the actual packet forwarding is handled by the network switches based on the packet’s Ethernet multicast address, and by routers based on the packet’s IPv4 multicast address.

In many switches, you may only be able to inspect IPv4 or Ethernet multicast address, and it is important to know how they are related in order to debug forwarding issues.

Some switches may also be configured with static multicast group memberships based on the IPv4 or Ethernet address of the group. The implementation of this is left entirely to the network switch vendor and varies significantly between switches.

IPv4 Multicast Address Allocation

IPv4 multicast addresses are allocated by the IANA. Generally, these addresses are allocated to protocols that operate on local networks and which need unique identifiers.

Of special note is the 239.0.0.0/24 range, which is allocated for use within an organization and which should not be routed across the internet. This range is defined by RFC-2365. multicast address allocations.

IGMP snooping switches without an IGMP Querier

IGMP snooping is a feature that can be enabled on many higher-end switches. The switch listens for IGMP group membership advertisements, and for each incoming group membership, it adds that port to the advertised multicast groups, so that any multicast packets to those groups will be sent to that port. This prevents multicast traffic from being delivered to ports that aren’t interested in it. This saves bandwidth, and in some cases can prevent small devices from being overwhelmed by large amounts of multicast traffic.

Switches with IGMP snooping also implement a group membership timeout: if the IGMP membership report hasn’t be received recently, then the group memberships are removed from that port. This is necessary for proper handling of devices that are attached through multiple switches, but there’s a trap here: devices only send multicast membership reports when they join a group, or when they receive an IGMP membership query.

The network device that sends IGMP membership queries is a IGMP Querier. All of the devices that I’ve seen so far either don’t include an IGMP Querier, or have it turned off by default. This results in the very fun “multicast works for a few minutes and then stops” issue. Of course, this happens because the multicast group memberships time out.

Of course, now that you know all of this, the fix is simple: set up an IGMP Querier! All of the IGMP-aware switches that I’ve seen include one, and turning it on is usually simple. The only configuration parameter for it is a querier address, which leads to the next issue…

Choosing a poor IGMP Querier Address

Some posts on the network tell you that you can choose any address for your IGMP querier, and some even suggest choosing 1.0.0.0, 2.0.0.0, etc so that it is easy to control the priority of IGMP querier election. They are WRONG.

On multi-homed Linux machines (and probably other operating systems too), the OS will drop any packets which originate from an implausible IP address. If your Linux machine has multiple interfaces, and receives a packet from, say, 1.0.0.0 on an interface which only has local connectivity, Linux correctly deduces that this packet could not have possibly come from that network, and drops it. Since the IGMP querier is sending packets from the querier address, this will cause Linux to drop the IGMP queries instead of responding to them.

Example Multi-homed network

Here is an example network that exhibits this issue. The Linux Laptop is attached to two networks, one wifi and the other wired. Its internet connection and default route go through wifi so its default route is 192.168.1.1 (the wifi router).

Since we have not added any additional routes to the Linux Laptop’s routing table, it knows that the only traffic that can come from the wired network should be from the 192.168.2.0/24 address range, or multicast traffic.

If the switch’s IGMP querier is configured with an IP address such as 1.0.0.0, when the Linux Laptop receives that traffic on its wired network port, it will drop it because it is impossible.

Again, the fix here is pretty simple: allocate an IP address to your switch, and set the IGMP to the switch’s IP address, or allocate a dedicated IP address on the same subnet that is explicitly for use by the IGMP querier.

In the example above, the switch should be allocated an IP address from the 192.168.2.0/24 range; perhaps 192.168.2.3, since this address is not currently used.

Sniffing Multicast traffic

Since switches with IGMP snooping only send multicast data to the ports that are joined to those groups, common packet capture tools such as tcpdump and wireshark will not capture any multicast traffic if they device where they are run is not joined to the multicast groups of interest.

The best solution that I have found for this is to either write custom python tools to join the relevant multicast groups and capture the traffic, or to configure the switch to forward all multicast traffic to the port in question; most switches call this “router mode.” isn’t sending you any packets!

Inbound Multicast on Multi-homed Hosts

When joining a multicast group with setsockopt IP_ADD_MEMBERSHIP, Linux allows you to specify a local IP address to bind to, or if you specify INADDR_ANY, it will choose an address for you. If you choose INADDR_ANY, Linux chooses “an appropriate interface”! On a computer with a single network interface this is fine, but if you have more than one interface, there’s a chance that INADDR_ANY will choose the wrong interface.

The fix here is a bit painful: you have to write your software to detect all of the configured IP addresses and either ask the user to pick one, or join the multicast group on all of them. Luckily, you can join multiple multicast groups on the same socket, so this isn’t too hard to manage.

You can do this in python with the following code (using psutil to detect interfaces):

import socket
import psutil

# Multicast address and group
MCAST_GROUP = "239.0.0.1"
MCAST_PORT = 2020

# Create an IPv4 UDP socket.
listen = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

# Set REUSEADDR so that we can also bind transmit sockets on the same port.
listen.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR)

# Bind to our multicast port.
listen.bind((socket.INADDR_ANY, MCAST_PORT))

# For each interface.
for interface, addrs in psutil.net_if_addrs().items():
    # For each address on the interface.
    for addr in addrs:
        # If the address is an IPv4 address and not localhost.
        if addr.family == socket.AF_INET and not addr.address.startswith("127."):
            # Join our multicast group on this address.
            listen.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP,
                socket.inet_aton(MCAST_GROUP) + socket.inet_aton(addr.address))

# Receive data from our socket!
data = listen.recvfrom(1500)

Sadly, outbound multicast behaves differently.

Outbound Multicast on Multi-homed Hosts

Linux chooses the outbound interface for a multicast packet, not based on the the interface that has the multicast subscription(s), but instead based on the system’s default route. This means that even if you’ve joined a multicast group and explicitly specified an interface, Linux will ignore that interface and send your multicast packets out the default route.

It also doesn’t matter if you have multiple memberships on the same socket, Linux still only sends one packet.

Linux provides setsockopt IP_MULTICAST_IF to specify the IP address of the outbound interface for a socket, but this option can only have one option on a socket, so if you want to send multicast packets to many interfaces, you have to open a separate socket for each interface!