Andrew Mercer
on this page

The problem

An MTU (Maximum Transmission Unit) mismatch along a path causes packets above the smallest MTU on that path to either fragment (if allowed) or get silently dropped (if the Don't Fragment bit is set and something in between can't fragment, e.g. certain tunnel encapsulations) — symptoms are usually "small transfers work fine, large ones hang or reset," which is a classic sign to check MTU before anything else.

Binary-searching the actual path MTU

Use ping with the Don't Fragment bit set (-M do) and vary the payload size — remember to account for the 28 bytes of IP+ICMP header overhead (a -s 1472 payload plus 28 bytes header = 1500 total, the standard Ethernet MTU):

ping -M do -s 1472 -c 1 host        # standard Ethernet MTU (1500 total)
ping -M do -s 8972 -c 1 host        # jumbo frame test (9000 MTU total)

A failing size returns:

ping: local error: Message too long, mtu=1500

Working example: a controller-to-controller link where 8972 failed but 8952 succeeded (some encapsulation layer eating 20 bytes of the jumbo budget):

for host in ctrl4 ctrl5 ctrl6; do
  echo "${host}"
  ping6 -c1 -M do -s 8952 "${host}"
  echo ""
done

Setting the MTU on an interface

sudo ip link set dev eth0 mtu 9000       # runtime change

Persist via Netplan (see the Linux router guide for the full Netplan syntax) or your distro's interface config.

Checklist when large transfers hang but small ones work

  1. ping -M do -s <size> host — binary search for the actual path MTU (see above)
  2. Check both ends and anything in between agree on jumbo frame support — a single hop stuck at 1500 breaks the whole path even if source and destination are both configured for 9000
  3. For tunnels (VPN, overlay networks) — subtract the encapsulation overhead from your target MTU (a common miss: forgetting the outer IP+UDP+VXLAN header when sizing a VXLAN underlay MTU)
  4. Check for a firewall dropping ICMP "fragmentation needed" (type 3, code 4) messages — this breaks Path MTU Discovery silently, causing exactly the "large transfers hang" symptom above with no obvious cause. Verify: iptables -L -v -n | grep icmp shouldn't show it blocked.

Further reading