Fixing Asymmetric Routing Issues in a Homelab

Discovering the Asymmetric Routing Problem

This week I wanted to improve the network monitoring strategy of my homelab. Instead of checking DHCP leases, I wanted to monitor actual devices connected to the network. To do so, I installed WatchYourLAN .

WatchYourLAN is a nice project that uses arp-scan to check for new devices. Since I have five VLANs, I needed the server where WatchYourLAN would run to have an interface on each of those VLANs to run this app.

After successfully setting up WatchYourLAN I started to have an issue. The laptop I was using to configure it started dropping connections after a few minutes.

I knew it had something to do with the VLANs because the server was in the 10.0.101.x network, and the laptop on the 10.30.101.x network. So, at first, I thought it was OPNSense’s fault, I checked the rules and everything seemed fine.

The next suspect was the Ubuntu server’s SSH server, but everything looked normal there too.

After some research online, I concluded this was being caused by Asymmetric Routing.

Asymmetric Routing occurs when two nodes establish communication and one way of the communication uses a different path than the return trip.

It was evident when I used traceroute in both machines. First on the client checking the route to the server:

❯ traceroute 10.0.101.100
traceroute to 10.0.101.100 (10.0.101.100), 64 hops max, 52 byte packets
 1  10.30.101.1 (10.30.101.1)  13.731 ms  6.764 ms  6.735 ms
 2  10.0.101.100 (10.0.101.100)  4.485 ms  3.546 ms  3.335 ms

We can note that in this case, the communication requires a jump via the 10.30.101.1 gateway. Now let’s inspect the other side.

jorge@inteleon:~$ traceroute 10.30.101.114
traceroute to 10.30.101.114 (10.30.101.114), 30 hops max, 60 byte packets
 1  10.30.101.114 (10.30.101.114)  8.928 ms  8.689 ms  8.571 ms

In this case, since the server also has an interface on the 10.30.101.0/24 network, the connection is direct. This is how I confirmed what the cause of the problem was.

Fixing the Asymmetric Routing Issue

To fix the problem, I approached it from two perspectives. The SSH server and the overall networking aspect.

I made sure the SSH server only listened on one interface to remove any ambiguity.

...

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
ListenAddress 10.0.101.100
#ListenAddress ::

After setting the IP, it’s just a matter of restarting the service

jorge@inteleon:~$ sudo systemctl restart ssh
[sudo] password for jorge:

After making sure SSH only listens in one interface, I needed to make sure that the server always returns via the 10.0.101.0/24 interface. I saved the file in /etc/init.d/set_ssh_routing.sh.

Here’s the script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

#!/bin/bash

# Wait for a few seconds to ensure all network services are fully up
sleep 10

# Variables
IP_RULE="from 10.0.101.100 table SSH"
IP_ROUTE="default via 10.0.101.1 dev ens18 table SSH"
TABLE_NAME="SSH"
TABLE_NUMBER="200"

# Echo current rt_tables for debugging
cat /etc/iproute2/rt_tables | grep SSH

# Check if the routing table exists and create it if it doesn't
ensure_table_exists() {
    if ! grep -q "$TABLE_NUMBER $TABLE_NAME" /etc/iproute2/rt_tables; then
        echo "$TABLE_NUMBER $TABLE_NAME" >> /etc/iproute2/rt_tables
        ip route flush cache
        echo "Table $TABLE_NAME added to rt_tables."
    else
        echo "Table $TABLE_NAME already exists."
    fi
}

# Functions to add rule and route
add_ip_rule() {
    echo "Adding IP rule..."
    ip rule add $IP_RULE
}

add_ip_route() {
    echo "Adding IP route..."
    ip route add $IP_ROUTE
}

# Main execution function
main() {
    ensure_table_exists

    # Check and add IP rule if it doesn't exist
    if ! ip rule show | grep -q "$IP_RULE"; then
        add_ip_rule
    else
        echo "Rule already exists."
    fi

    # Check and add IP route if it doesn't exist
    if ! ip route show table SSH | grep -q "$IP_ROUTE"; then
        add_ip_route
    else
        echo "Route already exists."
    fi
}

# Execute main function
main

Skipping the error validations and other verifications, the first step is adding a rule with the name SSH to /etc/iproute2/rt_tables.

jorge@inteleon:~$ tail /etc/iproute2/rt_tables
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep
200 SSH

After that, the script tries to execute two commands:

jorge@inteleon:~$ sudo ip rule add from 10.0.101.100 table SSH
jorge@inteleon:~$ sudo ip route add default via 10.0.101.1 dev ens18 table SSH

The first command defines that the rules with the SSH name will affect traffic originating from 10.0.101.100.

After that, we add a rule so that the default gateway for traffic originating from 10.0.101.100 is forced to 10.0.101.1. This prevents the traffic from following “closer” paths, such as in the case where the server has an interface in the network where the client is.

Now that we have a script, we can set up a systemd service for this.

First, we need to make sure that our file is executable.

jorge@inteleon:~$ sudo chmod 755 /etc/init.d/set_ssh_routing.sh
[sudo] password for jorge:

After that, we need to create the system file in /etc/systemd/system/set_ssh_routing.service:

[Unit]
Description=Set SSH Specific Routing
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/init.d/set_ssh_routing.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

Now that we are ready, we need to enable and start the service.

jorge@inteleon:~$ sudo systemctl daemon-reload
jorge@inteleon:~$ sudo systemctl start set_ssh_routing.service
jorge@inteleon:~$ sudo systemctl enable set_ssh_routing.service

After executing these commands, just to be sure, we can proceed to reboot the server and test it. By now the asymmetric routing should be fixed.