Raspberry Pi lab network with external wifi access

By Robert Russell

January 25, 2023 - 16 minutes read - 3379 words

Overview

I wanted to build a self-sufficient portable lab network consisting entirely of Raspberries Pi. The lab network should be able to access the internet if there’s wifi available. And it should be easy to connect an external computer to the network and access any of the Raspberries Pi over the lab network. Basically I’d like to be able to run the whole thing on battery power and use it with or without upstream internet.

I’m pretty happy with the solution I came up with. One of the Raspberries Pi acts as a wifi access point (AP) and another one connects to an available external wifi network for an upstream internet connection. All of the Raspberries Pi are connected to a wired router as well. This is the same system I tweeted about a while ago. This set of instructions is for myself or anyone who wants to build their own portable lab network. I’m not the first one to think of this - see the References section at the end for some other examples.

My scheme takes advantage of the fact that the Raspberry Pi 4B has two network connections - wired and wifi - and I only needed the wired connection for the machines on the lab network to talk to each other. The wifi radio can only act as either a wifi AP or a wifi client, not both at the same time. But there are multiple Raspberries Pi in this network so I configured one to act as an AP and a different one to act as a wifi client. I call the wifi client an upstream wifi connection and the access point provided by the lab network the downstream wifi connection for this document.

Network Participants

The lab network exists to connect the Raspberries Pi 4B to each other, these are the permanent residents of the network. Each of the 6 machines is simply plugged in to an unmanaged wired ethernet switch. Other participants come and go by connecting either to the wired switch or by connecting to the lab network wifi access point. Here’s a list of the machines involved in the lab network. I’ve given each an arbitrary name so as not to get hung up on what the kind of computer it is or what IP address it has.

RPi with no special servers, wifi turned off: Jules, Marion, and Ethel.
RPi with DHCP and DNS server, also provides downstream wifi: Micheaux.
RPi with upstream wifi, default gateway to the internet: Parker.
Laptop connected to wifi AP: Ackroyd.
Desktop occasionally plugged in to router: Stravinsky.

When an external machine like Ackroyd or Stravinsky connects they’ll also get an IP address and can reach all the Raspberries Pi on the network. All devices on the network, including Ackroyd and Stravinsky, can access the internet through the shared connection provided by Parker.

Planning

Drawing the network topology and writing down parameter values ahead of time takes some patience but it helps a lot later on. The initial setup will involve lots of turning computers on and off. Good notes save a lot of time.

Topology

Here’s the topology of the network, shown with 6 nodes but the same topology could be built with as few as two.

Lab network connections

IP addresses and other lab network parameters

Subnet: 10.20.0.xxx
SSID: array
Passphrase for AP: echobravojacketpotato
IP address pool for dynamic allocation: 10.20.0.201 to 10.20.0.250
Permanent IP addresses:
10.20.0.1: satchmo
10.20.0.2: micheaux
10.20.0.3: jules
10.20.0.4: marion
10.20.0.5: parker
10.20.0.6: ethel

Static and reserved IPs

Consistent IP addresses for all the nodes on the lab network are nice since there’s always going to be some situation where you want to know the IP address even if the DNS usually works. Reserving addresses via DHCP is better than setting up a static IP on each machine since it keeps all that logic on the DHCP server. But setting up the reserved addresses can be a pain since the mac addresses have to be collected from each node.

The DHCP server is given a static IP since the address will show up in other config files. The default gateway IP address can be assigned with a DHCP reservation but the address will also be used directly. These are 10.20.0.2 (Micheaux) and 10.20.0.5 (Parker) in the example. All other lab network nodes have IP addresses reserved via DHCP so the addresses will be consistent and can be managed centrally in the dnsmasq DHCP server configuration.

Remember: a static IP address is in a config file on the client that has the static IP. A DHCP reserved address is in a config file on the machine running the DHCP server. A client with a reserved address will still ask for an address when the network comes up - the reservation means that the DHCP server will always assign the same IP address.

Install software

Temporarily connect to wifi or another network to install packages on each of the Raspberries Pi. The easiest workflow is (unfortunately) to use a keyboard, mouse, and monitor while setting up each RPi. Wired ethernet might be the easiest since it’s easy to control externally. If wifi is used, don’t forget to turn it off later on nodes like satchmo, jules, marion, and ethel. A good time to do that is after the DHCP server on micheaux is up and running. Only the default gateway, parker, should have the wifi client enabled. Use raspi-config to turn wifi on or off when needed.

For each Raspberry Pi write a fresh microSD card using the Raspberry Pi Imager. Open the advanced options dialog with ctrl-shift-x and configure a unique hostname, username, and enable wireless LAN if needed.

Tip: After writing the card create a text file on it with any notes for yourself. That file can be found under the /boot/ directory when the RPi is up and running.

All lab network machines

Software installation is grouped together here so that packages can be installed and then internet access won’t be needed again until after the lab network has upstream internet access.

The dnsutils package includes dig and other commands that can help diagnose DNS issues during setup.

sudo apt update
sudo apt install -y dnsutils

DNS and DHCP server

Micheaux will act as both DNS and DHCP server because dnsmasq performs both those tasks.

sudo apt install -y dnsmasq

Downstream wifi

Micheaux also hosts the wifi access point by running hostapd. If this role were separated from the DHCP server then it would be difficult to bridge an external wifi client (like a laptop) to the lab network, that is to say it would be difficult for the laptop to get an address in the 10.20.0.xxx range.

sudo apt install -y hostapd

Upstream wifi

Parker provides the upstream wifi connection to the internet. The internet connection is called the default gateway on a network. The IP address of the default gateway is provided to clients on the network by the DHCP server. This could have been set up on micheaux but separating the responsibilities is more interesting.

sudo apt install -y netfilter-persistent iptables-persistent

Configuration

Most of the configuration happens on micheaux since it’s going to answer DNS and DHCP requests with dnsmasq and run hostapd to provide the downstream wifi network. The other special configuration happens on parker. It will need a permanent route from the lab network to the wifi interface.

These are the general steps:

Add DNS hostnames. A list of DNS to IP address mappings in /etc/hosts.
Set up dnsmasq. Configuration for dnsmasq goes in /etc/dnsmasq.d/.
Set up a network bridge. Bridge wlan0 to the wired interface on eth0.
Set up hostapd. The hostapd configuration holds parameters for downstream wifi.
Add the upstream wifi route. Use

Add DNS hostnames

The hosts file at /etc/hosts on micheaux will be used by dnsmasq to answer DNS queries from other nodes.

sudo vi /etc/hosts

Contents of /etc/hosts:

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

127.0.1.1       micheaux

10.20.0.1   satchmo
10.20.0.2   micheaux
10.20.0.3   jules
10.20.0.4   marion
10.20.0.5   parker
10.20.0.6   ethel

The first section of the file, up to the entry for 127.0.1.1 isn’t relevant. The lines for the lab network names look like:

10.20.0.N labmachinen

These addresses are for DNS but they have to match the addresses in the dnsmasq configuration since the latter is used for assigning addresses over DHCP.

Set up dnsmasq

dnsmasq provides both a DNS server and a DHCP server. The configuration files are under /etc/dnsmasq.d/.

This config file is on micheaux, notably micheaux doesn’t show up in the config since it can’t request a DHCP address from itself.

The configuration file will need the mac address of eth0 from all the machines that are going to have reserved addresses. This can mean booting them up one by one to retrieve a long text string.

Getting all the MAC addresses

Here are a couple easy ways to get the mac addresses for all ethernet interfaces and put it in a text file.

for i in /sys/class/net/* ; do echo $i , $(cat $i/address) ; done > /boot/mac.txt
sudo cp mac.txt /boot/

For just the mac address of eth0 this is shorter

sudo cp /sys/class/net/eth0/address /boot/mac-$(hostname).txt

The $(hostname) will be replaced with the name of the RPi so the file will be something like mac-raspberrypi.txt and the only thing in it will be the mac address of eth0.

In either case, the text file ends up in the /boot/ directory. That way the file can be read by popping the card out of the RPi (after shutting down) and putting it into a card reader of another computer. Be sure to label those cards if you go this route.

The ip command from the iproute2 suite of tools can help find information like this as well. For example

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether e4:5f:01:db:xx:yy brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DORMANT group default qlen 1000
    link/ether e4:5f:01:db:xx:zz brd ff:ff:ff:ff:ff:ff

Configuring dnsmasq

Create the configuration file.

sudo vi /etc/dnsmasq.d/array.conf

Initial contents of /etc/dnsmasq.d/array.conf follow. Note the dhcp-range line has a low timeout for address assignments. This makes it quicker for changes to propagate while setting up. Once it’s working change from 5m to a long time like 1h or 12h.

interface=br0
domain-needed
bogus-priv

# Set the default route (gateway) to parker.
dhcp-option=option:router,10.20.0.5

# Set DNS servers to announce
dhcp-option=6,0.0.0.0

# Set to 5 min for debug, 1h or more after things work.
# The range here will be assigned dynamically. Reservations
# need to be in the same subnet but outside this range.
dhcp-range=10.20.0.201,10.20.0.250,5m
#dhcp-range=10.20.0.201,10.20.0.250,1h
# satchmo
dhcp-host=e4:5f:01:45:f3:11,10.20.0.1
# jules
dhcp-host=dc:a6:32:d5:a9:33,10.20.0.3
# marion
dhcp-host=e4:5f:01:db:02:44,10.20.0.4
# parker
dhcp-host=e4:5f:01:02:3e:55,10.20.0.5
# ethel
dhcp-host=e4:5f:01:02:3c:66,10.20.0.6

# Upstream DNS servers
server=1.1.1.1
server=1.0.0.1

The options domain-needed and bogus-priv allow made up single-word names on the network. Otherwise domain names need to have dots and follow some other rules.

The first dhcp-option says that when a client is assigned IP address the client will also be notified that the default gateway is parker - the IP address 10.20.0.5. Clients will use the default gateway for any traffic that isn’t going to the local network.

Each DHCP address reservation is a dhcp-host line. The mac addresses collected earlier are used here. The IP address on the dhcp-host line has to match the one used in /etc/hosts/. This file contains DHCP assignments, not the DNS names.

Any names that dnsmasq doesn’t recognize are forwarded to upstream DNS servers. The upstream DNS servers given with the server lines are Cloudflare’s Public DNS. Google also offers a public DNS service, use it with these lines instead:

server=8.8.8.8
server=8.8.4.4

Make dnsmasq start at boot using systemd enable. The extra unmask command ensures that it’s possible to enable the service.

sudo systemctl unmask dnsmasq
sudo systemctl enable dnsmasq

Other configuration options

There are a lot of other interesting things that can be done to improve on this simple configuration. For example, names can provided from a different file using the addn-hosts option or the hostsdir option.

Another easy way to give a fixed address for a hostname is with the address option. For example:

address=/labnethomepage.robr.dev/10.20.0.2

Would be a simple way to serve a page at a well-known name to a user connecting to the downstream wifi.

Testing and debugging

Test the config is well-formatted with:

dnsmasq --test

Restart dnsmasq every time the configuration is changed:

sudo service dnsmasq restart

Logs from dnsmasq are collected by systemd. They can be viewed with:

journalctl --unit dnsmasq.service --boot

The --boot restricts to the most recent service run. Using --follow instead will show some recent log entries and show more log entries as they appear:

journalctl --unit dnsmasq.service --lines=30 --follow

Leave out the --lines flag for 10 lines. Press ctrl-C to quit.

DHCP debugging

Clients which have been given an IP address are listed in /var/lib/misc/dnsmasq.leases. This includes the reserved addresses as well as any dynamically assigned from the dhcp-range in the dnsmasq configuration.

DNS debugging

Use the dig command to test name resolution. dig will give the resulting name as well as the server it came from. It’s also possible to query a specific nameserver using dig rather than the defaults.

For example:

$ dig jules

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> jules
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6386
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;jules.                                IN      A

;; ANSWER SECTION:
jules.                 0       IN      A       10.20.0.3

;; Query time: 1020 msec
;; SERVER: 172.26.0.1#53(172.26.0.1) (UDP)
;; WHEN: Fri Jan 27 11:06:31 PST 2023
;; MSG SIZE  rcvd: 46

The line SERVER: 172.26.0.1#53(172.26.0.1) (UDP) indicates the answer came from 172.26.0.1 port 53 (default for DHCP). The 172.26.xx.xx subnet is some part of the WSL networking on my client machine, plugged in over a wired connection to the lab network router.

Specifying the server to query bypasses this local caching

$ dig @micheaux jules

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> @micheaux jules
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55994
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;jules.                                IN      A

;; ANSWER SECTION:
jules.                 0       IN      A       10.20.0.3

;; Query time: 0 msec
;; SERVER: 10.20.0.2#53(micheaux) (UDP)
;; WHEN: Fri Jan 27 11:09:54 PST 2023
;; MSG SIZE  rcvd: 51

The answer here is the same in this case - jules has the address 10.20.0.3 but the answer came directly from micheaux instead of a local cache. It’s very common for different DNS servers and caches to give different answers, especially when changes have been made recently.

Using ping can be a good stand-in to quickly confirm connectivity and it can be scripted in bash to check all machines on the network. For example:

for h in satchmo micheaux jules marion parker ethel ; do echo $h ; ping -c 1 $h ; done

The output is messy but it will tell whether that machine can be reached from the current machine using the given name.

When connecting with ssh, the ssh configuration file at ~/.ssh/config is also consulted for a name. A name can be used in the config file but that name won’t be seen by other services like ping or a web browser.

Set up a network bridge

A network bridge can be seen as a way to combine two network interfaces. The bridge gets a name like br0. Once the bridge is created between the wired interface eth0 and the wifi interface wlan0 the bridge will need an IP instead of the devices in the bridge.

Create the network bridge configuration

Add the bridge device by creating a systemd virtual network definition file:

sudo vi /etc/systemd/network/bridge-br0.netdev

Contents of /etc/systemd/network/bridge-br0.netdev:

[NetDev]
Name=br0
Kind=bridge

Add the interface eth0 to the bridge br0 by creating a systemd network configuration file:

sudo vi /etc/systemd/network/br0-member-eth0.network

Contents of /etc/systemd/network/br0-member-eth0.network:

[Match]
Name=eth0

[Network]
Bridge=br0

The Match section says the file applies to eth0 and the Network section says to add this link to the bridge br0. Don’t create a file for wlan0, hostapd will add it to the bridge.

Configure the DHCP client

To set a static IP address for micheaux and to tell it to use the same options dnsmasq is sending out, edit /etc/dhcpcd.conf.

sudo vi /etc/dhcpcd.conf:

Insert this line at the top:

denyinterfaces wlan0 eth0

and add this line at the bottom

interface br0
static ip_address=10.20.0.2/16
static routers=10.20.0.5
static domain_name_servers=10.20.0.2

The denyinterfaces directive will prevent an IP address from being assigned to wlan0 or eth0. The lines at the end will set up br0 with a static IP address and set the default gateway to parker.

Enable systemd-networkd

Start systemd-networkd at boot, it will manage the network using the files created under /etc/systemd/network.

sudo systemctl unmask systemd-networkd
sudo systemctl enable systemd-networkd

Debugging

Here’s the output of ip addr on the Raspberry Pi micheaux where the eth0 and wlan0 are bridged by br0:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether e4:5f:01:37:xx:yy brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:5f:01:37:xx:yy brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.2/16 brd 10.20.255.255 scope global noprefixroute br0
       valid_lft forever preferred_lft forever
    inet6 fe80::a69d:bf89:be76:db63/64 scope link
       valid_lft forever preferred_lft forever
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether e4:5f:01:37:xx:zz brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e65f:1ff:fe37:997b/64 scope link
       valid_lft forever preferred_lft forever

Set up hostapd

hostapd uses a wifi radio to create an access point, similar to a wifi router.

Enable the wireless access point service and set it to start at boot.

sudo systemctl unmask hostapd
sudo systemctl enable hostapd

Configure hostapd by editing /etc/hostapd/hostapd.conf:

country_code=US
interface=wlan0
bridge=br0
ssid=array
hw_mode=g
channel=7
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=echobravojacketpotato
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

The country_code must be set correctly so that the correct radio channels are used for your location.

Like other services, restart hostapd every time the configuration is changed:

sudo service hostapd restart

Logs from hostapd are collected by systemd. They can be viewed with:

journalctl --unit hostapd.service --boot

Add the upstream wifi route

Parker provides the upstream wifi and it’s the default gateway for all external traffic on the network. Up to here all of the special configuration has been on micheaux.

At the end of /etc/dhcpcd.conf, add:

interface wlan0
metric 100

The metric means the wlan0 interface will be preferred. A lower metric gives a higher priority, the default is around 1000 or 3000 for wireless.

Create a sysctl config file to enable forwarding. The name should be 99-something.conf. This one is named after the lab network:

sudo vi /etc/sysctl.d/99-array.conf

Contents of /etc/sysctl.d/99-array.conf

net.ipv4.ip_forward=1

The files under /etc/sysctl.d/ are read at boot by systemd-sysctl.service.

The kernel parameter set by this file also shows up under /proc/sys so the value can be checked by reading from the filesystem:

$ cat /proc/sys/net/ipv4/ip_forward
1

Use iptables to create the route and netfilter-persistent to save it.

sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
sudo netfilter-persistent save

Use ip route to check routing tables. This shows defaults as well as metrics which helps explain why one route or another is chosen.

References

The official Raspberry Pi documentation covers setting up hostapd and dnsmasq to create a bridged wireless access point

This forum post covers a very similar setup.

Names

The names used in this article were only used temporarily during setup. My actual hostnames follow a pattern like labmachine1, labmachine2, … labmachine6. The temporary names are mostly useful to separate the role of the machine (like DHCP server, DNS server, other participant) from the type of machine. These specific names were selected arbitrarily from an article on public domain day 2022.

gammacam