Raspberry Pi lab network with external wifi access
By Robert Russell
- 16 minutes read - 3379 wordsOverview
I wanted to build a self-sufficient portable lab network consisting entirely of Raspberries Pi. The lab network should be able to access the internet if there’s wifi available. And it should be easy to connect an external computer to the network and access any of the Raspberries Pi over the lab network. Basically I’d like to be able to run the whole thing on battery power and use it with or without upstream internet.
I’m pretty happy with the solution I came up with. One of the Raspberries Pi acts as a wifi access point (AP) and another one connects to an available external wifi network for an upstream internet connection. All of the Raspberries Pi are connected to a wired router as well. This is the same system I tweeted about a while ago. This set of instructions is for myself or anyone who wants to build their own portable lab network. I’m not the first one to think of this - see the References section at the end for some other examples.
My scheme takes advantage of the fact that the Raspberry Pi 4B has two network connections - wired and wifi - and I only needed the wired connection for the machines on the lab network to talk to each other. The wifi radio can only act as either a wifi AP or a wifi client, not both at the same time. But there are multiple Raspberries Pi in this network so I configured one to act as an AP and a different one to act as a wifi client. I call the wifi client an upstream wifi connection and the access point provided by the lab network the downstream wifi connection for this document.
Network Participants
The lab network exists to connect the Raspberries Pi 4B to each other, these are the permanent residents of the network. Each of the 6 machines is simply plugged in to an unmanaged wired ethernet switch. Other participants come and go by connecting either to the wired switch or by connecting to the lab network wifi access point. Here’s a list of the machines involved in the lab network. I’ve given each an arbitrary name so as not to get hung up on what the kind of computer it is or what IP address it has.
- RPi with no special servers, wifi turned off: Jules, Marion, and Ethel.
- RPi with DHCP and DNS server, also provides downstream wifi: Micheaux.
- RPi with upstream wifi, default gateway to the internet: Parker.
- Laptop connected to wifi AP: Ackroyd.
- Desktop occasionally plugged in to router: Stravinsky.
When an external machine like Ackroyd or Stravinsky connects they’ll also get an IP address and can reach all the Raspberries Pi on the network. All devices on the network, including Ackroyd and Stravinsky, can access the internet through the shared connection provided by Parker.
Planning
Drawing the network topology and writing down parameter values ahead of time takes some patience but it helps a lot later on. The initial setup will involve lots of turning computers on and off. Good notes save a lot of time.
Topology
Here’s the topology of the network, shown with 6 nodes but the same topology could be built with as few as two.
IP addresses and other lab network parameters
Subnet: 10.20.0.xxx
SSID: array
Passphrase for AP: echobravojacketpotato
IP address pool for dynamic allocation: 10.20.0.201 to 10.20.0.250
Permanent IP addresses:
10.20.0.1: satchmo
10.20.0.2: micheaux
10.20.0.3: jules
10.20.0.4: marion
10.20.0.5: parker
10.20.0.6: ethel
Static and reserved IPs
Consistent IP addresses for all the nodes on the lab network are nice since there’s always going to be some situation where you want to know the IP address even if the DNS usually works. Reserving addresses via DHCP is better than setting up a static IP on each machine since it keeps all that logic on the DHCP server. But setting up the reserved addresses can be a pain since the mac addresses have to be collected from each node.
The DHCP server is given a static IP since the address will show up in other config files. The default gateway IP address can be assigned with a DHCP reservation but the address will also be used directly. These are 10.20.0.2
(Micheaux) and 10.20.0.5 (Parker) in the example. All other lab network nodes have IP addresses reserved via DHCP so the addresses will be consistent and can be managed centrally in the dnsmasq
DHCP server configuration.
Remember: a static IP address is in a config file on the client that has the static IP. A DHCP reserved address is in a config file on the machine running the DHCP server. A client with a reserved address will still ask for an address when the network comes up - the reservation means that the DHCP server will always assign the same IP address.
Install software
Temporarily connect to wifi or another network to install packages on each of the Raspberries Pi. The easiest workflow is (unfortunately) to use a keyboard, mouse, and monitor while setting up each RPi. Wired ethernet might be the easiest since it’s easy to control externally. If wifi is used, don’t forget to turn it off later on nodes like satchmo, jules, marion, and ethel. A good time to do that is after the DHCP server on micheaux is up and running. Only the default gateway, parker, should have the wifi client enabled. Use raspi-config
to turn wifi on or off when needed.
For each Raspberry Pi write a fresh microSD card using the Raspberry Pi Imager. Open the advanced options dialog with ctrl-shift-x
and configure a unique hostname, username, and enable wireless LAN if needed.
Tip: After writing the card create a text file on it with any notes for yourself. That file can be found under the /boot/
directory when the RPi is up and running.
All lab network machines
Software installation is grouped together here so that packages can be installed and then internet access won’t be needed again until after the lab network has upstream internet access.
The dnsutils
package includes dig
and other commands that can help diagnose DNS issues during setup.
sudo apt update
sudo apt install -y dnsutils
DNS and DHCP server
Micheaux will act as both DNS and DHCP server because dnsmasq
performs both those tasks.
sudo apt install -y dnsmasq
Downstream wifi
Micheaux also hosts the wifi access point by running hostapd
. If this role were separated from the DHCP server then it would be difficult to bridge an external wifi client (like a laptop) to the lab network, that is to say it would be difficult for the laptop to get an address in the 10.20.0.xxx
range.
sudo apt install -y hostapd
Upstream wifi
Parker provides the upstream wifi connection to the internet. The internet connection is called the default gateway on a network. The IP address of the default gateway is provided to clients on the network by the DHCP server. This could have been set up on micheaux but separating the responsibilities is more interesting.
sudo apt install -y netfilter-persistent iptables-persistent
Configuration
Most of the configuration happens on micheaux since it’s going to answer DNS and DHCP requests with dnsmasq
and run hostapd
to provide the downstream wifi network. The other special configuration happens on parker
. It will need a permanent route from the lab network to the wifi interface.
These are the general steps:
- Add DNS hostnames. A list of DNS to IP address mappings in
/etc/hosts
. - Set up dnsmasq. Configuration for
dnsmasq
goes in/etc/dnsmasq.d/
. - Set up a network bridge. Bridge
wlan0
to the wired interface oneth0
. - Set up hostapd. The
hostapd
configuration holds parameters for downstream wifi. - Add the upstream wifi route. Use
Add DNS hostnames
The hosts file at /etc/hosts
on micheaux will be used by dnsmasq
to answer DNS queries from other nodes.
sudo vi /etc/hosts
Contents of /etc/hosts
:
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 micheaux
10.20.0.1 satchmo
10.20.0.2 micheaux
10.20.0.3 jules
10.20.0.4 marion
10.20.0.5 parker
10.20.0.6 ethel
The first section of the file, up to the entry for 127.0.1.1
isn’t relevant. The lines for the lab network names look like:
10.20.0.N labmachinen
These addresses are for DNS but they have to match the addresses in the dnsmasq
configuration since the latter is used for assigning addresses over DHCP.
Set up dnsmasq
dnsmasq
provides both a DNS server and a DHCP server. The configuration files are under /etc/dnsmasq.d/
.
This config file is on micheaux, notably micheaux doesn’t show up in the config since it can’t request a DHCP address from itself.
The configuration file will need the mac address of eth0
from all the machines that are going to have reserved addresses. This can mean booting them up one by one to retrieve a long text string.
Getting all the MAC addresses
Here are a couple easy ways to get the mac addresses for all ethernet interfaces and put it in a text file.
for i in /sys/class/net/* ; do echo $i , $(cat $i/address) ; done > /boot/mac.txt
sudo cp mac.txt /boot/
For just the mac address of eth0 this is shorter
sudo cp /sys/class/net/eth0/address /boot/mac-$(hostname).txt
The $(hostname) will be replaced with the name of the RPi so the file will be something like mac-raspberrypi.txt
and the only thing in it will be the mac address of eth0
.
In either case, the text file ends up in the /boot/
directory. That way the file can be read by popping the card out of the RPi (after shutting down) and putting it into a card reader of another computer. Be sure to label those cards if you go this route.
The ip
command from the iproute2 suite of tools can help find information like this as well. For example
$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether e4:5f:01:db:xx:yy brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DORMANT group default qlen 1000
link/ether e4:5f:01:db:xx:zz brd ff:ff:ff:ff:ff:ff
Configuring dnsmasq
Create the configuration file.
sudo vi /etc/dnsmasq.d/array.conf
Initial contents of /etc/dnsmasq.d/array.conf
follow. Note the dhcp-range
line has a low timeout for address assignments. This makes it quicker for changes to propagate while setting up. Once it’s working change from 5m
to a long time like 1h
or 12h
.
interface=br0
domain-needed
bogus-priv
# Set the default route (gateway) to parker.
dhcp-option=option:router,10.20.0.5
# Set DNS servers to announce
dhcp-option=6,0.0.0.0
# Set to 5 min for debug, 1h or more after things work.
# The range here will be assigned dynamically. Reservations
# need to be in the same subnet but outside this range.
dhcp-range=10.20.0.201,10.20.0.250,5m
#dhcp-range=10.20.0.201,10.20.0.250,1h
# satchmo
dhcp-host=e4:5f:01:45:f3:11,10.20.0.1
# jules
dhcp-host=dc:a6:32:d5:a9:33,10.20.0.3
# marion
dhcp-host=e4:5f:01:db:02:44,10.20.0.4
# parker
dhcp-host=e4:5f:01:02:3e:55,10.20.0.5
# ethel
dhcp-host=e4:5f:01:02:3c:66,10.20.0.6
# Upstream DNS servers
server=1.1.1.1
server=1.0.0.1
The options domain-needed
and bogus-priv
allow made up single-word names on the network. Otherwise domain names need to have dots and follow some other rules.
The first dhcp-option
says that when a client is assigned IP address the client will also be notified that the default gateway is parker - the IP address 10.20.0.5
. Clients will use the default gateway for any traffic that isn’t going to the local network.
Each DHCP address reservation is a dhcp-host
line. The mac addresses collected earlier are used here. The IP address on the dhcp-host
line has to match the one used in /etc/hosts/
. This file contains DHCP assignments, not the DNS names.
Any names that dnsmasq
doesn’t recognize are forwarded to upstream DNS servers. The upstream DNS servers given with the server
lines are Cloudflare’s Public DNS. Google also offers a public DNS service, use it with these lines instead:
server=8.8.8.8
server=8.8.4.4
Make dnsmasq
start at boot using systemd enable. The extra unmask command ensures that it’s possible to enable the service.
sudo systemctl unmask dnsmasq
sudo systemctl enable dnsmasq
Other configuration options
There are a lot of other interesting things that can be done to improve on this simple configuration. For example, names can provided from a different file using the addn-hosts
option or the hostsdir
option.
Another easy way to give a fixed address for a hostname is with the address
option. For example:
address=/labnethomepage.robr.dev/10.20.0.2
Would be a simple way to serve a page at a well-known name to a user connecting to the downstream wifi.
Testing and debugging
Test the config is well-formatted with:
dnsmasq --test
Restart dnsmasq
every time the configuration is changed:
sudo service dnsmasq restart
Logs from dnsmasq
are collected by systemd. They can be viewed with:
journalctl --unit dnsmasq.service --boot
The --boot
restricts to the most recent service run. Using --follow
instead will show some recent log entries and show more log entries as they appear:
journalctl --unit dnsmasq.service --lines=30 --follow
Leave out the --lines
flag for 10 lines. Press ctrl-C to quit.
DHCP debugging
Clients which have been given an IP address are listed in /var/lib/misc/dnsmasq.leases
. This includes the reserved addresses as well as any dynamically assigned from the dhcp-range
in the dnsmasq
configuration.
DNS debugging
Use the dig
command to test name resolution. dig
will give the resulting name as well as the server it came from. It’s also possible to query a specific nameserver using dig
rather than the defaults.
For example:
$ dig jules
; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> jules
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6386
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;jules. IN A
;; ANSWER SECTION:
jules. 0 IN A 10.20.0.3
;; Query time: 1020 msec
;; SERVER: 172.26.0.1#53(172.26.0.1) (UDP)
;; WHEN: Fri Jan 27 11:06:31 PST 2023
;; MSG SIZE rcvd: 46
The line SERVER: 172.26.0.1#53(172.26.0.1) (UDP)
indicates the answer came from 172.26.0.1 port 53 (default for DHCP). The 172.26.xx.xx
subnet is some part of the WSL networking on my client machine, plugged in over a wired connection to the lab network router.
Specifying the server to query bypasses this local caching
$ dig @micheaux jules
; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> @micheaux jules
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55994
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;jules. IN A
;; ANSWER SECTION:
jules. 0 IN A 10.20.0.3
;; Query time: 0 msec
;; SERVER: 10.20.0.2#53(micheaux) (UDP)
;; WHEN: Fri Jan 27 11:09:54 PST 2023
;; MSG SIZE rcvd: 51
The answer here is the same in this case - jules has the address 10.20.0.3
but the answer came directly from micheaux instead of a local cache. It’s very common for different DNS servers and caches to give different answers, especially when changes have been made recently.
Using ping
can be a good stand-in to quickly confirm connectivity and it can be scripted in bash to check all machines on the network. For example:
for h in satchmo micheaux jules marion parker ethel ; do echo $h ; ping -c 1 $h ; done
The output is messy but it will tell whether that machine can be reached from the current machine using the given name.
When connecting with ssh
, the ssh configuration file at ~/.ssh/config
is also consulted for a name. A name can be used in the config file but that name won’t be seen by other services like ping
or a web browser.
Set up a network bridge
A network bridge can be seen as a way to combine two network interfaces. The bridge gets a name like br0
. Once the bridge is created between the wired interface eth0
and the wifi interface wlan0
the bridge will need an IP instead of the devices in the bridge.
Create the network bridge configuration
Add the bridge device by creating a systemd virtual network definition file:
sudo vi /etc/systemd/network/bridge-br0.netdev
Contents of /etc/systemd/network/bridge-br0.netdev
:
[NetDev]
Name=br0
Kind=bridge
Add the interface eth0
to the bridge br0
by creating a systemd network configuration file:
sudo vi /etc/systemd/network/br0-member-eth0.network
Contents of /etc/systemd/network/br0-member-eth0.network
:
[Match]
Name=eth0
[Network]
Bridge=br0
The Match
section says the file applies to eth0
and the Network
section says to add this link to the bridge br0
. Don’t create a file for wlan0
, hostapd
will add it to the bridge.
Configure the DHCP client
To set a static IP address for micheaux and to tell it to use the same options dnsmasq
is sending out, edit /etc/dhcpcd.conf
.
sudo vi /etc/dhcpcd.conf:
Insert this line at the top:
denyinterfaces wlan0 eth0
and add this line at the bottom
interface br0
static ip_address=10.20.0.2/16
static routers=10.20.0.5
static domain_name_servers=10.20.0.2
The denyinterfaces
directive will prevent an IP address from being assigned to wlan0
or eth0
. The lines at the end will set up br0
with a static IP address and set the default gateway to parker.
Enable systemd-networkd
Start systemd-networkd
at boot, it will manage the network using the files created under /etc/systemd/network
.
sudo systemctl unmask systemd-networkd
sudo systemctl enable systemd-networkd
Debugging
Here’s the output of ip addr on the Raspberry Pi micheaux where the eth0
and wlan0
are bridged by br0
:
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether e4:5f:01:37:xx:yy brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e4:5f:01:37:xx:yy brd ff:ff:ff:ff:ff:ff
inet 10.20.0.2/16 brd 10.20.255.255 scope global noprefixroute br0
valid_lft forever preferred_lft forever
inet6 fe80::a69d:bf89:be76:db63/64 scope link
valid_lft forever preferred_lft forever
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether e4:5f:01:37:xx:zz brd ff:ff:ff:ff:ff:ff
inet6 fe80::e65f:1ff:fe37:997b/64 scope link
valid_lft forever preferred_lft forever
Set up hostapd
hostapd
uses a wifi radio to create an access point, similar to a wifi router.
Enable the wireless access point service and set it to start at boot.
sudo systemctl unmask hostapd
sudo systemctl enable hostapd
Configure hostapd
by editing /etc/hostapd/hostapd.conf
:
country_code=US
interface=wlan0
bridge=br0
ssid=array
hw_mode=g
channel=7
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=echobravojacketpotato
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP
The country_code
must be set correctly so that the correct radio channels are used for your location.
Like other services, restart hostapd
every time the configuration is changed:
sudo service hostapd restart
Logs from hostapd
are collected by systemd. They can be viewed with:
journalctl --unit hostapd.service --boot
Add the upstream wifi route
Parker provides the upstream wifi and it’s the default gateway for all external traffic on the network. Up to here all of the special configuration has been on micheaux.
At the end of /etc/dhcpcd.conf
, add:
interface wlan0
metric 100
The metric
means the wlan0
interface will be preferred. A lower metric gives a higher priority, the default is around 1000 or 3000 for wireless.
Create a sysctl config file to enable forwarding. The name should be 99-something.conf
. This one is named after the lab network:
sudo vi /etc/sysctl.d/99-array.conf
Contents of /etc/sysctl.d/99-array.conf
net.ipv4.ip_forward=1
The files under /etc/sysctl.d/
are read at boot by systemd-sysctl.service.
The kernel parameter set by this file also shows up under /proc/sys
so the value can be checked by reading from the filesystem:
$ cat /proc/sys/net/ipv4/ip_forward
1
Use iptables to create the route and netfilter-persistent to save it.
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
sudo netfilter-persistent save
Use ip route
to check routing tables. This shows defaults as well as metrics which helps explain why one route or another is chosen.
References
The official Raspberry Pi documentation covers setting up hostapd
and dnsmasq
to create a bridged wireless access point
This forum post covers a very similar setup.
Names
The names used in this article were only used temporarily during setup. My actual hostnames follow a pattern like labmachine1, labmachine2, … labmachine6. The temporary names are mostly useful to separate the role of the machine (like DHCP server, DNS server, other participant) from the type of machine. These specific names were selected arbitrarily from an article on public domain day 2022.