Building a NanoPi Cluster - Part 2
Now the hardware for the NanoPi-Datacenter is done, its time for the system setup. I chose Armbian 5.25 stable as base system for the master and the worker nodes since this is based on a Debian 8 (Jessie) with a 4.9.4 Kernel version.
The Debian image is available at armbian
Preparing the SD-Card
All nodes should have the same base image and default configuration, so the following steps are equal to all five SD-Cards execpt of the network configuration.
After inserting the SD-Card, dmesg | tail displays the most recent system
messages which contains information about the name of the SD-Card device.
In my case the name is /dev/mmcblk0
To be sure that the SD-Card is not mounted I checked the output of df -h.
The following command writes the image to the SD-Card:
$ dd -if=./Armbian_5.25_Nanopineo_Debian_jessie_dev_4.9.4.img of=/dev/mmcblk0 bs=4M
Once the image is written to the SD-Card, I inserted the Card into one of the NanoPis to boot the System once. This is necessary because Armbian does some initializations like preparing the partions etc. After that, I configured the network connections.
My internal network is 192.168.230.0/24 and the external address for the master comes via dhcp.
The easiest way to perform this task is to mount the Card on the laptop again and do the following steps:
Mount the SD-Card
$ mount /dev/mmcblk0p1 /mnt/sdcardEdit network settings
$ vim /mnt/sdcard/etc/network/interfacesChange content to
#for master:
iface eth0 inet static
address 192.168.230.1
netmask 255.255.255.0
network 192.168.230.0
broadcast 192.168.230.255
iface eth1 inet dhcp
# for worker
iface eth0 inet static
address 192.168.230.2 // .3; .4; .5 for the other nodes
netmask 255.255.255.0
gateway 192.168.230.1
Now the base system is ready, bootable and accessible on every cluster node. All futher configurations can now be done after the nodes are up and running!
Node configuration
Using the nodes in a cluster requires a bit more configuration on each node. Things like setting the hostname and adding ssh-keys make it easier to work on the cluster and bridging the network on the master node is required to make the internet available to the worker.
When connecting to an Armbian system for the first time it asks to change the root password, a new user and a hostname. I entered ‘raimund’ as user and
huiwainaas masters hostnamewahi-0[1-4]as workers hostname
A default step that I do on every system is to explicitly set the locale by
typing dpkg-reconfigure locales. I use en_US.UTF-8 on almost every system
as default locale.
Hostnames and SSH
To make all node reachable via hostname instead of the IP I added
192.168.230.1 huiwaina
192.168.230.2 wahi-01
192.168.230.3 wahi-02
192.168.230.4 wahi-03
192.168.230.5 wahi-04
at the end of /etc/hosts/.
After all the configuration a system reboot on every node is useful to make sure all settings are loaded correctly.
For the complete cluster the manager node reacts as the only reachable system
from the outside of the cluster. To login to huiwaina via ssh as the user
‘raimund’ I add my public ssh-key to the ~/.ssh/authorized_keys file in
the users home directory.
Since huiwaina is a proxy for the conntection to the other nodes and
sometimes it could be necessary to automatically connect to a worker node I
generate a new ssh key on the manager node without passphrase and add the public
key to the authorized_keys file on the worker nodes. The ssh key is saved
as id_rsa_cluster.
raimund@huiwaina $ ssh-keygen
[...]
raimund@wahi-0* $ mkdir .ssh && vim .ssh/authorized_keys
Insert public key
Additionaly on the manager node I create the .ssh/config file to store
configuration about connections to the worker nodes. The following configuration
tells ssh to use the newly generated key, the user ‘raimund’ and the hostname:
Host wahi-01
HostName wahi-01
User raimund
IdentityFile ~/.ssh/id_rsa_cluster
Host wahi-02
HostName wahi-02
User raimund
IdentityFile ~/.ssh/id_rsa_cluster
Host wahi-03
HostName wahi-03
User raimund
IdentityFile ~/.ssh/id_rsa_cluster
Host wahi-04
HostName wahi-04
User raimund
IdentityFile ~/.ssh/id_rsa_cluster
Connecting the Internet from worker nodes
The worker nodes are currently separated from the internet cause the only
connection to the world is configured on the manager. For updates, downloads
etc. it is necessary to have access to internet on each node. To archive this I
configure an IP forwarding on huiwaina.
sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
vim /etc/sysctl.conf
-> add "net.ipv4.ip_forward=1"
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
sh -c "iptables-save > /etc/iptables.ipv4.nat"
First thing after all cluster nodes gained internet connection, they get an update.
apt-get update && apt-get upgrade
Docker and the swarm
Docker went a long way since setting up the cluster and writing this Blog. In the first version I had to compile docker on my own for arm architecture. Meanwhile docker provides packages for many plattforms and distributions. That makes it much easier to get the swarm up and running.
In my case following the description for Debian is most expedient. The following steps summarize the installation of docker-ce. Each node gets docker on this way.
# Instal dependencies
apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common
# Get the GPG key
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt-key fingerprint 0EBFCD88
# Add the repository
add-apt-repository \
"deb [arch=amdhf] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
# Install docker-ce
apt-get update
apt-get install docker-ce
# Add the user to the docker group
adduser raimund docker
Now we have a fully system setup including docker!
But what about the swarm?
The swarm is only a few steps away! After a logout/login to really join the docker
group the output of docker version shows the installed client and server
version:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:30:35 2018
OS/Arch: linux/arm
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:26:28 2018
OS/Arch: linux/arm
Experimental: false
The manager node is the one I start initializing the swarm. Since the manager node has two network interfaces, I have to tell the swarm which one is the advertised address for the cluster.
One single command on huiwaina is enough to perform the initialization
of the swarm.
docker swarm init --advertise-addr $IP_OF_ETH1
The output contains a token that is used on the nodes to join the swarm. There
is a token for worker and one for manager nodes.
The output of docker node ls now looks like this:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
arumq7jkm22paj6y1kmmyci6u * huiwaina Ready Active Leader
huiwaina is automatically a manager of the swarm so I only need to add the
worker to the swarm using the given token.
docker swarm join --token SWMTKN-1-2jcd5hobd55yhuh6t05cxsum0943ndxk28bs0zes79iowui39p-dwxxk629e6wqjyjp6zoqgyqn6 $IP_OF_ETH1:2377
After executing the command on each worker the output of docker node ls
is
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
jaq1h1juyn56row9jrpr2avmg * huiwaina Ready Active Leader 18.06.1-ce
hk0sckes27g1l5af3an6cqqij wahi-01 Ready Active 18.06.1-ce
z9la82cw7e0uyhhr29p14lws6 wahi-02 Ready Active 18.06.1-ce
jr43l2r0q5z927gl1pdewhbr0 wahi-03 Ready Active 18.06.1-ce
24ofgf6tf2j4s8ychmq973mje wahi-04 Ready Active 18.06.1-ce
Done. Jeah! Next is to test the cluster and have a UI to manage it.
