Building a NanoPi Cluster - Part 2

Page content

Now the hardware for the NanoPi-Datacenter is done, its time for the system setup. I chose Armbian 5.25 stable as base system for the master and the worker nodes since this is based on a Debian 8 (Jessie) with a 4.9.4 Kernel version.

The Debian image is available at armbian

Preparing the SD-Card

All nodes should have the same base image and default configuration, so the following steps are equal to all five SD-Cards execpt of the network configuration.

After inserting the SD-Card, dmesg | tail displays the most recent system messages which contains information about the name of the SD-Card device. In my case the name is /dev/mmcblk0 To be sure that the SD-Card is not mounted I checked the output of df -h. The following command writes the image to the SD-Card:

$ dd -if=./Armbian_5.25_Nanopineo_Debian_jessie_dev_4.9.4.img of=/dev/mmcblk0 bs=4M

Once the image is written to the SD-Card, I inserted the Card into one of the NanoPis to boot the System once. This is necessary because Armbian does some initializations like preparing the partions etc. After that, I configured the network connections.

My internal network is and the external address for the master comes via dhcp.

The easiest way to perform this task is to mount the Card on the laptop again and do the following steps:

  • Mount the SD-Card $ mount /dev/mmcblk0p1 /mnt/sdcard

  • Edit network settings $ vim /mnt/sdcard/etc/network/interfaces

  • Change content to

  #for master:
  iface eth0 inet static

  iface eth1 inet dhcp
  # for worker
  iface eth0 inet static
       address // .3; .4; .5 for the other nodes

Now the base system is ready, bootable and accessible on every cluster node. All futher configurations can now be done after the nodes are up and running!

Node configuration

Using the nodes in a cluster requires a bit more configuration on each node. Things like setting the hostname and adding ssh-keys make it easier to work on the cluster and bridging the network on the master node is required to make the internet available to the worker.

When connecting to an Armbian system for the first time it asks to change the root password, a new user and a hostname. I entered ‘raimund’ as user and

  • huiwaina as masters hostname
  • wahi-0[1-4] as workers hostname

A default step that I do on every system is to explicitly set the locale by typing dpkg-reconfigure locales. I use en_US.UTF-8 on almost every system as default locale.

Hostnames and SSH

To make all node reachable via hostname instead of the IP I added huiwaina wahi-01 wahi-02 wahi-03 wahi-04

at the end of /etc/hosts/.

After all the configuration a system reboot on every node is useful to make sure all settings are loaded correctly.

For the complete cluster the manager node reacts as the only reachable system from the outside of the cluster. To login to huiwaina via ssh as the user ‘raimund’ I add my public ssh-key to the ~/.ssh/authorized_keys file in the users home directory.

Since huiwaina is a proxy for the conntection to the other nodes and sometimes it could be necessary to automatically connect to a worker node I generate a new ssh key on the manager node without passphrase and add the public key to the authorized_keys file on the worker nodes. The ssh key is saved as id_rsa_cluster.

raimund@huiwaina $ ssh-keygen

raimund@wahi-0* $ mkdir .ssh && vim .ssh/authorized_keys

  Insert public key

Additionaly on the manager node I create the .ssh/config file to store configuration about connections to the worker nodes. The following configuration tells ssh to use the newly generated key, the user ‘raimund’ and the hostname:

Host wahi-01
  HostName wahi-01
  User raimund
  IdentityFile ~/.ssh/id_rsa_cluster
Host wahi-02
  HostName wahi-02
  User raimund
  IdentityFile ~/.ssh/id_rsa_cluster
Host wahi-03
  HostName wahi-03
  User raimund
  IdentityFile ~/.ssh/id_rsa_cluster
Host wahi-04
  HostName wahi-04
  User raimund
  IdentityFile ~/.ssh/id_rsa_cluster

Connecting the Internet from worker nodes

The worker nodes are currently separated from the internet cause the only connection to the world is configured on the manager. For updates, downloads etc. it is necessary to have access to internet on each node. To archive this I configure an IP forwarding on huiwaina.

sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
vim /etc/sysctl.conf
  -> add "net.ipv4.ip_forward=1"

iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
sh -c "iptables-save > /etc/iptables.ipv4.nat"

First thing after all cluster nodes gained internet connection, they get an update.

apt-get update && apt-get upgrade

Docker and the swarm

Docker went a long way since setting up the cluster and writing this Blog. In the first version I had to compile docker on my own for arm architecture. Meanwhile docker provides packages for many plattforms and distributions. That makes it much easier to get the swarm up and running.

In my case following the description for Debian is most expedient. The following steps summarize the installation of docker-ce. Each node gets docker on this way.

# Instal dependencies
apt-get install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg2 \

# Get the GPG key
curl -fsSL | sudo apt-key add -
apt-key fingerprint 0EBFCD88

# Add the repository
add-apt-repository \
   "deb [arch=amdhf] \
   $(lsb_release -cs) \

# Install docker-ce
apt-get update

apt-get install docker-ce

# Add the user to the docker group
adduser raimund docker

Now we have a fully system setup including docker!

But what about the swarm?

The swarm is only a few steps away! After a logout/login to really join the docker group the output of docker version shows the installed client and server version:

 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:30:35 2018
 OS/Arch:           linux/arm
 Experimental:      false

  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:26:28 2018
  OS/Arch:          linux/arm
  Experimental:     false

The manager node is the one I start initializing the swarm. Since the manager node has two network interfaces, I have to tell the swarm which one is the advertised address for the cluster.

One single command on huiwaina is enough to perform the initialization of the swarm.

docker swarm init --advertise-addr $IP_OF_ETH1

The output contains a token that is used on the nodes to join the swarm. There is a token for worker and one for manager nodes. The output of docker node ls now looks like this:

    arumq7jkm22paj6y1kmmyci6u *  huiwaina  Ready   Active        Leader

huiwaina is automatically a manager of the swarm so I only need to add the worker to the swarm using the given token.

docker swarm join --token SWMTKN-1-2jcd5hobd55yhuh6t05cxsum0943ndxk28bs0zes79iowui39p-dwxxk629e6wqjyjp6zoqgyqn6 $IP_OF_ETH1:2377

After executing the command on each worker the output of docker node ls is

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
jaq1h1juyn56row9jrpr2avmg *   huiwaina            Ready               Active              Leader              18.06.1-ce
hk0sckes27g1l5af3an6cqqij     wahi-01             Ready               Active                                  18.06.1-ce
z9la82cw7e0uyhhr29p14lws6     wahi-02             Ready               Active                                  18.06.1-ce
jr43l2r0q5z927gl1pdewhbr0     wahi-03             Ready               Active                                  18.06.1-ce
24ofgf6tf2j4s8ychmq973mje     wahi-04             Ready               Active                                  18.06.1-ce

Done. Jeah! Next is to test the cluster and have a UI to manage it.