Herring Cove
Duplicate ED25519 SSH Host Keys at Hetzner

  2015-12-30

It turned out I was too busy to write the article indicated in the first post due to various reasons, but I assume many blogs suffer from this. I still hope to deliver that at a later point, but for now, here is this blog’s first real content:

Update 2016-01-05

Hanno Böck published ed25519hetzner on GitHub, a script to scan OpenSSH host key and known_hosts files for shared keys from server hoster Hetzner.

Coverage in the news:

Yesterday

Yesterday I ordered two new servers at Hetzner (a well-known data center operator in Germany), pre-installed with Debian jessie. I noticed, that both servers have identical fingerprints for their ED25519 SSH host key:

7f:0e:75:35:5b:fe:bd:a6:df:97:7b:fd:0f:b7:65:7b

When I checked other machines, I found that on a CentOS system the ED25519 key file was much older than the other keys, with a time stamp of April 2015, so I had to assume that the same problem exists here. The fingerprint (which I found on a single other web site) is:

ad:95:9c:0a:09:14:be:23:90:c2:10:2e:83:f9:7c:93

At this point I contacted Hetzner and they corrected the problem in their install script within less than two hours for all 14 affected installation images.

Today

Today all customers with possibly affected systems received a mail (see below), containing a list of all systems ordered since the issue was introduced, including links to their public wiki with instructions in German and English language to solve the problem.

I must say, I’m impressed. Especially at this time of the year I would have expected a slower reaction or a less detailed announcement.

This is the full mail we received:

Subject: Wichtige Hetzner Online Kundeninformation: Server-Sicherheitshinweis
 / Important Hetzner Online Customer Information: Server Security Notice

[English version below]


Sehr geehrte Damen und Herren,

wir wenden uns heute an Sie, um auf ein mögliches Sicherheitsrisiko im
SSH-Dienst der folgenden angemieteten Server hinzuweisen:

EX40 #nnnnnn (aaa.bbb.ccc.ddd)
EX41 #nnnnnn (aaa.bbb.ccc.ddd)
EX41-SSD #nnnnnn (aaa.bbb.ccc.ddd)

SSH-Server verwenden Host-Keys, um sich Benutzern gegenüber eindeutig zu
identifizieren. Diese Keys werden normalerweise bei jeder Installation
eines Betriebssystems neu generiert.

Aufgrund eines Fehlers innerhalb einer Installationsroutine wurden seit
dem 10. April 2015 bei der Installation unserer Betriebssystem-Images die
Ed25519 SSH Host-Keys (/etc/ssh/ssh_host_ed25519_key) nicht mehr individuell
neu generiert.

Resultierend daraus verwenden betroffene Installationen einen einheitlichen
und Betriebssystem-Image-spezifischen Ed25519 SSH Host-Key.

Dieser Umstand könnte gegebenenfalls ausgenutzt werden, um mittels eines
Man-in-the-Middle-Angriffs SSH-Kommunikation zu entschlüsseln bzw.
zu manipulieren.

Aufgrund unseres Netzwerksetups ist ein Man-in-the-Middle-Angriff
innerhalb unseres Netzwerks eher unwahrscheinlich, da jeder Server nur
direkt mit dem jeweiligen Router kommunizieren kann.

Wir möchten Sie dennoch bitten, zeitnah die Ed25519 SSH Host-Keys Ihres
Servers zu ersetzen. Die weiteren Host-Keys (RSA, DSA, ECDSA) sind hiervon
nicht betroffen und sind individuell.

Eine Anleitung zum Austauschen der Host-Keys und weitere Informationen
haben wir in unserem DokuWiki unter
http://wiki.hetzner.de/index.php/Ed25519 hinterlegt.

An dieser Stelle möchten wir uns bei unserem Kunden Thomas Arendsen Hein,
Intevation GmbH bedanken. Er hat uns gestern Nachmittag auf die Problematik
aufmerksam gemacht. Der Fehler wurde umgehend behoben und Installationen
nach dem 29. Dezember 2015, 16:00 Uhr sind nicht mehr betroffen.

Bei Rückfragen stehen wir Ihnen gerne zur Verfügung. Bitte senden Sie
dazu eine Supportanfrage an ed25519@hetzner.de.

Vielen Dank für Ihr Verständnis.

Mit freundlichen Grüßen

Hetzner Online GmbH
Industriestr. 25
91710 Gunzenhausen
Tel: +49 9831 505-0
Fax: +49 9831 505-3
info@hetzner.de
www.hetzner.de

Registergericht Ansbach, HRB 6089
Geschäftsführer: Martin Hetzner

--------------------------------------------------------------------------

Dear Sir or Madam

We are contacting you today to inform you of a potential security risk
in the SSH service of the following rented servers:

EX40 #nnnnnn (aaa.bbb.ccc.ddd)
EX41 #nnnnnn (aaa.bbb.ccc.ddd)
EX41-SSD #nnnnnn (aaa.bbb.ccc.ddd)

An SSH server uses host keys to uniquely identify itself to connecting
clients. These keys are normally automatically regenerated each time a
new installation of the operating system is done.

Due to an error in the installation software introduced on April 10th,
2015, the Ed25519 SSH host keys (/etc/ssh/ssh_host_ed25519_key) on our
standard images were no longer individually regenerated.

This resulted in identical Ed25519 SSH host keys for each affected OS
image.

An attacker could use this situation to compromise or eavesdrop on the
SSH communication between the client and the server using a
man-in-the-middle attack.

However, due to the security of our network setup, such an attack within
our network is highly unlikely as each server can only directly
communicate with the corresponding router.

Nevertheless we would like to urge you to replace the Ed25519 SSH host
key of your server as soon as possible. The other host keys (RSA, DSA,
ECDSA) are not affected and are unique.

Instructions and information on how to replace the host key can be
found in our wiki under http://wiki.hetzner.de/index.php/Ed25519/en

At this point we would like to thank our customer Arendsen Thomas Hein,
Intevation GmbH. He alerted us to this problem yesterday afternoon. The
error was promptly corrected, and installations after December 29th,
4 pm are no longer affected.

For any queries please do not hesitate to contact us. In this case
please send a support request to ed25519@hetzner.de.

Thank you for your understanding.

Kind regards

Hetzner Online GmbH
Industriestr. 25
91710 Gunzenhausen / Germany
Tel: +49 9831 505-0
Fax: +49 9831 505-3
info@hetzner.com
www.hetzner.com

Register Court: Registergericht Ansbach, HRB 6089
CEO: Martin Hetzner