Let's Encrypt, DNS over TLS and Android
Because of the recent expiry of the certificate used to cross-sign Let's Encrypt certificates, Android gets confused when you try to use them for DoT.
Today I ran into a weird problem. I've set up my Pihole as a DNS server for my home LAN and have since added DNS over HTTPS and DNS over TLS support to it. This all worked great, and even works flawlessly from outside my LAN.
Android supports DoT servers natively under the name "Private DNS", which works very well.
Today, the DoT DNS server suddenly stopped working. Stubby could query for domains fine and the certificate chain was reported valid, but Android insisted that the server could not be reached.
As it turns out, I've used the standard Let's Encrypt setup to proxy DNS requests over TCP+TLS to the DNS server, and the standard setup includes a certificate in the certificate chain sent by the server that expired today.
The certificate was used to cross-sign Let's Encrypt certificates before their root CA was included in common devices. All modern devices have the dedicated Let's Encrypt certificate in their system trust stores, so they don't need that certificate in the chain. However, due to a bug in some crypto library, any expired certificate in the chain will throw off the verification code.
This means that the expired certificate that wasn't needed anymore anyway caused the DNS client to fail certificate validation.
The quick fix? Simply remove the bad intermediate certificate from the chain. For me this was as simple as just finding the appropriate fullchain.pem and chain.pem and removing the bottom certificate from the file. After reloading nginx the entire thing started working again immediately.
The real fix would be to patch the broken TLS library in my phone's Android operating system, but I don't have control over that. Luckily, my device was barely broken. Tons of other, older devices can no longer access a large part of the web today. If those devices don't have the ability to install the Let's Encrypt root certificate, they probably won't get their previous internet access back any time soon.
Written by Jeroen on October 1, 2021