Friday, April 15, 2016

Fixing the Internets

Fixing the Internets
The last couple evenings have not exactly been fun. The flash memory card in my trusty Soekris net-4801 died. Yes, that's the box I set up 10 years ago. The net-4801 acted as my firewall, dhcp server, and primary DNS server for Lostentry and related domains. I'm using secondary DNS services from RollerNet. Without Internet and working from home, I switched on the hot spot on my work phone and finished the work day.

After the pets were taken care of, I rebooted the firewall again. When it didn't come back on, I figured "No problem. I just connect the null modem cable to my MacBook Pro and ... oh, right. No serial port."

"No problem, I pull out the trusty laptop from my data center days and I'll be connected in no time ... oh, right that Compaq got stripped off it's hard disk several years ago."

While I was contemplating which monitor and desktop computer to lug to the workbench in the garage, I found another laptop in my stash. "Ah yes. This Dell Inspiron has a serial port. Let's use that, ... oh, there's no minicom installed. Nor any other program that can talk to a serial port ... and it's running an ages old Debian install, I don't feel like connecting that directly to the Internet to find the right minicom installation." [though it probably would have been fine]

"No problem. I download the Grml Live CD to the MacBook Pro, and ... ah right, it doesn't have a CD drive, let alone a burner."

"No problem. I download the Live CD to the desktop in the bed room, and burn the CD there ... ah right, that desktop isn't anywhere near the Comcast cable modem and I really shouldn't connect my LAN to the Internet." [though, it likely would not have been a problem, because by now every device in the house had lost its dhcp lease, and Comcast wouldn't route my RFC1918 LAN address range anyways]

So I went upstairs and found oldest daughter's laptop, which has a CD-R drive, downloaded the Live CD, burned it, and ... the burn failed, because the download aborted shortly before the end, while I wasn't looking. It went through properly on the second try.

Soekris net4801-60
"I have a CD, I have a serial port, I have a null modem cable, I have a terminal program." That's when I found out that the CompactFlash from 10 years ago was dead.

"No problem. I pull out the OpenBSD machine from 11 years ago and load another CompactFlash card with the OpenBSD configuration + DNS and dhcp configs I had ... oh right, that config was on the CompactFlash that died ... and I really should not put OpenBSD 3.9 if I rebuild ..." has been superseeded by flashboot, which has been superseeded by flashrd for most uses. I should use the latest OpenBSD if I go that route. Which means I first need to build an OpenBSD server again.

"Maybe I should look at a Linux option." The OpenBSD install has always been somewhat of an odd man out in this household, eclipsed only by the NetBSD on SPARC install I did for fun one time.
Turns out, there are many small Linux distros around, the vast majority of which focus on running a minimal footprint desktop system. OpenWRT looks feasible ...  It might take a little while to get it the way I want, but certainly feasible.

"OH CRAP! How long will the secondary DNS servers hold on to my zone data?" Expiration on most of my domains is 2 weeks. But on the one domain that really matters, it's one week.
"When did RollerNet do the last successful zone transfer from my primary DNS server? ...  April 10 according to the logs on the admin Web site".
Yikes! I have to have this fixed before Sunday. Otherwise email for Lostentry accounts hosted on Google@Work won't get delivered anymore.

The phone rings. It's my family calling in from their road trip. The news of no Internet in the house is not taken up enthusiastically.

The focus of attention changes to building a temporary solution that gives me WiFi, and dhcp service for the home network so that I can bootstrap a DNS server and also get the Lostentry Web server back online.

"Hmmm, I have this old Netgear router in a box somewhere..." At the end of the evening yesterday, this Netgear WNR2000 is hooked up to the cable modem, configured, and provides Wifi to the household.
I briefly investigate external DNS services, but decide to first get back to a stable state before making further changes. Along the way I read through RollerNet's conditions for disabling secondary DNS service, which include "if ICMP ping is successful, but the DNS port is closed, the zone will be disabled". Whoops. I got lucky by having the router set to drop ICMP pings, instead of replying.

Tonight, I recreated the required BIND9 configs, brought up BIND9, and port-forwarded requests from the Netgear to the DNS server.

I now have a working DNS server again, with valid (and hopefully correct) configuration. Next I need to bring the remaining systems in the house back into a functional state. Mostly by matching up an internal DNS zone with dhcp assignments. Then I can dig deeper into deciding how I want to proceed with the firewall setup.

And there I thought I would have some time this week to work on trains, or refinish the kitchen counter while the family is away ...


Paga said...

The solution I use is:

- Raspberry Pi 2 (around ~$30-35) to work as my DNS, dhcp and everything else (I don't serve DNS to the outside, so in my case, dnsmasq does both DNS and DHCP very well. You might want consider a more robust dns server if this is your main DNS server.)

- TP-LINK TL-SG108E 8-Port Gigabit Easy Smart Switch: This sells for $34 at Amazon, but if you have time (looks like you don't) you can buy it for cheaper (it comes down to $20-something once in a blue moon.)

What I did:

- Download and install raspbian on the pi (basic install, no GUIs or anything). Good thing here is that this is Debian we all know well. Configure the pi to do minimal writes on the SD card (I got a collection of notes, if you want).

- Since the pi only has one ethernet port (100Mbps, grrr), you need to configure VLAN tagging (802.11q) on the switch and on the Pi.

- Use good ol' iptables to set up your firewall in any way you want.

- Boom! Done!

Small annoyances in this setup:

- This switch requires a stupid Windows program to configure VLANs. Since I have no Windows machines at home, I had to run windows inside a VM (it worked well inside VirtualBox).

- The pi can only do 100Mbps (and even so, poorly). This is not a problem for me, because my Internet is slower than that, and packets only go through the Pi when coming in or out of the Internet.

What I like about this setup:

- It's debian, which we all know and can be updated without much hassle.

- The rpi is a nice little machine that just sits there day after day. Never had a lockup of any kind.

- The whole thing is expandable: If in a few years someone launches a 1Gbps Rpi (or equivalent running Debian), a relative upgrade path exists.

Good luck!

Bernhard Beck said...

Thank you for the reminder. I did consider a small, low-power computer like the Pi before, but dislike the need for VLANs with the single NIC, and that my LAN is physically connected to the Internet.
I'll think about this some more, and we'll see which solution I end up with. For now I'm catching up on other commitments that I didn't get to due to this mess.