[PLUG] [PLUG-ANNOUNCE] REMINDER: Portland Linux/Unix Group General Meeting Announcement: Two half-talks

Russell Senior russell at pdxlinux.org
Sat May 25 10:10:45 UTC 2024



On 3/7/24 00:15, Russell Senior wrote:
>
>
> On 3/1/24 17:40, Russell Senior wrote:
>> Portland Linux/Unix Group General Meeting Announcement
>>
>> Who: Russell Senior
>> What: Part 1: A Network Relay via Cloud Instance ; Part 2: Retro 
>> Linux Tape Recovery Show and Tell
>> Where: 5500 SW Dosch Rd, Portland
>> When: Thursday, March 7, 2024 at 7pm (Help with chairs a few minutes 
>> early is always appreciated)
>> Why: The pursuit of technology freedom
>>
>> https://pdxlinux.org
>>
>> This is going to be a two-part talk, because each of the parts alone 
>> isn't enough to fill an hour (let's hope).
>>
>> The first part is going to be a description of how I relay network 
>> connections from the Internet to my low-volume home-based email 
>> server to evade potential ISP blockages.
>>

Earlier this week, you might have heard about a large Pacific Power 
outage in Northeast Portland. It only lasted for 30 or 40 minutes, but 
it affected a wide area and reportedly on the order of 30k customers. I 
was one of those affected. I was at home at the time. When all the UPSes 
started screaming at me, and the power didn't come back on immediately, 
I thought I'd better be pro-active and start shutting down machines. And 
it's good I did because the batteries I have wouldn't have sustained the 
load for that long. I wandered around the neighborhood, chatting with 
neighbors, exchanging information, and began contemplating building a 
soapbox racer or possibly pushing wheels with sticks in the dark time 
with no internet connections. It turns out, reports local journalists, a 
beaver up near the Columbia Slough had chewed through a tree that fell 
into some transmission lines and (i'm guessing now) caused a brief fault 
and opened up a circuit breaker. There must not have been any 
significant damage to the line because power was restored pretty much as 
soon as they'd identified the cause.

Lights come back on, and with some relief I commenced to go around and 
turn back on the machines I'd turned off. Some of the machines had been 
up for a long time, sustaining long running sessions, so the downage was 
a chance to catch up on the deferred maintenance. For example, I'd 
purchased a Core i7 975 on ebay to replace a first-gen i7 920, to max 
out the CPU in one of my desktop boxes. This was a chance to replace the 
CPU, which I did and got that box powered back up. I also power my 
mailserver back on, which had been running without a reboot for nearly a 
year. It gets regular updates, but I hadn't rebooted into a new kernel. 
You might recall, I gave a PLUG talk in March describing the cloud-based 
tunnel I used to connect the internet to the mailserver in my house, 
bypassing any obstructions my ISP might employ. It has been working 
great. Life seemingly returned to normal.

Then, Friday morning, I caught wind in a meeting that some mail (turns 
out, it was just mail being forwarded from gmail to my home server) was 
bouncing and the senders were seeing this odd domain they hadn't 
emailed. Uh-oh. But I had plans today and was away from home most of the 
day. This evening, I remembered about the mailserver and decided I'd 
better figure out what was going wrong. I had also recently updated my 
letsencrypt certificate, and that sometimes causes trouble if the 
mailserver doesn't use the new certificate, and I need to restart or 
reload the service.

Oh, and the machine runs Arch. And yes, I don't mind that it sometimes 
gives me paper cuts. We're coming to that.

So I look at my cloud hosted relay. If you recall the talk, the relay is 
just relaying packets, there's no server there other than the vpn I use 
to do the tunneling. And some tricky port forwarding, masquerading, ip 
rules, etc. At first, I'm just looking at the postfix logs on my 
mailserver, and I'm not seeing anything inbound. I look in iptables on 
the mail server to see if I'm dropping anything overzealously. Not that 
I can tell. I run tcpdump on the cloud-based relay, and I see TCP 
connections coming in but no answers. Weird. And then I run tcpdump on 
my mail server and I see TCP connection attempts there as well, but 
nothing going back over the tunnel interface, as they should. And then I 
think: "Hey, wait a minute, didn't someone just talk about this? And, 
hey, wait a minute, wasn't that person ME??? Where the hell are my 
slides?" and I go and find them, and flip through until I find the 
relevant bits. I had annoyingly obfuscated some of the addresses for the 
audience, so I had to translate the examples in my slides back to my 
actual context. And I start checking things like, are the fwmark rules 
intact (they were) and how about that ip rule? What? No ip rule? So I 
type in my translation, guessing a little at the table name. And ip 
tells me, "no such table". What? I remind myself that the table names 
are listed in a file called /etc/iproute2/rt_tables. I look in my 
/etc/iproute2 directory and I find rt_tables.pacsave, but no rt_tables. 
The pacsave version has my table name in it. Where did my file go? Well, 
I can just copy it back, which I do, and then run the ip command again 
with the table name and it works. And pretty much instantly, emails 
start flowing again.

So, where the hell did someone get the idea that they should remove my 
custom rt_tables file? I look in /var/log/pacman.log and notice that 
iproute2 was updated recently and I go look at its commit and don't seen 
anything particularly guilty looking. Then I realize the box has been up 
since late June 2023, nearly a year and realize that my file could have 
disappeared anytime since then and it probably would continue working. 
So, I hop on the #archlinux channel and describe my observations and 
asking what might have caused this kind of rude file move that broke my 
perfectly working network. After 10 minutes or so, someone pipes up with 
the commit from a number of versions ago. It appears that the files in 
/etc/iproute2 are mostly commented examples, and that the modern place 
for such examples is in /usr/share/iproute2/ and that the transition 
had, through an oversight, moved (thankfully not deleted) the file I was 
depending on.

So, my mail wasn't being delivered correctly for a few days. And I spent 
an hour or so puzzling out what had gone wrong. And I'm reminded that 
people can be sloppy (whether paid or volunteers), and they can make 
mistakes and distributions like Arch are particularly susceptible to 
moving fast and occasionally breaking things, but I got a nice puzzle 
out of it and was reminded of some things that I might have otherwise 
forgotten, and I didn't even need to pay for a subscription to the 
nytimes puzzle service.

-- 
Russell Senior
russell at pdxlinux.org



More information about the PLUG mailing list