[PLUG] [PLUG-ANNOUNCE] REMINDER: Portland Linux/Unix Group General Meeting Announcement: Two half-talks

Kevin Williams kevin at k9w.org
Sat May 25 22:57:36 UTC 2024


Russel that's a great story! Nice job troubleshooting.

On 5/25/24 03:10, Russell Senior wrote:
>
>
> On 3/7/24 00:15, Russell Senior wrote:
>>
>>
>> On 3/1/24 17:40, Russell Senior wrote:
>>> Portland Linux/Unix Group General Meeting Announcement
>>>
>>> Who: Russell Senior
>>> What: Part 1: A Network Relay via Cloud Instance ; Part 2: Retro 
>>> Linux Tape Recovery Show and Tell
>>> Where: 5500 SW Dosch Rd, Portland
>>> When: Thursday, March 7, 2024 at 7pm (Help with chairs a few minutes 
>>> early is always appreciated)
>>> Why: The pursuit of technology freedom
>>>
>>> https://pdxlinux.org
>>>
>>> This is going to be a two-part talk, because each of the parts alone 
>>> isn't enough to fill an hour (let's hope).
>>>
>>> The first part is going to be a description of how I relay network 
>>> connections from the Internet to my low-volume home-based email 
>>> server to evade potential ISP blockages.
>>>
>
> Earlier this week, you might have heard about a large Pacific Power 
> outage in Northeast Portland. It only lasted for 30 or 40 minutes, but 
> it affected a wide area and reportedly on the order of 30k customers. 
> I was one of those affected. I was at home at the time. When all the 
> UPSes started screaming at me, and the power didn't come back on 
> immediately, I thought I'd better be pro-active and start shutting 
> down machines. And it's good I did because the batteries I have 
> wouldn't have sustained the load for that long. I wandered around the 
> neighborhood, chatting with neighbors, exchanging information, and 
> began contemplating building a soapbox racer or possibly pushing 
> wheels with sticks in the dark time with no internet connections. It 
> turns out, reports local journalists, a beaver up near the Columbia 
> Slough had chewed through a tree that fell into some transmission 
> lines and (i'm guessing now) caused a brief fault and opened up a 
> circuit breaker. There must not have been any significant damage to 
> the line because power was restored pretty much as soon as they'd 
> identified the cause.
>
> Lights come back on, and with some relief I commenced to go around and 
> turn back on the machines I'd turned off. Some of the machines had 
> been up for a long time, sustaining long running sessions, so the 
> downage was a chance to catch up on the deferred maintenance. For 
> example, I'd purchased a Core i7 975 on ebay to replace a first-gen i7 
> 920, to max out the CPU in one of my desktop boxes. This was a chance 
> to replace the CPU, which I did and got that box powered back up. I 
> also power my mailserver back on, which had been running without a 
> reboot for nearly a year. It gets regular updates, but I hadn't 
> rebooted into a new kernel. You might recall, I gave a PLUG talk in 
> March describing the cloud-based tunnel I used to connect the internet 
> to the mailserver in my house, bypassing any obstructions my ISP might 
> employ. It has been working great. Life seemingly returned to normal.
>
> Then, Friday morning, I caught wind in a meeting that some mail (turns 
> out, it was just mail being forwarded from gmail to my home server) 
> was bouncing and the senders were seeing this odd domain they hadn't 
> emailed. Uh-oh. But I had plans today and was away from home most of 
> the day. This evening, I remembered about the mailserver and decided 
> I'd better figure out what was going wrong. I had also recently 
> updated my letsencrypt certificate, and that sometimes causes trouble 
> if the mailserver doesn't use the new certificate, and I need to 
> restart or reload the service.
>
> Oh, and the machine runs Arch. And yes, I don't mind that it sometimes 
> gives me paper cuts. We're coming to that.
>
> So I look at my cloud hosted relay. If you recall the talk, the relay 
> is just relaying packets, there's no server there other than the vpn I 
> use to do the tunneling. And some tricky port forwarding, 
> masquerading, ip rules, etc. At first, I'm just looking at the postfix 
> logs on my mailserver, and I'm not seeing anything inbound. I look in 
> iptables on the mail server to see if I'm dropping anything 
> overzealously. Not that I can tell. I run tcpdump on the cloud-based 
> relay, and I see TCP connections coming in but no answers. Weird. And 
> then I run tcpdump on my mail server and I see TCP connection attempts 
> there as well, but nothing going back over the tunnel interface, as 
> they should. And then I think: "Hey, wait a minute, didn't someone 
> just talk about this? And, hey, wait a minute, wasn't that person 
> ME??? Where the hell are my slides?" and I go and find them, and flip 
> through until I find the relevant bits. I had annoyingly obfuscated 
> some of the addresses for the audience, so I had to translate the 
> examples in my slides back to my actual context. And I start checking 
> things like, are the fwmark rules intact (they were) and how about 
> that ip rule? What? No ip rule? So I type in my translation, guessing 
> a little at the table name. And ip tells me, "no such table". What? I 
> remind myself that the table names are listed in a file called 
> /etc/iproute2/rt_tables. I look in my /etc/iproute2 directory and I 
> find rt_tables.pacsave, but no rt_tables. The pacsave version has my 
> table name in it. Where did my file go? Well, I can just copy it back, 
> which I do, and then run the ip command again with the table name and 
> it works. And pretty much instantly, emails start flowing again.
>
> So, where the hell did someone get the idea that they should remove my 
> custom rt_tables file? I look in /var/log/pacman.log and notice that 
> iproute2 was updated recently and I go look at its commit and don't 
> seen anything particularly guilty looking. Then I realize the box has 
> been up since late June 2023, nearly a year and realize that my file 
> could have disappeared anytime since then and it probably would 
> continue working. So, I hop on the #archlinux channel and describe my 
> observations and asking what might have caused this kind of rude file 
> move that broke my perfectly working network. After 10 minutes or so, 
> someone pipes up with the commit from a number of versions ago. It 
> appears that the files in /etc/iproute2 are mostly commented examples, 
> and that the modern place for such examples is in /usr/share/iproute2/ 
> and that the transition had, through an oversight, moved (thankfully 
> not deleted) the file I was depending on.
>
> So, my mail wasn't being delivered correctly for a few days. And I 
> spent an hour or so puzzling out what had gone wrong. And I'm reminded 
> that people can be sloppy (whether paid or volunteers), and they can 
> make mistakes and distributions like Arch are particularly susceptible 
> to moving fast and occasionally breaking things, but I got a nice 
> puzzle out of it and was reminded of some things that I might have 
> otherwise forgotten, and I didn't even need to pay for a subscription 
> to the nytimes puzzle service.
>


More information about the PLUG mailing list