[PLUG] [PLUG-ANNOUNCE] REMINDER: Portland Linux/Unix Group General Meeting Announcement: Two half-talks
Kevin Williams
kevin at k9w.org
Sat May 25 22:57:36 UTC 2024
Russel that's a great story! Nice job troubleshooting.
On 5/25/24 03:10, Russell Senior wrote:
>
>
> On 3/7/24 00:15, Russell Senior wrote:
>>
>>
>> On 3/1/24 17:40, Russell Senior wrote:
>>> Portland Linux/Unix Group General Meeting Announcement
>>>
>>> Who: Russell Senior
>>> What: Part 1: A Network Relay via Cloud Instance ; Part 2: Retro
>>> Linux Tape Recovery Show and Tell
>>> Where: 5500 SW Dosch Rd, Portland
>>> When: Thursday, March 7, 2024 at 7pm (Help with chairs a few minutes
>>> early is always appreciated)
>>> Why: The pursuit of technology freedom
>>>
>>> https://pdxlinux.org
>>>
>>> This is going to be a two-part talk, because each of the parts alone
>>> isn't enough to fill an hour (let's hope).
>>>
>>> The first part is going to be a description of how I relay network
>>> connections from the Internet to my low-volume home-based email
>>> server to evade potential ISP blockages.
>>>
>
> Earlier this week, you might have heard about a large Pacific Power
> outage in Northeast Portland. It only lasted for 30 or 40 minutes, but
> it affected a wide area and reportedly on the order of 30k customers.
> I was one of those affected. I was at home at the time. When all the
> UPSes started screaming at me, and the power didn't come back on
> immediately, I thought I'd better be pro-active and start shutting
> down machines. And it's good I did because the batteries I have
> wouldn't have sustained the load for that long. I wandered around the
> neighborhood, chatting with neighbors, exchanging information, and
> began contemplating building a soapbox racer or possibly pushing
> wheels with sticks in the dark time with no internet connections. It
> turns out, reports local journalists, a beaver up near the Columbia
> Slough had chewed through a tree that fell into some transmission
> lines and (i'm guessing now) caused a brief fault and opened up a
> circuit breaker. There must not have been any significant damage to
> the line because power was restored pretty much as soon as they'd
> identified the cause.
>
> Lights come back on, and with some relief I commenced to go around and
> turn back on the machines I'd turned off. Some of the machines had
> been up for a long time, sustaining long running sessions, so the
> downage was a chance to catch up on the deferred maintenance. For
> example, I'd purchased a Core i7 975 on ebay to replace a first-gen i7
> 920, to max out the CPU in one of my desktop boxes. This was a chance
> to replace the CPU, which I did and got that box powered back up. I
> also power my mailserver back on, which had been running without a
> reboot for nearly a year. It gets regular updates, but I hadn't
> rebooted into a new kernel. You might recall, I gave a PLUG talk in
> March describing the cloud-based tunnel I used to connect the internet
> to the mailserver in my house, bypassing any obstructions my ISP might
> employ. It has been working great. Life seemingly returned to normal.
>
> Then, Friday morning, I caught wind in a meeting that some mail (turns
> out, it was just mail being forwarded from gmail to my home server)
> was bouncing and the senders were seeing this odd domain they hadn't
> emailed. Uh-oh. But I had plans today and was away from home most of
> the day. This evening, I remembered about the mailserver and decided
> I'd better figure out what was going wrong. I had also recently
> updated my letsencrypt certificate, and that sometimes causes trouble
> if the mailserver doesn't use the new certificate, and I need to
> restart or reload the service.
>
> Oh, and the machine runs Arch. And yes, I don't mind that it sometimes
> gives me paper cuts. We're coming to that.
>
> So I look at my cloud hosted relay. If you recall the talk, the relay
> is just relaying packets, there's no server there other than the vpn I
> use to do the tunneling. And some tricky port forwarding,
> masquerading, ip rules, etc. At first, I'm just looking at the postfix
> logs on my mailserver, and I'm not seeing anything inbound. I look in
> iptables on the mail server to see if I'm dropping anything
> overzealously. Not that I can tell. I run tcpdump on the cloud-based
> relay, and I see TCP connections coming in but no answers. Weird. And
> then I run tcpdump on my mail server and I see TCP connection attempts
> there as well, but nothing going back over the tunnel interface, as
> they should. And then I think: "Hey, wait a minute, didn't someone
> just talk about this? And, hey, wait a minute, wasn't that person
> ME??? Where the hell are my slides?" and I go and find them, and flip
> through until I find the relevant bits. I had annoyingly obfuscated
> some of the addresses for the audience, so I had to translate the
> examples in my slides back to my actual context. And I start checking
> things like, are the fwmark rules intact (they were) and how about
> that ip rule? What? No ip rule? So I type in my translation, guessing
> a little at the table name. And ip tells me, "no such table". What? I
> remind myself that the table names are listed in a file called
> /etc/iproute2/rt_tables. I look in my /etc/iproute2 directory and I
> find rt_tables.pacsave, but no rt_tables. The pacsave version has my
> table name in it. Where did my file go? Well, I can just copy it back,
> which I do, and then run the ip command again with the table name and
> it works. And pretty much instantly, emails start flowing again.
>
> So, where the hell did someone get the idea that they should remove my
> custom rt_tables file? I look in /var/log/pacman.log and notice that
> iproute2 was updated recently and I go look at its commit and don't
> seen anything particularly guilty looking. Then I realize the box has
> been up since late June 2023, nearly a year and realize that my file
> could have disappeared anytime since then and it probably would
> continue working. So, I hop on the #archlinux channel and describe my
> observations and asking what might have caused this kind of rude file
> move that broke my perfectly working network. After 10 minutes or so,
> someone pipes up with the commit from a number of versions ago. It
> appears that the files in /etc/iproute2 are mostly commented examples,
> and that the modern place for such examples is in /usr/share/iproute2/
> and that the transition had, through an oversight, moved (thankfully
> not deleted) the file I was depending on.
>
> So, my mail wasn't being delivered correctly for a few days. And I
> spent an hour or so puzzling out what had gone wrong. And I'm reminded
> that people can be sloppy (whether paid or volunteers), and they can
> make mistakes and distributions like Arch are particularly susceptible
> to moving fast and occasionally breaking things, but I got a nice
> puzzle out of it and was reminded of some things that I might have
> otherwise forgotten, and I didn't even need to pay for a subscription
> to the nytimes puzzle service.
>
More information about the PLUG
mailing list