[PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )

[Ted Mittelstaedt]

Raised flooring went out with IBM servers, lol.  Far easier to run overhead cable management.  I'm not a fan of pulling a raised floor to get at a cable and finding a dead rat down there.

The Dell and HP systems require an extra license fee be paid to enable the remote tools and most of my customers are smaller.  Their tendency is to try to press workstations into use as servers, it's a big stretch to get them to actually buy a real server like a Proliant, let alone pay the additional fees to enable ILO. It's also kind of hard to reach those servers when the Internet connection itself is down.  I have actually in a few cases in the past gotten 2-3 year old servers off Ebay for a particular customer who was resistant to the idea of paying real money to replace the typical 5 year old workstation box out of warranty under the CEO's desk that gets kicked occasionally.  I've also supplied at very little cost (since I picked them up used for free or very little cost) relay racks and shelving and other accessories to some customers to outfit a closet as a "server room"

Once I get them setup with real server hardware and they notice wow - the server isn't going down every week - then they start to become believers.  But it takes a lot of baby steps and time for this.  And there's a LOT of hack techs running around out there who are happy to continue nursing the 5 year old workstation boxes out of warranty under the CEO's desk that get kicked occasionally.  I guess their MO is make money from service calls so they encourage that nonsense.  I only do retainers so as I explain to my customers, _I_ have a financial incentive for things to NOT go down because if they are going down all the time, my retainer fee isn't going to cover my time, whereas if you are paying that fee and you never see me, then that's good for you because then things are never going down, got it?  It's like a revelation to some of them.

The other thing is that most smaller customers do not, in fact, have a real Terminal Server.  What I do in those cases is either setup VPNs using Untangle as a firewall (Untangle has very slick support for OpenVPN) to replace the usual 4 port Netgear router or cablemodem/router combo, or I load Microsoft Remote Desktop Gateway Server on one of their servers than setup the RDP clients to use the GW server.  They RDP into their desktops not a terminal server.

The remote KVM's are cool but once more, you have to have an operating Internet connection for them to work.  With my customers most of their downtime is due to workstation issues and Internet connectivity so a KVM is not going to help.  And there's a whole circus to discuss on what is called "solid workstation hardware"   It's why I only buy HP Elites and Pros nowadays for myself and my family members instead of the crappy stuff.  But when a small business is looking at upgrading 10-20 desktops it's very hard for them to see why they should double their spend for good gear when they can get the cheap crap for half the cost of good gear.

Ted


-----Original Message-----
From: PLUG <plug-bounces at lists.pdxlinux.org> On Behalf Of Robert Citek
Sent: Sunday, February 26, 2023 10:19 PM
To: Portland Linux/Unix Group <plug at lists.pdxlinux.org>
Subject: [PLUG] Remote work on downed server ( Re: ANNOUNCEMENT: March PLUG Meeting: Anatomy of a Mailing List Meltdown )

On Sun, Feb 26, 2023 at 9:52 PM Ted Mittelstaedt <tedm at portlandia-it.com>
wrote:

> However during the entire pandemic I was still out and about - since 
> you can't do IT consulting on a server that's down remotely.


By "server", I am assuming that you mean some system on rails in a rack in a datacenter with raised flooring, hot/cold aisles, redundant power/networking, and physical security.  In that environment, you usually can ( and want to ) be able to work on a downed server remotely.  For example, Dell has iDRAC/DRAC and HP has iLO.  For those systems that don't have built-in out-of-band ( OOB ) management, there are multi-port KVM over IP switches with many having virtual USB/CDs and power control.[1]  For single use, there is the Lantronix Spider which is also available with remote power control.[2]  In other words, you can connect over the internet to the DRAC/KVM ( e.g. ssh ), upload an ISO of your OS onto the virtual CD, power cycle the box, and have full remote control from BIOS to RAID to OS repair/installation.

If the issue is hardware, e.g. bad drive, bad power supply, you put in a service request to remote hands at the data center and have them hot-swap your cold spare for the bad device.  You've given them a copy of your runbook.  They know what to do.

lf the system has truly failed, you have a new system sent to the data center.  When it arrives, have remote hands swap the bad for the good, plugging it into the OOB so you can once again access it remotely.  And they package and send the bad system back to wherever.

On the other hand, if by "server" you mean the five year old box that's out of warranty, sitting under the CEO's desk, and gets kicked every time they reach to answer the phone, then that's a different scenario.  Although, attaching a Spider to it would be a nice option.

[1] https://en.wikipedia.org/wiki/KVM_switch
[2] https://www.lantronix.com/products/lantronix-spider/

Regards,
- Robert