[PLUG] rebuilding your network after a crash... literally!

Russell Senior russell at personaltelco.net
Sat Aug 5 06:19:40 UTC 2006


Here is a moderately amusing story that occurred today.  Some of you
may be aware of the neighborhood-wide wireless network that Personal
Telco Project built (and continues to build) on North Mississippi
Avenue.  This morning, I checked the cacti graphs for the network and
noticed downage.  It was strange.  It was strange because different
nodes disappeared at different times.  The "total associations" graph
pretty much lawn-darted at about 3:15 or 3:30 this morning.  It wasn't
like any of the typical outages we've seen, where someone unplugs
something, a flakey cat5 connector gets bumped, a neighborhood-wide
power outage, etc.  

Hmm, I thought.  I'll need to get over there ASAP to figure out what
is going on.  I have the only PTP key with access to the building
where we bring in the DSL.  I stew a little.

A little while later, I check my email and find a message from the
building manager.  He reports in three terse sentences, that: someone
got on the roof last night and *threw* our gear off onto the sidewalk;
the police arrested the guy (who was drunk); and could we please call
to help figure out how to fix it!

Oh.  Shit.

Our gear there consists of non-penetrating "tripods", which are held
down with cinderblocks, with a vertical mast about 4-feet tall, to
which we have attached an additional 6 foot mast (with some overlap),
to which is attached our radio gear and antennas.  Each of the radios
are connected to the building via a length of UV-rated cat5 (power is
supplied over the unused pairs of the cat5 [1]).

I call and get a few more details.  There are three tripods, all three
went over the side.

Building manager tells me two of the masts are damaged.  The other one
got held up by a tree and was dangling from the cat5 until morning
(this explains the non-simultaneous failures).  The perp had
(thankfully) removed the cinderblocks from the tripod before hurling
it over.  Of course, with the cinderblocks it weighed about 400 lbs.
The cinderblocks stayed on the roof.  The building manager (and,
later, Don Park) took some photos of the carnage [2].

So, I rustle up some helpers and head over.  Don and I arrive first.
I find two of the masts sitting inside partially recombobulated.  One
of the radio enclosures has some damage (broke off the polycarbonate
mounting bolts at one end, and a small crack at the top.  Just the
small crack penetrates to the weatherproof section.  Don starts
opening up the boxes with a screwdriver, while I run home to get some
power supplies to fire up the radios to see if they still work, and
some rope with which to haul the gear back up onto the roof.

It's about 1pm when I get back, Don has the boxes open.  In the
cracked-case box, the board is loose.  The four plastic standoffs
that hold the single-board computer inside either broke or pulled
out.  I find one that isn't damaged and hang the board from it.  One
of the mini-PCI radios is slightly ajar in its socket.  We reseat the
radio.  We apply power, to each of these two, and happily they still
function.  We identify which one goes where, and start hauling the
gear outside to where we can rope-lift it up the side of the building.

Meanwhile, Caleb Phillips and Michael Weinberg let me know they are on
their way to help.  Don and I get the masts up on the roof.  When
Caleb arrives, a little after 2pm, he and Don carefully pull up the
tripods. 

In the process we discover one of the sector antennas we are using for
local coverage was also damaged.  A couple-inch long hole was broken
in the polycarbonate radome.  It doesn't rattle, but peaking through
the opening it appears the antenna elements might be damaged.

I run off to get a spare replacement antenna from my house and a
replacement bolt from the hardware store for one that got munged in
the fall.  By the time I get back, they have finished determining that
the cat5 runs are still okay, but they've fried my cat5 tester in the
process.  Well, two leds of my cat5 tester.  The leds didn't like the
48V supplied by the PoE.  Oops.

The only remaining problem is disassembling and reassembling the
Amphenol IP67 weatherproof ethernet plugs [3] on a newly crimped tip.
There is a special tool to do the disassembly, but small screwdrivers
and similar devices work also, with persistance.

That solved, we plug in the devices again, a little electrical tape to
temporarily patch the radio enclosure crack, and bingo, back online.
Total downtime, about 13 hours.

The only remaining questions are:  

  a) Why did the perp do it?  Was it just random drunken exhuberance,
     and a crime of opportunity?  Or did he have something in for PTP?

and

  b) Can we recruit this guy?  He's obviously not scared of climbing
     on roofs.  PTP needs people like that, we just need to curb his
     caber tossing[4] impulses a bit.

oh, and maybe:

  c) is this what I get for asking questions of last night's PLUG
     speaker regarding backhoes (or equivalent) taking out his fiber
     runs?

Notes:

[1] <http://en.wikipedia.org/wiki/IEEE_802.3af>
[2] <http://www.personaltelco.net/gallery/ballroomvandal>
[3] <http://metrix.net/images/MET-RJ45-M-01.jpg>
[4] <http://en.wikipedia.org/wiki/Caber_toss>

-- 
Russell Senior, Secretary
russell at personaltelco.net



More information about the PLUG mailing list