[PLUG] Multiple Internet Links...

[Martin A. Brown]

Hello there,

 : > I'm set up via two gateways to route a lan onto the Internet via a DSL
 : > connection.  I'm adding Comcast temporarily to get a feel for the service
 : > and possibly be able to order a month or so of service in case of DSL
 : > outages or need.  I want to route through Comcast on demand.  To effect this
 : > change, I'm thinking of adding a physical nic on each gateway that will
 : > connect to the cable modem.
 : >
 : > First question, how do I set the default gateway properly?  Second question,
 : > how do I set the name servers I want, not Comcast's?
 : 
 : This is actually a *really* hard thing to do right, I fear.  
 : Linux doesn't do "link health" detection or anything, so if you 
 : have a gateway on the dead device listed in the routing table it 
 : will get used even if a perfectly good second path exists.

Strictly speaking, Linux can do link-layer gateway health detection, 
but that still falls short of link-health detection (in the sense of 
checking the up/down status of a physical/logical Internet link).

If you are using multipath routes and one of your gateways is dead, 
Linux is clever enough not to use that route.  On the other hand, 
if the nexthop beyond one of your gateways is dead, you are out of 
luck.

So, practically speaking, Daniel is correct.

 : So, you actually need something on your router to sit there, keep 
 : an eye on the state of the connection, and update the routing 
 : table.

Yes--this is the usual solution, where somebody writes a little 
utility to initiate a ping to a known, expected-responding 
destination out a particular interface, and then takes some action 
if the destination stops responding and then another action once the 
destination resumes responding.  Fragile, but usually viable.

 : > Do I really need to add another nic or can I alias a second address on an
 : > existing nic and use free ports on my DSL connected ethernet switch for the
 : > cable modem?
 : 
 : You should be able to configure all the devices into the same 
 : internal range, and use the gateway server(s) to manage that 
 : appropriately.  You certainly can run different network ranges as 
 : well, though that can sometimes require additional work to make 
 : routing do the right thing.

Yes, certainly, this is probably most easily accomplished using a 
privately addressed network (RFC 1918) for your internal network 
and, employing NAT (that omnipresent small evil).

 : Anyway, the basic list of things to do looks something like this:
 : 
 : 1. Hook up both Internet routers so they can talk (somehow) to the gateways.
 : 2. Configure the gateways so they can talk through those routers to the
 :    Internet successfully.
 :    - you may need to adjust the preferred source address on the link
 :    - you may need to do some source routing
 : 3. Get both upstreams working when you route through the gateways
 : 4. Configure something to monitor the two Internet services and verify they
 :    are working correctly
 : 5. Have whatever monitors the service bring up and down the relevant routes
 :    when the service is healthy or dead.
 :    - since you want preferential routing, just assign the appropriate metric
 :      to them as default routes and it will all just work(tm) routing-wise.
 : 
 : 
 : With regards the question of using the right DNS servers: turn 
 : off the DHCP services on all the Internet router devices (cable 
 : and DSL), I would suggest, and then run DNS and DHCP service on 
 : your gateways.
 : 
 : That way you can supply your local resolver to client machines, 
 : configure whatever forwarders you want, and be happily in control 
 : of the whole thing.

N.B.  Let's assume that Michael Robinson has two providers, OpusNet 
      and Comcast.  If packets with a source address of OpusNet end 
      up hitting Comcast resolvers, Comcast has no obligation to
      return replies.  Also, either provider could (quite 
      legitimately) drop packets arriving from customer equipment
      that purport to be from the other provider's network.  This
      sort of filtering is often called BCP38 [0].

In short, if you would like to be using resolvers which could work 
on both providers, then you may wish to consider an open resolver 
(sometimes called public resolvers).  There are a variety to choose 
from:  OpenDNS, Google, DynDNS, Scrubit, and the old GTEI (now 
Level3) IPs.  There are quite a few others I didn't list.  The point 
is that you can use one of these if you want, or run your own 
resolver.

 : All of which should be reasonable straight-forward, but much of 
 : it will not exactly be simple.

I would certainly agree that running a network with multiple uplinsk 
is not exactly simple.  Getting everything running correctly can be 
a bit of a hassle.

OK, so I wrote this quite some time ago and haven't revisited it in 
several years, however, the kernel still supports all of the same 
functionality, and bulk of this is a description of the tools and 
behaviour of Linux when used in the fashion you are trying to use 
it.  It should still be useful:

  http://linux-ip.net/html/adv-multi-internet.html

Points to be aware of as you undertake this effort:

  * Yes, you can use a single NIC and put multiple IPs on that NIC, 
    however, this is more complicated than using a dedicated NIC for 
    each interface (1 for internal, 1 for provider A and 1 for 
    provider B).

  * If using fewer than 3 interfaces, be aware of ARP issues [1].

  * If you do not already know them, learn these commands well:
    tcpdump, ip route, ip rule, arp.

  * Learn how to use ping to send packets from an IP or interface of
    your choice.  This is good for testing the routing on your box
    which is performing the fancy routing.

  * If you are using multiple routing tables [1] (and you will 
    probably need to), DO NOT forget to account for the return 
    packets.  Routing the packet out of your internal network
    is the first hurdle.  Routing the returning packet is a separate 
    issue.

Anyway, good luck as you embark on this little adventure,

-Martin

 [0] http://www.faqs.org/rfcs/bcp/bcp38.html
 [1] http://linux-ip.net/html/ether-arp.html#ether-arp-flux
 [2] Yes, Linux supports multiple routing tables since the 2.2 
     series of kernels.

-- 
Martin A. Brown
http://linux-ip.net/