[PLUG] Multi host init system?
Martin A. Brown
martin at linux-ip.net
Fri Jul 10 06:54:46 UTC 2015
Hello there,
> I have an app that is distributed across a dozen servers.
>
> There are several processes involved, some with dependencies on
> processes running on other servers.
> What app would you recommend for starting the whole thing up in an
> orderly manner?
Is it possible to adjust the pieces of software so that there is no
required 'orderly' startup?
I ask because--if the application requires synchronized startup of
services across multiple machines, then what happens when one of the
services (or nodes) early in that dependency chain fails during
operation?
For example, let's imagine services A through I, each of which must
be launched before the subsequent can launch:
A -> B -> C -> D -> E -> F -> G -> H -> I
Assuming normal, orderly, coordinated startup, great. Now,
everything is running.
Suppose that service C fails.
What happens?
Will the application still run?
Do D through I need to be restarted (or just D)?
If it is possible to adjust the individual services so that each of
them can run and retry, fail gracefully, or even fail hard (as fast
as possible, please) to contend with dependency issues, I would
recommend that.
Perhaps you have already addressed that question or are in the
(unenviable) position of contending with feature-complete software
that is ready for deployment.
Since you are in the 10+ node realm, I think I'd also agree with
using some sort of configuration management (somebody suggested
Ansible). With this many nodes, it's an operational truism that one
of them will kick the bucket during your dog's midnight birthday
party [0] and you'll want to be able to move the service quickly to
another node.
Hurrah for the well-worn configuration management tools.
This is the modern take on startup script dependencies, just now
with more network in-between! Everybody needs more network
in-between! Not an easy problem.
Anyway, good luck with this conundrum!
-Martin
[0] Silicon devices sense these moments and cherish destroying our
equanimity.
--
Martin A. Brown
http://linux-ip.net/
More information about the PLUG
mailing list