[PLUG] Multi host init system?

[Michael Rasmussen]

On Thu, Jul 09, 2015 at 11:54:46PM -0700, Martin A. Brown wrote:
> 
> Hello there,
> 
> > I have an app that is distributed across a dozen servers.
> >
> > There are several processes involved, some with dependencies on 
> > processes running on other servers.
> 
> > What app would you recommend for starting the whole thing up in an 
> > orderly manner?
> 
> Is it possible to adjust the pieces of software so that there is no 
> required 'orderly' startup?
> 
> I ask because--if the application requires synchronized startup of 
> services across multiple machines, then what happens when one of the 
> services (or nodes) early in that dependency chain fails during 
> operation?
> 
> For example, let's imagine services A through I, each of which must 
> be launched before the subsequent can launch:
> 
>    A -> B -> C -> D -> E -> F -> G -> H -> I
> 
> Assuming normal, orderly, coordinated startup, great.  Now, 
> everything is running.
> 
> Suppose that service C fails.
>    What happens?

Given your chain, each node consumes data from the following one. 
Node I generates the raw data. The system is designed on a pull 
model. If C fails B and A think there's no work to do - and eventually 
generate alarms because they expect there always should be something 
to do.  D will write to local storage. When C is brought back it will 
process the queued data on D. 

Aside from getting disk arrays, databases and message queuing services started
there are no actual hard dependencies. The desire to control and coordinate 
startups is threefold.

 - ensure health of component before going forward
 - not populate logs with messages about lack of services
 - not generate a backlog of data to be processed when the full system is up.


-- 
      Michael Rasmussen, Portland Oregon  
    Be Appropriate && Follow Your Curiosity
Complaining is silly. Either act or forget.
    ~ Sagmeister