[PLUG] LAN broadcast?

Wayne E. Van Loon Sr. wevl at pacifier.com
Thu Jun 2 13:18:00 PDT 2011


Aaron:
Thanks for your reply.
Aaron Burt wrote:
> On Wed, Jun 01, 2011 at 09:13:03PM -0700, Wayne E. Van Loon Sr. wrote:
>   
>> At the start of each optimization job, I will have a few data files that 
>> need to be distributed to each of the helper optimization processes. 
>> Rather than have the central optimizer / server send these files 24 
>> times, some way to broadcast these files to all machines / processes at 
>> the same time might be nice.
>>     
>
> They do something like that on large compute clusters, using MTFTP, UFTP
> or UDPcast (I think- I've been out of the loop a while).  For a cluster as
> small as yours, I suspect it's not going to help much.  If the files are
> big enough for it to make a difference, they're probably too big for
> reliable multicast distribution.  Better to make sure you have a nice
> healthy gigabit backbone, and perhaps let the hosts pull their files over
> NFS rather than pushing them.
>
> Can you comment on the nature of the compute problem?
>   
The compute problem is testing various products such as posts, poles, 
blocks for a stud mill, etc., that can be removed (bucked) from a 
tree-length log (stem), and selecting the most profitable solution. The 
stem is scanned as it passes length-wise through 3D LASER scanners. 
Products are trial placed into the stem and evaluated to make sure that 
the product meets a number of criteria such as diameter, crookedness, 
taper, etc.

I have this software running very successfully in a sawmill with simpler 
conditions - shorter logs, fewer products, etc. That software running in 
a single process on a uniprocessor machine finds a solution in 1 - 2 
seconds.  We process somewhere around 10 stems / minute.

The conditions for the next application are longer logs, more products, 
etc. The number of solutions go into the millions and the testing / 
optimization process takes minutes. I need to solve several logs / minute.

I think I can divide the optimization process into zones and distribute 
the stem scan data, the current products desired, their values and 
requirement, etc. to multiple optimization helper processes and get the 
job done quickly enough. The scan data derived from each stem needs to 
be disseminated before any of the helper optimization processes can 
begin. The dissemination needs to begin immediately following imaging.

The dissemination of scan data and current product information is the 
task for which I was thinking about some sort of broadcast / multicast. 
But if multicast is any sort of hassle, or there is a latency to "pull" 
the data, it isn't worth it.

Wayne

> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>
>
>   




More information about the PLUG mailing list