[PLUG] LAN broadcast?

[Bill Barry]

On Wed, Jun 1, 2011 at 9:13 PM, Wayne E. Van Loon Sr. <wevl at pacifier.com> wrote:
> I have a question about user applications using broadcast on a LAN.
>
> I usually have several processes that connect via TCP/IP to a central
> server that solves some optimization problems. In more demanding project
> that I am working on now, I hope to distribute the optimization problem
> between 24 to 36 helper processes.
>
> At the start of each optimization job, I will have a few data files that
> need to be distributed to each of the helper optimization processes.
> Rather than have the central optimizer / server send these files 24
> times, some way to broadcast these files to all machines / processes at
> the same time might be nice.
>
> Is that the kind of thing that broadcasting could accomplish?
>
> And if so, and in a big picture way, how would one do it?
>
> An example of my network topology would be:
> TCP/IP server @ 192.168.1.40
> 4 helper processes @ 192.168.1.40
> 4 helper processes @ 192.168.1.41
> 4 helper processes @ 192.168.1.42
> 4 helper processes @ 192.168.1.43
> 4 helper processes @ 192.168.1.44
> 4 helper processes @ 192.168.1.45
>
> Thanks for looking.
> Wayne


This project looks like it might help  http://en.wikipedia.org/wiki/GridFTP
I have never used it, but apt-cache search shows there are Debian
packages for it.

The real question to ask I think is where is your bottleneck in the
computation.  What should you optimize first. If you are going to
distribute the computing,  are you transferring large amounts of data
to the nodes or just small datasets. Do the nodes take a long time to
do the computation compared to the data transfer? If so, maybe you can
make more progress optimizing the computation than the data transfer.

Bill