[PLUG] LAN broadcast?

Wayne E. Van Loon Sr. wevl at pacifier.com
Thu Jun 2 12:18:11 PDT 2011


Bill:
Thanks for your reply.

Bill Barry wrote:
> On Wed, Jun 1, 2011 at 9:13 PM, Wayne E. Van Loon Sr. <wevl at pacifier.com> wrote:
>   
>> I have a question about user applications using broadcast on a LAN.
>>
>> I usually have several processes that connect via TCP/IP to a central
>> server that solves some optimization problems. In more demanding project
>> that I am working on now, I hope to distribute the optimization problem
>> between 24 to 36 helper processes.
>>
>> At the start of each optimization job, I will have a few data files that
>> need to be distributed to each of the helper optimization processes.
>> Rather than have the central optimizer / server send these files 24
>> times, some way to broadcast these files to all machines / processes at
>> the same time might be nice.
>>
>> Is that the kind of thing that broadcasting could accomplish?
>>
>> And if so, and in a big picture way, how would one do it?
>>
>> An example of my network topology would be:
>> TCP/IP server @ 192.168.1.40
>> 4 helper processes @ 192.168.1.40
>> 4 helper processes @ 192.168.1.41
>> 4 helper processes @ 192.168.1.42
>> 4 helper processes @ 192.168.1.43
>> 4 helper processes @ 192.168.1.44
>> 4 helper processes @ 192.168.1.45
>>
>> Thanks for looking.
>> Wayne
>>     
>
>
> This project looks like it might help  http://en.wikipedia.org/wiki/GridFTP
> I have never used it, but apt-cache search shows there are Debian
> packages for it.
>   
I see that GridFTP is on top of FTP which is a client / server paradigm 
that it uses TCP/IP.
Is GridFTP intended for many sources of data to one receiver?
Or many receivers from one source of data?

> The real question to ask I think is where is your bottleneck in the
> computation.
Unquestionably, the optimization.
>   What should you optimize first. If you are going to
> distribute the computing,  are you transferring large amounts of data
> to the nodes or just small datasets. Do the nodes take a long time to
> do the computation compared to the data transfer?
Yes.
>  If so, maybe you can
> make more progress optimizing the computation than the data transfer.
>   
For sure. That is how I came to distributing the optimization.

I have been exploring some sort of broadcast or multicast. If it wasn't 
difficult to implement and keep synchronized, I would consider it. Your 
question is insightful, the optimization is where the time is.

Again, thanks very much for your reply.

Wayne




More information about the PLUG mailing list