is read from socket blocking?

[adventuresofgreg]

Hi: I would like to be able to read from more than 1 open port at the same time - is this possible? The reason is I am downloading data from another application through a socket connection. I would like to speed the process up by dividing the data I require into 2 parts, and downloading data from 2 open sockets simultaneously. I CAN open 2 sockets, and when I attempt to read from both of them at the same time, it works, but, only 1 data stream is 'collected' at any one time (the other waits until the first is complete).


write GetInstruction to socket lookupPort
read from socket lookupPort until "!END" with message "gotdata"
(when !END is reached, gotdata handler puts the data into MasterData[ActiveSymb] )
wait while MasterData[ActiveSymb] is empty with messages

Perhaps the problem is my "wait" command, but I need it because the script wont wait until the data from one request is finished before attempting to get the data from the second request - and you can't make more than 1 request PER open socket.

[FourthWorld]

Building Hadoop, Greg?

You could have two instances of LC running, each listening on a separate socket. While multiprocessing carries a bit more overhead than multithreading, we don't have direct support for the latter and the difference is usually not quite as dramatic as it may seem.

After all, if multiprocessing is good enough for Hadoop, it should be good enough for more modest needs.

[adventuresofgreg]

Hi Richard - I don't know much about Hadoop (sorry). But your idea of having 2 instances of LC running would probably work. Am I correct in assuming that you can't read from 2 open ports simultaneously?

[FourthWorld]

Am I correct in assuming that you can't read from 2 open ports simultaneously?

I haven't tried it so I can't say for sure, but other programs can listen on multiple sockets and you can open multiple sockets for read/write, so I can't imagine why you couldn't listen on multiple sockets.

AFAIK the only limitation is that any given socket can only have one listener at a time, so you should be covered.

However, having a single instance reading from two sockets won't be nearly as fast as using two different instances of your app.

[adventuresofgreg]

Thanks Richard. I'm running a second instance now, and each instance is listening on it's own socket and the data download is twice as fast. This works very well. I'm still not sure that downloading data from 2 sockets with a single instance is possible. I've tried a dozen different ways, and it seems that the reading from any open socket can't occur while another socket is being read from.

Any ideas on how the first instance of the running app could launch a second instance automatically, and automatically switch the port name?

... As for the different port names, I guess when I launch an instance, I can give the port name a random number ie: "127.0.0.0:20|foo265765745" This way, it would be unlikely that 2 ports would ever have the same name.

[FourthWorld]

You could pass the port as an argument on the command line, and the app being launched can get that with $1. If you need more args you can check $# to get the number of args, and walk through them with $1, $2, $3, etc.