using multi cores

[adventuresofgreg]

Hi: I am currently running about 20 instances of a LC standalone. Each SA imports a pile of data, does a bunch of number crunching, then repeats this every minute. With all 20 instances running, each instance of the SA will simultaneously open it's data file, and crunch it's numbers - with EVERYTHING done in a total of about 5 to 10 seconds.

I've been trying to write a version of this app that will do all of this from ONE app. But, my script won't open and process the 20 data files simultaneously, it crunches them one at a time, sequentially, and this takes (5 seconds * 20 = 100 minutes) to complete one epoch.

Is there a way to run stuff simultaneously that takes advantage of multiple processor cores?

thanks!
Greg

[FourthWorld]

Why not spawn 20 standalones? LC doesn't take up much memory, and if you make those command-line apps they'll be even more lean.

[adventuresofgreg]

Hi Richard - that is exactly what i have been thinking. I'm trying to learn more about the command line apps - but can't find much info. Is this a "open process for read" ? I've played with that and can't seem to read from the SA applet. Do I need to open a socket?

Cheers,
Greg

[FourthWorld]

This thread may help, though the sample code may be Mac-specific:
http://forums.runrev.com/viewtopic.php?f=9&t=15059

I've been toying with the idea of making a Hadoop-like library for LC, where we could break up tasks and distribute them among multiple machines for even faster parallel processing. But at the moment I don't actually have any task big enough to require that, so it's been a far back-burner project for me.

For the work I do, even parsing up 1GB text files, I've found the biggest gains from optimizing the parsing code itself. Of course there are limits to how much something can be optimized, and if you need parallel processing you need parallel processing. But any parallelization, whether via threads or separate processes or separate machines, will add its own overhead in coordinating them, so the net benefit is rarely as high as we might hope.

Can you post the processing code to see if there are any opportunities for optimization?

Also, when you wrote "(5 seconds * 20 = 100 minutes) ", did you mean "5 minutes" or "100 seconds"?

[adventuresofgreg]

Hi Richard: I meant 100 seconds, not minutes.

I agree 100% with you. I know that my code can probably be optimized a lot. The problem is that it is the result of a few years of adding functions, changing functions, etc, etc. And now, it probably needs to be totally re-written from scratch with an eye toward making it as efficient as possible. How is your time these days? want a job?

Greg

[townsend]

Very intriguing topic!

So... the big question... does livecode have some way for one app (exe) communicate with another?

Visual Basic had something called DDE: dynamic data exchange, specifically for this purpose.

[FourthWorld]

I agree 100% with you. I know that my code can probably be optimized a lot. The problem is that it is the result of a few years of adding functions, changing functions, etc, etc. And now, it probably needs to be totally re-written from scratch with an eye toward making it as efficient as possible. How is your time these days? want a job?

If you can post the relevant code here you can probably get a lot of eyeballs to optimize it for free. My own schedule's a bit hectic lately, but sometimes I can squeeze in a Saturday afternoon here and there - feel free to write me at ambassador AT fourthworld.com if you like.

[FourthWorld]

So... the big question... does livecode have some way for one app (exe) communicate with another?

Visual Basic had something called DDE: dynamic data exchange, specifically for this purpose.

DDE is nice for Windows, but unsupported everywhere else.

Most OSes (certainly all the ones LC runs on) support sockets, and for desktop apps LC's socket implementation is fairly easy to use.

There are other ways to exchange data and instructions between apps too. Macs can use AppleScript (though it's of course exclusive to Mac and not very efficient), and sometimes even simply polling for a file can be a good option.

[adventuresofgreg]

re: exchanging data bwtn LC SA apps, I often just write and read text files. It actually is pretty fast, and very easy to implement.

[FourthWorld]

re: exchanging data bwtn LC SA apps, I often just write and read text files. It actually is pretty fast, and very easy to implement.

While sockets can be faster for smaller amounts of data, polling for files can work surprisingly well: since most modern OSes cache directory contents, much of the disk-touching activity with polling happens in RAM.

[adventuresofgreg]

Richard I sent you an email. Did you get?

[FourthWorld]

I did, thanks. I'll be replying shortly.