How to feed data to standalone stack via 'write to process' issued by other standalone?

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
Tonaras
Posts: 18
Joined: Mon Jul 03, 2017 1:25 am

How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by Tonaras » Tue Sep 27, 2022 9:23 am

Hello fellow LiveCoders,

I have two standalone stacks, "MainApp".exe and "SubApp".exe. Think of the "SubApp".exe standalone as of an independend command line app. I'm trying to pass data from "MainApp".exe directry to "SubApp".exe via
- write myData to process "/SubApp . exe"

How does the "SubApp".exe standalone receive the 'myData' that is 'written' to it by the "MainApp".exe and how does it even know when data is sent to it that way??? I have no clue whatsoever.

I don't want to go through the pain of exchanging data via writing to and reading from files. Also, I cannot use shell() with standalone stacks on Windows, because data written to stdout don't get through correctly through the console.

Any help is HIGHLY appreciated!!!

PS.: For anyone wondering why I want to accomplish this: LC is really terribly slow on string processing. I have large XML files with up to 100k+ records / 80 columns wide that need to be parsed and read into a SQLite database. The revXML is quite fast as it is written in C++. But the parsing and converting in LCS is a total nightmare, despite all handler optimisations that I have attempted. I still wonder how it can be that LibreOffice opens and parses the same ultra heavy XML file within seconds, whereas LC needs 12-15 minutes to parse the same data. Running some benchmarks in LC and checking the Windows Task Manager, I noticed that LC barely exceeds 13-15% of CPU usage, even during the most heavy string processing. I wonder why this is so painfully slow and why LC doesn't expoit all the processing power of the CPU.

And, since LC is a non-multithreading snail-o-saurus, and I cannot wait for ever for the upcoming LC Compiler, I came up with the idea of splitting the task into smaller packets and "outsourcing" the task to several external processes (aka SubApp Apps) that run in parallel. I have no idea if that makes sense and how stable that will be in the end, but it's my last hope to tackle the task in LiveCode in a timely manner.

I've even tried to get my hands dirty with LCB, but I'was surprized to discover that this extravaganza is crawling even slower than LCS.

What are your guy's thoughts?

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by FourthWorld » Tue Sep 27, 2022 11:21 am

For inter-app comms the same classic options apply: file and sockets.

But I'm curious to see the parsing script. I'll bet we can get that down to acceptable levels in a single process.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Tonaras
Posts: 18
Joined: Mon Jul 03, 2017 1:25 am

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by Tonaras » Tue Sep 27, 2022 12:08 pm

Hi Richard, and thanks a ton for jumping in so promptly, I'm really grateful!!

There are several handlers involved actually.

The first is for extracting the data headers from XML nodes to be able to match input columns to output columns. Once the user has matched the i/o,
the second handler is called, which is for parsing the XML recursively and converting all relevant nodes into an array, based on fabulous code by Trevor, my all time LC super hero.
The third is a "router" that basically separates that array according to the header column destinations of the data set (part 1 going to the products table, part 2 to the media table and part 3 to the specifications table). The result are 3 new arrays.
The fourth handler assembles a list of SQL insert queries based on these 3 arrays (1 query per array key).
And the fifth finally executes the queries.

Beyond the speed problems of LC, the major headache I have is that the XML data structure that the end users will feed in is not constant and not known to me beforehand. So there's a lot of stuff that can go wrong. I thought that parsing crappy CSVs was a nightmare until I discovered the hell of XML's data structure diversity.

I wouldn't mind sharing the code if you're interested to dig in. It would be a honor for me!!

I'm so desperate with LC's lame and unacceptable string processing performance that I'm seriously considering to hire someone proficient in C++ to write an extension for this thing. It cannot be that bloody LibreOffice does all the above in a matter of seconds and LC needs 15-25 minutes to crunch the data...
Last edited by Tonaras on Thu Sep 29, 2022 5:22 am, edited 2 times in total.

stam
Posts: 2635
Joined: Sun Jun 04, 2006 9:39 pm
Location: London, UK

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by stam » Tue Sep 27, 2022 12:13 pm

As Richard says, many have posted their algorithms on this forum and have had their code optimised from something that takes 10+ minutes to mere seconds. Definitely worth posting that here.

I know other IDEs like XOJO have promised 'helper apps' that are meant to do exactly what you describe, but i gather that doesn't work so well...

I'd suggest looking and seeing if code can be optimised before going into sockets, paying someone to write an external etc.

Perhaps also include a sample xml as well?

S.

Tonaras
Posts: 18
Joined: Mon Jul 03, 2017 1:25 am

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by Tonaras » Tue Sep 27, 2022 12:31 pm

Hi Stam! Great to have you here! And reliefing too not to be alone!!

I'll prep a stack with the basic handlers and post it here along with a sample of these fat overweight XML files asap.

I've certainly banned anything in my code that even nearly smells like "repeat with..." or "put x+1 into x". How much more room for code optimizing is left there, I wonder.

No so sure about sockets as I've go no experience with these beasts yet. The idea of splitting long repeat loops and "outsourcing" them to multiple instances of helper apps that run in parallel is still highly intriguing though.

stam
Posts: 2635
Joined: Sun Jun 04, 2006 9:39 pm
Location: London, UK

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by stam » Tue Sep 27, 2022 12:45 pm

It sounds like you may have tried this already, but I came across this from Trevor DeVore:

https://gist.github.com/trevordevore/5584753

The essence of this gist is a set of methods to convert xml to a multidimensional LC array and back.

Have you tried this and if so, what is useful? (it is 10 years old, so may not work as advertised I suppose...)

S.

Tonaras
Posts: 18
Joined: Mon Jul 03, 2017 1:25 am

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by Tonaras » Tue Sep 27, 2022 1:03 pm

It sounds like you may have tried this already, but I came across this from Trevor DeVore:
Yes, THAT'S the ONE!!! I've just made some minor addaptations mainly for the detection of the XML's UTF encoding/decoding. The only problem I have with the ConvertXMLNodeToArray function is when a parent node in the XML contains multiple child nodes with the same child node name (it then get's stuck in an never ending loop). Otherwise it is brilliantly and elegantly coded like everything Trevor has shared so generously over the years.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by FourthWorld » Tue Sep 27, 2022 7:08 pm

Tonaras wrote:
Tue Sep 27, 2022 12:08 pm
I wouldn't mind sharing the code if you're interested to dig in. It would be a honor for me!!
Chances are it would be a mutually rewarding adventure. Some of the best threads in these forums have been where someone posts slow code, a lot of folks chip in to make it faster, it gets much faster, then Dick Kreisel jumps in and makes it faster still. :)

No guarantees of course. You may have already hit all the opportunities for performance enhancement. But worth trying.

Please use the Code tags when posting so things are more readable. Thanks.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Tonaras
Posts: 18
Joined: Mon Jul 03, 2017 1:25 am

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by Tonaras » Thu Sep 29, 2022 5:16 am

Hello Richard, so cool to have you arround mate!
Chances are it would be a mutually rewarding adventure. Some of the best threads in these forums have been where someone posts slow code, a lot of folks chip in to make it faster, it gets much faster, then Dick Kreisel jumps in and makes it faster still. :)
Yeah, that's exactly how I see it. And I can only confirm what you say, I've too learned A GREAT LOT from these threads here. So I think it's valuable for the entire community.

I'm already on my way, preparing a stack with all required elements, cleaning up and documenting code as much as I can so that it can be more easily understood. Stay tuned :wink:

Tonaras
Posts: 18
Joined: Mon Jul 03, 2017 1:25 am

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by Tonaras » Sat Oct 08, 2022 12:36 pm

Hello Richard, Stam and all,

this is just a quick update to let you know that I've not forgotten this thread and to appologise that I haven't yet posted the code.
Since about a week I'm going through my 2nd burnout this year because of overworking, mostly coding more that 24-48h in one go. So, everything that even remotely looks like a mouse, a compy or LC is causing me distress. I'll certainly post the code once I kinda recover a bit.
Please rest assured that I greatly appreciate all the tips, support and reccomendations that you all have provided to the community and to me over the years.

Warmest regards,
Anton

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: How to feed data to standalone stack via 'write to process' issued by other standalone?

Post by FourthWorld » Sat Oct 08, 2022 5:13 pm

Tonaras wrote:
Sat Oct 08, 2022 12:36 pm
Hello Richard, Stam and all,

this is just a quick update to let you know that I've not forgotten this thread and to appologise that I haven't yet posted the code.
Since about a week I'm going through my 2nd burnout this year because of overworking...
No worries. We'll be here. The important thing is the you will be too. Taking time for breaks is very important. Life is an endurance sport, not a sprint. :)

Be well. Enjoy some long walks among trees or a bicycle adventure. Whenever you're rested and have resumed your zeal for optimization, we can dive in then.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”