Python in LC?

[danielrr]

Hi all,

Is there any way to use LiveCode to build an interface for python code? And if the answer is yes, how do you do it (a stack example would be much appreciated)?

PS Don't get me wrong, I love LC syntax (as everything else) but at least as of today it can be very slow (some times painfully slow) when it takes to deal with Unicode texts (specially large chunks of it)

[FourthWorld]

You can use the shell function to call anything that can be run from the command line, but I'm guessing that may be limited for what you're looking for.

Let's see if we can approach this more directly: do you have an example function written in Python with an equivalent one written in LiveCode where the LC one performs below expectations? There may be ways we can optimize that.

[danielrr]

Thanks,

Let's say you have a bilingual dictionary. It is Greek-English, or Chinese-English, so you need Unicode. You have it in three files, one for the Greek headwords, a second file for definitions and a third one (a really gigantic file) with the body of every article. Say you want to know how many words in Greek are translated by means of the English word "help". You look it up in the second file, see that you find "help" in lines 3678, 87634, 114536 etc for a total of 33 hits. Then you retrieve lines 3678, 87634, ..., 114536 etc. of files #1, #2 and #3 and you have all the Greek words translated by "languish" in your Greek or Chinese dictionary.

You can do it by brute force just looking up the lines in file #2 where you find the word "help" and then reading the full file #1 and #3 and retrieve the same lines, but this can be complicated due to memory constraints.

Theoretically at least you can speed up things if you count the number of characters of every line in every file, and you can go directly to line 3678 at char 456789 of file #1 and char 3456876 of file #3, etc.

Problem is: with LC 7#, 8# if you try to do something like this on a unicode file

Code: Select all

read from file nfile at nChars until return

you'll get a spectacular crash (reported). (There's no "endOfLine" constant in LC, I think.)

You can try to outsmart LC with something like

Code: Select all

read from file nfile at nChars for 30000
return line 1 of it

but you`ll crash again. Strangely, you can get away with:

Code: Select all

read from file nfile at nChars until EOF
return line 1 of it

But this is when things start going really slow, most likely because you are putting a lot of text in memory at every search.

With Python you can read a very large Unicode line by line extremely fast and return the line you are searching in almost no time. I hope I made myself clear.

[FourthWorld]

It would be helpful if you could provide the complete LC handler as you have it, the complete Python function you're comparing it to so we can gauge relative performance, and the data files so we can be assured that the revisions we make to your handler will account for their specific structure.

[shaosean]

There's no "endOfLine" constant in LC, I think.

EOF

[SparkOut]

End of line? Or file?

[danielrr]

Unless I'm wrong, in LiveCode, there is an EOF (end of file) constant, but not an EOL (end of line).

[shaosean]

CR, LF, CRLF which ever your file format uses..

[danielrr]

It would be helpful if you could provide the complete LC handler as you have it, the complete Python function you're comparing it to so we can gauge relative performance, and the data files so we can be assured that the revisions we make to your handler will account for their specific structure.

I have been trying to upload the stack, the files and the python script, but the compressed file is a 56MB file and the server doesn't allow me to upload such whopper to the forum, or so it seems. If you give me an address, I can send you a link to download the file

[FourthWorld]

For benchmarking that's probably more data than we need. You can post the LiveCode and Python handlers here, and post a link to Dropbox or other common store for a representative sample of the data (ideally < 5 MB).

[danielrr]

Sorry for this long delay (It certainly wasn't due to lack of interest).

I have been trying to upload a simplified version of the stack together with a reduced version of all the files (just the first 100 lines of every file--more than 140,000 lines each), but even so the whole bunch is almost 3MB, which means 1MB more than the space allowed by the server.

The reason the stack is so heavy is this: to speed searches I built a table for every file, containing the incremental number of chars of every line. if line 1 is 2 chars long (plus the return code) and line 2 is just 3 chars long then the file starts with

0
3
7
etc

With this tables (each one kept in its field), if you just want line 120,000 you pick up line 120,000 of the corresponding field, read as many characters from the file as specified in that line, then read just one line and you're done. All that info placed on the stack make it a heavy weight)

As a last solution, I have uploaded the file with zatauploader, so than you can download it from here:
https://www.zeta-uploader.com/es/dl-h?hash=1708332459

Sorry for the inconvenience. I hope the stack and the files are enough for you to see my point: how slow is to read big unicode files in LC7

[FourthWorld]

As a last solution, I have uploaded the file with zatauploader, so than you can download it from here:
https://www.zeta-uploader.com/es/dl-h?hash=1708332459

Unfortunately Chrome chokes on that URL with this error:

Duplicate headers received from server

ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION

The response from the server contained duplicate headers. This problem is generally the result of a misconfigured website or proxy. Only the website or proxy administrator can fix this issue.

[danielrr]

Try this, please:

http://s000.tinyupload.com/?file_id=311 ... 1160261995

[FourthWorld]

Thanks for the new link. I only took a quick look at the code, and while I won't be able to optimize those handlers myself right away I would encourage you to review this thread for optimization opportunities:
http://forums.livecode.com/viewtopic.php?f=8&t=24945

It's a long one, but well worth the read, touching on many aspects of optimizing LiveCode routines. In that thread a long process which had originally taken more than 9 minutes was reduced to under 3 milliseconds.

Your script won't be able to become quite that fast because there are things you need to do that are more computationally intensive than in the routine described in that thread.

Two tips to get you started: you may be able to use LiveCodes XML external instead of your scripted handlers for parsing XML, and while I believe your use of a field to strip HTML tags is pretty good I would be mindful if there are other places where you're using fields for just storing info, as field access is much slower than accessing variables.

Take a gander at that thread, and let us know if you're able to improve performance. Hopefully others will have an opportunity to look at the code and data you've provided to offer more specific recommendations to help move that along.