Params to Python external script?

Got a LiveCode personal license? Are you a beginner, hobbyist or educator that's new to LiveCode? This forum is the place to go for help getting started. Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Params to Python external script?

Post by Zax » Wed Jul 07, 2021 3:12 pm

Hello,

As I have a long calculation to perform, I would like to try if a python script would be faster but I don't know how python and LC communicate.

Let's say i have the following LC script:

Code: Select all

on mouseUp
   local tMinParam = 1
   local tMaxParam = 1000
   local tResult
   repeat with x = tMinParam to tMaxParam
        ... do something and modify tResult
   end repeat
   answer tResult
end mouseUp
How can I pass the loop job to a python script with tMinParam, tMaxParam, and retreive tResult?

(in fact, I don't anything to Python, but a friend of mine can write Python script for me ;) )

Thank you.

Bernard
Posts: 351
Joined: Sat Apr 08, 2006 10:14 pm
Location: London, England

Re: Params to Python external script?

Post by Bernard » Wed Jul 07, 2021 7:15 pm

Code: Select all

python -c "import sys; print 'hi',sys.argv[1], sys.argv[2], 'bye'" abc 'de f'
returns

hi abc de f bye

I'm sure your friend will want to pass the name of a .py script rather than -c switch.

So

Code: Select all

put shell("python myfile.py " && tMinParam && tMaxParam) into tResult
should work to pass your variables into the python script, and the result of shell() will be what the python script returns (assuming that the python executable is installed on a path automatically found by your command terminal).

This also assumes that your tMinParam and tMaxParam are not strings containing characters that will confuse the python interpreter.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Params to Python external script?

Post by FourthWorld » Wed Jul 07, 2021 7:47 pm

If you can post the Python script it would allow us to translate it to see if it's actually faster.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Re: Params to Python external script?

Post by Zax » Thu Jul 08, 2021 9:42 am

Thank you.

My LC script is about large string (imageData). If the Python version is faster, I'll post the two versions.

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Re: Params to Python external script?

Post by Zax » Thu Jul 08, 2021 10:15 am

Bernard wrote:
Wed Jul 07, 2021 7:15 pm
This also assumes that your tMinParam and tMaxParam are not strings containing characters that will confuse the python interpreter.
Oops, I missed that part... So I will not be able to send an imageData to the Python script :(

EDIT : so, I could I send to Python a file name? Maybe with some quotes?

EDIT2: strings don't seem to be a problem, maybe as long they don(t contain quote?

Code: Select all

put "hello" into p1
   put "World" into p2
   put shell("python " & pythonScripFile && p1 && p2) into tResult
   put tResult
This works.

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Re: Params to Python external script?

Post by Zax » Thu Jul 08, 2021 2:28 pm

I now realise that the inital idea wasn't good, as it's rather stupid to pass an imageData as parameters.
Thanks all for your replies.
Anyway, I've learned how to call Python and I will use it in other projects.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Params to Python external script?

Post by FourthWorld » Thu Jul 08, 2021 10:18 pm

If you're in a position to post both scripts it would be very interesting to see the differences in structure and performance.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Bernard
Posts: 351
Joined: Sat Apr 08, 2006 10:14 pm
Location: London, England

Re: Params to Python external script?

Post by Bernard » Fri Jul 09, 2021 12:01 am

Zax wrote:
Thu Jul 08, 2021 2:28 pm
I now realise that the inital idea wasn't good, as it's rather stupid to pass an imageData as parameters.
Thanks all for your replies.
Anyway, I've learned how to call Python and I will use it in other projects.
You could use some conversion from binary to a text format in livecode and pass that encoded imageData, then in python re-convert to binary data and process the image.

Or you could save the imagedata to a file, then pass that filename to the python script, where python would read the binary data from that named file.

Or, as Richard suggests, explain what the problem is you're trying to speed up by using python. Could be that Livecode has got a way of doing it that will be fast enough for your purposes.

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Re: Params to Python external script?

Post by Zax » Fri Jul 09, 2021 9:55 am

My purpose is to turn an image to greyscale, and set transparency on bright grey values.
I suppose the best way to do that would be to use some graphic libs ane send shell commands, but I don't want to use external libs.
Bernard wrote:
Fri Jul 09, 2021 12:01 am
Or you could save the imagedata to a file, then pass that filename to the python script, where python would read the binary data from that named file.
Don't you think reading and writing large files will be longer?

For informations, this is my LC (9.04) code. Sorry, it's not very easy to read, as I tried to optimize it (and it's french commented ;) )
Capture d’écran 2021-07-09 à 10.53.37.jpg

Code: Select all

local arrParams

on preProcess
   if ("button" is in the target) then set hilite of the target to true
   set cursor to watch
   ------- reset PixEnd (delete image pour full reset si l'alphaData est corrompu)
   if (there is an image arrParams["destName"]) then delete image arrParams["destName"]
   create image arrParams["destName"]
   set threeD of image arrParams["destName"] to (the threeD of image arrParams["sourceName"])
   set borderWidth of image arrParams["destName"] to (the borderWidth of image arrParams["sourceName"])
   set showBorder of image arrParams["destName"] to (the showBorder of image arrParams["sourceName"])
   ------- visuel
   set the rect of image arrParams["destName"] to (the rect of image arrParams["sourceName"])
   set the topLeft of image arrParams["destName"] to \
         (the left of image arrParams["sourceName"] & "," & the bottom of image arrParams["sourceName"] + 6)
   -------
   put the milliseconds into fld "Timer" -- /////
end preProcess

on getParams
   put "Pix" into arrParams["sourceName"]
   put "PixEnd" into arrParams["destName"]
   put the hilite of btn "ChkInvert" into arrParams["invert"] -- négatif
   get the thumbPos of scrollbar "SliderLighten" -- de 0 (aucun éclaircissement) à 10 (éclaircissement maxi)
   put round(it * 200 / 10) into arrParams["brightness"] -- de 0 à 200
   get the thumbPos of scrollbar "SliderContrast" -- de 0 (aucune modif) à 10 (contraste maxi)
   put it / 10 into arrParams["contrast"] -- de 0 à 1
   
   put 0.30 into arrParams["xRed"] -- augmenter pour éclaircir les rouges (avec total = 1)
   put 0.59 into arrParams["xGreen"] -- augmenter pour éclaircir les verts (avec total = 1)
   put 0.11 into arrParams["xBlue"] -- augmenter pour éclaircir les bleus (avec total = 1)
   
   put the width of image arrParams["sourceName"] into arrParams["w"]
   put the height of image arrParams["sourceName"] into arrParams["h"]
end getParams

on postProcess
   ------- visual effect
   show image "PixEnd"
   put (the milliseconds - fld "Timer") && "ms" into fld "Timer"
   if ("button" is in the target) then set hilite of the target to false
end postProcess

on mouseUp
   getParams
   preProcess
   
   turnToGrayTranspa arrParams["sourceName"], arrParams["destName"], arrParams["w"], arrParams["h"], arrParams["xRed"], arrParams["xGreen"], arrParams["xBlue"], arrParams["brightness"], arrParams["contrast"], arrParams["invert"] 
   
   postProcess
end mouseUp

on turnToGrayTranspa pixSourceName, pixDestName, imgWidth, imgHeight, xRed, xGreen, xBlue, brightness, fc, invertBWboolean
   put "" into log
   addToLog log, imgWidth && "x" && imgHeight && "pixels"
   addToLog log, "r = " & xRed && "  g = " & xGreen && "  b = " & xBlue
   addToLog log, "brightness = " & brightness
   
   ----------------------------------------------------- prépa : négatif
   if (invertBWboolean) then
      put 255 into grayCorrection
   else put 0 into grayCorrection
   addToLog log, "invertBWboolean = " & invertBWboolean && "  --> grayCorrection = " & grayCorrection
   
   ----------------------------------------------------- prépa : contraste
   put 0 - (fc^5 + 0.05) into fp5 -- pré-calcul
   put sqrt(fc) into racf -- pré-calcul
   addToLog log, "fc = " & fc
   
   ----------------------------------------------------- prépa : image et transparence
   put the imageData of image pixSourceName into tImage
   local aImage -- pour alphaData
   
   ----------------------------------------------------- 
   local tPixel, tImgPosition, tGray
   # repeat for all the pixels in the image
   repeat with tPixel = 0 to ((imgWidth * imgHeight) - 1)
      ----------------------------------------------------- extractions des composantes RGB
      # multiply the image pixel position by 4 (Alpha + Red + Green + Blue) 
      put tPixel * 4 into tImgPosition
      /* plus long
      # calculate the gray level
      put charToNum(char(tImgPosition + 2) of tImage) into tRed
      put charToNum(char(tImgPosition + 3) of tImage) into tGreen
      put charToNum(char(tImgPosition + 4) of tImage) into tBlue
      ----------------------------------------------------- gris
      # here you can use any grayscale formula
      # set the RGB of the pixel to the same value this gives us the gray level (0-255)
      --put (xRed * tRed) + (xGreen * tGreen) + (xBlue * tBlue) into tGray
      --------------------- application auto "négatif"
      put abs(grayCorrection - ((xRed * tRed) + (xGreen * tGreen) + (xBlue * tBlue))) into tGray
      */
      ------------ calcul valeurs gris et application auto "négatif"
      put abs(grayCorrection - ((xRed * charToNum(char(tImgPosition + 2) of tImage)) + \
            (xGreen * charToNum(char(tImgPosition + 3) of tImage)) + \
            (xBlue * charToNum(char(tImgPosition + 4) of tImage)))) into tGray
      
      ----------------------------------------------------- contraste
      if (fc > 0) then
         put floor(tGray * (1 - racf)) + racf * (255 / (1 + (0.9921875 * exp(fp5 * (tGray - 128))))) into tGray
         -- Rislon : (x * (1 - sqrt(fc))) + sqrt(fc) * (255 / (1 + (0.9921875 * exp(-(fc^5 + 0.05) * (x - 128)))))
         /* -- plus précis mais plus long
         if (fc >=  0.5) then
            put exp(-2 * (tGray - 128) * fc^4) into tmg
            put floor(128 + 127.5 * (1 - tmg) / ( 1 + tmg)) into tGray
         else
            put exp(-2 * (tGray - 128) * 0.5^4) into tmg
            get 128 + 127.5 * (1 - tmg) / ( 1 + tmg)
            put floor(tGray * (1 - 2 * fc) + 2 * fc * it) into tGray
         end if
         */
      end if
      
      ----------------------------------------------------- éclaircissement général
      put min(255, brightness + tGray) into tGray
      
      ----------------------------------------------------- application des nouvelles valeurs de gris
      put numToChar(tGray) into ntcGray
      put ntcGray & ntcGray & ntcGray into char (tImgPosition + 2) to (tImgPosition + 4) of tImage
      
      ----------------------------------------------------- transparence toutes valeurs (alphaData)
      --------- transparence proportionnelle à la valeur de gris : plus c'est clair -> plus c'est transparent
      /*
      binaryEncode("C",0) --> transparentPixel
      binaryEncode("C",255) --> opaquePixel
      */
      put binaryEncode("C", 255 - tGray) after aImage -- 0 = transparent et 255 : opaque
   end repeat
   
   -------------------------------------------------------- affectation image dest
   set the imageData of image pixDestName to tImage
   set the alphaData of img pixDestName to aImage
   if (there is a fld "Log") then put log into fld "Log" --/////
end turnToGrayTranspa

on addToLog @log, str
   put str & return after log
end addToLog

function func0 x, f -- fcn Rislon pour f >= 0.5 : UNUSED telle quelle
   return (128 + 127.5 * (1 - exp(-2 * (x - 128) * f^4)) / ( 1 + exp(-2 * (x - 128) * f^4)))
end func0

function func1 x, f -- fcn Rislon pour f < 0.5 : UNUSED telle quelle
   return (x * (1 - 2 * f) + 2 * f * func0(x, 0.5))
end func1

Bernard
Posts: 351
Joined: Sat Apr 08, 2006 10:14 pm
Location: London, England

Re: Params to Python external script?

Post by Bernard » Mon Jul 12, 2021 12:13 pm

Image processing isn't something I ever do. However, as no-one else has made any suggestions, I'll have a shot.

Can you use your logging to work out where this is slow? My suspicion is that it will be in your loop where you turnToGray by pixel.

If so then you may get an improvement by pulling out the 4 pixels on which you work and appending them to a new variable, rather than repeatedly addressing (groups of 4) pixels in the imagedata. Then set the imageData of your image to the data in the new variable.

Here's a high level example:

Code: Select all

on turnToGrayTranspa
   put the imagedata of img 1 into t1
   put zero into tPixCount
   repeat for each char tPix in t2 -- only makes one pass through imagedata
      add 1 to tPixCount
      if tPixCount = 4 then
         -- do pixel processing here
         -- append processed pixels to t2
         put zero into tPixCount
      end if
   end repeat
   set the imageData of image 1 to t2 -- ty2now contains your new imageData
end turnToGrayTranspa
Those who do image processing may be able to come up with something better.

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Re: Params to Python external script?

Post by Zax » Mon Jul 12, 2021 5:45 pm

Bernard wrote:
Mon Jul 12, 2021 12:13 pm
My suspicion is that it will be in your loop where you turnToGray by pixel.
You're right Bernard: a huge count of charToNum and numToChar take some time.
I tried the "after" method to fill an empty variable but found that in my case, process was longer.

Process is a little bit faster with a repeat... step 4.

Bernard
Posts: 351
Joined: Sat Apr 08, 2006 10:14 pm
Location: London, England

Re: Params to Python external script?

Post by Bernard » Mon Jul 12, 2021 6:15 pm

Can you give us an idea of how long things take?

Also I'm surprised that a single pass through the imageData takes longer than repeated scans of it looking for sets of pixels. Perhaps include your modified turnToGrayTranspa handler so others can see.

Another technique is memoization. Once you have done a calculation for 4 pixel values, you don't need to recalculate when you run into those 4 identical values again. You could use an array as a lookup table (the original values would be the lookup key, the calculated values the associated value of the array). Obviously this will give huge performance benefits on the image you included in your post, since it is bars of the same colour, but will offer less benefit with an image that has more random pixel data.

Finally, since your interest has changed from Python to LC performance, you might be better asking a new question specifically about your turnToGrayTranspa handler, and see if you can get the attention of someone who specialises in the area of image manipulation.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Params to Python external script?

Post by FourthWorld » Mon Jul 12, 2021 7:32 pm

If I ask a fourth time might I have a copy of the Python script that was originally used? If it has some sort of licensing restriction I'd understand, but I would love the opportunity to make stylistic and performance comparisons of a non-trivial algo between the two languages.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Zax
Posts: 431
Joined: Mon May 28, 2007 10:12 am
Location: France

Re: Params to Python external script?

Post by Zax » Tue Jul 13, 2021 7:34 am

Sorry Richard if I wasn't clear (me english is rather poor) : I don't have Python script. A soon I realised passing an imageData as parameter was a bad idea, I told my fried not to write a Python script.

@Bernard: process time depends of course of source image, and CPU.
For exemple, it takes 10 secondes for a 2500x600 pixels JPEG image with an old MacBook Pro.
Concerning memoization, the image shown on my screen capture was just an example to see gray values.

This is the actual code:

Code: Select all

local arrParams

on preProcess
   if ("button" is in the target) then set hilite of the target to true
   set cursor to watch
   ------- reset PixEnd (delete image pour full reset si l'alphaData est corrompu)
   if (there is an image arrParams["destName"]) then delete image arrParams["destName"]
   create image arrParams["destName"]
   set threeD of image arrParams["destName"] to (the threeD of image arrParams["sourceName"])
   set borderWidth of image arrParams["destName"] to (the borderWidth of image arrParams["sourceName"])
   set showBorder of image arrParams["destName"] to (the showBorder of image arrParams["sourceName"])
   ------- visuel
   set the rect of image arrParams["destName"] to (the rect of image arrParams["sourceName"])
   set the topLeft of image arrParams["destName"] to \
         (the left of image arrParams["sourceName"] & "," & the bottom of image arrParams["sourceName"] + 6)
   -------
   put the milliseconds into fld "Timer" -- /////
end preProcess

on getParams
   put "Pix" into arrParams["sourceName"]
   put "PixEnd" into arrParams["destName"]
   put the hilite of btn "ChkInvert" into arrParams["invert"] -- négatif
   get the thumbPos of scrollbar "SliderLighten" -- de 0 (aucun éclaircissement) à 10 (éclaircissement maxi)
   put round(it * 200 / 10) into arrParams["brightness"] -- de 0 à 200
   get the thumbPos of scrollbar "SliderContrast" -- de 0 (aucune modif) à 10 (contraste maxi)
   put it / 10 into arrParams["contrast"] -- de 0 à 1
   
   put 0.30 into arrParams["xRed"] -- augmenter pour éclaircir les rouges (avec total = 1)
   put 0.59 into arrParams["xGreen"] -- augmenter pour éclaircir les verts (avec total = 1)
   put 0.11 into arrParams["xBlue"] -- augmenter pour éclaircir les bleus (avec total = 1)
   
   put the width of image arrParams["sourceName"] into arrParams["w"]
   put the height of image arrParams["sourceName"] into arrParams["h"]
end getParams

on postProcess
   ------- visual effect
   show image "PixEnd"
   put (the milliseconds - fld "Timer") && "ms" into fld "Timer"
   if ("button" is in the target) then set hilite of the target to false
end postProcess

on mouseUp
   getParams
   preProcess
   
   turnToGrayTranspa arrParams["sourceName"], arrParams["destName"], arrParams["w"], arrParams["h"], arrParams["xRed"], arrParams["xGreen"], arrParams["xBlue"], arrParams["brightness"], arrParams["contrast"], arrParams["invert"] 
   
   postProcess
end mouseUp

on turnToGrayTranspa pixSourceName, pixDestName, imgWidth, imgHeight, xRed, xGreen, xBlue, brightness, fc, invertBWboolean
   put "" into log
   addToLog log, imgWidth && "x" && imgHeight && "pixels"
   addToLog log, "r = " & xRed && "  g = " & xGreen && "  b = " & xBlue
   addToLog log, "brightness = " & brightness
   
   ----------------------------------------------------- prépa : négatif
   if (invertBWboolean) then
      put 255 into grayCorrection
   else put 0 into grayCorrection
   addToLog log, "invertBWboolean = " & invertBWboolean && "  --> grayCorrection = " & grayCorrection
   
   ----------------------------------------------------- prépa : contraste
   put 0 - (fc^5 + 0.05) into fp5 -- pré-calcul
   put sqrt(fc) into racf -- pré-calcul
   addToLog log, "fc = " & fc
   
   ----------------------------------------------------- prépa : image et transparence
   put the imageData of image pixSourceName into tImage
   local aImage -- pour alphaData
   
   ----------------------------------------------------- 
   local tPixel, tImgPosition, tGray
   # repeat for all the pixels in the image
   repeat with tImgPosition = 0 to 4*((imgWidth * imgHeight) - 1) step 4
      ----------------------------------------------------- extractions des composantes RGB
      # multiply the image pixel position by 4 (Alpha + Red + Green + Blue)
      ------------ calcul valeurs gris et application auto "négatif"
      put abs(grayCorrection - ((xRed * charToNum(char(tImgPosition + 2) of tImage)) + \
            (xGreen * charToNum(char(tImgPosition + 3) of tImage)) + \
            (xBlue * charToNum(char(tImgPosition + 4) of tImage)))) into tGray
      
      ----------------------------------------------------- contraste
      if (fc > 0) then
         put (tGray * (1 - racf)) + racf * (255 / (1 + (0.9921875 * exp(fp5 * (tGray - 128))))) into tGray
      end if
      
      ----------------------------------------------------- éclaircissement général
      put min(255, brightness + tGray) into tGray
      
      ----------------------------------------------------- application des nouvelles valeurs de gris
      put numToChar(tGray) into ntcGray
      put ntcGray & ntcGray & ntcGray into char (tImgPosition + 2) to (tImgPosition + 4) of tImage
      
      ----------------------------------------------------- transparence toutes valeurs (alphaData)
      --------- transparence proportionnelle à la valeur de gris : plus c'est clair -> plus c'est transparent
      put binaryEncode("C", 255 - tGray) after aImage -- 0 = transparent et 255 : opaque
   end repeat
   
   -------------------------------------------------------- affectation image dest
   set the imageData of image pixDestName to tImage
   set the alphaData of img pixDestName to aImage
   if (there is a fld "Log") then put log into fld "Log" --/////
end turnToGrayTranspa

on addToLog @log, str
   put str & return after log
end addToLog

Bernard
Posts: 351
Joined: Sat Apr 08, 2006 10:14 pm
Location: London, England

Re: Params to Python external script?

Post by Bernard » Tue Jul 13, 2021 10:48 am

I've just done a test and it took < 4 seconds to loop through an imagedata that had 4 million chars in it using "repeat for each". I wasn't doing all the various calculations you are doing, I was just running charToNum() on each char. I'm testing speed of loop format rather than anything connected to imaging.

Using your loop format was taking so long that I stopped after 4 seconds, and it had only processed 75,000 chars. This is the kind of difference one would expect.

Can you show us the times for "repeat for each" and show us the handler using that loop format. Contrast this with your loop.

Memoization might still be of use in your situation. Also I'd suggest you try speed comparisons using the kind of image that you think you will eventually process.

Post Reply

Return to “Getting Started with LiveCode - Complete Beginners”