[SOLVED] Advice/Tutorials/Guides for Asynchronous Process Polling

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
speedbump1981
Posts: 14
Joined: Fri Aug 08, 2014 7:24 am

[SOLVED] Advice/Tutorials/Guides for Asynchronous Process Polling

Post by speedbump1981 » Tue Dec 11, 2018 10:06 pm

I'm looking for Advice/Guides/Tutorials on how to implement an asynchronous process polling mechanism so I can monitor and react to the output of multiple python processes. I searched a few LC resources and I'm striking out. Any suggestions? :)

My stack utilizes youtube-dl to populate a Plex "YouTube Videos" library (i.e, my "subscriptions") on a daily basis. My current implementation works just fine (code below), but several channels have a few thousand videos that blocks the UI for hours while youtube-dl checks each and every video within the channel (something I'm still working on fixing).

Process command: (modified for forum posting: removed youtube url)

Code: Select all

youtube-dl --write-info-json --write-thumbnail -v -i --prefer-free-formats --add-metadata -r 3.0M -f bestvideo[height<=720]+bestaudio/best[height<=720] -o "/home/speedbump/Desktop/YouTube Library/%(uploader)s/%(title)s.%(ext)s" --dateafter now-1week /channel/UCGhjwFyTUfA9W5m02JEbdlQ
Current LC execution:

Code: Select all

   put cd fld "command" into tCommand
   open process tCommand for text update
   read from process tCommand at start until end
   put cr& "[channel_surfer]" && it after cd fld "result"
   close process tCommand
My goal is to be able to launch 1-4 youtube-dl processes, in a non-blocking manor, and to be able to act upon those process as they output new data (progress, video/channel info, errors and warning, etc). The desired result is the ability to download newly released videos from my YouTube Subscriptions in just a few hours as opposed to most of the day (my first run time was 12 hours :shock:).

My head spins out when I try to figure out how to break down the async process and build it. I'd love some help on this. Thanks in advance! :)
Last edited by speedbump1981 on Thu Dec 13, 2018 10:24 pm, edited 1 time in total.
Derek Bump | Oshkosh, WI

Dreamscape Software (Closed)
JPEGCompress, "Side-scrolling Game Test"

Factory Direct Shippers, LLC (Retired)
Online Shopping Cart & Customer Information Manager

speedbump1981
Posts: 14
Joined: Fri Aug 08, 2014 7:24 am

[SOLVED] Re: Advice/Tutorials/Guides for Asynchronous Process Polling

Post by speedbump1981 » Thu Dec 13, 2018 10:24 pm

I was able to come up with a solution after hammering my way through the LC documentation. I setup a library stack called Process Monitor with a console log and a mechanism for displaying open process IDs. I plan to develop it out into a more versatile process monitor but it runs primarily like this:

To initiate a process: dispatchProcess { string | object data }

Code: Select all

command dispatchProcess tCommand
   global _OpenProcesses
   local tProcessID
   
   ### Display the Command in the Console
   put cr& "[channel_surfer]" && tCommand after cd fld "output" of stack "processMonitor"
   
   ### Generate an ID for this process
   put random(99999) into tProcessID
   put tCommand into _OpenProcesses[tProcessID]["command"]
   put 0 into _OpenProcesses[tProcessID]["bytes"]
   put cr& "[channel_surfer] launching process id:" && tProcessID after cd fld "output" of stack "processMonitor"
   set the text of cd btn "switcher" of stack "processMonitor" to (the keys of _OpenProcesses)
   
   ### Launch Process
   open process tCommand for text read
   if it is not empty then put cr& "[channel_surfer] it:" && it after cd fld "output" of stack "processMonitor"
   if the result is not empty then put cr& "[channel_surfer] the result:" && the result after cd fld "output" of stack "processMonitor"
   put cr after cd fld "output" of stack "processMonitor" # Bug Fix: next read from process will need to start on a blank line
   set the vScroll of cd fld "output" of stack "processMonitor" to (the formattedHeight of cd fld "output" of stack "processMonitor")
   
   ### Start Monitoring
   send "processMonitor" to stack "processMonitor" in 10 milliseconds
end dispatchProcess
The dispatcher queues up the Process Monitor and begins polling any processes still in the _OpenProcesses array. This seems to allow for process threads, resulting in a 30-50% reduction in processing time (depending on computer hardware):

Code: Select all

command processMonitor
   global _OpenProcesses
   local opID, opData
   
   ### Check each open process for new data and re-queue the processMonitor
   repeat for each line opID in the keys of _OpenProcesses
      
      ### Read from the byte counter and update the UI
      read from process _OpenProcesses[opID]["command"] at _OpenProcesses[opID]["bytes"] until linefeed
      
      ### Process EOF
      if the result is "eof" then
         put cr& "[channel_surfer] Closing process" && opID & ": eof" after cd fld "output" of stack "processMonitor"
         close process _OpenProcesses[opID]["command"]
         delete global _OpenProcesses[opID]
         next repeat
      end if
      
      ### Process Closed
      if the result is "process is not open for read" then
         put cr& "[channel_surfer] Closing process" && opID & ": process is not open for read" after cd fld "output" of stack "processMonitor"
         close process _OpenProcesses[opID]["command"]
         delete global _OpenProcesses[opID]
         next repeat
      end if
      
      if the result is not empty then put cr& "[channel_surfer] result:" && the result after cd fld "output" of stack "processMonitor"
      put it into opData
      if opData is not empty then add (the number of chars in opData) to _OpenProcesses[opID]["bytes"]
      put opData after cd fld "output" of stack "processMonitor"
      set the vScroll of cd fld "output" of stack "processMonitor" to (the formattedHeight of cd fld "output" of stack "processMonitor")
      set the text of cd btn "switcher" of stack "processMonitor" to (the keys of _OpenProcesses)
      
   end repeat
   delete local opData
   
   ### One final cleanup, just in case there are no processes open
   set the vScroll of cd fld "output" of stack "processMonitor" to (the formattedHeight of cd fld "output" of stack "processMonitor")
   set the text of cd btn "switcher" of stack "processMonitor" to (the keys of _OpenProcesses)
   
   
   ### Queue the Process Monitor
   if the keys of _OpenProcesses is not empty then send "processMonitor" to me in 50 milliseconds
   
end processMonitor
This seems to be giving me what I was looking for. A lot more refinement is necessary, but I'm now able to launch concurrent processes in a non-blocking manor within LiveCode, all while slurping up the output in an actionable manor.
Derek Bump | Oshkosh, WI

Dreamscape Software (Closed)
JPEGCompress, "Side-scrolling Game Test"

Factory Direct Shippers, LLC (Retired)
Online Shopping Cart & Customer Information Manager

Mikey
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 755
Joined: Fri Jun 27, 2008 9:00 pm

Re: [SOLVED] Advice/Tutorials/Guides for Asynchronous Process Polling

Post by Mikey » Sat Dec 15, 2018 5:40 pm

Oh that's sexy.

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”