Player Object and AlwaysBuffer on = massive delays in script execution

Visuals, audio, animation. Blended, not stirred. If LiveCode is part of your rich media production toolbox, this is the forum for you.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

okk
Posts: 176
Joined: Wed Feb 04, 2015 11:37 am

Player Object and AlwaysBuffer on = massive delays in script execution

Post by okk » Sun Feb 23, 2020 1:24 pm

Hi,
I was looking again at the player object in the latest LC version. There had been issues earlier on with AlwaysBuffer and the player object, I wanted to see if there are any improvements. I tested with LC 9.5. and also the latest 9.6. DP2 on MacOS 10.12.6. Here is what I did:
I placed a player object on a stack, linked a video to the player, the video is not even Full HD resolution, I set AlwaysBuffer to true for the player object.

I created a script that is supposed to print every 50 milliseconds a number on top of the player object for 50 times. The number actually indicates how long it took to execute the script for each iteration.

Code: Select all

global counter, tcstart, tcstartabsolut

on mouseUp
   set the visible of player "test" to the hilite of button "play"
   set the paused of player "test" to not the hilite of button "play"
   put empty into field "info"
   put empty into field "result"
   put the milliseconds into tcstart
   put tcstart into tcstartabsolut
   put 0 into counter
   send printtext to me in 50 milliseconds
end mouseUp

on printtext
   put counter + 50 into counter
   
   if counter < 2500 then
      put the milliseconds into tcend
      put tcend-tcstart & CR after field "info"
      put tcend into tcstart
      send printtext to me in 50 milliseconds
   else
      put the milliseconds into tcend
      put "ideal: 2500 ms / real: " & tcend-tcstartabsolut & " ms" into field "result"
      set the paused of player "test" to true
   end if
   
end printtext

My expectation would be that the total execution time for the script would be a bit over 50*50=2500 milliseconds. In reality it is almost 10 x longer or even more! It takes even up to a second to execute one iteration of the 50 ms loop. You can try it yourself with the attached stack. The video file I used can be downloaded from here: http://speechkaraoke.org/files/benigni_1280x800.mov Can you repeate my test results? Is there any rational explanation? Any hint is appreciated. Thanks. Oliver

PS: this issue is not just of cursory interest, our project depends on the ability to print text on top of videos in order to function. My last working version I run is on 6.5.1. on an ancient macbook with an ancient MacOS. People buy new macs and our software wont run on the latest machines anymore.
Attachments
playbacktest.zip
(2.75 KiB) Downloaded 309 times

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by FourthWorld » Sun Feb 23, 2020 5:42 pm

I wonder if setting the acceleratedRendering of the stack to true would help, perhaps with also setting the player object's layerMode to dynamic.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7214
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by jacque » Sun Feb 23, 2020 11:04 pm

I see a few things that might speed up the process but it probably won't be perfect. Any handler will take some time to run, but you might be able to reduce the amount of time in this case a bit.

Field access is one of the slowest things LC does. That would be the primary issue I think, but you can't avoid that since the display is what you are aiming for. But as a test, you could comment out the line that updates the field text, which will give you the actual timing of the rest of the handler. I also suspect that 50 milliseconds isn't long enough for the handler to complete, so if it's possible to extend that time between updates you might have better results.

Another thing to do is to quote the word "printtext". This line:

Code: Select all

send printtext to me in 50 milliseconds
requires LC to determine whether "printtext" is a variable or a literal by scanning the script to see if it can find a variable of that name. You can avoid that ovehead by using quotes around the command. The time saving will be smallish but since the command occurs every 50 ms it can make a difference overall:

Code: Select all

send "printtext" to me in 50 milliseconds
When you put text into a field, LC needs to extract the existing text, make changes and put the text back. You could avoid half the processing by keeping a copy of the field text in a variable, altering it there, and replacing the entire field text all at once. That way LC doesn't have to retrieve and process the existing text, it just needs to wipe out the whole thing and replace it. I'm not sure how much difference this will make but it's worth a try. Changing text in a variable is way faster than changing text in a field.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by bogs » Sun Feb 23, 2020 11:07 pm

Would lock / unlock screen help in relation to updating the text at all? Just curious, I don't really do much with the player object even in the versions I run.
Image

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7214
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by jacque » Mon Feb 24, 2020 2:21 am

bogs wrote:
Sun Feb 23, 2020 11:07 pm
Would lock / unlock screen help in relation to updating the text at all? Just curious, I don't really do much with the player object even in the versions I run.
It usually does. I just wasn't sure how that would affect player playback.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by bogs » Mon Feb 24, 2020 11:03 am

Heh, I was actually going to test this out myself out of idle curiosity, but as usual, even the simplest things in Lc don't work at all on 'nix.

What I found out was (9.01 to 9.6 DP) :
1. on 'nix, you can't drag the player object from the tools palette to the stack. While any other tool in the palette drags over (if it isn't, apparently, a widget), you have to double click the player object to get one on the stack. I wasted out about 5 minutes figuring that one out. Go figure.

2. on 'nix, the player object apparently won't play (at least, not with the stack as posted, did not take the time to investigate very thoroughly).

3. the rest of the code as posted works as expected, returning a real time of 2537 ms, which is not far off the 2500 ms expected. Unfortunately, since nothing was happening in the video player, I suspect the whole excersise was a waste of time here.

Sorry Oliver.

Just so it won't be a complete waste, I will also test on a Windows Vm later, not enough time left this morning.
Image

okk
Posts: 176
Joined: Wed Feb 04, 2015 11:37 am

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by okk » Mon Feb 24, 2020 12:57 pm

Hi and thanks for the replies. Thanks bogs for making all the effort to replicate the test on your machine. I tested the different suggestions that were put forward but only with minimal improvements. I went further and simplified the script. I do not print anything on top of the player during the loop. I let the video run (with AlwaysBuffer on) while looping through my simple script in the background.

Code: Select all

global counter, tcstart

on mouseUp
   set the paused of player "test" to false
   put the milliseconds into tcstart
   put 0 into counter
   send "printtext" to me in 50 milliseconds
end mouseUp

on printtext
   put counter + 50 into counter
   if counter < 2500 then
      send "printtext" to me in 50 milliseconds
   else
      put the milliseconds into tcend
      put "ideal: 2500 ms / real: " & tcend-tcstart & " ms" into field "result"
      set the paused of player "test" to true
   end if
end printtext
Even without printing anything on top of the player object I get a total runtime of about 7 to 8 seconds. That equals 160 milliseconds per loop instead of 50ms or 6 frames per second. The issue is also that execution is very inconistent, it could take 1 second to excuted the loop sometimes, sometimes it is way faster. A 50 millisecond gameloop is not particular ambitious, for our project we at least need reliably 15 fps. What could possible be going on here? Thanks for your help. It is appreciated. Oliver

PS: this worked fine last time in LC 6.5.2, after that the issue wit playerobject/alwaysbuffer became persistent.

okk
Posts: 176
Joined: Wed Feb 04, 2015 11:37 am

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by okk » Mon Feb 24, 2020 1:22 pm

Hi,
just tried the same as above but with a FullHD video (1920x1080), the execution time of my script, that should be in total not much more than 2500 ms went up to a whopping 357646 milliseconds execution time, so about 6 minutes, or about 7 seconds per loop. The playback effectively blocks the execution of scripts in the background.

Best
Oliver

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by FourthWorld » Mon Feb 24, 2020 4:24 pm

If you can point us to other apps that render text over video we might be able to brainstorm some solutions.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

okk
Posts: 176
Joined: Wed Feb 04, 2015 11:37 am

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by okk » Mon Feb 24, 2020 5:47 pm

Hi Richard, I guess VLC could be a start. I tried to overlay a subtitle file with 20 captions that last 50 ms each, it seems to be no problem for VLC to display.

But as I mentioned in the second test, the playback even affects the execution of background scripts when nothing is printed on top of the player. That is somehow worrying. I assume this is related to one of the older bugs: https://quality.livecode.com/show_bug.cgi?id=12811 and https://quality.livecode.com/show_bug.cgi?id=19444 and perhaps https://quality.livecode.com/show_bug.cgi?id=15488

Thanks for looking into this.

Best
Oliver

bn
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3990
Joined: Sun Jan 07, 2007 9:12 pm
Location: Bochum, Germany

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by bn » Tue Feb 25, 2020 1:03 am

Hi Oliver,

have you considered using callbacks? I tested your stack and using 100 milliseconds intervals and adapting the "firing" time (taking into account the elapsed time) it kind of worked.

But it worked a lot better using callbacks. In the stak I set the callbacks to fire every second and put the internet date into a field that is positioned on top of the player. It actually worked quite well and if I remember correctly you have some kind of karaoke in mind. Maybe second intervals are not to bad. I tested with a video file that was not too demanding as a proof of principle. I did not test with FullHD video (1920x1080).

You might have to save the stack after setting the callbacks for it to work.

Kind regards
Bernd
Attachments
testCallbacks okk.livecode.zip
(1.79 KiB) Downloaded 332 times

okk
Posts: 176
Joined: Wed Feb 04, 2015 11:37 am

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by okk » Tue Feb 25, 2020 4:13 pm

Hi Bernd,
thanks a lot for your test stack. I can confirm that call-backs work better than the "gameloop" approach. Nevertheless the best I could achieve with a Full HD movie was about 200 milliseconds, comparing to about 5 fps. Still the execution time of your "BOING" script varied considerably, I could not get it run consistently on the time as specified in the callback property. If I tried to set callbacks within shorter timespan, like every 50 ms, the execution would delay more and more over time, so after 300 callbacks I had a dfelay of 7 seconds in total.

You remember correctly, we have a project that is karaoke for speeches. The issue is that we animate the coloring of the text (layered on top of a video) based on the speed and rythm of the original speech. Since many speeches are rather fast, we need at least 10 fps (reliably) to have a usable effect.

I for now will not use alwaysbuffer and animate the text on a separate area of the screen, separted from the video. That is a pity, since it worked very well in in the good old quicktime days.

Still, my concern is valid, please use my stack with a FullHD video and you will see how massive the script execution is delayed. This is worrysome, I use livecode to make different kind of interactive video installations, for that I need reliable and precise timed execution of code in relation to video.

Thanks!
Oliver

bn
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3990
Joined: Sun Jan 07, 2007 9:12 pm
Location: Bochum, Germany

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by bn » Tue Feb 25, 2020 5:27 pm

okk wrote:
Tue Feb 25, 2020 4:13 pm
Still, my concern is valid, please use my stack with a FullHD video and you will see how massive the script execution is delayed. This is worrysome, I use livecode to make different kind of interactive video installations, for that I need reliable and precise timed execution of code in relation to video.
I also noted the slowdown of the player once the dimensions of the video go up. That gave me the idea of using callbacks. It works well with a 1920 by 1080 video down to one callback per second. Beyond that the callbacks stack up and fire after the movie stopped. Maybe one could get it down a bit but 10 callbacks per second is too much for the player object.

Another idea would be to play the video in a browser widget. But one would have to add javascript functions to get the callbacks from the browser widget. And I have no idea how to script such a beast.
Hermann on this forum does amazing things using a browser widget and javascript.

Kind regards
Bernd

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by FourthWorld » Tue Feb 25, 2020 6:24 pm

okk wrote:
Tue Feb 25, 2020 4:13 pm
You remember correctly, we have a project that is karaoke for speeches.
This is encouraging. If I understand that correctly, the text elements need to be timed with specific points in sync with the video, and that timing does not change - is that correct?

The core issue here is that this once worked acceptably well and currently does not. It would be great if Mark Waddingham or other engineer could chime in here to discuss the feasibility of restoring that performance with the current rendering model.

But to be honest, I'm surprised it ever worked acceptably well, given the layers of work LC needs to do for something like this vs an app like VLC.

VLC handles decompression, rendering, and buffering internally to itself, so it has the luxury of hooking into any part of that pipeline most suited for adding other elements like text.

In contrast, LC hands off all of that to whatever video engine is supplied by the OS. When alwaysBuffer is off (the default), the OS playback engine renders directly into the window, which is why it's super-efficient but doesn't let us have any LC objects on top of it. When alwaysBuffer is true, the OS playback engine renders into an offscreen buffer, where LC then composits its own rendering of its own objects, and then transfers that composited rendering to the visible screen - with every frame. Ouch.

One way to think of this problem is in terms of moving the text rendering as close to the innards of the playback engine as possible. If we could put the text directly into the video, the OS renderer would be handling this in compiled machine code at the most advantageous point.

Fortunately, many multimedia formats support text tracks, and a couple XML-based formats exist as playable containers for multiple tracks which can include text. SMIL, for example, has a wonderful implementation of text tracks, and while I haven't looked into this for newer formats I believe most provide a means of adding them, many dynamically.

So if you can find the simplest way to put the text into the file being played - perhaps even dynamically if needed using one of the XML-based formats - you'd get OS-level rendering efficiency, and could play the video in LC very well, since you'd no longer need to set the alwaysBuffer to true.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

okk
Posts: 176
Joined: Wed Feb 04, 2015 11:37 am

Re: Player Object and AlwaysBuffer on = massive delays in script execution

Post by okk » Tue Feb 25, 2020 10:58 pm

Hi Bernd and Richard, thanks a lot for both of your thoughts and help. The browser widget might be a possible direction, but I am not very fluent with javascript and wouldnt know how to do this. Thanks for Richard for the explanation about alwaysbuffer. The tip with emebdding the text as track in the mediacontainer was something we looked into in the very beginning, but we couldnt find any solution for our usecase, it was precesily the lack of solutions that made us to code it ourselves in LC. Normal subtitles have a timecodestart and end for the a whole line of text, in our case we time-code each word. A word is colored based on the start and end timecode for that word which again is based on the original speed of the speaker, which can be sometimes quite fast.

In any case, I fiddled around a bit more and things got more curious. Here is what I tried: I have a stack with just a player object. AlwaysBuffer is OFF. I created a second stack, placed it on top of the first stack, set the blendlevel to 50, so I have a composite image. On this second stack I put my LC controls, my text field etc, I have my game loop on the card of this stack. I start and stop the player located on the first stack from a button on this second stack. Now, everything runs super smooth, the video runs perfectly in the background, my loop is executed every 10 milliseconds, the text is printed almost every 10 ms. I scratch my head here: why is this possible when layering two stacks on top, but not possible in one stack with two layers? Thanks! Oliver

Post Reply

Return to “Multimedia”