Multi-track playback synchronization

Visuals, audio, animation. Blended, not stirred. If LiveCode is part of your rich media production toolbox, this is the forum for you.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
pklum
Posts: 3
Joined: Fri Jan 17, 2025 6:47 pm

Multi-track playback synchronization

Post by pklum » Sat Jan 18, 2025 3:43 pm

Hello,
this is my first post ever on a forum, so bear with me ...

A little about me!
I consider myself quite knowledgeable in HyperTalk (started back in 1993). During 2000-2010 programming in HyperTalk was down (cheated a bit in Flash/ActionScript during that time) before I discovered Livecode in 2011. Since then have been faithful to Livecode.

I am currently developing a desktop application (macOS/Win) for musicians where you can play music (mp3) synchronized with lyrics and chords, called BandAid. I plan to release the application on the Mac App Store and Microsoft Store this summer 2025.

In BandAid I use Spleeter (search on GitHub) using "open process" and CLI calls to Spleeter to create separate stems from an mp3 file. I then want to be able to play the stems in sync and mix these together in BandAid.

The problem is that I have a hard time getting the stems to play in sync with each other.
As an example, I have six audio files (*.mp3) loaded into six hidden players:
  • audio_0_player loaded with the file "original" - contains the original mp3 not Spleeted. (reference audio track)
  • audio_1_player loaded with the file "drums" - contains the drum stem.
  • audio_2_player loaded with the file "other" - contains guitar/keyboard stem
  • audio_3_player loaded with the file "piano" - contains the piano stem.
  • audio_4_player loaded with the file "bass" - contains the bass stem.
  • audio_5_player loaded with the file "vocals" - contains vocal stem.
All files have the same duration and timeScale.

If I start the player one after the other, the previously started player widget will continue playing until the time it takes to execute the start command for the next player widget - why they are not synchronized in time. This is something that I find difficult to control.
I have tried a variety of ways to get these players to play synchronized with "send in time", "wait 0 with messages", "lock messages, ...

The latest and most successful version is to use audio_0_player (muted to avoid slapback echo) as a reference and create callbacks in audio_0_player to start the other players after each other. To separate the start of the different player, I move currentTime for each player before start timeScale/10 intervals after each player.

global gState["PlayerList"] contains a list of all player with loaded files with their short names.
global gState["MainTimescalePlayer"] contains the timeScale of the files.
global gState["PlayRate"] contains the current set playrate.
The command Players_Stop stops all player widgets

The command "Player_Start" starts the sequence.

Code: Select all

-- Start player(s)
-----------------------------------------------------------------------------------------
command Player_Start
   Players_Align
   wait 10 ticks with messages -- Flush messages
   Player_Start_Players
end Player_Start 
The "Players_Align" moves currentTIme of each player and defines callbacks in in audio_0_player when they should start.

Code: Select all

-- Move currentTime of each player {1:5} and create player {1:5} start callbacks in audio_0_player
-----------------------------------------------------------------------------------------
command Players_Align
   Players_Stop -- Stops all players
   set the callbacks of player "audio_0_player" of me to empty
   if the number of lines in gState["PlayerList"] > 1 then
      -- Multi-track
      put the currentTime of player "audio_0_player" of me into tRef
      put gState["MainTimescalePlayer"]/10 into tIntervalDelay
      put empty into tCallbackList
      repeat with tPlayerNum = 2 to number of lines in gState["PlayerList"]
      -- First line is audio_0_player
         if line tPlayerNum of gState["PlayerList"] is empty then next repeat
         put (tRef + tIntervalDelay*(tPlayerNum-1)) into tStartInterval 
         put line tPlayerNum of gState["PlayerList"] into tPlayer
         set the currentTime of player tPlayer of me to tStartInterval
         put tStartInterval, "Player_Start_Player" &  (tPlayerNum-1) into line (tPlayerNum-1) of tCallbackList
      end repeat
      set the callbacks of player "audio_0_player" to tCallbackList
   end if
end Players_Align
The "Player_Start_Players" start audio_0_player. audio_0_player will send messages to start the other players according to tCallbackList in the audio_0_player.

Code: Select all

-- Start Reference player audio_0_player
-----------------------------------------------------------------------------------------
command Player_Start_Players
   set the playrate of player "audio_0_player" of me to gState["PlayRate"]
end Player_Start_Players

-- Start audio_1_player from audio_0_player callback
-----------------------------------------------------------------------------------------
command Player_Start_Player1
   set the playrate of player "audio_1_player" of me to gState["PlayRate"]
end Player_Start_Player1

-- Start audio_2_player from audio_0_player callback
-----------------------------------------------------------------------------------------
command Player_Start_Player2
   set the playrate of player "audio_2_player" of me to gState["PlayRate"]
end Player_Start_Player2

-- Start audio_3_player from audio_0_player callback
-----------------------------------------------------------------------------------------
command Player_Start_Player3
   set the playrate of player "audio_3_player" of me to gState["PlayRate"]
end Player_Start_Player3

-- Start audio_4_player from audio_0_player callback
-----------------------------------------------------------------------------------------
command Player_Start_Player4
   set the playrate of player "audio_4_player" of me to gState["PlayRate"]
end Player_Start_Player4

-- Start audio_5_player from audio_0_player callback
-----------------------------------------------------------------------------------------
command Player_Start_Player5
   set the playrate of player "audio_5_player" of me to gState["PlayRate"]
end Player_Start_Player5
I have also put the code below in the audio_{1:5}_player's, which I think improves the sync a bit:

Code: Select all

on playerStarted
	set the currentTime of me to the currentTime of player "audio_0_player"
end playerStarted
Despite this, I can sometimes hear that the players are not playing in sync.

Now my questions!
  • Does anyone have an idea how to sync player widgets?
  • I have not find a method to measure the synchronization between the players (must use my ears!). Any idea how to measure this?
  • Is mp3 an inappropriate format for synchronizing playback?
I hope someone else has tried and successfully synced players or has an idea how to improve the sync.

A completely different question!
I use . (period) as a delimiter in my handler names, but had to change it to _ (underscore) as the Forum editor thought I was creating external links. Is there any way around it? Strangely enough, it was possible to write (*.mp3) above!?!?


Regards,
pklum


I develop (today) on macOS Sequoia 15_2, Mac Mini M1, Livecode 10_0_1 (rc 3). I also check that the code works under Windows 10

richmond62
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 9864
Joined: Fri Feb 19, 2010 10:17 am

Re: Multi-track playback synchronization

Post by richmond62 » Sat Jan 18, 2025 4:06 pm

Probably because LiveCode is not multithreaded.

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7358
Joined: Sat Apr 08, 2006 8:31 pm
Contact:

Re: Multi-track playback synchronization

Post by jacque » Sun Jan 19, 2025 6:48 pm

See the "prepare" command in the dictionary. It preloads videos into memory so that they will play instantly on demand. I'm not sure if this will complely solve the problem but it should avoid most of the delay when starting playback. While sound files are not exactly videos, it should work with any file played in a player control.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

pklum
Posts: 3
Joined: Fri Jan 17, 2025 6:47 pm

Re: Multi-track playback synchronization

Post by pklum » Sun Jan 19, 2025 10:44 pm

Hi Jacque,
Isn't the prepare command when you use video/audio clips with the play command? I am using Player objects and the fileName parameter.
How should I use the prepare command using a player object?

One could guess that alwaysBuffer parameter in the Player object could be the same as prepare. But the documentation for alwaysBuffer states that "If a player's movie contains only sound with no visual track, the setting of its alwaysBuffer property has no effect.".

Nevertheless, I tried and turned on alwaysBuffer on all my 6 players but then the callbacks in audio_0_player would not fire. If I turned alwaysBuffer off on audio_0_player and used alwaysBuffer on the other players (multi-track players), they played but i could still hear them out of sync as before. Going back to not use alwaysBuffer!

I guess each Player lives in its own thread (to be able to process the amount of data it handles)? In Player documentation the messages socketClose, socketError and socketTimeout are mentioned. Could these be used for synchronization, ie. NTP ? Haven't found any examples of this.

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7358
Joined: Sat Apr 08, 2006 8:31 pm
Contact:

Re: Multi-track playback synchronization

Post by jacque » Mon Jan 20, 2025 2:50 am

It's been a while since I did this but I think you're right. Sorry about that. I wonder if setting the start time to, say, 1 would preload the player. It's something to try anyway.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

pklum
Posts: 3
Joined: Fri Jan 17, 2025 6:47 pm

Re: Multi-track playback synchronization

Post by pklum » Mon Jan 20, 2025 10:09 pm

Hi Jacque,
Thanks for your suggestion! I tried your suggestion but unfortunately it did not improve synchronization.
Are there any possibility to prioritize a message?

Post Reply

Return to “Multimedia”