Performance and background jobs

Got a LiveCode personal license? Are you a beginner, hobbyist or educator that's new to LiveCode? This forum is the place to go for help getting started. Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller

Post Reply
ace16vitamine
Posts: 130
Joined: Fri Apr 13, 2018 1:53 pm

Performance and background jobs

Post by ace16vitamine » Thu Jan 31, 2019 11:45 am

Dear all,

I have a simple (and stupid) question.

My application is like this one:

Code: Select all

on open_sock
Transfer file bla bla
sql_select
end open_sock

on sql_select
select bla bla bla
calculate something
show_message
end sql_select

on show_message
answer bla bla bla
end show_message
On "sql_select" I am calculating thinks, this takes about 20-40 secs. During this time the application GUI Frontend seems to be frozen, thinks are not calculating in background and there is enough CPU Power left. sql_select is integrated to the frozen card, I think this is the problem.

What can I do that operations like this are running in background?

Thanks
Stefan

Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: Performance and background jobs

Post by Klaus » Thu Jan 31, 2019 12:11 pm

Hi Stefan,

unfortunately LC is single-threaded, so there is not much you can do about this.
Maybe you can "cut" your long handler into smaller units?
Maybe you can post the handler and we check if we could make it faster and/or see
where we can cut it into little pieces.


Best

Klaus

ace16vitamine
Posts: 130
Joined: Fri Apr 13, 2018 1:53 pm

Re: Performance and background jobs

Post by ace16vitamine » Thu Jan 31, 2019 2:52 pm

Hi Klaus (again),

this is some of the biggest block. I am merge some SQL outputs different times to create a laaaaarge variable.


I am starting with the Order IDs from t_array["vBestellnummer"] to get a list with all existing Order IDs. Then, in the first section, I am matching each line of s_array["vSendungsnummern"] and find the right ID which a matching to create a new variable (v_order).

The selects are always the same:

ID TAB TrackingID (or if paid, or name, surname etc)


I THINK the Problem is that it is very heavy to calculate this again and again... Maybe we can make this faster (and shorter).

Code: Select all


####### SECTION 1

 put t_array["vBestellnummer"] into tNames
 put s_array["vSendungsnummern"] into tIds

#Example  t_array["vBestellnummer"] 
#AU123
#AU456
#AU789

#Example s_array["vSendungsnummern"]
# AU123 A123
# AU456 B456

   set itemdel to TAB
   
   ## Loop through all NAME pairs
   repeat for each line tLine in tNames
      put item 1 of tLine & TAB into tItem
      put lineoffset(tItem,tIds) into tOffset
      
      ## No match, modify this line if you want to add an empty ITEM 
      ## to the current NAME line or whatever you want to do in that case
      
      if tOffset = 0 then
         put tab into item 2 of line tOffset of tIds
         put tLine & TAB & item 2 of line tOffset of tIds & CR after v_order
         
         next repeat
         
      else
         
         ## repeat for each is READ only, so we just create a new list form the resulting data:
         put tLine & TAB & item 2 of line tOffset of tIds & CR after v_order # [b]<- This is the Target for section 1[/b]
         
      end if
   end repeat

delete char -1 of v_order
   
######### RESULT:
# AU123 A123
# AU456 B456

######################################## Next Section: Match if DHL, DPD...
####### SECTION 2

#Example  t_array["vVersandart2"]
#AU123 DHL
#AU456 DPD
#AU789 GLS

#Example v_order (result from the 1st section)
# AU123 DHL123
# AU456 DHL456



put v_order into tNames
   put t_array["vVersandart2"] into tIds
   set itemdel to TAB
   
   ## Loop through all NAME pairs
   repeat for each line tLine in tNames
      put item 1 of tLine & TAB into tItem
      put lineoffset(tItem,tIds) into tOffset
      
      ## No match, modify this line if you want to add an empty ITEM 
      ## to the current NAME line or whatever you want to do in that case
      
      if tOffset = 0 then
         
         next repeat
         
      else
         
         ## repeat for each is READ only, so we just create a new list form the resulting data:
         put tLine & TAB & item 2 of line tOffset of tIds & CR after v_order2 # [b]<- This is the Target for section 2[/b]
         
         
         
         
         
      end if
   end repeat

   delete char -1 of v_order2
   
   ######### RESULT:
# AU123 A123 DHL
# AU456 B456 DPD


######################################## Next Section: Match if paid
####### SECTION 3


#Example  t_array["paid"]
#AU123 yes
#AU456 no
#AU789 no

#Example v_order2 (result from the 2st section)
# AU123 DHL123 A123
# AU456 DHL456 B456



#   (..) with v_order3, v_order4, v_order5, v_order6



Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: Performance and background jobs

Post by Klaus » Thu Jan 31, 2019 5:18 pm

Hm, OK, I would probably use some local variables or custom properties
and split the long handler into 6 smaller handlers -> v_order1 to v_order6.
And use SEND to call them from the previous handler, maybe with a little delay

This way you can shorten the processing time, but may still take a couple
of seconds for every hanbdler, so I would show the user a "fake dialog"
(a group or whatever) indicating that the app is not dead but working. :D
Seeing is believing!

Know what I mean?

AndyP
Posts: 614
Joined: Wed Aug 27, 2008 12:57 pm
Location: Seeheim, Germany (ex UK)
Contact:

Re: Performance and background jobs

Post by AndyP » Thu Jan 31, 2019 5:50 pm

I its a long process then this could be done in a separate hidden stack launched from the main client facing stack. You could then use socket server and client to communicate between the stacks and pass the results back to the main stack. Examples of client and server stacks can be found here

http://livecodeshare.runrev.com/stack/5 ... ket-server

http://livecodeshare.runrev.com/stack/5 ... ket-client

and lessons

http://lessons.livecode.com/m/4071/l/12 ... ng-sockets
Andy Piddock
https://livecode1001.blogspot.com Built with LiveCode
https://github.com/AndyPiddock/TinyIDE Mini IDE alternative
https://github.com/AndyPiddock/Seth Editor color theming
http://livecodeshare.runrev.com/stack/897/ LiveCode-Multi-Search

Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: Performance and background jobs

Post by Klaus » Thu Jan 31, 2019 5:53 pm

Hi Andy,

hm, great idea, but the ENGINE is single threaded, so the same engine running two stacks will cause the same problems as using only one stack, not?


Best

Klaus

ace16vitamine
Posts: 130
Joined: Fri Apr 13, 2018 1:53 pm

Re: Performance and background jobs

Post by ace16vitamine » Thu Jan 31, 2019 8:58 pm

Klaus wrote:
Thu Jan 31, 2019 5:18 pm
Hm, OK, I would probably use some local variables or custom properties
and split the long handler into 6 smaller handlers -> v_order1 to v_order6.
And use SEND to call them from the previous handler, maybe with a little delay
OK, i do this later (in the case that my wife is sleeping... ) :D

This way you can shorten the processing time, but may still take a couple
of seconds for every hanbdler, so I would show the user a "fake dialog"
(a group or whatever) indicating that the app is not dead but working. :D
Seeing is believing!

Know what I mean?
Yes, BUT... Ive already installed a spinner. In the case that my "construction" is working the spinner hangs :roll:

OK, first, let me check if this is really my problem and not a slow DB Connection. I can put whatever to the message box to check on which step the construction is.

Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: Performance and background jobs

Post by Klaus » Thu Jan 31, 2019 9:14 pm

Yes, BUT... Ive already installed a spinner. In the case that my "construction" is working the spinner hangs.
Well, a repeat loop in general takes all CPU power, UNLESS you add a little "wait" statement:

Code: Select all

...
repeat for each line tLine in tNames
     ## Do your stuff here...
     ## ...
     wait 0 with messages
end repeat
...
That should give the SPINNER time to do its job, at least worth a try. :D

ace16vitamine
Posts: 130
Joined: Fri Apr 13, 2018 1:53 pm

Re: Performance and background jobs

Post by ace16vitamine » Thu Jan 31, 2019 10:36 pm

Well, a repeat loop in general takes all CPU power, UNLESS you add a little "wait" statement:
Hm... good to know!

But I have another idea to get more performance :lol:

I have a key... The Order Number. The key is always the same. I can put the SQL Select in another "Transfer" SQLite DB. That means instead of the beast the CPU Power will take over to SQL.

But I need to do this for each line... :-/. I am not sure if the SQLlite Connection is really faster (maybe, a little bit) because in this case (for each line) I have hundreds of inserts

Let me first check to split it into different handlers and then call the handler from each before.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Performance and background jobs

Post by FourthWorld » Thu Jan 31, 2019 10:50 pm

Mutli-threading isn't an option, but multi-processing is. We could explore that if needed, but I'm wondering if the query itself could be optimized. 40 seconds seems like a long time. Can you tell us more about the query, and the number of records, and whether the query involved unindexed fields?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

AndyP
Posts: 614
Joined: Wed Aug 27, 2008 12:57 pm
Location: Seeheim, Germany (ex UK)
Contact:

Re: Performance and background jobs

Post by AndyP » Fri Feb 01, 2019 6:39 am

Klaus wrote:
Thu Jan 31, 2019 5:53 pm
Hi Andy,

hm, great idea, but the ENGINE is single threaded, so the same engine running two stacks will cause the same problems as using only one stack, not?


Best

Klaus
Yes, but once compiled, each instance will be its own entity and allocated its own slice of processor and memory.
Andy Piddock
https://livecode1001.blogspot.com Built with LiveCode
https://github.com/AndyPiddock/TinyIDE Mini IDE alternative
https://github.com/AndyPiddock/Seth Editor color theming
http://livecodeshare.runrev.com/stack/897/ LiveCode-Multi-Search

AndyP
Posts: 614
Joined: Wed Aug 27, 2008 12:57 pm
Location: Seeheim, Germany (ex UK)
Contact:

Re: Performance and background jobs

Post by AndyP » Fri Feb 01, 2019 7:00 am


But I need to do this for each line... :-/. I am not sure if the SQLlite Connection is really faster (maybe, a little bit) because in this case (for each line) I have hundreds of inserts
Another option might be to offload these inserts to a server script and let the server do the heavy lifting .
Andy Piddock
https://livecode1001.blogspot.com Built with LiveCode
https://github.com/AndyPiddock/TinyIDE Mini IDE alternative
https://github.com/AndyPiddock/Seth Editor color theming
http://livecodeshare.runrev.com/stack/897/ LiveCode-Multi-Search

AndyP
Posts: 614
Joined: Wed Aug 27, 2008 12:57 pm
Location: Seeheim, Germany (ex UK)
Contact:

Re: Performance and background jobs

Post by AndyP » Fri Feb 01, 2019 7:05 am

Ah..just noticed its a sqlite db and so probably local..in which case ignore my last post.
Andy Piddock
https://livecode1001.blogspot.com Built with LiveCode
https://github.com/AndyPiddock/TinyIDE Mini IDE alternative
https://github.com/AndyPiddock/Seth Editor color theming
http://livecodeshare.runrev.com/stack/897/ LiveCode-Multi-Search

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Performance and background jobs

Post by AxWald » Fri Feb 01, 2019 3:21 pm

Hi,
ace16vitamine wrote:
Thu Jan 31, 2019 10:36 pm
[...] hundreds of inserts [...]
Oh oh. You will want to avoid this, it's a crazy slowdown! Better CREATE TEMPORARY TABLE and feed it the data all at once, then move it around inside the db (UPDATE, INSERT).

Anyways, I don't really understand what you're doin here, but to me it looks like it should be done within the database itself. This is almost always the better way - databases are optimized to do such things. You might need to make yourself familiar with things like JOIN, UNION, temporary tables, stored procedures & views - but once done you'll not want to miss it anymore.

Hmm. I read your example again. Pity you left out the SQL so I cannot visualize the data. But I can't help thinking you could replace all this with a single SELECT ... FROM ... JOIN statement?

Have fun!
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

ace16vitamine
Posts: 130
Joined: Fri Apr 13, 2018 1:53 pm

Re: Performance and background jobs

Post by ace16vitamine » Sun Feb 03, 2019 8:09 pm

Hey guys,

thank you for all your informations.

Ive checked my App step by step to identify WHAT is the performance hungry monster.

What I found out... My "construct" needs < 1sec, the calculation is not the problem. The SQL Selects are also fast, the bottle neck is:

- The Hard Disc (Performance on virtual environment is extrem slow, don't know why (tested on parallels and ESXi))
- The Sock opening process is slow and raised the CPU (open socket to sock_server with message "push_transferfile_socket_auth" takes 5-10 secs), the communication and transfer itself is fast (< 1 sec).

Now I need to check why the sock opening process is slow. There is no firewall or filter between both applications (localhost connection) and the problem comes only while I am open a socket. After the socket is open the system is running fast. If I open a connection with telnet the answer is immediately there.



Stefan

Post Reply

Return to “Getting Started with LiveCode - Complete Beginners”