Sort multidimensional array

[stam]

Mmm, actually I warned you about this...

Yep, you did indeed. As always i'm ever so grateful for all your advice and contributions which i've found absolutely invaluable during my short time on this forum (especially with regex!) - didn't mean anything by my comment, just that that was needed for multidimensional array sorting.

On some cursory testing, i now see why you left it out - you can't use this for 1-dimensional arrays as Merge() doesn't evaluate anything in brackets it seems...

Based on this i've revised the code to cater for both 1- and multi-dimensional arrays. Includes a conditional (without pKey as per your code, but also with pKey when needed)... can't get away from at least 1 if statement i guess...

Code: Select all

function sortArray @pArray, pDirection, pSortType, pKey
   local tNextIndex, tSortedArray, tSortText
   if (pDirection is not in "ascending,descending" and pDirection is not empty) or \
         ( pSortType is not in "international,numeric,datetime,text,binary" and pSortType is not empty) then
      answer "Malformed parameters - sortDirection must be either Ascending, Descending or empty" & return & \
            "sortType must be either 'international', 'numeric', 'datetime', 'text', 'binary' or empty"
      return pArray
   end if
   get the keys of pArray
   if pKey is empty then 
      put merge("sort lines of IT [[pDirection]] [[pSortType]] by pArray[each]") into tSortText //1-dimensional array
   else
      put merge("sort lines of IT [[pDirection]] [[pSortType]] by pArray[each][pKey]") into tSortText //multidimensional array
   end if
   do tSortText
   repeat for each line tKey in IT
      add 1 to tNextIndex
      put pArray[tKey] into tSortedArray[tNextIndex]
   end repeat
   return tSortedArray
end sortArray

Usage: pArray is mandatory, all other parameters can be passed as empty or ""
- Empty pSortDIrection will default to ascending
- Empty pSortType will default to text
- Empty pKey will treat array as 1-dimensional

Thanks once again @Thierry for helping improve my coding

[Thierry]

Yep, you did indeed. As always i'm ever so grateful for all your advice and contributions which i've found absolutely invaluable during my short time on this forum (especially with regex!) - didn't mean anything by my comment, just that that was needed for multidimensional array sorting.

You're welcome

On some cursory testing, i now see why you left it out - you can't use this for 1-dimensional arrays as Merge() doesn't evaluate anything in brackets it seems...

Based on this i've revised the code to cater for both 1- and multi-dimensional arrays. Includes a conditional (without pKey as per your code, but also with pKey when needed)... can't get away from at least 1 if statement i guess...

Code: Select all

function sortArray @pArray, pDirection, pSortType, pKey
   ....
   if pKey is empty then 
      put merge("sort lines of IT [[pDirection]] [[pSortType]] by pArray[each]") into tSortText //1-dimensional array
   else
      put merge("sort lines of IT [[pDirection]] [[pSortType]] by pArray[each][pKey]") into tSortText //multidimensional array
   end if
   .....

OR as it seems we have kind of same concerns with simple and short code:

Code: Select all

function sortArray @pArray, pDirection, pSortType, pKey
   ....
   local sortCmd = "sort lines of IT [[pDirection]] [[pSortType]] by pArray[each]"
   
   if pKey is not  empty then put "[pkey]" after sortCmd
   
   put merge( sortCmd) into tSortText

   .....

Warning: you have to check/test everything I'm saying

Regards,

Thierry

[stam]

Excellent - 3 lines less

Final code (so far!) tested with one 1-dimensional and one multidimensional array:

Code: Select all

function sortArray @pArray, pDirection, pSortType, pKey
   local tNextIndex, tSortedArray, tSortText
   if (pDirection is not in "ascending,descending" and pDirection is not empty) or \
         ( pSortType is not in "international,numeric,datetime,text,binary" and pSortType is not empty) then
      answer "Malformed parameters - sortDirection must be either Ascending, Descending or empty." & return & \
            "sortType must be either 'international', 'numeric', 'datetime', 'text',  'binary' or empty."
      return pArray
   end if
   get the keys of pArray
   put "sort lines of IT [[pDirection]] [[pSortType]] by pArray[each]" into tSortText //1-dimensional array
   if pKey is not empty then put "[pKey]" after tSortText //mutidimensional array
   do merge(tSortText)
   repeat for each line tKey in IT
      add 1 to tNextIndex
      put pArray[tKey] into tSortedArray[tNextIndex]
   end repeat
   return tSortedArray
end sortArray

Again - feedback welcome

[Simon Knight]

Hi all,

Crikey how this thread has grown! All good stuff. In my recent project I have been reading file details into an array. The files are image files and the folder structure contains something like 180,000 files. Two lessons I have learnt to speed processing are the use of the pass by reference @ qualifier as included in the code examples above and the use of the lock screen command.

Without locking the screen my handlers processed the file names in just under two minutes which I thought was o.k. then I had a "what would Bernd do" moment? So I added the lock screen commands and the file collection is now processed in about 0.36 seconds.

[stam]

I often enjoy these randomly/organically growing discussions
Good to know about lock screen - i've used it but haven't seen a massive increase in speed but maybe i've not been doing stuff that was taxing enough to make it obvious (and haven't bothered time-logging)

I guess it will (almost?) never hurt to use lock screen though

[FourthWorld]

Where appropriate, the combination of lockScreen with lockMessages can be quite dramatic.

[Thierry]

Good to know about lock screen
i've used it but haven't seen a massive increase in speed but maybe i've not been doing stuff that was taxing enough to make it obvious (and haven't bothered time-logging)

Well, I don't believe this will change the speed at all in this use case,
plus I think using or not the @ prefix in your parameter won't change much too...

I guess it will (almost?) never hurt to use lock screen though

It could

Regards,

Thierry

[Simon Knight]

plus I think using or not the @ prefix in your parameter won't change much too...

it depends on the size of the data being passed: somewhere on this forum I posted about a problem where the routine was taking hours and hours to complete, adding the prefix stopped the duplication of data and reduced the processing to tens of seconds. I'll dig it out later but its getting late.

best wishes

Simon

[FourthWorld]

How many records are in the returned data?

[bobcole]

This may not be the topic to which Simon Knight referred but it is recent and, I think, relevant to the @ question:
viewtopic.php?f=7&t=35369
FYI,
Bob

[bobcole]

Also, I seem to recall that there was an issue with using the "do" command.
If I remember correctly, it has somehow related to Apple not liking programs that have potentially self-altering code in their App Store.
Anyone familiar with that issue?
Sorry, it is only a vague recollection.
Bob

[Simon Knight]

Hi

Here is a link to my post from 2011

viewtopic.php?f=32&t=7069&p=32207&hilit ... nce#p32207

In it I estimate that an operation would take 56 hours, this was reduced to 70 seconds mostly by switching to referenced variable passing. Mind you the variable was quite large at 70Mbytes.

Thanks to Bernd the estimated 56 hours for processing a datafile is down to a real 70 seconds. The main speed problem was caused by passing a copy of the complete data variable to a function that was being called over a million times. Livecode just choked. The solution was to add the @ character before the parameter name in the function declaration, this makes livecode pass a reference to the data rather than the default of sending a copy. the data is stored in theBinarydata e.g.

[stam]

...there was an issue with using the "do" command....
...somehow related to Apple not liking programs that have potentially self-altering code in their App Store....

I must say, that seems unlikely.... I’d be surprised if Apple did a code-level review of apps (but really have no idea).

Plus there’s a ton of iOS apps (mainly games) that introduce substantial changes by altering themselves (downloading stuff after install into the app) or by acting as a “player” for essentially “unauthorised” apps

In this instance the do command is just replacing a ton of conditional statements. And not sure Do does anything that can’t be replicated with with other code structures?

Can anyone shed more light on this?

[stam]

Here is a link to my post from 2011

viewtopic.php?f=32&t=7069&p=32207&hilit ... nce#p32207

In it I estimate that an operation would take 56 hours, this was reduced to 70 seconds mostly by switching to referenced variable passing. Mind you the variable was quite large at 70Mbytes.

Yes but they have optimised as of v8 to always use a pass by reference when possible - Bob links to an interesting discussion above, in which Richard explains:

In fact, it's so useful to pass by reference that the engine now does that automatically by default (as of v8 or so).

It uses "copy on write" by default, so even if you don't explicitly declare a param as reference ("@"), it won't actually make a copy into you perform some action which would alter it. If you want that alteration to be reflected in the calling handler, use @ before the arg name in the definition handler.

In your case, the only operation is read, so you should see no difference in performance whether you use @ or not, since only a pointer to the original data is what's being used by default.

@Richard: I presume this means of moving around large data and writing to it would still benefit from passing by reference?

[Simon Knight]

I didn't know about the changes in V8 so thanks for the link and quotes.

Simon