Advice & tips for array

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
thatkeith
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 346
Joined: Mon Mar 01, 2010 7:13 pm
Location: London, UK
Contact:

Advice & tips for array

Post by thatkeith » Sat Feb 13, 2021 10:29 pm

I have a moderately large (~5000 lines and counting) text file, and I'd like to parse it and list how many entries there are for each day. It's important that I do this as efficiently as reasonably possible; the file is a simple log of the times an online LC server tool is used, and I'd like to generate the count on the fly as a web page is loaded.

The line structure is: date (currently M/D/Y), time (numbers), AM or PM, and file name (which often have spaces).
I'm guessing arrays will be the secret here – but I have a shameful secret: I've never got my teeth into arrays as they make my head hurt. I apologise. Be gentle, please! :lol:

Sample data:

Code: Select all

2/11/21 8:24:12 PM 360 blender image hell 2-fixed.jpg
2/11/21 9:45:34 PM IMG_20210212_122747_00_005(1)-fixed.jpg
2/11/21 10:23:51 PM 360-fixed.jpg
2/11/21 10:47:44 PM PXL_20210212_055619107.PHOTOSPHERE-fixed.jpg
2/11/21 11:14:27 PM IMG_20210211_181554-fixed.jpg
2/12/21 12:13:03 AM F69230D9-0DC7-40EC-93AE-9265C5FF536A-fixed.jpg
2/12/21 12:28:22 AM R0010069_xmp_e-fixed.jpg
2/12/21 2:51:24 AM bx - 1-fixed.jpg
Any suggestions?

k
Technical Writer, Meta
University Lecturer
Technical Editor, MacUser (1996-2015)
360 VR media specialist

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Advice & tips for array

Post by bogs » Sat Feb 13, 2021 10:45 pm

See if this helps you any https://youtu.be/E_kDfR75AMk
Image

Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: Advice & tips for array

Post by Klaus » Sat Feb 13, 2021 10:55 pm

Hi Keith,

do this:

Code: Select all

on mouseUp
  ## Your data here:
   put fld 2 into tData
   
   ## Looks like your example uses a space, change to your needs...
   set itemdel to SPACE
   put empty into tArray
   repeat for each line tLine in tData
      add 1 to tArray[item 1 of tLine]
   end repeat
   
   ## Display if neccessary
   repeat for each key tKey in tArray
      put tKey & TAB & tArray[tKey] & CR after tNewData
   end repeat
      put "Date" & TAB & "Number of entries" & CR & tNewData into fld 1
end mouseUp
:-)

Best

Klaus

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Advice & tips for array

Post by bogs » Sat Feb 13, 2021 11:00 pm

Hm, I could swear I included that in the video. I may have to re-watch it and see if I forgot it or something... :twisted:
Image

thatkeith
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 346
Joined: Mon Mar 01, 2010 7:13 pm
Location: London, UK
Contact:

Re: Advice & tips for array

Post by thatkeith » Sun Feb 14, 2021 12:13 am

Wow.
Bogs, Klaus, thank you! The code works and is blazingly fast. It's a great kick-off for me! No playing with this tomorrow as it's Feb 14th, but when I can get back to it I'll get the output sorting itself in date order. And I'll watch your video Bogs, thanks for that!

k
Technical Writer, Meta
University Lecturer
Technical Editor, MacUser (1996-2015)
360 VR media specialist

SparkOut
Posts: 2839
Joined: Sun Sep 23, 2007 4:58 pm

Re: Advice & tips for array

Post by SparkOut » Sun Feb 14, 2021 12:21 am

All good, and Klausimausi gave you a good answer to your original question. But as time goes by it will be an ever-increasing overhead if you just have one monolithic log file.

Why not make a new log file every day? Slim and instant. You can always combine the previous files if you need to do processing on date ranges, or one whole load of everything.

Xero
Posts: 152
Joined: Sat Jun 23, 2018 2:22 pm

Re: Advice & tips for array

Post by Xero » Sun Feb 14, 2021 3:04 am

Is there a way that you can do that without going through an array?
Essentially, you know where the pertinent data is in your line structure, and if all you want is the data from ONE day (not a range, that's a different beast), you should just be able to do a "search" for want of a better term...

Code: Select all

put empty into tData
   --put txt file data into variable--
   put url("file:" & "C:/Your File Path Here/Document.txt") into tData
   set the itemdelimiter to space
   -- look at each line and determine date from an input field called 'Date"--
   repeat with tLine = 1 to the number of lines of tData
      if word 1 of line tLine of tData = field "Date" then
         put cr& line tline of tData after tOutput
      end if
   end repeat
   delete line 1 of tOutput
   -- display tOutput if you want to show all lines containing that date.--
   put tOutput into --wherever you want to display it
   
   --or, if all you want is a count of how many times that date appeared (how many accesses, not what was accessed--
   put the number of lines of tOutput into --wherever you want to display it--
This should bypass the array function and also bypass any issues with date formatting. Just as long as you input date format exactly as per text file (i.e. D/M/YYYY, not DD/MM/YYYY or DD/MM/YY. You need to search for the exact same string!

I tested it out and it seems to work...

Hope that helps.
XdM
Last edited by Xero on Mon Feb 15, 2021 7:58 am, edited 1 time in total.

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Advice & tips for array

Post by bogs » Sun Feb 14, 2021 11:45 am

SparkOut wrote:
Sun Feb 14, 2021 12:21 am
Why not make a new log file every day? Slim and instant. You can always combine the previous files if you need to do processing on date ranges, or one whole load of everything.
While I would (generally) tend to agree with this, that could add up to a LOT of files in a short period of time, which could be an issue all its own after a while.

In a case where you are generating multiples of files like this, I'd tend to generate just one log file and make a differentiation (like a date or time stamp, or better yet, both) at the start of each part of it that would equate to a separate file, then use the read/write/update/append functions to get what you need out of it at the time you need it.

Not overly hard to do, and can be made to be *almost* as fast, but sure is a lot easier to keep track of.
Image

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Advice & tips for array

Post by AxWald » Sun Feb 14, 2021 12:29 pm

Hi,

why not just throw those logs into a SQLite & use them from there?
No more reading/ converting of large files, most comfortable adding/ deleting of records, and lots of possibilities of querying even the most elaborate results via SQL. Simple, fast, reliable.

Have fun!
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

thatkeith
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 346
Joined: Mon Mar 01, 2010 7:13 pm
Location: London, UK
Contact:

Re: Advice & tips for array

Post by thatkeith » Sun Feb 14, 2021 2:58 pm

I agree, how it's working now isn't necessarily the best method. Not the worst either, I'm sure, but it was an afterthought in my first 'for real' experimental use of LiveCode Server. :) All I really want to do now is be able to display a chart of the daily uses of the process, which seems to be around 80 per day on average.

@AxWald, do you know of a useful 'getting started' guide for SQLite on a web server, specifically relating to LC? I don't want to stray into PHP land, but only because I'm using this as an ongoing learning exercise for LC in server territory.

k
Technical Writer, Meta
University Lecturer
Technical Editor, MacUser (1996-2015)
360 VR media specialist

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7215
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Advice & tips for array

Post by jacque » Sun Feb 14, 2021 7:24 pm

Xero wrote:
Sun Feb 14, 2021 3:04 am
Is there a way that you can do that without going through an array?
Essentially, you know where the pertinent data is in your line structure, and if all you want is the data from ONE day (not a range, that's a different beast), you should just be able to do a "search" for want of a better term...
The main reason is that the array method and "repeat for each" will be much faster.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

Post Reply

Return to “Talking LiveCode”