Advice & tips for array

[thatkeith]

I have a moderately large (~5000 lines and counting) text file, and I'd like to parse it and list how many entries there are for each day. It's important that I do this as efficiently as reasonably possible; the file is a simple log of the times an online LC server tool is used, and I'd like to generate the count on the fly as a web page is loaded.

The line structure is: date (currently M/D/Y), time (numbers), AM or PM, and file name (which often have spaces).
I'm guessing arrays will be the secret here – but I have a shameful secret: I've never got my teeth into arrays as they make my head hurt. I apologise. Be gentle, please!

Sample data:

Code: Select all

2/11/21 8:24:12 PM 360 blender image hell 2-fixed.jpg
2/11/21 9:45:34 PM IMG_20210212_122747_00_005(1)-fixed.jpg
2/11/21 10:23:51 PM 360-fixed.jpg
2/11/21 10:47:44 PM PXL_20210212_055619107.PHOTOSPHERE-fixed.jpg
2/11/21 11:14:27 PM IMG_20210211_181554-fixed.jpg
2/12/21 12:13:03 AM F69230D9-0DC7-40EC-93AE-9265C5FF536A-fixed.jpg
2/12/21 12:28:22 AM R0010069_xmp_e-fixed.jpg
2/12/21 2:51:24 AM bx - 1-fixed.jpg

Any suggestions?

k

[bogs]

See if this helps you any https://youtu.be/E_kDfR75AMk

[Klaus]

Hi Keith,

do this:

Code: Select all

on mouseUp
  ## Your data here:
   put fld 2 into tData
   
   ## Looks like your example uses a space, change to your needs...
   set itemdel to SPACE
   put empty into tArray
   repeat for each line tLine in tData
      add 1 to tArray[item 1 of tLine]
   end repeat
   
   ## Display if neccessary
   repeat for each key tKey in tArray
      put tKey & TAB & tArray[tKey] & CR after tNewData
   end repeat
      put "Date" & TAB & "Number of entries" & CR & tNewData into fld 1
end mouseUp

Best

Klaus

[bogs]

Hm, I could swear I included that in the video. I may have to re-watch it and see if I forgot it or something...

[thatkeith]

Wow.
Bogs, Klaus, thank you! The code works and is blazingly fast. It's a great kick-off for me! No playing with this tomorrow as it's Feb 14th, but when I can get back to it I'll get the output sorting itself in date order. And I'll watch your video Bogs, thanks for that!

k

[SparkOut]

All good, and Klausimausi gave you a good answer to your original question. But as time goes by it will be an ever-increasing overhead if you just have one monolithic log file.

Why not make a new log file every day? Slim and instant. You can always combine the previous files if you need to do processing on date ranges, or one whole load of everything.

[Xero]

Is there a way that you can do that without going through an array?
Essentially, you know where the pertinent data is in your line structure, and if all you want is the data from ONE day (not a range, that's a different beast), you should just be able to do a "search" for want of a better term...

Code: Select all

put empty into tData
   --put txt file data into variable--
   put url("file:" & "C:/Your File Path Here/Document.txt") into tData
   set the itemdelimiter to space
   -- look at each line and determine date from an input field called 'Date"--
   repeat with tLine = 1 to the number of lines of tData
      if word 1 of line tLine of tData = field "Date" then
         put cr& line tline of tData after tOutput
      end if
   end repeat
   delete line 1 of tOutput
   -- display tOutput if you want to show all lines containing that date.--
   put tOutput into --wherever you want to display it
   
   --or, if all you want is a count of how many times that date appeared (how many accesses, not what was accessed--
   put the number of lines of tOutput into --wherever you want to display it--

This should bypass the array function and also bypass any issues with date formatting. Just as long as you input date format exactly as per text file (i.e. D/M/YYYY, not DD/MM/YYYY or DD/MM/YY. You need to search for the exact same string!

I tested it out and it seems to work...

Hope that helps.
XdM

[bogs]

Why not make a new log file every day? Slim and instant. You can always combine the previous files if you need to do processing on date ranges, or one whole load of everything.

While I would (generally) tend to agree with this, that could add up to a LOT of files in a short period of time, which could be an issue all its own after a while.

In a case where you are generating multiples of files like this, I'd tend to generate just one log file and make a differentiation (like a date or time stamp, or better yet, both) at the start of each part of it that would equate to a separate file, then use the read/write/update/append functions to get what you need out of it at the time you need it.

Not overly hard to do, and can be made to be *almost* as fast, but sure is a lot easier to keep track of.

[AxWald]

Hi,

why not just throw those logs into a SQLite & use them from there?
No more reading/ converting of large files, most comfortable adding/ deleting of records, and lots of possibilities of querying even the most elaborate results via SQL. Simple, fast, reliable.

Have fun!

[thatkeith]

I agree, how it's working now isn't necessarily the best method. Not the worst either, I'm sure, but it was an afterthought in my first 'for real' experimental use of LiveCode Server. All I really want to do now is be able to display a chart of the daily uses of the process, which seems to be around 80 per day on average.

@AxWald, do you know of a useful 'getting started' guide for SQLite on a web server, specifically relating to LC? I don't want to stray into PHP land, but only because I'm using this as an ongoing learning exercise for LC in server territory.

k

[jacque]

Is there a way that you can do that without going through an array?
Essentially, you know where the pertinent data is in your line structure, and if all you want is the data from ONE day (not a range, that's a different beast), you should just be able to do a "search" for want of a better term...

The main reason is that the array method and "repeat for each" will be much faster.