How to get date and time from a file

[mfstuart]

Hi all,
I need to get the modified date and time from a file that resides in a folder with thousands of files.
Using the "detailed files" is way too slow as it has to get all the files.

So is there a direct way, with the path and file name, to get the information from the file?

Thanx,
Mark Stuart

[WaltBrown]

In Win7 I use the shell.

Code: Select all

  local tFilePath, tCmd
   put "C:\Users\Walt\output.txt" into tFilePath
   put "dir" && tFilePath into tCmd
   put return & shell(tCmd) after fld "fFileData"

The returned data is the last access time and date and file size.
Linux would use "ls" of course. I don't know the Mac shell command.

[dunbarx]

Hi.

Goodness, if you have the pathname, use the "read from file" command, and work it in LC directly. See the dictionary...

Craig Newman

[FourthWorld]

@mfstuart: What is the amount of time your routine takes for obtaining that info from the "detailed files" list, and what does your code look like? Lineoffset is usually pretty fast.

[mfstuart]

Hi Craig,
I'm not sure how to get the modified date and time of the file from your suggestion of using "read from file".
Can you please elaborate?

Thanx,
Mark Stuart

[Klaus]

Yes Craig, we are waiting!

[mfstuart]

Hi Richard,
On my Win7 computer, it takes 30 seconds just to process the script line "put the detailed files into tFiles".

Here's the script:

Code: Select all

command StartProcess pFolder, pFilename
   put the defaultFolder into defFolder
   set the defaultFolder to pFolder
   
   put the detailed files into tFiles    --this line takes 30 seconds to process with close to 15,000 files in the folder pFolder
   filter tFiles with "inprocess.txt*"   --I'm looking for this file to get the modified date and time of it

   set the defaultFolder to defFolder   
end StartProcess

Since I have the path and filename, I'll use revCopy to copy the file to a folder with no files in it and apply the "put the detailed files into tFiles" there.
It should process in no time.

Thanx,
Mark Stuart

[FourthWorld]

How many files are in that folder, and what is the average length of their file names?

[mfstuart]

Hi Richard and Walt,
The folder that I need to read contains files that contain logging information about processes that run in batch oriented applications - fetch records and send email. When the application starts up it creates a text file called "inprocess.txt". This is so that another process cannot start if this file exists. When the application process is complete, it deletes the file.

Windows Task Scheduler launches the batch application executable every 15 minutes, starting at 6am until 10pm, 7 days a week.
The application creates and writes to a log file while processing. So there's 4 log files for each hour.
So that's 64 files per day to read with "put the detailed files into tFiles".

I haven't "archived" these files in some time so the file count is at 15,100.

The other day the application crashed leaving the "inprocess.txt" in the folder. So we didn't have any email going out for a week.
Yes, I hadn't created a watchdog for this situation, and I hadn't manually checked the folder for this file. So hence the need for an app to check for the existence of this inprocess.txt file. It will read the modified date and time and compare it with today's date.
If it's say, more than 3 hours difference then send myself an email of the issue.

Now that I'm writing this reply, I'm thinking I need to archive these files daily. That will certainly return a better performance with the LC script.

The average file name length is 45 characters. e.g.: ECS_ECS_OUTBOUND_MISSED_B_20140813_104701.log

Walt - I'm currently using your "shell" command to get and parse the date and time directly from the folder. Works great. Thanx for that.

Thanx,
Mark Stuart

[FourthWorld]

Now that I'm writing this reply, I'm thinking I need to archive these files daily. That will certainly return a better performance with the LC script.

No doubt, but it's an interesting problem just the same. I did a quick test on Mac, Linux, and Win, and Win took about 6 times as long as the others to obtain the mod dates on the same random selection of 100 files in a folder of 15,100 files. Some of the speed difference may be accountable by my Win 8.1 install running in a VM, but for most CPU-intensive tasks like filtering I don't see that much of a difference, so I'm wondering if there may be an opportunity to optimize your NTFS by turning off short file name support if possible, or defragging the directory index with contig.exe as describe here:
http://stackoverflow.com/questions/1971 ... irectories

Either way, on all three platforms my notion of lineoffset being faster than filter didn't pan out - they took almost the same amount of time, with lineoffset being only about 5% faster.

Interestingly, my Win 8.1 system still only took about 6.5 seconds to run through all 100 file mod dates, or just 650 ms per file. Considering this was in a VM, this suggests there may be something going on with your NTFS if it takes roughly 50 times longer to do the task (I'm running a 3.0 GHz Haswell duel core, but it's just a Pentium so it's not exactly the fastest CPU out there). I'd be interested to know if any file system optimizations improve performance for your system.

Ultimately, you've hit the nail on the head in your OP: we could really use a way to obtain file info like this for a single file, so I just submitted a request for this:
http://quality.runrev.com/show_bug.cgi?id=13182

[marksmithhfx]

Ultimately, you've hit the nail on the head in your OP: we could really use a way to obtain file info like this for a single file, so I just submitted a request for this:
http://quality.runrev.com/show_bug.cgi?id=13182

Hi Richard, that link doesn't work anymore... but just wondering (since I need to do it), was this ever addressed in LC?

Thanks
Mark

[AxWald]

Hi,

[...] Win took about 6 times as long as the others to obtain the mod dates on the same random selection of 100 files in a folder of 15,100 files. [...]

It may be caused by this. I wrote about it here.

tl;dr:
Text operations with huge lists are unbearably slow in the Windows versions of the contemporary LCs. This is acknowledged by the mothership, a crude workaround exists, and maybe it will be fixed some day.

Have fun!