Allocation of memory when processing large files

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Allocation of memory when processing large files

Post by Simon Knight » Sat Oct 03, 2020 8:35 am

Hi,

I'm using a Mac and I am trying to extract certain data points for large XML files created by and available from the OpenStreetMap project. I have downloaded an XML file that describes Scotland. This file size is 3.2 Gbytes which all my text processors choke on. However the excellent "HexFiend" reads it without problem.

To date I have not used the XML tools that Livecode provides as I'm not a fan of xml, so I am following one of the tutorials. Basically this loads the data into a variable and then this variable is passed to the XML routines to create a tree. Later I plan to convert the tree to an array but I'm digressing.

My mac has 16Gbytes of memory so the 3.2 Gbytes is well within memory limits and so far the simple code is using just over 10Gbytes of memory.

Other larger regions of the world obviously require larger files. I wonder how Livecode handles instructions that cause the available memory to overflow. I know I can and should try such a file but even the 3.2Gbyte file is taking ages to process and I'm not sure I have the patience to wait for Livecode to attempt to process a 50 Gbyte file. So does anyone know if Livecode automatically starts to use the hard drive to store data in its variables or is it up to me?

I should say that I have no intention of having my final version process whole files in one go as it is obvious that reading in bite size chunks of data is a superior method of both developing the app and reading the data. However, I am interested how Livecode manages memory overflow. My guess is that it will use resources until the OS starts to post warnings then die.
best wishes
Skids

Post Reply

Return to “Talking LiveCode”