Large number of images in a data grid

Got a LiveCode personal license? Are you a beginner, hobbyist or educator that's new to LiveCode? This forum is the place to go for help getting started. Welcome!

Moderators: Klaus, FourthWorld, heatherlaine, kevinmiller

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 5402
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Large number of images in a data grid

Post by jacque » Tue Dec 25, 2018 6:57 pm

Datagrids natively page their contents as they scroll, so you don't need to script that. The problem is likely the number and size of the images. Have you saved the thumbnails at their smallest size and resolution, or are you using full-sized images that are just scaled down in the IDE? A lot of work goes on to expand, scale, and draw images to screen, so smaller is faster.

Smaller refers primarily to the number of pixels in the image rather than the actual dimensions. Color depth matters too, if you are saving the images with millions of colors the actual file will be a magnitude larger than the same image saved at 8 or 16 bits. 8-bit color isn't usually great but 16 might do for such small images.

You could also save the image in an uncompressed format like BMP which would eliminate the need for the engine to decompress the file. The files would be larger on disk but would load faster.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Wed Dec 26, 2018 1:28 am

Thanks. I tried setting the paintCompression global setting to PICT before saving the thumbnails, and it seems to make the display a little quicker.

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 5402
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Large number of images in a data grid

Post by jacque » Wed Dec 26, 2018 5:45 pm

How large are the images in bytes? That's the main thing. Is the width of the image the same size as the formattedWidth?
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Wed Dec 26, 2018 7:50 pm

The images are 150x150 and are not resized in the data grid. They are roughly 30 to 50 Kb each. That corresponds to 75MB for 1500 pictures at 50kb.

I made some tests, and the size can be halved by saving them as 256 color BMP, or to about 5Kb in JPEG, without losing quality, it seems.

How would I go about transforming the format and number of colors within LiveCode ? EDIT: maybe use "export with palette" to copy it to another image with a specific number of colors ? seems to have been done here: https://forums.livecode.com/viewtopic.php?t=22080

Got it to work. Will play around with various formats. Thanks for the inputs.

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 5402
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Large number of images in a data grid

Post by jacque » Wed Dec 26, 2018 11:49 pm

Glad you found that, I wasn't aware of the "with palette" parameter. If you use JPG format then each image will have to go through the decompresion process which adds a slight overhead, so if you you can get it work with an uncompressed image format you'll save a bit of time.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Fri Jan 04, 2019 1:42 am

The performance of data grids is really disapppointing. I am testing instead a browser widget with generated html code containing the url of the images, and it seems to work well. I will confirm...

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Fri Jan 04, 2019 2:50 am

It is "lightning" fast with a widget !

trevordevore
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 986
Joined: Sat Apr 08, 2006 3:06 pm
Location: Overland Park, Kansas
Contact:

Re: Large number of images in a data grid

Post by trevordevore » Fri Jan 04, 2019 5:40 am

For your needs a browser may give you the best results. I just skimmed through this thread and wanted to mention a couple of things though -

1. For a large data set that is displaying lots of images do not turn caching on. There is an initial slowdown when loading the data and it won't scale well if your data set increases.
2. Rather than converting the SQL cursor to an array and assigning it to the dgData it would be much more efficient use the GetDataForLine callback to feed data to the DataGrid on an as-needed basis. Here is a brief article on it:

http://lessons.livecode.com/m/datagrid/ ... ts-of-data

When using a database connection the approach would be as follows:

1. Open a database cursor using revQueryDatabase() and store the cursor id in a script local variable that you can pass to revdb handlers.
2. Set the dgNumberOfRecords of the DataGrid to revNumberOfRecords().
3. In the GetDataForLine command that you define pass pLine to revMoveToRecord() in order to move the cursor to the correct record.
4. Use revDatabaseColumnNamed() or revDatabaseColumnNamed() or revDatabaseColumnNumbered() to extract columns from the current row in the cursor and populate the @pDataA parameter passed to GetDataForLine.

This will be the most efficient way to streaming data from the database cursor into the DataGrid.
Trevor DeVore
ScreenSteps - https://www.screensteps.com

LiveCode Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode
LiveCode Builder Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode-builder

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Fri Jan 04, 2019 4:17 pm

Thanks. I have another data grid which I use to display album tracks, which is also seems slow when there are a large number of tracks (> 100), and I will optimize it as you indicated.

trevordevore
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 986
Joined: Sat Apr 08, 2006 3:06 pm
Location: Overland Park, Kansas
Contact:

Re: Large number of images in a data grid

Post by trevordevore » Fri Jan 04, 2019 6:24 pm

After I responded to you last night I decided to write a database cursor behavior for the DataView control I recently made available. You can read about it in this thread:

viewtopic.php?f=108&t=31987

A DataView is a more modern DataGrid and is more efficient. It is also easy to add behaviors to it in order to provide an interface to a specific type of data source. There is an interface for arrays, tree nodes, and now a database cursor.

If you want to take a look you can find the demo project here:

https://github.com/trevordevore/dataview_demo

The actual code that feeds a cursor to the DataView can be found here:

https://github.com/trevordevore/levureh ... codescript

You could do something similar for your DataGrids. Look at the `DataForRow` command which is similar to `GetDataForLine` in a DataGrid.
Trevor DeVore
ScreenSteps - https://www.screensteps.com

LiveCode Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode
LiveCode Builder Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode-builder

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Sat Jan 05, 2019 12:11 am

Excellent, thank you, will definitely look into it. I just happened to be working also on a way to display folders & files, and I see you worked on that as well.

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Sat Jan 05, 2019 11:13 am

I started out by downloading the example stack from LiveCode (Data Grid and Databases.rev). While the initial load of the database is fast, there are issues when scrolling through the grid, especially when clicking in the vertical bar.

trevordevore
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 986
Joined: Sat Apr 08, 2006 3:06 pm
Location: Overland Park, Kansas
Contact:

Re: Large number of images in a data grid

Post by trevordevore » Sat Jan 05, 2019 3:17 pm

Where did you find the example stack? I'm not seeing it in my searches.
Trevor DeVore
ScreenSteps - https://www.screensteps.com

LiveCode Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode
LiveCode Builder Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode-builder

trevordevore
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 986
Joined: Sat Apr 08, 2006 3:06 pm
Location: Overland Park, Kansas
Contact:

Re: Large number of images in a data grid

Post by trevordevore » Sat Jan 05, 2019 7:10 pm

@hopkins - do you have the Project Browser open when you are testing the Data Grid? If so, try closing it. The Project Browser will slow things down as the Data Grid is creating/deleting/showing controls.
Trevor DeVore
ScreenSteps - https://www.screensteps.com

LiveCode Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode
LiveCode Builder Repos - https://github.com/search?q=user%3Atrevordevore+topic:livecode-builder

hopkins
Posts: 41
Joined: Wed Nov 28, 2018 11:09 am

Re: Large number of images in a data grid

Post by hopkins » Sat Jan 05, 2019 8:39 pm

No I do not have it open.

I found the example stack (livecode's) in the link you gave a few posts up: http://lessons.livecode.com/m/datagrid/ ... ts-of-data

Post Reply

Return to “Getting Started with LiveCode - Complete Beginners”