How does a datagrid achieve almost unlimited rows?

[Simon Knight]

The background to this question is that I have a stack that uses a group to display and scroll a virtual column of custom controls. When data is loaded into the group a template is copied to a new X Y coordinate. This works well but suffers from the situation where the nth custom control has its Y coordinate set to a value greater that 32768 which throws an error. This overflow occurs at approximately the 140th custom control.

Now Stam has pointed out that I should/could be using a datagrid form and has helped by designing a prototype. This datagrid has no issues with loading 500 rows and I am interested in how this is achieved behind the scenes. I have tried looking inside a datagrid but have been unable to discover how it is achieved but that is probably because I have been looking in the wrong place.

At the moment my non datagrid group displays its contents through the use of its built in vertical scroll bar. I suspect that this will have to be removed and an external scroll tool used to select a number of records from the data and redraw/repopulate all the visible custom controls each time the user moves the external scroll tool. But I thought I would ask for ideas before I try breaking my existing code

[paul@researchware.com]

The basic approach is to display only the visible rows of data and so render only the enough custom controls to hold the visisle rows).

As for how you figure and calculate the visible rows, I don't have an algorithm to offer you. I would assume it would neeed the following elements:

1) Get the total number of rows of data to be displayed (number of keys from an array, number of lines from a delimited field, number of rows froma SLQ query)
2) Determine, based on the size of the containing GROUP (for the custom controls) how many rows can be render (say by rendering the first N records until the height of the number of visible controls > than the group height).
3) Contructs a scroll bar that when scrolled updates the data displayed in the visible controls.
4) Create a resizeControl handler for the containing GROUP that reduced or adds additions rows of visible controls and updated the count of how many rows are visible.

I am sure I am missing something (probably several somethings).

[Simon Knight]

Hi Paul,

I think you are correct as looking at a form displaying a number of image thumbnails its possible to see it stutter when being scrolled. My guess is that the stutter is caused when new rows are added just out of view and other rows are deleted from the "end" opposite to the direction of scroll.

[FourthWorld]

32,765 pixels is about 30 feet of scrolling. That's a lot of scrolling.

If you need it, you need it. And the DG handles it well.

But do you need it?

Does the user need to see all those records at the same time?

Is this an opportunity waiting to be discovered for simplifying record selection?

I don't know, I haven't seen your app. But sometimes questions outside the problem are worth considering.

[Simon Knight]

But do you need it?
Does the user need to see all those records at the same time?
Is this an opportunity waiting to be discovered for simplifying record selection?

The simple answer is no, this app does not need to display more than the 149 odd records it can presently display. The final output are copies of the images reduced in size for use on a web site along with a CSV file used by the web design software. I suppose my interest is academic.

In truth a list is a poor way of displaying more than twenty or thirty images but I am more interested in how to replace my present system that copies a template control into the group and allows the user to scroll up and down with a system that just adds / deletes controls as needed for display.

Trevor worked out how to do it back in 2009 with the Datagrid and I am sure others have also produced similar scrolling controls since. When it comes to image selection and sorting a proper scrolling grid is the method employed by most image applications. I feel that I need to work out the straight forward list version before I venture into rows and columns.

My present custom control works pretty well with 125 images with smoother scrolling than a datagrid form and far better drag and drop (if I say it myself). In defence of the datagrid my attempts so far have not been refined and the datagrid is general purpose with features that I do not need. However, the datagrid form's dragging and dropping of rows seems clunky to me.

[paul@researchware.com]

For what it's worth, I have replaced a few instances where I used Datagrids to display table data with Polygrid widgets. In my case all the table data is text or numbers, the dictionary entry for the pgColumns property of the polygrid suggests it can display all the following data types: columnIndex["contentType"] -- text|svg-lcname|svg-path|imagedata|imagefile|color|boolean|autoincrement|progress

I have found the polygrid widget to be easier to use than the datagrid in general unless some very specific customer object is needed in a row of data.

[Simon Knight]

I have found the polygrid widget to be easier to use than the datagrid

I have not tried using the polygrid but will have a go as it sounds interesting.

[Simon Knight]

Just tried the polygrid. Very simple to set up and similar scroll performance to a datagrid. Both the datagrid and the polygrid benefit from being supplied with smaller thumbnail images.

Surprise surprise feeding both of them both with thumbnails improves the scrolling removing the stutter previously reported.

The issue with the polygrid is does not appear to be an inbuilt method of editing text fields. It may be possible to use a second field as a field editor or just have a hidden field / form that gets displayed to allow text editing.

A further limitation is that there is no inbuilt row reordering, at least I can't find one.

[paul@researchware.com]

The issue with the polygrid is does not appear to be an inbuilt method of editing text fields. It may be possible to use a second field as a field editor or just have a hidden field / form that gets displayed to allow text editing.

Yea, this is probably THE biggest reason to still use a datagrid vs a polygrid. If you need field editing, the datagrid is definitely easier.

A further limitation is that there is no inbuilt row reordering, at least I can't find one.

Column sorting is not built into the polygrid, but is easy to do. Panos provided a lesson. I can't seem to find the boomark to the link at the moment.

[stam]

Just tried the polygrid. Very simple to set up and similar scroll performance to a datagrid. Both the datagrid and the polygrid benefit from being supplied with smaller thumbnail images.

Surprise surprise feeding both of them both with thumbnails improves the scrolling removing the stutter previously reported.

The issue with the polygrid is does not appear to be an inbuilt method of editing text fields. It may be possible to use a second field as a field editor or just have a hidden field / form that gets displayed to allow text editing.

A further limitation is that there is no inbuilt row reordering, at least I can't find one.

In general I find PG slightly more performant than DG, but has a significantly smaller feature set, so I usually end up reverting back to the DG. On the plus side it’s a self contained widget, whereas the DG is a complex group that requires a behaviour sub stack that makes it copying to other stacks a much more complex affair.

It good that we have more options now with the tableField (and in particular Bernd’s modTableField), the datagrid, the polygrid and not forgetting polylist.

[Simon Knight]

I wonder if either of you would use a data/polygrid to display a true grid of image thumbnails in the way of most image cataloging programs. I can imagine its quite possible to populate a datagrid form with multiple thumbnails per row but imagine that drag drop reordering would be difficult.

[stam]

I would use the polyList for that, it's designed to be resizeable with variable number of columns.

if I understood correctly you want to do what I've done in one of my apps, but with images instead - here is a list of medications presented as 1 column or 3 columns

If I understood correctly that's what you're hinting at - if yes, PolyList is an easy solution to this. However not sure if it's possible to do drag'n'drop reordering... it might be.

[stam]

Playing around with PolyList, that may be an interesting solution.
Set the image control to be of type 'image-file'. Position the stuff as you want (less intuitive as it means playing around with margins rather than directly dragging/manipulating - I find it helpful to temporarily set the 'fill' tickbox to true while resizing).

what I've seen so far, using full size images, no thumbnails:
1. loading is as fast as DG if set to 1 column, but signficnalty slower if setting to multiple columns
2. on scrolling the PL with full size images loaded, the first scroll is slow, but it caches the images so subsequent scrolls are lightening fast
3. it's a great way to set up a true grid:

I've added this side-by-side to the datagrid solution for comparison (just visually, not implemented time counts), stack attached.
Edit: obviously you'd need to have access to the external PolyList for this to work....
As per everything else, I suspect that using thumbnails instead of full size images would greatly speed up things even more...

[stam]

I also wonder if speed would be better if getting the binary data from the image and storing that in the array (the PL has a data type of image-data which presumably can be set to the binary data) might be faster... but not tested.

[Simon Knight]

Unfortunately your last demo stack does not display any images in the polylist. The data is being set but there are no items set. Given I have never looked at the polylist before I'm failing to get it to work.