Scrolling a group with a very long field - basic concept?

[dunbarx]

Made a test about the capacity of a field. Interesting and odd. I am starting a new thread about it in the "Talking Livecode" area.

Bottom line, though there are issues, a field can hold at least 500,000 characters, but getting them into that field is time consuming.

Craig

[FourthWorld]

A LiveCode field can hold up to ~4GB of data, theoretically. Obviously practical memory limits will come into play far sooner, and usability considerations sooner still.

Fields scroll. Everywhere. On all platforms, in all development systems.

Jumping through hoops to get something this basic is a usability bug.

[richmond62]

A LiveCode field can hold up to ~4GB of data, theoretically.

Does that mean that when a stack containing a field containing 4GB of text is open
it will take up at least 4GB of memory?

[FourthWorld]

A LiveCode field can hold up to ~4GB of data, theoretically.

Does that mean that when a stack containing a field containing 4GB of text is open
it will take up at least 4GB of memory?

No, because practical memory limits will come into play far sooner, and usability considerations sooner still. That is, such a field is extremely unlikely to exist.

But for practical sizes, yes, the size of the text on disk is the minimum memory required to load it into RAM; operations on it, including rendering it, will require more memory.

[AxWald]

Hi again :)

Craig,
the "32,767" are a LC specific limitation for coordinates: "The current architecture uses 16-bit signed integers for all co-ordinates", from the current dict. I assume the mothership thinks that's by far enough for apps.
About field limitations, besides what was said elsewhere: You can put really huge chunks of data into a field. "Length of lines" is to consider, but that should only hit when using raw picture (or artificial) data. But that doesn't mean you should do it - I made some speed tests, results below.

Jaque,
Group scrolling smoother? Didn't notice yet on my (few) tests, but I'll keep this in mind.
About char counts & the results, see below.
And, for sure, I need to implement some housekeeping with the mobile scrollers. I assume I'll have max 2 of 'em visible at the same time, so I could create these as dummys at PreOpen and only set/ reset the rect/ contentRect/ visibility when needed? (Btw., during my testing today exactly this happened - I had too many scrollers left unclosed, and creating a new one suddenly took very long)

Richard,

Jumping through hoops to get something this basic is a usability bug.

I agree - after all, it's not this complicated, as I see now. Actually, it's quite simple once you find out yourself!
But there's no clear & concise documentation anywhere, and the examples given at lessons.livecode.com are all a bit different & contradictory. (And I didn't see a single picture of an Android screen ...)
Further, any wisdom from the devs is buried within the mailing list that is such a wasteland of quoting madness that even googlebot shudders, turns away in sheer terror & moves the results to the backwards pages to avoid class action suits for reckless causing of eye cancer.

Richmond,
it will take up ways more! See below, 2nd table.


So, I did some tests.
I used a demo dataset ("load_employees.dump"), cleaned up & reduced to 300K lines.
This I read from file ("put URL ("file:" & it) into myData"), ask "How many lines?" and do a "repeat for each line L in myData - put [...] & CR after myVar", with "exit repeat" after the number of lines given.
In the loop I add a counter column, and shuffle the columns a bit. Finally I sort the result & put it into a customProperty. The data inserted are 15,809,783 Bytes, ~15 MB.
This time is "Processing". (Data read is always the same big file, I only process the given number of lines)

Then I "put the c_Data of this cd into fld "data_fld"". Time "Display", stopped manually.

Finally (mobile only) I lay a mobScroller over the field (code somewhere above, "mobScrollMake"). This happens instantly (as long as don't have bazillions of old ones unclosed ...).

Code: Select all

Lines used:		300K		30K		3K
Data (Chars):		15809783	1530390		150119
FormattedHeight px:	9000010		900010		90010

Processing Desktop, ms:	1400		120		20
Processing Android, ms:	8700		800		165

Display Desktop, sec:	17		<1		0
Display Android, sec:	20		<3		0

The test shows that LC & the field easily handles data with 30K lines and about 1.5MB, even on Android. 300K lines & 15MB looks like a bit much, but more potent hardware *) & a patient user may handle it. I get a system window then like "[appName] doesn't respond, wanna quit?" But it finishes the task, and the scroller is exactly as agile as it is with the small dataset.

It may be noticed that a quick test with LC 9.6/64 was disappointing again - for it you may apply ~ x6 to the times above. I work with 6.7.10 ("latest stable" ;-) )

While doing these tests, I watched the RAM usage of the IDE in taskmanager. The values remain consistent during repeated tests:

Code: Select all

RAM used in IDE - dataset "300K" (~15MB), all lines:
Stack & IDE initial: 	~25 MB
after "Processing":	~46 MB
after "Display":	~110 MB
after "clear all":	~54 MB

I "delete variable ..." anything used in the initial reading & processing as soon as possible. "Clear all" means put/ set empty for the customProp and the field. So LC must keep additional copies of the data somewhere, it uses way more RAM than expected & doesn't free it completely afterwards.

*) Test configuration:
Tablet: Lenovo Yoga Tab3 850L, Android 6.0.1, 2GB RAM, Snapdragon 210 4x 1.3GHz
Desktop: HP Compaq Pro 6300, Win10/64, 8GB RAM, Pentium G2020 2x 2.9GHz, GeForce GT 730
LiveCode: 6.7.10/32bit

[jacque]

Versions of LC before LC 7 were much faster. I'm not sure the timing tests will be accurate for LC 9.6 and Android 10. Field scrolling worked with that configuration but wasn't as smooth, though on high powered devices it was quite acceptable. Testers with older hardware and lesser CPUs said they got headaches when scrolling, obviously hyperbole, but that's why I converted everything to groups. When I tested on my 4 year old tablet with Android 6.0.1 I did see jitters.

I misspoke about the capacity of a field, which can indeed hold a huge amount of text. My roadblock was the formatted dimensions of the field, which is more limited.

[richmond62]

Versions of LC before LC 7 were much faster.

Why?

[jacque]

Versions of LC before LC 7 were much faster.

Why?

The most commonly cited reason is the implementation of native unicode management. It affected nearly every aspect of the engine, and for field behavior, requires juggling twice as many bytes as before. But there was also a complete rewrite of the engine structure, which allowed the addition of many new features.

[FourthWorld]

Reading blogs for other scripting languages reveals a similar pattern with Unicode adoption: initial significant performance loss, later some of it regained with optimization in other areas in addition to the computationally-expensive encoding coercion itself, and a lot of new features along the way that come with adoption of industry-standard text libraries (eg in LC trueWord, multi-char itemDel, and more).

And with LiveCode we had one feature few others even dreamed of: almost complete backwards compatibility with earlier versions of the language.

I don't know why so many other projects don't often prioritize backwards compatibility the way LiveCode does, but it really hurts when that's not there. The gulf between Python 2 and Python 3 threw the whole ecosystem into disarray for years. And in systems like Drupal, and to some degree WordPress, moving between any two major versions means a scramble to replace third-party plugins and rewriting large chunks of custom code.

LiveCoders who don't have to use other technologies often don't realize just how good we have it.

[dunbarx]

I do not use other technologies, and I am acutely aware of how good we have it.

Craig

[richmond62]

The most commonly cited reason is the implementation of native unicode management.

Hot D*mn; that's what really "did it for me" and turned LiveCode into something extra special.

That, AND 1000% the open source version.