Not a number is a number, problem

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

johnf923
Posts: 18
Joined: Tue Feb 23, 2021 8:20 pm

Re: Not a number is a number, problem

Post by johnf923 » Tue Mar 23, 2021 12:12 am

Right: I think I've worked on this as much as I can so I hope you'll let me have a go at explaining my take on the issues in this thread. I'm sorry it's such a long post but I've tried to be as clear as possible. You're allowed to disagree with that...

## 1. Floating Point Arithmetic, Numeric Comparisons and NaN

The rules of floating point arithmetic are laid down in the IEEE 754 standard. They are built into the hardware of just about every computer that supports floating point and anyone who wants to learn a new language is likely to find the arithmetic rules easy to get to grips with because they are the same across the board.

The LiveCode engine, in nearly every case, follows the standard rules. The engine is written in C++, and the numeric expressions are evaluated using the corresponding C++ operators and functions, thus guaranteeing compliance with IEEE 754. One of the standard rules involves the special value NaN, which is actually a range of values used for error detection and other purposes. The standard says that when NaN appears on either or both sides of an arithmetic comparison operator all the comparison operators must return false, except for <> which must return true. This may seem non-intuitive at first; nonetheless it’s an integral part of the standard.

I said that LiveCode follows the IEEE 754 rules in “nearly every case” because the LiveCode engine does not correctly follow the rules regarding numeric comparisons and NaN. I submit that this is an error in LiveCode because it is out of step with the rest of the world. In saying this I am only expressing my support for bug report 22274 which has already addressed the issue.

For some further reading on NaN, see https://ntgard.medium.com/why-nan-nan-3d41af988d30. That blog entry discusses NaN in the context of JavaScript but because of the universal nature of floating point arithmetic it is just as valid for LiveCode.

## 2. Perils of Numeric Comparisons

When strings like NAN and INF appear in a collection of data the < and > operators will not work as one might at first expect because these strings will be converted to numbers. The question that started this thread gave the use case of working with Australian stock market codes, which Wikipedia describes as three-character strings with alphanumeric characters as the first and third characters, and an alphabetic character between them. Different stock exchanges have different rules but if we stick with the Australian use case here are some examples of codes that might or might not be in use yet but could be some day. The first five of these are of the digit-letter-digit form and they are only five out of a total of 116 similar possibilities.

Code: Select all

0X0, 0XA, 4E3, 5E2, 6E1, INF, NAN
All of these will be treated as numbers by the < and > operators, so the issue is much wider than the interpretation of NAN.

If we consider just the (4E3, 5E2, 6E1) set, then under the usual rules for weakly typed languages they can be taken as numbers so they will be, and we find that

6E1 < 5E2 < 4E3 (in decimal, 60 < 500 < 4000)

But consider the very similar set with H instead of E. These cannot be taken as numbers so they will be compared as strings and we find that

4H3 < 5H2 < 6H1

The order appears reversed but this is not an error. It is simply a consequence of the rules of the language.

## 3. String Comparisons

The issue of comparing string values (for example to see which of two strings would appear earlier in a dictionary) is (IMHO) somethng that’s made more difficult by LiveCode’s lack of a dedicated string comparison operator. There are (at least) two problems:

1. Because the LiveCode comparison operators convert strings to numbers where possible, the idea of alphabetical ordering breaks in the face of values that can be interpreted as numbers. A useful string comparison method must suppress the automatic conversion of these values.

2. There are many ways of comparing strings. LiveCode’s sort command, for example, allows a sortType parameter which can be international, numeric, datetime, text or binary. Other languages offer other possibilities, including what’s sometimes known as dictionary sorting where the string A20 will sort before the string A100.

I believe LiveCode would benefit from a new string comparison operator. One possibility would be
this syntax:

Code: Select all

<string> sorts [sortType] [by sortKey] [not] before|equal|after <string>
to emphasize the similarity with the sort command and to give the same flexibility. As with existing operators, the new operator would also respect the caseSensitive property.

## 4. Summary and Workround

I would like to see string comparison implemented along the lines I have described, but in the meantime it is still possible to enforce string comparison with the construct

Code: Select all

put ("x" & string1) > ("x" & string2)
It may not be elegant but it is at least effective.

Davidv
Posts: 77
Joined: Sun Apr 09, 2006 1:51 am
Location: Australia

Re: Not a number is a number, problem

Post by Davidv » Tue Mar 23, 2021 2:39 am

That is interesting information about IEEE 754 treatment of NAN in maths processing (albeit not all unknown to me). My considerable lack of conviction has nothing to do with the description but with the implied assumption and ultimately with the supposed benefit.

A straw poll:
Would all of those who have found a need within LiveCode scripting to compare NAN with NAN as IEEE 754 conforming numerics please say so? I will take it that all non-answers are straws on the other side.

Secondly, if you did that numeric form NAN comparison, I would like to know in what context you managed it? Where does LIveCode throw NAN as a result rather than supplying a consequent error message? Please show the code example (relevant documentation in LiveCode would be handy too).

In the message box and in a mouseUp script, bare and within a Try...Catch structure, I tried to deal with and to compare sqrt(-1). Undoubtedly LiveCode was managing NAN in its code because errors were thrown, messages provided, but none comprising "NAN", none including the string of characters NAN, and no comparisons were undertaken owing to the error. Hence my previous question, I would like to know how it is done.

Try these bits of code, in the message box, multiline message box, or a mouseUp handler:

Code: Select all

get sqrt(-1)

Code: Select all

put sqrt(-1) = sqrt(-1)

Code: Select all

try
	answer sqrt(-1)
catch mice
	put mice into msg
end try
Were any mice actual NANs, rather than consequential error responses?

Meanwhile, the current behaviour of LiveCode is that every use of NAN in sort ordering and (relatedly) string comparison, with itself or adjacent strings like NAM and NAO, works as one would expect for strings with the sole exception of NAN > NAN. It seems Livecode needs change only one thing for NAN to work perfectly as a string, where it invariably occurs.

Alternatively, we could introduce a dedicated string comparison to LiveCode or make sure every string comparison we ever do concatenates another character just in case the string NAN turns up, with the benefit that we solve a problem which, until someone can give me a specific LiveCode example, appears not to exist at all.

And yes, I am questioning entirely the relevance of bug 22274 for LiveCode scripting. At a glance, many of Dar's results appear a consequence of of LC doing string comparisons, right in that context, wrong for IEE 754 maths processing, which in LC is under the hood.

David

rkriesel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 118
Joined: Thu Apr 13, 2006 6:25 pm

Re: Not a number is a number, problem

Post by rkriesel » Tue Mar 23, 2021 3:14 am

johnf923 wrote:
Tue Mar 23, 2021 12:12 am
...
I believe LiveCode would benefit from a new string comparison operator. One possibility would be
this syntax:

Code: Select all

<string> sorts [sortType] [by sortKey] [not] before|equal|after <string>
to emphasize the similarity with the sort command and to give the same flexibility. As with existing operators, the new operator would also respect the caseSensitive property.
...
Another possibility is very similar to the caseSensitive property: the typeSensitive property. TRUE would suppress type conversion for all comparisons.
This possibility would obviate new operators, reducing the load on the LC team and the developers.
Thanks for your post, johnf923.
-- Dick

Davidv
Posts: 77
Joined: Sun Apr 09, 2006 1:51 am
Location: Australia

Re: Not a number is a number, problem

Post by Davidv » Tue Mar 23, 2021 3:53 am

I spent some minutes after my earlier post checking something.

NAN is not in the Dictionary.

NAN is not in the User Guide.

These make sense because NAN is not in LiveCode.

So, why document something not in LiveCode language, regardless of whether it is used under the hood in the engine? Does your vehicle user guide document the axle bearings for the driver?


Even if NAN were special, every other command or literal in LiveCode is capable of being compared correctly when quoted as a string.

So, why not NAN?


I am very interested in answers which address the question of relevance before dreaming up [in]elegant fixes for an apparent non-issue. What is the value to LiveCode scripting of defining NAN in the Dictionary, contorting the rest of the language around it, if it is never used in the script language? Examples of the LiveCode problem first, please.

SparkOut
Posts: 2839
Joined: Sun Sep 23, 2007 4:58 pm

Re: Not a number is a number, problem

Post by SparkOut » Tue Mar 23, 2021 9:15 am

Well this is further distillation of the issues such that the internal comparisons of NaN in the engine are not always compliant with standards. This probably should be fixed, but doesn't really affect me that I can tell*, just for the sake of compliance really.
Documentation is lacking, but NaN is not an operator in LiveCode Script. There's no glossary entry even or reference in passing. If there is a new concept introduced to the language you would expect some notes, such as with 'infinity'. There's no need to document engine-only operators*.
*The NaN operator bleeds out of the engine to affect LiveCode Script in certain specific cases, where a string is being interpreted as a "number" for comparison. This shouldn't happen, and doesn't always. (Seemingly it does when it is taken as an item from a list.)
John highlights that other comparisons also first attempt to convert operands to numbers. I can see the difference in result making sense, with this knowledge. But I am still fundamentally against the engine taking a value I have defined as a string and applying it as a number without my explicit intent.

johnf923
Posts: 18
Joined: Tue Feb 23, 2021 8:20 pm

Re: Not a number is a number, problem

Post by johnf923 » Tue Mar 23, 2021 1:08 pm

Davidv wrote:
Tue Mar 23, 2021 3:53 am
NAN is not in the Dictionary.

NAN is not in the User Guide.

These make sense because NAN is not in LiveCode.
Just because you didn't see it doesn't mean you'll never meet it. Try this in a button script:

Code: Select all

on mouseUp
   put 1e10000 into a
   put 1 / a into b
   put a / b into c
   put c - c into d
   answer "c =" && c & "; d =" && d
end mouseUp
(To those who may not have the IDE handy, the script demonstrates one of the myriad ways of encountering both INF and NAN in LiveCode.)

The omission of NAN from the dictionary is another matter that needs fixing.

Davidv
Posts: 77
Joined: Sun Apr 09, 2006 1:51 am
Location: Australia

Re: Not a number is a number, problem

Post by Davidv » Tue Mar 23, 2021 11:58 pm

Code: Select all

put 1e10000 = infinity
--> true

or

Code: Select all

put "n & "a" & "n"
--> nan

Easy to do. It depends with what you start.

However, no-one has put their hand up for practical working examples or relevance.

nan is not a reserved word in LiveCode, unlike infinity (not INF).

However, as it states in LiveCode documentation, quoting infinity ("infinity") treats it as a literal string like treatment of every other command or reserved word.

So why not nan? Which is not even a reserved word, no matter the myriad ways of generating that string.

LiveCode gets it right mostly but a little oil has leaked from the engine making a mess on the floor, where "nan" > "nan". The leak needs fixing, because the controls and documented dashboard are otherwise fine.

Or maybe LiveCode needs strcmp. This may help.

Code: Select all

function strcmp a,b
	return a = b
end strcmp
Works for everything. :)

LiveCode is C in the sense that I am atoms. It is not a typed language. Any documentation of the ways of generating nan has nothing to do with the discovered bug that comparing nan strings where nan > nan fails. The fix is to compare string literals correctly, no new typing or functions required.

Davidv
Posts: 77
Joined: Sun Apr 09, 2006 1:51 am
Location: Australia

Re: Not a number is a number, problem

Post by Davidv » Wed Mar 24, 2021 4:07 am

This may help to clarify my comments, again.

It was back on page 2 that I noted the sole reference to NaN that I could find for LC was in release notes for 9.5.0 dp-1 (hereafter RN9.5). It was most likely those 9.5 comments which triggered Dar into submitting 22274 (Review), 22275 (Resolved Unsuitable) and 22291 (Hibernated). I can see some theoretical sense in the suggestion within hibernated 22291 that by default LC not raise maths exceptions, as clearly defined in RN9.5, but perhaps have a property which allows maths exceptions to propagate into the code, although no particular reasons for that to happen are advanced.

Currently, the position as described in RN9.5 is that there are three execution errors which can be thrown.
  • numeric: domain error
  • numeric: range error
  • numeric: divide by zero
Execution errors are not thrown when the input is non-finite, e.g. sqrt(-infinity) = NaN.

None of these are engine errors. They relate to common code for error checking and handling. NAN is not among the exceptions thrown although it can cause them (e.g. a domain error).

Now, if we were to take up 22291 then we would expect that NaN would be added to Infinity as reserved. In that case, we would expect it to behave as a string literal when quoted, just like Infinity does. In either case (22291 or not) it is a bug if either of nan or infinity fails to behave like a string literal when quoted.

Currently, nan fails that test in one of three possible comparison cases. That is what needs fixing.

Until such time as nan is reserved, should that ever eventuate, then nan is strictly a string, precisely as LiveCode says today.

Code: Select all

put nan is strictly a string
--> true
Note I did not even quote nan. You will find false for nan for all of the other options in strictly in the dictionary

johnf923
Posts: 18
Joined: Tue Feb 23, 2021 8:20 pm

Re: Not a number is a number, problem

Post by johnf923 » Wed Mar 24, 2021 6:17 pm

Well, let's leave it there. Good luck with the project!

Post Reply

Return to “Talking LiveCode”