NumberFormat and rounding

[dunbarx]

Since when did the numberFormat property become a rounding function?

Code: Select all

on mouseUp
   put 1.2345678 into temp
   answer the length of temp
   set the numberFormat to "0.##"
   put temp * 1 into temp ----Rounds to 1.234568

  -- side issue follows
   answer the length of temp
   answer the length of temp && char 6 of temp
end mouseUp

If you step through this, and watch the variable temp, it rounds the last two decimals at line 4, either up or down as appropriate. (Appropriate??)

As a side issue (this based on something Colin posted on the use-group) the numberFormat property should only affect the number of chars displayed. In HC, if I step through the above, the numberFormat truncs the value to two decimal places immediately. In LC, it does nothing (except for that rounding thing). Isn't there a disconnect between the value shown in the debugger (eight chars, never mind it should be 9) and the value you get when you try to do anything with that value, such as execute that last line, or place the variable into a field. By disconnect, I mean "when is that value really truncated? Or rather, why is it not truncated as shown in the debugger?

Craig Newman

[jacque]

I mentioned this on the list, but... numberFormat is only applied when the variable becomes a string. Placing the value into a field makes it a string. Since the variable watcher is comprised of fields, the values are strings and you see the effect of that. This is probably a sort of bug, though I think it could be tricky for the editor to work around it because the transformation happens in the engine and the script editor is just a stack.

NumberFormat does do rounding if the requested format is shorter than the variable length, which is different than HC.

The last two lines of the example are puzzling at first, but it looks like getting the length of a variable requires it to be converted to a string first (numbers don't have "length" per se.) So the engine does that before calculating the length, but the numberFormat is still in effect, so you get 1.23 as the value of temp -- which is 4 characters long.

Adjusting the numberFormat fixes it:

Code: Select all

 -- side issue follows
  set the numberformat to "0.#############"
  get the length of temp
  get the length of temp && char 6 of temp

[dunbarx]

Jacque.

I keep forgetting that all IDE gadgetry, like the script editor. are just stacks, and therefore their fields behave like fields. This thinking, which I must abandon, goes back to HC, where such things were XCMDS.

Thanks.

Craig

[FredBeck]

What we need is a statTrunc fonction :)