Chi-square

[danielrr]

This is a long shot, I know: is there a LiveCode stack already mad that helps the user to perform the chi-square test on the user on his or her own data?

[danielrr]

To make any possible answer easier, what I try to do is perform the chi-square test on the distribution of a certain feature along the year. The expected results (given that the feature is climate independent, etc) would be the same for all 30 days months (30 * total) / 365, 31 days months (31 * total) / 365 and february (28 * total) / 365.
The actual results are as follows:
Jan. 376
Febr. 469
März 518
Apr. 470
Mai 546
Juni 618
Juli 614
Aug. 517
Sept. 453
Okt. 423
Nov. 459
Dez. 373

[stam]

I'm not aware of a statistics library for LC and would welcome one myself as well...
I wonder if it's possible to use the R command line app?

[stam]

You may find these links helpful if wanting to recreate in LiveCode:

https://www.simplilearn.com/tutorials/s ... quare-test
https://visualstudiomagazine.com/articl ... -test.aspx
https://www.codeproject.com/Articles/43 ... ed-P-Value

The last URL provides a full code listing of calculating this in C - which I've tried to replicate in LiveCode but getting some overflow errors as it seems there are numbers too large for LC to handle so had to modify the code a bit and I'm not sure it's that accurate.

If you have any know-how, or know someone who can, I'm sure you can get someone to port this to a library external.
Or give the code below a try - it might be a pile of poo because LC <> C, but maybe you can modify it and make it work better.
I literally just transcribed this to LiveCodeScript as best I could - mainly out of idle curiosity

The inputs (obsered/expected) are comma-separated lists, but you can change that:

Code: Select all

local sExpected, sObserved, sDoF, sCriticalValue

# sExpected, SObserved are comma delimited lists
function chiSquare pObserved, pExpected
    local K, X, peeValue
    set the numberformat to "0.##################"
    setValues pExpected, pObserved
    if sCriticalValue < 0 or sDoF < 1 then return 0
    put sDoF  * 0.5 into K
    put sCriticalValue * 0.5 into X
    //special case
    if sDoF = 2 then return exp(-1.0 * X)
    put igf(K, X) into peeValue
    if peeValue is not a number or peeValue = infinity or peeValue <= 1e-8  then return 1e-14;
    divide peeValue by gamma(K)
    return 1 - peeValue
end chiSquare

command setValues pExpected, pObserved
    if pExpected is empty or pObserved is empty then exit to top
    put pExpected into sExpected
    put pObserved into sObserved
    put getCriticalValue() into sCriticalValue
    put getDegreesOfFreedeom() into sDoF
end setValues

function getCriticalValue // test statistic
    local tDiff, tValue
    repeat with x = 1 to the number of items of sObserved
        put item x of sObserved - item x of sExpected into tDiff
        add ((tDiff * tDiff)/item x of sExpected) to tValue
    end repeat
    return tValue
end getCriticalValue

function getDegreesOfFreedeom
   return the number of items of sObserved - 1
end getDegreesOfFreedeom

function igf S, Z
   local Sc, tSum = 1, Nom = 1, Denom = 1
   if Z < 0 then return 0
   put 1/S into Sc
   multiply Sc by  Z^S 
   multiply Sc by exp(-1 * Z)
   repeat with x = 0 to 160 // 200 causes overlow so reduced to 160
       multiply Nom by Z
       add 1 to S
       multiply denom by S
       add (Nom/Denom) to tSum
   end repeat
   return  tSum * Sc
end igf

function gamma N
    local K, Z, Sc, F = 1, Ck, tSum 
    constant A = 15
    constant SQRT2PI = 5066282746310005024157652848110452530069867406099383
    put N into Z
    put (Z+A)^(z+0.5) into Sc
    multiply Sc by exp(-1*(Z+A))
    divide Sc by z
    put SQRT2PI into tSum
    repeat
        Add 1 to z
        Add 1 to k
        put (a-k)^(k-0.5) into Ck
        multiply Ck by exp(A-K)
        divide Ck by F
        add (Ck/Z) to tSum
        multiply F by (-1*K)
        exit repeat if K >= A
    end repeat
    return tSum * Sc
end gamma

--function gamma Z //  approx_gamma algorithm
--    local D, RECIP_Z
--    constant RECIP_E =  0.36787944117144232159552377016147
--    constant TWOPI = 6.283185307179586476925286766559
--    put  (1/Z) into RECIP_Z
--    put 0.1 * RECIP_Z into D
--    put 1/((12*Z) - D) into D
--    put (D+Z) * RECIP_E into D
--    put D^Z into D
--    multiply D by sqrt(TWOPI * RECIP_Z)
--    return D
--end gamma

[FourthWorld]

Nicely done, Stam. Thanks for posting that.

Back in the open source era I'd hoped it might last long enough to attract contributions for a stats lib and others. Much better than shelling out to some other language. Maybe this will inspire more.

[stam]

Nicely done, Stam. Thanks for posting that.

Back in the open source era I'd hoped it might last long enough to attract contributions for a stats lib and others. Much better than shelling out to some other language. Maybe this will inspire more.

Thanks Richard.

Long doubles seem to be a datatype that cause overflow errors in LC and it's probably not the right tool for statistics.
I had to reduce a repeat loop from the recommended 200 interactions to 160, greatly reducing accuracy, because it kept crashing with "overflow error" until I did this.

That's why I still think R is the best solution - it's an extremely well established FOSS CLI app with vast scientific following and support. There is literally no statical method you can't do with this. Sadly I never really learned it in depth but may do at some point. There are some graphical interfaces for the CLI and I can't really see reason why this couldn't be done with LiveCode either.

Maybe a project for the future....