CheckSum Verification Failure

[dcpbarrington]

LiveCode Forum,

I'm saving a group of files and want to verify if the file contents has changed. When I create the file I calculate a MD5 CheckSum with the following function.

Code: Select all

function hexDigest pvalue 
   local tRes 
   put md5Digest(pValue) into tMD5 
   get binaryDecode("H*",tMD5,tRes) 
   return tRes 
end hexDigest 

I place the checkSum in the first line of the file and save the file with put tFullOutput into URL ("file:" & pFileName)

Now I also save the CheckSum in an array for later reference. To determine if the file has changed I use the following function.

Code: Select all

function validLocalStore pCheckSum
   
   put getLocalStoreName( ) into tFileName
   -- Open the Cach File
   put URL( "file:" & tFileName ) into tDataInput
   if tDataInput is Empty then
      return "ERROR: File Empty"
   end if
   
   put line 1 of tDataInput into tStoredCheckSum
   replace CR with Empty in tStoredCheckSum
   replace LF with Empty in tStoredCheckSum
   replace CR with Empty in pCheckSum
   replace LF with Empty in pCheckSum
   
   if tStoredCheckSum = pCheckSum then
      return "VALID"
   else
      return "ERROR: CheckSum Does Not Match"
   end if
   
end validLocalStore

Here is an example of a checkSum that the function says that it is NOT VALID
Array CheckSum = a07dd580783edd56c26dd4555e2b4b95
File CheckSum = a07dd580783edd56c26dd4555e2b4b95

As far as I can tell they are IDENTICAL, but the IF statement always says that they are not the same.

When I change the put URL( "file:" & tFileName ) into tDataInput ---> put URL( "binfile:" & tFileName ) into tDataInput

The IF statement says that the two strings are the same.
There is something going on that I don't understand.
I thought it was the CRLF, but I have done a replace statement for each.

Any Ideas??????

[Mark]

Hi,

Try this at the beginning of your script:

Code: Select all

function validLocalStore pCheckSum
   put getLocalStoreName( ) into tFileName
   -- Open the Cach File
   put URL( "file:" & tFileName ) into tDataInput
   put the result into rslt
   if tDataInput is empty then
      return "ERROR: File Empty"
   else if rslt is not empty then
      return "ERROR:" && rslt
   end if

Do you get an error?

Kind regards,

Mark

[Cairoo]

Hello dcmbarrington,

This looks like a classic null-terminated string problem. LiveCode doesn't seem to handle null-terminated strings very well. To check whether you have a null-terminated string in a variable, try concatenate some text to the variable. If concatenation fails, then you have a null-terminated string.

If indeed you have a null-terminated string then you have to remove the null-terminator before using the variable in further text operations, or else it might drive you insane. But removing a null-terminator from a string in LiveCode is not an easy task! There is a way, but it's complicated and would be too much trouble if in your case you've already found a way around it, even when your workaround doesn't make sense.

Regards,

Cairoo

[Mark]

Hi Cairoo,

Try

Code: Select all

put "a" & NULL & "b" 

in the message box. What makes you think that you can't concatenate strings ending with NULL?

Although it is possible that either string contains an invisible character, I consider this quite unlikely, having looked at the scripts.

Kind regards,

Mark

[Cairoo]

@Mark: I know about the null-terminated string issue because I've had to deal with it myself before. I had a variable in LiveCode that received a string from a JavaScript function in a browser. LiveCode simply REFUSED to concatenate anything to that variable because it had a null-terminator, which I could only remove using a JavaScript function that spliced the string into its individual bytes and rejoined the bytes leaving out the null terminator. You cannot reproduce the problem by simply concatenating NULL in LiveCode.

[Cairoo]

Oops, I take it all back... It was JavaScript that refused to concatenate anything to a null-terminated string, not LiveCode! Sorry guys, it's my bad.

[Mark]

Hi Cairoo,

Actually, I could imagine a case where NULL's would cause a problem, that's somewhat similar to the concatenating problem you describe. Sometimes, if you set the unicodeText of a field to a corrupted string of unicode data, the string is usually displayed correctly up to the last correctly positioned NULL character. This isn't what OP described, though (as far as I can see).

Kind regards,

Mark

[dcpbarrington]

Mark,

I added your code to verify that there is not a File Read Error.
I tested the function and there was not an error.

I think that the problem is that the CheckSum was being written by a different system which was leaving the CRLF at the end of the CheckSum.
I have tried to verify that everywhere it is being written the CRLF is being removed.

Thanks for the help.

dcp

[dunbarx]

I played around with this for just a second, back when it seemed that the strings read the same but did not pass muster. I tested in one way simply by measuring the length of each string, knowing that a CR or even a null constant (both perhaps invisible) would add to the count. They were equal. I suspected, though, that some invisible char or chars was making the strings unequal...

On another note, null is so ephemeral, that if I click on its entry in the dictionary, nothing comes up.

Now that is a shady character.

Craig Newman

[FourthWorld]

On another note, null is so ephemeral, that if I click on its entry in the dictionary, nothing comes up.

There's a missing chunk of XML in the Dictionary files. This has been reported, and is expected to be fixed in the next build.

[Simon]

I recently came upon another data-point.

I see a difference when using
put url("binfile:"...
and
open file tFilePath for binary read
They return different checksum values.
Not sure if that helps.

Simon

[FourthWorld]

I see a difference when using
put url("binfile:"...
and
open file tFilePath for binary read

What is the difference in the data read with each method?

[Simon]

Hi Richard,
It was only the checksum value of opening the same file using the two methods.
I didn't think that they would be different.

Simon

[FourthWorld]

Hi Richard,
It was only the checksum value of opening the same file using the two methods.
I didn't think that they would be different.

If the checksum value is different, the data going into the checksum value is different.

Were you able to determine how they differed?

[Simon]

If the checksum value is different, the data going into the checksum value is different.

Were you able to determine how they differed?

Ahh... that was the question.
Actually I haven't taken it apart yet. Used the same sqlite file for testing.
The complete open file;

Code: Select all

 open file theFilepath for binary read
      read from file theFilePath until end
      put it into theBinaryData
      close file theFilePath 
      put hexDigest(theBinaryData) into tCheck

Not anything special.

Simon