revxml

[malte]

Thanks Mark.

We were a bit puzzled, as the same document parses just fine with the old external. So it would be good if the "old" behaviour could be restored.

[mwieder]

Indeed. I am also at a loss to explain what problem the hard limits are attempting to solve.

[malte]

Ok folks, not good that RC2 wasn't in test for long enough to being able to check on. a couple of days more would not have hurt. The external in this release appears quite broken I will set up a test stack with all the things we have found and keep you posted. First findings were that the copyXMLNode and COPYRemoteXML node appear to invalidate the XML. Not good.

Best,

Malte

[SirWobbyTheFirst]

Ok folks, not good that RC2 wasn't in test for long enough to being able to check on. a couple of days more would not have hurt. The external in this release appears quite broken I will set up a test stack with all the things we have found and keep you posted. First findings were that the copyXMLNode and COPYRemoteXML node appear to invalidate the XML. Not good.

Best,

Malte

Uh oh, I just upgraded to 6.6 last night, I am heavily dependent on XML at the moment, any idea what other functions may have been compromised?

[malte]

Hi,

I really can not say in detail yet. My plan was to give it a go over the weekend with RC2 however, timeframe was too short for that. The regular XML stuff appears to be working as far as I can see. revXMLCopyNode and / or revXMLCopyRemoteNode seem to be broken, not sure which of those or both yet.

#11748 is not fixed. Still crashes.

As the scope of the external is rather limited, I will be trying to write a unit test stack for it that tests all commands / functions. This will be a bit of work, but as XML and DB is bread and butter here, it seems to be worth it. If anybody of you wants to chime in setting up such a beast I really would like opinions and maybe a helping hand or 2. For now the only way I see until this is fixed is using the old external. Not ideal...

Best,

Malte

[LCMark]

@malte: Is the breakage of CopyNode / CopyRemoteNode only in 6.6-rc-2 and later - i.e. it was fine in 6.6-rc-1? The reason I ask is that the fix to the 'long XML input' problem didn't touch anything to do with operations on XML, only the act of reading the XML in - so if the tree you get after reading in the XML appears to be sound (before doing any operations on it) I'm a little puzzled as to what has broken.

[malte]

Hi Mark,

this appears to be the other issue I was not able to pin down as mentioned earlier in this topic. The problem was that we could not really see it before as the XML in the app in question is always really large. We are just now finishing the transition of the app from the 5.5 engine towards the 6.x engine, because it is so complex that changing the engine is always pretty risky.

What I am going to do (unless you have something like it yourselves already) is writing a unit test stack that works with our real life XML data. What I imagine is a stack that tests all XML commands and compares them to a known result from an oldr engine where we found stuff stable.

All the best,

Malte

[LCMark]

@malte: Ah! That makes sense... It sounds like moving to the newer version of libxml might be the cause of the Copy(Remote)Node problem and others you are seeing then - now that you can actually load your XML files again!

In regards to the test stack - if you could write something like that then that would be great. If you get in contact with Hanson (quality@runrev.com) then we could even integrate it into the test system we've been developing, so it will get checked on each build we do. (We've not yet reached the point of writing tests to cover externals functionality).

[mwieder]

Malte- indeed. Send me a copy of the test stack as well. I've got a test stack for the xpath and xslt functions, but I didn't expand it to the other xml functions since I didn't change them. Possibly something is broken^H^H^H^Hchanged in the new libxml library.

[malte]

That sounds like a plan.

Will set up this stuff over the weekend and on Monday hopefully. I will keep you posted.

Best,

Malte

[mwieder]

@runrevmark:

revxml no longer compiles after your last change to attempt to handle big data. This is unfortunately in the master branch. You're attempting to call two non-existent functions: xmlSAXParseMemoryWithDataAndOptions() and xmlSAXParseFileWithDataAndOptions().

...lines 245 and 269 of file revxml/src/xmldoc.cpp

Edit: it's from commit 9944ce3e897a57e94b8e7fb05a5a520a05f3d800 from 10 March 2014, which got merged into the develop branch on 14 March by Michael. It's not clear to me yet from gitk how that got merged into master, but my guess is it was part of a mass merge from develop, and so there may well be other things that got merged in before their time.

[mwieder]

I updated the submodules and got past the error. I could have sworn I had updated them previously... possibly I wasn't on the master branch in the thirdparty directory when I did it.

I hate git submodules.

[malte]

Quick update on the unit test thingy. This is a little more involved than I would have hoped, but making progress. I still could not boil the anomaly I have seen down to atomic level. It fails all the time in the application (suddenly ending up with parts of script that is in a library in the XML), but as soon as I do the same thing isolated from the app, it suddenly works as expected. This is the kind of thing that really drives one nuts... That said, still working on it.

Best,

Malte

[mwieder]

Malte-

I experimented a bit here with linux mint 14 with the test stack you attached to bug 11903. I've got some interesting results:

LC 5.5.4:
bytes read from file: 12999879
xml tree data size: 12999735
length of field data: 9749801

LC 6.6:
bytes read from file: 12999879
xml tree data size: 12999735
length of field data: 9749801

LC 7.0:
bytes read from file: 12999873
xml tree data size: 12999735
length of field data: 9499801

Where the xml tree data size is reported just before the base64encode line in your script. So the xml tree creation looks to be producing the same tree in all the incarnations, but the base64encode statement is producing truncated data in LC 7.0. I'm also mystified as to why a binary file read would be reporting 6 fewer bytes in 7.0 (the actual filesize is 12999879), but that's possibly a different bug. I *am* seeing the field populated in all three LC versions.

I filed bug 12190 on the base64encode truncation and 12191 on the file size discrepancy.

[LCMark]

@malte: Oh dear - this sounds like its going to be a bit of a mare to track down. It sounds like a memory overrun related thing if data that should not be in the XML is appearing in the XML data. Which platforms does you application run on? If it runs on Linux then we can try valgrind'ing it in the office to see if we can track down the issue (sometimes valgrind works on Mac too - but it has various issues which make it somewhat difficult) - contact Hanson at quality@runrev.com and we'll sort something out.

Also, don't forget that these problems seem to have been introduced with the libxml2 library upgrade in 6.5 - if you aren't using the xslt functions, then you can just revert to the pre-6.5 XML external... Just copy the files from pre-6.5 into the appropriate places in Externals and Runtime.

@mwieder: Thanks for noticing these issues with 7.0 - I've seen the bug reports in the DB, so the guys working on 7.0 will get to them soon