Issues with Encoding UTF-8 JSON Files in LiveCode for Python Compatibility

[Fermin]

Hello everyone,

I’m working on a project where I need to export configuration data from LiveCode to a JSON file. The ultimate goal is for this JSON file to be compatible with a Python program that reads and processes it. The challenge is that the data includes special characters (such as accented letters in Spanish), and I need to ensure the JSON file is encoded in UTF-8 so it can be read properly by Python.

Here’s the problem I’m facing:

I’m using ArrayToJSON to convert LiveCode arrays to JSON format.
When I attempt to save the JSON output to a file (using methods like put ... into url "binfile:" or revFileOpen), the special characters appear corrupted in the saved file.
I've tried various approaches, such as:

Using textEncode to convert the JSON text to UTF-8 before saving.
Writing with revFileOpen to directly handle binary encoding.
Despite these attempts, the special characters (like á, é, ñ) still display incorrectly in the output JSON file.

My questions are:

Has anyone successfully managed to export JSON with special characters in UTF-8 encoding from LiveCode in a way that is fully compatible with Python?
Are there any recommended workarounds or specific functions that ensure UTF-8 encoding for JSON output in LiveCode?
Any advice or examples would be greatly appreciated. Thank you!

[stam]

I thought th the internal methods would respect UTF-8, no?

In any case a very nice alternative to the internal method is this 3rd partly library from FerrusLogic:
https://github.com/Ferruslogic/PhotonJSON

Certainly does respect UTF-8...

Stam

[Fermin]

Thank you, Stam, for your recommendation. I appreciate the suggestion of FerrusLogic's PhotonJSON library, especially since it respects UTF-8. While I am still exploring and investigating alternative methods to ensure compatibility with UTF-8, your advice is valuable and certainly something I will consider as I proceed with my project.

Best regards,
[Fermin]

[bn]

Fermin,

Would this work for you:

Code: Select all

on mouseUp
   put "á, é, ñ" into tText
   put tText into tDataA["Text"]
   put arraytoJson(tDataA) into tJson
   put textDecode(tJson, "utf-8") into tJsonDecoded
   -- save tJsonDecoded
end mouseUp

Kind regards
Bernd

[Fermin]

Hello Bernd,

Thank you for your suggestion on handling UTF-8 encoding in LiveCode. Although I haven’t tested it exhaustively yet, it seems that the compatibility issue with special characters in JSON files is resolved, which simplifies their use in my text animation application in Python.

Much appreciated.

Best regards,
Fermín

[bn]

Hello Fermin,

I feel a bit unconfortable with the hack I suggested for "arrayToJson"

Currently there are two ways to convert an array to a json file in LC:

First using what we were talking about:
arraytoJson
jsontoArray
This uses the mergExternal

Second:
Jsonexport
JsonImport
This uses com.livecode.library.json

I tried the very simple example to create a json with non-ascii characters and it worked.

Maybe you could try that version of creating json and see if it works for you.

I consider the workaround I suggested a workaround for a bug in arrayToJson.

I filed a bug report for this and unless it is decided that it is not a bug I would not use the workaround because it would probably break in case the "bug" is fixed.
https://quality.livecode.com/show_bug.cgi?id=24620

Kind regards
Bernd

[stam]

Or you can just use PhotonJSON - this works surprisingly well and it's quite fast...

[bn]

Hello all,

Mark Waddingham (CTO of LC) responded to the bug report:

https://quality.livecode.com/show_bug.cgi?id=24620

Thanks for the report.
This is not a bug - the documentation for ArrayToJSON does state in its return value entry 'A UTF8 encoded JSON string'.
i.e. it returns UTF-8 encoded text and thus needs a textDecode() to be turned into a normal LiveCode (Unicode) string.
Warmest Regards,
Mark.

So it turns out that I misunderstood the documentation and what I considered a workaround is actually the way to go.

@Fermin: You should be safe to use the "workaround".

Kind regards
Bernd

[Fermin]

Thank you very much for all your help. It is very reassuring to be able to count on this forum.