convert encode big5 text to utf8

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
terryho
Posts: 126
Joined: Mon Nov 05, 2012 2:53 pm

convert encode big5 text to utf8

Post by terryho » Wed Oct 24, 2018 3:54 pm

Hi

I want to covert the big5 encode text file into utf8 encode text file.

The attach zip file contain big5.txt and uft8.txt file

I read the big5.txt file and use the following method to convert it into utf8, but cannot display correct chinese character.

on mouseup
answer file "A Text file" //(select the big5.txt file)
put URL("file:" & it) into aval
answer textencode(textdecode(aval, "ansi"), "utf8")
end mouseup

The correct encode file "utf8.txt" will display correctly in Livecode.

Can I use Livecode to convert Big5 encode text into unicode encode text file ?

Regards

Terry Ho
Attachments
text.zip
zip contain big5.txt and utf8.txt
(233 Bytes) Downloaded 228 times

richmond62
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 9287
Joined: Fri Feb 19, 2010 10:17 am
Location: Bulgaria

Re: convert encode big5 text to utf8

Post by richmond62 » Wed Oct 24, 2018 6:27 pm

I don't think LiveCode can do that.

Unfortunately this is NOT a problem due to Unicode, it is a problem due to the differing origins of big5 and Unicode:
WARNING! It is currently impossible to provide round-trip compatibility between BIG5 and Unicode.
http://www.unicode.org/Public/MAPPINGS/ ... R/BIG5.TXT

This seems about as close as you can get: http://technology.chtsai.org/b5touni/

although I have 2 questions:

1. Is it confined only to the Windows family of operating systems? (it reallt annoys me when People assume everyone uses Windows).

2. Can the algorithm be extracted and implemented in a LiveCode stack?

I have a feeling that the document that ends in ".c" which contains the source code may be written in C++
(one of the very many programming languages I don't know): if you can understand C++, then . . . 8)

richmond62
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 9287
Joined: Fri Feb 19, 2010 10:17 am
Location: Bulgaria

Re: convert encode big5 text to utf8

Post by richmond62 » Wed Oct 24, 2018 6:38 pm

If you enjoy working very, very hard, for a long, long time, the "other" way to do
things would be to download ALL the Chinese Unicode tables:

https://www.unicode.org/charts/

and ALL the big5 tables:

http://ash.jp/code/cn/big5tbl.htm

and write a mapping routine to convert from big5 to Unicode.

This would involve at least 22,713 mappings.

However, I suspect that would involve a lot of work, and require more than a single person to check for
mistakes and so on, and at least one of the people involved would have to be a Chinese native speaker who KNEW most, if not all, of that characters involved.

Although, theoretically, I could write the code, as I am NOT either a Chinese native speaker, nor even a Chinese speaker/reader, I would not trust my ability to recognise characters in big5 and their equivalents in Unicode.

Unfortunately I don't think there is a simple correspondence between the big5 encoding and Unicode, or
any of the other Chinese encodings.

The other thing that has to be considered very carefully, is whether such an effort is ultimately justified.

Um:

http://glyph.iso10646hk.net/ccs/ccs.jsp?lang=zh_TW

terryho
Posts: 126
Joined: Mon Nov 05, 2012 2:53 pm

Re: convert encode big5 text to utf8

Post by terryho » Thu Oct 25, 2018 9:23 am

Hi

Thanks your info.

I had find that with building of Linux external it can use iconv.dll to convert between different code page

http://newsletters.livecode.com/october ... etter1.php

If Livecode can adopt this dll, which has a window version.

http://gnuwin32.sourceforge.net/packages/libiconv.htm

Then I think it can make Livecode has a function to convert between different code page.

Regards

Terry Ho

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”