unicode (utf-8) over socket

Got a LiveCode personal license? Are you a beginner, hobbyist or educator that's new to LiveCode? This forum is the place to go for help getting started. Welcome!

Moderators: heatherlaine, Klaus, FourthWorld, kevinmiller

Post Reply
pajito
Posts: 23
Joined: Thu Apr 13, 2017 8:08 pm

unicode (utf-8) over socket

Post by pajito » Sun Nov 12, 2017 1:18 pm

Hi everybody,
I am facing a problem with unicode - unidecode etc, I know it has been a classic problem but still it gives me a hard time. So as you can understand from the uploaded file in the end of this post, I load a csv with unicode(utf-8) greek characters, in fld"1" , at first I couldn't even read the characters correct,then I found the following code, so the greek characters were properly presented in my fld.

Code: Select all

 answer file "Select a file"
   put it into tFile
   put url ("file:" & tFile) into tUnicodeText
   set the unicodeText of field 1 to uniEncode(tUnicodeText,"UTF8")
   
   put url("file:" & tFile) into fld 1


Then I had to split those data into fields so i can send them to a socket, I managed to do that, and managed to send them in greek with the following code

Code: Select all

 repeat with x=1 to the number of lines  of field "1"
      add 1 to counter
   end repeat
   repeat with x=1 to the number of flds of group "pl"
      add 1 to counter
      put textencode(item 2 of line x of fld 1,"UTF-8") into fld x of grp "pl"
   end repeat


The problem is that the data in the fields are again presented """chinese""" with all the respect to Chinese characters. I tried again to add the first code, and the "chinese" turned to greek, but only "question marks" come to the socket.

Any suggestions
Thanks in advance
Ioannis

https://ufile.io/spmql <---- Project
https://uploadfiles.io/kac60 <---- Csv FIle

Klaus
Posts: 10425
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: unicode (utf-8) over socket

Post by Klaus » Sun Nov 12, 2017 5:11 pm

Hi Ioannis,

ouch, lots of bad things!

1. Do NOT and NEVER name your LC objects with a number! -> field "1"
The engine does not differ between numbers and strings in that case and think you will mean the field with LAYER 1 (first field on the card) which may be different from its actual layer!
I used -> field "f1" in my script

2. In the script of that field you have a "mouseover" handler.
This is not a valid message in LC, maybe you mean "mouseenter"?

3. "uniencode" etc. has been deperecated since version 7 in LC.

4. Your CSV file is in fact saved as UTF-16! 8)

5. This does work:

Code: Select all

on mouseUp
   answer file "Select a file"
   put it into tFile

   ## We need to treat unicode stuff as BINARY data, which it is!
   put url ("binfile:" & tFile) into tUnicodeText
   
   ## Always lock screen when updating contents of field(s) to speed things up!
   lock screen
   
   ## See above...
   # set the unicodeText of field 1 to uniEncode(tUnicodeText,"UTF16")
   set the Text of field 1 to textdecode(tUnicodeText,"UTF16")
   
   ## We already have the content of that file in the field!?
   ## put url("file:" & tFile) into fld 1
   set the itemDelimiter to ";"
   delete line 1 of fld 1
   delete line 2 of fld 1
   
   ## Since you do not use the variable COUNTER in this script, we do not need it!?
   #   repeat with x=1 to the number of lines  of field "f1"
   #      add 1 to counter
   #   end repeat
   repeat with x=1 to the number of flds of group "pl"
      # add 1 to counter
      # put textencode(item 2 of line x of fld 1,"UTF-8") into fld x of grp "pl"

      ## The text in fld 1 already has the correct format, so we only need to distribute ist content as is!
      put item 2 of line x of fld "f1" into tText
      put tText into fld x of grp "pl"
   end repeat 
    
   unlock scfeen
end mouseUp
Tested and woirks!


Best

Klaus

pajito
Posts: 23
Joined: Thu Apr 13, 2017 8:08 pm

Re: unicode (utf-8) over socket

Post by pajito » Sun Nov 12, 2017 5:57 pm

Hi Klaus

first of all I need to apologize :oops: for the UTF-16 file and for forgetting to mention that my socket accepts only UTF-8 encoding. Your code of course works, I get the greek characters correct both in field"f1" and all fields in grp "pl", but when I send them over the socket again they are displayed as question marks.
I saved the file in UTF-8 encoding, made some changes both in my script and your, but still ???? come out of the socket. Just an idea, is there a way to encode the socket itself, so we can use the rest of the code as it is?
As it concerns fld 1, I made this mistake lot and lot of times, but still some times I do it again.

Thanks again
Ioannis

Klaus
Posts: 10425
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: unicode (utf-8) over socket

Post by Klaus » Sun Nov 12, 2017 6:19 pm

Hi Ioannis,

try with urlencode and HTMLtext before sending over the socket like this.
Field "f1" contains the greek text:

Code: Select all

on mouseUp
   set itemdel to ";"
   put urlencode(the htmltext of item 2 of line 2 of fld "f1") into tData
   set the htmltext of fld "f2" to urldecode(tData)
end mouseup
This way you send simple ASCII over the sockets and that should work.


Best

Klaus

pajito
Posts: 23
Joined: Thu Apr 13, 2017 8:08 pm

Re: unicode (utf-8) over socket

Post by pajito » Sun Nov 12, 2017 7:11 pm

Hi again

Unfortunately no, still question marks are coming out. I will give it a try again by changing or merging the scripts, hoping fot the best.

Thanks for your time Klaus
Ioannis

Klaus
Posts: 10425
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: unicode (utf-8) over socket

Post by Klaus » Sun Nov 12, 2017 8:36 pm

Can you plase post the script(s) for reading/writing to/from socket?

pajito
Posts: 23
Joined: Thu Apr 13, 2017 8:08 pm

Re: unicode (utf-8) over socket

Post by pajito » Sun Nov 12, 2017 9:02 pm

Here you are Klaus,
please note that the one we care about is fld "p1a" , t1 is the dynamic field from Adobe Animate.
Thanks again

Code: Select all

global casparServer
on mouseUp
   
   write "CG 1-81 ADD 1" & quote & "superiera/SW1" & quote & "1" & quote &  "<templateData><componentData id=\" & quote & "t1\"& quote & "><data id=\"& quote &"text\"&quote &" value=\"& quote & fld "p1a" &"\"& quote &"/></componentData></templateData>"& quote & format("\r\n") to socket casparServer
   
end mouseUp

 
bellow is the code that I use to connect to the socket

Code: Select all

on mouseUp
   global casparServer
  
   if field "Server2" is empty then
      set the backgroundcolor of button "Server2" to "Blue "
   else 
      put field "Server2" into casparServer
      if casparServer is among the lines of the opensockets then
         close socket casparServer
         set the backgroundcolor of button "Server2" to "Gray "
         set the label of button "Server2" to " Disconnected"
      else
         open socket to casparServer
        
         set the backgroundcolor of button "Server2" to "green"
         set the label of button "Server2" to "Connected"
      end if
   end if
end mouseUp
end mouseUp

Klaus
Posts: 10425
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: unicode (utf-8) over socket

Post by Klaus » Mon Nov 13, 2017 12:14 pm

Hi Ioannis,

I mean the scripts where you write and receive (and put itno fields) data via the socket.

General hint:
In cases like this one you should use this little function to save a lot of typing:

Code: Select all

function q tString
  return QUOTE & tString & QUOTE
end q
Get the picture?


Best

Klaus

pajito
Posts: 23
Joined: Thu Apr 13, 2017 8:08 pm

Re: unicode (utf-8) over socket

Post by pajito » Mon Nov 13, 2017 8:50 pm

Hi Klaus

sorry for the late response, there are no scripts writing to the fields, the data are coming straight from the cvs field "1" or "f1", the other way I use is by typing myself into the fields. If I got something wrong please let me know and I will send you what you need.

Best Regards
Ioannis

Post Reply

Return to “Getting Started with LiveCode - Complete Beginners”