Text Problems over Socket Connection

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Klaus
Posts: 14177
Joined: Sat Apr 08, 2006 8:41 am
Contact:

Re: Text Problems over Socket Connection

Post by Klaus » Sun Sep 15, 2013 5:27 pm

Oh, you replied while I was replying :D

Well, I am no socket expert, but it sound like they only like ASCII somehow.
As proposed earlier in this thread, you could base64encode your UTF8 string
so it gets "converted" to ASCII.

No idea this will work, but at least worth a try :D

aircooled76
Posts: 38
Joined: Mon May 20, 2013 3:14 pm

Re: Text Problems over Socket Connection

Post by aircooled76 » Sun Sep 15, 2013 5:36 pm

Yeah... I figured that out eventually... other problems now... one solved next one crops up!!! (see my more recent post)

aircooled76
Posts: 38
Joined: Mon May 20, 2013 3:14 pm

Re: Text Problems over Socket Connection

Post by aircooled76 » Sun Sep 15, 2013 5:39 pm

The commands I need to send over the socket need to be UTF8... this is good as I need special Charactors (ANSII is no good)

The problem is I have a whole lot of ANSII text in LiveCode along with the one field that needs to be UTF8 or 16... Need to figure a way how to get it all to pass through nicely... Ill have to have a crack tomorrow... it is getting late over here... Hopefully there can be some more suggestions I can test out in the meantime!

Klaus
Posts: 14177
Joined: Sat Apr 08, 2006 8:41 am
Contact:

Re: Text Problems over Socket Connection

Post by Klaus » Mon Sep 16, 2013 12:10 pm

Sorry, no more ideas...

aircooled76
Posts: 38
Joined: Mon May 20, 2013 3:14 pm

Re: Text Problems over Socket Connection

Post by aircooled76 » Mon Sep 16, 2013 2:45 pm

Just noticed something interesting...

when I use the code:

Code: Select all

on mouseUp
   set the unicodeText of field "testfield2" to the unicodeText of field "testfield"
   put the unicodetext of fld "testfield2" into tData
   put uniDecode(tData,"utf16") into field "testfield3"
end mouseUp
I get "Č Ž Š Đ Ć" in testfield2

but only get "C Ž Š Ð C" in testfield3... the "Č" and the "Ć" just turn into "C" :cry:

There must be a solution for this... or do I say to my Croatian friend... "sorry livecode won't play nicely with your countries names"

Klaus
Posts: 14177
Joined: Sat Apr 08, 2006 8:41 am
Contact:

Re: Text Problems over Socket Connection

Post by Klaus » Tue Sep 17, 2013 2:00 pm

Hi,

just made a test with a couple of fields and this seems to do the job:

Code: Select all

on mouseup
   ## Some chinese text in fld 1, works, same strange characters in fld 2 :-D
   set the unicodetext of fld 2 to the unicodetext of fld 1

   ## put unicodetext into variable
   put the unicodetext of fld 2 into tt

   ## as mentioned earlier, "turn" binary(sic!) unicodetext to ASCII
   put base64encode(tt) into tt2
   
   ## Test: Yes, really two lines of pure ASCII :-)
   put tt2

   ## When "converted" back to unicode, this line works as exspected:
   set the unicodetext of fld 3 to base64decode(tt2)
end mouseup
Best

Klaus

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Text Problems over Socket Connection

Post by FourthWorld » Tue Sep 17, 2013 3:25 pm

People sometimes say it's overkill, but I've found time and again that when I have transport issues with binary data like Unicode, converting it to base64 seems to take care of it.

There may be ways to adjust the header to address that without converting to base64, but base64 is so convenient in LC that it's often a pretty good way to resolve such issues.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

BvG
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 1239
Joined: Sat Apr 08, 2006 1:10 pm
Contact:

Re: Text Problems over Socket Connection

Post by BvG » Wed Sep 18, 2013 10:27 am

FourthWorld wrote:People sometimes say it's overkill, but I've found time and again that when I have transport issues with binary data like Unicode, converting it to base64 seems to take care of it.

There may be ways to adjust the header to address that without converting to base64, but base64 is so convenient in LC that it's often a pretty good way to resolve such issues.
completely not the problem, please read threads more carefully :P
Various teststacks and stuff:
http://bjoernke.com

Chat with other RunRev developers:
chat.freenode.net:6666 #livecode

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Text Problems over Socket Connection

Post by FourthWorld » Wed Sep 18, 2013 5:32 pm

BvG wrote:completely not the problem, please read threads more carefully :P
My apologies. With so many posts and my volunteer admin tasks here, I rarely have the time to completely re-read every thread here before replying, and admittedly this may sometimes mean a post from me that doesn't account for errors in other posts.

So if I understand you correctly, the base64Encode included in so many prior posts is completely unnecessary?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

BvG
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 1239
Joined: Sat Apr 08, 2006 1:10 pm
Contact:

Re: Text Problems over Socket Connection

Post by BvG » Thu Sep 19, 2013 7:30 pm

FourthWorld wrote:So if I understand you correctly, the base64Encode included in so many prior posts is completely unnecessary?
that too of course, but actually the problems Aircooled has are more in the direction of basic programming, like how to use functions and repeat loops. The somewhat weird way how LC does Unicode doesn't help either (Great job Klaus, in trying to continuingly offering help btw).
Various teststacks and stuff:
http://bjoernke.com

Chat with other RunRev developers:
chat.freenode.net:6666 #livecode

aircooled76
Posts: 38
Joined: Mon May 20, 2013 3:14 pm

Re: Text Problems over Socket Connection

Post by aircooled76 » Tue Dec 10, 2013 12:57 pm

OK... so I got a bit frustrated and went onto other projects... but now I am back to slay this Unicode thing... (hopefully just in time for v7 of LiveCode to be released with a code-wide fix!

I digress...

My current attempt to fix this is this:

Code: Select all

write "CG 1-" & field "coach3rdvl" of card settings && "add" && field coach3rdfl of card settings &&quote& field coach3rd of the card settings &"_home" &quote&" 1 "&quote&"<templateData><componentData id=\"&quote&"_name\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& uniDecode (the unicodetext of field T1Coach,"UTF8")  & "\"&quote&">" & cr & \
"</data> </componentData><componentData id=\"&quote&"_club\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& field t1fullname of card Controller & "\"&quote&">" & cr & \
      "</data></componentData></templateData>"&quote&"" & format("\r\n") to socket field IP of card settings
The field T1Coach has " ščćđž ČĆŽŠĐ" in it but what comes out in the destination program is just ????? but what gets to the graphics is šžŽŠ. No sign of the čćđČĆĐ. I have tried using

Code: Select all

uniDecode (the unicodetext of field T1Coach,"UTF16")
but that just gives me " ?ccd? CC???" in the destination program or "ccd CC" or in the graphics.

I feel like I am bumbling along in the dark... if anyone can shed some light this way I would appreciate it!

bangkok
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 937
Joined: Fri Aug 15, 2008 7:15 am

Re: Text Problems over Socket Connection

Post by bangkok » Tue Dec 10, 2013 1:57 pm

You might use base64encode, no ?

Code: Select all

Use the base64Encode function to encode binary data for transmission over communication channels that don't accept raw binary data.

aircooled76
Posts: 38
Joined: Mon May 20, 2013 3:14 pm

Re: Text Problems over Socket Connection

Post by aircooled76 » Tue Dec 10, 2013 3:47 pm

OK... I did a bit more testing...

I downloaded a SocketTest and tuned into the port LiveCode is sending to http://sockettest.sourceforge.net/

When i sent

Code: Select all

uniDecode (the unicodetext of field T1Coach,"UTF8") 


I get at SocketTest:

Code: Select all

CG 1-2 add 4 "coach_home" 1 "<templateData><componentData id=\"_name\">
<data id=\"text\" value=\" š�ćđž ČĆŽŠ�\">
</data> </componentData><componentData id=\"_club\">
<data id=\"text\" value=\"HOME TEAM NAME\">
</data></componentData></templateData>"
When i sent

Code: Select all

uniDecode (the unicodetext of field T1Coach,"UTF16") 


I get at SocketTest:

Code: Select all

CG 1-2 add 4 "coach_home" 1 "<templateData><componentData id=\"_name\">
<data id=\"text\" value=\" šccdž CCŽŠÐ\">
</data> </componentData><componentData id=\"_club\">
<data id=\"text\" value=\"HOME TEAM NAME\">
</data></componentData></templateData>"
I also sent the following code to save to a text file:

Code: Select all

on mouseUp
   ask file "Select a location to save to"
    if the result is not "cancel" then
put "CG 1-" & field "coach3rdvl" of card settings && "add" && field coach3rdfl of card settings &&quote& field coach3rd of the card settings &"_home" &quote&" 1 "&quote&"<templateData><componentData id=\"&quote&"_name\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& uniDecode (the unicodetext of field T1Coach,"UTF8")  & "\"&quote&">" & cr & \
"</data> </componentData><componentData id=\"&quote&"_club\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& field t1fullname of card Controller & "\"&quote&">" & cr & \
      "</data></componentData></templateData>"&quote&"" & format("\r\n") into url ("file:" & it) 
end if
end mouseUp
And that resulted in:

Code: Select all

CG 1-2 add 4 "coach_home" 1 "<templateData><componentData id=\"_name\">
<data id=\"text\" value=\" ščćđž ČĆŽŠĐ\">
</data> </componentData><componentData id=\"_club\">
<data id=\"text\" value=\"HOME TEAM NAME\">
</data></componentData></templateData>"
That seems to be correct (checked in NotePad++ and it displays correctly in UTF8

On reading CasparCG's (the destination software) website all data is assumed to be UTF8...

I have more information but still feel in the dark! :roll:

I havn't tried base64encode yet... but I seemed to have found the problem... the font in the graphics templates does not seem to accept all the Characters I am sending... I tested with another graphics template (with Arial font) and all the characters went through OK...

Big shout out to Monte from MergEXT who gave me a tip on email... cheers guys...

visionmill
Posts: 6
Joined: Fri Sep 12, 2014 2:32 pm

Re: Text Problems over Socket Connection

Post by visionmill » Fri Sep 12, 2014 2:37 pm

Hi

I am looking to do exactly the same thing with CasparCG. Did you ever get a way of generating the UTF8 correctly?

aircooled76
Posts: 38
Joined: Mon May 20, 2013 3:14 pm

Re: Text Problems over Socket Connection

Post by aircooled76 » Fri Sep 12, 2014 2:50 pm

Yeah... go it working.. More info on CasparCG forum here: http://casparcg.com/forum/viewtopic.php?f=9&t=1467

Post Reply