UrlEncode and ArrayEncode

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

UrlEncode and ArrayEncode

Post by trevix » Wed Feb 21, 2018 1:07 pm

Someone can explain me why this does not works?

Code: Select all

on mouseUp 
   put "me" into tArrayData["one"]
   put "me you" into tArrayData["two"]
   put arrayEncode(tArrayData) into tArrayData
   put urlEncode(tArrayData) into tArrayData
   
   put urlDecode(tArrayData) into tArrayData
   put arrayDecode(tArrayData) into tArrayData
   --tArrayData is no longher an array
   put the keys of tArrayData
end mouseUp
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: UrlEncode and ArrayEncode

Post by Klaus » Wed Feb 21, 2018 2:46 pm

Hi Trevix,

use "base64encode" instead of "urlencode"!
"ArrayEncode" produces BINARY data and "urlencode" will corrupt it, "base64encode" does not.

Works:

Code: Select all

on mouseUp 
   put "me" into tArrayData["one"]
   put "me you" into tArrayData["two"]
   put arrayEncode(tArrayData) into tArrayData
   put base64encode(tArrayData) into tArrayData
   
   put base64Decode(tArrayData) into tArrayData
   put arrayDecode(tArrayData) into tArrayData
   put the keys of tArrayData
end mouseUp
Best

Klaus

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Thu Jun 07, 2018 8:30 pm

LC 8.1.10 OSX 12
I am having some problem using
base64encode(arrayEncode(pTheData))
and
arrayDecode(base64decode(Thedata))
, trying to send a multiple keys array over sockets.

While I checked encoding and decoding before transmission and they work correctly, using sockets on some occasion the decoding does not works and, as we know (bug 9552), no error is trow and no breakpoint will intercept.

Could it be because of some strange chars used on the array? I am not sure to understand the dictionary stating:
base64Encode function can include uppercase and lowercase letters, digits, +, /, and =, but no other characters.
Or the length?

Thanks for any help.

Trevix
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: UrlEncode and ArrayEncode

Post by FourthWorld » Thu Jun 07, 2018 8:55 pm

How are you sending them and what problem are you encountering?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Thu Jun 07, 2018 11:07 pm

On receiving the data, after the arrayDecode(base64decode(Thedata)) the script just end without messages.
I send them like this:

Code: Select all

...
put base64encode(arrayEncode(pTheData)) into TheInvio
 put "**," & TheInvio & FineTrasmissione  into TheInvio --FineTrasmissione is declared constant = ",#F#"
write TheInvio to socket tSocket
and receive with:

Code: Select all

..
read from socket W until FineTrasmissione with message "chatReceived"
...
on chatReceived W,TheData
      if item 1 of TheData is startOfMessage then -- é un messaggio valido 
         delete item 1 of TheData --startOfMessage= "**"
         delete last item of TheData --endOfMessage
         put arrayDecode(base64decode(Thedata)) into TheData
        --at this line the script just stop and no breakpoint will work. The data is received but don't get converted
...
end chatReceived
As I said, since for some content of the array the transmission is fine, it looks like could be something in the array.
Please note: the same data works fine doing the encode and decode in the same script.
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: UrlEncode and ArrayEncode

Post by FourthWorld » Fri Jun 08, 2018 1:12 am

In some recent versions the "read socket...until <char>" option was not working. I missed if you included the version number, but you may want to try the most recent build.

FWIW you'll not only bypass that issue, but also eliminate any issues that may arise from have your ending flag constant confused when data content, and also get a most performance boost by using "read socket...for <length>" instead.

One challenge with that is finding a fixed & length representation for any number. You could pad ASCII to come up with a consistent 10 characters, which will allow you to handle data lengths from 0 to 4GB, the largest size LC can handle. Use smaller if smaller will work.

Or you can use binaryEncode to get a four-byte value that can cover the same range.

Then you do two reads: first for a fixed length to get the says length, and then read for that specified amount.

If you have a lot of transactions the modest time savings may even be measurable, as it saves a lot of substring comparisons in the incoming buffer, instead handing that task off in one move to the OS to grab the specified length from the buffer.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Fri Jun 08, 2018 10:10 am

Thankyou. As always you are very helpful.(Livecode 8.1.10)
I just wish I could just code instead of constantly chasing LC bugs.

If I understand you:

Code: Select all

put "**" & comma & base64encode(arrayEncode(tArray))  into Thedata
put "000000000" & the length of Thedata into tLength  --get the size of the sent
put char -10 to -1 of tLength into tLenght --alway 10 chars
write tLenght to socket gTCPref["setup"]["ChatSocket"]
write  Thedata to socket gTCPref["setup"]["ChatSocket"]
...
read from socket W for 10 with message "chatReceived"
...
on chatReceived W,pNum
read from socket W for pNum
put it into TheData
if item 1 of TheData = "**" then --I need this because trying to arrayDecode something that is not an array will screw the script 
put arrayDecode(base64decode(it)) into TheData
..
This apparently complicates my protocol, having to write and read twice for each message.

Note that the most complete example of socket communication, is this Livecode lesson:
http://lessons.livecode.com/m/4071/l/12 ... ng-sockets
It uses "return" as an end char. Not very helpful for this.
My protocol was based on the "Chat Server" example, improved but not working anymore when I tried to go from passing "strings" to passing "encoded array".
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Fri Jun 08, 2018 4:15 pm

I've put together a small example of communication using sockets (see attachment for LC 8.1.10).

It uses data length and has been reduced to the minimum, so to be easily handled.
There is no error correction and the only use of it is to establish a connection and send an array from the server to the client.

All scripts are in the button "Start Server" (stack TestSocketServer) and "Connect to server" (stack TestSocketClient)
Can be used on the same computer. Just fill the local IP and a port number on both stacks.
Press "Start Server" and then "Connect to server": on line 37 of the "Connect to server" btn you can see that the array is correctly received.

I spent a couple of hours for doing that, because I often rely on socket communication between standalone and I needed to check if this way to do it works (it does) and if it is the correct way to do it.
Timing between the calls are fundamentals.
Attachments
TestSockets_2018.zip
(4.92 KiB) Downloaded 181 times
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Fri Jun 08, 2018 4:16 pm

Any suggestions are welcomed...

Regards
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: UrlEncode and ArrayEncode

Post by FourthWorld » Fri Jun 08, 2018 4:52 pm

trevix wrote:
Fri Jun 08, 2018 4:16 pm
Any suggestions are welcomed...
Is the example working?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Fri Jun 08, 2018 5:43 pm

Yes, it does.
But I wonder about the size of this protocol, once completed with error handling.
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: UrlEncode and ArrayEncode

Post by FourthWorld » Fri Jun 08, 2018 8:40 pm

What are the average and maximum sizes of the payloads being transferred?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: UrlEncode and ArrayEncode

Post by FourthWorld » Fri Jun 08, 2018 9:53 pm

trevix wrote:
Fri Jun 08, 2018 10:10 am
Thankyou. As always you are very helpful.(Livecode 8.1.10)
I just wish I could just code instead of constantly chasing LC bugs.
The most recent build will always contain the most complete set of fixes.

As for the added step of a length indicator:
This apparently complicates my protocol, having to write and read twice for each message.
A good scripting language like LC prevents us from having to think like computers. But when performance matters (and it doesn't always, but sometimes it does) thinking like the computer can help.

On the sending side, you could prepend four bytes for the length (if using binaryEncode, or padded with the same function to get 10-char output if you prefer), and write in one pass.

It's only the receiver that needs two reads.

But reading from an I/O buffer is not a big task; the stream is already open, the data already collected by the host OS ready to do your bidding. In some respects, a small read on a stream is more like parsing a variable than opening and reading a file, in terms of total system overhead.

But the benefits can be tremendous, depending on the payload size.

Lines, items, words, even characters -- all these things mean nothing to the computer. LC lets us conceptualize data that way, but at the cost of having the engine sift through bytes to understand the boundaries.

But bytes - now there's something computers grasp easily. It's pretty much the only chunk type they understand.

When we read an I/O stream for a match against a string, every byte must be compared to the string in hopes of eventually finding that match. LC's pretty good about that so it's not like it takes that long, but consider what happens when we already know exactly how much to read:

Asking LC to read for a specified size allows it to pass that request almost directly to the I/O buffering in the OS, where the OS will then reach out and grab that in one step. Byte operations are the only things computers can do, but they do them very well.

So while for us it means typing an extra line of code, for the computer it replaces potentially hundreds or thousands (depending on payload size) of strcmp operations with a gulp performed in a single step.

This is why HTTP 1.1 added the length indicator in the header. Speeds up and simplifies a great many things.

We can learn from that when crafting our own custom protocols as well.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Sat Jun 09, 2018 4:36 pm

Thanks. Very useful indeed.
Regards
Trevix
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

trevix
Posts: 958
Joined: Sat Feb 24, 2007 11:25 pm
Location: Italy
Contact:

Re: UrlEncode and ArrayEncode

Post by trevix » Sun Jun 10, 2018 11:33 am

Oh my...
I found out what was the problem. To me looks like a bug:

Code: Select all

on mouseUp pMouseButton
   put "Bg0AAAABAAN1bm8GAAVhc2lubw==" into TheData
   put base64decode(Thedata) into tData
   put arrayDecode(tData) into TheResult
   put (TheResult is an array) into msg ---YES, it's a plain
   breakpoint
   
   put arrayDecode(base64decode(Thedata)) into TheResult
   put (TheResult is an array) into msg ---YES, it's a bird
   breakpoint
   
    put arrayDecode(base64decode(Thedata)) into Thedata
   put (Thedata is an array) into msg ---------------------------NOOOOO, it's Superman
   breakpoint --breakpoint will not work (bug 9552)
end mouseUp
Trevix
OSX 14.3.1 xCode 15 LC 10 DP7 iOS 15> Android 7>

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”