FTP - force ASCII instead of Binary

Bringing the internet highway into your project? Building FTP, HTTP, email, chat or other client solutions?

Moderators: Klaus, FourthWorld, heatherlaine, kevinmiller, robinmiller

Post Reply
robl
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 62
Joined: Wed Dec 22, 2010 9:50 pm
Location: 3rd planet from the Sun

FTP - force ASCII instead of Binary

Post by robl » Wed Dec 28, 2016 1:57 am

Ran into a situation here that I haven't figured out how to work around the issue. I am using FTP to talk to an old IBM Mainframe to retrieve some raw data. The problem is that if I use the following code

Code: Select all

put url "ftp://user:password@servername.server.com/SOME.DATA.REPORTS/REPORT1" into fld "output"
it retrieves the data as binary, which ends being raw EBCDIC (not ASCII) text. If I use the old command line ftp client or a GUI FTP client like FileZilla, I can switch it to ASCII and then transfer the file, resulting in the correct format. "ascii" and "binary" are standard command/modes for the FTP protocol, but LiveCode seems to only do binary?

With the Linux/Unix FTP CLI client, this works as follows

Code: Select all

ftp X.X.X.X
Connected to X.X.X.X.
220 *** MVS FTP Daemon on X.X.X.X ***
Name (X.X.X.X:USER): 
331 Okay, waiting for password.
Password: 
230 You are now logged in.
Remote system type is MVS.
ftp> ascii
200 Type set to A
ftp> cd SOME.DATA.REPORTS
250 CWD command successful
ftp> get REPORT1
local: REPORT1 remote: REPORT1
227 Entering Passive Mode (X,X,X,X,194,140)
150 Now opening data connection
 18615        1.69 MiB/s 
226 Transfer complete!
18615 bytes received in 00:00 (1.67 MiB/s)
ftp> quit
221 Bye!
I have yet to find a way to tell LiveCode to grab the file as ASCII. Any suggestions?

shaosean
Posts: 878
Joined: Thu Nov 04, 2010 7:53 am

Re: FTP - force ASCII instead of Binary

Post by shaosean » Wed Dec 28, 2016 7:51 am

I think that the default transfer mode for FTP is binary and seeing as there is no file extension, the server (and perhaps LC) do not know to switch to ASCII mode.. If you want more control over the transfer method, you might want to look at the other FTP library which allows you to switch modes as you see fit..

strongbow
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 111
Joined: Mon Jul 31, 2006 1:39 am
Location: New Zealand, Australia, Germany, Japan
Contact:

Re: FTP - force ASCII instead of Binary

Post by strongbow » Wed Dec 28, 2016 9:22 am

Can you use

libURLftpCommand

to set it to ASCII before getting the file?

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3978
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: FTP - force ASCII instead of Binary

Post by jacque » Wed Dec 28, 2016 4:49 pm

Another likely possibility is that the files are unicode and need to be run through textDecode().
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 5462
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: FTP - force ASCII instead of Binary

Post by FourthWorld » Wed Dec 28, 2016 5:29 pm

Interesting problem. I've never worked with EBCDIC before, so I checked out the Wikipedia page on it, which includes a section titled "Criticism and Humor":
https://en.wikipedia.org/wiki/EBCDIC#Cr ... _and_humor

It's such an odd and old format that I'm wondering: when you use text mode in other software, where is the translation from EBCDIC to ASCII taking place? Does the FTP client include this, or is there something in the FTP server that handles that?

If the latter then using the suggestions here to request text mode may suffice. If the former, you may need to write a translation function for it. There may already be an EBCDICtoASCII function somewhere in our community, but it's such an old format and so seldom used it seems at least as likely that there may not be one yet.
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3978
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: FTP - force ASCII instead of Binary

Post by jacque » Wed Dec 28, 2016 6:01 pm

I missed the ebcdic reference. There's a conversion table here:

http://www.simotime.com/asc2ebc1.htm
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

robl
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 62
Joined: Wed Dec 22, 2010 9:50 pm
Location: 3rd planet from the Sun

Re: FTP - force ASCII instead of Binary

Post by robl » Wed Dec 28, 2016 8:02 pm

FourthWorld wrote:Interesting problem. I've never worked with EBCDIC before, so I checked out the Wikipedia page on it, which includes a section titled "Criticism and Humor":
https://en.wikipedia.org/wiki/EBCDIC#Cr ... _and_humor

It's such an odd and old format that I'm wondering: when you use text mode in other software, where is the translation from EBCDIC to ASCII taking place? Does the FTP client include this, or is there something in the FTP server that handles that?

If the latter then using the suggestions here to request text mode may suffice. If the former, you may need to write a translation function for it. There may already be an EBCDICtoASCII function somewhere in our community, but it's such an old format and so seldom used it seems at least as likely that there may not be one yet.
It's actually the server that converts it to ascii when requested to do so. The client sends a "TYPE A" and the server should translate it to ASCII, which happens with using ncftp, "ftp" (CLI), and FileZilla GUI.

robl
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 62
Joined: Wed Dec 22, 2010 9:50 pm
Location: 3rd planet from the Sun

Re: FTP - force ASCII instead of Binary

Post by robl » Wed Dec 28, 2016 8:10 pm

strongbow wrote:Can you use

libURLftpCommand

to set it to ASCII before getting the file?
Thank you! It almost works. I'm not sure if it is something I am missing, but I can get to download ASCII using your solution, but I have to download it TWICE! If I send a "QUIT" to the ftp server to formally disconnect, and then try, the first time I download, I get a binary file. I download the SAME file, I get ASCII. Very odd!

The code below works -- but notice the 2 downloads of the same file. the first one returns binary (raw EBCDIC) and the second will return ASCII.

Code: Select all

on mouseUp
    put empty into fld "output"
    downloadTEST
end mouseUp

on downloadTEST
   put libURLftpCommand("TYPE A",lServer) & return into fld "status"  -- tells the server to send ASCII for ALL downloads on this connection until I tell otherwise
   libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile, lLocalFile,"downloadFile" -- (I get EBCDIC)
end downloadTEST

on downloadFile
   libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile, lLocalFile,"downloadDone" -- (this is ASCII)
end downloadFile

on downloadDone
   put URL ("file:"& lLocalFile) after fld "output"
   killFTP
end downloadDone

on killFTP
   put libURLftpCommand("QUIT",lServer) & return into fld "status"
end killFTP

strongbow
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 111
Joined: Mon Jul 31, 2006 1:39 am
Location: New Zealand, Australia, Germany, Japan
Contact:

Re: FTP - force ASCII instead of Binary

Post by strongbow » Thu Dec 29, 2016 12:26 am

Did you try a small "wait" between setting to Ascii and getting the file?

robl
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 62
Joined: Wed Dec 22, 2010 9:50 pm
Location: 3rd planet from the Sun

Re: FTP - force ASCII instead of Binary

Post by robl » Thu Dec 29, 2016 3:13 am

strongbow wrote:Did you try a small "wait" between setting to Ascii and getting the file?
Indeed, I tried a delay of a varying amounts up to 10 seconds, still the same results. I'm almost wondering if I am missing something in the "raw" FTP commands that isn't fully initializing it correctly and the libURLdownloadToFile finishes the initialization as part of the first download.

If I download 10 files, the first is binary, the remaining 9 are ASCII. Something is happening differently during the first download attempt and I haven't quite figured it out yet. I may end up trying wireshark and comparing what livecode is sending to that of another client.

robl
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 62
Joined: Wed Dec 22, 2010 9:50 pm
Location: 3rd planet from the Sun

Re: FTP - force ASCII instead of Binary

Post by robl » Thu Dec 29, 2016 1:24 pm

I think I found a solution, and it is very simple, but not documented in the LiveCode docs.

Code: Select all

libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile & ";TYPE=A", lLocalFile,"downloadDone"
That's it. No need to use any libURLftpCommand. No need to download the file twice.

Just add the string ";TYPE=A" to the end of the "ftp://" URL. i.e., "ftp://my.server.com/some/path/file;TYPE=A". It only seems to work with libURLDownloadToFile, and not the "put URL" command.

I found the solution while looking at the documentation for the command line "curl" tool.
https://curl.haxx.se/docs/manpage.html
-B, --use-ascii

(FTP LDAP) Enable ASCII transfer. For FTP, this can also be enforced by using an URL that ends with ";type=A". This option causes data sent to stdout to be in text mode for win32 systems.

strongbow
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 111
Joined: Mon Jul 31, 2006 1:39 am
Location: New Zealand, Australia, Germany, Japan
Contact:

Re: FTP - force ASCII instead of Binary

Post by strongbow » Thu Dec 29, 2016 11:54 pm

Ahh, that's right! I'd forgotten that! Probably last time I used that form was in the 80s! Great that you found it tho'.

Post Reply

Return to “Internet”