FTP - force ASCII instead of Binary

[robl]

Ran into a situation here that I haven't figured out how to work around the issue. I am using FTP to talk to an old IBM Mainframe to retrieve some raw data. The problem is that if I use the following code

Code: Select all

put url "ftp://user:password@servername.server.com/SOME.DATA.REPORTS/REPORT1" into fld "output"

it retrieves the data as binary, which ends being raw EBCDIC (not ASCII) text. If I use the old command line ftp client or a GUI FTP client like FileZilla, I can switch it to ASCII and then transfer the file, resulting in the correct format. "ascii" and "binary" are standard command/modes for the FTP protocol, but LiveCode seems to only do binary?

With the Linux/Unix FTP CLI client, this works as follows

Code: Select all

ftp X.X.X.X
Connected to X.X.X.X.
220 *** MVS FTP Daemon on X.X.X.X ***
Name (X.X.X.X:USER): 
331 Okay, waiting for password.
Password: 
230 You are now logged in.
Remote system type is MVS.
ftp> ascii
200 Type set to A
ftp> cd SOME.DATA.REPORTS
250 CWD command successful
ftp> get REPORT1
local: REPORT1 remote: REPORT1
227 Entering Passive Mode (X,X,X,X,194,140)
150 Now opening data connection
 18615        1.69 MiB/s 
226 Transfer complete!
18615 bytes received in 00:00 (1.67 MiB/s)
ftp> quit
221 Bye!

I have yet to find a way to tell LiveCode to grab the file as ASCII. Any suggestions?

[shaosean]

I think that the default transfer mode for FTP is binary and seeing as there is no file extension, the server (and perhaps LC) do not know to switch to ASCII mode.. If you want more control over the transfer method, you might want to look at the other FTP library which allows you to switch modes as you see fit..

[strongbow]

Can you use

libURLftpCommand

to set it to ASCII before getting the file?

[jacque]

Another likely possibility is that the files are unicode and need to be run through textDecode().

[FourthWorld]

Interesting problem. I've never worked with EBCDIC before, so I checked out the Wikipedia page on it, which includes a section titled "Criticism and Humor":
https://en.wikipedia.org/wiki/EBCDIC#Cr ... _and_humor

It's such an odd and old format that I'm wondering: when you use text mode in other software, where is the translation from EBCDIC to ASCII taking place? Does the FTP client include this, or is there something in the FTP server that handles that?

If the latter then using the suggestions here to request text mode may suffice. If the former, you may need to write a translation function for it. There may already be an EBCDICtoASCII function somewhere in our community, but it's such an old format and so seldom used it seems at least as likely that there may not be one yet.

[jacque]

I missed the ebcdic reference. There's a conversion table here:

http://www.simotime.com/asc2ebc1.htm

[robl]

Interesting problem. I've never worked with EBCDIC before, so I checked out the Wikipedia page on it, which includes a section titled "Criticism and Humor":
https://en.wikipedia.org/wiki/EBCDIC#Cr ... _and_humor

It's such an odd and old format that I'm wondering: when you use text mode in other software, where is the translation from EBCDIC to ASCII taking place? Does the FTP client include this, or is there something in the FTP server that handles that?

If the latter then using the suggestions here to request text mode may suffice. If the former, you may need to write a translation function for it. There may already be an EBCDICtoASCII function somewhere in our community, but it's such an old format and so seldom used it seems at least as likely that there may not be one yet.

It's actually the server that converts it to ascii when requested to do so. The client sends a "TYPE A" and the server should translate it to ASCII, which happens with using ncftp, "ftp" (CLI), and FileZilla GUI.

[robl]

Can you use

libURLftpCommand

to set it to ASCII before getting the file?

Thank you! It almost works. I'm not sure if it is something I am missing, but I can get to download ASCII using your solution, but I have to download it TWICE! If I send a "QUIT" to the ftp server to formally disconnect, and then try, the first time I download, I get a binary file. I download the SAME file, I get ASCII. Very odd!

The code below works -- but notice the 2 downloads of the same file. the first one returns binary (raw EBCDIC) and the second will return ASCII.

Code: Select all

on mouseUp
    put empty into fld "output"
    downloadTEST
end mouseUp

on downloadTEST
   put libURLftpCommand("TYPE A",lServer) & return into fld "status"  -- tells the server to send ASCII for ALL downloads on this connection until I tell otherwise
   libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile, lLocalFile,"downloadFile" -- (I get EBCDIC)
end downloadTEST

on downloadFile
   libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile, lLocalFile,"downloadDone" -- (this is ASCII)
end downloadFile

on downloadDone
   put URL ("file:"& lLocalFile) after fld "output"
   killFTP
end downloadDone

on killFTP
   put libURLftpCommand("QUIT",lServer) & return into fld "status"
end killFTP

[strongbow]

Did you try a small "wait" between setting to Ascii and getting the file?

[robl]

Did you try a small "wait" between setting to Ascii and getting the file?

Indeed, I tried a delay of a varying amounts up to 10 seconds, still the same results. I'm almost wondering if I am missing something in the "raw" FTP commands that isn't fully initializing it correctly and the libURLdownloadToFile finishes the initialization as part of the first download.

If I download 10 files, the first is binary, the remaining 9 are ASCII. Something is happening differently during the first download attempt and I haven't quite figured it out yet. I may end up trying wireshark and comparing what livecode is sending to that of another client.

[robl]

I think I found a solution, and it is very simple, but not documented in the LiveCode docs.

Code: Select all

libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile & ";TYPE=A", lLocalFile,"downloadDone"

That's it. No need to use any libURLftpCommand. No need to download the file twice.

Just add the string ";TYPE=A" to the end of the "ftp://" URL. i.e., "ftp://my.server.com/some/path/file;TYPE=A". It only seems to work with libURLDownloadToFile, and not the "put URL" command.

I found the solution while looking at the documentation for the command line "curl" tool.
https://curl.haxx.se/docs/manpage.html

-B, --use-ascii

(FTP LDAP) Enable ASCII transfer. For FTP, this can also be enforced by using an URL that ends with ";type=A". This option causes data sent to stdout to be in text mode for win32 systems.

[strongbow]

Ahh, that's right! I'd forgotten that! Probably last time I used that form was in the 80s! Great that you found it tho'.