HTTPD Library - Capturing Form Data

Bringing the internet highway into your project? Building FTP, HTTP, email, chat or other client solutions?

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Mon Jul 26, 2021 8:28 am

SoapDog wrote:
Sun Jul 25, 2021 11:37 pm
This should be simple enough if the multipart/form-data gets processed. Currently the HTTPD Library has a "case" construct defined for multipart/form-data but the construct does not contain any code. Instead it has a comment saying "--todo".
If you're in control of the form, you can use JS to change to make a POST request with the values in a JSON instead, I believe that should work even though I haven't tried. You would need to change the content type to "application/json" on the request, and decode the JSON yourself on your callback handler in your stack. It should be inside pRequestA["content"], if I understood the library correctly.
maybe that would work, but wouldn't the HTTPD Library need to be updated to detect and process application/json content types? If that's the case, it might be as well to just fix the multipart/form-data rather than adding another type?

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Mon Jul 26, 2021 8:31 am

matthiasr wrote:
Fri Jul 23, 2021 9:27 pm
You cannot use html to process the formdata. You would need PHP, Perl or LivecodeServer for example. But this is not possible with the HTTPD library.
Hi, I'm not using HTML to process the formdata. After testing, I discovered that POST sends the data back to LC in an array and you can handle it there. It's just not very well documented.

matthiasr
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 190
Joined: Sat Apr 08, 2006 7:55 am
Location: Lübbecke, Germany
Contact:

Re: HTTPD Library - Capturing Form Data

Post by matthiasr » Mon Jul 26, 2021 10:55 am

Hi,
i played a little bit with your stack and i am able to get the .csv displayed.

I've copied the httpd library to the stackscript and modiefied it, so it does not interfere with the original one. But additionally i've unloaded the original one and unchecked (load at startup) in the Extension Manager.

I ended up with adding an answer dialog at the end of the __NewRequest handler. With that answer dialog i get the content of the uploaded file displayed, but only if i use an other form.

I am attaching my sample script. There is a 2nd start button "mrStart" which make use of my custom html form.
HTTP-Test_mr.livecode.zip
(6.67 KiB) Downloaded 202 times

There's a 2nd start button "mrStart". It makes use of my upload form.
There is also a switch button which enables or disables the answer dialog. With the answer dialog enabled and using the mrStart button i get the .csv displayed. Without the answer dialog, the content isn't displayed. Maybe it's a timing thing. I tried a wait ... instead of the answer dialog, but that doesn't work.

Maybe someone other find out what might be wrong with the httpd library.

Btw. when running the stack in the IDE and connecting from the same computer to the webserver, sometimes the the content of the csv isn't displayed also, although the answer dialog is enabled. So i would suggest to test with 2 different devices. One's running the app, the other is accessing the webserver.

[Edit] Please see my comments below. There is no problem with the httpd library. Using an other html form for the upload, returns the posted data.
Last edited by matthiasr on Mon Jul 26, 2021 11:18 am, edited 2 times in total.

matthiasr
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 190
Joined: Sat Apr 08, 2006 7:55 am
Location: Lübbecke, Germany
Contact:

Re: HTTPD Library - Capturing Form Data

Post by matthiasr » Mon Jul 26, 2021 11:06 am

I've replaced the attached stack. The original button still used the original library.
Last edited by matthiasr on Mon Jul 26, 2021 11:14 am, edited 1 time in total.

matthiasr
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 190
Joined: Sat Apr 08, 2006 7:55 am
Location: Lübbecke, Germany
Contact:

Re: HTTPD Library - Capturing Form Data

Post by matthiasr » Mon Jul 26, 2021 11:13 am

Oh my... ;)

Testing with 2 devices, one's running the app, the other's uploading the file, the content is displayed even without the answer dialog enabled, but only when the webserver is started with mrStart button. So it seems there is something wrong with your form.

Sorry for all the confusion.

I am not sure, if i should modify or delete my original post, so i leave it for now.

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Mon Jul 26, 2021 11:38 am

matthiasr wrote:
Mon Jul 26, 2021 11:13 am
Oh my... ;)

Testing with 2 devices, one's running the app, the other's uploading the file, the content is displayed even without the answer dialog enabled, but only when the webserver is started with mrStart button. So it seems there is something wrong with your form.

Sorry for all the confusion.

I am not sure, if i should modify or delete my original post, so i leave it for now.
Hi Matthias - thanks for the stacks. This is very interesting and it's good to see that it kinda works too.

Ok, so I downloaded your 1st stack and ran the stack on one computer, then accessed the webpage from another. With the Answer dialog box checked I could see the content of the CSV file, but nothing with it unchecked.

Then I downloaded the 2nd stack and I still get similar results if I do multiple runs.

The first time I run it shows me an answer dialog with the file name and other header info and the same info is displayed on the web page
The second time I run it shows me a blank answer dialog and no content info is displayed on the web page
and the Third time I run it shows me an answer dialog with the file content and the file content also displayed on the web page.

Its definitely progress though!

matthiasr
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 190
Joined: Sat Apr 08, 2006 7:55 am
Location: Lübbecke, Germany
Contact:

Re: HTTPD Library - Capturing Form Data

Post by matthiasr » Mon Jul 26, 2021 11:50 am

I had one or two additional answer dialogs in the script which i did not mention. ;)

With this stack here
HTTP-Test_mr_orgHTTPD.livecode.zip
(3.07 KiB) Downloaded 205 times
which uses LC's httpd library i am able to get the posted raw data displayed when using the mrStart button.

In my case i get this here
pRequest[content]=------WebKitFormBoundaryKGBLNMBegBFY5XHW Content-Disposition: form-data; name="datei"; filename="test.csv" Content-Type: text/csv Peter;Müller Peter;Maier Peter;Meyer Petra;Welbert Petra;Wiemann Petra;Weiß Dieter;Peh Dieter;Beh Dieter;Ceh ------WebKitFormBoundaryKGBLNMBegBFY5XHW--
I am testing with an real iOS device and Safari on macOS 10.15.7

How large is your .csv file? I am using a 1kb file for testing.

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Mon Jul 26, 2021 12:27 pm

v3 seems to work even better. Cool!

I'm just using a 1kb test csv file which works ok on second load. However it seems to fail with larger files or when I switch to another csv. I'm guessing some cleanup might be required when re-loading csv files.

I think the html form is a bit scrappy in my original LC file. I'll take a look at it and see if I can fix it... (probably some spurious or malformed tag structures).

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Tue Jul 27, 2021 9:07 am

matthiasr wrote:
Mon Jul 26, 2021 11:50 am
I had one or two additional answer dialogs in the script which i did not mention. ;)

With this stack here
HTTP-Test_mr_orgHTTPD.livecode.zip
which uses LC's httpd library i am able to get the posted raw data displayed when using the mrStart button.

In my case i get this here
pRequest[content]=------WebKitFormBoundaryKGBLNMBegBFY5XHW Content-Disposition: form-data; name="datei"; filename="test.csv" Content-Type: text/csv Peter;Müller Peter;Maier Peter;Meyer Petra;Welbert Petra;Wiemann Petra;Weiß Dieter;Peh Dieter;Beh Dieter;Ceh ------WebKitFormBoundaryKGBLNMBegBFY5XHW--
I am testing with an real iOS device and Safari on macOS 10.15.7

How large is your .csv file? I am using a 1kb file for testing.
I'm not sure if you have found this, but if I try using larger files, then I generally get a value in pRequest["headers"]["content-length"] (e.g. 7641) but pRequest["content"] is empty.

This was really bugging me, so I ran a few tests with various file sizes.

What I have found is that when I choose a file and immediately hit submit very small files (< 1KB) will upload without any problems. However when I choose a larger file (e.g. 5KB) and immediately hit submit it does not upload.

Interestingly if I choose a larger file then wait a while before hitting the Submit button the file does get uploaded and both pRequest["header"]["content-length"] and pRequest["content"] are populated.

I guess I was just being too impatient and hitting Submit before the file had loaded.

I've attached a new Zip of my updated HTTP-Test.livecode and HTTP-Test.csv for reference as this is what I have been using.

Cheers, Al
Archive.zip
HTTP Test & data
(2.55 KiB) Downloaded 199 times

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Tue Jul 27, 2021 5:23 pm

j9v6 wrote:
Tue Jul 27, 2021 9:07 am
matthiasr wrote:
Mon Jul 26, 2021 11:50 am
I had one or two additional answer dialogs in the script which i did not mention. ;)

With this stack here
HTTP-Test_mr_orgHTTPD.livecode.zip
which uses LC's httpd library i am able to get the posted raw data displayed when using the mrStart button.

In my case i get this here
pRequest[content]=------WebKitFormBoundaryKGBLNMBegBFY5XHW Content-Disposition: form-data; name="datei"; filename="test.csv" Content-Type: text/csv Peter;Müller Peter;Maier Peter;Meyer Petra;Welbert Petra;Wiemann Petra;Weiß Dieter;Peh Dieter;Beh Dieter;Ceh ------WebKitFormBoundaryKGBLNMBegBFY5XHW--
I am testing with an real iOS device and Safari on macOS 10.15.7

How large is your .csv file? I am using a 1kb file for testing.
I'm not sure if you have found this, but if I try using larger files, then I generally get a value in pRequest["headers"]["content-length"] (e.g. 7641) but pRequest["content"] is empty.

This was really bugging me, so I ran a few tests with various file sizes.

What I have found is that when I choose a file and immediately hit submit very small files (< 1KB) will upload without any problems. However when I choose a larger file (e.g. 5KB) and immediately hit submit it does not upload.

Interestingly if I choose a larger file then wait a while before hitting the Submit button the file does get uploaded and both pRequest["header"]["content-length"] and pRequest["content"] are populated.

I guess I was just being too impatient and hitting Submit before the file had loaded.

I've attached a new Zip of my updated HTTP-Test.livecode and HTTP-Test.csv for reference as this is what I have been using.

Cheers, Al
Archive.zip
This all seems to be an issue specifically with the Safari browser. With Edge, Chrome & Firefox the files will upload first time every time without any waiting around to press the Submit button. I've tested this with the following browsers:

Safari 13.1.2 (13609.3.5.1.5) - Fails to upload consistently
Safari 14.1.1 (16611.2.7.1.4) - Fails to upload consistently
Edge 92.0.902.55 - No problems found
Chrome 91.0.4472.164 - No problems found
Firefox 89.0.3 - No problems found
Firefox 90.0.2 - No problems found

Thanks for all the help and input from everyone.
Cheers,
Al

matthiasr
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 190
Joined: Sat Apr 08, 2006 7:55 am
Location: Lübbecke, Germany
Contact:

Re: HTTPD Library - Capturing Form Data

Post by matthiasr » Tue Jul 27, 2021 5:49 pm

Just fyi,
i tried to upload a bigger .csv (2.9mb) with Firefox and got an error message

Something like "Error: Connection interrupted...".
I have here a german error message, so i am not sure what the exact english one is.

Uploading with Safari btw does not give an error message, but then only about 9968 characters of the 2.9mb file are shown back in the browser.

Seems the httpd library does not handle larger files. So if you can make sure, that the file size does not exceed a special amount, then you are fine. But what, if the file size increases?

Did you ever consider to create also a desktop companion app instead of using a webserver on the mobile device, that uses sockets to transfer the .csv?
On mobile app you would open a socket and accept connections and one the desktop app would then connect to the mobile app and send the file directly.

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Tue Jul 27, 2021 6:40 pm

Thanks for the additional info. I may just set an upper limit on the file size that can be uploaded and deal with it that way. As a rule the csv files that will be uploaded won’t be more than a few hundred k, so if it can handle that size without a problem I will be happy.

I have considered a thick client for Mac, Windows and Linux, but the browser option is more appealing at this point in time as it means less code to support, plus the browsers are cross platform and ubiquitous.

On the subject of browsers am I right in thinking you are having no problems uploading with Safari? I am finding it flakey and have tried it on two separate macs.

Al

matthiasr
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 190
Joined: Sat Apr 08, 2006 7:55 am
Location: Lübbecke, Germany
Contact:

Re: HTTPD Library - Capturing Form Data

Post by matthiasr » Tue Jul 27, 2021 7:26 pm

I am testing with a real iOS Device running your last sample and an iMac with Safari for uploading the csv file.
I just tested with an 1kb file and today with a 2.9 mb.
With 1kb i never have the problem you've described. With the 2.9MB i see only about 9900 characters of the file in the browser

Running your stack on my iMac and uploading the 1kb from the same machine using Safari sometimes also does not work and the content of the file is not shown.

j9v6
Posts: 72
Joined: Tue Aug 06, 2019 9:27 am
Location: U.K.
Contact:

Re: HTTPD Library - Capturing Form Data

Post by j9v6 » Tue Jul 27, 2021 8:51 pm

matthiasr wrote:
Tue Jul 27, 2021 7:26 pm
Running your stack on my iMac and uploading the 1kb from the same machine using Safari sometimes also does not work and the content of the file is not shown.
This is what I get when I run the server on one Mac and upload from the other - always with Safari but not with other browsers. I tested the first version of HTTP-Test on Android & iOS sims, and on a physical Android device - but then I couldn't upload anything at all as the html was a little screwy.

I'll try building this latest version and deploying it to the Android device and see if I can upload with Safari from the Mac. Would be interesting if that works.

Post Reply

Return to “Internet”