Post Form Data to Browser Widget

[wl5]

I am developing a program that interacts with a website that I do not control the HTML of (I am associated with it but have no interaction/say in how the site is ran or coded). The website uses complex 2FA which requires JavaScript and the login process to be completed in a browser widget due to the flow of user interaction. Once the user is logged in, I would like to hide the browser widget and have the user be able to send queries/receive responses via a custom interface built in LC which makes the site more user friendly.

All interaction on the site is through forms that POST data. If I use the LC "Post xxx to URL xxx" the site rejects the request and logs the user out since the POST command does not contain any requisite data passed along through certain JavaScript and session data. I have tried multiple ways of doing this and get the same result every time. Also, any incorrectly formatted POST (wrong name or incorrect option) will cause the site to error out and log the user out. (It is very large, very legacy, and very much cobbled together through years of "upgrades").

I can use "do xxx in widget" to do a JavaScript .click() function and simulate clicking the form which returns the appropriate result in the browser widget. This works for some basic functions but becomes tedious with a large form and having to write a lot of JavaScript to manipulate the DOM.

What I would ultimately like to do is be able to POST data directly to the browser widget so that I don't have to write a bunch of JavaScript handlers that manipulate the innerHTML form data. The Browser widget would maintain the session throughout and just act as an invisible data source where the HTML responses could be parsed.

The more I search the more I realize I may just have to go down the "do xxx in widget" route but hopefully I'm wrong and overlooked something.

[FourthWorld]

All interaction on the site is through forms that POST data. If I use the LC "Post xxx to URL xxx" the site rejects the request and logs the user out since the POST command does not contain any requisite data passed along through certain JavaScript and session data. I have tried multiple ways of doing this and get the same result every time. Also, any incorrectly formatted POST (wrong name or incorrect option) will cause the site to error out and log the user out. (It is very large, very legacy, and very much cobbled together through years of "upgrades").

Forget the browser widget. It just adds complication. Let's see what we can do to get you going with how the site owner wants you to, with POST.

What additional data is needed that's missing to cause the logout?

[stam]

Not sure i entirely understand the problem, but if it's a POST API you're trying to debug and construct correct calls to, you may want to look at postman. Excellent tool for this type of work and free for up to 3 users without the features for larger teams.

Maybe seeing what works in postman will help sort out the LC code...

HTH
Stam

[strongbow]

I also highly recommend using Postman to assist you in sorting out your problem. I found it invaluable for sorting out working with APIs.

[wl5]

Forget the browser widget. It just adds complication. Let's see what we can do to get you going with how the site owner wants you to, with POST.

What additional data is needed that's missing to cause the logout?

Is it possible to load a page that requires JavaScript outside of a browser widget? By requires JS I mean the page explicity will not load and comes back with a "JavaScript Required" type error if not enabled, no other content is sent. All of my current attempts have failed due to this problem.

[FourthWorld]

A web page is just a client-side display. When submitting a form, that is sent from the browser using POST.

LiveCode offers a POST command, which lets you provide the same functionality of most web form submissions, but without the overhead of needing a browser to display a web UI.

Look at the form in the source of the page you want to submit to. You'll see an "action" attribute there. That will provide the URL to send the POST data to. The form fields provide the data. The field names in the form along with their data are submitted as name-value pairs in the POST data. See the LC Dictionary on the POST command, examine the source of the page you're interested in emulating, and come back with the questions you'll have on how the two come together.

[wl5]

As I stated in the initial question, the page requires JavaScript to even load. The LC Post command just returns a page stating JavaScript is required since it is just a rote command with no context or associated browser. In addition, the POST command does not carry session data along with it so the page being posted to doesn't know the user is logged in.

The need is for the post to be processed within the browser widget. Think of it as "post xxx to widget Browser" or "put post xxx into the URL of widget Browser" type behavior.

I can get the results I want from DOM manipulation using innerHTML to change form entries and JS .click() form submissions but I was hoping to write some cleaner code than that. After reading all of the topics on the matter and responses here I think there really is no good way to do this outside of writing a bunch of "do xxx in widget" JS to modify the DOM.

[bwmilby]

I’ve done something along the same conceptual lines (MS Access controlling a web site in IE). My approach was to abstract as much as possible and use simple commands to do things like setValue(pObjectRef, pObjectValue) and clickItem(pObjectRef). Should be able to do something similar where the handler builds the JS and executes it for you.

[FourthWorld]

As I stated in the initial question, the page requires JavaScript to even load. The LC Post command just returns a page stating JavaScript is required since it is just a rote command with no context or associated browser. In addition, the POST command does not carry session data along with it so the page being posted to doesn't know the user is logged in.

The need is for the post to be processed within the browser widget. Think of it as "post xxx to widget Browser" or "put post xxx into the URL of widget Browser" type behavior.

I can get the results I want from DOM manipulation using innerHTML to change form entries and JS .click() form submissions but I was hoping to write some cleaner code than that. After reading all of the topics on the matter and responses here I think there really is no good way to do this outside of writing a bunch of "do xxx in widget" JS to modify the DOM.

It sounds like the service is working really hard to prevent people from accessing it through automated means.

Is it a public service? If so, may I ask what it is?

[wl5]

It sounds like the service is working really hard to prevent people from accessing it through automated means.

Is it a public service? If so, may I ask what it is?

It is not a public system, it is owned by my employer, hence the lack of functioning APIs and methods for easily accessing the data. There are at least 2 apps on the market that already have the functionality I am replicating but they are iOS only. I am attempting to make the first cross platform solution. These apps are used by close to 75% of employees with access to this specific system so it has a strong backing.

[wl5]

I’ve done something along the same conceptual lines (MS Access controlling a web site in IE). My approach was to abstract as much as possible and use simple commands to do things like setValue(pObjectRef, pObjectValue) and clickItem(pObjectRef). Should be able to do something similar where the handler builds the JS and executes it for you.

Thanks. That looks like the route I’m going down.

[FourthWorld]

It sounds like the service is working really hard to prevent people from accessing it through automated means.

Is it a public service? If so, may I ask what it is?

It is not a public system, it is owned by my employer, hence the lack of functioning APIs and methods for easily accessing the data. There are at least 2 apps on the market that already have the functionality I am replicating but they are iOS only. I am attempting to make the first cross platform solution. These apps are used by close to 75% of employees with access to this specific system so it has a strong backing.

It's such an uncommon way to move data I hadn't anticipated it would be a choice; most companies find it more cost-effective to provide an API.

Sounds like management has their mind made up on this, but if a lighter API-driven approach becomes a consideration down the road, LC also has a server edition that could be used to handle this more directly (or PHP, Node.js, Python, or pretty much anything else that plays into team strengths).

I don't know the business case specifics here, but if they ever anticipate making other services available most API code is easily reusable, often streamlining development and maintenance of both native and web clients.

[wl5]

It's such an uncommon way to move data I hadn't anticipated it would be a choice; most companies find it more cost-effective to provide an API.

Sounds like management has their mind made up on this, but if a lighter API-driven approach becomes a consideration down the road, LC also has a server edition that could be used to handle this more directly (or PHP, Python, or pretty much anything else that plays into team strengths).

I don't know the business case specifics here, but if they ever anticipate making other services available most API code is easily reusable, often streamlining development and maintenance of both native and web clients.

This is a completely internal system never designed for public interface. It was only designed for about ~7000 employees operating in the core business. The current apps that have been made to interact with the system all use browser type backends that parse out HTML behind the scenes and present it to the user in more friendly manner. There literally is no API and there never will be because the intended use case is via the browser and the company wants it that way for control. Those of us working on these alternative solutions are not "sanctioned' but not necessarily afoul of the rules either as long as we don't actively try to go outside the guardrails.

JavaScript is explicitly required on all pages because there are a lot of trackers and counters that are intended to rate limit and reduce the usage of certain resources as well as closely track session data/activity due to some large security breaches in the past. There are absolutely no AJAX or JS driven UI elements on the entire site, yet JS is explicitly required or else you are logged out.

So essentially to make the whole app work it has to fool the server into believing an actual user is accessing the site. This is the requirement for the browser widget to access the site and be the portal for data transfer back and forth. At this point it looks like building a bunch of custom functions for each page and doing a lot of "do xxx in widget" JS is the only way to go. I have been successful using Selenium in Python, however I prefer the cross-platform build of LC.

[FourthWorld]

What happens on the client is ultimately known to the server by signals the client sends. Those signals can be emulated.

But I've never seen the page source, and since the exercise is to bypass elements put into place by the site owner to obviate automation, further assistance on how to resolve this by emulating the signals doesn't seem a good idea.

Your initial focus of using "do as JavaScript" to control the browser widget, though tedious, might be your best bet.

[wl5]

I understand how headers, HTML requests, cookies, and all of that works. This question really had nothing to do with that. I think all of the responses have missed the point of the question.

It is not "What is POST and how do I deconstruct this web app to use the LC POST command?"

It is "Is it possible to send a POST command to the Browser Widget?"

Apparently it is not possible so I will work from there.