Browser Widget Enhancements

Bringing the internet highway into your project? Building FTP, HTTP, email, chat or other client solutions?

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
Brahmanathaswami
Posts: 52
Joined: Mon Jun 24, 2013 4:10 am
Location: Hawaii
Contact:

Browser Widget Enhancements

Post by Brahmanathaswami » Tue Jan 31, 2017 3:14 am

Hello everyone, there is some talk on the list about make having stronger integration between LC and the browser widget. Jonathan Lynch provided an enhancement request here:

http://quality.livecode.com/show_bug.cgi?id=19184

right now HQ says they can do it, but for a price, it's a non-trivial job, and hours to get it done need to be covered, cost wise. Actually the quote they gave me was very reasonable, but still beyond our budgets. if anyone is interested, we could all "kick in" some amount to get the job done ..

Just few of the requirements we are interested in having working

1) The ability to pass variables from LiveCode to Javascript (current method has issues, see the enhancement request for details)

2) Synchronous, non-callback methods for integrating functions both from LC to JS and from JS to LC. (not possible now. if you have a function in LC and ping it with JS, the browser widget cannot get any value in return)

3) Full layering: Right now, the browser widget can be layered a little bit by putting it inside of a group, but not very well.

4) multi-threading

5) masking of browser widgets

Jonathan has more ideas.

Basically it means that html5 "apps" could be part of your LC "package" and the browser widget behaves like any other control and is a fully integrated into the message path as any other control (well, almost). at a simpler level if you have good html/css skills you could create small "views" like

Code: Select all

<ul class="panel">
  <li class="row1> Oranges </li>
  <li class="row2> Apples </li>
  <li class="row1> Papaya </li>
  <li class="row2> Pinapple </li>
<ul>
+ a little CSS and bingo you have a new control that is responsive and flows like any web page <div>  Changing attributes of multiple elements become super easy because you touchone or two values in the CSS vs having to tweak many object in a group.  But then you would want the JS calls fromt his small html to beable to "talk" to LC on the back end.
i.e. in a MVC kind of environment, you could use HTML5 to build views where the JS is an API to your view, but the deep models and controllers running on the back end are all live code. typicall example is: you want the html browser widget to save data to the same sqllite dbase on mobile that the LC package is reading and write to and from.

On the high end, you could use LC to wrap existing and "big" web apps and have them talk to each other synchronously.

Any one interested?

JonathanLynch
Posts: 22
Joined: Mon Apr 10, 2006 1:42 pm
Location: Atlanta
Contact:

Re: Browser Widget Enhancements

Post by JonathanLynch » Tue Jan 31, 2017 3:06 pm

Hi everyone,

Just to explain a bit more. Think of all of things you have wanted to do with LiveCode, but you decided not to because it would be impractical. Have you wanted to make multiuser online 3D games with LC? That would be difficult as things currently stand, because LC does not have a native 3D object.

With a maximally functional browser widget, here is how you could do that:

1. Store the 3D landscape data on a server.
2. Store the current position (and other characteristics) of other players on a separate table online.
3. Asynchronously load the information from step 2. This can be done using the Load command, using HTML webworkers, or using an external designed for quick-loading data. (I believe LC has just such an external available for indy and business users).
4. Pass the environmental data and other player data into the browser
5. Present the updated environmental data and other player data through a browser window using webgl technology. This is 3D javascript technology that is currently available.

I have a working 3D globe with markers in an app right now, using webglearth as the javascript source and openstreetmaps as the map source. This would work a lot better with a fully functional browser widget, but the fact that it is working right now is proof that this can be done. It looks as good as the 3D street walking effect found in Pokemon Go.
The key to a happy life is to practice love for all things.

JonathanLynch
Posts: 22
Joined: Mon Apr 10, 2006 1:42 pm
Location: Atlanta
Contact:

Re: Browser Widget Enhancements

Post by JonathanLynch » Tue Jan 31, 2017 3:10 pm

Nonblocking spinning wait indicator

Here is another example of how this would benefit.

Several LC users have commented on the need for a nonblocking spinning wait indicator. I posted an example of how to do this in a browser widget on livecode share. Of course, the wait indicator takes up a rectangle shape on screen and cannot have any transparency. If browser widgets could have an alpha channel, just like an image, then we could have a nonblocking wait indicator that fades on top of the app, lets the user know that the app is thinking, and fades off when it is done.
The key to a happy life is to practice love for all things.

JonathanLynch
Posts: 22
Joined: Mon Apr 10, 2006 1:42 pm
Location: Atlanta
Contact:

Re: Browser Widget Enhancements

Post by JonathanLynch » Tue Jan 31, 2017 3:41 pm

3D Image Manipulation

Javascript is excellent for manipulating images, using transform, translate, and other JS commands.

If we could pass large variables directly to the browser widget from LC, we could do the following:

1. Get image data from an image in LC. (it would probably be 'put image "myImage" into tImage' rather that using the actual 'imagedata' property)
2. Pass that image variable to a canvas object in JS
3. Perform a 3D transform (or whatever transform you want) on the image, probably by setting CSS characteristics of the object
4. Use the JS getImageData() method to get the transformed image information back into a variable
5. Synchronously pass that image variable back to LC
6. Put that image information back into an LC image object.

This would be many many times faster than any sort of scripted image manipulations that could be done in an LC script. Here is an example of the kind of 3D transformations you can make in javascript/html5/CSS: https://stemkoski.github.io/Three.js/Reflection.html
The key to a happy life is to practice love for all things.

JonathanLynch
Posts: 22
Joined: Mon Apr 10, 2006 1:42 pm
Location: Atlanta
Contact:

Re: Browser Widget Enhancements

Post by JonathanLynch » Tue Jan 31, 2017 4:00 pm

Seamlessly Using Media Devices, Like a built-in Camera

If we had the ability to properly layer the browser widget, we could do the following.

1. Set up a browser widget that uses the getUserMedia() method. (https://developer.mozilla.org/en-US/doc ... tUserMedia). It would have to obtain user permission to do this.
2. Present the user's camera image through the browser widget.
3. Superimpose a LiveCode image (perhaps a gif animation) on top of the browser widget.

For example, Pokemon Go uses this technique for the part where users can capture wild pokemon. You could also use this for an augmented reality application that uses your position, orientation, and tilt to determine exactly what you are looking at, and superimposes information on top.
The key to a happy life is to practice love for all things.

JonathanLynch
Posts: 22
Joined: Mon Apr 10, 2006 1:42 pm
Location: Atlanta
Contact:

Re: Browser Widget Enhancements

Post by JonathanLynch » Tue Jan 31, 2017 6:02 pm

Zoomable Controls

We can do some zooming with LC objects, such as by changing imagesize inside of a group or by changing the pixelscale. However, it is very difficult to smoothly zoom in and out of a complex livecode object. For Ready Wrigley Activities, I had to limit the zoom levels to 2X and 4X, because any other scaling resulted in a raggedy look.

However, browsers are designed for making smooth presentations. With improved browser widget control, one could easily do the following:

1. Create a browser-based control, such as a 3D carousel of images
2. Pass images into the browser from LC
3. Set up javascript functions to zoom in and out of the div that contains the carousel
4. Detect when a user clicks on the zoomed carousel image and do something with that click

With alpha channel control, this browser widget could be seamlessly integrated into the rest of LC stack.
The key to a happy life is to practice love for all things.

rblackmore245
Posts: 67
Joined: Fri Jun 12, 2015 9:42 am

Re: Browser Widget Enhancements

Post by rblackmore245 » Mon Mar 06, 2017 5:14 pm

Would love to be able to use the browser object with the getUserMedia() being able to use ready made javascript API's with my apps is a must.

Simple things like a javascript barcode reader which QuaggaJS does a great job, but you need the getUserMedia permission.

Brahmanathaswami
Posts: 52
Joined: Mon Jun 24, 2013 4:10 am
Location: Hawaii
Contact:

Post by Brahmanathaswami » Mon Mar 06, 2017 6:40 pm

@rblackmore... I have gotten an estimate from HQ on browser enhancement development. We would need a minimum of $7,000.00 in a kitty, $11,000 if we want robust error checking, to enhance the browser widget to a point where can get it to communicate "runtime" with the LC context it is running in. I don't know if you or your company has any coin for this. I'm willing to throw in $1,000.00 but that's about the limit I could get approved here for this. If we did it, perhaps the getUserMedia() could be rolled in... We need a way to crowd source funding these small projects, because HQ may not want to do this on their feature upgrade framework. I'm not sure how they decide, things like FileMaker and WordPress hardly seem to have a greater priority than the internet browser, (what is more ubiquitous than fetching web content!) yet for those HQ launched enhancement fund raising. For this we need to figure out how to set that up yourselves. If you are interested. Send me a note or reply here. I can put the estimate from HQ somewhere for you to read, it is very detailed and quite reasonable given the scope of the requirements.

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Browser Widget Enhancements

Post by [-hh] » Sat Mar 11, 2017 12:47 am

I can't see any reason why the enhancements you list should be made via the browser widget.

You currently can't even have full access to the widget in linux. And it has (also incl. the enhancements you wish to have) much less properties available than other LC controls.

None of your examples actually relies on the language javascript, but you wish to use packages that access user's hardware. This makes the speed, not the language.
HTML5/javascript is not faster than LC in handling images, but _compiled_ javascript is faster and even more faster are packages like WebGL which use the users hardware (GPU).

So, what you essentially want is a FFI for HTML5/javascript that allows to access the CPU/other media by such packages. Has a priori nothing to do with the browser widget which could even become a hurdle/bottleneck if it is not enough up-to-date.

TMHO it would be much better to bundle these efforts in the development branch HTML5, which is a separately licensed but meagre and sickly child of the company. The amount mentioned above is certainly still several times available for such features from the HTML5-funding.

A HTML5 standalone could be as fast as all examples that you describe and could use all these packages, having the _full_ power of user's professional web browser available.

In sum: The enhancements that you request now for the browser widget are _part_ only of the list of features that I had awaited to have, derived from the early announcements of the company, for a HTML5 standalone, may be also depending on other LC progress (FFI).
shiftLock happens

Brahmanathaswami
Posts: 52
Joined: Mon Jun 24, 2013 4:10 am
Location: Hawaii
Contact:

Re: Browser Widget Enhancements

Post by Brahmanathaswami » Sat Mar 11, 2017 4:39 am

HH I understand your wish for a broader implementation. But I don't think we are on the same page.

You write: "So, what you essentially want is a FFI for HTML5/javascript that allows to access the CPU/other media by such packages. "

Actually no: the use case I envision/need/would hope for, is quite simple

if we have a function in JS that calls a livecode function, we want to have the value generated by the livecode script to be immediately registered by the javascript *before* the JS steps into it's next line. i.e. synchronous communication.. And vice versa. the JS in the browser widget simply becomes a transparent addition to the message hierarchy.

That's really all I'm talking about for now. A typical use case would be: if you have an HTML5 developer who prefers to work in that platform/language, you could hire him to do some interesting module; load this in your LC app package and then, e.g. save data from that "module to the SQLLite database which is currently inaccessible via the browser webkit. The html5 module would pass data to an LC script which would save it to the database, and the LC script would return values from the database to the HTML5 module, but it has to happen synchronously.

The problem with the "meager and sickly child" branch is, IMHO, never going to be fixed, because, quite simply: you have to wait while Emscripten is downloaded. I can't envision *any* users I am deal with who would have the patience for that. I see the you think otherwise... but that's not really our use case:

Since we already have a "giant plug-in" pre-installed on virtually all platforms: the web browser/kit... doesn't it make sense to just be able to drop a small html/js/css package into the mix of LC modules?

Someone else here on our team created a marvelous app in ionic, using Angular 2...you can get it on line "Spiritual Workout" ... I was able to extract the core of that app, once you strip away the ungodly mess of IDE libraries/spaghetti and snakepit of ionic libs (10,000+!)... I had a small core of code <900K of (html/js/css) that actually runs *really, really* well in the browser widget.

I would use it (and maybe will later) if it were not for the heavy load of sound files that go with it that push our app package over the deliver-by-wi-fi limits... BUT! and this is the deal breaker: we cannot get anything in that app to talk intelligently to the rest of the larger LC frame work(stack/card/widget) that hosts it. And that's what I want... simple requirements like save a date reminder to the MySQL dbase or add somepage in theHTML5 module to the list of "favorites" in the overall app framework. Without the synchronous comm; having the JS synchronous part of the message path, it doesn't really work well enough to go there.

For now we are planning to go the way of using URL schema to fire links to other apps and because of this inability to get the JS in to the msg path, we have to abandon LC for a lot of future development and instead work on a multi-app ecosystem using the cross app linking URL schema. The kind of mega integrated app I envisioned is only partially achievable.

Again the use case is: engage HTML 5 talent and skill sets into the mix...so let's not talking about "doing it in LC".. you just cannot achieve the same level of smooth motion graphics/transitions which are mission critical visual production values in my world--if it doesn't flow, the platform is not considered viable by other stack holders, and yes, I have used and continue to use Animation Engine and No, it still does not meet the level of expectations required by those who have honed this video production skills where ken burns and transitions etc are smooth a silk. They "sniff" at LC like it was some kind of retro grade toy when it comes to the presentation layer, meanwhile they are seeing html5 apps where the visual production values are top-notch.

FWIW, you wrote me before on the list about how badly I had implemented some attempts at this in another stack and I appreciated that feedback and perhaps, one day, when I get back to it I will try again to build some small things in LC and get your guidance and to it right and get much better results... but that's not the issue: the issue is utilizing talent that has no idea what Livecode is and who will never, ever bother to learn it, the new ID's for building HTML5 productively will eclipse the LC IDE by the end of this year. It's already happening, big time. I want to leverage that trend back into LC if we can. I think it would a enormous boost for Livecode worldwide if this one simple thing could happen: JS in the widget becomes a true part of the message path.

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Browser Widget Enhancements

Post by [-hh] » Sat Mar 11, 2017 8:27 pm

BR wrote:... A typical use case would be: if you have an HTML5 developer who prefers to work in that platform/language, you could hire him to do some interesting module; load this in your LC app package and then, e.g. save data from that "module to the SQLLite database which is currently inaccessible via the browser webkit. The html5 module would pass data to an LC script which would save it to the database, and the LC script would return values from the database to the HTML5 module, but it has to happen synchronously ...
Thanks BR. I see now better your special needs. And I'm happy the team found a solution. Hope the slow down in speed on the HTML5 side by the massive amount of callbacks is not too extreme.
BR wrote: ... those who have honed this video production skills where ken burns and transitions etc are smooth a silk. They "sniff" at LC ...
Once again, the smooth transitions are only partially due to special production skills. It's mainly the fact that these packages have the option to access the client's hardware/GPU. If LC could open the access to such routines it could also produce such smoothing.
But I accept the sniffing, if someone is able to write the source code for such transitions (accessing the hardware).
BR wrote: ... you wrote me before on the list about how badly I had implemented some attempts at this in another stack ...
Please point me to that post. I can't remember one that could have caused such a negative feeling and I would like to apologize and avoid such wording in the future.

I would like to support your "widget-project" but I already arrived at my limit by supporting HTML5 deployment (I'm not selling anything made with LC). I'm waiting now until the end of the year for further progress there. If there is no, I'll cancel that and switch this part to your "widget-project".

p.s. A new addition to HTML5/javascript, 'Promise', is also going in your direction (lets asynchronous methods return values like synchronous methods, not immediately returning the final value).
shiftLock happens

trice001
Posts: 1
Joined: Sat Mar 18, 2017 10:03 am

Re: Browser Widget Enhancements

Post by trice001 » Sat Mar 18, 2017 10:05 am

thanks for that.

Monox18
Posts: 118
Joined: Tue Nov 25, 2014 9:48 pm
Location: Deggendorf, Germany

Re: Browser Widget Enhancements

Post by Monox18 » Fri Jan 08, 2021 4:24 pm

Has there been any updates on the original request? Couple years have passed by and the bug stays "hibernated". The browser widget has a ton of potential and benefits. Of which many were mentioned in this post and the bug report.

The 3D transformations example https://stemkoski.github.io/Three.js/Reflection.html is wonderful, it works in the browser widget and it shows great potential for 3D rendering.

The getUserMedia() method is disabled because the CEF browser requires the --enable-media-stream flag to be passed along with the command line launch request. LC has a hardcoded way of launching the CEF. So you need to modify how LC launches the CEF browser by adding this extra flag and then getUserMedia() would work. I made a a post about this http://forums.livecode.com/viewtopic.php?f=66&t=35193
Monox
Developing a Cyber Physical System.
https://www.monoxware.com/

Post Reply

Return to “Internet”