Ultimate backscript in engine?

[DarScott]

I was wondering how much of the engine could be or should be in LiveCode, virtually (or really) in a stack in the engine. Would this simplify the engine or complicate it?

If any enhancements to LiveCode are needed to make this work, those may well be enhancements that would open the doors to writing better script libraries. The thought experiment might help us think of ways to make LiveCode better, even if the ultimate stack is not put into the engine.

The engine, stack libraries, and maybe even externals should all be able to look good to the LiveCode programmer and have the same feel (at some level).

[monte]

Can you have another stab at explaining what you mean because I have no idea what you're talking about

[DarScott]

I'll give it a try, Monte. I was hoping to not say much so I can see what others are thinking.

Uh, you know how, except for files, GET URL is translated to a command that is dropped into the message path? It is virtually an engine operation, but is really in LiveCode. That has a feel for what I'm talking about and might suggest some of the problems that might come up. It uses some more fundamental commands and functions involving sockets. Now, even sockets can be made out of some simple private BSD sockets engine commands, if good support for creating blocking commands is available. (And those don't really have to be private.)

Um, an easier example. This might be a bad example because the coding is a sunk cost. But it is a good example because it is simple and doesn't raise issues. Consider the function annuity(). It can be in the backest backscript. It can be fully independent. This ultimate backscript would be in the engine. That is if annuity() falls all the way down the pinball machine and drops into the black hole, it still has one more chance, kinda like those messages that can be blocked. The engine backscript would work exactly like the most back backscript. Except, it might have some special features which have to be determined. And it does not have to be implemented so the function goes down the message path, there might be some other way; maybe the compiler can create a call to the ultimate back script.

Maybe this would have some advantages. It might bring in some expertise. It might have performance costs.

But it also reminds us that even if it is not done, having good support for making blocking commands. I don't like to use blocking commands, but some customers want them. So, it would be nice to do a full callback based implementation and put a blocking wrapper around that. (Now, it may well be that partial blocking like with GET URL is fine with most people and this is a nonissue.) It could remind us of other things, like what would be nice in open language.

If worth pondering, then I encourage it. If not, then I don't want to bother folks; we have lots to do.

[monte]

OK, I understand what you mean now... I think you can add a blocking wrapper around things by having a completed function and use wait until completed() with messages... or do you want complete blocking (without messages?)... To do that we would need something like a reference counted wait...

Code: Select all

on someCommand
    doSomething "somethingCallback"
    lock wait
end someCommand

on somethingCallback
   unlock wait
end somethingCallback

Personally I think we need less blocking in the platform rather than more...

[DarScott]

Yes, concerning implementing blocking, I currently do wait "with messages" or "for message" depending on the case.

And I would want all of my callbacks to work. And all the callbacks of the modules I use. But block all else.

I agree with you on less blocking. Maybe that is really a user education issue. I can create partial blocking with the above, explain the limitations, and encourage nonblocking. But, when it gets to the bottom line, when a customer wants blocking, I provide it, even hard blocking which can complicate a whole module.

But, in any case, my blocking does not work like the engine's blocking. That means it is a new concept and it emphases that the stack library is not like the engine.

I wonder if that is a distraction from the notion of the engine being partial implemented in a script.

Another distraction: if a pcode machine is coming, then maybe the script doesn't have to be there. Maybe the script is really pcode.

[mwieder]

@Dar- are you suggesting (I've read through this a few times and I *think* I'm starting to get a feel for it) a backscript that sits *behind* the engine?

[DarScott]

@MWeider, in the simplest view, in front, but that might be misleading.

The engine would consist of two parts, the LiveCode implemented part (maybe only in the form compiled by LiveCode), and the part compiled in some C-like language. The first would be like a stack in front of the second in a simple approach. But the two make up the engine.

By being in front, we are making presumptions about how this fits in with the message path. It might have some advantages that way. If annuity() is removed from the C-like implementation and put into such a stack in front of the C-like engine, I can replace annuity() with my own without a special name. But that might have a performance hit. And it might break some scripts that relied on a quirk of the built-in version. So,instead of in front, it might be some floating "stack" that the LiveCode compiler sends to or calls or something even more effective for this. But, in front can work and the compiler need not know anything about it.

I don't know if am ready to say I'm suggesting anything. I'm somewhat musing. Maybe others have had similar wonderings or have strong opinion one way or the other.

The use of LiveCode in the engine is not really a prerequisite for putting some engine commands and functions on the message path, that is just one natural way to handle one approach to putting LiveCode in the engine.

Maybe the gain is little and this might be a lot of work for perhaps slowing things down. As I said, my thinking here is early and mostly wondering.

[mwieder]

I'm missing why that's different from putting an annuity() function into a frontscript.

[DarScott]

Sure an engine stack can be virtually the first frontscript or the last backscript.

And engine features can even be implemented as a frontscript or backscript. However, depending on what features are implemented there, the classical engine view can break and even the whole execution can break if those are removed. Also, it would be harder to hide primitives (if desired). That is if sockets is implemented from core operations like BSD sockets, then the issue of whether those are undocumented commands or what.

A frontscript, virtually in front or a real one, could be a way to go.

The problem with implementing this as a real frontscript is that it would require changes in everybody's code to insert the script in front and where is the stack kept and so on. If this is in the engine, the behavior can be such that there are no breaking changes, the enhancements are things that would have caused errors before. A hybrid scheme might be used in which the engine executes a bit of LiveCode before startup and that can insert an embedded stack into the front. Even then scripts might break that do not expect it to be there, but with the implementation of some add-ons, that is not likely.

Consider the property 'left'. Maybe that can be implemented with getprop and setprop. Whether it should be or not, is of course a big question. Thought for some it might be a trivial and easy question. And whether that is in front or in back is another. The property left can be defined in terms of the property rectangle.

And, to confuse things, my pondering is conflating this with the notion of allowing overrides of some commands, functions and property operations, that is, virtually dropping those onto the message path as with current user defined commands functions and property operations. I like the ability to override, even if to add some patches while bug fixes are in the works. For example, a bug in sockets causes a crash. (Maybe that is fixed.) A layer over that might avoid the crash. That would need some way of using the internal offending socket command in the safe cases and a way to create an error in others. And it may be that putting some operations on the message path should be a fully separate topic.

Also, conflated is the ability to add new commands that are like engine commands.

The definition of the engine would be the same. The difference really does not have to be visible. However, the embedded scripts might be handy for documentation writers. "Ah, so that is what 'left' does!" It can add people to development. It might make the engine smaller.

[mwieder]

I think a lot of (actually I think it's *all*) of what you're asking for is going to be delivered in the Open Language doodad.

[DarScott]

I don't know if I'm at the point of asking for anything. My intent was to make a comment and see what discussion emerged if any. It seems this is alien for most people. That is fine.

And yeah, perhaps much much of this will come with open language.

If I was to ask for anything related to what we discussed, it would be to blur the contrast between engine commands and script library commands. I'm not sure what this involves. Maybe the ability to create errors at the caller's line, but good error documentation and throw enhancements might be enough. Maybe the ability to set 'it' on return can be blessed and documented. The ability to get the numberFormat and other local properties of the caller would be nice, but maybe documentation of the ability of 'do' to work in the caller's context would be enough. The ability to have unevaluated parameters might be nice. Support for blocking might be nice, but maybe the pseudo-blocking we have is good enough. The ability to have easy internal externals without the overhead of unloading and loading would be nice. The ability to have private commands in a stack's external would be nice. But all that is a distraction to the main topic and might be part of some other distraction, I mean, discussion.

Long ago in some other century I implemented my own Forth. Perhaps 80% of that was in Forth.

Whether because it is a bad idea, or the idea is too alien, this might not be the right thing for a LiveCode engine.

[mwieder]

Long ago in some other century I implemented my own Forth. Perhaps 80% of that was in Forth.

My guess is more like 90%, IIRC. You have a bare number of primitives to implement, and then by the time you've implemented colon and semicolon, the world is yours.

But you can get away with that in Forth because it's so close to the metal. There are tradeoffs in something as high-level as xtalk between putting something in the engine for speed and putting it in a script for flexibility.

One thing I'd like to bring in from Forth is the idea of vocabularies. It seems like a way to ease into namespaces.

[LCMark]

There is a certain elegance with the idea of a language being 'self-hosting' - i.e. written in itself, and perhaps one day LiveCode will become that (although I'm not sure the language features are quite yet at a point where this would become a possibility - from the point of writing it).

That being said, the plan is to allow 'modules' to be written in LiveCode as well as native code. These 'modules' don't sit in the message path, but instead extend what you could call the engine feature-set... i.e. LiveCode just becomes another implementation language for them, alongside C/C++ and any other language that bindings are developed for (such as Java...).

The LiveCode modules will still (most likely) have to sit on native code to a certain degree (just as Java needs 'native' methods to talk to the system), however at some point when the native interfaces are well-enough refined (there has to be some sanitization due to cross-platformness and such) then it is quite possible that a lot of current functinality could be moved to LiveCode from native code.