Sockets: Multicast or Socket Options

[DarScott]

It is very likely that in a short time I will need the ability to send and receive multicast packets. I can, of course, make a couple of externals, one for Windows and one for OS X. This has some advantages. But, I will be duplicating a lot of what is already done in LiveCode.

An alternate approach would be to enhance LiveCode. This allows a contribution to the community. I'd like to be part of that.

Multicasting is variation of UDP datagrams in which datagrams are sent to groups and a socket can be set up to receive from the group. It is like broadcasting but has important differences.

One approach is to enhance the syntax and implementation of 'accept' and maybe add a few things to tie up loose ends. Another is to add socket options. There are several approaches that might work for socket options: global setup properties, socket properties (new concept, see my other post), or new commands/functions. Socket options can open other doors. I can elaborate on the syntax possibilities.

I have a few questions on being a contributor.

Do all five platforms have to be accounted for in a change? That is, can I do Windows and OS X and encourage others to help with other platforms?

How do I make sure what I create is acceptable?

[DarScott]

Perhaps one way to do socket options is to have some way to get the handle of the socket. Like windowID. Then an external can be created to do socket options.

[DarScott]

Socket options would also workarounds for bugs or design decisions.

For example the bug in which a datagram accept would close when its buffer is too small for incoming data could have socket option workaround.

[DarScott]

I plan to find and join the right mailing list and do the right things to get started in helping.

If I add multicast to Livecode without adding socket options to LiveCode, it might be like this:

An 'open datagram socket' is allowed to have an IP address in the multicast range. If a nonempty value is specified in defaultNetworkInterface then, it is set appropriately for the outgoing source socket (IP_MULTICAST_IF?). Loopback is enabled to allow receivers to be on the same computer (IP_MULTICAST_LOOP). The TTL is left at 1; allowing use only in the local LAN. No syntax change is needed.

The 'accept datagram socket' command is enhanced to allow the optional clause 'from group <IP-address>'. This allows datagrams to be received from the specified multicast group (IP_ADD_MEMBERSHIP?). No other groups can be added for that port. No source list or source blocking is available. If a nonempty value is specified in defaultNetworkInterface then that is used (also IP_ADD_MEMBERSHIP?). Reuse the socket (SO_REUSEADDR and SO_REUSEPORT?). A language syntax change is needed.

Buffer sizes are set to at least the maximum size of a datagram socket in both cases (SO_RCVBUF and SO_SNDBUF?).

I think the socket options I mentioned are applicable on both Windows and OS X, except SO_REUSEPORT is not used on Windows.

This makes a lot of default assumptions. If more flexibility is needed then more syntax can be added or some sort of socket option capability can be examined.

[LCMark]

An alternate approach would be to enhance LiveCode. This allows a contribution to the community. I'd like to be part of that.

Your contributions would be more than welcome - making sockets more flexible would be good for everyone using LiveCode... It is an area of the engine which has not seen much love in recent years.

Do all five platforms have to be accounted for in a change? That is, can I do Windows and OS X and encourage others to help with other platforms?

Ideally contributions are cross-platform - at least for the platforms it makes sense for. In this case though, most of the code for sockets is cross-platform - there are some minor differences in the code for Windows, but the UNIX based systems all use the same codebase. As it stands, only the desktop platforms support sockets - we've made some movement towards getting them up and running on mobile platforms with the integration of SSL there, but haven't yet had a chance to finish the plumbing in (mainly to do with servicing the socket's fd's in an appropriate way). Apart from the socket code (which mostly relies in opensslsocket.cpp), the rest of the code for syntax and such is all cross-platform - so if it works on one platform it will (perhaps I should say 'should') work on the rest.

How do I make sure what I create is acceptable?

The main process is to setup your own github account and create a clone of the main livecode repository (you only need to have your own clones of the sub-modules if any work you do touches them - in this case I think it is unlikely that it will) then create a branch off of either master or develop to start working on your contribution (in this case I'd probably suggest 'master' since it makes it the most portable in terms of patching into releases). When you're happy with it, or want someone here to take a look for guidance or help just send a pull-request with that branch on github - this makes it easy for us to look at.

In terms of coding style and such, we generally ask that when modifying existing code that the style is similar to the surrounding code - I'd imagine what you'd be proposing to do will be patches to various parts of the existing source so just use common sense and use comments to explain what the additions do (as our engineers take on the burden of maintenance of third-party patches we have to have at least some idea of how the code works).

If in the process of modifying code you also fix some bugs you notice, do mention this as we can then see if we can tie things up to a bug number - looking at the Quality Center at the moment there do seem to be a fair few issues with various areas of sockets.

I plan to find and join the right mailing list and do the right things to get started in helping.

This forum is essentially the 'mailing-list' you are looking for (we did originally intend to have a mailing list, but the general consensus at the time was that forums were better).

If I add multicast to Livecode without adding socket options to LiveCode, it might be like this:

I fear that your knowledge of sockets far out-weighs mine in this area so I might defer you to @runrevfraser to help out here since he knows more about this kind of thing than I

[LCfraser]

As Mark said in your other topic, the idea of sockets having properties is nicer but more problematic to implement at this stage. A more straightforward approach might be something like the following:

Code: Select all

local tIpMReq
put "224.0.0.1" into tIpMReq["multiaddr"]
put "10.1.2.3" into tIpMReq["interface"]
setSocketOption(tSocket, "IP_ADD_MEMBERSHIP", tIpMReq)

One downside to this approach is that each desired socket option has to be added to the engine - most of them are fairly simple integer values though and shouldn't be too problematic. Structures would be done using arrays with appropriate keys. I think there are also some options that take strings. I bet there are quite a few options that it would be useful to expose this way. It isn't the most LiveCode-y interface but that can always be added later.

[mwieder]

I for one have stayed away from doing any engine work in the sockets area because a "complete" cross-platform rework of the socket layer has been on the roadmap since the kickstarter project. I've put enough headbanging time in on projects that ended up being irrelevant. If it's really true that this is not the case then I'd be happy to look at the socket interface again.

Why, for instance, is there no "accept" command for the server?

At any rate, Dar, I concur that *this* is the place where these discussions go on. Sometimes way too long <g> but this forum is the source of all things engine-related, and the place to look up any previous engine work.

Well, this and the git tree, of course.

[FourthWorld]

Why, for instance, is there no "accept" command for the server?

How would accept work in a CGI context? Does Server offer some daemon mode I missed?

[malte]

If there was a "daemon-mode" and accept worked... Wow, the opportunities...

[FourthWorld]

If there was a "daemon-mode" and accept worked... Wow, the opportunities...

You an write a Linux standalone that uses accept - go for it, and let is know what you cook up.

[DarScott]

@runrevmark, thanks so much for the encouragement and information. That helps a lot.

I hope to jump in intently in a week or so. At this point, I don't even know how to create a correct building environment suitable for this (Xcode version, studio version and so on). I'll have to search the forums.

I realize how things "should" work on multiple operating systems in sockets, but I know Windows has different behavior with some socket options and OS X seems to have some quirks (bugs) in others. So, for an implementation that meets some goals, some differences might be needed.

[DarScott]

@runrevfraser, Hi!

I like your IP_ADD_MEMBERSHIP example.

I wonder if there are any advantages to this one-line variation:

Code: Select all

setSocketOption tSocket, "IP_ADD_MEMBERSHIP", "124.0.0.1,10.1.2.3"

The first concern I have (and I don't know how concerned I should be) is that some options will need to be set between socket() and either connect() or bind(). Those will need to be set in the middle of 'open' or 'accept'. Those values would have to be setup beforehand. Models we already have for doing this kind of thing are allowDatagramBroadcasts, serialControlString and object templates. If the last one is our inspiration (and this is really needed) then maybe it can be setup like this:

Code: Select all

setSocketOption socketTemplate, "SO_REUSEPORT", true

The other is that the socket might be on the same port as a TCP port or a port on a different interface. This will probably need the socket ID suffix as is allowed for 'open' so we can tell them apart. like this:

Code: Select all

setSocketOption "3105|myMulticast", "IP_ADD_MEMBERSHIP", tIpMReq

We can think of this as levels of abstractions. In some sense, the most abstract is TCP/IP. The next level is an abstract socket view, a LiveCode view. The lower level is the OS implementation of sockets. Even though socket option names are the same among operating systems, some have slightly different meanings, some have other defaults, and some have quirks that require goofy procedures. Thought the LiveCode socket is the best view, implementers will not know all the gotchas and it will take longer to implement than say "set IP_ADD_MEMBERSHIP on this OS" and have the meaning be OS dependent. This can be handy if there is a bug in the higher level implementation of something. It allows doing things exactly the way something works in some other language.

There might be some higher level properties to be set as well as the lower level socket options. The lower level socket options can simply be properties, too, just with socket option names. Because of that, the example might be something like this:

Code: Select all

setSocketProperty tSocket, "IP_ADD_MEMBERSHIP", tIpMReq

Note the Property in the name.

[DarScott]

I'd like to do multicast, either directly as I described or in making the setting and getting of socket options visible, as described. If the latter, I'm interested in a property syntax, as described in the topic on properties for opened things, if the merge schedule is appropriate.

For the property syntax, @runrevali mentioned his repo in the other topic on properties of opened things. In the topic on tools, @runrevmark mentioned branch refactor-syntax-unicode in the context of "intending to play around with the pseudo-object stuff". How far off is refactor-syntax-unicode from being merged into something that will become a release candidate?

I'm hoping to get something into a release candidate soon. Say a month? Or am I grossly ignorant of how things work?

Above, @runrevmark suggested working off the master branch. That sounds like something that would get released sooner. But, if I can do the property syntax, that would be cool.

If the above doesn't expose how much of a git noob I am, this will. @runrevmark mentioned creating my own account and creating a clone of the "main livecode repository". What is the URL for that? Do I pull that to my local git and then push it to my own repository? (Or is this a question I should figure out in git studying?) Mark then said to send a pull request on that branch. I'm not sure what that means--is it pull the main livecode repository to my local repository, merge master with my branch, push that to my repository and then send an email to RunRev with the repository and branch? Or do I push to the "main livecode repository"? Or is all this detailed in the README?

[DarScott]

I've been reading about git. Maybe I should be learning about github.

[smihaila]

Wait....no IGMP / multicast options?
I was currently evaluating LiveCode / Transcript as one viable cross-platform solution, targeting both non-mobile/desktop and mobile platforms.
Hearing this is a major disappointment. No finer / granular control via the API / Transcript.

I guess I have to consider this platform just a nice toy, and not ready for real, production-grade code and real-live deployments to customers.
Too bad, another 4GL fad, like Rebol...Even if it seems to have at least of decade of history backing it up, via HyperCard.