Peer to Peer Networking Stack

[townsend]

I am VERY interested in Peer to Peer networking, and would be glad to participate in a Livecode open source project that aimed to build a simple framework for peer to peer communication. Rather than have one server app, and many clients, I'd like to help develop code for a single stack that could act as both client and server, allowing data to reliably be sent or received, even under heavy traffic. It would then be up to each individual developer to build their own send/receive protocol, depending on the function of their peer to peer application.

Just brainstorming here.

[FourthWorld]

See the chat example included with LiveCode.

[townsend]

I've spent a LOT of time with that chat example. It's a great starting point. But it is two separate apps: Server & client. The problem is, all client to client communication has to be is relayed through the server. I've made some good progress in combining them, into one multi-function stack, but still have a long way to go.

Basically I was just putting it out there-- to see if others were also interested in peer to peer applications.

[splash21]

Check out this feature request. If we could specify local ports then p2p would be possible with LiveCode using the same technique that Skype, etc. use. You can check out 'UDP hole punching' on Wikipedia for more info.

http://quality.runrev.com/show_bug.cgi?id=5097

[townsend]

"Typically required fof client-to-client networking appliations"

Thanks for pointing this. So, this means that my goal of creating a peer to peer, client to client, app in LiveCode is currently not feasible??

[FourthWorld]

Depends what you want to do. What are you building?

[townsend]

Actually this isn't even the project I'm working on right now. I saw that post about TOR and when I went to reply, noticed my peer-to-peer comments had nothing to do with TOR, so I went ahead and started new thread.

Okay, so I just dug out that old code where I was trying to combine both client and server into one stack. I can post it if anyone is interested. It's pretty rudimentary, but if I remember correctly, it does work. That is, it works fine when two clients are run on the same machine, or when run on two machines connected to the same router. The point where I got stuck was getting it to connect through the internet, with an IP address.

Basically, all it does is set up a simple chat between two clients. I thought that would be a good foundation for any type of protocol that might be built afterwards. Nothing specific at this point, but the possiblity of 100's or even 1000's of clients all talking to each other and sharing information, without a server, could lead to some real stand out projects.

So, at this point, due to the limited success of my experimentation, and your comment that LiveCode's ability to do peer-to-peer "Depends" I can only conclude that using using LiveCode for a peer to peer application IS POSSIBLE. That's good news.

So, another question, how would a LiveCode peer to peer application fall short as compared to a full on "UDP Hole Punching" peer-to-peer application?

[FourthWorld]

...how would a LiveCode peer to peer application fall short as compared to a full on "UDP Hole Punching" peer-to-peer application?

I rarely use UDP and have no experience with "hole punching" so I'll leave that specific aspect for others to address.

The thing with "peer-to-peer" is that it's a very broad term, reflecting only that the system isn't dependent on a centralized server, but beyond that there are a great many implementations:
http://en.wikipedia.org/wiki/Peer-to-peer

UDP hole punching appears to be one technique that can be useful for some P2P scenarios, but a great many use TCP, and both UDP and TCP seem to be used with a wide range of specific methods.

About the only thing they have in common is that, in absence of a centralized server, a given node can act as both client and server.

This is why the chat example included in the LC package can be a useful start for exploring many practical P2P applications: as provided, the functionality is split between two stacks, but as you've found it's not hard to simply put the scripts into a single stack.

With most socket communications, the process works similarly to make a phone call: one party is sitting by the phone ready to answer ("accept connections on port <portNumber>"), and the other party dials the number to initiate the call ("open socket to <host>:<portNumber>"), and from that point on the nature of the conversation depends on the participants.

In this view, the only significant difference between "client-server" and "peer-to-peer" is that with P2P a given application can both accept calls and dial out to others prepared to accept calls.

Shao Sean has good judgment on such things, and as noted in the bug report cited above no doubt the UDP hole punching option would be useful in some scenarios.

But as Sean also notes, it's not entirely necessary for every type of P2P application you might want to build.

In fact, as the Wikipedia article I linked to notes, many P2P apps use a hybrid approach, with a centralized server acting as a discovery mechanism, or initial transaction broker, or other roles. And with LiveCode server, using HTTP protocols for that side of things would let you implement those using LiveCode for everything in the system as well.

With P2P, client-server, and hybrid approaches, the number of useful networking things you can make with LiveCode is nearly limitless.

Get crackin'! We wanna see what you come up with.

[splash21]

I hadn't realised that feature request was filed so long ago. I managed to get p2p working with livecode way back then, between 2 machines - both with local ip addresses behind routers. The big kludge of a trick to get it working was to have the app listen on a specific port then continually open sockets to that port and send a small message. You can then note (in the receiving handler) the local port that was used to send the data. When you get to the local port number just before the one you want to use for your p2p communication, you quit the loop and - all going well - the next socket you open will have the correct local port number to open one half of the p2p link - the other machine has to do the same. It was pretty slow looping over socket connections to get near the target (up to 2 minutes) on my Windows machine back then. I tried something similar recently on OSX and it didn't work - the local ports assigned seemed to be more random rather than in sequence, so I patiently wait......

So, another question, how would a LiveCode peer to peer application fall short as compared to a full on "UDP Hole Punching" peer-to-peer application?

It worked in exactly the same way as Skype - once the p2p connection is established, you have direct 2 way communication between the client machines.

Mark W. did say to me (back at the 2009 conference) that it MIGHT not be a big job to add the feature, but it's been buried in the QCC for a while now - I reckon it would be a pretty cool addition

[townsend]

Wow-- thanks guys--

Potential Problem

The point where I got stuck was getting it to connect through the internet, with an IP address.

Can be solved

I managed to get p2p working with livecode way back then, between 2 machines - both with local ip addresses behind routers.

Bigger Problem

The big kludge of a trick to get it working was to have the app listen on a specific port then continually open sockets to that port and send a small message.

Points to Real Problem

It was pretty slow looping over socket connections to get near the target (up to 2 minutes) on my Windows machine back then

Which is why we need

UDP hole punching appears to be one technique that can be useful for some P2P scenarios, but a great many use TCP, and both UDP and TCP seem to be used with a wide range of specific methods.

But it's not currently on the the roadmap

Mark W. did say to me (back at the 2009 conference) that it MIGHT not be a big job to add the feature, but it's been buried in the QCC for a while now

So a work-around is required

Once the p2p connection is established, you have direct 2 way communication between the client machines. It worked in exactly the same way as Skype.

So the solution is

Many P2P apps use a hybrid approach, with a centralized server acting as a discovery mechanism, or initial transaction broker, or other roles. And with LiveCode server, using HTTP protocols for that side of things would let you implement those using LiveCode for everything in the system as well.

[FourthWorld]

So the solution is

Many P2P apps use a hybrid approach, with a centralized server acting as a discovery mechanism, or initial transaction broker, or other roles. And with LiveCode server, using HTTP protocols for that side of things would let you implement those using LiveCode for everything in the system as well.

That's one solution, useful for the open Internet where the range of addresses to attempt connections to would be prohibitive. Skype, DynDNS, and other similar services use a central server as a known point to establish connections between peers, which subsequently takes place between the peers themselves.

But for a local subnet, there are only 255 possible addresses. On my slow Mac here I can attempt connections to all 255 of them using TCP in about a quarter second. Even if I had a list of several possible alternate ports to try rather than just one, I can't figure out how it could take two minutes.

@splash21: What exactly was your app doing?

[splash21]

Typically, a central server is used as a meta server - a directory to find other p2p clients. But once a direct link is established between two clients, they can send data directly to each other, with no further need for the central server. Sending directly is much faster than using a server as a go between. You can always fall back to using a server if a p2p link just can't be established.

From the bug report;
open datagram socket to "a.b.c.d:x" from local port y

It took 2 mins for the client to continually open sockets to itself on port x, sending a packet each time, reading the packet to find the value of 'port y' - this is the local port used to send the data - NOT the port you are sending data to. For p2p, you need control over the port y bit. Back then, my windows machine sequentially opened ports up to a maximum number then looped back round to its' mimimum port value and started counting up again.

Imagine my app wants to send data from local port 10320. I can't specify the local port to be used (YET!) and the initial local port assigned by the OS may be higher than that, so windows opened all the ports up to 65535 or whatever it's maximum value was, then started again at whatever the low port value was - and eventually would get to port 10319 (after around 2 mins or so...). At that point, I would stop the loop - the next socket connection would be to the machine I want to connect to. The data would be sent out on local port 10320. Initially, the data will go nowhere as the other machine is behind a router too, BUT when the other machine sends data to port 10320 to me, my router sees it as a reply to the data I sent out on that port, so the data gets through. The same process happens at the other end of the connection...

The p2p link was eventually made, but it was an unreliable way to do it as any other program running on the machine opening ports could potentially grab the local port that you require.

Hopefully this explains some of the madness!

Townsend, are you interested in p2p over the internet, or just local client / server networking? - which is pretty straight forward with LC

[FourthWorld]

Hopefully this explains some of the madness!

It sees like a good explanation, but I'm only two sips into my morning coffee so it seems I'm somewhat dense.

I've been able to pick an arbitrary port unlikely to be in use, have one machine listening on that, and any others who want to connect with it discover it by walking through the subnet range with "connect" statements until they get a meaningful response. Here that process takes less than a quarter second.

For connections outside the subnet I've used a server as a known connection point, as you describe and as most open Internet solutions use as trackers and such. Given the size of the haystack in which we're attempting to find the needle machine we want to connect to, I'm not familiar with anything other than a hybrid approach for discovering other peers. Is that what UDP hole punching does? If so, I'm having a difficult time understanding how.

More coffee....

[splash21]

Is that what UDP hole punching does?

No. The hole punching is the act of breaking through the routers' barriers. It allows 2 machines - both behind (different) routers and with private ip addresses - to communicate directly with one and other over the net.

As for discovering other peers - you either know them already, or you need a directory.

Have another coffee.

[FourthWorld]

Yes, it seems I need another coffee, because if both peers already have each other's IP address, what more do they need to communicate with each other?