Peer to Peer Networking Stack

[splash21]

My IP address on this machine right now is 192.168.0.2 - a private ip address. A lot of people will be on machines with the same ip address. You're machine could be using the very same ip address. If you want your machine to talk directly to mine, then you need something else... and it ain't coffee.

[FourthWorld]

Sounds like what routers do. What am I missing?

[splash21]

OK - you write a little app that let's you and me send messages directly to each other's machines. Shouldn't take you more than 5 minutes. Let me know when you're finished and I'll give you my internal IP address and my router's address.

[townsend]

Yes, it seems I need another coffee, because if both peers already have each other's IP address, what more do they need to communicate with each other?

I think you need more than just the IP address, as all machines using the same router have the same IP Address. http://whatismyipaddress.com/ Mabe a MAC address suffix?

Anyone have the code for getting full IP:MAC address of the local machine?

Townsend, are you interested in p2p over the internet, or just local client / server networking? - which is pretty straight forward with LC

Local p2p wouldn't help much. Not if the app is to be used by 100's or 1000's of users. Once a p2p connection is established, then I'd use LiveCode's built in SQLite for data management. Without using MySQL or some other major SQL manager, the LiveCode executable remains as a self functioning app, without any dependencies.

It worked in exactly the same way as Skype - once the p2p connection is established, you have direct 2 way communication between the client machines.

Again-- so we know this is possible.

No. The hole punching is the act of breaking through the routers' barriers. It allows 2 machines - both behind (different) routers and with private ip addresses - to communicate directly with one and other over the net.

As for discovering other peers - you either know them already, or you need a directory.

Excellent definition of UDP Hole Punching!! Now-- as for "discovering peers," This is NOT necessary. What is necessary is being able to accept a connection from an anonymous client. Then depending on the initial packet received, the receiving client app, could decide whether to disconnect OR exchange data.

[townsend]

Okay-- back from dinner. I think we've established that a direct connection is certainly possible, with the correct IP:mac address. Next, to make this stack truly effective, there are two more BIG problems we might discuss.

1-In order for 1000+ clients to be able to exchange information back and forth we will need some kind of directory. This directory should be able to both, accommodate people who have dynamic IP address, AND be able to start fresh after an internet outage. I have a suggestion. There are all these free dropbox services, which allow anyone freely read and write data. Couldn't I put my current IP:mac address into a dropbox file that is set up to be publiclly accessable. Then if my IP:mac address didn't connect you could go out to this dropbox text file and grab my most current address. Why stop there? May in the preferences section there could be options for: iDrive, Mozy, Google Drive and iCloud.

2- The other big problem is dealing with multiple connections at the same time. Since LiveCode does not support multiple threads, I think the only solution is to spawn each connection to a separate executable. Communication between the main stack and the spawned executables can be done through reading and writing files, in which case, there would probably be two files for each spawn, one for sending orders to the spawn, and the another for receiving completions messages from the spawn.

Then looking ahead, once all this was taken care of, I think some sort of basic protocol would needed for all the common and essential operations. Ideally it would be in such a structure that other commands and operations could easily be added to the same structure, by each developer, depending on the needs of their p2p network. .

Again... just brainstorming.

[FourthWorld]

OK - you write a little app that let's you and me send messages directly to each other's machines. Shouldn't take you more than 5 minutes. Let me know when you're finished and I'll give you my internal IP address and my router's address.

It seems my questions ruffled your feathers to the point of provoking sarcasm. Please accept my apologies. My intention wasn't to annoy, but simply to understand. I appreciate the explanations you've provided thus far, and RFC 512 seems helpful so I'll continue my reading there.

[FourthWorld]

Anyone have the code for getting full IP:MAC address of the local machine?

http://lists.runrev.com/pipermail/use-l ... 03289.html

[FourthWorld]

Excellent definition of UDP Hole Punching!! Now-- as for "discovering peers," This is NOT necessary. What is necessary is being able to accept a connection from an anonymous client. Then depending on the initial packet received, the receiving client app, could decide whether to disconnect OR exchange data.

I have little experience with UDP, since I've been able to do what I need with TCP, and given Dr. Raney's comments about UDP's unreliability and unpredictable data size limits:
http://lists.runrev.com/pipermail/metac ... 01537.html

However, most of what I need to do has been client-server, with the little experience I've had in P2P being limited to LANs, mostly for managing appliances and task-sharing systems where we very much don't want them available from the outside.

That said, It seems most routers ship with port forwarding features, and many provide defaults for a wide range of apps from popular games to iTunes and more, suggesting it's not an uncommon method for allowing external connections to reach internal machines.

LC's chat example stacks use port 1987, and making an entry in my router to forward that port to the internal address of the machine running that stack allows me to connect easily from the outside.

It would certainly be more convenient to the end-user if we could avoid having to require them to allow forwarding the relevant port in their router settings, but given the apparent number of apps that ask this of users it doesn't appear too onerous.

Hopefully we'll see the ability to specify an internal port added to the engine soon, and now that the engine's source code is open it could be added by any interested community member at any time.

But in the meantime, it seems the workarounds available can at least allow the connection to be established, so we can move on to the bigger challenges:

It may be worth considering that even Apache, which is not only fully threaded but also presumes it's running on a machine dedicated as a server, ships with a default setting for MaxClients of just 150. The overhead of socket connections, whether handled in spawned threads or spawned processes, may pose an issue on some systems if each client attempts to connect to all other clients.

It's been long time since I looked at the Gnutella protocol (in my youth I was ambitious enough to attempt an LC library for that, long since abandoned because existing Gnutella clients were pretty good), but IIRC each client has relatively few open connections to other clients, with the Gnutella network as a whole running by chaining small clusters of connected devices to other small clusters.

I think you hit the nail on the head here:

I think some sort of basic protocol would needed for all the common and essential operations. Ideally it would be in such a structure that other commands and operations could easily be added to the same structure, by each developer, depending on the needs of their p2p network.

Would an existing protocol do what you want to do? If an existing protocol could be used we'd be able to avoid replicating the lengthy design effort, and would have the additional benefit of being interoperable with other systems.

[townsend]

Thanks Richard-- But looking at that function to get the MAC address, it does so by doing a shell statement. Usually I try to avoid all shell statements because, apps that include these are often flagged by virus protection software as dangerous.

Let me ask you this. When setting up a connection between two machines over the Internet, is the MAC address even necessary. Maybe just an real IP address would be enough. Maybe the router would take care of the MAC stuff. I'm not even sure how to get the IP address, as all my test were with two machines off the same router.

Apologies for my lack of knowledge here. IP & MAC addresses-- and even ports-- I have almost no experience in this area.

[townsend]

It would certainly be more convenient to the end-user if we could avoid having to require them to allow forwarding the relevant port in their router settings, but given the apparent number of apps that ask this of users it doesn't appear too onerous.

Requiring a user to go into their router settings would knock out all casual users. I often do these mental estimates. Maybe 95% of potential users, would not be able to do this.

So-- let me ask you this. Are you saying that a LiveCode app on my machine could not generate a complete address that I could give to you via-- say via email-- that would allow your app to contact my app, listening on a designated port, without messing with the router settings?

If this is the case, then I would ask, would the same limitations be found when establishing a FTP connection between two machines?

[FourthWorld]

Thanks Richard-- But looking at that function to get the MAC address, it does so by doing a shell statement. Usually I try to avoid all shell statements because, apps that include these are often flagged by virus protection software as dangerous.

Let me ask you this. When setting up a connection between two machines over the Internet, is the MAC address even necessary. Maybe just an real IP address would be enough. Maybe the router would take care of the MAC stuff. I'm not even sure how to get the IP address, as all my test were with two machines off the same router.

Apologies for my lack of knowledge here. IP & MAC addresses-- and even ports-- I have almost no experience in this area.

This may well be the blind leading the blind, since I've built a fair number of connected systems but using only a small subset of methods, so my experience is very vertical. But hopefully that won't stop us from experimenting and learning.

So far the only references I've seen to MAC addresses used as network identifiers have been in discussions of IP v6, like this one:
http://www.iab.org/wp-content/IAB-uploa ... review.txt

And as that article points out, since those uniquely identify not merely a temporarily-assigned address but a permanent reference to a specific device, many have raised privacy concerns, suggesting random or hashed values be used instead.

For the sake of getting something up and running, I'd consider a hybrid approach using a tracker/directory server instead. This seems common enough that it has some merit, and many of us in the community have a surplus of underutilized server resources that could be made available to assist.

Of course that would only help with discovery. The directory server would only know the public address of the clients, so we'd still need to rely on port forwarding to be able to reach the client.

But if both a hybrid approach and requiring setting up port forwarding were acceptable, the bigger question remains:

What exactly do we want to do with this network?

The answer to that would inform options for the protocol to drive it.

[splash21]

It seems my questions ruffled your feathers to the point of provoking sarcasm.

It wasn't sarcasm. You obviously didn't understand the problem and a five minute practical exercise would have demonstrated what the problem is and cleared things up. That's what I usually do if things aren't clear or obvious - knock out a quick demo stack.

That said, It seems most routers ship with port forwarding features...

Not everyone has access to port forwarding and many would not even know where to start an attempt to set it up. Really depends on how IT savvy your typical client will be. That is why a solution that just works is desirable.

In order for 1000+ clients to be able to exchange information back and forth we will need some kind of directory. This directory should be able to both, accommodate people who have dynamic IP address, AND be able to start fresh after an internet outage.

I'd definitely look at the database idea - pick up from where you left off if it crashes. Dynamic ip address could be taken care of with a username/password - even an auto login option in the client. That way the server can update according to user name.

This thread's got my interest in p2p going again - it's been shelved for quite a while, but I'd really like to get something working with LC.

[FourthWorld]

It seems my questions ruffled your feathers to the point of provoking sarcasm.

It wasn't sarcasm. You obviously didn't understand the problem and a five minute practical exercise would have demonstrated what the problem is and cleared things up. That's what I usually do if things aren't clear or obvious - knock out a quick demo stack.

Yes, I'd already noted several times that I didn't understand why you were using the method you were. Using the existing chat client example (zero development time) yielded a useful result for me using the common method of port forwarding. I now understand that the UDP hole punching is merely a way to obviate the need for setting up port forwarding, and I appreciate your explanation.

This thread's got my interest in p2p going again - it's been shelved for quite a while, but I'd really like to get something working with LC.

Good to hear. I look forward to seeing what you come up with.

[townsend]

But if both a hybrid approach and requiring setting up port forwarding were acceptable, the bigger question remains:

What exactly do we want to do with this network?

The hybrid approach is fine. And I'm all on board with experimenting. But if each user has to modify their router settings, this would certainly defeat my ambition of creating a widely used peer-to-peer application. As I stated early on, we might work together, to develop a reliable peer-to-peer stack that any LiveCode developer could use as a basis to develop their own unique vision of a peer-to-peer network.

Personally, I'm thinking in terms of a peer-to-peer social network. But IF each user has to set up port forwarding on their router, the potential market would be so small, it would not be worth developing.

Yes, it seems I need another coffee, because if both peers already have each other's IP address, what more do they need to communicate with each other?

That's what I originally thought, but my knowledge in this area is limited. Maybe an experiment is in order. I know I can go out to the web and get my IP address http://whatismyipaddress.com/ but the solution to being able to get my IP address in LiveCode without going out to the web, I have not found. IF a shell statement is required, I would rather open up a web page, that have my app flagged, as dangerous, by virus protection software.

Yes-- coffee sounds good-- me too.

[FourthWorld]

Requiring a user to go into their router settings would knock out all casual users. I often do these mental estimates. Maybe 95% of potential users, would not be able to do this.

Perhaps, but it really depends on the target audience. Like many other aspects of this discussion it comes back to the central question: what do we want this network do?

Until the goal of the system is defined, it would seem difficult to anticipate the specific audience who would participate in it.

But if gamers are any indication, it seems many are quite comfortable with port forwarding - indeed, sometimes too comfortable - this article points out how frequently gamers use port forwarding when they don't even need to:
http://pcgamingtips.blogspot.com/2010/0 ... ragon.html

So-- let me ask you this. Are you saying that a LiveCode app on my machine could not generate a complete address that I could give to you via-- say via email-- that would allow your app to contact my app, listening on a designated port, without messing with the router settings?

Never say never. In the worst case, a member of the community with some networking and C++ skills could add the request in RQCC#5097 to the engine.

We've seen a lot of valuable features added by community members in recent months. Hopefully this can be one more.

If this is the case, then I would ask, would the same limitations be found when establishing a FTP connection between two machines?

Considering how often that's come up on the use-livecode list, perhaps so:
https://www.google.com/search?as_sitese ... FTP+socket

Public servers are identifiable via DNS, and generally have their own firewalls. Ubuntu and Debian, to cite the currently most popular server distros, ship with some pretty tight restrictions with their ufw, allowing only the services you explicitly turn on (aside from HTTP, which is of course open by default).

With most private machines behind the firewalls built into routers, without the sort of hole-punching splash21 describes, it seems a good many apps require port forwarding to route incoming requests.