Keeping Sockets Open, Clients, Servers
Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller
-
- VIP Livecode Opensource Backer
- Posts: 118
- Joined: Thu Sep 20, 2012 5:11 pm
Keeping Sockets Open, Clients, Servers
Hello,
The attached stacks are completed versions of the bits of code found in various lessons and responses to questions in the forums.
They demonstrate using "accept connections", read from socket, write to socket, open socket, socketError, socketClosed, socketTimeout.
While they mostly work, I don't know that they are the proper way to implement this functionality.
They do provide anyone interested in this functionality a pretty good basis for building a client or server, with examples of putting your functionality in a Card script, or Stack script or a separate subStack script.
I added something different, an "echo server" that is an intermediate application that echos client sends to the target server and also echos server responses back to the sending client. The server can also subsequently send back to the last client it received a transmission from.
One thing you will find in this code is that the "server" does not "accept" connections when in echo mode. This is to allow the server to be running on a computer where you can not access the port forwarding of the router. The "Echo Server" DOES require that it be on a machine where you can access the port forwarding in your router. Of course if you are only doing socket communication on a local network then the router is not involved. You can ignore that kind of functionality by running the "client-server" stack in NON-echo mode.
I have NOT been able to discover how to stop the Server ports from shutting down after a period of no use.
I have not found what message that is generated to notify the program that the socket is shutting down (will not respond to an open socket request from a client or not responding to a write socket request on what is expected to be an open socket).
How long that period of no use is before the socket on the server no longer responds, is something else I have not been able to determine.
It has nothing to do with the "Socket timeout" setting that I can tell because as these attached stacks demonstrate, you can send and receive after seeing socketTimeout messages on any of the connected machines.
So how does one keep sockets on the server open when there are no clients around?
How does one keep the socket open to a server on a client when there are long periods of no need for communication?
Thanks,
Andrew
The attached stacks are completed versions of the bits of code found in various lessons and responses to questions in the forums.
They demonstrate using "accept connections", read from socket, write to socket, open socket, socketError, socketClosed, socketTimeout.
While they mostly work, I don't know that they are the proper way to implement this functionality.
They do provide anyone interested in this functionality a pretty good basis for building a client or server, with examples of putting your functionality in a Card script, or Stack script or a separate subStack script.
I added something different, an "echo server" that is an intermediate application that echos client sends to the target server and also echos server responses back to the sending client. The server can also subsequently send back to the last client it received a transmission from.
One thing you will find in this code is that the "server" does not "accept" connections when in echo mode. This is to allow the server to be running on a computer where you can not access the port forwarding of the router. The "Echo Server" DOES require that it be on a machine where you can access the port forwarding in your router. Of course if you are only doing socket communication on a local network then the router is not involved. You can ignore that kind of functionality by running the "client-server" stack in NON-echo mode.
I have NOT been able to discover how to stop the Server ports from shutting down after a period of no use.
I have not found what message that is generated to notify the program that the socket is shutting down (will not respond to an open socket request from a client or not responding to a write socket request on what is expected to be an open socket).
How long that period of no use is before the socket on the server no longer responds, is something else I have not been able to determine.
It has nothing to do with the "Socket timeout" setting that I can tell because as these attached stacks demonstrate, you can send and receive after seeing socketTimeout messages on any of the connected machines.
So how does one keep sockets on the server open when there are no clients around?
How does one keep the socket open to a server on a client when there are long periods of no need for communication?
Thanks,
Andrew
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Keeping Sockets Open, Clients, Servers
Ah, I think you're concerned here with Keep-Alive Interval. The default is 1800 seconds.
https://www.rfc-editor.org/rfc/rfc1122#page-101
You may want to send an ACK packet once in a while to ensure that the socket connection stays open.
https://www.rfc-editor.org/rfc/rfc1122#page-101
You may want to send an ACK packet once in a while to ensure that the socket connection stays open.
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Keeping Sockets Open, Clients, Servers
So... if you really want to change the keep-alive interval, it's an operating system change, so it will depend on what your server is running on.
And it's a serious change, so you need to think about whether you really want to do this, especially if it's public-facing... it's possible to change this setting for a single socket on linux, but I'm not sure about other systems.
And it's a serious change, so you need to think about whether you really want to do this, especially if it's public-facing... it's possible to change this setting for a single socket on linux, but I'm not sure about other systems.
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- VIP Livecode Opensource Backer
- Posts: 118
- Joined: Thu Sep 20, 2012 5:11 pm
Re: Keeping Sockets Open, Clients, Servers
Hello,
Thanks for that information.
So I am at this point modifying the stacks to generate an empty write as well as a follow on "ACK" write every so often.
So based on the idea of a default keep alive interval of 1800 seconds (30 minutes)....
I am setting the timeOutInterval to 10000 (10 seconds) and then planning to do the empty writes every 5 minutes and an ACK write every 50 minutes IF there is no other traffic.
Any traffic will cause the counters to reset their countdown.
Once I get it implemented and tested I will post the new stacks.
Thanks for that information.
So I am at this point modifying the stacks to generate an empty write as well as a follow on "ACK" write every so often.
So based on the idea of a default keep alive interval of 1800 seconds (30 minutes)....
I am setting the timeOutInterval to 10000 (10 seconds) and then planning to do the empty writes every 5 minutes and an ACK write every 50 minutes IF there is no other traffic.
Any traffic will cause the counters to reset their countdown.
Once I get it implemented and tested I will post the new stacks.
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Keeping Sockets Open, Clients, Servers
Amazing... I asked chatCGT
To adjust the TCP keep-alive value on a server, you will need to modify the appropriate operating system configuration. The steps for doing this will vary depending on the specific operating system that you are using.
On Linux systems, you can use the setsockopt function to set the TCP keep-alive interval for a socket. You can also use the /proc filesystem to set the global keep-alive interval for all sockets on the system.
For example, to set the keep-alive interval to 2 hours (7200 seconds) for all sockets on a Linux system, you could use the following command:
On macOS and other BSD-based systems, you can use the sysctl command to set the keep-alive interval. For example, to set the keep-alive interval to 2 hours for all sockets on a macOS system, you could use the following command:
On Windows systems, you can use the registry editor to set the keep-alive interval. You can find the relevant settings under the following key:
There are several values that you can modify to control the TCP keep-alive behavior, including KeepAliveTime, KeepAliveInterval, and TcpMaxDataRetransmissions. You can find more information about these values in the Windows documentation.
Keep in mind that modifying these values may have unintended consequences and could potentially affect the performance of your system. It is always a good idea to thoroughly test any changes that you make before implementing them in a production environment.
and I got the following:how do i adjust a tcp keep-alive server value?
To adjust the TCP keep-alive value on a server, you will need to modify the appropriate operating system configuration. The steps for doing this will vary depending on the specific operating system that you are using.
On Linux systems, you can use the setsockopt function to set the TCP keep-alive interval for a socket. You can also use the /proc filesystem to set the global keep-alive interval for all sockets on the system.
For example, to set the keep-alive interval to 2 hours (7200 seconds) for all sockets on a Linux system, you could use the following command:
Code: Select all
echo 7200 > /proc/sys/net/ipv4/tcp_keepalive_time
Code: Select all
sysctl -w net.inet.tcp.keepidle=7200
Code: Select all
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Keep in mind that modifying these values may have unintended consequences and could potentially affect the performance of your system. It is always a good idea to thoroughly test any changes that you make before implementing them in a production environment.
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
Re: Keeping Sockets Open, Clients, Servers
I, for one, welcome our new AI overlords...
-
- VIP Livecode Opensource Backer
- Posts: 118
- Joined: Thu Sep 20, 2012 5:11 pm
Re: Keeping Sockets Open, Clients, Servers
Well the results of my testing so far are mixed.
(all this on Macintoshs)
IF the client and server are inside your local network and you keep them alive with regular posts, then everything stays alive.
My regular posts, with the applications sitting doing nothing else are every 5 minutes the client does a no data write to the server.
BUT, if you have a client that logs into the server by referencing your external IP address, then the outside world decides to shut down the client connection after a couple hours.
I am beginning to think that the empty writes are not being transmitted, I don't see them being logged by the EchoServer, but the every 50 minute "ACK" writes are showing up in the EchoServer log.
I have to do more testing under circumstances where the client machine is not ending up going to sleep to be certain that is not a factor in all this as well. (the server is set to never sleep - it has no monitor to worry about).
(all this on Macintoshs)
IF the client and server are inside your local network and you keep them alive with regular posts, then everything stays alive.
My regular posts, with the applications sitting doing nothing else are every 5 minutes the client does a no data write to the server.
BUT, if you have a client that logs into the server by referencing your external IP address, then the outside world decides to shut down the client connection after a couple hours.
I am beginning to think that the empty writes are not being transmitted, I don't see them being logged by the EchoServer, but the every 50 minute "ACK" writes are showing up in the EchoServer log.
I have to do more testing under circumstances where the client machine is not ending up going to sleep to be certain that is not a factor in all this as well. (the server is set to never sleep - it has no monitor to worry about).
-
- VIP Livecode Opensource Backer
- Posts: 118
- Joined: Thu Sep 20, 2012 5:11 pm
Re: Keeping Sockets Open, Clients, Servers
OK, been a while but Christmas, New Years, lots of testing.
I have attached updated stacks that demonstrate working code.
IF you are using sockets to communicate within your network, they don't get arbitrarily closed down after a while - at least not with my routers and on Macintoshs.
IF you go out on the public network, my test client would get a socketClosed message after an hour or so. It is probably dependent upon whoever's systems you are connecting through. What I found is that unless there is traffic, it seems in both directions, Spectrum will send out a socketClosed to the client side (the side that did open socket) after about an hour.
BUT, if you have at least some traffic - from the server and the client - I sent out a "kaw" (keep Alive Write) every 5 minutes and that kept the connections opened. Eventually just so I could see who was sending what I made the messages unique, but it seems that 3 characters is enough. Maybe you can get by with doing kaws less often, I did not check.
SECOND - if you are sending a lot of data - such as a database query that returns a lot of data, you REALLY REALLY need to add a length field or have some means of knowing when you have received all the data from the query. I found that sometimes despite the fact that I sent out a single "write", the data would arrive in multiple packets and I needed to have code to check to see if all the data had been received and if not, continue reading.
I also used a somewhat different field delimiter for the socket fields because my database can have just about any character in its fields. One bug I encountered was truncating the incoming data because a database record had my "delimiter" character in it.
So I switched to two "~" characters for my socket fields. Weird, but easy to implement.
That specialized reading code is not in the stacks, but shown below.
There are function calls in the code below that are present in the stacks I have attached.
CLIENT SIDE - GETS A LENGTH AND KEEPS READING UNTIL ALL DATA RECEIVED.
SERVER SIDE - PREPENDS A LENGTH FIELD TO THE DATA TO SEND TO THE CLIENT.
Hopefully this is helpful to others.
(and if anybody has better code, please feel free to fix these stacks and post the updates!!)
Andrew
I have attached updated stacks that demonstrate working code.
IF you are using sockets to communicate within your network, they don't get arbitrarily closed down after a while - at least not with my routers and on Macintoshs.
IF you go out on the public network, my test client would get a socketClosed message after an hour or so. It is probably dependent upon whoever's systems you are connecting through. What I found is that unless there is traffic, it seems in both directions, Spectrum will send out a socketClosed to the client side (the side that did open socket) after about an hour.
BUT, if you have at least some traffic - from the server and the client - I sent out a "kaw" (keep Alive Write) every 5 minutes and that kept the connections opened. Eventually just so I could see who was sending what I made the messages unique, but it seems that 3 characters is enough. Maybe you can get by with doing kaws less often, I did not check.
SECOND - if you are sending a lot of data - such as a database query that returns a lot of data, you REALLY REALLY need to add a length field or have some means of knowing when you have received all the data from the query. I found that sometimes despite the fact that I sent out a single "write", the data would arrive in multiple packets and I needed to have code to check to see if all the data had been received and if not, continue reading.
I also used a somewhat different field delimiter for the socket fields because my database can have just about any character in its fields. One bug I encountered was truncating the incoming data because a database record had my "delimiter" character in it.
So I switched to two "~" characters for my socket fields. Weird, but easy to implement.
That specialized reading code is not in the stacks, but shown below.
There are function calls in the code below that are present in the stacks I have attached.
CLIENT SIDE - GETS A LENGTH AND KEEPS READING UNTIL ALL DATA RECEIVED.
Code: Select all
local sResponse, sPartial, sLength
function sendToDBserver pData
doOpenConnection;
local tSock;
put getSocket() into tSock;
if tSock is not empty then
resetKeepAlive
put urlEncodeRFC(pData) into pData
write pData to socket tSock; //send databases request
put empty into sResponse
put empty into sPartial
read from socket tSock with message "doReading"
wait while sResponse is empty with messages
//yeah, infinite loop - can't do anything more until server responds
return sResponse
end if
return 0;
end sendToDBserver
on doReading pSocket, pData
local tMsg, tLen
put urlDecodeRFC(pData) into tMsg
set itemDelimiter to "~~"
if item 1 of tMsg is "LEN" then
delete item 1 of tMsg
put item 1 of tMsg into sLength
delete item 1 of tMsg
end if
put tMsg after sPartial
put length(tMsg) into tLen
put sLength - tLen into sLength
if sLength <= 0 then
put sPartial into sResponse
else
read from socket pSocket with message "doReading"
end if
end doReading
SERVER SIDE - PREPENDS A LENGTH FIELD TO THE DATA TO SEND TO THE CLIENT.
Code: Select all
command sendStuffToClient pClientIPAddress, pData
//how to send stuff to the client-Server
logger " :sendStuffToClient Sending:<" & pClientIPAddress & "><" & pData & ">"
resetKeepAlive false
put pClientIPAddress into sLastClientAddressPort
local tLen
put length(pData) into tLen
put "LEN~~" & tLen & "~~" & pData into pData
put urlEncodeRFC(pData) into pData
write pData to socket pClientIPAddress with message "writeClientServerResponse"
end sendStuffToClient
on writeClientServerResponse pClientServerAddress
//got response from writing to the client-Server.
//this is just indicating that the write to client-Server finished
end writeClientServerResponse
(and if anybody has better code, please feel free to fix these stacks and post the updates!!)
Andrew
- Attachments
-
- DemoSource.zip
- (25.71 KiB) Downloaded 66 times
-
- VIP Livecode Opensource Backer
- Posts: 9857
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: Keeping Sockets Open, Clients, Servers
Good that you have a solution, but the journey to arrive at it reminds me of journeys of my own over the years, most of which led me to stop creating novel protocols and use HTTP instead.
Is HTTP unsuitable for this application?
Is HTTP unsuitable for this application?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
-
- VIP Livecode Opensource Backer
- Posts: 118
- Joined: Thu Sep 20, 2012 5:11 pm
Re: Keeping Sockets Open, Clients, Servers
Hi,
Ultimately I will end up with HTTP for parts of this system, but as I understand it, the server side of HTTP can't directly interact with local files such as a database for some reason. So there needs to be a separate application that manages the database that the HTTP application interrogates for the information it needs. I am using sockets for that communication.
Since this whole system will ultimately be physically located in a different state, I want to be able to run a test system separately that mirrors how the deployed system works for debugging and enhancement purposes.
Ultimately I will end up with HTTP for parts of this system, but as I understand it, the server side of HTTP can't directly interact with local files such as a database for some reason. So there needs to be a separate application that manages the database that the HTTP application interrogates for the information it needs. I am using sockets for that communication.
Since this whole system will ultimately be physically located in a different state, I want to be able to run a test system separately that mirrors how the deployed system works for debugging and enhancement purposes.
-
- VIP Livecode Opensource Backer
- Posts: 118
- Joined: Thu Sep 20, 2012 5:11 pm
Re: Keeping Sockets Open, Clients, Servers
OH,
One more thing.
I forget if the posted code watches for the "kaw" and "ACK" to be received.
When those do get received, you need to note them and delete them from being processed as legitimate data in your "read the socket" code.
(I did forget in my application derived from the code I posted and had to track that bug down!)
One more thing.
I forget if the posted code watches for the "kaw" and "ACK" to be received.
When those do get received, you need to note them and delete them from being processed as legitimate data in your "read the socket" code.
(I did forget in my application derived from the code I posted and had to track that bug down!)
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Keeping Sockets Open, Clients, Servers
That's safest in any event. You don't want direct access to the database from the client.So there needs to be a separate application that manages the database that the HTTP application interrogates for the information it needs.
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Keeping Sockets Open, Clients, Servers
Ouch. That must have been fun tracking down.(I did forget in my application derived from the code I posted and had to track that bug down!)
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev