Unable to stop socket server while there are connected clients

Hi All,
I realized the previous link (THRIFT-2696) was related to Delphi
(gosh!). C++ implementation had the same problem but seems to be already
fixed:

https://github.com/apache/thrift/commit/1684c429501e9df9387cb518e660691f032d7926#diff-73a64b6e865fd1b207b30764b9e06033

Is it possible to open a ticket for C# too?

Regards,
Gianni

Post by Gianni Ambrosio
Hi All,
https://issues.apache.org/jira/browse/THRIFT-2696
So I moved to the latest thrift version (i.e. 0.11.0) but the problem
is still there!
public class ThriftServer
{
   public void start(){
      Service.Processor processor = new
Service.Processor(serviceHandler);
      TServerSocket serverTransport = new
TServerSocket(ServiceConstants.ServicePort);
      server = new TSimpleServer(processor, serverTransport);
      workerThread = new Thread(new ThreadStart(run));
      workerThread.Start();
   }
   private void run(){
      server.Serve();
   }
   public void stop(){
      server.Stop();
   }
}
void ThriftClient::connect(const std::string& serverIp) {
boost::shared_ptr<apache::thrift::transport::TTransport> socket(new
apache::thrift::transport::TSocket(serverIp.c_str(),
g_Service_constants.ServicePort));
   transport =
boost::make_shared<apache::thrift::transport::TBufferedTransport>(socket);
   transport->open();
   client =
boost::make_shared<ServiceClient>(boost::make_shared<apache::thrift::protocol::TBinaryProtocol>(transport));
}
The problem is that when I close the C# application the application
does not close. But in that state it closes as soon as I close the C++
client application.
Debugging the C# application, server.Stop() is called but
server.Serve() call does not exit unless the client has been
disconnected.
Since the above thrift ticket seems reporting exactly the same issue
and it should be fixed in thrift 0.9.2, what's wrong in my code?
Best regards,
Gianni

Mario Emmenlauer

2018-02-27 16:31:46 UTC

I have the same problem with Java, reported here:
https://issues.apache.org/jira/browse/THRIFT-4252

Its a problem for us, and would be great to be fixed.

All the best,

Mario

Post by Gianni Ambrosio
Hi All,
https://github.com/apache/thrift/commit/1684c429501e9df9387cb518e660691f032d7926#diff-73a64b6e865fd1b207b30764b9e06033
Is it possible to open a ticket for C# too?
Regards,
Gianni

Post by Gianni Ambrosio
Hi All,
https://issues.apache.org/jira/browse/THRIFT-2696
So I moved to the latest thrift version (i.e. 0.11.0) but the problem is still there!
public class ThriftServer
{
   public void start(){
      Service.Processor processor = new Service.Processor(serviceHandler);
      TServerSocket serverTransport = new TServerSocket(ServiceConstants.ServicePort);
      server = new TSimpleServer(processor, serverTransport);
      workerThread = new Thread(new ThreadStart(run));
      workerThread.Start();
   }
   private void run(){
      server.Serve();
   }
   public void stop(){
      server.Stop();
   }
}
void ThriftClient::connect(const std::string& serverIp) {
boost::shared_ptr<apache::thrift::transport::TTransport> socket(new apache::thrift::transport::TSocket(serverIp.c_str(), g_Service_constants.ServicePort));
   transport = boost::make_shared<apache::thrift::transport::TBufferedTransport>(socket);
   transport->open();
   client = boost::make_shared<ServiceClient>(boost::make_shared<apache::thrift::protocol::TBinaryProtocol>(transport));
}
The problem is that when I close the C# application the application does not close. But in that state it closes as soon as I close the C++ client application.
Debugging the C# application, server.Stop() is called but server.Serve() call does not exit unless the client has been disconnected.
Since the above thrift ticket seems reporting exactly the same issue and it should be fixed in thrift 0.9.2, what's wrong in my code?
Best regards,
Gianni

Viele Gruesse,

Mario Emmenlauer

--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/

Robin Coe

2018-02-28 13:19:01 UTC

Not having looked at the bug report, so take this advice as appropriate,
but I would rethink the implementation. The underlying network protocol
has no expectation that a server closes a socket, it is always the client's
responsibility. Data integrity is in question when the server just
forcibly closed the connection, and why java responds with a socket
exception in the client.

A safer approach is to implement a callback mechanism that allows the
server to broadcast that it is being shutdown and allow the clients to
respond. Once clients have closed the connection, then the server daemon
can terminate.

Post by Gianni Ambrosio
Hi All,
I realized the previous link (THRIFT-2696) was related to Delphi (gosh!).
https://github.com/apache/thrift/commit/1684c429501e9df9387c
b518e660691f032d7926#diff-73a64b6e865fd1b207b30764b9e06033
Is it possible to open a ticket for C# too?
Regards,
Gianni

Post by Gianni Ambrosio
Hi All,
https://issues.apache.org/jira/browse/THRIFT-2696
So I moved to the latest thrift version (i.e. 0.11.0) but the problem is
still there!
public class ThriftServer
{
public void start(){
Service.Processor processor = new Service.Processor(serviceHandl
er);
TServerSocket serverTransport = new TServerSocket(ServiceConstants
.ServicePort);
server = new TSimpleServer(processor, serverTransport);
workerThread = new Thread(new ThreadStart(run));
workerThread.Start();
}
private void run(){
server.Serve();
}
public void stop(){
server.Stop();
}
}
void ThriftClient::connect(const std::string& serverIp) {
boost::shared_ptr<apache::thrift::transport::TTransport> socket(new
apache::thrift::transport::TSocket(serverIp.c_str(),
g_Service_constants.ServicePort));
transport = boost::make_shared<apache::thr
ift::transport::TBufferedTransport>(socket);
transport->open();
:thrift::protocol::TBinaryProtocol>(transport));
}
The problem is that when I close the C# application the application does
not close. But in that state it closes as soon as I close the C++ client
application.
Debugging the C# application, server.Stop() is called but server.Serve()
call does not exit unless the client has been disconnected.
Since the above thrift ticket seems reporting exactly the same issue and
it should be fixed in thrift 0.9.2, what's wrong in my code?
Best regards,
Gianni

James King

2018-02-28 13:49:25 UTC

The server should never be held hostage by a client. As such, I disagree
with your proposal.
When the server needs to shutdown, it shall:

1. Drain outstanding requests to zero. During this time any outstanding
requests are processed to completion, but no new requests or connections
are allowed.
2. Deliver the responses by properly closing the socket using
well-documented means. This is by no means a guaranteed delivery of the
response, given how sockets work.
3. Shutdown the service.

Significant work went into the C++ engine to support proper shutdown in
this manner, back in the 0.9.3 release.
Prior to that, any client could hold the server hostage and prevent a
shutdown (see: prevent an upgrade; etc...) - even a simple telnet
connection to the port.
Clearly the other language server libraries need to catch up which is why
we have items in the backlog for it. (Thank you for reporting it!)

- Jim

Post by Robin Coe
Not having looked at the bug report, so take this advice as appropriate,
but I would rethink the implementation. The underlying network protocol
has no expectation that a server closes a socket, it is always the client's
responsibility. Data integrity is in question when the server just
forcibly closed the connection, and why java responds with a socket
exception in the client.
A safer approach is to implement a callback mechanism that allows the
server to broadcast that it is being shutdown and allow the clients to
respond. Once clients have closed the connection, then the server daemon
can terminate.

Post by Gianni Ambrosio
Hi All,
https://issues.apache.org/jira/browse/THRIFT-2696
So I moved to the latest thrift version (i.e. 0.11.0) but the problem

Post by Gianni Ambrosio
still there!
public class ThriftServer
{
public void start(){
Service.Processor processor = new Service.Processor(serviceHandl
er);
TServerSocket serverTransport = new TServerSocket(ServiceConstants
.ServicePort);
server = new TSimpleServer(processor, serverTransport);
workerThread = new Thread(new ThreadStart(run));
workerThread.Start();
}
private void run(){
server.Serve();
}
public void stop(){
server.Stop();
}
}
void ThriftClient::connect(const std::string& serverIp) {
boost::shared_ptr<apache::thrift::transport::TTransport> socket(new
apache::thrift::transport::TSocket(serverIp.c_str(),
g_Service_constants.ServicePort));
transport = boost::make_shared<apache::thr
ift::transport::TBufferedTransport>(socket);
transport->open();
client = boost::make_shared<ServiceClient>(boost::make_
:thrift::protocol::TBinaryProtocol>(transport));
}
The problem is that when I close the C# application the application does
not close. But in that state it closes as soon as I close the C++ client
application.
Debugging the C# application, server.Stop() is called but server.Serve()
call does not exit unless the client has been disconnected.
Since the above thrift ticket seems reporting exactly the same issue and
it should be fixed in thrift 0.9.2, what's wrong in my code?
Best regards,
Gianni

Robin Coe

2018-02-28 17:49:34 UTC

I agree with the followups to my post, which I did point out was just
advice on implementing a client that can be informed of a pending
shutdown. I see that as prudent, regardless of the issue. And your points
on how best to terminate connections cleanly is part of that prudence.

Certainly nothing stops java from handling an interrupt and terminating a
socket, which is why the socket connection libraries use caught
exceptions. My point was that a designer of a service should take into
account the needs of the client, which obviously it doesn't know. However,
a clean contract that can message the client at least provides a best
effort before the socket is closed.

On Feb 28, 2018 08:49, "James King" <***@gmail.com> wrote:

The server should never be held hostage by a client. As such, I disagree
with your proposal.
When the server needs to shutdown, it shall:

1. Drain outstanding requests to zero. During this time any outstanding
requests are processed to completion, but no new requests or connections
are allowed.
2. Deliver the responses by properly closing the socket using
well-documented means. This is by no means a guaranteed delivery of the
response, given how sockets work.
3. Shutdown the service.

Significant work went into the C++ engine to support proper shutdown in
this manner, back in the 0.9.3 release.
Prior to that, any client could hold the server hostage and prevent a
shutdown (see: prevent an upgrade; etc...) - even a simple telnet
connection to the port.
Clearly the other language server libraries need to catch up which is why
we have items in the backlog for it. (Thank you for reporting it!)

- Jim

Post by Gianni Ambrosio
Hi All,
https://issues.apache.org/jira/browse/THRIFT-2696
So I moved to the latest thrift version (i.e. 0.11.0) but the problem

Post by Gianni Ambrosio
still there!
public class ThriftServer
{
public void start(){
Service.Processor processor = new Service.Processor(serviceHandl
er);
TServerSocket serverTransport = new

TServerSocket(ServiceConstants

Post by Gianni Ambrosio
.ServicePort);
server = new TSimpleServer(processor, serverTransport);
workerThread = new Thread(new ThreadStart(run));
workerThread.Start();
}
private void run(){
server.Serve();
}
public void stop(){
server.Stop();
}
}
void ThriftClient::connect(const std::string& serverIp) {
boost::shared_ptr<apache::thrift::transport::TTransport> socket(new
apache::thrift::transport::TSocket(serverIp.c_str(),
g_Service_constants.ServicePort));
transport = boost::make_shared<apache::thr
ift::transport::TBufferedTransport>(socket);
transport->open();
client = boost::make_shared<ServiceClient>(boost::make_
:thrift::protocol::TBinaryProtocol>(transport));
}
The problem is that when I close the C# application the application does
not close. But in that state it closes as soon as I close the C++ client
application.
Debugging the C# application, server.Stop() is called but

server.Serve()

Post by Gianni Ambrosio
call does not exit unless the client has been disconnected.
Since the above thrift ticket seems reporting exactly the same issue and
it should be fixed in thrift 0.9.2, what's wrong in my code?
Best regards,
Gianni

Mario Emmenlauer

2018-02-28 14:18:42 UTC

Dear Robin,

I fully agree to your argument as long as we are considering well-
behaving clients. However a problem can arise if the client is not so
well-behaving. Basically, if the client fails to disconnect for whatever
reasons, the server can not cleanly shut down and must be killed. In
many applications (or at least in mine) the server side stores central,
valuable information that under no circumstances should be harmed. A
forced server shut down is the last thing I'd like to happen!

I think your proposed protocol is very good! But on top of that, if
clients fail to obey the disconnect-request, the server must have a fall
back to ensure a clean exit. Currently this is not implemented in
some of the thrift servers :-(

PS: I'm not necessarily saying that it must be malicious intentions of
the client to fail to disconnect. If a significant number of clients
are involved, there is a certain chance that some of them may suffer
from unreliable internet connection, may be busy, may be stale, etc...

All the best,

Mario

Post by Gianni Ambrosio
Hi All,
https://issues.apache.org/jira/browse/THRIFT-2696
So I moved to the latest thrift version (i.e. 0.11.0) but the problem is
still there!
public class ThriftServer
{
public void start(){
Service.Processor processor = new Service.Processor(serviceHandl
er);
TServerSocket serverTransport = new TServerSocket(ServiceConstants
.ServicePort);
server = new TSimpleServer(processor, serverTransport);
workerThread = new Thread(new ThreadStart(run));
workerThread.Start();
}
private void run(){
server.Serve();
}
public void stop(){
server.Stop();
}
}
void ThriftClient::connect(const std::string& serverIp) {
boost::shared_ptr<apache::thrift::transport::TTransport> socket(new
apache::thrift::transport::TSocket(serverIp.c_str(),
g_Service_constants.ServicePort));
transport = boost::make_shared<apache::thr
ift::transport::TBufferedTransport>(socket);
transport->open();
:thrift::protocol::TBinaryProtocol>(transport));
}
The problem is that when I close the C# application the application does
not close. But in that state it closes as soon as I close the C++ client
application.
Debugging the C# application, server.Stop() is called but server.Serve()
call does not exit unless the client has been disconnected.
Since the above thrift ticket seems reporting exactly the same issue and
it should be fixed in thrift 0.9.2, what's wrong in my code?
Best regards,
Gianni

Adam Marcionek

2018-02-28 15:24:08 UTC

Why would the implementation differ from the NFS V3 standard, which is an RPC over either TCP or UDP? UDP is completely stateless. Using TCP, clients connect to the port and then start sending messages. The connection can be torn away at any time by either end. There is no guaranteed delivery or guaranteed response. If the client doesn't receive a response, the server may, or may not, have received and processed it. Naturally this is a problem for non-idempotent messages but there are client-side ways to deal with that, like server-side request cache, transactions or metadata exclusivity (e.g. the EXCLUSIVE mode on a CREATE: https://tools.ietf.org/html/rfc1813#page-54)

I would argue a server can stop at any time for any reason. One can even add to the RCP session creation/tear down which could be used for graceful shutdown. But that happens at the RPC implementation layer, not the protocol layer.

From: Mario Emmenlauer [mailto:***@emmenlauer.de]
Sent: Wednesday, February 28, 2018 9:19 AM
To: ***@thrift.apache.org
Subject: Re: Unable to stop socket server while there are connected clients

Dear Robin,