best way to wrap a retry in C++ client?

[Mario Emmenlauer]

Dear All,

I'm super happy with Thrift, for its speed and exception handling!
I've developed a Java server and C++ client.

However there are occasional failures to connect (and disconnects)
beyond my understanding. Its possible that the server is just busy,
and a retry would help. To implement a generic retry in C++ I've
found that lambda functions are useful. But I'm curious what other
people are using and/or if there are ready-made solutions for this
problem? I guess most thrift users face the same issue?

On a related note, I've found the excellent discussion of "Designing
robust distributed systems" from JensG at stackoverflow:
https://stackoverflow.com/questions/23013942/handling-failures-in-thrift-in-general

Its from 2014. Is there an implementation of such patterns? What do
people commonly use for retry and associated issues?

All the best,

Mario Emmenlauer


--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: [phone]
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/

[Adam Marcionek]

Hi Mario,

We're pretty new to thrift, so take this advice with a grain of salt. Our C++ client's generic retry functionality on ANY thrift RPC call is as the following pseudocode:

catch TTransportException& tx
If the TTransportExceptionType (tx.getType()) is UNKNOWN, NOT_OPEN or TIMEDOUT, we close the transport, reset the socket, transport and client and reopen the transport. Then retry the command.
Else (getType() is BAD_ARGS, CORRUPTED_DATA, END_OF_FILE, INTERNAL_ERROR, INTERRUPTED) just retry the command.
catch (TException& tx)
Simply fail the call (i.e. do nothing with retry or reconnect.)

After a retry failure, we simply fail the call and let higher level code handle it. This is all done from a macro. If you'd like that code, send me an email and I'll reply with it.

I would actually LOVE to see C++ lamba function do this so if you have something, please share!

Adam



From: Mario Emmenlauer [mailto:***@emmenlauer.de]
Sent: Tuesday, October 3, 2017 6:14 AM
To: ***@thrift.apache.org
Subject: best way to wrap a retry in C++ client?


Dear All,

I'm super happy with Thrift, for its speed and exception handling!
I've developed a Java server and C++ client.

However there are occasional failures to connect (and disconnects)
beyond my understanding. Its possible that the server is just busy,
and a retry would help. To implement a generic retry in C++ I've
found that lambda functions are useful. But I'm curious what other
people are using and/or if there are ready-made solutions for this
problem? I guess most thrift users face the same issue?

On a related note, I've found the excellent discussion of "Designing
robust distributed systems" from JensG at stackoverflow:
https://stackoverflow.com/questions/23013942/handling-failures-in-thrift-in-general

Its from 2014. Is there an implementation of such patterns? What do
people commonly use for retry and associated issues?

All the best,

Mario Emmenlauer


--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: [phone]
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/

________________________________

[Mario Emmenlauer]

Hi Adam,

thanks for your reply! I am currently using something quite
similar as you do, but with lambda functions and templates.
Its not complete yet, because I'm missing the closing of socket.
Currently its just a retry on the actual API methods.

Attached is the rough idea: I use a template<typename Func>
RetryAPICall(Func aAPIMethod) that accepts a function *with*
parameters, and executes this function in a try-catch-block.
Due to this construct, the functions can have arbitrarily many
parameters, and the can return values by reference.

The try-catch is wrapped in a incremental backoff loop with a
number of retries (set the variable mMaxRetries).

The actual lambda function is constructed from the RPC method
via the following construct (all parameters passed as reference):
auto fAPIMethod = [&](){ mThriftAPI->UserPassLogin(mSessionId, aUsername, aPassword); };

Then the retry-aware API call can be executed simply with:
RetryAPICall(fAPIMethod);


Does the attached code make sense to you? I've just hacked it
together yesterday, so it might not be perfect yet. Let me know
if you need more context to make sense of it.

All the best,

Mario

Post by Adam Marcionek
Hi Mario,
catch TTransportException& tx
If the TTransportExceptionType (tx.getType()) is UNKNOWN, NOT_OPEN or TIMEDOUT, we close the transport, reset the socket, transport and client and reopen the transport. Then retry the command.
Else (getType() is BAD_ARGS, CORRUPTED_DATA, END_OF_FILE, INTERNAL_ERROR, INTERRUPTED) just retry the command.
catch (TException& tx)
Simply fail the call (i.e. do nothing with retry or reconnect.)
After a retry failure, we simply fail the call and let higher level code handle it. This is all done from a macro. If you'd like that code, send me an email and I'll reply with it.
I would actually LOVE to see C++ lamba function do this so if you have something, please share!
Adam
Sent: Tuesday, October 3, 2017 6:14 AM
Subject: best way to wrap a retry in C++ client?
Dear All,
I'm super happy with Thrift, for its speed and exception handling!
I've developed a Java server and C++ client.
However there are occasional failures to connect (and disconnects)
beyond my understanding. Its possible that the server is just busy,
and a retry would help. To implement a generic retry in C++ I've
found that lambda functions are useful. But I'm curious what other
people are using and/or if there are ready-made solutions for this
problem? I guess most thrift users face the same issue?
On a related note, I've found the excellent discussion of "Designing
https://stackoverflow.com/questions/23013942/handling-failures-in-thrift-in-general
Its from 2014. Is there an implementation of such patterns? What do
people commonly use for retry and associated issues?
All the best,
Mario Emmenlauer
--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: [phone]
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/
________________________________

Viele Gruesse,

Mario Emmenlauer


--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: [phone]
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/