Error Handling

Mcrouter returns different errors on different scenarios. This article's goal is to help you understand the ways mcrouter handle errors.

Errors types

Mcrouter has basically two main categories of errors: SERVER_ERROR and CLIENT_ERROR. As the name implies, server errors refers to when there is something wrong with the server, and a client error is reported when there is a problem with the request.

Error forwarding

Mcrouter sometimes forward errors received from servers and sometimes handle the errors and returns something different. In this section we will see what kind of errors are forwarded from the underlying server (usually memcached) to the client.

ASCII

If the underlying server (e.g. memcached) is talking to mcrouter using the ASCII protocol, these are the errors that will be forwarded:

• For get: By default no errors are returned for gets, mcrouter will always return a miss. However, if the option --disable-miss-on-get-errors is provided, all errors (i.e. both SERVER_ERROR and CLIENT_ERROR) will be forwarded.
• For set: All errors (i.e. both SERVER_ERROR and CLIENT_ERROR).
• For delete: By default, only errors of type: CLIENT_ERROR are forwarded, meaning that all other errors result in NOT_FOUND. This behavior can be modified if the option --asynclog-disable is specified, in which case all errors (i.e. both SERVER_ERROR and CLIENT_ERROR) will be forwarded to the client.

List of errors

Bellow is a list of all errors and in which case they are used.

ooo

Error returned by facebook's memcached fork. This error is not generated by mcrouter. It is just forwarded if returned by memcached.
ASCII representation: SERVER_ERROR out of order\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above).

timeout

Error that is generated by mcrouter every time the underlying server fails to respond to a request in a given amount of time.
ASCII representation: SERVER_ERROR timeout\r\n.
Use cases: Underlying server timeout.

connect_timeout

Hard error that indicates a timeout while trying to establish connection with the underlying server.
ASCII representation: SERVER_ERROR connection timeout\r\n.
Use cases: Connection timeout.

connect_error

Hard error that means that a generic error occurred while trying to connect to underlying server.
ASCII representation: SERVER_ERROR connection error\r\n.
Use cases: Connection error.

busy

Error that indicates that the underlying server refused to respond due to load shedding. This error is not generated by mcrouter.
ASCII representation: SERVER_ERROR 307 busy\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above).

try_again

Error that indicates that the underlying server refused to respond, but it can still receive requests. This error is not generated by mcrouter.
ASCII representation: SERVER_ERROR 302 try again\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above).

tko

Error generated by mcrouter indicating that the underlying server is marked as TKO, so the request was not sent to that server.
ASCII representation: SERVER_ERROR unavailable\r\n.
Use cases: Should not send request to server.

bad_command

Error generated by mcrouter indicating that the user sent a command that (despite existing) is not allowed in current context.
ASCII representation: CLIENT_ERROR bad command\r\n.
Use cases: When "shutdown" command goes too deep into the stack - this should never happen though.

bad_key

Error generated by mcrouter telling that the key provided is invalid.
ASCII representation: CLIENT_ERROR bad key\r\n.
Use cases: Key provided is too large (just for ASCII protocol). Key too small (i.e. empty key) does not generate a bad_key error, but a malformed_request error.

bad_flags

Error that indicates that the flags provided are invalid. This error is not generated by mcrouter.
ASCII representation: CLIENT_ERROR bad flags\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above).

bad_exptime

Error that indicates that the exptime provided is invalid. This error is not generated by mcrouter.
ASCII representation: CLIENT_ERROR bad exptime\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above)

bad_lease_id

Error that indicates that the lease_id provided is invalid. This error is not generated by mcrouter.
ASCII representation: CLIENT_ERROR bad lease_id\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above)

bad_cas_id

Error that indicates that the cas_id provided is invalid. This error is not generated by mcrouter.
ASCII representation: CLIENT_ERROR bad cas_id\r\n.
Use cases: Not generated by mcrouter. This error might be returned to the client only if forwarded from underlying server (see rules above)

aborted

Error that indicates that the request was aborted by mcrouter.
ASCII representation: SERVER_ERROR aborted\r\n.
Use cases: When mcrouter is aborting communication to underlying server. Usually, this scenario takes place when either mcrouter is being shut down or mcrouter is being reconfigured online and the underlying server in question was removed from config.

local_error

This error indicates that something went wrong with mcrouter itself.
ASCII representation: SERVER_ERROR local error\r\n.
Use cases: When we go past the maximum number of allowed pending requests, when an unexpected error happens while serializing a request, when an exception happens while routing a request, when we rate limit a request and when we fail to build the request.

remote_error

This error indicates that something went wrong with the remote (underlying) server.
ASCII representation: SERVER_ERROR remote error\r\n. Although this is the default ASCII representation of this error, it commonly is displayed with other error messages (e.g. SERVER_ERROR Command not supported\r\n).
Use cases: When we shutdown communication without "notice", when we fail to parse an umbrella message, when an exception is thrown while parsing an umbrella request, when an error occur while parsing an umbrella reply, when we fail to parse an umbrella reply value, when we get a request with a command not supported, when we get an invalid command.