Hadoop RPC protocol description¶
Snakebite currently implements the following protocol in
snakebite.channel.SocketRpcChannel to communicate with the NameNode.
Connection¶
The Hadoop RPC protocol works as described below. On connection, headers are sent to setup a session. After that, multiple requests can be sent within the session.
| Function | Type | Default |
|---|---|---|
| Header | bytes |
“hrpc” |
| Version | uint8 |
7 |
| Auth method | uint8 |
80 (Auth method SIMPLE) |
| Serialization type | uint8 |
0 (protobuf) |
| IpcConnectionContextProto length | uint32 |
|
| IpcConnectionContextProto | bytes |
Sending messages¶
When sending a message, the following is sent to the sever:
| Function | Type |
|---|---|
| Length of the next two parts | uint32 |
| RpcPayloadHeaderProto length | varint |
| RpcPayloadHeaderProto | protobuf serialized message |
| HadoopRpcRequestProto length | varint |
| HadoopRpcRequestProto | protobuf serialized message |
varint is a Protocol Buffer variable int.
Note
The Java protobuf implementation uses writeToDelimited to prepend
the message with their lenght, but the python implementation doesn’t implement
such a method (yet).
Next to an rpcKind (snakebites default is RPC_PROTOCOL_BUFFER),
an rpcOp (snakebites default is RPC_FINAL_PAYLOAD), the
RpcPayloadHeaderProto message defines a callId that is added
in the RPC response (described below).
The HadoopRpcRequestProto contains a methodName field that defines
what server method is called and a has a property request that contains the
serialized actual request message.
Receiving messages¶
After a message is sent, the response can be read in the following way:
| Function | Type |
|---|---|
| Length of the RpcResponseHeaderProto | varint |
| RpcResponseHeaderProto | bytes |
| Length of the RPC response | uint32 |
| Serialized RPC response | bytes |
The RpcResponseHeaderProto contains the callId of the request
and a status field. The status can be SUCCESS, ERROR or
FAILURE. In case SUCCESS the rest of response is a complete
protobuf response.
In case of ERROR, the response looks like follows:
| Function | Type |
|---|---|
| Length of the RpcResponseHeaderProto | varint |
| RpcResponseHeaderProto | bytes |
| Length of the RPC response | uint32 |
| Length of the Exeption class name | uint32 |
| Exception class name | utf-8 string |
| Length of the stack trace | uint32 |
| Stack trace | utf-8 string |