-
Notifications
You must be signed in to change notification settings - Fork 1
Network protocols
This page specifies the network protocol used by robonobo. All data is exchanged using protocol buffers, using data types defined in the files src/java/com/robonobo/core/api/proto/coreapi.proto in the api project, src/java/com/robonobo/mina/message/proto/mina.proto in the mina project, and src/java/com/robonobo/wang/proto/wang.proto in the wang-client project. The term 'msg' in the below explanation refers to a protocol buffer message defined in one of these files.
robonobo uses a hub and spoke network topology for finding and querying sources. All leaf nodes connect to a supernode.
Messages are exchanged between nodes using a SEON connection (see EON Overview) called a Control Connection. In addition to a control connection, other SEON connections are established to send stream data. These are called Listen/Broadcast Connections (depending on which end you are).
Stream data is sent in the form of pages. Each page is numbered (starting at zero), and has a byte offset and time offset from the beginning of the stream. In the initial implementation, pages are all 128KB in size, but this is not required, and we anticipate later implementations will use variable-length pages to conform to underlying stream data formats, to allow seeking within a stream and other features.
The below is only an outline of the network messages, not an implementation specification. To examine the message handling in detail, see the classes in the com.robonobo.mina.message.handlers package.
New node contacts its configured sonar server via http. Sonar returns a list of Node msgs, one per available supernode. New node selects best supernode and establishes a connection to it.
New node constructs a Node msg for itself, and sends it via DEON to the local broadcast ip address. When a node receives this DEON packet, it establishes a connection to the new node.
New node creates a SEON connection to the specified SEON endpoint (ip address, udp port, eon port) for the target node, as specified in a previously-received Node msg. When this connection is established, the connecting node sends a Hello msg. The target node replies with another Hello msg.
This takes place if the target node has no endpoints that can be reached directly, but the new node has. New node sends a ReqConn msg, containing a Node msg for itself, to its connected supernode. The supernode forwards this ReqConn msg to the target node. The target node looks at the Node msg in the ReqConn, and establishes a connection to the new node directly.
Every 30000 +/- rand(1,3000) millisecs, if a node has not received any traffic from a connection, it sends a Ping msg. On reception, the target node sends a Pong msg. If no reply is received to the Ping within 30 seconds, the connection is closed.
When a node starts broadcasting a stream, it sends an AdvSource msg to its supernode. When broadcasting several streams, these are batched into a single AdvSource. The supernode keeps track of which nodes are sources for the stream, and which are requesting sources. If any nodes are currently requesting sources for this stream, the supernode sends to them a GotSource msg with this new source.
When a node wishes to listen to a stream, it sends a WantSource msg to its supernode. The supernode replies with one or more GotSource msgs.
When a listening node receives a GotSource msg, and determines that this source might be useful, it sends a ReqSourceStatus msg to its supernode, which forwards it to the source - unless the listener is already connected to the source, in which case the ReqSourceStatus is sent directly. The source replies with a SourceStatus msg, which is forwarded by the supernode back to the listener unless there is a direct connection.
If the listener determines from the SourceStatus that the source is useful, it establishes a connection to the source (unless already connected), opens an account (unless one is already open), and bids for the source's bandwidth (unless it already has an accepted bid).
New node must have a control connection to the target node. New node sends a TopUp msg containing one or more currency tokens to the target. Target pays currency tokens into the bank, and credits the new node with the appropriate balance.
New node must have an open account with the target node. New node sends a Bid msg to the target containing their opening bid, given as currency units/byte (double-precision value). If there was no existing auction, the target node sends an AuctionResult msg to the new node. If there was an existing auction result, the target node now initiates a new auction by sending a BidUpdate msg to all bidders except the new node.
When nodes receive a BidUpdate, they decide whether or not to counter-bid, and send Bid or NoBid msgs as appropriate. Each node has only a specified amount of time to reply, otherwise they are removed from the auction. After all nodes have sent Bid/NoBid msgs, or the timeout is reached, the target node sends another BidUpdate to remaining bidders.
This continues until there is at most one remaining bidder, at which point the auction is over and the target node sends out an AuctionResult msg to all bidders. This auction result defines the ɣ values (see EON Overview): bidders are given ɣ of 1.0 * (theirBid/maxBid) - the highest bidder always has a ɣ of 1.0.
After an auction, the target will not allow another auction to be triggered for an amount of time (default 30s). This is indicated as part of AuctionResult and other msgs. Bids received before this time (the 'bid freeze') are ignored.
Nodes may not lower their bid once an auction has started, except as their first bid of an auction. Counter-bids must increment by a minimum amount, advertised by the target node. A node may leave an auction by bidding zero, but only if they have not already bid in the current auction.
The top bidder in an auction undertakes to pay for receiving data at a certain minimum rate (advertised by the target node, default 5 KB/s). This prevents a DoS attack by bidding very highly (and thus making everyone else's gammas very low) and then not requesting any data. When an auction finishes, the target node informs the top bidder of this charge by sending a MinCharge msg if their data rate has been lower than the minimum.
A receiving node must keep track of how much data is has received, and must send TopUp msgs (containing currency tokens) as necessary to top up their account. A sending node may send out PayUp demands when the receiving node is about to run out of currency, but this is not required. When a receiving node's account is zero, the sender will not send it any data, will ignore any bids from that node, and will remove the node from the auction as soon as the bid freeze expires.
New node must have an accepted bid in a finished auction with the target node. New node sends a StartSource msg to the target, advertising the SEON endpoint to receive data, and including initial requested pages if any. Target node connects a SEON connection to the target endpoint, and starts sending the serialized pages. To request more pages, the new node sends ReqPage msgs. To close the connection, the new node sends a StopSource msg.
Each page contains an auction index, which increments every time there is a new auction result (and is advertised with that result). This allows nodes to keep track of the cost of each page.
A broadcasting node announces its position in the stream using a StreamStatus msg. These are sent as part of a SourceStatus, and also sent to all listeners when the broadcaster receives a new page (rate limited to max 1/s). The StreamStatus contains the last contiguous page that the broadcaster has, plus a bitmap specifying which of the 32 subsequent pages it has (the 'pagemap'). Listeners keep track of the pagemaps of all their broadcasters.
To decide which pages to request, each listener has a 'page window' consisting of the page numbers after the last contiguous page already received. The page window is ten seconds' worth of pages, min 4 and max 32. When requesting pages from a broadcaster, the listener looks at the pagemaps of all its broadcasters, and chooses the pages that are least common. If several pages are equally uncommon, one is chosen at random.
Under this algorithm, two listeners who start receiving simultaneously from a single source will be able to use each other as additional sources, improving the throughput of the whole network.
New node sends a CloseAcct msg to the target node. Target node replies with an AcctClosed msg containing currency tokens for any remaining balance.
Node sends a Bye msg, then closes the SEON control connection. Other node closes the connection.