diff --git a/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.md b/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.md index 982dc4d1b..3b46ccc04 100644 --- a/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.md +++ b/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.md @@ -10,165 +10,168 @@ Created: 2018-07-25 Glossary of Terms -- Chain: a list of blocks starting with genesis block and consisted of successive blocks. -- Best Chain: a chain with the most accumulated PoW, and starting with a common genesis block which nodes agree with the consensus. -- Best Header Chain: a chain with the most PoW and consisted only of blocks in the status of Connected, Downloaded or Accepted. Please refer to block status part for more details. -- Tip: the latest block of a chain and Tip can be used to determine a specific chain. -- Best Chain Tip: the tip of Best Chain. +- Chain: a list of blocks starting with genesis block and consisted of successive blocks +- Best Chain: a chain with the most accumulated PoW, and starting with a common genesis block which nodes agree with the consensus +- Best Header Chain: a chain with the most PoW and consisted only of blocks in the status of `Connected`, `Downloaded` or `Accepted`. Please refer to block status part for more details +- Tip: the latest block of a chain. Tip can be used to determine a specific chain +- Best Chain Tip: the tip of Best Chain ## Abstract -Block synchronization **must** be performed in stages with [Bitcoin Headers First](https://bitcoin.org/en/glossary/headers-first-sync) style. Blocks are downloaded in parts in each stage and are validated using the obtained parts. +Block synchronization **must** be performed in stages with [Bitcoin Headers First](https://btcinformation.org/en/glossary/headers-first-sync) style. Blocks are downloaded in parts in each stage and are validated using the obtained parts. -1. Connecting Header: Get block header, and validate format and PoW. -2. Downloading Block: Get and validate the complete block. Transactions in ancestor blocks are not required. -3. Accepting Block: Validate the block in the context of the chain. +1. Connect Header: Get block header, validate format and PoW. +2. Download Block: Get and validate the complete block. Transactions in ancestor blocks are not required. +3. Accept Block: Validate the block in the context of the chain. +4. New Block Announcement: Push the updated Best Chain Tip to other nodes. -The purpose of stage execution is trying to preclude most of the attacks with the least cost. For example, in the first step, header connecting only accounts for 5% workload while there would be 95% possibility to say the block is valid. +The purpose of stage execution is trying to preclude most of the attacks with the least cost. For example, in the first stage, header connecting only accounts for 5% workload, but has 95% confidence to determine if the block is valid. -According to the execution stages, there are 5 statuses of blocks: +According to the execution stages, there are 6 statuses of blocks: -1. Unknown: the status of a block is unknown before header connecting. -2. Invalid: A block and all of its descendant blocks are marked as 'Invalid' if any above steps failed. -3. Connected: A block succeeds in stage Connecting Header, and all its ancestor blocks are in a status of Connected, Downloaded or Accepted. -4. Downloaded: A block succeeds in stage Downloading Block, and all its ancestor blocks are in a status of Downloaded or Accepted. -5. Accepted: A block succeeds in stage Accepting Block, and all its ancestor blocks are in the status of Accepted. +- `Unknown`: the status of a block is unknown before header connecting +- `Invalid`: A block and all of its descendant blocks are marked as `Invalid` if any of above steps failed +- `Connected`: A block succeeds in stage Connect Header, and all its ancestor blocks are in a status of `Connected`, `Downloaded` or `Accepted` +- `Downloaded`: A block succeeds in stage Download Block, and all its ancestor blocks are in a status of `Downloaded` or `Accepted` +- `Accepted`: A block succeeds in stage Accept Block, and all its ancestor blocks are in the status of `Accepted` +- `Sent`: A block is sent to peer nodes. -Block status is propagated from the previous block to the later ones. Using the list index number above, the status number of a block is always less than or equal to its parent block. Here are conditions, if a block is invalid, all of its descendant blocks must be invalid. The cost of every step for synchronization is higher than the previous one and every step may fail. In this scenario, work will be wasted if a child block enters a later status before its parent block, and parent block is approved to be Invalid later. +The representation of blocks in different statuses shown in the following figure will be used throughout the article. -Initially, Genesis block is in status Accepted and the rest is in status Unknown. +

+ +

-Below figures are used to indicate blocks in different status later on. +Block status is propagated from the previous block to the later ones. Using the serial number above, the status number of a block is always less than or equal to its parent block. If a block is invalid, all of its descendant blocks must be invalid. The cost of every step for synchronization is higher than the previous one and every step may fail. In this scenario, work will be wasted if a child block enters a later status before its parent block, and parent block is approved to be invalid later. -![](images/block-status.jpg "Block Status") +Initially, Genesis block is in status `Accepted` and the rest is in status `Unknown`. The Genesis block of the nodes who participate in synchronization **must be** the same, and all blocks can be constructed as a tree with the Genesis block being the root. Blocks will be removed if they cannot connect to the root eventually. -Genesis block of the nodes synchronizing **must be** the same, and all blocks can be constructed as a tree with the genesis block being the root. Blocks will be removed if they cannot connect to the root eventually. - -Every participating node forms its local status tree where the chain consisting of Accepted blocks with the most PoW is considered as Best Chain. The chain that consists of blocks in the status of connected, downloaded or accepted with the most PoW is Best Header Chain. +Every participating node forms its local status tree where the chain consisting of `Accepted` blocks with the most PoW is considered as the Best Chain. The chain that consists of blocks in the status of `Connected`, `Downloaded` or `Accepted` with the most PoW is the Best Header Chain. The graph below is an example of Status Tree formed by Alice and blocks signed with name Alice is this node's current Best Chain Tip. -![](images/status-tree.jpg "Status Tree by Alice") +

+ +

-## Connecting Header - -Headers first synchronization helps to validate PoW with the least cost. Since it costs the same work to construct PoW whether the included transactions are valid or not, attackers may use other more efficient ways. It means it's highly possible to regard the whole block as valid when the PoW is valid. This is why headers first synchronization would avoid resource-wasting on invalid blocks. +## Connect Header -Because of the low cost, Headers synchronization can be processed in parallel with all peers and construct a highly reliable global graph. In this way, block downloading can be scheduled in the most efficient way to avoid wasting resource on lower PoW branch. +Headers first synchronization helps to validate PoW with the least cost. Since it costs the same work to construct PoW whether the included transactions are valid or not, attackers may use other more efficient ways, for example, fake headers. As a result, it's highly possible that the whole block is valid when the PoW is valid. This is why Headers first synchronization would avoid resource-wasting on invalid blocks. -The goal of connecting header is demonstrated using the following example. When Alice connects to Bob, Alice asks Bob to send all block headers in Bob's Best Chain but not in Alice's **Best Header Chain** and then validate them to decide the blocks status are either Connected or Invalid. +Because of the low cost, Headers synchronization can be processed in parallel with all peers. Each nodes can construct a highly reliable global graph with all forks of the chain. In this way, block downloading can be scheduled in the most efficient way to avoid wasting resource on lower PoW branch. -When Alice connects header, keeping Best Header Chain Tip updated could help to decrease numbers of receiving headers already existed. +The goal of Connect Header is demonstrated with the following example. When Alice connects to Bob, Alice asks Bob to send all block headers in Bob's **Best Chain** but not in Alice's **Best Header Chain** and then validate them to decide the blocks status are either `Connected` or `Invalid`. When Alice connects Headers, keeping Best Header Chain Tip updated could help decrease the number of received headers which already existed. -![](images/seq-connect-headers.jpg) +The process of Connect Header is illustrated below. After the first round of Connect Header, nodes keep up-to-date using New Block Announcement, which will be introduced in the later section. -The graph above instructs the process of connecting headers. After a round of connecting headers, nodes are supposed to keep up-to-date using new block notification. +

+ +

-Take Alice and Bob above as an example, firstly Alice samples blocks from her Best Header Chain and sent the hashes to Bob. The basic principle of sampling is that later blocks are more possible to be selected than early blocks. For example, choose latest 10 blocks from the chain, then sample other blocks backward with 2's exponential increased intervals, a.k.a, 2, 4, 8, and etc. The list of hashes of the sampled blocks is called a Locator. In the following figure, the undimmed blocks are sampled. The genesis block should be always in the Locator. +Firstly Alice samples blocks from her Best Header Chain and sends the hashes to Bob. The basic principle of sampling is that later blocks are more likely to be selected than early blocks. For example, Alice can choose latest 10 blocks from the chain, then sample other blocks backward with 2's exponential increased intervals, a.k.a, `2, 4, 8, …` The list of hashes of the sampled blocks is called a Locator. The Genesis block should be always in the Locator. -![](images/locator.jpg) +

+ +

-Bob can get the latest common block between these two chains according to Locator and his own Best Chain. Because the genesis block is identical, there must be such kind of block. Bob will send all block headers from the common block to his Best Chain Tip to Alice. +Bob can get the latest common block between these two chains according to the Locator and his own Best Chain. Because the Genesis block is identical, there must be such kind of a block. Bob will send all block headers from the common block to his Best Chain Tip to Alice. -![](images/connect-header-conditions.jpg) +

+ +

-In the figure above, blocks with undimmed color should be sent from Bob to Alice, and golden bordered one is the latest common block. There are three possible cases in the process: +In the figure above, blocks with dark green color should be sent from Bob to Alice. There are three possible cases in the process: -1. If Bob's Best Chain Tip is in Alice's Best Header Chain, the latest common block will be Bob's Best Chain Tip and there are no block headers for Bob to send. -2. If Alice's Best Header Chain Tip is in Bob's Best Chain but is not the Tip, the latest common block will be Alice's Best Header Chain Tip. -3. If Alice's Best Header Chain and Bob's Best Chain fork, the latest common block will be the one before the fork occurs. +- If Bob's Best Chain Tip is in Alice's Best Header Chain, the latest common block will be Bob's Best Chain Tip and there are no block headers for Bob to send. +- If Alice's Best Header Chain Tip is in Bob's Best Chain but is not the Tip, the latest common block will be Alice's Best Header Chain Tip. +- If Alice's Best Header Chain and Bob's Best Chain fork, the latest common block will be the one before the fork occurs. Bod will send `DE'F'` in such case. -If there are too many blocks to send, pagination is required. Bob sends the first page, Alice will ask Bob for the next page if she finds out that there are more block headers. A simple pagination solution is to limit the maximum number of block headers returned each time, 2000 for example. If the number of block headers returned is equal to 2000, it means there may be more block headers could be returned. If the last block of a certain page is the ancestor of Best Chain Tip or Best Header Chain Tip, it can be optimized to get next page starting with the corresponding tip. +If there are too many blocks to send, pagination is required. Bob sends the first page, and Alice will ask Bob for the next page if she finds out that there are more block headers. A simple pagination solution is to limit the maximum number of block headers returned each time. If the number of block headers returned reaches the maximum, it means there may be more block headers to be returned. If the last block of a certain page is the ancestor of Best Chain Tip or Best Header Chain Tip, it can be optimized to get next page starting with the corresponding Tip. Alice could observe Bob's present Best Chain Tip, which is the last block received during each round of synchronization. If Alice's Best Header Chain Tip is exactly Bob's Best Chain Tip, Alice couldn't observe Bob's present Best Chain because Bob has no block headers to send. Therefore, it should start building from the parent block of Best Header Chain Tip when sending the first request in each round. -In the following cases, a new round of connection block header synchronization must be performed. +A new round of Header synchronization must be performed if a node receives a new block notification from the others, but the parent block status of the new block is `Unknown`. -- Received a new block notification from the others, but the parent block status of the new block is Unknown. +The following exceptions may occur during Connect Header: -The following exceptions may occur when connecting a block header: +- Alice observes that Bob's Best Chain Tip has not been updated for a long time, or its timestamp is old. In this case, Bob does not provide valuable data. When the number of connections reaches a limit, Bob could be disconnected first. +- Alice observes that the status of Bob's Best Chain Tip is `Invalid`. This can be found in any page without waiting for the end of a round of Connect Header. Since Bob is on an invalid branch, Alice can stop synchronizing with Bob and add Bob to the blacklist. +- There are two possibilities if the Headers Alice received are all on her own Best Header Chain. One is that Bob sends them deliberately. The other is that Alice's Best Chain changes when she connects headers. In this case, those block headers can only be ignored because they are difficult to distinguish. However, the proportion of received blocks already in Best Header Chain would be recorded. If the proportion is above a certain threshold value, Bob could be disconnected or added to the blacklist. -- Alice observed that Bob's Best Chain Tip has not been updated for a long time, or its timestamp is old. In this case, Bob does not provide valuable data. When the number of connections reaches a limit, Bob could be disconnected first. -- Alice observed that the status of Bob's Best Chain Tip is Invalid. This can be found in any page without waiting for the end of a round of Connect Head. There, Bob is on an invalid branch, Alice can stop synchronizing with Bob and add Bob to the blacklist. -- There are two possibilities if the block headers Alice received are all on her own Best Header Chain. One is that Bob sends them deliberately. The other is that Best Chain changes when Alice wants to Connect Head. In this case, those block headers can only be ignored because they are difficult to distinguish. However, the proportion of received blocks already in Best Header Chain would be recorded. If the proportion is above a certain threshold value, Bob may be added to the blacklist. - -Upon receiving the block header message, the format should be verified first. +Upon receiving the block headers message, the format should be verified first. - The blocks in the message are continuous. -- The status of all blocks and the parent block of the first block are not Invalid in the local Status Tree. -- The status of the parent block of the first block is not Unknown in the local Status Tree, which means Orphan Block will not be processed in synchronizing. - -In this stage, verification includes checking if block header satisfies the consensus rules and if Pow is valid or not. Since Orphan Blocks are not processed, difficulty adjustment can be verified as well. +- The status of all blocks and the parent block of the first block are not `Invalid` in the local Status Tree. +- The status of the parent block of the first block is not `Unknown` in the local Status Tree, which means Orphan Block will not be processed during synchronization. +- The consensus rules are satisfied and the PoW is valid. -![](images/connect-header-status.jpg) +

+ +

-The figure above is the Status Tree of Alice after synchronized with Bob, Charlie, Davis, Elsa. The observed Best Chain Tip of each peer is also annotated in the figure. +The figure above is the Status Tree of Alice after synchronized with Bob, Charlie, Davis, Elsa. The observed Best Chain Tip of each peer is also annotated in the figure. The `Unknown` blocks are not considered on the Status Tree. New blocks in the status of `Connected` or `Invalid` will be extended to the leaves of the Status Tree during Connect Header. As a result, Connect Header stage explores and extends the status tree. -If the Unknown status block is considered not on the Status Tree, new blocks in the status of Connected or Invalid will be extended to the leaves of the Status Tree during Connecting Header. As a result, Connecting Header stage explores and extends the status tree. +## Download Block -## Downloading Block +After Connect Header is completed, the branch of some observed Best Chain Tip ends with one or more `Connected` block, a.k.a., Connected Chain. Download Block stage should start to request complete blocks from peers and perform verification. -After Connecting Header is completed, the branch of some observed Best Chain Tip ends with one or more Connected block, a.k.a., Connected Chain. Downloading Block stage should start to request complete blocks from peers and perform verification. +With the status tree, synchronization can be scheduled to avoid useless work. An effective optimization is to download the block only if the Best Chain of the observed peer is better than the local Best Chain. And priority can be ordered that the connected chain with more accumulated PoW should be processed first. The branch with lower PoW can only be attempted if a branch is confirmed to be `Invalid` or if the download times out. -With the status tree, synchronization can be scheduled to avoid useless work. An effective optimization is to download the block only if the Best Chain of the observed peer is better than the local Best Chain's. And priority can be ordered that the connected chain with more accumulated PoW should be processed first. The branch with lower PoW can only be attempted if a branch is confirmed to be invalid or if the download times out. +When downloading a branch, earlier blocks should be downloaded first due to the block dependency, and should be downloaded concurrently from different peers to utilize full bandwidth. A sliding window can be applied to solve the problem. -When downloading a branch, earlier blocks should be downloaded firstly due to the dependency of blocks, and should be downloaded concurrently from different peers to utilize full bandwidth. A sliding window can be applied to solve the problem. +Assume that the number of the first `Connected` status block to be downloaded is `M` and the length of the sliding window is `N`, then only the blocks numbered `M` to `M+N-1` can be downloaded. After the block M is downloaded and verified, the sliding window moves to the next `Connected` block. If verification of block M fails, then the remaining blocks of this branch are all `Invalid`, and there is no need to continue downloading. If the window does not move towards the right for a long time, it is considered as time out. The node should try again later, or waits until the branch has new `Connected` blocks. -Assume that the number of the first Connected status block to be downloaded is M and the length of the sliding window is N, then only the blocks numbered M to M+N-1 can be downloaded. After the block M is downloaded and verified, the sliding window moves to the next Connected block. If verification of block M fails, then the remaining blocks of this branch are all Invalid, and there is no need to continue downloading. If the window does not move towards the right for a long time, it is considered as time out. The node should try again later, or waits until the branch has new connected blocks. +

+ +

-![](images/sliding-window.jpg) +The figure above is an example of a sliding window with length `8`. In the beginning, the downloadable blocks are `[3, 10]`. After block 3 is downloaded, the window will move to block 5 because block 4 has already been downloaded in advance. -The figure above is an example of an 8 length sliding window. In the beginning, the downloadable block range from 3 to 10. After block 3 is downloaded, the window will move to block 5 because block 4 has already been downloaded in advance (as the figure above illustrated). +The Best Chains of peers are already known in Connect Header stage, it is assumed that the peer has a block if it is in the peer's Best Chain and that peer is a full node. During the downloading, blocks in the sliding window can be split into several small stripes and those stripes could be scheduled among peers who have the blocks. -The Best Chains of peers are already known in stage Connecting Header, it is assumed that the peer has a block if it is in the peer's Best Chain and that peer is a full node. During the downloading, blocks in the sliding window can be split into several small stripes and those stripes could be scheduled among peers who have the blocks. +The downloaded transactions in a block may be mismatched with the Merkle Hash Root in the header, or the list contains duplicated txid. It doesn't mean that the block is invalid. It can only approve the downloaded block is incorrect. The block content provider could be added to the blacklist, but the block status should not be marked as `Invalid`. Otherwise, the malicious nodes may pollute the nodes' Status Tree by sending the wrong block contents. -The downloaded transactions in a block may be mismatched with the Merkle Hash Root in the header, or the list contains duplicated txid. It doesn't mean that the block is invalid since it can only approve the downloaded block is incorrect. The block content provider could be added to the blacklist, but the block status should not be marked as invalid. Otherwise, the malicious nodes may pollute the nodes' Status Tree by sending the wrong block contents. - -Verification of transaction lists and block header matching is required in this stage, but any validation that relies on the transaction contents in the ancestor block is not required, which will be placed in the next stage. +Verification of transaction lists and block header matching is required in this stage, but any validation that relies on the transaction contents in the ancestor block is not required and they will be conducted in the next stage. Several validations can be checked in this phase, for example, Merkle Hash validation, transaction txid cannot be repeated, transaction list cannot be empty, inputs and outputs cannot be blank at the same time, or only the first transaction can be generation transaction, etc. -Downloading Block will update the status of blocks in the best Connected Chain, from Connected to Downloaded or Invalid. - -## Accepting Block +Download Block will update the status of blocks in the best Connected Chain, from `Connected` to `Downloaded` or `Invalid`. -In the previous stage, there will be some chains which ended with one or more Downloaded status, hereinafter referred to as Downloaded Chain. If those chains' cumulative work is more than Best Chain Tip's, the complete validation in the chain context should be performed in this stage. If there are more than one chains satisfied, the chain with the most work should be performed first. +## Accept Block -All the verification must be completed in this stage, including all rules that depend on historical transactions. +In the previous stage, there will be some chains which ended with `Downloaded` status, hereinafter referred to as Downloaded Chain. If those chains' cumulative work is more than the Best Chain Tip's, the complete validation in the chain context should be performed in this stage. If there are more than one chain satisfy, the chain with the most work should be performed first. -Because it involves UTXO (unspent transaction outputs) indexes, the cost of verification is huge in this phase. One set of UTXO indexes is sufficient in this simple solution. First rollback local Best Chain Tip necessarily. After that, verify blocks in the candidate best Downloaded Chain and add them to Best Chain one by one. If there is an invalid block during verification, the remain blocks in Downloaded Chain are also considered as Invalid. If so, Best Chain Tip would even have lower work than the previous Tip. It can be resolved in several different ways: +All the verification must be completed in this stage, including all rules that depend on historical transactions. Because it involves UTXO (unspent transaction outputs) indexes, the cost of verification is huge in this phase. It can be simplified to keep only one set of UTXO indexes. First rollback local Best Chain Tip when necessary. After that, verify blocks in the candidate best Downloaded Chain and add them to the Best Chain one by one. If there is an invalid block during verification, the remain blocks in Downloaded Chain are also considered as `Invalid`. With the simplified solution, the Best Chain Tip would even have lower work than the previous Tip, which can be resolved in several different ways: -- If the work of Best Chain before rollback is more than present Tip, then restore the previous Best Chain. -- If the work of other Downloaded Chains is more than Best Chain that before rollback, try rollback and relocate to that chain. +- If the work of the Best Chain before rollback is more than the present Tip, then restore the previous Best Chain. +- If the work of other Downloaded Chains is more than the Best Chain before rollback, try rollback and relocate to that chain. -The process of Accepting Block will change the status of blocks in the Downloaded chain, from Downloaded to Accepted or Invalid. The verified Downloaded Chain which has the most work will become the new local Best Chain. +The process of Accept Block will change the status of blocks in the Downloaded chain, from `Downloaded` to `Accepted` or `Invalid`. The verified Downloaded Chain which has the most work will become the new local Best Chain. -## New block announcement +## New Block Announcement -When the local Best Chain Tip changes, the node should push an announcement to peers. The best header with most cumulative work sent to each peer should be recorded, to avoid sending duplicate blocks in the announcement and sending blocks only peer doesn't know. This does not only record headers sent for new blocks, but also the ones sent as the responses in stage Connecting Header. +When the local Best Chain Tip changes, the node should push an announcement to peers. The best header with most cumulative work sent to each peer should be recorded, to avoid sending duplicate blocks before the announced new Best Chain Tip. The best header recorded not only includes the headers in New Block Announcement, but also the ones sent as the responses in Connect Header stage. It is assumed that the peers already know the Best Sent Header and its ancestors, so these blocks can be excluded when sending new block announcements. -![](images/best-sent-header.jpg "Best Sent Header") +

+ +

-From the above example, Alice's Best Chain Tip is annotated with her name. The best header sent to Bob is annotated as "Best Sent To Bob". The undimmed blocks are the ones Alice should send to Bob as new blocks announcement. Following is the detailed description for each step: +From the above example, Alice's Best Chain Tip is annotated with her name. The best header sent to Bob is annotated as "Best Sent To Bob". The dark green blocks are the ones Alice should send to Bob in New Block Announcement. Following is the detailed description for each step: -1. In the beginning, Alice only has Best Chain Tip to send -2. Another new block is added to the best chain before Alice has a chance to send the headers. In this case, the last two blocks of Best Chain need to be sent. -3. Alice sends the last two blocks to Bob and updates Best Sent to Bob. +1. Alice sends `DE`, which are blocks from "Best Sent to Bob" to Best Chain Tip, to Bob. +2. Alice updates "Best Sent to Bob" to be the Best Chain Tip. 4. Alice's Best Chain relocates to another fork. Only blocks after the last common block should be sent to Bob. How to send the announcement is determined by connection negotiated parameters and the number of new blocks to be announced: -- If there is only one block and the peer prefers Compact Block [^1], then use Compact Block. -- In other cases, just send block header list with an upper limit on the number of blocks to send. For example, if the limit is 8 and there are 8 or more blocks need to be announced, only the latest 7 blocks will be announced. - -When receiving a new block announcement, there may be a situation the parent block's status is Unknown, also called Orphan Block. If so, a new round of Connecting Header is required immediately. When a Compact Block is received, and its parent block is the local Best Chain Tip, then the full block may be recovered from the transaction pool. If the recovery succeeds, the work of these three stages can be compacted into one. Otherwise, it falls back to a header-only announcement. +If there is only one block and the peer prefers Compact Block [^1], then use Compact Block. For other cases, just send the block header list with an upper limit `k` on the number of blocks to send. If exceeded, the node only sends latest `k-1` blocks. As a result, when receiving a new block announcement, there may be a situation the parent block's status is `Unknown`, also called Orphan Block. If so, a new round of Connect Header is required immediately. When a Compact Block is received, and its parent block is the local Best Chain Tip, then the full block may be recovered from the transaction pool. If the recovery succeeds, the work of these three stages can be compacted into one. Otherwise, it falls back to a header-only announcement. ## Synchronization Status ### Configuration -- `GENESIS_HASH`: hash of genesis block +- `GENESIS_HASH`: hash of Genesis block - `MAX_HEADERS_RESULTS`: the max number of block headers can be sent in a single message - `MAX_BLOCKS_TO_ANNOUNCE`: the max number of new blocks to be announced - `BLOCK_DOWNLOAD_WINDOW`: the size of the download window @@ -192,19 +195,19 @@ Compact Block [^1] messages `cmpctblock` and `getblocktxn` will be described in ### getheaders -It is used to request a block header from a peer in stage Connecting Header. The first-page request, and subsequent pages request can share the same getheaders message format. The difference between them is that the first page requests generate a Locator from the parent block of the local Best Header Chain Tip, and the subsequent page request generates the Locator using the last block in the last received page. +It is used to request a block header from a peer in Connect Header stage. The first-page request, and subsequent pages request can share the same getheaders message format. The difference between them is that the first page request generates a Locator from the parent block of the local Best Header Chain Tip, and the subsequent page request generates the Locator using the last block in the last received page. -- `locator`: Sampled hashes of the already known blocks +- `Locator`: Sampled hashes of the already known blocks ### headers -It is used to reply `getheaders` and announce new blocks. There is no difference in processing logic, but if an Orphan Block is founded when the number of block headers is less than `MAX_BLOCKS_TO_ANNOUNCE`, a new round of Connecting Header is required. If the number of block `headers` received equals is equal to `MAX_HEADERS_RESULTS`, it indicates that there are more blocks to request. +It is used to reply `getheaders` and announce new blocks. There is no difference in processing logic, but if an Orphan Block is founded when the number of block headers is less than `MAX_BLOCKS_TO_ANNOUNCE`, a new round of Connect Header is required. If the number of block `headers` received is equal to `MAX_HEADERS_RESULTS`, it indicates that there are more blocks to request. - `headers`:block headers list ### getdata -It is used in Downloading Block stage. +It is used in Download Block stage. - `inventory`: object lists for download, with following fields in each list element: - `type`: type of the object, only "block" here diff --git a/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.zh.md b/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.zh.md index 6862b4643..de42c23fb 100644 --- a/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.zh.md +++ b/rfcs/0004-ckb-block-sync/0004-ckb-block-sync.zh.md @@ -10,160 +10,161 @@ Created: 2018-07-25 术语说明 -- Chain: 创世块开头,由连续的块组成的链。 -- Best Chain: 节点之间要达成最终一致的、满足共识验证条件的、PoW 累积工作量最高的、以共识的创世块开始的 Chain。 -- Best Header Chain: 累积工作量最高,由状态是 Connected, Downloaded 或者 Accepted 的块组成的 Chain。详见下面块状态的说明。 -- Tip: Chain 最后一个块。Tip 可以唯一确定 Chain。 -- Best Chain Tip: Best Chain 的最后一个块。 +- Chain: 创世块开头,由连续的块组成的链 +- Best Chain: 节点之间要达成最终一致的、满足共识验证条件的、PoW 累积工作量最高的、以共识的创世块开始的 Chain +- Best Header Chain: 累积工作量最高,由状态是 `Connected`, `Downloaded` 或者 `Accepted` 的块组成的 Chain。详见下面块状态的说明。 +- Tip: Chain 最后一个块。Tip 可以唯一确定 Chain +- Best Chain Tip: Best Chain 的最后一个块 ## 同步概览 -块同步**必须**分阶段进行,采用 [Bitcoin Headers First](https://bitcoin.org/en/glossary/headers-first-sync) 的方式。每一阶段获得一部分块的信息,或者基于已有的块信息进行验证,或者两者同时进行。 +Ckb Layer1 的块同步采用 [Bitcoin Headers First](https://btcinformation.org/en/glossary/headers-first-sync) 的方式,**必须**分阶段进行。同步的每一阶段或者获得一部分块的信息,或者基于已有的块信息进行验证,亦可两者同时进行。 -1. 连接块头 (Connect Header): 获得块头,验证块头格式正确且 PoW 工作量有效 -2. 下载块 (Download Block): 获得块内容,验证完整的块,但是不依赖祖先块中的交易信息。 -3. 采用块 (Accept Block): 在链上下文中验证块,会使用到祖先块中的交易信息。 +以下为节点同步的 4 个步骤: -分阶段执行的主要目的是先用比较小的代价排除最大作恶的可能性。举例来说,第一步连接块头的步骤在整个同步中的工作量可能只有 5%,但是完成后能有 95% 的可信度认为块头对应的块是有效的。 +1. 连接块头 (Connect Header): 获得块头,验证块头格式正确且 PoW 工作量有效。 +2. 下载块 (Download Block): 获得块内容,验证完整的块,但是不依赖祖先块中的交易信息。 +3. 采用块 (Accept Block): 在链上下文中验证块,会使用到祖先块中的交易信息。 +4. 新块通知 (New Block Announcement): 向其他节点推送更新的 Best Chain Tip。 + +分阶段执行的主要目的是先用比较小的代价排除最大作恶的可能性。如第一步连接块头的步骤在整个同步中的工作量可能只有 5%,但是完成后能有 95% 的可信度认为块头对应的块是有效的。 按照已经执行的阶段,块可以处于以下 5 种状态: -1. Unknown: 在连接块头执行之前,块的状态是未知的。 -2. Invalid:任意一步失败,块的状态是无效的,且当一个块标记为 Invalid,它的所有子孙节点也都标记为 Invalid。 -3. Connected: 连接块头成功,且该块到创世块的所有祖先块都必须是 Connected, Downloaded 或 Accepted 的状态。 -4. Downloaded: 下载块成功,且该块到创世块的所有祖先块都必须是 Downloaded 或者 Accepted 的状态。 -5. Accepted: 采用块成功,且该块到创世块的所有祖先块都必须是 Accepted 的状态。 +- `Unknown`: 在连接块头执行之前,块的状态是未知的 +- `Invalid`:任意一步失败,块的状态是无效的,且当一个块标记为 `Invalid`,它的所有子孙节点也都标记为 `Invalid` +- `Connected`: 连接块头成功,且该块到创世块的所有祖先块都必须是 `Connected`, `Downloaded` 或 `Accepted` 的状态 +- `Downloaded`: 下载块成功,且该块到创世块的所有祖先块都必须是 `Downloaded` 或者 `Accepted` 的状态 +- `Accepted`: 采用块成功,且该块到创世块的所有祖先块都必须是 `Accepted` 的状态 -块的状态是会沿着依赖传递的。按照上面的编号,子块的状态编号一定不会大于父块的状态编号。首先,如果某个块是无效的,那依赖它的子孙块自然也是无效的。另外,同步的每一步代价都远远高于前一步,且每一步都可能失败。如果子节点先于父节点进入下一阶段,而父节点被验证为无效,那子节点上的工作量就浪费了。而且,子块验证是要依赖父块的信息的。 +本文将使用下图示例表示不同状态的块: -初始时创世块状态为 Accepted,其它所有块为 Unknown。 +

+ +

-之后会使用以下图示表示不同状态的块: +块的状态会沿着依赖传递。按照上面的编号,子块的状态编号一定不会大于父块的状态编号。首先,如果某个块是无效的,那依赖它的子孙块自然也是无效的。另外,同步的每一步代价都远远高于前一步,且每一步都可能失败。如果子节点先于父节点进入下一阶段,而父节点被验证为无效,那子节点上的工作量就浪费了。而且,子块验证是要依赖父块的信息的。 -![](images/block-status.jpg "Block Status") +在链初始时创世块状态为 `Accepted`,其它所有块为 `Unknown`。而参与同步的节点创世块**必须**相同,所有的块组成由创世块为根的一颗树。如果一个块无法最终连接到创世块,与之相连的所有块将被不被处理。 -参与同步的节点创世块**必须**相同,所有的块必然是组成由创世块为根的一颗树。如果块无法最终连接到创世块,这些块都可以丢弃不做处理。 -参与节点都会在本地构造这颗状态树,其中全部由 Accepted 块组成的累积工作量最大的链就是 Best Chain。而由状态可以是 Connected, Downloaded 或 Accepted 块组成的累积工作量最大的链就是 Best Header Chain. +所有链同步的参与节点都会在本地构造这颗状态树,其中全部由 `Accepted` 块组成的累积工作量最大的链就是 Best Chain。而由状态可以是 `Connected`, `Downloaded` 或 `Accepted` 块组成的累积工作量最大的链就是 Best Header Chain. 下图是节点 Alice 构造的状态树的示例,其中标记为 Alice 的块是该节点当前的 Best Chain Tip。 -![](images/status-tree.jpg "Status Tree by Alice") +

+ +

## 连接块头 -先同步 Headers 可以用最小的代价验证 PoW 有效。构造 PoW 时,不管放入无效的交易还是放入有效的交易都需要付出相同的代价,那么攻击者会选择其它更高性价比的方式进行攻击。可见,当 PoW 有效时整个块都是有效的概率非常高。所以先同步 Headers 能避免浪费资源去下载和验证无效块。 - -因为代价小,同步 Headers 可以和所有的节点同时进行,在本地能构建出可信度非常高的、当前网络中所有分叉的全局图。这样可以对块下载进行规划,避免浪费资源在工作量低的分支上。 +先同步 Headers 可以用最小的代价验证 PoW 有效。构造 PoW 时,不管放入无效的交易还是放入有效的交易都需要付出相同的代价,因此攻击者常会选择其它更高性价比的方式进行攻击,如篡改 Headers。可见,当 PoW 有效时整个块都是有效的概率非常高。所以先同步 Headers 能避免浪费资源去下载和验证无效块。 -连接块头这一步的目标是,当节点 Alice 连接到节点 Bob 之后,Alice 让 Bob 发送所有在 Bob 的 Best Chain 上但不在 Alice 的 **Best Header Chain** 上的块头,进行验证并确定这些块的状态是 Connected 还是 Invalid。 +因为代价小,同步 Headers 可以和所有的节点同时进行。节点在本地能构建出可信度非常高的、当前网络中所有分叉的全局图。这样可以对块下载进行规划,避免浪费资源在工作量低的分支上。 -Alice 在连接块头时,需要保持 Best Header Chain Tip 的更新,这样能减少收到已有块头的数量。 +连接块头的目标是,当节点 Alice 连接到节点 Bob 之后,Alice 让 Bob 发送所有在 Bob 的 **Best Chain** 上但不在 Alice 的 **Best Header Chain** 上的块头,进行验证并确定这些块的状态是 `Connected` 还是 `Invalid`。Alice 在连接块头时,需要保持 Best Header Chain Tip 的更新,这样能减少收到已有块头的数量。 -![](images/seq-connect-headers.jpg) +下图是一轮连接块头的流程。完成了第一轮连接块头后,节点之间通过新块通知保持之后的同步。 -上图是一轮连接块头的流程。完成了一轮连接块头后,节点之间应该通过新块通知保持之后的同步。 +

+ +

-以上图 Alice 从 Bob 同步为例,首先 Alice 将自己 Best Header Chain 中的块进行采样,将选中块的哈希作为消息内容发给 Bob。采样的基本原则是最近的块采样越密,越早的块越稀疏。比如可以取最后的 10 个块,然后从倒数第十个块开始按 2, 4, 8, … 等以 2 的指数增长的步长进行取样。采样得到的块的哈希列表被称为 Locator。下图中淡色处理的是没有被采样的块,创世块应该始终包含在 Locator 当中。 +首先 Alice 将自己 Best Header Chain 中的块进行采样,将选中块的哈希作为消息内容发给 Bob。采样的基本原则是越新的块采样越密,越老的块越稀疏。比如可以取最后 10 个块,然后从倒数第 10 个块开始按 `2, 4, 8, …` 等以 2 的指数增长的步长进行取样。采样得到的块的哈希列表被称为 Locator。创世块应该始终包含在 Locator 当中。 -![](images/locator.jpg) +

+ +

Bob 根据 Locator 和自己的 Best Chain 可以找出两条链的最后一个共同块。因为创世块相同,所以一定存在这样一个块。Bob 把共同块之后一个开始到 Best Chain Tip 为止的所有块头发给 Alice。 -![](images/connect-header-conditions.jpg) - -上图中未淡出的块是 Bob 要发送给 Alice 的块头,金色高亮边框的是最后共同块。下面列举了同步会碰到的三种情况: +

+ +

-1. Bob 的 Best Chain Tip 在 Alice 的 Best Header Chain 中,最后共同块就是 Bob 的 Best Chain Tip,Bob 没有块头可以发送。 -2. Alice 的 Best Header Chain Tip 在 Bob 的 Best Chain 中并且不等于 Tip,最后共同块就是 Alice 的 Best Header Chain Tip。 -3. Alice 的 Best Header Chain 和 Bob 的 Best Chain 出现了分叉,最后共同块是发生发叉前的块。 +注意在上图中深绿的块是 Alice 发送的 Locator。下面列举了同步会碰到的三种情况: -如果要发送的块很多,需要做分页处理。Bob 先发送第一页,Alice 通过返回结果发现还有更多的块头就继续向 Bob 请求接下来的页。一个简单的分页方案是限制每次返回块头的最大数量,比如 2000。如果返回块头数量等于 2000,说明可能还有块可以返回,就接着请求之后的块头。如果某页最后一个块是 Best Header Chain Tip 或者 Best Chain Tip 的祖先,可以优化成用对应的 Tip 生成 Locator 发送请求,减少收到已有块头的数量。 +- Bob 的 Best Chain Tip 在 Alice 的 Best Header Chain 中,最后共同块就是 Bob 的 Best Chain Tip,Bob 没有块头可以发送。 +- Alice 的 Best Header Chain Tip 在 Bob 的 Best Chain 中并且不等于 Tip,最后共同块就是 Alice 的 Best Header Chain Tip。Bob 此时发送 `F`。 +- Alice 的 Best Header Chain 和 Bob 的 Best Chain 出现了分叉,最后共同块是发生发叉前的块。Bob 此时发送 `DE'F'`。 -在同步的同时,Alice 可以观察到 Bob 当前的 Best Chain Tip,即在每轮同步时最后收到的块。如果 Alice 的 Best Header Chain Tip 就是 Bob 的 Best Chain Tip ,因为 Bob 没有块头可发,Alice 就无法观测到 Bob 目前的 Best Chain。所以在每轮连接块头同步的第一个请求时,**应该**从 Best Header Chain Tip 的父块开始构建,而不包含 Tip。 +如果要发送的块很多,需要做分页处理。一个简单的分页方案是限制每次返回块头的最大数量。如果返回块头数量等于最大数量,说明可能还有块可以返回,就接着请求之后的块头。如果某页最后一个块是 Best Header Chain Tip 或者 Best Chain Tip 的祖先,可以优化成用对应的 Tip 生成 Locator 发送请求,减少收到已有块头的数量。 -在下面的情况下**必须**做新一轮的连接块头同步。 +在同步的同时,Alice 可以观察到 Bob 当前的 Best Chain Tip,即在每轮同步时最后收到的块。如果 Alice 的 Best Header Chain Tip 就是 Bob 的 Best Chain Tip ,因为 Bob 没有块头可发,Alice 就无法观测到 Bob 目前的 Best Chain。所以在每轮连接块头同步的第一个请求时,应该从 Best Header Chain Tip 的父块开始构建,而不包含 Tip。 -- 收到对方的新块通知,但是新块的父块状态是 Unknown +当收到对方的新块通知,但是新块的父块状态是 `Unknown`时,节点**必须**做新一轮的连接块头进行同步。 连接块头时可能会出现以下一些异常情况: -- Alice 观察到的 Bob Best Chain Tip 很长一段时间没有更新,或者时间很老。这种情况 Bob 无法提供有价值的数据,当连接数达到限制时,可以优先断开该节点的连接。 -- Alice 观察到的 Bob Best Chain Tip 状态是 Invalid。这个判断不需要等到一轮 Connect Head 结束,任何一个分页发现有 Invalid 的块就可以停止接受剩下的分页了。因为 Bob 在一个无效的分支上,Alice 可以停止和 Bob 的同步,并将 Bob 加入到黑名单中。 -- Alice 收到块头全部都在自己的 Best Header Chain 里,这有两种可能,一是 Bob 故意发送,二是 Alice 在 Connect Head 时 Best Chain 发生了变化,由于无法区分只能忽略,但是可以统计发送的块已经在本地 Best Header Chain 上的比例,高于一定阈值可以将对方加入到黑名单中。 +- Alice 观察到的 Bob Best Chain Tip 长时间未更新,或者时间很老。这种情况 Bob 无法提供有价值的数据,当连接数达到限制时,可以优先断开该节点的连接。 +- Alice 观察到的 Bob Best Chain Tip 状态是 `Invalid`。在任何一个分页发现有 `Invalid` 的块就可以停止同步。因为 Bob 在一个无效的分支上,Alice 可以停止和 Bob 的同步,并将 Bob 加入到黑名单中。 +- Alice 收到块头全部都在自己的 Best Header Chain 里,这有两种可能,一是 Bob 故意发送,二是 Alice 在 Connect Head 时 Best Chain 发生了变化,由于无法区分只能忽略,但是可以统计发送的块已经在本地 Best Header Chain 上的比例,高于一定阈值可以断开与 Bob 的连接,或者将对方加入到黑名单中。 -在收到块头消息时可以先做以下格式验证: +在收到块头消息时需做以下验证: - 消息中的块是连续的 -- 所有块和第一个块的父块在本地状态树中的状态不是 Invalid -- 第一个块的父块在本地状态树中的状态不是 Unknown,即同步时不处理 Orphan Block。 - -这一步的验证包括检查块头是否满足共识规则,PoW 是否有效。因为不处理 Orphan Block,难度调整也可以在这里进行验证。 - -![](images/connect-header-status.jpg) +- 所有块和第一个块的父块在本地状态树中的状态不是 `Invalid` +- 第一个块的父块在本地状态树中的状态不是 `Unknown`,即同步时不处理 Orphan Block +- 检查块头是否满足共识规则,PoW 是否有效 -上图是 Alice 和 Bob, Charlie, Davis, Elsa 等节点同步后的状态树情况和观测到的其它节点的 Best Chain Tip。 +

+ +

-如果认为 Unknown 状态块是不在状态树上的话,在连接块头阶段,会在状态树的末端新增一些 Connected 或者 Invalid 状态的节点。所以可以把连接块头看作是拓展状态树,是探路的阶段。 +上图是 Alice 和 Bob, Charlie, Davis, Elsa 等节点同步后的状态树情况和观测到的其它节点的 Best Chain Tip。`Unknown` 状态块不在状态树上。在连接块头阶段,会在状态树的末端新增一些 `Connected` 或者 `Invalid` 状态的节点。所以可以把连接块头看作是拓展状态树,是探路的阶段。 ## 下载块 -完成连接块头后,一些观测到的邻居节点的 Best Chain Tip 在状态树上的分支是以一个或者多个 Connected 块结尾的,即 Connected Chain,这时可以进入下载块流程,向邻居节点请求完整的块,并进行必要的验证。 +完成连接块头后,一些观测到的邻居节点的 Best Chain Tip 在状态树上的分支是以一个或者多个 `Connected` 块结尾的,即 Connected Chain,这时可以进入下载块流程,向邻居节点请求完整的块,并进行必要的验证。 -因为有了状态树,可以对同步进行规划,避免做无用工作。一个有效的优化就是只有当观测到的邻居节点的 Best Chain 的累积工作量大于本地的 Best Chain 的累积工作量才进行下载块。而且可以按照 Connected Chain 累积工作量为优先级排序,优先下载累积工作量更高的分支,只有被验证为 Invalid 或者因为下载超时无法进行时才去下载优先级较低的分支。 +因为有了状态树,可以对同步进行规划,避免做无用工作。一个有效的优化就是只有当观测到的邻居节点的 Best Chain 的累积工作量大于本地的 Best Chain 的累积工作量才进行下载块。而且可以按照 Connected Chain 累积工作量为优先级排序,优先下载累积工作量更高的分支,只有被验证为 `Invalid` 或者因为下载超时无法进行时才去下载优先级较低的分支。 下载某个分支时,因为块的依赖性,应该优先下载更早的块;同时应该从不同的节点去并发下载,充分利用带宽。这可以使用滑动窗口解决。 -假设分支第一个要下载的 Connected 状态块号是 M,滑动窗口长度是 N,那么只去下载 M 到 M + N - 1 这 N 个块。在块 M 下载并验证后,窗口往右移动到下一个 Connected 状态的块。如果块 M 验证失败,则分支剩余的块也就都是 Invalid 状态,不需要继续下载。如果窗口长时间没有向右移动,则可以判定为下载超时,可以在尝试其它分支之后再进行尝试,或者该分支上有新增的 Connected 块时再尝试。 +假设分支第一个要下载的 `Connected` 状态块号是 `M`,滑动窗口长度是 `N`,那么只去下载 `M` 到 `M + N - 1` 这 `N` 个块。在块 M 下载并验证后,窗口往右移动到下一个 `Connected` 状态的块。如果块 `M` 验证失败,则分支剩余的块也就都是 `Invalid` 状态,不需要继续下载。如果窗口长时间没有向右移动,则可以判定为下载超时,可以在尝试其它分支之后再进行尝试,或者该分支上有新增的 `Connected` 块时再尝试。 -![](images/sliding-window.jpg) +

+ +

-上图是一个长度为 8 的滑动窗口的例子。开始时可下载的块是从 3 到 10。块 3 下载后,因为 4 已经先下载好了,所以窗口直接滑动到从 5 开始。 +上图是一个长度为 `8` 的滑动窗口的例子。开始时可下载的块是`[3, 10]`。块 3 下载后,因为 4 已经先下载好了,所以窗口直接滑动到从 5 开始。 因为通过连接块头已经观测到了邻居节点的 Best Chain,如果在对方 Best Chain 中且对方是一个全节点,可以认为对方是能够提供块的下载的。在下载的时候可以把滑动窗口中的块分成小块的任务加到任务队列中,在能提供下载的节点之间进行任务调度。 -下载块如果出现交易对不上 Merkle Hash Root 的情况,或者能对上但是有重复的交易 txid 的情况,并不能说明块是无效,只是没有下载到正确的块内容。可以将对方加入黑名单,但是不能标记块的状态为 Invalid,否则恶意节点可以通过发送错误的块内容来污染节点的状态树。 +下载块如果出现交易对不上 Merkle Hash Root 的情况,或者能对上但是有重复的交易 txid 的情况,并不能说明块是无效,只是没有下载到正确的块内容。可以将对方加入黑名单,但是不能标记块的状态为 `Invalid`,否则恶意节点可以通过发送错误的块内容来污染节点的状态树。 -这一阶段需要验证交易列表和块头匹配,但是不需要做任何依赖祖先块中交易内容的验证,这些验证会放在下一阶段进行。 +这一阶段需要验证交易列表和块头匹配,但是不需要做任何依赖祖先块中交易内容的验证,这些验证会放在下一阶段进行。可以进行的验证比如 Merkel Hash 验证、交易 txid 不能重复、交易列表不能为空、所有交易不能 inputs outputs 同时为空、只有第一个交易可以是 generation transaction 等等。 -可以进行的验证比如 Merkel Hash 验证、交易 txid 不能重复、交易列表不能为空、所有交易不能 inputs outputs 同时为空、只有第一个交易可以是 generation transaction 等等。 - -下载块会把状态树中工作量更高的 Connected Chain 中的 Connected 块变成 Downloaded 或者 Invalid。 +下载块会把状态树中工作量更高的 Connected Chain 中的 `Connected` 块变成 `Downloaded` 或者 `Invalid`。 ## 采用块 -在上一阶段中会产生一些以一个或多个 Downloaded 状态的块结尾的链,以下简称为 Downloaded Chain。如果这些链的累积工作量大于 Best Chain Tip, 就可以对这条链进行该阶段完整的合法性验证。如果有多个这样的链,选取累积工作量最高的。 - -这一阶段需要完成所有剩余的验证,包括所有依赖于历史交易内容的规则。 +在上一阶段中会产生若干以 `Downloaded` 状态的块结尾的链,以下简称为 Downloaded Chain。如果这些链的累积工作量大于 Best Chain Tip, 就可以对这条链进行该阶段完整的合法性验证。如果有多个这样的链,选取累积工作量最高的。 -因为涉及到 UTXO (未消耗掉的交易 outputs) 的索引,这一步的验证开销是非常大的。为了简化系统,可以只保留一套 UTXO 索引,尝试将本地的 Best Chain Tip 进行必要回退,然后将 Downloaded Chain 上的块进行一次验证,再添加到 Best Chain 上。如果中间有块验证失败则 Downloaded Chain 上剩余的块也就都是 Invalid 状态不需要再继续。这时 Best Chain Tip 甚至会低于之前的 Tip,如果遇到可以采取以下的方案处理: +这一阶段需要完成所有剩余的验证,包括所有依赖于历史交易内容的规则。因为涉及到 UTXO (未消耗掉的交易 outputs) 的索引,这一步的验证开销很大。为了简化系统,可以只保留一套 UTXO 索引,尝试将本地的 Best Chain Tip 进行必要回退,然后将 Downloaded Chain 上的块进行验证,再添加到 Best Chain 上。如果中间有块验证失败则 Downloaded Chain 上剩余的块也就都是 `Invalid` 状态不需要再继续。这时 Best Chain Tip 有可能会低于之前的 Tip,如果遇到可以采取以下的方案处理: - 如果回退之前的 Best Chain 工作量比当前 Tip 更高,恢复之前的 Best Chain -- 如果有其它 Downloaded Chain 比回退之前的 Best Chain 工作量更高,可以继续使用下一个 Downloaded Chain 进行采用块的步骤。 +- 如果有其它 Downloaded Chain 比回退之前的 Best Chain 工作量更高,可以继续使用下一个 Downloaded Chain 进行采用块的步骤 -采用块会将工作量更高的 Downloaded Chain 中的 Downloaded 状态块变成 Accepted 或者 Invalid,而累积工作量最高的 Downloaded Chain 应该成为本地的 Best Chain。 +采用块会将工作量更高的 Downloaded Chain 中的 `Downloaded` 状态块变成 `Accepted` 或者 `Invalid`,而累积工作量最高的 Downloaded Chain 应该成为本地的 Best Chain。 ## 新块通知 -当节点的 Best Chain Tip 发生变化时,应该通过推送的方式主动去通知邻居节点。为了避免通知重复的块,和尽量一次性发送邻居节点没有的块,可以记录给对方发送过的累积工作量最高的块头 (Best Sent Header)。发送过不但指发送过新块通知,也包括发送过在连接块头时给对方的块头的回复。 +当节点的 Best Chain Tip 发生变化时,应该通过推送的方式主动去通知邻居节点。为了避免通知重复的块和尽量一次性发送邻居节点没有的块,可以记录给对方发送过的累积工作量最高的块头 (Best Sent Header)。发送过不但指发送过新块通知,也包括发送过在连接块头时给对方的块头的回复。 因为可以认为对方节点已经知道 Best Sent Header,及其祖先节点,所以发送新块通知时可以排除掉这些块。 -![](images/best-sent-header.jpg "Best Sent Header") - -上面的例子中标记为 Alice 的块是节点 Alice 的 Best Chain Tip。标记为 Best Sent to Bob 是记录的发送给 Bob 工作量最高的块头。其中未淡化的块是 Alice 需要通知给 Bob 的新块。数字对应的每一步说明如下: - -1. 开始时 Alice 只有 Best Chain Tip 需要发送 -2. Alice 还没有来得及发送,就又多了一个新块,这时需要发送 Best Chain 最后两个块头 -3. Alice 将最后两个块头发送给了 Bob 并同时更新了 Best Sent to Bob -4. Alice 的 Best Chain 发生了分支切换,只需要发送和 Best Sent to Bob 最后共同块之后的块。 +

+ +

-基于连接的协商参数和要通知的新块数量: +上面的例子中标记为 Alice 的块是节点 Alice 的 Best Chain Tip。标记为 Best Sent to Bob 是记录的发送给 Bob 工作量最高的块头。其中深绿色的块是 Alice 需要通知给 Bob 的新块。数字对应的每一步说明如下: -- 数量为 1 且对方偏好使用 Compact Block [^1],则使用 Compact Block -- 其它情况直接发送块头列表,但要限制发送块的数量不超过某个阈值,比如 8,如果有 8 个或更多的块要通知,只通知最新的 7 个块。 +1. Alice 发送 Best Sent to Bob 至 Best Chain Tip 的块 `DE` 给Bob。 +2. Alice 更新 Best Sent to Bob 为 Best Chain Tip. +2. Alice 的 Best Chain 发生了分支切换,只需要发送和 Best Sent to Bob 最后共同块之后的块 `E'F'`。 -当收到新块通知时,会出现父块状态是 Unknown 的情况,即 Orphan Block,这个时候需要立即做一轮连接块头的同步。收到 Compact Block 且父块就是本地的 Best Chain Tip 的时候可以尝试用交易池直接恢复,如果恢复成功,直接可以将三阶段的工作合并进行,否则就当作收到的只是块头。 +当配置同步新块数量为 1 且对方偏好使用 Compact Block [^1],使用 Compact Block 发送块数据。在其它情况下直接发送块头列表,限制发送块的数量不超过某个阈值 `k`。若超过该阈值,则只通知最新的 `k-1` 个块。因此当收到新块通知时,会出现父块状态是 `Unknown` 的情况,即 Orphan Block,这个时候需要立即做一轮连接块头的同步。而当收到 Compact Block 且父块就是本地的 Best Chain Tip 的时候可以尝试用交易池直接恢复,如果恢复成功,可直接将三阶段的工作合并进行,否则就当作收到的只是块头。 ## 同步状态 @@ -177,7 +178,7 @@ Bob 根据 Locator 和自己的 Best Chain 可以找出两条链的最后一个 ### 存储 - 块状态树 -- Best Chain Tip,决定是否要下载块和采用块。 +- Best Chain Tip,决定是否要下载块和采用块 - Best Header Chain Tip,连接块头时用来构建每轮第一个请求的 Locator 每个连接节点需要单独存储的 @@ -197,7 +198,7 @@ Compact Block [^1] 需要使用到的消息 `cmpctblock` 和 `getblocktxn` 会 用于连接块头时向邻居节点请求块头。请求第一页,和收到后续页使用相同的 getheaders 消息,区别是第一页是给本地的 Best Header Chain Tip 的父块生成 Locator,而后续页是使用上一页的最后一个块生成 Locator。 -- `locator`: 对 Chain 上块采样,得到的哈希列表 +- `Locator`: 对 Chain 上块采样,得到的哈希列表 ### headers diff --git a/rfcs/0004-ckb-block-sync/images/alice-status-tree.png b/rfcs/0004-ckb-block-sync/images/alice-status-tree.png new file mode 100644 index 000000000..1efc26905 Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/alice-status-tree.png differ diff --git a/rfcs/0004-ckb-block-sync/images/best-sent-header.jpg b/rfcs/0004-ckb-block-sync/images/best-sent-header.jpg deleted file mode 100644 index ec79d18f4..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/best-sent-header.jpg and /dev/null differ diff --git a/rfcs/0004-ckb-block-sync/images/block-status.jpg b/rfcs/0004-ckb-block-sync/images/block-status.jpg deleted file mode 100644 index f4f3042df..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/block-status.jpg and /dev/null differ diff --git a/rfcs/0004-ckb-block-sync/images/connect-header-conditions.jpg b/rfcs/0004-ckb-block-sync/images/connect-header-conditions.jpg deleted file mode 100644 index 5e9bc1b0c..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/connect-header-conditions.jpg and /dev/null differ diff --git a/rfcs/0004-ckb-block-sync/images/connect-header-conditions.png b/rfcs/0004-ckb-block-sync/images/connect-header-conditions.png new file mode 100644 index 000000000..13a1e27df Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/connect-header-conditions.png differ diff --git a/rfcs/0004-ckb-block-sync/images/connect-header-status.jpg b/rfcs/0004-ckb-block-sync/images/connect-header-status.jpg deleted file mode 100644 index 628854d6c..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/connect-header-status.jpg and /dev/null differ diff --git a/rfcs/0004-ckb-block-sync/images/connect-header-status.png b/rfcs/0004-ckb-block-sync/images/connect-header-status.png new file mode 100644 index 000000000..1e81503ce Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/connect-header-status.png differ diff --git a/rfcs/0004-ckb-block-sync/images/header-status.png b/rfcs/0004-ckb-block-sync/images/header-status.png new file mode 100644 index 000000000..18d9cde36 Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/header-status.png differ diff --git a/rfcs/0004-ckb-block-sync/images/header-sync.png b/rfcs/0004-ckb-block-sync/images/header-sync.png new file mode 100644 index 000000000..70d278c3f Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/header-sync.png differ diff --git a/rfcs/0004-ckb-block-sync/images/locator.jpg b/rfcs/0004-ckb-block-sync/images/locator.jpg deleted file mode 100644 index 213b010fa..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/locator.jpg and /dev/null differ diff --git a/rfcs/0004-ckb-block-sync/images/locator.png b/rfcs/0004-ckb-block-sync/images/locator.png new file mode 100644 index 000000000..53d732da7 Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/locator.png differ diff --git a/rfcs/0004-ckb-block-sync/images/seq-connect-headers.jpg b/rfcs/0004-ckb-block-sync/images/seq-connect-headers.png similarity index 100% rename from rfcs/0004-ckb-block-sync/images/seq-connect-headers.jpg rename to rfcs/0004-ckb-block-sync/images/seq-connect-headers.png diff --git a/rfcs/0004-ckb-block-sync/images/sliding-window.jpg b/rfcs/0004-ckb-block-sync/images/sliding-window.jpg deleted file mode 100644 index 5cdc2e32a..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/sliding-window.jpg and /dev/null differ diff --git a/rfcs/0004-ckb-block-sync/images/sliding-window.png b/rfcs/0004-ckb-block-sync/images/sliding-window.png new file mode 100644 index 000000000..28f4abf21 Binary files /dev/null and b/rfcs/0004-ckb-block-sync/images/sliding-window.png differ diff --git a/rfcs/0004-ckb-block-sync/images/status-tree.jpg b/rfcs/0004-ckb-block-sync/images/status-tree.jpg deleted file mode 100644 index 50e9fe79c..000000000 Binary files a/rfcs/0004-ckb-block-sync/images/status-tree.jpg and /dev/null differ