From c4ecf135a75aecde0608d3c868e9a4d814f83114 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Tue, 10 Nov 2020 16:38:45 -0800 Subject: [PATCH 01/10] wip: hash consistent sorted trees --- design/sorted-tree.md | 109 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 design/sorted-tree.md diff --git a/design/sorted-tree.md b/design/sorted-tree.md new file mode 100644 index 00000000..7e6098df --- /dev/null +++ b/design/sorted-tree.md @@ -0,0 +1,109 @@ +# Sorted Tree + +These is a living document. The purpose is to capture the current status of research on IPLD sorted trees. + +## Problem Statement + +There are numerous well sorted data structures we want to us in IPLD. Queues, large lists, sorted maps, sparse arrays, etc. All of these +have the same basic requirements: + +* Multi-block linear list of **ENTRIES** sorted by any user defined sorting method. +* Hash consistent structure regardless of inserion order. +* Managable churn rate on mutation. +* Random access for reads and mutations w/ predictable performance. + +Our favorite family of data structures for multi-block collections is HAMT's. By applying bucket configurations to the addressable +space inside of a hash we can then apply a hashing algorithm to any key and create a collection that is balanced and has predictable +churn on mutation. But it has problems: + +* Since the key is hashed we lose the ability to sort the structure by other means. +* Since the bucket settings are fixed the shape and churn rate of the data structure does not + alter itself all that well to different sizes without altering the bucket settings. + +### The Chunking Problem + +Since we need a sorted structure that is stored in IPLD we'll end up with a merkle tree of some sort. + +In IPLD, the challenge we always face with tree structures is how to break large lists of nodes into +individually serialized blocks. We'll call this "the chunking problem." + +Take this simple sorted tree example: + +``` + root ++-------------------------------+ +| +-----------+ +-----------+ | +| | 1, block1 | | 4, block2 | | +| +-----------+ +-----------+ | ++-------------------------------+ + + block1 block2 ++-------------+ +-------------+ +| +---+ +---+ | | +---+ +---+ | +| | 1 | | 2 | | | | 4 | | 5 | | +| +---+ +---+ | | +---+ +---+ | ++-------------+ +-------------+ +``` + +Here we have a small and simple balanced tree using a chunking function that holds only 2 nodes per block. + +If we want to insert `3` into the structure we're confronted with several problems. If we append `block1` or +prepend to `block2` you'll end up with a different tree shape, and a different root hash, than if you created +the structure anew using the chunking function. + +**Rule: all sorted tree's must be consisently chunked in order to produce consistent tree structures which then +produce consistent hashes.** This means that hash consistent tree will, in effect, be self-balancing upon mutation. + +If we stick with our "only 2 nodes per block" chunker then minor mutations to our list will produce mutations +to the entire tree on the right side of that mutation. This is obviously not ideal. + +What we need is a new chunking technique that produces nodes of a desirable length but also consistently splits +on particular **entries**. If we can find a way to consistently split on particular entries then we can avoid +large mutations to the rest of the tree. + +## First Tree: Sorted CID Set + +At this point we're ready to build our first tree. This data structure is a `Set` of CIDs sorted by binary comparison. +The nice thing about working with this use case is each CID is both the *key* and the *value*. + +CID's end in a multihash, and the multihash typically ends in a hash digest. For this use case, we'll say that CID's that are allowed +in this `Set` are limited to those that use sufficiently secure hashing functions. + +Since hash digests are, effectively, a form of randomization, we can simply convert the tail of each digest to an fixed size integer and designate +some part of that address space to cause splits in our chunker. + +For simplicity, let's convert the last byte to Uint8. Now, let's make the chunker close every chunk when it sees a `0`. This will give +use nodes that have, on average, 256 entries. + +``` +Digest Uint8 Tails ++----------------+ +| DIGEST-A | 56 | ++----------------+ ++----------------+ +| DIGEST-B | 123 | ++----------------+ ++----------------+ +| DIGEST-D | 0 | ++----------------+ +// split ++----------------+ +| DIGEST-M | 123 | ++----------------+ ++----------------+ +| DIGEST-N | 56 | ++----------------+ ++----------------+ +| DIGEST-L | 113 | ++----------------+ ++----------------+ +| DIGEST-O | 6 | ++----------------+ ++----------------+ +| DIGEST-P | 45 | ++----------------+ +``` + +Now, if we need to insert a new entry at `DIGEST-C` it will not effect the leaf nodes to the right, and as a result will have a limited +effect on the rest of the tree. + From 479bb99e218d63ec514761046fe8683e69e6d714 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Tue, 10 Nov 2020 17:27:40 -0800 Subject: [PATCH 02/10] wip: second tree --- design/sorted-tree.md | 52 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 7e6098df..1d37e70d 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -61,7 +61,21 @@ What we need is a new chunking technique that produces nodes of a desirable leng on particular **entries**. If we can find a way to consistently split on particular entries then we can avoid large mutations to the rest of the tree. -## First Tree: Sorted CID Set +## First Tree: Sorted CID Set (Tail Chunker) + +```ipldschema +type Entry link +type Leaf [ Entry ] +type Branch struct { + start Link + link Link +} representation tuple + +type Node union { + | "leaf" Leaf + | "branch" Branch +} representation keyed +``` At this point we're ready to build our first tree. This data structure is a `Set` of CIDs sorted by binary comparison. The nice thing about working with this use case is each CID is both the *key* and the *value*. @@ -107,3 +121,39 @@ Digest Uint8 Tails Now, if we need to insert a new entry at `DIGEST-C` it will not effect the leaf nodes to the right, and as a result will have a limited effect on the rest of the tree. +Since every node in this tree will be a content addressed block so we can continue to use the hash digest of every branch and apply the same chunking +technique. Whenever an entry or branch is changed we need to re-run the chunker on it and merge the entries in every node with the node to the right if +they are no longer closed. This keeps the hash of the tree consistent regardless of what order any mutations were made and it also incrementally re-balances +the tree on each mutation. + +We can control the performance of this tree by altering the chunking algorithm. If we use more of the tail we'll have a larger number space and can +increase the average chunk size beyond 256. Larger blocks mean more orphaned data on mutation. Smaller blocks mean deeper trees with more traversal. + +## Second Tree: Sorted Map (Tail Chunker) + +Now we'll build a tree that can illustrate a few attacks against this structure. Understanding how you can attack this structure +plays a key role in designing secure trees for different use cases. + +Let's make the entries in our tree a key/value pair. Let's make the schema require that values be links so that we can rely on the hash digest +for chunking. + +```ipldschema +type Entry struct { + key String + value Link +} representation tuple + +type Leaf [ Entry ] +type Branch [ Entry ] + +type Node union { + | "leaf" Leaf + | "branch" Branch +} representation keyed +``` + +You can use the tail chunker on this tree if you have complete control over the keys and values being inserted. You'll need to ensure that +the values being inserted have something unique or random. If you don't have this assurance and you end up inserting the same value into +the same part of the tree the leaf block will never close and you'll eventually cause an error when you go over MAX_BLOCK_SIZE. + +### First Attack: Unclosed Leaf From b7fa7bad0b6d39fc28fff39ec92512616cf12419 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Tue, 10 Nov 2020 20:56:51 -0800 Subject: [PATCH 03/10] wip: better chunker definition --- design/sorted-tree.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 1d37e70d..41d5bc46 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -61,6 +61,13 @@ What we need is a new chunking technique that produces nodes of a desirable leng on particular **entries**. If we can find a way to consistently split on particular entries then we can avoid large mutations to the rest of the tree. +## Chunker + +The chunker is a simple state machine. You feed the chunker unique identifiers for each entry. Each identifier must +use derived from the entry, typically through a hashing function or from a digest already present in the entry. + +The chunker should break as consistently as possible while also protecting against various attacks (see below). + ## First Tree: Sorted CID Set (Tail Chunker) ```ipldschema From a1179d89c57c04ab0e52bcefd239279fd7658539 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Wed, 11 Nov 2020 09:43:56 -0800 Subject: [PATCH 04/10] wip: new method --- design/sorted-tree.md | 187 ++++++++++-------------------------------- 1 file changed, 45 insertions(+), 142 deletions(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 41d5bc46..c73848da 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -1,166 +1,69 @@ -# Sorted Tree +# Hash Consistent Sorted Trees -These is a living document. The purpose is to capture the current status of research on IPLD sorted trees. +This document describes a technique for creating, mutating, and reading merkle DAG's that: -## Problem Statement +* Provide consistently performant random access. +* Self-balance on mutation. +* Every branch is a concurrency vector that can be safely mutated (lock-free thread safety). +* Are sorted by an arbitrary sort function. +* Can contain entries with any key and/or value type as long as each entry is unique. (Note: + this is not a *unique key constraint* as the entry is potentially a unique pairing of key **and value**). -There are numerous well sorted data structures we want to us in IPLD. Queues, large lists, sorted maps, sparse arrays, etc. All of these -have the same basic requirements: +## Basic Technique -* Multi-block linear list of **ENTRIES** sorted by any user defined sorting method. -* Hash consistent structure regardless of inserion order. -* Managable churn rate on mutation. -* Random access for reads and mutations w/ predictable performance. +Terms: -Our favorite family of data structures for multi-block collections is HAMT's. By applying bucket configurations to the addressable -space inside of a hash we can then apply a hashing algorithm to any key and create a collection that is balanced and has predictable -churn on mutation. But it has problems: +* NODE: LEAF or BRANCH. +* LEAF: Sorted list of ENTRIES. +* BRANCH: Sorted list of [ START_KEY, CHILD_NODE_HASH ]. -* Since the key is hashed we lose the ability to sort the structure by other means. -* Since the bucket settings are fixed the shape and churn rate of the data structure does not - alter itself all that well to different sizes without altering the bucket settings. +In order to produce a balanced and hash consistent tree over an arbitrarily sorted list +we need to find a way to chunk this list in parts that are: -### The Chunking Problem +* An average target size. +* Consistently chunked. This means that the same boundaries should be found in a newly created + tree as those found in a tree we are modifying. -Since we need a sorted structure that is stored in IPLD we'll end up with a merkle tree of some sort. +First, we guarantee that every entry is unique, which we can do with any arbitrary sorting function. +Then, we hash every entry and designate a portion of the address space in that hash to CLOSE each +chunk. This means that the same entries will always closed the structure and as we modify the tree +we will have very limited churn in the surrounding blocks. Since the hash randomizes the assignedment +of identifiers to each entry the structure will self-balance with new splits as entries are added +to any part of the tree. -In IPLD, the challenge we always face with tree structures is how to break large lists of nodes into -individually serialized blocks. We'll call this "the chunking problem." +That covers how the leaves are created. Branch creation is almost identical. Every branch is list of entries +where the START_KEY is ENTRY_KEY_MAP(ENTRY) of the child LEAF or simply the START_KEY of the child BRANCH, and the value +is the CHILD_NODE_HASH. The START_KEY + CHILD_NODE_HASH are hashed and the same chunking technique is applied +to branches as we apply to the leaves. -Take this simple sorted tree example: +## Settings -``` - root -+-------------------------------+ -| +-----------+ +-----------+ | -| | 1, block1 | | 4, block2 | | -| +-----------+ +-----------+ | -+-------------------------------+ - - block1 block2 -+-------------+ +-------------+ -| +---+ +---+ | | +---+ +---+ | -| | 1 | | 2 | | | | 4 | | 5 | | -| +---+ +---+ | | +---+ +---+ | -+-------------+ +-------------+ -``` - -Here we have a small and simple balanced tree using a chunking function that holds only 2 nodes per block. - -If we want to insert `3` into the structure we're confronted with several problems. If we append `block1` or -prepend to `block2` you'll end up with a different tree shape, and a different root hash, than if you created -the structure anew using the chunking function. - -**Rule: all sorted tree's must be consisently chunked in order to produce consistent tree structures which then -produce consistent hashes.** This means that hash consistent tree will, in effect, be self-balancing upon mutation. +Every tree needs the following settings in order to produce consistent and balanced trees. -If we stick with our "only 2 nodes per block" chunker then minor mutations to our list will produce mutations -to the entire tree on the right side of that mutation. This is obviously not ideal. +* SORT: defines the sort order of every ENTRY in the list. -What we need is a new chunking technique that produces nodes of a desirable length but also consistently splits -on particular **entries**. If we can find a way to consistently split on particular entries then we can avoid -large mutations to the rest of the tree. +Chunker Settings -## Chunker +* HASH_TAIL_SIZE: The size, in bytes, of the tail to be used for a calculating the close. +* HASH_TAIL_CLOSE: The highest integer that will close a chunk. +* MAX_ENTRIES: The maximum allowed number of entries in a single leaf. This protects against + insertion attacks when an attacker is allowed to define the entire ENTRY. -The chunker is a simple state machine. You feed the chunker unique identifiers for each entry. Each identifier must -use derived from the entry, typically through a hashing function or from a digest already present in the entry. +Leaf Chunker Settings -The chunker should break as consistently as possible while also protecting against various attacks (see below). +* ENTRY_BYTE_MAP: converts an ENTRY to ENTRY_BYTES. +* HASH_FN: the hashing function used on ENTRY_BYTES. +* ENTRY_KEY_MAP: takes an entry and returns the KEY defined by that ENTRY. -## First Tree: Sorted CID Set (Tail Chunker) +Branch Chunker Settings -```ipldschema -type Entry link -type Leaf [ Entry ] -type Branch struct { - start Link - link Link -} representation tuple +* KEY_BYTE_MAP: converts a START_KEY to KEY_BYTES. +* HASH_FN: the hashing function used on ENTRY_BYTES. -type Node union { - | "leaf" Leaf - | "branch" Branch -} representation keyed -``` - -At this point we're ready to build our first tree. This data structure is a `Set` of CIDs sorted by binary comparison. -The nice thing about working with this use case is each CID is both the *key* and the *value*. - -CID's end in a multihash, and the multihash typically ends in a hash digest. For this use case, we'll say that CID's that are allowed -in this `Set` are limited to those that use sufficiently secure hashing functions. +# Tree Creation -Since hash digests are, effectively, a form of randomization, we can simply convert the tail of each digest to an fixed size integer and designate -some part of that address space to cause splits in our chunker. +The following diagram uses ` -For simplicity, let's convert the last byte to Uint8. Now, let's make the chunker close every chunk when it sees a `0`. This will give -use nodes that have, on average, 256 entries. - -``` -Digest Uint8 Tails -+----------------+ -| DIGEST-A | 56 | -+----------------+ -+----------------+ -| DIGEST-B | 123 | -+----------------+ -+----------------+ -| DIGEST-D | 0 | -+----------------+ -// split -+----------------+ -| DIGEST-M | 123 | -+----------------+ -+----------------+ -| DIGEST-N | 56 | -+----------------+ -+----------------+ -| DIGEST-L | 113 | -+----------------+ -+----------------+ -| DIGEST-O | 6 | -+----------------+ -+----------------+ -| DIGEST-P | 45 | -+----------------+ ``` -Now, if we need to insert a new entry at `DIGEST-C` it will not effect the leaf nodes to the right, and as a result will have a limited -effect on the rest of the tree. - -Since every node in this tree will be a content addressed block so we can continue to use the hash digest of every branch and apply the same chunking -technique. Whenever an entry or branch is changed we need to re-run the chunker on it and merge the entries in every node with the node to the right if -they are no longer closed. This keeps the hash of the tree consistent regardless of what order any mutations were made and it also incrementally re-balances -the tree on each mutation. - -We can control the performance of this tree by altering the chunking algorithm. If we use more of the tail we'll have a larger number space and can -increase the average chunk size beyond 256. Larger blocks mean more orphaned data on mutation. Smaller blocks mean deeper trees with more traversal. - -## Second Tree: Sorted Map (Tail Chunker) - -Now we'll build a tree that can illustrate a few attacks against this structure. Understanding how you can attack this structure -plays a key role in designing secure trees for different use cases. - -Let's make the entries in our tree a key/value pair. Let's make the schema require that values be links so that we can rely on the hash digest -for chunking. - -```ipldschema -type Entry struct { - key String - value Link -} representation tuple - -type Leaf [ Entry ] -type Branch [ Entry ] - -type Node union { - | "leaf" Leaf - | "branch" Branch -} representation keyed ``` - -You can use the tail chunker on this tree if you have complete control over the keys and values being inserted. You'll need to ensure that -the values being inserted have something unique or random. If you don't have this assurance and you end up inserting the same value into -the same part of the tree the leaf block will never close and you'll eventually cause an error when you go over MAX_BLOCK_SIZE. - -### First Attack: Unclosed Leaf From 52e6762264eb9669ad465d9bcb32e48d557ba7c5 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Wed, 11 Nov 2020 10:39:06 -0800 Subject: [PATCH 05/10] wip: more work on chunker function --- design/sorted-tree.md | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index c73848da..7abefb27 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -60,10 +60,42 @@ Branch Chunker Settings * KEY_BYTE_MAP: converts a START_KEY to KEY_BYTES. * HASH_FN: the hashing function used on ENTRY_BYTES. +# Chunking Function + +The chunker converts the last HASH_TAIL_SIZE bytes to an integer. Any integer at or below +HASH_TAIL_CLOSE will terminate a chunk + +*Note: the following section in not complete and needs some testing and simulations to finalize +which is why some of the parameters are still loosely defined.* + +When untrusted users can insert ENTRIES into the structure it's vulnerable to an attack because +you can insert an unlimited number of entries that will never cause a close. To protect against +this the chunker requires a MAX_ENTRIES integer. + +If the chunker were to simply cut off at MAX_ENTRIES the attack would still be quite effective as +mutations in a particular section would all be of MAX_ENTRIES and mutations would cause a large +number of node merges in order to handle overflow. + +Instead, we should increase HASH_TAIL_CLOSE as we aproach MAX_ENTRIES. This will give +us some consistency to closing entries even when nodes overflow and will increase the difficulty of an attack +since an attacker will need to generate much more data to find hashes that fail to close since closes use more +of the address space. + +We'll need to run simulations in order to find the ideal technique for increasing HASH_TAIL_CLOSE and at what point +we should begin to apply it as we approach MAX_ENTRIES. A logorithmic scale may increase the hit rate too quickly which would +end up failing to match consistently enough, but an exponential scale may leave a little too much room for an attacker to +generate entries that won't close. + +We could also consider feeding some % of each integer into a randomized calculation that increases HASH_TAIL_CLOSE. This +would make it harder to produce entries you know will keep the structure open but it'll be hard to find the right math that +still produces consistent matches. + # Tree Creation -The following diagram uses ` +The following diagram uses `O` to represent entries that have a hash that keeps the chunk open and `C` for entries that +have a hash that closes a chunk. Every entry is unique even though many look identical in this diagram. ``` - ++---- +| ``` From a8361ea9ceb6e4db9d6792596227cdb0367d4db4 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Wed, 11 Nov 2020 15:08:01 -0800 Subject: [PATCH 06/10] wip: better chunker --- design/sorted-tree.md | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 7abefb27..a8a6204c 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -46,8 +46,6 @@ Chunker Settings * HASH_TAIL_SIZE: The size, in bytes, of the tail to be used for a calculating the close. * HASH_TAIL_CLOSE: The highest integer that will close a chunk. -* MAX_ENTRIES: The maximum allowed number of entries in a single leaf. This protects against - insertion attacks when an attacker is allowed to define the entire ENTRY. Leaf Chunker Settings @@ -69,26 +67,19 @@ HASH_TAIL_CLOSE will terminate a chunk which is why some of the parameters are still loosely defined.* When untrusted users can insert ENTRIES into the structure it's vulnerable to an attack because -you can insert an unlimited number of entries that will never cause a close. To protect against -this the chunker requires a MAX_ENTRIES integer. - -If the chunker were to simply cut off at MAX_ENTRIES the attack would still be quite effective as -mutations in a particular section would all be of MAX_ENTRIES and mutations would cause a large -number of node merges in order to handle overflow. - -Instead, we should increase HASH_TAIL_CLOSE as we aproach MAX_ENTRIES. This will give -us some consistency to closing entries even when nodes overflow and will increase the difficulty of an attack -since an attacker will need to generate much more data to find hashes that fail to close since closes use more -of the address space. - -We'll need to run simulations in order to find the ideal technique for increasing HASH_TAIL_CLOSE and at what point -we should begin to apply it as we approach MAX_ENTRIES. A logorithmic scale may increase the hit rate too quickly which would -end up failing to match consistently enough, but an exponential scale may leave a little too much room for an attacker to -generate entries that won't close. - -We could also consider feeding some % of each integer into a randomized calculation that increases HASH_TAIL_CLOSE. This -would make it harder to produce entries you know will keep the structure open but it'll be hard to find the right math that -still produces consistent matches. +you can insert an unlimited number of entries that will never cause a close. + +Hard limits on the number of entries is not effective because mutations to the left most leaf +will cause an overflow that generates subsequent mutations in every leaf to the right that is also +at its limit. Introducing entropy in HASH_TAIL_CLOSE has the same problem because it's too easy for an attacker to +generate entries that are at the highest boundary of the address space for keeping the chunk open. + +What we probably need to do is define a point at which we change the algorithm for closing the structure. If we +keep a floating fixed size list of previous hashes we can start generating a consistent "sequence identity" to use instead +of just the hash of the entry. As long as we keep the list size fixed we will tend to get consistent entries for the tail +and the number of hashing you would have to generate to cause an overflow will be far higher. We can *then* apply +a gradual increase in the HASH_TAIL_SIZE which will reduce the address space of a successful attack but still result +in fairly consistent break points. # Tree Creation From 0778d01901f5dc6b781bef4be6aebb3adb8d786f Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Thu, 12 Nov 2020 15:04:16 -0800 Subject: [PATCH 07/10] wip: overflow protection --- design/sorted-tree.md | 92 +++++++++++++++++++++++++++++++++---------- 1 file changed, 71 insertions(+), 21 deletions(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index a8a6204c..99ecafd3 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -44,8 +44,7 @@ Every tree needs the following settings in order to produce consistent and balan Chunker Settings -* HASH_TAIL_SIZE: The size, in bytes, of the tail to be used for a calculating the close. -* HASH_TAIL_CLOSE: The highest integer that will close a chunk. +* TARGET_SIZE: The size, in bytes, of the tail to be used for a calculating the close. Leaf Chunker Settings @@ -60,26 +59,77 @@ Branch Chunker Settings # Chunking Function -The chunker converts the last HASH_TAIL_SIZE bytes to an integer. Any integer at or below -HASH_TAIL_CLOSE will terminate a chunk - -*Note: the following section in not complete and needs some testing and simulations to finalize -which is why some of the parameters are still loosely defined.* - -When untrusted users can insert ENTRIES into the structure it's vulnerable to an attack because -you can insert an unlimited number of entries that will never cause a close. - -Hard limits on the number of entries is not effective because mutations to the left most leaf -will cause an overflow that generates subsequent mutations in every leaf to the right that is also -at its limit. Introducing entropy in HASH_TAIL_CLOSE has the same problem because it's too easy for an attacker to -generate entries that are at the highest boundary of the address space for keeping the chunk open. +First, we divide `MAX_UINT32` by `TARGET_SIZE` and `FLOOR()` it to give us our +`THRESHOLD` integer. Now, randomly generated numbers will occur an average of 1 +every `TARGET_SIZE`. + +The tail of each `HASH(ENTRY_BYTE_MAP(ENTRY))` is converted to Uint32. If that +integer is at or below `THRESHOLD` then the entry causes the chunk to close. + +## Overflow Protection + +When a chunk reaches twice the target size we start applying overflow protection. + +It's not very hard to generate entries that will fail to close a chunk under our +prior rules. An attack can be crafted to overflow a particular leaf until it +reaches `MAX_BLOCK_SIZE`. This overflow can't be resolved by simply setting a hard +limit because an attacker can continue to overflow a node and every mutation will +overflow into the next leaf and cause recursive mutations in as many nodes as the attacker +can insert. The only way to overcome this problem is to find a way to still find +common break points at a reasonable probability when in an overflow state. + +In overflow protection we still close on the same entries as we would otherwise but +we also compute a *`SEQUENCE_IDENTITY`*. This is calculated by applying the hash +function to the prior `TARGET_SIZE` number of hashes and the current hash. This gives us a +unique identifier for each entry based on its placement in the back half(ish) of the +list, which is important because the identity needs to remain consistent even when data +overflows into the left side of the list. We convert the tail +of this new hash (`SEQUENCE_IDENTITY`) to a Uint32 and compare it against the OVERFLOW_LIMIT. + +Our OVERFLOW_THESHOLD is an integer that increases an equal amount from 0 on every +entry until it reaches MAX_UINT32. The increase in OVERFLOW_LIMIT on each entry +is our existing `THRESHOLD` which ensures an absolute maximum size of a leaf is 3x the `TARGET_SIZE`. + +This makes generating sequential data that will keep the chunk open highly difficult +given a sufficient TARGET_SIZE and still produces relatively (although not completely) +consistent chunk splits in overflow protection. + +```js +const createChunker = (TARGET_SIZE) => { + // a new chunker must be created after each split in order + // to reset the state + const THRESHOLD = FLOOR(MAX_UINT32 / TARGET_SIZE) + const RECENT_HASHES = [] + let COUNT = 0 + let OVERFLOW_THRESHOLD = 0 + const chunker = (ENTRY) => { + COUNT += 1 + const ENTRY_HASH = HASH(ENTRY) + const IDENTITY = UINT32(TAIL(4, ENTRY_HASH)) + if (IDENTITY < THRESHOLD) return true + + PUSH(RECENT_HASHES, ENTRY_HASH) + if (RECENT_HASHES.length > TARGET_SIZE) { + SHIFT(RECENT_HASHES, 1) + } + if (COUNT > (TARGET_SIZE * 2)) { + OVERFLOW_THRESHOLD += THRESHOLD + // overflow protection + const SEQUENCE_IDENTITY = UINT32(TAIL(4, HASH(JOIN(RECENT_HASHES)))) + if (SEQUENCE_IDENTITY < OVERFLOW_THRESHOLD) { + return true + } + } + return false + } + return chunker +} +``` -What we probably need to do is define a point at which we change the algorithm for closing the structure. If we -keep a floating fixed size list of previous hashes we can start generating a consistent "sequence identity" to use instead -of just the hash of the entry. As long as we keep the list size fixed we will tend to get consistent entries for the tail -and the number of hashing you would have to generate to cause an overflow will be far higher. We can *then* apply -a gradual increase in the HASH_TAIL_SIZE which will reduce the address space of a successful attack but still result -in fairly consistent break points. +Any state that is built up in the chunker must be cleared on every chunk split. It's +very important that entries after a split do not effect the identities of prior entries +because if they did we wouldn't be able to safely split and merge with on mutation without +re-chunking every entry in the tree. # Tree Creation From a303ac0ba9d970ce328efdc5d692ae65c40b0224 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Thu, 12 Nov 2020 15:17:01 -0800 Subject: [PATCH 08/10] wip: edits --- design/sorted-tree.md | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 99ecafd3..5e8b008b 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -5,7 +5,7 @@ This document describes a technique for creating, mutating, and reading merkle D * Provide consistently performant random access. * Self-balance on mutation. * Every branch is a concurrency vector that can be safely mutated (lock-free thread safety). -* Are sorted by an arbitrary sort function. +* Are sorted by any user provided sort function. * Can contain entries with any key and/or value type as long as each entry is unique. (Note: this is not a *unique key constraint* as the entry is potentially a unique pairing of key **and value**). @@ -18,20 +18,20 @@ Terms: * BRANCH: Sorted list of [ START_KEY, CHILD_NODE_HASH ]. In order to produce a balanced and hash consistent tree over an arbitrarily sorted list -we need to find a way to chunk this list in parts that are: +we need to find a way to chunk this list into parts that are: * An average target size. -* Consistently chunked. This means that the same boundaries should be found in a newly created +* Fully consistent. This means that the same boundaries should be found in a newly created tree as those found in a tree we are modifying. First, we guarantee that every entry is unique, which we can do with any arbitrary sorting function. Then, we hash every entry and designate a portion of the address space in that hash to CLOSE each -chunk. This means that the same entries will always closed the structure and as we modify the tree -we will have very limited churn in the surrounding blocks. Since the hash randomizes the assignedment +chunk. This means that the same entries will always close the structure and as we modify the tree +we will have very limited churn in the surrounding blocks. Since the hash randomizes the assignment of identifiers to each entry the structure will self-balance with new splits as entries are added to any part of the tree. -That covers how the leaves are created. Branch creation is almost identical. Every branch is list of entries +That covers how the leaves are created. Branch creation is almost identical. Every branch is a list of entries where the START_KEY is ENTRY_KEY_MAP(ENTRY) of the child LEAF or simply the START_KEY of the child BRANCH, and the value is the CHILD_NODE_HASH. The START_KEY + CHILD_NODE_HASH are hashed and the same chunking technique is applied to branches as we apply to the leaves. @@ -44,18 +44,19 @@ Every tree needs the following settings in order to produce consistent and balan Chunker Settings -* TARGET_SIZE: The size, in bytes, of the tail to be used for a calculating the close. +* TARGET_SIZE: The target number of entries per NODE. Leaf Chunker Settings -* ENTRY_BYTE_MAP: converts an ENTRY to ENTRY_BYTES. -* HASH_FN: the hashing function used on ENTRY_BYTES. -* ENTRY_KEY_MAP: takes an entry and returns the KEY defined by that ENTRY. +* `ENTRY_BYTE_MAP`: converts an `ENTRY` to `ENTRY_BYTES`. +* `HASH_FN`: the hashing function used on `ENTRY_BYTES`. +* `ENTRY_KEY_MAP`: takes an entry and returns the `KEY` defined by that `ENTRY`. (This + is used to create a `BRANCH` in order to reference the first `KEY` of a `LEAF`) Branch Chunker Settings -* KEY_BYTE_MAP: converts a START_KEY to KEY_BYTES. -* HASH_FN: the hashing function used on ENTRY_BYTES. +* `KEY_BYTE_MAP`: converts a `START_KEY` to `KEY_BYTES`. +* `HASH_FN`: the hashing function used on `ENTRY_BYTES`. # Chunking Function @@ -84,14 +85,14 @@ function to the prior `TARGET_SIZE` number of hashes and the current hash. This unique identifier for each entry based on its placement in the back half(ish) of the list, which is important because the identity needs to remain consistent even when data overflows into the left side of the list. We convert the tail -of this new hash (`SEQUENCE_IDENTITY`) to a Uint32 and compare it against the OVERFLOW_LIMIT. +of this new hash (`SEQUENCE_IDENTITY`) to a Uint32 and compare it against the `OVERFLOW_LIMIT`. -Our OVERFLOW_THESHOLD is an integer that increases an equal amount from 0 on every -entry until it reaches MAX_UINT32. The increase in OVERFLOW_LIMIT on each entry +Our `OVERFLOW_THRESHOLD` is an integer that increases an equal amount from 0 on every +entry until it reaches `MAX_UINT32`. The increase in `OVERFLOW_LIMIT` on each entry is our existing `THRESHOLD` which ensures an absolute maximum size of a leaf is 3x the `TARGET_SIZE`. This makes generating sequential data that will keep the chunk open highly difficult -given a sufficient TARGET_SIZE and still produces relatively (although not completely) +given a sufficient `TARGET_SIZE` and still produces relatively (although not completely) consistent chunk splits in overflow protection. ```js From e6956c53d39053a2a0b7476de6636f7225372007 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Thu, 12 Nov 2020 15:18:41 -0800 Subject: [PATCH 09/10] wip: more edits --- design/sorted-tree.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 5e8b008b..3d9a4232 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -13,9 +13,9 @@ This document describes a technique for creating, mutating, and reading merkle D Terms: -* NODE: LEAF or BRANCH. -* LEAF: Sorted list of ENTRIES. -* BRANCH: Sorted list of [ START_KEY, CHILD_NODE_HASH ]. +* `NODE`: `LEAF` or `BRANCH`. +* `LEAF`: Sorted list of `ENTRIES`. +* `BRANCH`: Sorted list of `[ START_KEY, CHILD_NODE_HASH ]`. In order to produce a balanced and hash consistent tree over an arbitrarily sorted list we need to find a way to chunk this list into parts that are: @@ -25,26 +25,26 @@ we need to find a way to chunk this list into parts that are: tree as those found in a tree we are modifying. First, we guarantee that every entry is unique, which we can do with any arbitrary sorting function. -Then, we hash every entry and designate a portion of the address space in that hash to CLOSE each +Then, we hash every entry and designate a portion of the address space in that hash to close each chunk. This means that the same entries will always close the structure and as we modify the tree we will have very limited churn in the surrounding blocks. Since the hash randomizes the assignment of identifiers to each entry the structure will self-balance with new splits as entries are added to any part of the tree. That covers how the leaves are created. Branch creation is almost identical. Every branch is a list of entries -where the START_KEY is ENTRY_KEY_MAP(ENTRY) of the child LEAF or simply the START_KEY of the child BRANCH, and the value -is the CHILD_NODE_HASH. The START_KEY + CHILD_NODE_HASH are hashed and the same chunking technique is applied +where the `START_KEY` is `ENTRY_KEY_MAP(ENTRY)` of the child `LEAF` or simply the `START_KEY` of the child `BRANCH`, and the value +is the `CHILD_NODE_HASH`. The `START_KEY + CHILD_NODE_HASH` are hashed and the same chunking technique is applied to branches as we apply to the leaves. ## Settings Every tree needs the following settings in order to produce consistent and balanced trees. -* SORT: defines the sort order of every ENTRY in the list. +* `SORT`: defines the sort order of every `ENTRY` in the list. Chunker Settings -* TARGET_SIZE: The target number of entries per NODE. +* `TARGET_SIZE`: The target number of entries per `NODE`. Leaf Chunker Settings From 94fdef2742332db6b7b1261d5ac38b57a8d479b9 Mon Sep 17 00:00:00 2001 From: Mikeal Rogers Date: Thu, 12 Nov 2020 15:26:05 -0800 Subject: [PATCH 10/10] wip: more text --- design/sorted-tree.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/design/sorted-tree.md b/design/sorted-tree.md index 3d9a4232..680fc0cd 100644 --- a/design/sorted-tree.md +++ b/design/sorted-tree.md @@ -24,6 +24,20 @@ we need to find a way to chunk this list into parts that are: * Fully consistent. This means that the same boundaries should be found in a newly created tree as those found in a tree we are modifying. +In order to make the tree efficient we need to find a way for the chunks to be split on consistent +boundaries even when the chunks are modified. This means you can use a typical chunker because you'd +have to run the chunker over the entire right side of the list on every mutation in order to produce +a consistent tree shape. + +We do this by giving each entry a unique `IDENTITY` (`UINT32`) using a hashing function. This distributes +randomly distributed unique and consistent identifiers to every entry in the list. If we set a threshold +for which numeric identity we consider a `CLOSE` we can calculate the average occurance of those numbers +which gives us an average distribution of `CLOSES` throughout the list. Entries that are added or removed +will cause splits and merges that balance the tree and ensure a consistent shape which will guarantee hash +consistency for the entire tree. + +Here's how it works. + First, we guarantee that every entry is unique, which we can do with any arbitrary sorting function. Then, we hash every entry and designate a portion of the address space in that hash to close each chunk. This means that the same entries will always close the structure and as we modify the tree