Code

Splitstore Enhanchements#6474

Code

Merged

magik6k merged 197 commits intomasterfilecoin-project/lotus:masterfrom feat/splitstore-reduxfilecoin-project/lotus:feat/splitstore-reduxCopy head branch name to clipboardJul 13, 2021

+2,058-1,131

Commits

Commits on Jul 4, 2021

• noop blockstore

  vyzo

  committed
  4d3c73f

  Copy full SHA for 4d3c73f

• hook noop blockstore for splitstore in DI

  vyzo

  committed
  5cca29d

  Copy full SHA for 5cca29d

• kill full splitstore compaction, simplify splitstore configuration

  vyzo

  committed
  04f2e10

  Copy full SHA for 04f2e10

• implement chain walking

  vyzo

  committed
  e3cbeec

  Copy full SHA for e3cbeec

• decrease CompactionThreshold to 3 finalities

  vyzo

  committed
  d7ceef1

  Copy full SHA for d7ceef1

• mark from current epoch to boundary epoch when necessary

  Show description for 3a9b7c5
  vyzo

  committed
  3a9b7c5

  Copy full SHA for 3a9b7c5

• metrics: increment misses in View().

  

  raulk

  authored and

  vyzo

  committed
  b2b7eb2

  Copy full SHA for b2b7eb2

• don't open bolt tracking store with NoSync, it might get corrupted

  vyzo

  committed
  e9f531b

  Copy full SHA for e9f531b

• keep genesis-linked state hot

  vyzo

  committed
  7cf75e6

  Copy full SHA for 7cf75e6

• more robust handling of sync gap walks

  vyzo

  committed
  bdb97d6

  Copy full SHA for bdb97d6

• first visit the cid, then short-circuit non dagcbor objects

  vyzo

  committed
  d33a44e

  Copy full SHA for d33a44e

• fix test

  vyzo

  committed
  fda291b

  Copy full SHA for fda291b

• reduce SyncGapTime to 1 minute

  Show description for fa64814
  vyzo

  committed
  fa64814

  Copy full SHA for fa64814

• also walk parent message receipts when including messages in the walk

  vyzo

  committed
  41573f1

  Copy full SHA for 41573f1

• refactor genesis state loading code into its own method

  vyzo

  committed
  7c814cd

  Copy full SHA for 7c814cd

• keep headers hot when running with a noop splitstore

  vyzo

  committed
  997f2c0

  Copy full SHA for 997f2c0

• don't try to visit genesis parent blocks

  vyzo

  committed
  7b02673

  Copy full SHA for 7b02673

• don't attempt compaction while still syncing

  vyzo

  committed
  3fe4261

  Copy full SHA for 3fe4261

• refactor warmup to trigger at startup and not wait for sync

  vyzo

  committed
  9b64485

  Copy full SHA for 9b64485

• save the warm up epoch only if successful in warming up

  vyzo

  committed
  421f05e

  Copy full SHA for 421f05e

• track writeEpoch relative to current wall clock time

  Show description for bb17608
  vyzo

  committed
  bb17608

  Copy full SHA for bb17608

• fix lint issue

  vyzo

  committed
  66f1630

  Copy full SHA for 66f1630

• update write epoch in the background every second

  vyzo

  committed
  933c786

  Copy full SHA for 933c786

• augment current epoch by +1

  Show description for b789759
  vyzo

  committed
  b789759

  Copy full SHA for b789759

• coalesce back-to-back compactions

  Show description for c4d95de
  vyzo

  committed
  c4d95de

  Copy full SHA for c4d95de

• fix test

  vyzo

  committed
  a178c1f

  Copy full SHA for a178c1f

• reintroduce compaction slack

  vyzo

  committed
  a25ac80

  Copy full SHA for a25ac80

• fix test

  vyzo

  committed
  79d2148

  Copy full SHA for 79d2148

• CompactionThreshold should be 4 finalities

  Show description for a21f559
  vyzo

  committed
  a21f559

  Copy full SHA for a21f559

• adjust compaction range

  vyzo

  committed
  30dbe49

  Copy full SHA for 30dbe49

• always do full walks, not only when there is a sync gap

  vyzo

  committed
  0390285

  Copy full SHA for 0390285

• add debug log skeleton

  vyzo

  committed
  fc247e4

  Copy full SHA for fc247e4

• flush move log when cold collection is done

  vyzo

  committed
  fce7b8d

  Copy full SHA for fce7b8d

• implement debug log

  vyzo

  committed
  a53c4e1

  Copy full SHA for a53c4e1

• fix lint

  vyzo

  committed
  b187b5c

  Copy full SHA for b187b5c

• reset counters after flush

  vyzo

  committed
  375a179

  Copy full SHA for 375a179

• don't log read misses before warmup

  vyzo

  committed
  50ebaf2

  Copy full SHA for 50ebaf2

• add config option for hot headers

  vyzo

  committed
  649b7dd

  Copy full SHA for 649b7dd

• fix transactional race during compaction

  Show description for cb665d0
  vyzo

  committed
  cb665d0

  Copy full SHA for cb665d0

• minor tweaks in purge

  Show description for 65ccc99
  vyzo

  committed
  65ccc99

  Copy full SHA for 65ccc99

• use a map for txn protection mark set

  vyzo

  committed
  6af3a23

  Copy full SHA for 6af3a23

• use internal get during walk to avoid blowing the compaction txn

  Show description for 31497f4
  vyzo

  committed
  31497f4

  Copy full SHA for 31497f4

• move code around for better readability

  vyzo

  committed
  4a71c68

  Copy full SHA for 4a71c68

• fix error check for unreachable cids

  vyzo

  committed
  9fda61a

  Copy full SHA for 9fda61a

• log put errors in splitstore log

  vyzo

  committed
  40ff5bf

  Copy full SHA for 40ff5bf

• better log message

  vyzo

  committed
  7ebef6d

  Copy full SHA for 7ebef6d

• deduplicate stack logs and optionally trace write stacks

  vyzo

  committed
  dec61fa

  Copy full SHA for dec61fa

• fix index out of range

  vyzo

  committed
  0b315e9

  Copy full SHA for 0b315e9

• fix debug panic

  vyzo

  committed
  b2b13bb

  Copy full SHA for b2b13bb

• use succint timetamp in debug logs

  vyzo

  committed
  57e25ae

  Copy full SHA for 57e25ae

• cache stack repr computation

  vyzo

  committed
  7307eb5

  Copy full SHA for 7307eb5

• fix broken purge count log

  vyzo

  committed
  4bed316

  Copy full SHA for 4bed316

• check both markset and txn liveset before declaring an object cold

  vyzo

  committed
  e29b64c

  Copy full SHA for e29b64c

• count txn live objects explicitly for logging

  vyzo

  committed
  7de0771

  Copy full SHA for 7de0771

• check for lookback references to block headers in walk

  vyzo

  committed
  09efed5

  Copy full SHA for 09efed5

• walk tweaks

  vyzo

  committed
  40f42db

  Copy full SHA for 40f42db

• better logging for chain walk

  vyzo

  committed
  90dc274

  Copy full SHA for 90dc274

• store the hash in map markset

  vyzo

  committed
  f97535d

  Copy full SHA for f97535d

• treat Has as an implicit Write

  Show description for 6a3cbea
  vyzo

  committed
  6a3cbea

  Copy full SHA for 6a3cbea

• add missing return

  vyzo

  committed
  e472cac

  Copy full SHA for e472cac

• batch implicit write tracking

  Show description for be6cc2c
  vyzo

  committed
  be6cc2c

  Copy full SHA for be6cc2c

• flush implicit writes in all paths in updateWriteEpoch

  vyzo

  committed
  a29947d

  Copy full SHA for a29947d

• flush implicit writes before starting compaction

  vyzo

  committed
  7f473f5

  Copy full SHA for 7f473f5

• flush implicit writes at the right time before starting compaction to avoid races

  vyzo

  committed
  d0bfe42

  Copy full SHA for d0bfe42

• track all writes using async batching, not just implicit ones

  vyzo

  committed
  3e8e927

  Copy full SHA for 3e8e927

• move comments about tracking perf issues into a more pertinent place

  vyzo

  committed
  aeaa59d

  Copy full SHA for aeaa59d

• debug log writes at track so that we get correct stack traces

  vyzo

  committed
  2faa4aa

  Copy full SHA for 2faa4aa

• fix panic at startup

  Show description for b3ddaa5
  vyzo

  committed
  b3ddaa5

  Copy full SHA for b3ddaa5

• move write log back to flush so that we don't crawl to a halt

  vyzo

  committed
  4de0cd9

  Copy full SHA for 4de0cd9

• transitively track dags from implicit writes in Has

  vyzo

  committed
  9828673

  Copy full SHA for 9828673

• add pending write check before tracking the object in Has

  vyzo

  committed
  13a6743

  Copy full SHA for 13a6743

• do the dag walk for deep write tracking during flush

  Show description for a98a062
  vyzo

  committed
  a98a062

  Copy full SHA for a98a062

• refactor txn reference tracking, do deep marking of DAGs

  vyzo

  committed
  bd92c23

  Copy full SHA for bd92c23

• first write, then track

  vyzo

  committed
  4071488

  Copy full SHA for 4071488

• downgrade a couple of logs to warnings

  vyzo

  committed
  da00fc6

  Copy full SHA for da00fc6

• optimize transitive write tracking a bit

  vyzo

  committed
  1d41e15

  Copy full SHA for 1d41e15

• reused cidset across all walks when flushing pending writes

  vyzo

  committed
  484dfae

  Copy full SHA for 484dfae

• avoid clown shoes: only walk links for tracking in implicit writes/refs

  vyzo

  committed
  9d6bcd7

  Copy full SHA for 9d6bcd7

• fix faulty if/else logic for implicit txn protection

  vyzo

  committed
  637fbf6

  Copy full SHA for 637fbf6

• bubble up dependent txn ref errors

  Show description for b87295d
  vyzo

  committed
  b87295d

  Copy full SHA for b87295d

• skip moving cold blocks when running with a noop coldstore

  Show description for 68bc5d2
  vyzo

  committed
  68bc5d2

  Copy full SHA for 68bc5d2

• fix some residual purge races

  vyzo

  committed
  e4bb4be

  Copy full SHA for e4bb4be

• create the transactional protect filter before walking

  vyzo

  committed
  5834231

  Copy full SHA for 5834231

• use a single map for tracking pending writes, properly track implicits

  vyzo

  committed
  39723bb

  Copy full SHA for 39723bb

• only treat Has as an implicit write within vm.Copy context

  vyzo

  committed
  736d6a3

  Copy full SHA for 736d6a3

• short-circuit marking walks when encountering a block and more efficient walking

  vyzo

  committed
  8157f88

  Copy full SHA for 8157f88

• if it's not a dag, it's not a block

  vyzo

  committed
  9d6cabd

  Copy full SHA for 9d6cabd

• rework tracking logic; do it lazily and far more efficiently

  vyzo

  committed
  228a435

  Copy full SHA for 228a435

• remove dead code

  vyzo

  committed
  184d380

  Copy full SHA for 184d380

• fix log message

  vyzo

  committed
  2b03316

  Copy full SHA for 2b03316

• remove vm copy context detection hack

  Show description for 6f58fdc
  vyzo

  committed
  6f58fdc

  Copy full SHA for 6f58fdc

• more accurate stats about mark set updates

  vyzo

  committed
  d79e4da

  Copy full SHA for d79e4da

• remove unnecessary code

  vyzo

  committed
  c5cf8e2

  Copy full SHA for c5cf8e2

• deal with memory pressure, don't walk under the boundary

  vyzo

  committed
  642f0e4

  Copy full SHA for 642f0e4

• add staging cache to bolt tracking store

  vyzo

  committed
  00fcf6d

  Copy full SHA for 00fcf6d

• fix bug that turned candidate filtering to dead code

  vyzo

  committed
  68a8350

  Copy full SHA for 68a8350

• BlockstoreIterator trait with implementation for badger

  vyzo

  committed
  d476a3d

  Copy full SHA for d476a3d

• RIP tracking store

  vyzo

  committed
  1f2b604

  Copy full SHA for 1f2b604

• update splistore DI constructor

  vyzo

  committed
  5f7ae1f

  Copy full SHA for 5f7ae1f

• simplify compaction model

  vyzo

  committed
  6fa2cd2

  Copy full SHA for 6fa2cd2

• fix panic from concurrent map writes in txnRefs

  vyzo

  committed
  36f9364

  Copy full SHA for 36f9364

• more efficient trackTxnRefMany

  vyzo

  committed
  eafffc1

  Copy full SHA for eafffc1

• reuse key buffer in badger ForEachKey

  Show description for 08cad30
  vyzo

  committed
  08cad30

  Copy full SHA for 08cad30

• fix log

  vyzo

  committed
  0a1d7b3

  Copy full SHA for 0a1d7b3

• deal with partially written objects

  vyzo

  committed
  19d1b1f

  Copy full SHA for 19d1b1f

• housekeeping

  Show description for 190cb18
  vyzo

  committed
  190cb18

  Copy full SHA for 190cb18

• fix test

  vyzo

  committed
  95c3aae

  Copy full SHA for 95c3aae

• walkChain should visit the genesis state root

  vyzo

  committed
  8e56fff

  Copy full SHA for 8e56fff

• make test do something useful again

  vyzo

  committed
  028a5c4

  Copy full SHA for 028a5c4

• short-circuit rescanning on block headers

  vyzo

  committed
  2c7a89a

  Copy full SHA for 2c7a89a

• fix lint

  vyzo

  committed
  1f02428

  Copy full SHA for 1f02428

• use deep object walking for more robust handling of transactional references

  vyzo

  committed
  680af8e

  Copy full SHA for 680af8e

• fix error message

  vyzo

  committed
  4d286da

  Copy full SHA for 4d286da

• recursively protect all references

  vyzo

  committed
  f124389

  Copy full SHA for f124389

• smarter trackTxnRefMany

  vyzo

  committed
  13d612f

  Copy full SHA for 13d612f

• sort cold objects before deleting

  Show description for 40c271c
  vyzo

  committed
  40c271c

  Copy full SHA for 40c271c

• simplify transactional protection logic

  Show description for f33d4e7
  vyzo

  committed
  f33d4e7

  Copy full SHA for f33d4e7

• reduce length of critical section

  Show description for 94efae4
  vyzo

  committed
  94efae4

  Copy full SHA for 94efae4

• fix lint

  vyzo

  committed
  b08e0b7

  Copy full SHA for b08e0b7

• reduce CompactionThreshold to 5 finalities

  Show description for db53859
  vyzo

  committed
  db53859

  Copy full SHA for db53859

Commits on Jul 5, 2021

• deal with incomplete objects that need to be marked and protected

  Show description for 1726eb9
  vyzo

  committed
  1726eb9

  Copy full SHA for 1726eb9

• remove the sleeps and busy loop more times when waiting for missing objects

  vyzo

  committed
  3597192

  Copy full SHA for 3597192

• add warning for missing objects for marking for debug purposes

  vyzo

  committed
  4c41f52

  Copy full SHA for 4c41f52

• add some comments about the missing business and anothre log

  vyzo

  committed
  c81ae5f

  Copy full SHA for c81ae5f

• only occur check for DAGs

  vyzo

  committed
  839f7bd

  Copy full SHA for 839f7bd

• short-circuit fil commitments

  Show description for 2ea2abc
  vyzo

  committed
  2ea2abc

  Copy full SHA for 2ea2abc

• a bit more fil commitment short-circuiting

  vyzo

  committed
  918a7ec

  Copy full SHA for 918a7ec

• improve logs and error messages

  vyzo

  committed
  3ec834b

  Copy full SHA for 3ec834b

• reduce memory pressure from marksets when the size is decreased

  vyzo

  committed
  d7709de

  Copy full SHA for d7709de

• readd minute delay before trying for missing objects

  vyzo

  committed
  d8b8d75

  Copy full SHA for d8b8d75

• use internal version of has for occurs checks

  vyzo

  committed
  0b7153b

  Copy full SHA for 0b7153b

• fix log

  vyzo

  committed
  59936ef

  Copy full SHA for 59936ef

• get rid of ugly missing reference handling code

  Show description for fa195be
  vyzo

  committed
  fa195be

  Copy full SHA for fa195be

• reinstate some better code for handling missing references.

  vyzo

  committed
  59639a0

  Copy full SHA for 59639a0

• more commentary on the missing refs situation

  vyzo

  committed
  5a099b7

  Copy full SHA for 5a099b7

• handle all missing refs together

  Show description for af8cf71
  vyzo

  committed
  af8cf71

  Copy full SHA for af8cf71

• dont needlessly wait 1 min in first retry for missing refs

  vyzo

  committed
  73d0799

  Copy full SHA for 73d0799

• unify the two marksets

  Show description for 3477d26
  vyzo

  committed
  3477d26

  Copy full SHA for 3477d26

• code cleanup: refactor txn state code into their own functions

  vyzo

  committed
  e859942

  Copy full SHA for e859942

• make endTxnProtect idempotent

  vyzo

  committed
  bd436ab

  Copy full SHA for bd436ab

• quiet linter

  Show description for 51ab891
  vyzo

  committed
  51ab891

  Copy full SHA for 51ab891

• make sure to nil everything in txnEndProtect

  vyzo

  committed
  2cbd3fa

  Copy full SHA for 2cbd3fa

• use walkObjectRaw for computing object weights

  Show description for c6ad8fd
  vyzo

  committed
  c6ad8fd

  Copy full SHA for c6ad8fd

• use hashes as keys in weight map to avoid duplicate work

  Show description for 525a2c7
  vyzo

  committed
  525a2c7

  Copy full SHA for 525a2c7

Commits on Jul 6, 2021

• cache cid strings in sort

  Show description for 0659235
  vyzo

  committed
  0659235

  Copy full SHA for 0659235

• optimize sort a tad

  Show description for bf7aeb3
  vyzo

  committed
  bf7aeb3

  Copy full SHA for bf7aeb3

• short-circuit block headers on sort weight computation

  vyzo

  committed
  55a9e0c

  Copy full SHA for 55a9e0c

• really optimize computing object weights

  Show description for 169ab26
  vyzo

  committed
  169ab26

  Copy full SHA for 169ab26

• minor tweak

  vyzo

  committed
  c4ae3e0

  Copy full SHA for c4ae3e0

• add some comments for debug only code

  vyzo

  committed
  dc8139a

  Copy full SHA for dc8139a

• remove unused GetGenesis method from ChainAccessor interface

  vyzo

  committed
  5c51450

  Copy full SHA for 5c51450

• move map markset implementation to its own file

  vyzo

  committed
  fdff1be

  Copy full SHA for fdff1be

• improve comments

  vyzo

  committed
  c1c2586

  Copy full SHA for c1c2586

• clean up: simplify debug log, get rid of ugly debug log

  vyzo

  committed
  f2f4af6

  Copy full SHA for f2f4af6

• prepare the transaction before launching the compaction goroutine

  vyzo

  committed
  0e2af11

  Copy full SHA for 0e2af11

• transactional protect incoming tipsets

  vyzo

  committed
  90da622

  Copy full SHA for 90da622

Commits on Jul 7, 2021

• rename som Txn methods for better readability

  vyzo

  committed
  05dbbe9

  Copy full SHA for 05dbbe9

• remove the curTs state variable; we don't need it

  vyzo

  committed
  6cc2112

  Copy full SHA for 6cc2112

• protect assignment of warmup epoch with the mutex

  vyzo

  committed
  83c30dc

  Copy full SHA for 83c30dc

• don't leak tracking errors through the API

  vyzo

  committed
  9dbb2e0

  Copy full SHA for 9dbb2e0

• RIP bbolt-backed markset

  vyzo

  committed
  451ddf5

  Copy full SHA for 451ddf5

• remove bbolt dependency from go.mod

  vyzo

  committed
  ec586a8

  Copy full SHA for ec586a8

• nil map/bf on markset close

  vyzo

  committed
  aec2ba2

  Copy full SHA for aec2ba2

• don't nil the mark sets on close, it's dangerous.

  Show description for c6421f8
  vyzo

  committed
  c6421f8

  Copy full SHA for c6421f8

• check the closing state on each batch during the purge.

  vyzo

  committed
  fee50b1

  Copy full SHA for fee50b1

• fix lint

  vyzo

  committed
  4f80836

  Copy full SHA for 4f80836

Commits on Jul 8, 2021

• check the closing state variable often

  Show description for f5c45bd
  vyzo

  committed
  f5c45bd

  Copy full SHA for f5c45bd

• intelligently close marksets and signal errors in concurrent operations

  vyzo

  committed
  48f13a4

  Copy full SHA for 48f13a4

• use RW mutexes in marksets

  vyzo

  committed
  e6eacbd

  Copy full SHA for e6eacbd

• add README for documentation

  vyzo

  committed
  9aa4f3b

  Copy full SHA for 9aa4f3b

• README: add instructions for how to enable

  vyzo

  committed
  5cf1e09

  Copy full SHA for 5cf1e09

• move check for closure in walkChain

  Show description for 00d7772
  vyzo

  committed
  00d7772

  Copy full SHA for 00d7772

• fix typo

  Show description for fa30ac8
  

  vyzo

  and

  magik6k

  authored
  fa30ac8

  Copy full SHA for fa30ac8

• fix typo

  Show description for c053784
  

  vyzo

  and

  magik6k

  authored
  c053784

  Copy full SHA for c053784

• fix potential deadlock in View

  Show description for 60dd97c
  vyzo

  committed
  60dd97c

  Copy full SHA for 60dd97c

• add environment variables to turn on the debug log without recompiling

  vyzo

  committed
  b661112

  Copy full SHA for b661112

Commits on Jul 9, 2021

• explicitly switch marksets for concurrent marking

  Show description for abdf4a1
  vyzo

  committed
  abdf4a1

  Copy full SHA for abdf4a1

• make badger Close-safe

  vyzo

  committed
  909f703

  Copy full SHA for 909f703

• correctly handle identity cids

  vyzo

  committed
  de5e21b

  Copy full SHA for de5e21b

• kill isOldBlockHeader; it's dangerous.

  vyzo

  committed
  4f89d26

  Copy full SHA for 4f89d26

• fix test

  vyzo

  committed
  565faff

  Copy full SHA for 565faff

• properly handle protecting long-running views

  vyzo

  committed
  acc4c37

  Copy full SHA for acc4c37

• dont mark references inline; instad rely on the main compaction thread to do concurrent marking

  Show description for da0feb3
  vyzo

  committed
  da0feb3

  Copy full SHA for da0feb3

• make view protection optimistic again, as there is a race window

  vyzo

  committed
  095d742

  Copy full SHA for 095d742

• remove unused lookback constructs

  vyzo

  committed
  18161fe

  Copy full SHA for 18161fe

• rename noopstore to discardstore

  vyzo

  committed
  c0a1cff

  Copy full SHA for c0a1cff

• update wording around discard store

  vyzo

  committed
  b9a5ea8

  Copy full SHA for b9a5ea8

• fix test

  vyzo

  committed
  4129038

  Copy full SHA for 4129038

• refactor debug log code to eliminate duplication

  vyzo

  committed
  f5ae10e

  Copy full SHA for f5ae10e

• handle id cids in internal versions of view/get

  vyzo

  committed
  870a47f

  Copy full SHA for 870a47f

Commits on Jul 10, 2021

• address review comments

  vyzo

  committed
  0c5e336

  Copy full SHA for 0c5e336

• fix lint

  vyzo

  committed
  df9670c

  Copy full SHA for df9670c

Commits on Jul 13, 2021

• always return the waitgroup in protectView

  Show description for 759594d
  vyzo

  committed
  759594d

  Copy full SHA for 759594d

• put a mutex around HeadChange

  vyzo

  committed
  60212c8

  Copy full SHA for 60212c8

• nit: remove useless goto

  Show description for 04abd19
  Stebalien

  committed
  04abd19

  Copy full SHA for 04abd19

• fix view waiting issues with the WaitGroup

  Show description for 257423e
  vyzo

  committed
  257423e

  Copy full SHA for 257423e

• finetune view waiting

  vyzo

  committed
  af39952

  Copy full SHA for af39952