Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bootstrapping #176

Merged
merged 62 commits into from
Jan 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
d0670d9
thoth grpc service.
Hellblazer Dec 4, 2023
986e829
clean up dep declarations
Hellblazer Dec 4, 2023
f03d054
Provide action upon Thoth inception.
Hellblazer Dec 4, 2023
7ede248
Inline action upon Thoth inception.
Hellblazer Dec 5, 2023
5996e03
decent amount of logging/clean up on thoth dynamic membership
Hellblazer Dec 7, 2023
f77db69
Refactor Seal to have Any message type value()
Hellblazer Dec 9, 2023
4565621
Fix mispelling of proto packages - d'oh
Hellblazer Dec 10, 2023
6d2ea93
basic skeleton of Binder and Reconciliation service infra
Hellblazer Dec 10, 2023
655f8f8
MVStore
Hellblazer Dec 10, 2023
9eb2ec4
Further flesh on the bone
Hellblazer Dec 10, 2023
0a6b2d0
Further flesh on the bone
Hellblazer Dec 10, 2023
8ddf2fa
The general idea.
Hellblazer Dec 19, 2023
ae531bf
Flesh out get/unbind API
Hellblazer Dec 19, 2023
03a7325
Woohoo! First successful replication.
Hellblazer Dec 20, 2023
1e54d72
correct reconciliation logic. track digests of bound.
Hellblazer Dec 20, 2023
3877f42
General cleanup. Add README.md's
Hellblazer Dec 22, 2023
f7b3614
Moar cleanup. Expose dht in proc domain
Hellblazer Dec 23, 2023
2fdceac
test comms cleanup
Hellblazer Dec 23, 2023
d4c2813
refactor out subdomain behaviour from top level ProcessDomain with Pr…
Hellblazer Dec 24, 2023
21013b2
correctly attach sources
Hellblazer Dec 25, 2023
61915ce
moar cleanup
Hellblazer Dec 25, 2023
5e8b052
remove
Hellblazer Dec 25, 2023
da1e78c
cache?
Hellblazer Dec 25, 2023
ce58802
process domain uses the event validation of the DHT's Ani
Hellblazer Dec 25, 2023
e13a72a
properly parameterize ProcessDomain
Hellblazer Dec 25, 2023
edaa49f
use active majority bootstrapping
Hellblazer Dec 26, 2023
82c987a
revert RingIterator bootstrap majority
Hellblazer Dec 26, 2023
76365d2
logging
Hellblazer Dec 26, 2023
2612016
use controlled identifier directly for seed from Node
Hellblazer Dec 26, 2023
417fc09
override additional
Hellblazer Dec 26, 2023
35d3d14
amp logging
Hellblazer Dec 26, 2023
7295616
amp logging
Hellblazer Dec 26, 2023
918309d
targeted loggin
Hellblazer Dec 26, 2023
1170241
better config. just model for now
Hellblazer Dec 26, 2023
f090591
arg
Hellblazer Dec 26, 2023
21a392a
again?
Hellblazer Dec 26, 2023
60f7d0c
moar logging
Hellblazer Dec 26, 2023
5ea27e2
decouple context
Hellblazer Dec 26, 2023
f9404b1
only unique
Hellblazer Dec 26, 2023
2a15db3
add BootstrapVerifiers to provide key state resolution via delegation…
Hellblazer Dec 27, 2023
1d83bdc
fix compilation error
Hellblazer Dec 27, 2023
71a61d9
use stereotomy msg, add BootstrapVerifiersTest
Hellblazer Dec 27, 2023
b984a14
Rename to Bootstrapper, add EventValidation functionality. Basically…
Hellblazer Dec 27, 2023
3b05df3
interim
Hellblazer Dec 28, 2023
e2176fc
Handle view join failure.
Hellblazer Dec 28, 2023
cbdd46b
logging
Hellblazer Dec 28, 2023
d0a54a9
Bootstrapping Fireflies.
Hellblazer Dec 30, 2023
64ee543
Cleanup and extend SwarmTest timeout
Hellblazer Dec 30, 2023
048acdd
moar clean up
Hellblazer Dec 30, 2023
92be787
Use non virtual threads on inbound servers
Hellblazer Dec 30, 2023
b3d94ae
halve swarm test for the poor build hamsters. Moar view cleanup
Hellblazer Dec 30, 2023
b3e9be5
Moar view stuff
Hellblazer Dec 30, 2023
2d8b234
The problem child
Hellblazer Dec 31, 2023
d79238b
This seems to make everything all right again.
Hellblazer Dec 31, 2023
dedc485
Reinstitute delegated event validation and verification to Fireflies.…
Hellblazer Dec 31, 2023
f265b82
fix isolates
Hellblazer Dec 31, 2023
a915749
Work around the Verifiers
Hellblazer Dec 31, 2023
40fe861
Minimize sig verification in replication. Re-enable sig verification …
Hellblazer Dec 31, 2023
6e5c22a
add keyState api for bootstrap verification
Hellblazer Jan 1, 2024
8e96d59
readd Bootstrapper
Hellblazer Jan 1, 2024
6dc1ad6
Use delegated Verifiers and EventValidation in the View
Hellblazer Jan 1, 2024
8de73a2
Moar cleanup wrt scheduling
Hellblazer Jan 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 8 additions & 1 deletion .github/workflows/maven.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,20 @@
name: Java CI

on: [push]
on: [ push ]

jobs:
build:

runs-on: ubuntu-latest

steps:
- name: Cache local Maven repository
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: Set up Maven
uses: stCarolas/[email protected]
with:
Expand Down
205 changes: 154 additions & 51 deletions README.md

Large diffs are not rendered by default.

22 changes: 11 additions & 11 deletions choam/src/main/java/com/salesforce/apollo/choam/CHOAM.java
Original file line number Diff line number Diff line change
Expand Up @@ -11,27 +11,27 @@
import com.google.protobuf.ByteString;
import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.Message;
import com.salesfoce.apollo.choam.proto.*;
import com.salesfoce.apollo.choam.proto.SubmitResult.Result;
import com.salesfoce.apollo.cryptography.proto.PubKey;
import com.salesfoce.apollo.messaging.proto.AgedMessageOrBuilder;
import com.salesforce.apollo.archipelago.RouterImpl.CommonCommunications;
import com.salesforce.apollo.bloomFilters.BloomFilter;
import com.salesforce.apollo.choam.comm.*;
import com.salesforce.apollo.choam.fsm.Combine;
import com.salesforce.apollo.choam.fsm.Combine.Merchantile;
import com.salesforce.apollo.choam.proto.*;
import com.salesforce.apollo.choam.proto.SubmitResult.Result;
import com.salesforce.apollo.choam.support.*;
import com.salesforce.apollo.choam.support.Bootstrapper.SynchronizedState;
import com.salesforce.apollo.choam.support.HashedCertifiedBlock.NullBlock;
import com.salesforce.apollo.cryptography.*;
import com.salesforce.apollo.cryptography.Signer.SignerImpl;
import com.salesforce.apollo.cryptography.proto.PubKey;
import com.salesforce.apollo.membership.Context;
import com.salesforce.apollo.membership.GroupIterator;
import com.salesforce.apollo.membership.Member;
import com.salesforce.apollo.membership.RoundScheduler;
import com.salesforce.apollo.membership.messaging.rbc.ReliableBroadcaster;
import com.salesforce.apollo.membership.messaging.rbc.ReliableBroadcaster.MessageAdapter;
import com.salesforce.apollo.membership.messaging.rbc.ReliableBroadcaster.Msg;
import com.salesforce.apollo.messaging.proto.AgedMessageOrBuilder;
import com.salesforce.apollo.utils.Utils;
import io.grpc.StatusRuntimeException;
import org.h2.mvstore.MVMap;
Expand Down Expand Up @@ -75,9 +75,9 @@ public class CHOAM {
private final AtomicReference<HashedCertifiedBlock> genesis = new AtomicReference<>();
private final AtomicReference<HashedCertifiedBlock> head = new AtomicReference<>();
private final ExecutorService linear;
private final AtomicReference<nextView> next = new AtomicReference<>();
private final AtomicReference<Digest> nextViewId = new AtomicReference<>();
private final Parameters params;
private final AtomicReference<nextView> next = new AtomicReference<>();
private final AtomicReference<Digest> nextViewId = new AtomicReference<>();
private final Parameters params;
private final PriorityBlockingQueue<HashedCertifiedBlock> pending = new PriorityBlockingQueue<>();
private final RoundScheduler roundScheduler;
private final Session session;
Expand Down Expand Up @@ -587,6 +587,10 @@ private void genesisInitialization(final HashedBlock h, final List<Transaction>
}
}

private String getLabel() {
return "CHOAM" + params.member().getId() + params.context().getId();
}

private boolean isNext(HashedBlock next) {
if (next == null) {
return false;
Expand Down Expand Up @@ -938,10 +942,6 @@ private void synchronizedProcess(CertifiedBlock certifiedBlock) {
pending.add(hcb);
}

private String getLabel() {
return "CHOAM" + params.member().getId() + params.context().getId();
}

public interface BlockProducer {
Block checkpoint();

Expand Down
20 changes: 9 additions & 11 deletions choam/src/main/java/com/salesforce/apollo/choam/Committee.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@

import org.slf4j.Logger;

import com.salesfoce.apollo.choam.proto.Certification;
import com.salesfoce.apollo.choam.proto.Reconfigure;
import com.salesfoce.apollo.choam.proto.SubmitResult;
import com.salesfoce.apollo.choam.proto.SubmitResult.Result;
import com.salesfoce.apollo.choam.proto.Transaction;
import com.salesfoce.apollo.choam.proto.ViewMember;
import com.salesforce.apollo.choam.proto.Certification;
import com.salesforce.apollo.choam.proto.Reconfigure;
import com.salesforce.apollo.choam.proto.SubmitResult;
import com.salesforce.apollo.choam.proto.SubmitResult.Result;
import com.salesforce.apollo.choam.proto.Transaction;
import com.salesforce.apollo.choam.proto.ViewMember;
import com.salesforce.apollo.choam.support.HashedCertifiedBlock;
import com.salesforce.apollo.cryptography.Digest;
import com.salesforce.apollo.cryptography.DigestAlgorithm;
Expand All @@ -33,21 +33,19 @@

/**
* @author hal.hildebrand
*
*/
public interface Committee {

static Map<Member, Verifier> validatorsOf(Reconfigure reconfigure, Context<Member> context) {
return reconfigure.getJoinsList()
.stream()
.collect(Collectors.toMap(e -> context.getMember(new Digest(e.getMember().getId())),
e -> new DefaultVerifier(publicKey(e.getMember()
.getConsensusKey()))));
e -> new DefaultVerifier(
publicKey(e.getMember().getConsensusKey()))));
}

/**
* Create a view based on the cut of the supplied hash across the rings of the
* base context
* Create a view based on the cut of the supplied hash across the rings of the base context
*/
static Context<Member> viewFor(Digest hash, Context<? super Member> baseContext) {
Context<Member> newView = new ContextImpl<>(hash, baseContext.getRingCount(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
import com.chiralbehaviors.tron.Fsm;
import com.google.protobuf.ByteString;
import com.google.protobuf.InvalidProtocolBufferException;
import com.salesfoce.apollo.choam.proto.*;
import com.salesfoce.apollo.cryptography.proto.PubKey;
import com.salesforce.apollo.choam.proto.*;
import com.salesforce.apollo.cryptography.proto.PubKey;
import com.salesforce.apollo.archipelago.RouterImpl.CommonCommunications;
import com.salesforce.apollo.choam.comm.Terminal;
import com.salesforce.apollo.choam.fsm.Genesis;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

import java.util.Collections;

import com.salesfoce.apollo.choam.proto.Validate;
import com.salesforce.apollo.choam.proto.Validate;
import com.salesforce.apollo.choam.CHOAM.BlockProducer;
import com.salesforce.apollo.cryptography.Signer;
import com.salesforce.apollo.cryptography.Verifier;
Expand All @@ -17,7 +17,6 @@

/**
* @author hal.hildebrand
*
*/
public class GenesisContext extends ViewContext {

Expand Down
64 changes: 32 additions & 32 deletions choam/src/main/java/com/salesforce/apollo/choam/Parameters.java
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@
import com.netflix.concurrency.limits.limit.AIMDLimit;
import com.netflix.concurrency.limits.limiter.LifoBlockingLimiter;
import com.netflix.concurrency.limits.limiter.SimpleLimiter;
import com.salesfoce.apollo.choam.proto.FoundationSeal;
import com.salesfoce.apollo.choam.proto.Join;
import com.salesfoce.apollo.choam.proto.Transaction;
import com.salesfoce.apollo.stereotomy.event.proto.KERL_;
import com.salesforce.apollo.choam.proto.FoundationSeal;
import com.salesforce.apollo.choam.proto.Join;
import com.salesforce.apollo.choam.proto.Transaction;
import com.salesforce.apollo.stereotomy.event.proto.KERL_;
import com.salesforce.apollo.archipelago.Router;
import com.salesforce.apollo.choam.CHOAM.TransactionExecutor;
import com.salesforce.apollo.choam.support.CheckpointState;
Expand Down Expand Up @@ -60,32 +60,36 @@ public static Builder newBuilder() {
return new Builder();
}

public int majority() {
return runtime.context.majority();
public Function<ULong, File> checkpointer() {
return runtime.checkpointer;
}

public SigningMember member() {
return runtime.member;
public Router communications() {
return runtime.communications;
}

public Context<Member> context() {
return runtime.context;
}

public Router communications() {
return runtime.communications;
public Function<Map<Member, Join>, List<Transaction>> genesisData() {
return runtime.genesisData;
}

public ChoamMetrics metrics() {
return runtime.metrics;
public Supplier<KERL_> kerl() {
return runtime.kerl;
}

public Function<ULong, File> checkpointer() {
return runtime.checkpointer;
public int majority() {
return runtime.context.majority();
}

public Function<Map<Member, Join>, List<Transaction>> genesisData() {
return runtime.genesisData;
public SigningMember member() {
return runtime.member;
}

public ChoamMetrics metrics() {
return runtime.metrics;
}

public TransactionExecutor processor() {
Expand All @@ -96,10 +100,6 @@ public BiConsumer<HashedBlock, CheckpointState> restorer() {
return runtime.restorer;
}

public Supplier<KERL_> kerl() {
return runtime.kerl;
}

public static class MvStoreBuilder implements Cloneable {
private int autoCommitBufferSize = -1;
private int autoCompactFillRate = -1;
Expand Down Expand Up @@ -580,10 +580,6 @@ public Limiter<Void> build(String name, MetricRegistry metrics) {
.build();
}

public Duration getbacklogDuration() {
return backlogDuration;
}

public int getBacklogSize() {
return backlogSize;
}
Expand Down Expand Up @@ -638,6 +634,10 @@ public LimiterBuilder setTimeout(Duration timeout) {
return this;
}

public Duration getbacklogDuration() {
return backlogDuration;
}

public LimiterBuilder setBacklogDuration(Duration backlogDuration) {
this.backlogDuration = backlogDuration;
return this;
Expand Down Expand Up @@ -679,14 +679,6 @@ public static class Builder implements Cloneable {
private SignatureAlgorithm viewSigAlgorithm = SignatureAlgorithm.DEFAULT;
private int crowns = 2;

public int getCrowns() {
return crowns;
}

public void setCrowns(int crowns) {
this.crowns = crowns;
}

public Parameters build(RuntimeParameters runtime) {
return new Parameters(runtime, combine, gossipDuration, maxCheckpointSegments, submitTimeout, genesisViewId,
checkpointBlockDelta, crowns, digestAlgorithm, viewSigAlgorithm,
Expand Down Expand Up @@ -739,6 +731,14 @@ public Builder setCombine(ReliableBroadcaster.Parameters combine) {
return this;
}

public int getCrowns() {
return crowns;
}

public void setCrowns(int crowns) {
this.crowns = crowns;
}

public DigestAlgorithm getDigestAlgorithm() {
return digestAlgorithm;
}
Expand Down
Loading