Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CT static API support #73

Merged
merged 15 commits into from
Jul 23, 2024
Merged

Conversation

AlCutter
Copy link
Collaborator

@AlCutter AlCutter commented Jul 18, 2024

This PR enables Tessera to support a CT Static API personality.

There's broadly two things which are necessary to be able to support that personality:

  1. Formatting of EntryBundle is different and incompatible
    tlog-tiles specifies a length/value scheme, whereas CT Static API specifies just value and relies on the fact that value in this case is self-describing.
  2. Each log entry contains its own index in the log.
    The SCT returned to the submitter contains the sequence number of the entry in the log in the CtExtensions field of the SCT struct. What's not obvious from the spec is that this must also be part of the Merkle leaf (otherwise compatibility with RFC6962 would be broken).
    This means that we don't actually know the LeafData or its MerkleLeafHash until we've sequenced it.

Beyond enabling a CT Static API personality to be built, other large goals for this PR are:

  1. To make it hard/impossible to build new transparency log ecosystems which attempt to use these patterns outside of CT.
  2. Avoid "overly invasive" changes spreading throughout Tessera, instead containing them as close as possible to the personality:
  3. Non-CT logs shouldn't have to pay a performance cost simply because the support exists
  4. Contain changes to the sequencing side, leaving integration concerned only with managing Merkle specifics.

Toward #41

@AlCutter AlCutter force-pushed the ct_static_api_support branch 8 times, most recently from c0989b7 to 46930b4 Compare July 19, 2024 11:51
@AlCutter AlCutter changed the title WIP: CT static API support CT static API support Jul 19, 2024
@AlCutter AlCutter force-pushed the ct_static_api_support branch 2 times, most recently from 1e70afd to 7ef226c Compare July 22, 2024 11:25
@AlCutter AlCutter requested a review from phbnf July 22, 2024 11:42
@AlCutter AlCutter marked this pull request as ready for review July 22, 2024 11:43
@AlCutter AlCutter requested a review from mhutchinson July 23, 2024 09:18
entry.go Show resolved Hide resolved
ct_only.go Outdated
type Storage interface {
// Add should duably assign an index to the provided Entry, and return it.
//
// Implementations MUST call the AssignIndex() method on the entry before marshaling and persisting it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is AssignIndex referring to?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot, that was a function from an earlier incarnation which went away.
I've updated the comments to remove the reference.

entry.go Outdated
}

// indexFunc will be called when an index is assigned to the entry.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this function MUST NOT modify the assigned index?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment to that effect.

@codecov-commenter
Copy link

codecov-commenter commented Jul 23, 2024

Codecov Report

Attention: Patch coverage is 29.48718% with 110 lines in your changes missing coverage. Please review.

Project coverage is 33.33%. Comparing base (46ec9c2) to head (1974787).
Report is 3 commits behind head on main.

Files Patch % Lines
ctonly/ct.go 0.00% 66 Missing ⚠️
storage/gcp/gcp.go 47.05% 16 Missing and 2 partials ⚠️
ct_only.go 0.00% 11 Missing ⚠️
entry.go 50.00% 6 Missing and 1 partial ⚠️
storage/queue.go 81.48% 3 Missing and 2 partials ⚠️
api/state.go 0.00% 2 Missing ⚠️
storage/integrate.go 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #73      +/-   ##
==========================================
- Coverage   35.80%   33.33%   -2.48%     
==========================================
  Files          16       18       +2     
  Lines        1363     1440      +77     
==========================================
- Hits          488      480       -8     
- Misses        801      888      +87     
+ Partials       74       72       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@AlCutter AlCutter force-pushed the ct_static_api_support branch from a805415 to dd4b26e Compare July 23, 2024 10:14
// See the License for the specific language governing permissions and
// limitations under the License.

package tessera
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tessera.Storage is somewhat independent from CT, but is defined inside ct_only.go. It seems a bit odd. Would it make sense to define NewCertificateTransparencySequencedWriter and convertCTEntry in their own package?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That interface is there currently just because there's not a corresponding tessera.NewSequencedWriter yet.

Ideally the CT stuff would be in its own package, but to prevent other new use-cases from using these patterns it needs be in here to do "private" stuff internal to Entry when setting those marshaling funcs.

ctonly/ct.go Show resolved Hide resolved
ct_only.go Outdated
type Storage interface {
// Add should duably assign an index to the provided Entry, and return it.
//
// Implementations MUST call MarshalBundleData method on the entry before marshaling and persisting it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now reads a bit funny because we must call the marshaling function before marshaling it? Maybe just remove the second marshaling?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d'oh, fixed - ta.

@AlCutter AlCutter force-pushed the ct_static_api_support branch from 272b47e to d1fc00d Compare July 23, 2024 16:01
storage/queue.go Outdated Show resolved Hide resolved
@phbnf
Copy link
Contributor

phbnf commented Jul 23, 2024

Summarizing our conversation here for the record.

The logic makes sense to me, so I'll approve this PR.

We could improve naming though to make it easier to read the code:
There are 2 things called entry: tessera.Entry, and storage.entry. And storage.entry stores a tessera.Entry in a data field. Not to mention storage.SequencedEntry.
There are 3 things called index: tessera.Entry.internal.Index (an integer pointer), tessera.Entry.Index (a method interface), storage.Entry.index (a future method).

@AlCutter
Copy link
Collaborator Author

Summarizing our conversation here for the record.

The logic makes sense to me, so I'll approve this PR.

We could improve naming though to make it easier to read the code: There are 2 things called entry: tessera.Entry, and storage.entry. And storage.entry stores a tessera.Entry in a data field. Not to mention storage.SequencedEntry. There are 3 things called index: tessera.Entry.internal.Index (an integer pointer), tessera.Entry.Index (a method interface), storage.Entry.index (a future method).

tessera.Index() is a getter for the similarly named internal field, so I think that's reasonable.
storage.entry is now storage.queueItem since that's the only place it's used.
queueItem.index is now queueItem.future.

@AlCutter AlCutter merged commit e8e136f into transparency-dev:main Jul 23, 2024
4 checks passed
@AlCutter AlCutter deleted the ct_static_api_support branch July 23, 2024 17:41
phbnf pushed a commit to phbnf/trillian-tessera that referenced this pull request Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants