Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Would there be any interest in more k8s "native" tooling for integration testing? #2672

Open
vaikas opened this issue Feb 9, 2022 · 6 comments · May be fixed by #2754
Open

Would there be any interest in more k8s "native" tooling for integration testing? #2672

vaikas opened this issue Feb 9, 2022 · 6 comments · May be fixed by #2754
Assignees
@mhutchinson @vaikas
Labels
feature

Comments

@vaikas
Copy link
Contributor

vaikas commented Feb 9, 2022 •
edited
Loading

Hey there :)

We in the Sigstore utilize Trillian as our Merkle tree (Thank you!!). As part of our desire to be able to run more extensive and comprehensive e2e testing, we're utilizing kind and utilizing containers for all of our testing. I have put together a sigstore-scaffolding that contains various components that need to be spun up for us to test Sigstore. One of these is the need to create a Trillian (and supporting operations, like creating 2 trees in Trillian). This may be related to what's looked for in #2329 @pavelkalinnikov ?

It's not ideal to have these be in a 'side' repo, and therefore, I wanted to see if there would be any interest in me upstreaming the Trillian related pieces into this repo. It would not be all of these, but just the components related to Trillian.

One nice side benefit also is that you can use 'kind' locally (assuming you have docker) and have a higher degree of confidence that what you're testing with will then also work in ci/cd as well as deployment to k8s, since the constructs are the same.

If there's interest, there's the overview of how we use the k8s constructs (like configmaps / jobs) to communicate the state between components that need information from Trillian (like, treeid created for example).

Anyways, I've yammered too long. Please let me know if there's any interest in this and I'd be happy to work with you and if there are any questions, comments or concerns, I'd be happy to answer them :)

Overview (for Sigstore, but contains information about Trillian, since we need it for both Rekor / CTLog)
https://github.com/vaikas/sigstore-scaffolding/blob/main/README.md

Configurations related to Trillian specific components:
https://github.com/vaikas/sigstore-scaffolding/tree/main/config/trillian

One example run in github actions:
https://github.com/vaikas/sigstore-scaffolding/runs/5097036599?check_suite_focus=true

I think the pieces that would be maybe interesting to upstream would be:

  • createtree that can update a configmap in a k8s cluster, plus a job that it runs as.
  • github workflow that creates a kind cluster, spins up logserver/logsigner and performs some operations on it
  • not sure if there's anything else at this point.
@mhutchinson mhutchinson self-assigned this Feb 10, 2022
@mhutchinson
Copy link
Contributor

Hello! Yeah this sounds cool and I think bringing these components into this repo would be great. I'm happy to review any PRs you want to send. The other addition to the list would obviously be a README that explains how to use these :-)

In terms of organizing this, we have top level directories for deployment and examples/deployment that are similar. Perhaps for now examples/deployment/kind would be the best place to organize these new files?

@vaikas
Copy link
Contributor Author

vaikas commented Feb 11, 2022

Fantastic! Thanks for getting back to me.
I also saw (and heavily cribbed the existing cmd/createtree) and produced a version that stashes the treeID created into a ConfigMap (so dependant pieces (like CTLog, Rekor) can utilize it in k8s fashion).
https://github.com/vaikas/sigstore-scaffolding/blob/main/cmd/trillian/createtree/main.go

I reckon that might live in:
cmd/createtree-k8s (?)

Another option would be to make it a flag to existing, but seems like that might not be desireable since it will take a dependency on k8s libraries?

I'm totes fine with the examples/deployment/kind but it's not only specific to kind (since any cluster will do). Wonder if examples/deployment/kubernetes/scaffolding might make it more clear that it can be used with other tooling.
Or something like that, I'm terrible with naming, and hence fine with kind in the first place.

Really that piece is only meant to allow Trillian dependent software be able to bootstrap by creating the tree using a k8s job (therefore lending itself nicely to e2e testing on k8s).

Thoughts?

@mhutchinson
Copy link
Contributor

Looks like you have a good grasp on naming; i.e. it's important, but it's hard!

I'm happy with the suggestion for cmd/createtree-k8s. We can roll them into one in the future if we like, but I think on balance I would prefer the duplication of a small amount of code than adding more options and dependencies to this small cmd.

Let's try examples/deployment/kubernetes/scaffolding and see how it looks. I'd originally suggested something other than this directory because a) I've never deployed it using the instructions in there; and b) that directory looked otherwise self-contained and single-purpose. That said, having all the k8s stuff grouped together is going to keep the codebase a lot more navigable. Really this is a signal that I should take some time to play with the trillian on k8s stuff :-)

@mhutchinson
Copy link
Contributor

Any updates here? :-)

@vaikas
Copy link
Contributor Author

vaikas commented May 31, 2022

Yes, thank you for the ping and sorry for the tardy reply. I upstreamed my code to sigstore and it now lives here:
https://github.com/sigstore/scaffolding

So, that's also been cleaned up a bit along the way which is nice :)

I'll make time this week to create the PR for it.

@vaikas vaikas linked a pull request Jun 7, 2022 that will close this issue
Open
2 tasks
@vaikas
Copy link
Contributor Author

vaikas commented Jun 7, 2022

@mhutchinson ok, finally got to it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mhutchinson mhutchinson

@vaikas vaikas

Labels
feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add scaffolding to stand up Trillian on k8s. vaikas/trillian
2 participants
@mhutchinson @vaikas