Document and test devops documentation for GO services

[kltm]

Project link

https://github.com/orgs/geneontology/projects/146

Project description

TL;DR: Developers at all levels should be able to test code, put out new code releases, and update data using our documentation.

Over the last year, with the help of @abessiari, we have built up a set of DevOps tools and scripts to setup and service our major components, namely the GO API, the RDF endpoint, and AmiGO/GOlr.

This development work is wrapping soon (end of January) and we want to make sure that this new infrastructure is fully documented to meet the following criteria and use cases

• Is usable for developers who have minimal experience with DevOps
• Introduces the broad concepts of our DevOps
  • Terraform and workspaces (via S3 buckets)
    • Our naming conventions
  • How we refresh and test our services for a data release
    • Services not updated, but new one out with fresh data and old one destroyed
    • New instance up at patterned URL; test; switch Cloudflare/DNS; old instance down
• Given the context above, can be used by a developer to
  • Deploy to a fresh instance at a new random domain for testing/development of software
  • Deploy a fresh instance to switch over to on a data release
  • TODO: Deploy a fresh instance with security updates

(Note that we also should have functional Docker images for local development, but that may be outside the scope of these Terraform-related docs.)

Work done so far: @abessiari has done initial documentation; @sierra-moxon did a pass on GO API, getting it ready for production; @kltm and @dustine32 started a more generalized version of the documentation, but did not complete it.

Useful resources:

• "Generalized" documentation attempt: https://github.com/geneontology/devops-documentation/ (@kltm best contact)
• Complete GO API documentation: https://github.com/geneontology/go-fastapi/blob/main/provision/production/PRODUCTION_PROVISION_README.md (@sierra-moxon best contact)
• AmiGO/GOlr: https://github.com/geneontology/amigo/tree/master/provision
• GO RDF endpoint/graphstore: https://github.com/geneontology/go-graphstore/tree/master/provision

This documentation, out of necessity, may have to touch on topics like the release pipeline, credential distribution, DNS/Cloudflare, and the cert server.

PI

Chris

Product owner (PO)

Seth

Technical lead (TL)

Seth

Other personnel (OP)

Patrick, Sierra, Dustin

Technical specs

n/a

Other comments

I was going to originally file these items under #19, but thought that it would be more useful and understandable as it's own "project", rather than a set of tickets in an ongoing project.

[kltm]

@pgaudet This would be a mostly a @pkalita-lbl project, as we need to make progress before the end of January (while feedback can easily be given) and he is a good target for the docs; no other people would really need to be brought in, besides clarification of current documentation and test of said docs. I was going to originally slide this in under the "needful software" project, but thought it would be nice to have it surfaced on its own. I've talked to @cmungall about having @pkalita-lbl for this span and got thumbs up.

[kltm]

Given that we are punting the software get-together for the moment, where a major push to closing this would have taken place, and the fact that the iron is no longer particularly hot, @pgaudet given thumbs up to pecking away at this as is convenient.