Merge pull request #452 from kbase/feature-SAM-209-organize-documemation

SAM-209 organize documentation
kbase · Apr 19, 2022 · aefcc32 · aefcc32
2 parents edd45fc + e0ef957
commit aefcc32
Show file tree

Hide file tree

Showing 29 changed files with 1,216 additions and 487 deletions.
diff --git a/.gitignore b/.gitignore
@@ -8,3 +8,4 @@ deploy.cfg
 .coverage
 test/test_temp/
 lib/SampleService/SampleServiceImpl.py.bak*
+_temp
diff --git a/README.md b/README.md
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
@@ -2,8 +2,10 @@
 
 ## Unreleased
 
+* organize documentation under a single `docs` directory; add some additional documentation and placeholders  [SAM-209]
+
 ## 0.2.4
-* Changes github actions: creates images from releases off master, adds test running on develop branch
+* Changes GitHub actions: creates images from releases off master, adds test running on develop branch
 * Bugfix for write-write error
 * Bugfix for incorrectly thrown `owner unexpectedly changed` errors
 * Improved error handling/messages for get_samples method

diff --git a/TODO.md → docs/TODO.md b/TODO.md → docs/TODO.md
diff --git a/docs/adrs/0000-TEMPLATE.md b/docs/adrs/0000-TEMPLATE.md
@@ -0,0 +1,43 @@
+# TITLE HERE
+
+Date: YYYY-MM-DD
+
+SUMMARY HERE
+
+## Author(s)
+
+@AUTHOR
+
+## Status
+
+WHAT ARE STATUSES?
+
+## Alternatives Considered
+
+* alt1
+* alt2
+
+## Decision Outcome
+
+DESCRIBE
+
+## Consequences
+
+DESCRIBE
+
+## Pros and Cons of the Alternatives
+
+### alt1
+
+* `+` a pro
+* `-` a con
+
+### alt2
+
+* `+` a pro
+* `-` a con
+
+## References
+
+* [alternative 1](https://www.example.com/)
+* [alternative 2](https://www.example.com/)
diff --git a/docs/adrs/index.md b/docs/adrs/index.md
@@ -0,0 +1,5 @@
+# Architectural Decision Records (ADRs)
+
+## Contents
+
+- [Template](./0000-TEMPLATE.md)
diff --git a/docs/api/index.md b/docs/api/index.md
@@ -0,0 +1,5 @@
+# API
+
+The SDK API specification for the service is contained in the `SampleService.spec` file. An indexed
+interactive version is
+[also available](http://htmlpreview.github.io/?https://github.com/kbaseIncubator/sample_service/blob/master/SampleService.html).
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -0,0 +1,131 @@
+# Configuration
+
+The server has several startup parameters beyond the standard SDK-provided parameters that must be configured in the Catalog Service by a Catalog Service administrator in order for the service to run. These are documented in the `deploy.cfg` file.
+
+## Configuration File
+
+The `deploy.cfg` configuration file contains a key, `metadata-validator-config-repo`, that if provided must be a relative GitHub path that points to a validator configuration GitHub repo.
+
+Setting `github-token` will help to avoid any rate limiting that may occur (1k/hr vs 60/hr requests.)
+
+The configuration repo should have chronological releases containing a configuration file. This file's name can be specified with `metadata-validator-config-filename` (`metadata_validation.yml` by default).
+
+The most recent release from the specified repo will be loaded. If pre-releases should also be included, set the `metadata-validator-config-prerelease` config variable to 'true'. 
+
+A direct file URL override can also be provided with the `metadata-validator-config-url` key. With this form, the url begins with `file://`, followed by a path to the directory containing the validation config file, which should be named `metadata_validation.yml` (unless overridden as described above.) This is utilized by tests.
+
+The configuration file is loaded on service startup and used to configure the metadata validators. If changes are made to the configuration file the service must be restarted to reconfigure the validators.
+
+The configuration file uses the YAML format and is validated against the following JSONSchema:
+
+```json
+{
+  "type": "object",
+  "definitions": {
+    "validator_set": {
+      "type": "object",
+      "additionalProperties": {
+        "type": "object",
+        "properties": {
+          "key_metadata": {
+            "type": "object",
+            "additionalProperties": {
+              "type": [
+                "number",
+                "boolean",
+                "string",
+                "null"
+              ]
+            }
+          },
+          "validators": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "properties": {
+                "module": {
+                  "type": "string"
+                },
+                "callable_builder": {
+                  "type": "string"
+                },
+                "parameters": {
+                  "type": "object"
+                }
+              },
+              "additionalProperties": false,
+              "required": [
+                "module",
+                "callable_builder"
+              ]
+            }
+          }
+        },
+        "required": [
+          "validators"
+        ]
+      }
+    },
+    "additionalProperties": false
+  },
+  "properties": {
+    "validators": {
+      "$ref": "#/definitions/validator_set"
+    },
+    "prefix_validators": {
+      "$ref": "#/definitions/validator_set"
+    }
+  },
+  "additionalProperties": false
+}
+```
+
+The configuration consists of a mapping of standard and prefix metadata keys to a further mapping of metadata key properties, including the list of validator specifications and static metadata about the key. Each validator is run against the metadata value in order. The `module` key is a python import path for the module containing a builder function for the validator, while the `callable_builder` key is the name of the function within the module that can be called to create the validator. `parameters` contains a mapping that is passed directly to the callable builder. The builder is expected to return a callable with the call signature as described previously.
+
+A simple configuration might look like:
+
+```yaml
+validators:
+    foo:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: noop
+        key_metadata:
+            description: test key
+            semantics: none really
+    stringlen:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: string
+              parameters:
+                  max-len: 5
+            - module: SampleService.core.validator.builtin
+              callable_builder: string
+              parameters:
+                  keys: spcky
+                  max-len: 2
+        key_metadata:
+            description: check that no strings are longer than 5 characters and spcky is <2
+prefix_validators:
+    gene_ontology_:
+        validators:
+            - module: geneontology.plugins.kbase
+              callable_builder: go_builder
+              parameters: 
+                  url: https://fake.go.service.org/api/go
+                  apitoken: abcdefg-hijklmnop
+        key_metadata:
+            description: The key value contains a GO ontology ID that is linked to the sample.
+            go_url: https://fake.go.service.org/api/go
+            date_added_to_service: 2020/3/8
+```
+
+In this case any value for the `foo` key is allowed, as the `noop` validator is assigned to the key. Note that if no validator was assigned to `foo`, including that key in the metadata would cause a validation error. The `stringlen` key has two validators assigned and any metadata under that key must pass both validators.
+
+The first validator ensures that no keys or value strings in in the metadata map are longer than 5 characters, and the second ensures that the value of the `spcky` key is a string of no more than two characters. See the documentation for the `string` validator (below) for more information.
+
+> TODO: we are not using prefix validators; should not encourage their use?
+
+Finally, the wholly fabricated `gene_ontology_` prefix validator will match **any** key starting with `gene_ontology_`. The validator code might look up the suffix of the key, say `GO_0099593`, at the provided url to ensure the suffix matches a legitimate ontology term. Without a prefix validator, a validator would have to be written for each individual ontology term, which is infeasible.
+
+All the metadata keys have static metadata describing the semantics of the keys and other properties that service users might need to properly use the keys.
diff --git a/docs/deployment/index.md b/docs/deployment/index.md
@@ -0,0 +1,4 @@
+# Deployment
+
+> TODO: this section should describe how to deploy the sample service and related concerns. E.g. assets within the repo related to deployment, required and optional env vars used in deploy.cfg, the process of preparing a release and deployment, and tools we use for deployment itself. It should also provide links to KBase deployment processes.
+> 
diff --git a/docs/design/built-in-validators.md b/docs/design/built-in-validators.md
@@ -0,0 +1,133 @@
+# Built-In Validators
+
+All built in validators are in the `SampleService.core.validator.builtin` module.
+
+## noop
+
+Example configuration:
+
+```yaml
+validators:
+    metadatakey:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: noop
+```
+
+This validator accepts any and all values.
+
+## string
+
+Example configuration:
+
+```yaml
+validators:
+    metadatakey:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: string
+              parameters:
+                  keys: ['key1', 'key2']
+                  required: True
+                  max-len: 10
+```
+
+* `keys` is either a string or a list of strings and determines which keys will be checked by the
+  validator. If the key exists, its value must be a string or `None` (`null` in JSON-speak).
+* `required` requires any keys in the `keys` field to exist in the map, although their value may
+  still be `None`.
+* `max-len` determines the maximum length in characters of the values of the keys listed in `keys`.
+  If `keys` is not supplied, then it determines the maximum length of all keys and string values
+  in the metadata value map.
+
+## enum
+
+Example configuration:
+
+```yaml
+validators:
+    metadatakey:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: enum
+              parameters:
+                  keys: ['key1', 'key2']
+                  allowed-values: ['red', 'blue', 'green]
+```
+
+* `allowed-values` is a list of primitives - strings, integers, floats, or booleans - that are
+  allowed metadata values. If `keys` is not supplied, all values in the metadata value mapping must
+  be one of the allowed values.
+* `keys` is either a string or a list of strings and determines which keys will be checked by the
+  validator. The key must exist and its value must be one of the `allowed-values`.
+
+## units
+
+Example configuration:
+
+```yaml
+validators:
+    metadatakey:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: units
+              parameters:
+                  key: 'units'
+                  units: 'mg/L'
+```
+
+* `key` is the metadata value key that will be checked against the `units` specification.
+* `units` is a **unit specification in the form of an example**. Any units that can be converted
+  to the given units will be accepted. For example, if `units` is `K`, then `degF`, `degC`, and
+  `degR` are all acceptable input to the validator. Similarly, if `N` is given, `kg * m / s^2` and
+  `lb * f / s^2` are both acceptable.
+
+## number
+
+Example configuration:
+
+```yaml
+validators:
+    metadatakey:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable_builder: number
+              parameters:
+                  keys: ['length', 'width']
+                  type: int
+                  required: True
+                  gte: 42
+                  lt: 77
+```
+
+Ensures all values are integers or floats.
+
+* `keys`, which is either a string or a list of strings, determines which keys in the metadata value
+  map are checked. If omitted, all keys are checked.
+* If `required` is specified, the keys in the `keys` list must exist in the metadata value map,
+  although their value may be `null`.
+* `type` specifies that the number or numbers must be integers if set to `int` or any number if
+  omitted or set to `float` or `null`.
+* `gt`, `gte`, `lt`, and `lte` are respectively greater than, greater than or equal,
+  less than, and less than or equal, and specify a range in which the number or numbers must exist.
+  If `gt` or `lt` are specified, `gte` or `lte` cannot be specified, respectively, and vice versa.
+
+## ontology_has_ancestor
+
+Example configuration:
+
+```yaml
+validators:
+    metadatakey:
+        validators:
+            - module: SampleService.core.validator.builtin
+              callable-builder: ontology_has_ancestor
+              parameters:
+                  ontology: 'envo_ontology'
+                  ancestor_term: 'ENVO:00010483'
+                  srv_wiz_url: 'https://kbase.us/services/service_wizard'
+```
+
+* `ontology` is the ontology that the meta value will be checked against.
+* `ancestor_term` is the ancestor ontology term that will be used to check whether meta value has such ancestor or not.
+* `srv_wiz_url` is the kbase service wizard url for getting OntologyAPI service.
diff --git a/docs/design/errors.md b/docs/design/errors.md
@@ -0,0 +1,49 @@
+# Errors
+
+Error messages returned from the API may be general errors without a specific structure to
+the error string or messages that have error codes embedded in the error string. The latter
+*usually* indicate that the user/client has sent bad input, while the former indicate a server
+error. A message with an error code has the following structure:
+
+```text
+Sample service error code <error code> <error type>: <message>
+```
+
+There is a 1:1 mapping from error code to error type; error type is simply a more readable
+version of the error code. The error type **may change** for an error code, but the error code
+for a specific error will not.
+
+The current error codes are:
+
+| Code | Message |  Meaning |
+|------|---------|----------|
+| 20000 | Unauthorized | | |
+| 30000 | Missing input parameter |  |
+| 30001 | Illegal input parameter |  |
+| 30010 | Metadata validation failed |  |
+| 40000 | Concurrency violation |  |
+| 50000 | No such user |  |
+| 50010 | No such sample |  |
+| 50020 | No such sample version |  |
+| 50030 | No such sample node |  |
+| 50050 | No such data link |  |
+| 60000 | Data link exists for data ID |  |
+| 60010 | Too many data links |  |
+| 100000 | unsupported operation |  |
+
+## Generic JSON-RPC Errors
+
+Errors with the form of the request, or the programming of a method, may result in "standard" JSON-RPC errors.
+
+The word *standard* is in quotes because this service utilizes [JSON-RPC 1.1](https://jsonrpc.org/historical/json-rpc-1-1-wd.html), which never actually settled on standard error codes (and was never actually released as a JSON-RPC standard). Rather, KBase utilizes error codes from [JSON-RPC 2.0](https://www.jsonrpc.org/specification).
+
+The error codes are:
+
+| Code | Message |  Meaning |
+|------|---------|----------|
+| -32700 | Parse error | Invalid JSON was received by the server. An error occurred on the server while parsing the JSON text. |
+| -32600 | Invalid Request | The JSON sent is not a valid Request object. |
+| -32601 | Method not found  | The method does not exist / is not available. |
+| -32602 | Invalid params | Invalid method parameter(s). |
+| -32603 | Internal error | Internal JSON-RPC error. |
+| -32000 to -32099| Server error | Reserved for implementation-defined server-errors. |