[WIP] DEV-1849: Add yaml linter, formatter and unix-script for replacing chars.

[wrkhenddher]

• Add pre-commit hook yamlfmt as manual stage.
• Add pre-commit hooks, replacer and yamllint, to the commit stage.

yamlfmt is already being used by gdcapi as a pre-commit hook

replacer is a custom shell script that changes occurrences of any en-hyphen with an ascii dash. This will remove the possibility of inadvertently using en-hyphen as list-item structural marker.

yamllint performs various extra structural and semantics checks.

The reason to leave yamlfmt in the manual stage is so that all files that are currently reporting warnings can be fixed gradually. Once they are all fixed, the hook can be permanently turned on by removing the stages: [manual] attribute.

To run yamlfmt on demand, use the following command:

$ pre-commit run --hook-stage manual yamlfmt --files <files...>

# E.g. 
# pre-commit run --hook-stage manual yamlfmt --files src/gdcdictionary/schemas/aligned_reads.yaml

[philtom-ctds]

When adding libraries, check the license. This code is GPL'd. We cannot use it.

We need more permissible licenses, such as MIT or Apache 2.0.

[wrkhenddher]

Even when we neither distribute the tool, nor modify it, nor include it in our distribution?

(wouldn't using this tool be equivalent to using any other gnu-tool in our build system?)

[philtom-ctds]

that's a fair question and, to my knowledge, a grey area. as an interpreted language, python (mostly) does not have the concept of "linking". so, there's little distinction between a library and source code. are you aware of any legal precedents that have clarified this? if not, we should err on the side of caution and not use GPL'd python code.

[wrkhenddher]

Good point.

@stilesj-uchicago and I had the same understanding => "we're good in this case".

We're using yamlfmt (mit) in gdcapi but not yamllint (gplv3).

Maybe I can remove yamllint until we find an alternative?

[philtom-ctds]

1. This isn't going to work for everyone, such as Windows users.
2. If we are going to enforce ascii, we could use require-ascii. It's catch much more than just the hyphen. We'll need to exclude these files, as described in this comment.

[wrkhenddher]

1. This isn't going to work for everyone, such as Windows users.

Good point. Should it be a deal breaker?

2. If we are going to enforce ascii, we could use require-ascii. It's catch much more than just the hyphen. We'll need to exclude these files, as described in this comment.

The catch is that unicode chars are allowed. Depending on the version of the yaml file (which we do NOT explicitly define) and the version of the parser (somehow loose atm), one should either quote them or do whatever applies.

When I first saw the failure and the traceback, I was inclined to remove the whole "ascii" enforcement and instead specify yaml versions and straighten up all the versions (yaml files and parser).

I think we should pursue that end. It might take some bit effort but as long as we can diff the current json schema with the new one and make sure the match, we should be ok. IMHO.

[philtom-ctds]

this is a great comment 👍