DTS Manifest parsing #210
DTS Manifest parsing #210
Conversation
| instructions = manifest[_DTS_INSTRUCTIONS_KEY] | ||
| for resource_obj in instructions[_DTS_INSTRUCTIONS_OBJECTS_KEY]: | ||
| datatype = resource_obj[_DTS_INSTRUCTIONS_DATATYPE_KEY] | ||
| parameters = frozendict(resource_obj[_DTS_INSTRUCTIONS_PARAMETERS_KEY]) |
There was a problem hiding this comment.
So the DTS has to know the app parameter names for each datatype it processes? IOW the DTS is coupled to the apps?
There was a problem hiding this comment.
I also assume the file names / locations in the staging area are in the parameters? They can't be anywhere else...
There was a problem hiding this comment.
Yes to both. We talked about doing some references from the parameters to file paths, but if we do that'll be in a later version.
There was a problem hiding this comment.
Well, that seems less than ideal, but if you've agreed that that's how it'll work so be it
| results[datatype].append(parameters) | ||
| # Re-package results as a dict of {datatype: ParseResult} | ||
| results = { | ||
| source: ParseResult(spcsrc, tuple(parsed)) for source, parsed in results.items() |
There was a problem hiding this comment.
| source: ParseResult(spcsrc, tuple(parsed)) for source, parsed in results.items() | |
| datatype: ParseResult(spcsrc, tuple(paramlist)) for datatype, paramlist in results.items() |
There was a problem hiding this comment.
you didn't like the paramlist name? That matched up better with the prior var names to me
There was a problem hiding this comment.
Oops. Meant to change that and got distracted. Changed.
| for parsed in res.results["gff_metagenome"].result: | ||
| assert parsed == {"param1": "value1", "param2": "value2"} |
There was a problem hiding this comment.
Why did you decide to make all the parameters the same?
There was a problem hiding this comment.
This was mostly meant as a smoke test / structure test, with more complicated ones elsewhere.
Doesn't matter, though, I updated with more complexity.
|
This does the work of parsing the DTS manifest and returning results.
It relies pretty heavily on the jsonschema to do the validation work, so the parser has relatively minimal checking.
Also a minor schema tweak to make sure that parameters are always primitive types.
This is on the big side, but the vast majority is either test data or tests that follow a pretty set pattern.