Packaging and Metadata Improvements / Questions

[alexrichey]

This is still a draft - I'll be adding notes here in preparation for our upcoming discussion. No feedback required yet.

Improvements / Changes

• Stop Bundling Metadata in the packages (e.g. what happens if we change destination four-fours, then have a bunch of stale metadata on OD). Or maybe just stop using it when we distribute?

• What to do about intermediate bundles/packages/"Assemblages"? (e.g. zip files of metadata + datasets, or fgdbs that contain multiple datasets). Where do they live on S3?

• Our current models are DCP specific (e.g. Bytes destination). How do we generalize them?

• How do we "migrate" metadata? How do we make reviewable changes?

• Refer to columns identifiers as ids/keys, not names

• column overrides at both dataset_file and destination level should always reference the column key

Questions

• Metadata versions / breaking changes in our code base.
• Flatten the Package model? e.g. remote attachment/dataset_file/zip distinction, both on S3 and in metadata
  • simplifies and addresses situations where, e.g. we have a csv that could function both as a dataset and as an attachment.

[alexrichey]

Proposed Meeting Agenda

• Review / Discuss: model changes for product.dataset.metadata

  • consistency of key vs id vs name. Which do we like?
  • Goals:
    • more consistency / "openness" / extensibility of types, esp for DatasetFiles, and Destinations
      becomes a question of what to hardcode in models. subgoal: BYO-Model.
    • generalize to remove anything dcp-specific, e.g. "BytesDestination"
  • Sample Proposed Metadata here

• Review: changes to Socrata metadata application. (Quick, just want to make sure we're all on the same page)

  • Consistency in overrides for dataset, and then destination. Issue here

• Discuss: Add metadata at the product level (vs dataset)

  • use cases: multi-dataset zips, shapefiles, and fgdbs.

• Determine: What to call the product-metadata repo?

  • Brainstorm: What do we / will we store there?

• Discuss: Documentation, and can we leverage pydantic models/descriptions

• Discuss: Refactoring YAML. (Probably quick) Thoughts:

  • We can order top-level keys in the yaml like this yaml.dump(data, default_flow_style=False, sort_keys=False)
  • I like having a newline between top-sections (makes it easier for me to traverse) which we can do with something like How can I add blank lines between top-level objects? yaml/pyyaml#127 (comment)
  • Action Items: Linter/Formatter/Style Choices.

Last 10 Minutes

• Chat about how best to structure our upcoming packaging meeting about generating README's / Data Dictionaries.

[alexrichey]

From our chat about product-level packages, e.g. an fgdb containing all of DCM, here's a folder structure that I think would accomodate our needs. Using LION as an example product, and atomic_polygons as an example dataset.

product_datasets 
  ...
  lion 
    package (somewhat duplicated with `packages` below, but I think is fine as it denotes the stage of the lifecycle, rather than anything file-related)
      ...
      24b 
        packages
            - all_lion_datasets.zip
        datasets `<--- New Folder`
            ...
            atomic_polygons
                dataset_files
                attachments
                packages `<--- New Folder`
                    atomic_poly_csv_and_attachments.zip