-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add versioning to readthedocs, small warning remaining
- Loading branch information
Showing
25 changed files
with
2,406 additions
and
49 deletions.
There are no files selected for viewing
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
Data Versioning | ||
================================ | ||
|
||
The `versioning` package of `impresso_commons` contains several modules and scripts with classes and functions that allow to version Impresso's data at various stages of the processing pipeline. | ||
|
||
The main goal of this approach is to version the data and track information at every stage to: | ||
1. **Ensure data consisteny and ease of debugging:** Data elements should be consistent across stages, and inconsistencies/differences should be justifiable through the identification of data leakage points. | ||
2. **Allow partial updates:** It should be possible to (re)run all or part of the processes on subsets of the data, knowing which version of the data was used at each step. This can be necessary when new media collections arrive, or when an existing collection has been patched. | ||
3. **Ensure transparency:** Citation of the various data stages and datasets should be straightforward; users should know when using the interface exactly what versions they are using, and should be able to consult the precise statistics related to them. | ||
|
||
|
||
Data Statistics and NewspaperStatistics | ||
------------------------------------------ | ||
|
||
.. automodule:: impresso_commons.versioning.data_statistics | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
Data Manifest | ||
-------------------------------------------- | ||
|
||
.. automodule:: impresso_commons.versioning.data_manifest | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
Versioning Helpers | ||
-------------------------------------------- | ||
|
||
.. automodule:: impresso_commons.versioning.helpers | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
Manifest Computing Script | ||
-------------------------------------------- | ||
|
||
.. automodule:: impresso_commons.versioning.compute_manifest | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Oops, something went wrong.