Creation of SOURCES.toml #634
Labels
Comments
|
@dsmedia just to echo #631 (comment)
If we wanted a non- vega-datasets/scripts/build_datapackage.py Lines 231 to 234 in 719c388 I'm not sure how familiar you are with |
|
Sounds good to me. I don't mind either format and having automated checks sounds great. |
|
Might something like this work for a TOML format, containing resource-level (i.e. dataset level) description, source and license information? This is just a proof-of-concept that includes three of the datasets: SOURCES.toml# Package-level license information
[package.license]
name = "BSD-3-Clause"
path = "https://opensource.org/license/bsd-3-clause"
title = "The 3-Clause BSD License"
# Resource metadata
budget.description = "Budget FY 2016 - Receipts data from the Office of Management and Budget (U.S.)"
[budget.sources]
title = "Office of Management and Budget (U.S.)"
path = "https://www.govinfo.gov/app/details/BUDGET-2016-DB/BUDGET-2016-DB-3"
countries.description = """
This dataset combines key demographic indicators (life expectancy at birth and fertility rate measured as babies per woman) for various countries from 1955 to 2000 at 5-year intervals. It includes both current values and adjacent time period values (previous and next) for each indicator. Gapminder's [data documentation](https://www.gapminder.org/data/documentation/) notes that its philosophy is to fill data gaps with estimates and use current geographic boundaries for historical data. Gapminder states that it aims to "show people the big picture" rather than support detailed numeric analysis.
"""
[[countries.sources]]
title = "Gapminder Foundation - Life Expectancy"
path = "https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676"
version = "v14"
[[countries.sources]]
title = "Gapminder Foundation - Fertility"
path = "https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676"
version = "v14"
[countries.licenses]
name = "CC-BY-4.0"
path = "https://www.gapminder.org/free-material/"
title = "Creative Commons Attribution 4.0 International"
gapminder.description = """
This dataset combines key demographic indicators (life expectancy at birth, population, and fertility rate measured as babies per woman) for various countries from 1955 to 2005 at 5-year intervals. It also includes a 'cluster' column, a categorical variable grouping countries. Gapminder's [data documentation](https://www.gapminder.org/data/documentation/) notes that its philosophy is to fill data gaps with estimates and use current geographic boundaries for historical data. Gapminder states that it aims to "show people the big picture" rather than support detailed numeric analysis.
"""
[[gapminder.sources]]
title = "Gapminder Foundation - Life Expectancy"
path = "https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676"
version = "v14"
[[gapminder.sources]]
title = "Gapminder Foundation - Population"
path = "https://docs.google.com/spreadsheets/d/1c1luQNdpH90tNbMIeU7jD__59wQ0bdIGRFpbMm8ZBTk/edit?gid=176703676#gid=176703676"
version = "v7"
[[gapminder.sources]]
title = "Gapminder Foundation - Fertility"
path = "https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676"
version = "v14"
[[gapminder.sources]]
title = "Gapminder Foundation - Data Geographies"
path = "https://docs.google.com/spreadsheets/d/1qHalit8sXC0R8oVXibc2wa2gY7bkwGzOybEMTWp-08o/edit?gid=1597424158#gid=1597424158"
version = "v2"
[gapminder.licenses]
name = "CC-BY-4.0"
path = "https://www.gapminder.org/free-material/"
title = "Creative Commons Attribution 4.0 International"Here are the relevant excerpts from the data package definition and data resource definition sources
licenses
|
|
Looks good. Why are some entries in single [ and some in double [[? |
|
@dsmedia I like the look of the proposed I wanted to provide a comparison to what The main difference is that the bulk of the content is within
|
@domoritz interesting timing on this - hopefully the examples in (#634 (comment)) can explain the nesting. I'm not sure @dsmedia's sample would translate directly into the target schema. This is what
|
|
Got it. So something like the following? sources.toml$schema = "https://datapackage.org/profiles/2.0/datapackage.json"
# Package-level metadata using inline tables
name = "vega-datasets"
description = "Common repository for example datasets used by Vega related projects."
homepage = "http://github.com/vega/vega-datasets.git"
sources = [
{ path = "https://github.com/vega/vega-datasets/blob/next/SOURCES.md" },
]
contributors = [
{ title = "UW Interactive Data Lab", path = "http://idl.cs.washington.edu" },
]
version = "2.11.0"
created = "2024-12-01T15:50:47.863271+00:00"
[[licenses]]
name = "BSD-3-Clause"
path = "https://opensource.org/license/bsd-3-clause"
title = "The 3-Clause BSD License"
# Resources array
[[resources]]
name = "budget"
path = "budget.json"
description = "Budget FY 2016 - Receipts data from the Office of Management and Budget (U.S.)"
[[resources.sources]]
title = "Office of Management and Budget (U.S.)"
path = "https://www.govinfo.gov/app/details/BUDGET-2016-DB/BUDGET-2016-DB-3"
[[resources]]
name = "countries"
path = "countries.json"
description = """
This dataset combines key demographic indicators (life expectancy at birth and fertility rate measured as babies per woman) for various countries from 1955 to 2000 at 5-year intervals. It includes both current values and adjacent time period values (previous and next) for each indicator. Gapminder's [data documentation](https://www.gapminder.org/data/documentation/) notes that its philosophy is to fill data gaps with estimates and use current geographic boundaries for historical data. Gapminder states that it aims to "show people the big picture" rather than support detailed numeric analysis.
"""
[[resources.sources]]
title = "Gapminder Foundation - Life Expectancy"
path = "https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676"
version = "v14"
[[resources.sources]]
title = "Gapminder Foundation - Fertility"
path = "https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676"
version = "v14"
[[resources.licenses]]
name = "CC-BY-4.0"
path = "https://www.gapminder.org/free-material/"
title = "Creative Commons Attribution 4.0 International"
[[resources]]
name = "gapminder"
path = "gapminder.json"
description = """
This dataset combines key demographic indicators (life expectancy at birth, population, and fertility rate measured as babies per woman) for various countries from 1955 to 2005 at 5-year intervals. It also includes a 'cluster' column, a categorical variable grouping countries. Gapminder's [data documentation](https://www.gapminder.org/data/documentation/) notes that its philosophy is to fill data gaps with estimates and use current geographic boundaries for historical data. Gapminder states that it aims to "show people the big picture" rather than support detailed numeric analysis.
"""
[[resources.sources]]
title = "Gapminder Foundation - Life Expectancy"
path = "https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676"
version = "v14"
[[resources.sources]]
title = "Gapminder Foundation - Population"
path = "https://docs.google.com/spreadsheets/d/1c1luQNdpH90tNbMIeU7jD__59wQ0bdIGRFpbMm8ZBTk/edit?gid=176703676#gid=176703676"
version = "v7"
[[resources.sources]]
title = "Gapminder Foundation - Fertility"
path = "https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676"
version = "v14"
[[resources.sources]]
title = "Gapminder Foundation - Data Geographies"
path = "https://docs.google.com/spreadsheets/d/1qHalit8sXC0R8oVXibc2wa2gY7bkwGzOybEMTWp-08o/edit?gid=1597424158#gid=1597424158"
version = "v2"
[[resources.licenses]]
name = "CC-BY-4.0"
path = "https://www.gapminder.org/free-material/"
title = "Creative Commons Attribution 4.0 International"
|
|
@dsmedia #634 (comment) looks good
Happy for this to be moved out of version = "2.11.0"
created = "2024-12-01T15:50:47.863271+00:00"
Whatever is in the But I imagine it could be helpful to manually define the parts we discover are detected incorrectly like: Incorrect schema detection
vega-datasets/datapackage.json Lines 542 to 547 in 719c388 vega-datasets/datapackage.json Lines 1229 to 1234 in 719c388 vega-datasets/datapackage.json Lines 2065 to 2070 in 719c388 vega-datasets/datapackage.json Lines 2091 to 2096 in 719c388 vega-datasets/datapackage.json Lines 2425 to 2430 in 719c388 vega-datasets/datapackage.json Lines 2551 to 2556 in 719c388 vega-datasets/datapackage.json Lines 2681 to 2690 in 719c388 |
|
For the package authors, we might want to change all of our packages to be "the Vega organization" or something like that. Can we do that separately from this? |
dsmedia
changed the title
|
Thanks for the guidance above. For the purposes of resolving this issue, I'll plan to generate a toml file compatible with the existing script that contains a resource description, and, where available, source and license, for each dataset in the repo. This will produce a more complete datapackage.json file when the script is run. |
|
@dangotbanned will the below work for the script? I assume some modifications may need to be done to the script, so if it helps to modify the format of the toml file just let me know. TOML Generation NotesFile Structure
Content Handling
Future Improvements
sources.toml# Path: 7zip.png
[[resources]]
name = "7zip.png"
path = "7zip.png"
description = """Application icons from open-source software projects."""
# Path: airports.csv
[[resources]]
name = "airports.csv"
path = "airports.csv"
# Path: annual-precip.json
[[resources]]
name = "annual-precip.json"
path = "annual-precip.json"
description = """A raster grid of global annual precipitation for the year 2016 at a resolution 1 degree of lon/lat per cell, from [CFSv2](https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2)."""
[[resources.sources]]
title = "Climate Forecast System Version 2"
path = "https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2"
# Path: anscombe.json
[[resources]]
name = "anscombe.json"
path = "anscombe.json"
description = """Graphs in Statistical Analysis, F. J. Anscombe, The American Statistician."""
# Path: barley.json
[[resources]]
name = "barley.json"
path = "barley.json"
description = """The result of a 1930s agricultural experiment in Minnesota, this dataset contains yields for 10 different varieties of barley at six different sites. It was first published by agronomists F.R. Immer, H.K. Hayes, and L. Powers in the 1934 paper \"Statistical Determination of Barley Varietal Adaption.\" R.A. Fisher's popularized its use in the field of statistics when he included it in his book [\"The Design of Experiments.\"](https://en.wikipedia.org/wiki/The_Design_of_Experiments) Since then it has been used to demonstrate new statistical techniques, including the [trellis charts](http://ml.stat.purdue.edu/stat695t/writings/TrellisDesignControl.pdf) developed by Richard Becker, William Cleveland and others in the 1990s."""
[[resources.sources]]
title = "The Design of Experiments Reference"
path = "https://en.wikipedia.org/wiki/The_Design_of_Experiments"
[[resources.sources]]
title = "Trellis Charts Paper"
path = "http://ml.stat.purdue.edu/stat695t/writings/TrellisDesignControl.pdf"
# Path: birdstrikes.csv
[[resources]]
name = "birdstrikes.csv"
path = "birdstrikes.csv"
description = """http://wildlife.faa.gov"""
[[resources.sources]]
title = "FAA Wildlife Strike Database"
path = "http://wildlife.faa.gov"
# Path: budget.json
[[resources]]
name = "budget.json"
path = "budget.json"
description = """Source: Office of Management and Budget (U.S.)
[Budget FY 2016 - Receipts](https://www.govinfo.gov/app/details/BUDGET-2016-DB/BUDGET-2016-DB-3)"""
[[resources.sources]]
title = "Office of Management and Budget - Budget FY 2016"
path = "https://www.govinfo.gov/app/details/BUDGET-2016-DB/BUDGET-2016-DB-3"
# Path: budgets.json
[[resources]]
name = "budgets.json"
path = "budgets.json"
# Path: burtin.json
[[resources]]
name = "burtin.json"
path = "burtin.json"
description = """The burtin.json dataset is based on graphic designer [Will Burtin's](https://en.wikipedia.org/wiki/Will_Burtin) 1951 visualization of antibiotic effectiveness, originally published in [Scope Magazine](https://graphicdesignarchives.org/projects/scope-magazine-vol-iii-5/). The dataset compares the performance of three antibiotics against 16 different bacteria. The numerical values in the dataset represent the minimum inhibitory concentration (MIC) of each antibiotic, measured in units per milliliter, with lower values indicating higher antibiotic effectiveness. The dataset was featured as an [example](https://mbostock.github.io/protovis/ex/antibiotics-burtin.html) in the Protovis project, a precursor to D3.js. The Protovis example notes that, \"Recreating this display revealed some minor errors in the original: a missing grid line at 0.01 μg/ml, and an exaggeration of some values for penicillin.\" The vega-datsets version is largely consistent with the Protovis version of the dataset, with one correction (changing 'Brucella antracis' to the correct 'Bacillus anthracis') and the addition of a new column, 'Genus', to group related bacterial species together.
The caption of the original 1951 [visualization](https://graphicdesignarchives.org/wp-content/uploads/wmgda_8616c.jpg) reads as follows:
> ## Antibacterial ranges of Neomycin, Penicillin and Streptomycin
>
> The chart compares the in vitro sensitivities to neomycin of some of the common pathogens (gram+ in red and gram- in blue) with their sensitivities to penicillin, and streptomycin. The effectiveness of the antibiotics is expressed as the highest dilution in μ/ml. which inhibits the test organism. High dilutions are toward the periphery; consequently the length of the colored bar is proportional to the effectiveness. It is apparent that neomycin is especially effective against Staph. albus and aureus, Streph. fecalis, A. aerogenes, S. typhosa, E. coli, Ps. aeruginosa, Br. abortus, K. pneumoniae, Pr. vulgaris, S. schottmuelleri and M. tuberculosis. Unfortunately, some strains of proteus, pseudomonas and hemolytic streptococcus are resistant to neomycin, although the majority of these are sensitive to neomycin. It also inhibits actinomycetes, but is inactive against viruses and fungi. Its mode of action is not understood."""
[[resources.sources]]
title = "Scope Magazine"
path = "https://graphicdesignarchives.org/projects/scope-magazine-vol-iii-5/"
[[resources.sources]]
title = "Protovis Antibiotics Example"
path = "https://mbostock.github.io/protovis/ex/antibiotics-burtin.html"
# Path: cars.json
[[resources]]
name = "cars.json"
path = "cars.json"
description = """http://lib.stat.cmu.edu/datasets/"""
[[resources.sources]]
title = "StatLib Datasets Archive"
path = "http://lib.stat.cmu.edu/datasets/"
# Path: co2-concentration.csv
[[resources]]
name = "co2-concentration.csv"
path = "co2-concentration.csv"
description = """https://scrippsco2.ucsd.edu/data/atmospheric_co2/primary_mlo_co2_record but modified to only include date, CO2, seasonally adjusted CO2 and only include rows with valid data."""
[[resources.sources]]
title = "Scripps CO2 Program"
path = "https://scrippsco2.ucsd.edu/data/atmospheric_co2/primary_mlo_co2_record"
# Path: countries.json
[[resources]]
name = "countries.json"
path = "countries.json"
description = """- **Original Data**: [Gapminder Foundation](https://www.gapminder.org/)
- **URLs**:
- Life Expectancy (v14): [Data](https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676) | [Reference](https://www.gapminder.org/data/documentation/gd004/)
- Fertility (v14): [Data](https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676) | [Reference](https://www.gapminder.org/data/documentation/gd008/)
- **Date Accessed**: July 31, 2024
- **License**: Creative Commons Attribution 4.0 International (CC BY 4.0) | [Reference](https://www.gapminder.org/free-material/)
This dataset combines key demographic indicators (life expectancy at birth and fertility rate measured as babies per woman) for various countries from 1955 to 2000 at 5-year intervals. It includes both current values and adjacent time period values (previous and next) for each indicator. Gapminder's [data documentation](https://www.gapminder.org/data/documentation/) notes that its philosophy is to fill data gaps with estimates and use current geographic boundaries for historical data. Gapminder states that it aims to \"show people the big picture\" rather than support detailed numeric analysis.
1. `year` (type: integer): Years from 1955 to 2000 at 5-year intervals
2. `country` (type: string): Name of the country
3. `fertility` (type: float): Fertility rate (average number of children per woman) for the given year
4. `life_expect` (type: float): Life expectancy in years for the given year
5. `p_fertility` (type: float): Fertility rate for the previous 5-year interval
6. `n_fertility` (type: float): Fertility rate for the next 5-year interval
7. `p_life_expect` (type: float): Life expectancy for the previous 5-year interval
8. `n_life_expect` (type: float): Life expectancy for the next 5-year interval"""
[[resources.sources]]
title = "Gapminder Foundation - Life Expectancy"
path = "https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676"
version = "14"
[[resources.sources]]
title = "Gapminder Foundation - Fertility"
path = "https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676"
version = "14"
[[resources.licenses]]
name = "countries.json"
title = "Creative Commons Attribution 4.0 International"
path = "https://www.gapminder.org/free-material/"
# Path: crimea.json
[[resources]]
name = "crimea.json"
path = "crimea.json"
# Path: disasters.csv
[[resources]]
name = "disasters.csv"
path = "disasters.csv"
description = """https://ourworldindata.org/natural-catastrophes"""
[[resources.sources]]
title = "Our World in Data - Natural Catastrophes"
path = "https://ourworldindata.org/natural-catastrophes"
# Path: driving.json
[[resources]]
name = "driving.json"
path = "driving.json"
description = """https://archive.nytimes.com/www.nytimes.com/imagepages/2010/05/02/business/02metrics.html"""
[[resources.sources]]
title = "New York Times"
path = "https://archive.nytimes.com/www.nytimes.com/imagepages/2010/05/02/business/02metrics.html"
# Path: earthquakes.json
[[resources]]
name = "earthquakes.json"
path = "earthquakes.json"
description = """https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson
(Feb 6, 2018)"""
[[resources.sources]]
title = "USGS Earthquake Feed"
path = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson"
# Path: ffox.png
[[resources]]
name = "ffox.png"
path = "ffox.png"
description = """Application icons from open-source software projects."""
# Path: flare-dependencies.json
[[resources]]
name = "flare-dependencies.json"
path = "flare-dependencies.json"
# Path: flare.json
[[resources]]
name = "flare.json"
path = "flare.json"
# Path: flights-10k.json
[[resources]]
name = "flights-10k.json"
path = "flights-10k.json"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-200k.arrow
[[resources]]
name = "flights-200k.arrow"
path = "flights-200k.arrow"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-200k.json
[[resources]]
name = "flights-200k.json"
path = "flights-200k.json"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-20k.json
[[resources]]
name = "flights-20k.json"
path = "flights-20k.json"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-2k.json
[[resources]]
name = "flights-2k.json"
path = "flights-2k.json"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-3m.parquet
[[resources]]
name = "flights-3m.parquet"
path = "flights-3m.parquet"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-5k.json
[[resources]]
name = "flights-5k.json"
path = "flights-5k.json"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: flights-airport.csv
[[resources]]
name = "flights-airport.csv"
path = "flights-airport.csv"
description = """Flight delay statistics from U.S. Bureau of Transportation Statistics. https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
Transformed using `/scripts/flights.py`"""
[[resources.sources]]
title = "U.S. Bureau of Transportation Statistics"
path = "https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr"
# Path: football.json
[[resources]]
name = "football.json"
path = "football.json"
description = """Football match outcomes across multiple divisions from 2013 to 2017. This dataset is a subset of a larger dataset from https://github.com/openfootball/football.json. The subset was made such that there are records for all five chosen divisions over the time period."""
[[resources.sources]]
title = "OpenFootball"
path = "https://github.com/openfootball/football.json"
# Path: gapminder-health-income.csv
[[resources]]
name = "gapminder-health-income.csv"
path = "gapminder-health-income.csv"
description = """**Original Data**: [Gapminder Foundation](https://www.gapminder.org/)
**Description** Per-capita income, life expectancy, population and regional grouping. Dataset does not specify the reference year for the data. Gapminder historical data is subject to revisions.
Gapminder (v30, 2023) defines per-capita income as follows:
>\"This is real GDP per capita (gross domestic product per person adjusted for inflation) converted to international dollars using purchasing power parity rates. An international dollar has the same purchasing power over GDP as the U.S. dollar has in the United States.\" | [Source](https://docs.google.com/spreadsheets/d/1i5AEui3WZNZqh7MQ4AKkJuCz4rRxGR_pw_9gtbcBOqQ/edit?gid=501532268#gid=501532268)
**License**: Creative Commons Attribution 4.0 International (CC BY 4.0) | [Reference](https://www.gapminder.org/free-material/)"""
[[resources.sources]]
title = "Gapminder Foundation"
path = "https://www.gapminder.org"
[[resources.sources]]
title = "Gapminder GDP Per Capita Data"
path = "https://docs.google.com/spreadsheets/d/1i5AEui3WZNZqh7MQ4AKkJuCz4rRxGR_pw_9gtbcBOqQ/edit?gid=501532268#gid=501532268"
# Path: gapminder.json
[[resources]]
name = "gapminder.json"
path = "gapminder.json"
description = """- **Original Data**: [Gapminder Foundation](https://www.gapminder.org/)
- **URLs**:
- Life Expectancy (v14): [Data](https://docs.google.com/spreadsheets/d/1RehxZjXd7_rG8v2pJYV6aY0J3LAsgUPDQnbY4dRdiSs/edit?gid=176703676#gid=176703676) | [Reference](https://www.gapminder.org/data/documentation/gd004/)
- Population (v7): [Data](https://docs.google.com/spreadsheets/d/1c1luQNdpH90tNbMIeU7jD__59wQ0bdIGRFpbMm8ZBTk/edit?gid=176703676#gid=176703676) | [Reference](https://www.gapminder.org/data/documentation/gd003/)
- Fertility (v14): [Data](https://docs.google.com/spreadsheets/d/1aLtIpAWvDGGa9k2XXEz6hZugWn0wCd5nmzaRPPjbYNA/edit?gid=176703676#gid=176703676) | [Reference](https://www.gapminder.org/data/documentation/gd008/)
- Data Geographies (v2): [Data](https://docs.google.com/spreadsheets/d/1qHalit8sXC0R8oVXibc2wa2gY7bkwGzOybEMTWp-08o/edit?gid=1597424158#gid=1597424158) | [Reference](https://www.gapminder.org/data/geo/)
- **Date Accessed**: July 11, 2024
- **License**: Creative Commons Attribution 4.0 International (CC BY 4.0) | [Reference](https://www.gapminder.org/free-material/)
This dataset combines key demographic indicators (life expectancy at birth, population, and fertility rate measured as babies per woman) for various countries from 1955 to 2005 at 5-year intervals. It also includes a 'cluster' column, a categorical variable grouping countries. Gapminder's [data documentation](https://www.gapminder.org/data/documentation/) notes that its philosophy is to fill data gaps with estimates and use current geographic boundaries for historical data. Gapminder states that it aims to \"show people the big picture\" rather than support detailed numeric analysis.
1. `year` (type: integer): Years from 1955 to 2005 at 5-year intervals
2. `country` (type: string): Name of the country
3. `cluster` (type: integer): A categorical variable (values 0-5) grouping countries. See Revision Notes for details.
4. `pop` (type: integer): Population of the country
5. `life_expect` (type: float): Life expectancy in years
6. `fertility` (type: float): Fertility rate (average number of children per woman)
1. Country Selection: The set of countries in this file matches the version of this dataset originally added to this collection in 2015. The specific criteria for country selection in that version are not known. Data for Aruba are no longer available in the new version. Hong Kong has been revised to Hong Kong, China in the new version.
2. Data Precision: The precision of float values may have changed from the original version. These changes reflect the most recent source data used for each indicator.
3. Regional Groupings: The 'cluster' column represents a regional mapping of countries corresponding to the 'six_regions' schema in Gapminder's Data Geographies dataset. To preserve continuity with previous versions of this dataset, we have retained the column name 'cluster' instead of renaming it to 'six_regions'. The six regions represented are:
`0: south_asia, 1: europe_central_asia, 2: sub_saharan_africa, 3: america, 4: east_asia_pacific, 5: middle_east_north_africa`."""
[[resources.sources]]
title = "Gapminder Foundation - Population"
path = "https://docs.google.com/spreadsheets/d/1c1luQNdpH90tNbMIeU7jD__59wQ0bdIGRFpbMm8ZBTk/edit?gid=176703676#gid=176703676"
version = "7"
[[resources.sources]]
title = "Gapminder Foundation - Data Geographies"
path = "https://docs.google.com/spreadsheets/d/1qHalit8sXC0R8oVXibc2wa2gY7bkwGzOybEMTWp-08o/edit?gid=1597424158#gid=1597424158"
version = "2"
[[resources.sources]]
title = "Gapminder Data Documentation"
path = "https://www.gapminder.org/data/documentation/"
# Path: gimp.png
[[resources]]
name = "gimp.png"
path = "gimp.png"
description = """Application icons from open-source software projects."""
# Path: github.csv
[[resources]]
name = "github.csv"
path = "github.csv"
description = """Generated using `/scripts/github.py`."""
# Path: global-temp.csv
[[resources]]
name = "global-temp.csv"
path = "global-temp.csv"
description = """Combined Land-Surface Air and Sea-Surface Water Temperature Anomalies (Land-Ocean Temperature Index, L-OTI), 1880-2023. Source: NASA's Goddard Institute for Space Studies https://data.giss.nasa.gov/gistemp/"""
[[resources.sources]]
title = "NASA Goddard Institute for Space Studies"
path = "https://data.giss.nasa.gov/gistemp/"
# Path: income.json
[[resources]]
name = "income.json"
path = "income.json"
# Path: iowa-electricity.csv
[[resources]]
name = "iowa-electricity.csv"
path = "iowa-electricity.csv"
description = """The state of Iowa has dramatically increased its production of renewable wind power in recent years. This file contains the annual net generation of electricity in the state by source in thousand megawatthours. The dataset was compiled by the [U.S. Energy Information Administration](https://www.eia.gov/beta/electricity/data/browser/#/topic/0?agg=2,0,1&fuel=vvg&geo=00000g&sec=g&linechart=ELEC.GEN.OTH-IA-99.A~ELEC.GEN.COW-IA-99.A~ELEC.GEN.PEL-IA-99.A~ELEC.GEN.PC-IA-99.A~ELEC.GEN.NG-IA-99.A~~ELEC.GEN.NUC-IA-99.A~ELEC.GEN.HYC-IA-99.A~ELEC.GEN.AOR-IA-99.A~ELEC.GEN.HPS-IA-99.A~&columnchart=ELEC.GEN.ALL-IA-99.A&map=ELEC.GEN.ALL-IA-99.A&freq=A&start=2001&end=2017&ctype=linechart<ype=pin&tab=overview&maptype=0&rse=0&pin=) and downloaded on May 6, 2018. It is useful for illustrating stacked area charts."""
[[resources.sources]]
title = "U.S. Energy Information Administration"
path = "https://www.eia.gov/beta/electricity/data/browser/#/topic/0?agg=2,0,1&fuel=vvg&geo=00000g&sec=g&linechart=ELEC.GEN.OTH-IA-99.A~ELEC.GEN.COW-IA-99.A~ELEC.GEN.PEL-IA-99.A~ELEC.GEN.PC-IA-99.A~ELEC.GEN.NG-IA-99.A~~ELEC.GEN.NUC-IA-99.A~ELEC.GEN.HYC-IA-99.A~ELEC.GEN.AOR-IA-99.A~ELEC.GEN.HPS-IA-99.A~&columnchart=ELEC.GEN.ALL-IA-99.A&map=ELEC.GEN.ALL-IA-99.A&freq=A&start=2001&end=2017&ctype=linechart<ype=pin&tab=overview&maptype=0&rse=0&pin="
# Path: jobs.json
[[resources]]
name = "jobs.json"
path = "jobs.json"
description = """U.S. census data on [occupations](https://usa.ipums.org/usa-action/variables/OCC1950#codes_section) by sex and year across decades between 1850 and 2000. The dataset was obtained from IPUMS USA, which \"collects, preserves and harmonizes U.S. census microdata\" from as early as 1790.
Originally created for a 2006 data visualization project called *sense.us* by IBM Research (Jeff Heer, Martin Wattenberg and Fernanda Viégas), described [here](https://homes.cs.washington.edu/~jheer/files/bdata_ch12.pdf). The dataset is also referenced in this vega [example](https://vega.github.io/vega/examples/job-voyager/).
Data is based on a tabulation of the [OCC1950](https://usa.ipums.org/usa-action/variables/OCC1950) variable by sex across IPUMS USA samples. The dataset appears to be derived from Version 6.0 (2015) of [IPUMS USA](https://usa.ipums.org/usa/), according to 2024 correspondence with the IPUMS Project. IPUMS has made improvements to occupation coding since version 6, particularly for 19th-century samples, which may result in discrepancies between this dataset and current IPUMS data. Details on data revisions are available [here](https://usa.ipums.org/usa-action/revisions).
The dataset is structured as follows:
- job: The occupation title
- sex: Sex (men/women)
- year: Census year
- count: Number of individuals in the occupation
- perc: Percentage of the workforce in the occupation
IPUMS USA confirmed in 2024 correspondence that hosting this dataset on vega-datasets is permissible, stating:
>We're excited to hear that this dataset made its way to this repository and is being used by students for data visualization. We allow for these types of redistributions of summary data so long as the underlying microdata records are not shared.
This dataset contains only summary statistics and does not include any underlying microdata records.
1. This dataset represents summary data. The underlying microdata records are not included.
2. Users attempting to replicate or extend this data should use the [PERWT](https://usa.ipums.org/usa-action/variables/PERWT#description_section) (person weight) variable as an expansion factor when working with IPUMS USA extracts.
3. Due to coding revisions, figures for earlier years (particularly 19th century) may not match current IPUMS USA data exactly.
When using this dataset, please refer to IPUMS USA [terms of use](https://usa.ipums.org/usa/terms.shtml). The organization requests use of the following citation for this json file:
Steven Ruggles, Katie Genadek, Ronald Goeken, Josiah Grover, and Matthew Sobek. Integrated Public Use Microdata Series: Version 6.0. Minneapolis: University of Minnesota, 2015. http://doi.org/10.18128/D010.V6.0"""
[[resources.sources]]
title = "IPUMS USA"
path = "https://usa.ipums.org/usa/"
# Path: la-riots.csv
[[resources]]
name = "la-riots.csv"
path = "la-riots.csv"
description = """More than 60 people lost their lives amid the looting and fires that ravaged Los Angeles for five days starting on April 29, 1992. This file contains metadata about each person, including the geographic coordinates of their death. It was compiled and published by the [Los Angeles Times Data Desk](http://spreadsheets.latimes.com/la-riots-deaths/)."""
[[resources.sources]]
title = "Los Angeles Times Data Desk"
path = "http://spreadsheets.latimes.com/la-riots-deaths/"
# Path: londonBoroughs.json
[[resources]]
name = "londonBoroughs.json"
path = "londonBoroughs.json"
description = """Boundaries of London boroughs reprojected and simplified from `London_Borough_Excluding_MHW` shapefile held at https://data.london.gov.uk/dataset/statistical-gis-boundary-files-london. Original data \"contains National Statistics data © Crown copyright and database right (2015)\" and \"Contains Ordnance Survey data © Crown copyright and database right [2015]."""
[[resources.sources]]
title = "London Datastore"
path = "https://data.london.gov.uk/dataset/statistical-gis-boundary-files-london"
# Path: londonCentroids.json
[[resources]]
name = "londoncentroids.json"
path = "londonCentroids.json"
description = """Calculated from `londongBoroughs.json` using `d3.geoCentroid`."""
# Path: londonTubeLines.json
[[resources]]
name = "londontubeLines.json"
path = "londonTubeLines.json"
description = """Selected rail lines simplified from `tfl_lines.json` at https://github.com/oobrien/vis/tree/master/tube/data"""
[[resources.sources]]
title = "London Tube Data"
path = "https://github.com/oobrien/vis/tree/master/tube/data"
# Path: lookup_groups.csv
[[resources]]
name = "lookup_groups.csv"
path = "lookup_groups.csv"
# Path: lookup_people.csv
[[resources]]
name = "lookup_people.csv"
path = "lookup_people.csv"
# Path: miserables.json
[[resources]]
name = "miserables.json"
path = "miserables.json"
# Path: monarchs.json
[[resources]]
name = "monarchs.json"
path = "monarchs.json"
description = """A chronological list of English and British monarchs from Elizabeth I through George IV.
Each entry includes:
- `name`: The ruler's name or identifier (e.g., \"W&M\" for William and Mary, \"Cromwell\" for the period of interregnum)
- `start`: The year their rule began.
- `end`: The year their rule ended
- `index`: A [zero-based sequential number](https://en.wikipedia.org/wiki/Zero-based_numbering) assigned to each entry, representing the chronological order of rulers
- `commonwealth`: A Boolean flag (true) for the period from 1649 to 1660. This field is omitted for all other entries.
The dataset contains two intentional inaccuracies to maintain compatibility with the [Wheat and Wages](https://vega.github.io/vega/examples/wheat-and-wages/) example visualization:
1. the start date for the reign of Elizabeth I is shown as 1565, instead of 1558;
2. the end date for the reign of George IV is shown as 1820, instead of 1830.
These discrepancies align the `monarchs.json` dataset with the start and end dates of the `wheat.json` dataset used i the visualization.
The entry \"W&M\" represents the joint reign of William III and Mary II. While the dataset shows their reign as 1689-1702, the official Web site of the British royal family indicates that Mary II's reign ended in 1694, though William III continued to rule until 1702.
The `commonwealth` field is used to flag the period from 1649 to 1660, which includes the Commonwealth of England, the Protectorate, and the period leading to the Restoration. While historically more accurate to call this the \"interregnum,\" the field name of `commonwealth` from the original dataset is retained for backwards compatibility.
The dataset was revised in Aug. 2024. James II's reign now ends in 1688 (previously 1689).
Source data has been verified against the [kings & queens](https://www.royal.uk/kings-and-queens-1066) and [interregnum](https://www.royal.uk/interregnum-1649-1660) [official website of the British royal family](https://www.royal.uk) pages of the official Web site of the British royal family (retrieved in Aug. 2024). Content on the site is protected by Crown Copyright. Under the [UK Government Licensing Framework](https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/crown-copyright/), most Crown copyright information is available under the [Open Government Licence v3.0](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/)."""
[[resources.sources]]
title = "The Royal Family"
path = "https://www.royal.uk/kings-and-queens-1066"
# Path: movies.json
[[resources]]
name = "movies.json"
path = "movies.json"
description = """The dataset has well known and intentionally included errors. This dataset is used for instructional purposes, including the need to reckon with dirty data."""
# Path: normal-2d.json
[[resources]]
name = "normal-2d.json"
path = "normal-2d.json"
# Path: obesity.json
[[resources]]
name = "obesity.json"
path = "obesity.json"
# Path: ohlc.json
[[resources]]
name = "ohlc.json"
path = "ohlc.json"
description = """This dataset contains the performance of the Chicago Board Options Exchange [Volatility Index](https://en.wikipedia.org/wiki/VIX) ([VIX](https://finance.yahoo.com/chart/%5EVIX?ltr=1#eyJpbnRlcnZhbCI6ImRheSIsInBlcmlvZGljaXR5IjoxLCJ0aW1lVW5pdCI6bnVsbCwiY2FuZGxlV2lkdGgiOjgsInZvbHVtZVVuZGVybGF5Ijp0cnVlLCJhZGoiOnRydWUsImNyb3NzaGFpciI6dHJ1ZSwiY2hhcnRUeXBlIjoibGluZSIsImV4dGVuZGVkIjpmYWxzZSwibWFya2V0U2Vzc2lvbnMiOnt9LCJhZ2dyZWdhdGlvblR5cGUiOiJvaGxjIiwiY2hhcnRTY2FsZSI6ImxpbmVhciIsInN0dWRpZXMiOnsidm9sIHVuZHIiOnsidHlwZSI6InZvbCB1bmRyIiwiaW5wdXRzIjp7ImlkIjoidm9sIHVuZHIiLCJkaXNwbGF5Ijoidm9sIHVuZHIifSwib3V0cHV0cyI6eyJVcCBWb2x1bWUiOiIjMDBiMDYxIiwiRG93biBWb2x1bWUiOiIjRkYzMzNBIn0sInBhbmVsIjoiY2hhcnQiLCJwYXJhbWV0ZXJzIjp7IndpZHRoRmFjdG9yIjowLjQ1LCJjaGFydE5hbWUiOiJjaGFydCJ9fX0sInBhbmVscyI6eyJjaGFydCI6eyJwZXJjZW50IjoxLCJkaXNwbGF5IjoiXlZJWCIsImNoYXJ0TmFtZSI6ImNoYXJ0IiwidG9wIjowfX0sInNldFNwYW4iOnt9LCJsaW5lV2lkdGgiOjIsInN0cmlwZWRCYWNrZ3JvdWQiOnRydWUsImV2ZW50cyI6dHJ1ZSwiY29sb3IiOiIjMDA4MWYyIiwiZXZlbnRNYXAiOnsiY29ycG9yYXRlIjp7ImRpdnMiOnRydWUsInNwbGl0cyI6dHJ1ZX0sInNpZ0RldiI6e319LCJzeW1ib2xzIjpbeyJzeW1ib2wiOiJeVklYIiwic3ltYm9sT2JqZWN0Ijp7InN5bWJvbCI6Il5WSVgifSwicGVyaW9kaWNpdHkiOjEsImludGVydmFsIjoiZGF5IiwidGltZVVuaXQiOm51bGwsInNldFNwYW4iOnt9fV19)) in the summer of 2009."""
[[resources.sources]]
title = "Yahoo Finance VIX Data"
path = "https://finance.yahoo.com/chart/%5EVIX"
# Path: penguins.json
[[resources]]
name = "penguins.json"
path = "penguins.json"
description = """Palmer Archipelago (Antarctica) penguin data collected and made available by [Dr. Kristen Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php) and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a member of the [Long Term Ecological Research Network](https://lternet.edu/). For more information visit [allisonhorst/penguins](https://github.com/allisonhorst/penguins) on GitHub."""
[[resources.sources]]
title = "Palmer Station Antarctica LTER"
path = "https://pal.lternet.edu/"
[[resources.sources]]
title = "Allison Horst's Penguins Repository"
path = "https://github.com/allisonhorst/penguins"
# Path: platformer-terrain.json
[[resources]]
name = "platformer-terrain.json"
path = "platformer-terrain.json"
description = """Assets from the video game [Celeste](http://www.celestegame.com/)."""
[[resources.sources]]
title = "Celeste Game"
path = "http://www.celestegame.com/"
# Path: points.json
[[resources]]
name = "points.json"
path = "points.json"
# Path: political-contributions.json
[[resources]]
name = "political-contributions.json"
path = "political-contributions.json"
description = """Summary financial information on contributions to candidates for U.S. elections. An updated version of this datset is available from the \"all candidates\" files (in pipe-delimited format) on the [bulk data download](https://www.fec.gov/data/browse-data/?tab=bulk-data) page of the U.S. Federal Election Commission, or, alternatively, via [OpenFEC](https://api.open.fec.gov/developers/). Information on each of the 25 columns is available from the [FEC All Candidates File Description](https://www.fec.gov/campaign-finance-data/all-candidates-file-description/). The sample dataset in `political-contributions.json` contains 58 records with dates from 2015.
FEC data is subject to the commission's:
- [Sale or Use Policy](https://www.fec.gov/updates/sale-or-use-contributor-information/)
- [Privacy and Security Policy](https://www.fec.gov/about/privacy-and-security-policy/)
- [Acceptable Use Policy](https://github.com/fecgov/FEC/blob/master/ACCEPTABLE-USE-POLICY.md)
Additionally, the FEC's Github [repository](https://github.com/fecgov/FEC) states:
> This project is in the public domain within the United States, and we waive worldwide copyright and related rights through [CC0 universal public domain](https://creativecommons.org/publicdomain/zero/1.0/) dedication. Read more on our license page. A few restrictions limit the way you can use FEC data. For example, you can't use contributor lists for commercial purposes or to solicit donations. Learn more on [FEC.gov](https://www.fec.gov/)."""
[[resources.sources]]
title = "Federal Election Commission Bulk Data"
path = "https://www.fec.gov/data/browse-data/?tab=bulk-data"
[[resources.sources]]
title = "OpenFEC API"
path = "https://api.open.fec.gov/developers/"
# Path: population.json
[[resources]]
name = "population.json"
path = "population.json"
description = """United States population statistics by sex and age group across decades between 1850 and 2000. The dataset was obtained from [IPUMS USA](https://usa.ipums.org/usa/), which \"collects, preserves and harmonizes U.S. census microdata\" from as early as 1790.
The dataset is structured as follows:
- year: four-digit year of the survey. - [IPUMS description](https://usa.ipums.org/usa-action/variables/YEAR#description_section)
- age: age group in 5-year intervals (0 represents ages 0-4, 5 represents 5-9, 10 represents 10-14, etc., up to 90 representing 90 and above) - [IPUMS description](https://usa.ipums.org/usa-action/variables/AGE#description_section)
- sex: Sex (men = 1 / women = 2) - [IPUMS description](https://usa.ipums.org/usa-action/variables/SEX#description_section)
- people: Number of individuals, equivalent to IPUMS variable name [PERWT](https://usa.ipums.org/usa-action/variables/PERWT#description_section).
IPUMS updates and revises datasets over time, which may result in discrepancies between this dataset and current IPUMS data. Details on data revisions are available [here](https://usa.ipums.org/usa-action/revisions).
When using this dataset, please refer to IPUMS USA [terms of use](https://usa.ipums.org/usa/terms.shtml). The organization requests the use of the following citation for this json file:
Steven Ruggles, Katie Genadek, Ronald Goeken, Josiah Grover, and Matthew Sobek. Integrated Public Use Microdata Series: Version 6.0. Minneapolis: University of Minnesota, 2015. http://doi.org/10.18128/D010.V6.0"""
[[resources.sources]]
title = "IPUMS USA"
path = "https://usa.ipums.org/usa/"
# Path: population_engineers_hurricanes.csv
[[resources]]
name = "population_engineers_hurricanes.csv"
path = "population_engineers_hurricanes.csv"
description = """Data about engineers from https://www.bls.gov/oes/tables.htm. Hurricane data from http://www.nhc.noaa.gov/paststate.shtml. Income data from https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ACS_07_3YR_S1901&prodType=table."""
[[resources.sources]]
title = "Bureau of Labor Statistics"
path = "https://www.bls.gov/oes/tables.htm"
[[resources.sources]]
title = "American Community Survey"
path = "https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ACS_07_3YR_S1901&prodType=table"
[[resources.sources]]
title = "NOAA National Climatic Data Center"
path = "https://www.ncdc.noaa.gov/cdo-web/datatools/records"
# Path: seattle-weather-hourly-normals.csv
[[resources]]
name = "seattle-weather-hourly-normals.csv"
path = "seattle-weather-hourly-normals.csv"
description = """Data from [NOAA](https://www.ncdc.noaa.gov/cdo-web/datatools/normals). Hourly weather normals with metric units. The 1981-2010 Climate Normals are NCDC's three-decade averages of climatological variables, including temperature and precipitation. Learn more in the [documentation](https://www1.ncdc.noaa.gov/pub/data/cdo/documentation/NORMAL_HLY_documentation.pdf). We only included temperature, wind, and pressure and updated the format to be easier to parse."""
[[resources.sources]]
title = "NOAA National Climatic Data Center (NCDC)"
path = "https://www.ncdc.noaa.gov/cdo-web/datatools/normals"
# Path: seattle-weather.csv
[[resources]]
name = "seattle-weather.csv"
path = "seattle-weather.csv"
description = """Data from [NOAA](https://www.ncdc.noaa.gov/cdo-web/datatools/records). Daily weather records with metric units. Transformed using `/scripts/weather.py`. We synthesized the categorical \"weather\" field from multiple fields in the original dataset. This data is intended for instructional purposes."""
[[resources.sources]]
title = "NOAA National Climatic Data Center"
path = "https://www.ncdc.noaa.gov/cdo-web/datatools/records"
# Path: sp500-2000.csv
[[resources]]
name = "sp500-2000.csv"
path = "sp500-2000.csv"
description = """S&P 500 index values from 2000 to 2020, retrieved from [Yahoo Finance](https://finance.yahoo.com/quote/%5EDJI/history/)."""
[[resources.sources]]
title = "Yahoo Finance"
path = "https://finance.yahoo.com/quote/%5EDJI/history/"
# Path: sp500.csv
[[resources]]
name = "sp500.csv"
path = "sp500.csv"
# Path: stocks.csv
[[resources]]
name = "stocks.csv"
path = "stocks.csv"
# Path: udistrict.json
[[resources]]
name = "udistrict.json"
path = "udistrict.json"
# Path: unemployment-across-industries.json
[[resources]]
name = "unemployment-across-industries.json"
path = "unemployment-across-industries.json"
description = """Industry-level unemployment statistics from the [Current Population Survey](https://www.census.gov/programs-surveys/cps.html) (CPS), published monthly by the U.S. Bureau of Labor Statistics. Includes unemployed persons and unemployment rate across 11 private industries, as well as agricultural, government, and self-employed workers. Covers January 2000 through February 2010. Industry classification follows format of CPS [Table A-31](https://www.bls.gov/web/empsit/cpseea31.htm).
Each entry in the JSON file contains:
- `series`: Industry name
- `year`: Year (2000-2010)
- `month`: Month (1-12)
- `count`: Number of unemployed persons (in thousands)
- `rate`: Unemployment rate (percentage)
- `date`: [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html)-formatted date string (e.g., \"2000-01-01T08:00:00.000Z\")
The dataset can be replicated using the BLS API. For more, see the `scripts` folder of this repository.
The BLS Web site states:
> \"Users of the public API should cite the date that data were accessed or retrieved using the API. Users must clearly state that “BLS.gov cannot vouch for the data or analyses derived from these data after the data have been retrieved from BLS.gov.” The BLS.gov logo may not be used by persons who are not BLS employees or on products (including web pages) that are not BLS-sponsored.\"
See full BLS [terms of service](https://www.bls.gov/developers/termsOfService.htm)."""
[[resources.sources]]
title = "U.S. Census Bureau Current Population Survey"
path = "https://www.census.gov/programs-surveys/cps.html"
[[resources.sources]]
title = "BLS Local Area Unemployment Statistics"
path = "https://www.bls.gov/lau/"
[[resources.sources]]
title = "BLS LAUS Data Tools"
path = "https://www.bls.gov/lau/data.htm"
[[resources.sources]]
title = "Bureau of Labor Statistics Table A-31"
path = "https://www.bls.gov/web/empsit/cpseea31.htm"
# Path: unemployment.tsv
[[resources]]
name = "unemployment.tsv"
path = "unemployment.tsv"
description = """This dataset contains county-level unemployment rates in the United States, with data generally consistent with levels reported in 2009. The dataset is structured as tab-separated values with two columns:
1. `id`: The combined [state and county FIPS code](https://www.census.gov/library/reference/code-lists/ansi.html)
2. `rate`: The unemployment rate for the county
The unemployment rate represents the number of unemployed persons as a percentage of the labor force. According to the [Bureau of Labor Statistics (BLS) glossary](https://www.bls.gov/opub/hom/glossary.htm#U):
> Unemployed persons (Current Population Survey) [are] persons aged 16 years and older who had no employment during the reference week, were available for work, except for temporary illness, and had made specific efforts to find employment sometime during the 4-week period ending with the reference week. Persons who were waiting to be recalled to a job from which they had been laid off need not have been looking for work to be classified as unemployed.
The labor force includes all persons classified as employed or unemployed in accordance with the BLS definitions.
This dataset is derived from the [Local Area Unemployment Statistics (LAUS)](https://www.bls.gov/lau/) program, a federal-state cooperative effort overseen by the Bureau of Labor Statistics (BLS). The LAUS program produces monthly and annual employment, unemployment, and labor force data for census regions and divisions, states, counties, metropolitan areas, and many cities and towns.
For the most up-to-date LAUS data:
1. **Monthly and Annual Data Downloads**:
- Visit the [LAUS Data Tools](https://www.bls.gov/lau/data.htm) page for [monthly](https://www.bls.gov/lau/tables.htm#mcounty) and [annual](https://www.bls.gov/lau/tables.htm#cntyaa) county data.
2. **BLS Public Data API**:
- The BLS provides an [API for developers](https://www.bls.gov/developers/) to access various datasets, including LAUS data.
- To use the API for LAUS data, refer to the [LAUS Series ID Formats](https://www.bls.gov/help/hlpforma.htm#LA) to construct your query.
- API documentation and examples are available on the [BLS Developers](https://www.bls.gov/developers/) page.
When using BLS public data API and datasets, users should adhere to the [BLS Terms of Service](https://www.bls.gov/developers/termsOfService.htm), which includes the following guidelines:
1. Cite the date that data were accessed or retrieved.
2. Acknowledge that \"BLS.gov cannot vouch for the data or analyses derived from these data after the data have been retrieved from BLS.gov.\"
3. Do not use the BLS logo without permission.
For detailed methodology and technical information about LAUS estimates, refer to the [BLS Handbook of Methods](https://www.bls.gov/opub/hom/lau/home.htm)."""
[[resources.sources]]
title = "BLS Handbook of Methods"
path = "https://www.bls.gov/opub/hom/lau/home.htm"
[[resources.sources]]
title = "BLS Developers API"
path = "https://www.bls.gov/developers/"
# Path: uniform-2d.json
[[resources]]
name = "uniform-2d.json"
path = "uniform-2d.json"
# Path: us-10m.json
[[resources]]
name = "us-10m.json"
path = "us-10m.json"
# Path: us-employment.csv
[[resources]]
name = "us-employment.csv"
path = "us-employment.csv"
description = """In the mid 2000s the global economy was hit by a crippling recession. One result: Massive job losses across the United States. The downturn in employment, and the slow recovery in hiring that followed, was tracked each month by the [Current Employment Statistics](https://www.bls.gov/ces/) program at the U.S. Bureau of Labor Statistics.
This file contains the monthly employment total in a variety of job categories from January 2006 through December 2015. The numbers are seasonally adjusted and reported in thousands. The data were downloaded on Nov. 11, 2018, and reformatted for use in this library.
Totals are included for the [22 \"supersectors\"](https://download.bls.gov/pub/time.series/ce/ce.supersector) tracked by the BLS. The \"nonfarm\" total is the category typically used by economists and journalists as a stand-in for the country's employment total.
A calculated \"nonfarm_change\" column has been appended with the month-to-month change in that supersector's employment. It is useful for illustrating how to make bar charts that report both negative and positive values."""
[[resources.sources]]
title = "U.S. Bureau of Labor Statistics Current Employment Statistics"
path = "https://www.bls.gov/ces/"
[[resources.sources]]
title = "BLS Supersectors"
path = "https://download.bls.gov/pub/time.series/ce/ce.supersector"
# Path: us-state-capitals.json
[[resources]]
name = "us-state-capitals.json"
path = "us-state-capitals.json"
# Path: volcano.json
[[resources]]
name = "volcano.json"
path = "volcano.json"
description = """Maunga Whau (Mt Eden) is one of about 50 volcanos in the Auckland volcanic field. This data set gives topographic information for Maunga Whau on a 10m by 10m grid. Digitized from a topographic map by Ross Ihaka, adapted from [R datasets](https://stat.ethz.ch/R-manual/R-patched/library/datasets/html/volcano.html). These data should not be regarded as accurate."""
[[resources.sources]]
title = "R Datasets"
path = "https://stat.ethz.ch/R-manual/R-patched/library/datasets/html/volcano.html"
# Path: weather.csv
[[resources]]
name = "weather.csv"
path = "weather.csv"
description = """Data from [NOAA](http://www.ncdc.noaa.gov/cdo-web/datatools/findstation). Transformed using `/scripts/weather.py`. We synthesized the categorical \"weather\" field from multiple fields in the original dataset. This data is intended for instructional purposes."""
[[resources.sources]]
title = "NOAA Climate Data Online"
path = "http://www.ncdc.noaa.gov/cdo-web/datatools/findstation"
# Path: weather.json
[[resources]]
name = "weather.json"
path = "weather.json"
description = """Instructional dataset showing actual and predicted temperature data."""
# Path: wheat.json
[[resources]]
name = "wheat.json"
path = "wheat.json"
description = """In an 1822 letter to Parliament, [William Playfair](https://en.wikipedia.org/wiki/William_Playfair), a Scottish engineer who is often credited as the founder of statistical graphics, published [an elegant chart on the price of wheat](http://dh101.humanities.ucla.edu/wp-content/uploads/2014/08/Vis_2.jpg). It plots 250 years of prices alongside weekly wages and the reigning monarch. He intended to demonstrate that “never at any former period was wheat so cheap, in proportion to mechanical labour, as it is at the present time.”"""
[[resources.sources]]
title = "1822 Playfair Chart"
path = "http://dh101.humanities.ucla.edu/wp-content/uploads/2014/08/Vis_2.jpg"
# Path: windvectors.csv
[[resources]]
name = "windvectors.csv"
path = "windvectors.csv"
description = """Simulated wind patterns over northwestern Europe."""
# Path: world-110m.json
[[resources]]
name = "world-110m.json"
path = "world-110m.json"
# Path: zipcodes.csv
[[resources]]
name = "zipcodes.csv"
path = "zipcodes.csv"
description = """GeoNames.org"""
[[resources.sources]]
title = "GeoNames"
path = "https://www.geonames.org"
|
|
Since name and path are the same, can we have only one in the file designed for human editing? Also I like that some files have column descriptions. Do we merge those into the data package? |
|
(#634 (comment))
I agree with @domoritz, just There seems to be what I'm assuming is an escaping issue with this description:
|
|
Sounds good on comments, and will do on the PR. Can you clarify with an example what you mean by your last point on referring to sources in the description? Also, some of the resource descriptions currently include column information. The resource spec has a special place for those (called a table schema), but I was thinking we could leave the column descriptions where they are (just within the source description field) for now, and migrate those into a resource table schema in a future PR. Or we could do that now as well. ** The table schema of course will still be part of the script-generated Json. But we'd need that to be merged with the column descriptions in the toml file to have them fully integrated. |
Sure, I thought we were talking about the same thing here:
As an example, in the section below: # Path: annual-precip.json
[[resources]]
name = "annual-precip.json"
path = "annual-precip.json"
description = """A raster grid of global annual precipitation for the year 2016 at a resolution 1 degree of lon/lat per cell, from [CFSv2](https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2)."""
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[[resources.sources]]
title = "Climate Forecast System Version 2"
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
path = "https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2"
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^I would rewrite this as: [[resources]] # Path: annual-precip.json
path = "annual-precip.json"
description = "A raster grid of global annual precipitation for the year 2016 at a resolution 1 degree of lon/lat per cell."
[[resources.sources]]
title = "Climate Forecast System Version 2"
path = "https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2"Or if there are cases where you need to refer to a source mid-sentence, use the |
dsmedia
added a commit
to dsmedia/vega-datasets
that referenced
this issue
Dec 8, 2024
- Add SOURCES.toml to provide supplemental (extrinisic) metadata on datasets, from SOURCES.md, in a form usable by build_datapackage.py - Include resource descriptions, sources, and licenses to supplement script output - Preserve existing markdown content for future documentation - TODO: Remove duplicated content between descriptions and sources - TODO: Incorporate resource-level column descriptions into table schema, where available - TODO: determine if root-level $schema property should be specified in the TOML file with the value "https://datapackage.org/profiles/2.0/datapackage.json" per Frictionless Data guidelines Resolves vega#634
8 tasks
5 tasks
dangotbanned
added a commit
to dsmedia/vega-datasets
that referenced
this issue
Dec 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Before migrating the extrinsic metadata of each dataset from markdown to a machine readable format, should we agree on a yaml template that would work well with the new Frictionless tooling? Should we make any required (like sourcing) to ensure future datasets are properly documented before release? Not all have sources now, but we can get those added. What should the yaml file be named?
The text was updated successfully, but these errors were encountered: