AirNow metadata options and other updates #146
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
that can take date arg and defaults to s3 URI, which is faster (700 ms vs 400 ms ish on my laptop)
like done for timezone finder
with a simple two-day test, found able to get slightly more data (100 or so rows) by disabling today metadata, since fewer rows with bad UTC offset were dropped
zmoon
changed the title
so codes and such are string
mainly for my current testing/development, but could be useful
as semicolon-separated strings
|
Current test failure is openaq-fetches denying access ("PermissionError: Access Denied"). |
- merging of AirNow, with lat/lon if new site IDs - correct airnow_flag - drop site ID dupe rows - express columns to keep instead of columns to drop
|
Need to conserve more info when duplicate-dropping for AQS. For example, currently only get some of the full set of unique |
|
Should use the CSV parameter list to translate code for ones we don't have custom names for. And/or to construct a new in-code translation table. |
Open
zmoon
commented
Oct 10, 2024
| else: | ||
| raise ValueError("Could not determine if file is daily or hourly") | ||
|
|
||
| dtype = {"siteid": str, "obs": float} |
There was a problem hiding this comment.
Need to ensure MSA code and such are read as str, not int/float. cmsa_name and msa_code seem to need this or they get convert to float64 (in order to represent missing with NaN).
|
Note that this PR fixes UTC offset to be float (integer conversion floors the few that are non-integer, making them incorrect). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #129 and makes some progress towards #111.