Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

missing values are not behaving properly #137

Open
RobLBaker opened this issue Jun 20, 2024 · 1 comment
Open

missing values are not behaving properly #137

RobLBaker opened this issue Jun 20, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@RobLBaker
Copy link
Member

getting reports that missing values such as "NULL" are causing issues if they are in numeric columns. This shouldn't be an issue. Haven't been able to get ahold of a dataset that contains this particular error yet, but it is worth testing further as multiple SCA data strike team members have experienced the issue.

@RobLBaker RobLBaker added the bug Something isn't working label Jun 20, 2024
@RobLBaker RobLBaker self-assigned this Jun 20, 2024
@RobLBaker
Copy link
Member Author

From the people who ran into this problem... maybe the issue is more about R not liking mixed data types than the DPchecker test:

"requested we use NULL for all missing values throughout the dataset, so that is what I did originally. I don't remember exactly what the wording of the warning was that came up from DPchecker, but it had the word "parse" or "parsing" in it. When I went on the github page that lists the functions in DPchecker and searched "parse" the function I landed on was Test data and metadata data formats match — test_dates_parse • DPchecker (nationalparkservice.github.io)". I think when I originally read the function name it I misread "dates" as "data" but when I called problems() for this function it returned to me 4 rows of NULLs in a numeric column that housed cubic feet per second values. This sparked a big conversation surrounding how to represent missing values in different data type columns. (NULL is a character type in R and creates the problem of having multiple data types in 1 column for numeric and datetime columns) We ended up deciding on NULL for missing values in character and categorical columns and NA for missing values in numeric and datetime columns, since NA is neither a character or numeric type in R. Once I changed the NULLs in numeric and datetime columns to NAs it ran through DPchecker just fine."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant