Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated process_report.py to avoid warnings for SettingWithCopyWarnings and FutureWarnings #40

Merged
merged 1 commit into from
May 10, 2024

Conversation

QuanMPhm
Copy link
Contributor

@QuanMPhm QuanMPhm commented May 2, 2024

Closes #34, I've added a few small changes which should remove all pandas SettingWithCopyWarning and FutureWarning warnings. These consisted of explicit type casting, and explicitly making copies of data-frames that would later be modified

@QuanMPhm QuanMPhm requested review from knikolla and naved001 May 2, 2024 20:05
@QuanMPhm QuanMPhm force-pushed the 34/patch_warnings branch from 4707b82 to 1b2716a Compare May 2, 2024 20:10
@naved001 naved001 requested a review from larsks May 3, 2024 18:35
Copy link
Member

@larsks larsks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the relationship between the title of this PR ("Patched pandas warnings for SettingWithCopyWarnings and FutureWarnings") and the changes it contains.

@QuanMPhm
Copy link
Contributor Author

QuanMPhm commented May 6, 2024

@larsks Without the changes, running the processing script will make Pandas print the 2 warnings mentioned in the title. The SettingWithCopyWarnings came from the fact that I was performing chained indexing on a few dataframes. Since chained indexing may return a copy of the dataframe or dataframe itself, this causes ambiguity on whether assignments to the indexed dataframes would change the original, or merely a copy. I resolved the warnings by explicitly using the copy() function to make copies of the dataframes.

As for the FutureWarnings, these were caused by assigning strings to empty columns, whose values default to NaN (float). Since float is incompatible with strings, this raises a warning.

@larsks
Copy link
Member

larsks commented May 6, 2024

Ah, so "Update code to avoid SettingWithCopyWarnings and FutureWarnings".

@naved001
Copy link
Collaborator

naved001 commented May 6, 2024

Without the changes, running the processing script will make Pandas print the 2 warnings mentioned in the title. The SettingWithCopyWarnings came from the fact that I was performing chained indexing on a few dataframes. Since chained indexing may return a copy of the dataframe or dataframe itself, this causes ambiguity on whether assignments to the indexed dataframes would change the original, or merely a copy. I resolved the warnings by explicitly using the copy() function to make copies of the dataframes.

As for the FutureWarnings, these were caused by assigning strings to empty columns, whose values default to NaN (float). Since float is incompatible with strings, this raises a warning.

@QuanMPhm You should probably put that into the commit message because it is a good explanation of what's going on.

@QuanMPhm QuanMPhm changed the title Patched pandas warnings for SettingWithCopyWarnings and FutureWarnings Updated process_report.py to avoid warnings for SettingWithCopyWarnings and FutureWarnings May 6, 2024
@QuanMPhm QuanMPhm force-pushed the 34/patch_warnings branch from 1b2716a to 2416397 Compare May 7, 2024 16:30
@QuanMPhm
Copy link
Contributor Author

QuanMPhm commented May 7, 2024

@naved001 I've updated the commit message

Without the patches, running the processing script will make Pandas print 2 warnings, SettingWithCopyWarnings and FutureWarnings.
The SettingWithCopyWarnings came from the fact that chained indexing was performed on a few dataframes.
Since chained indexing may return a copy of the dataframe or dataframe itself, this causes ambiguity on
whether assignments to the indexed dataframes would change the original, or merely a copy.
This is resolves the warnings by explicitly using the copy() function to make copies of the dataframes.

As for the FutureWarnings, these were caused by assigning strings to empty columns, whose values default to NaN (float).
Since float is incompatible with strings, this raises a warning. This is fixed by explicitly type casting certain columns
@QuanMPhm QuanMPhm force-pushed the 34/patch_warnings branch from 2416397 to a76b353 Compare May 10, 2024 19:03
@QuanMPhm QuanMPhm merged commit 0489e1f into CCI-MOC:main May 10, 2024
3 checks passed
@QuanMPhm QuanMPhm deleted the 34/patch_warnings branch May 10, 2024 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Some warnings I saw when I ran the scripts to produce the report for April
3 participants