Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPI-3429 - Fix mergesp3 utility #37

Merged
merged 7 commits into from
Aug 2, 2024

Conversation

ronaldmaj
Copy link
Collaborator

This PR is created to address the issue brought up here:
#28

Basically the mergesp3 utility in it's current form was not doing what it was supposed to.
Giving it a pair of any two sp3 files led to an error each time (a ValueError saying that the truth value of two series was ambiguous). This was coming up because in using the pd.concat function in gn_io.sp3.sp3merge there is a step where the equality of the df.attrs are checked. These are going to be different for sp3 files from different time periods.

Therefore, I've created a new variable that merges the attrs separately

merged_attrs = merge_attrs(sp3_dfs)

using an existing function merge_attrs, and then assigns the attrs of each df to an empty dict (thereby making them the same).

    for df in sp3_dfs:
        df.attrs = {}

One problem I encoutnered that took a while to debug was that even with no CLK files, I would keep ending up in the block of code that tries to incorporate CLK files into the dataframe, causing failure.
This is due to the mergesp3 utility producing an empty tuple rather than a None when no clkpaths are passed in.

I've manually set it to None now in that case.

@ronaldmaj ronaldmaj changed the title NPI-3429 NPI-3429 - Fix mergesp3 utility Jul 30, 2024
@seballgeyer seballgeyer self-requested a review July 30, 2024 12:11
Copy link
Collaborator

@seballgeyer seballgeyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be interesting also to have some unit test on this one, just making sure that the output is what we expect.

gnssanalysis/gn_io/sp3.py Show resolved Hide resolved
@ronaldmaj
Copy link
Collaborator Author

It would be interesting also to have some unit test on this one, just making sure that the output is what we expect.

Yeah, good point, forgot about this. I'll put together something now - I need to get into the habit of writing unit tests when I do work on gnssanalysis 😄

@ronaldmaj
Copy link
Collaborator Author

ronaldmaj commented Aug 1, 2024

It would be interesting also to have some unit test on this one, just making sure that the output is what we expect.

Okie doke, I've been able to write up a test for the sp3merge function. I did have to introduce a new py package unfortunately (pyfakefs) but that was the only way I saw to mimic a filesystem and therefore test the function itself.

Test seems to pass the pipeline ok:
Screenshot_20240801_135144

Copy link
Collaborator

@seballgeyer seballgeyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all good,
tested, pushed one correction wrt numpy v.2 compatibility

@seballgeyer seballgeyer self-assigned this Aug 2, 2024
seballgeyer
seballgeyer previously approved these changes Aug 2, 2024
epoch_vals["X"].replace(to_replace=[_np.inf, _np.NINF], value=SP3_POS_NODATA_STRING, inplace=True)
epoch_vals["Y"].replace(to_replace=[_np.inf, _np.NINF], value=SP3_POS_NODATA_STRING, inplace=True)
epoch_vals["Z"].replace(to_replace=[_np.inf, _np.NINF], value=SP3_POS_NODATA_STRING, inplace=True)
epoch_vals["X"].replace(to_replace=[_np.inf, -_np.inf], value=SP3_POS_NODATA_STRING, inplace=True)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need updating again for Pandas 3. I'll try to replace it with format() calls at the point of writing the data out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need updating again for Pandas 3. I'll try to replace it with format() calls at the point of writing the data out.

Yeah, there's a few changes that we'll need to make in order to make us Pandas 3 compliant. Running the sp3merge utility already has a whole bunch of warnings around this

treefern
treefern previously approved these changes Aug 2, 2024
Copy link
Collaborator

@treefern treefern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.
I examined the padded date-stamp modifying line and couldn't think of a way it could go wrong.
It's nice to see some more unit tests coming in.

Adding check on PV_FLAG to sp3merge unit test

Co-authored-by: Nathan <[email protected]>
@ronaldmaj ronaldmaj merged commit a903c44 into main Aug 2, 2024
1 check passed
@ronaldmaj ronaldmaj deleted the NPI-3429-fix-sp3merge-diff-attrs-issue branch August 2, 2024 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants