-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NPI-3429 - Fix mergesp3 utility #37
Conversation
…d.concat doesnt work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be interesting also to have some unit test on this one, just making sure that the output is what we expect.
Yeah, good point, forgot about this. I'll put together something now - I need to get into the habit of writing unit tests when I do work on |
Okie doke, I've been able to write up a test for the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all good,
tested, pushed one correction wrt numpy v.2 compatibility
epoch_vals["X"].replace(to_replace=[_np.inf, _np.NINF], value=SP3_POS_NODATA_STRING, inplace=True) | ||
epoch_vals["Y"].replace(to_replace=[_np.inf, _np.NINF], value=SP3_POS_NODATA_STRING, inplace=True) | ||
epoch_vals["Z"].replace(to_replace=[_np.inf, _np.NINF], value=SP3_POS_NODATA_STRING, inplace=True) | ||
epoch_vals["X"].replace(to_replace=[_np.inf, -_np.inf], value=SP3_POS_NODATA_STRING, inplace=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will need updating again for Pandas 3. I'll try to replace it with format()
calls at the point of writing the data out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will need updating again for Pandas 3. I'll try to replace it with
format()
calls at the point of writing the data out.
Yeah, there's a few changes that we'll need to make in order to make us Pandas 3 compliant. Running the sp3merge
utility already has a whole bunch of warnings around this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
I examined the padded date-stamp modifying line and couldn't think of a way it could go wrong.
It's nice to see some more unit tests coming in.
Adding check on PV_FLAG to sp3merge unit test Co-authored-by: Nathan <[email protected]>
This PR is created to address the issue brought up here:
#28
Basically the
mergesp3
utility in it's current form was not doing what it was supposed to.Giving it a pair of any two sp3 files led to an error each time (a
ValueError
saying that the truth value of two series was ambiguous). This was coming up because in using thepd.concat
function ingn_io.sp3.sp3merge
there is a step where the equality of thedf.attrs
are checked. These are going to be different for sp3 files from different time periods.Therefore, I've created a new variable that merges the
attrs
separatelyusing an existing function
merge_attrs
, and then assigns theattrs
of eachdf
to an emptydict
(thereby making them the same).One problem I encoutnered that took a while to debug was that even with no CLK files, I would keep ending up in the block of code that tries to incorporate CLK files into the dataframe, causing failure.
This is due to the
mergesp3
utility producing an empty tuple rather than aNone
when no clkpaths are passed in.I've manually set it to
None
now in that case.