Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jason's report of data issues at multiple sites #28

Open
BaptisteVandecrux opened this issue Jan 19, 2023 · 1 comment
Open

Jason's report of data issues at multiple sites #28

BaptisteVandecrux opened this issue Jan 19, 2023 · 1 comment

Comments

@BaptisteVandecrux
Copy link
Member

https://github.com/jasonebox/CARRA-TU_GEUS/blob/a75e187197c1816719ecb64df52b9ebda3460f70/GEUS_AWS_reformat_xmit_for_CARRA_TU_v20230111_new%20pipeline.py#L338-L394

            if site=='CP1':
                df=adjuster(site,df,['airpressureminus1000'],2022,7,1,'min_filter',2022,12,31,'xmit outlier?',-250)
                df=adjuster(site,df,['airpressureminus1000'],2022,7,1,'max_filter',2022,12,31,'xmit outlier?',-150)
                # print(np.nanmin(df['airpressureminus1000']))
                # plt.plot(df['airpressureminus1000'])


            if site=='HUM' and tx_or_standard=='tx':
                df=adjuster(site,df,['airpressureminus1000'],2022,11,1,'min_filter',2022,12,31,'xmit outlier?',-243)
                df=adjuster(site,df,['airpressureminus1000'],2022,11,1,'max_filter',2022,12,1,'xmit outlier?',-213)
            
            if site=='HUM' and tx_or_standard=='level_3':
                df=adjuster(site,df,['airpressureminus1000'],2022,11,1,'min_filter',2022,12,31,'xmit outlier?',-243)
                df=adjuster(site,df,['airpressureminus1000'],2022,11,1,'max_filter',2022,12,1,'xmit outlier?',-213)
                # df=adjuster(site,df,['airpressureminus1000'],2022,12,1,'min_filter',2022,12,31,'xmit outlier?',-243)
                df=adjuster(site,df,['airpressureminus1000'],2022,12,15,'max_filter',2022,12,31,'xmit outlier?',-200)                
                
            if site=='NAU':
                df=adjuster(site,df,['airpressureminus1000'],2022,12,23,'min_filter',2022,12,28,'xmit outlier?',-267)
                df=adjuster(site,df,['airpressureminus1000'],2022,12,23,'max_filter',2022,12,31,'xmit outlier?',-238)


            if site=='NEM':
                df=adjuster(site,df,['airpressureminus1000'],2022,7,1,'min_filter',2022,12,31,'xmit outlier?',-290)
                df=adjuster(site,df,['airpressureminus1000'],2022,7,1,'max_filter',2022,12,31,'xmit outlier?',-220)
                df=adjuster(site,df,['windspeed'],2022,11,1,'nan_var',2022,12,31,'instrument frozen?',0)


            if site=='NSE':
                df=adjuster(site,df,['airpressureminus1000'],2022,4,1,'min_filter',2022,5,31,'xmit outlier?',-350)
                # df=adjuster(site,df,['airpressureminus1000'],2022,5,31,'min_filter',2022,7,31,'xmit outlier?',-1300)
                # df=adjuster(site,df,['airpressureminus1000'],2022,5,31,'max_filter',2022,7,31,'xmit outlier?',-1100)
                df=adjuster(site,df,['temperature'],2022,4,1,'min_filter',2022,7,31,'xmit outlier?',-90)
                df=adjuster(site,df,['temperature'],2022,4,1,'max_filter',2022,7,31,'xmit outlier?',9)
                df=adjuster(site,df,['temperature'],2022,7,1,'min_filter',2022,12,31,'xmit outlier?',-50)
                df=adjuster(site,df,['relativehumidity'],2022,7,1,'min_filter',2022,12,31,'xmit outlier?',50)


                df.temperature[df.temperature<-90]=np.nan
                df.temperature[df.temperature>-9]=np.nan
    
            if site=='NUK_U':
                df=adjuster(site,df,['windspeed'],2022,4,1,'nan_var',2022,5,31,'propellor not secure?',0)
            
            if site=='QAS_M':
                # df=adjuster(site,df,['TA1','TA2'],2021,6,1,'max_filter',2022,4,1,'xmit outlier?',2)
                df=adjuster(site,df,['temperature','relativehumidity'],2022,4,1,'nan_var',2022,5,31,'instrument burial!?',0)
    
            if site=='QAS_U':
                # df=adjuster(site,df,['TA1','TA2'],2021,6,1,'max_filter',2022,4,1,'xmit outlier?',2)
                df=adjuster(site,df,['temperature','relativehumidity'],2022,4,1,'nan_var',2022,5,28,'instrument burial!?',0)
                # df=adjuster(site,df,['relativehumidity'],2022,4,1,'nan_var',2022,5,28,'instrument burial!?',0)
    
            if site=='TAS_A':
                df=adjuster(site,df,['temperature','relativehumidity','windspeed','winddirection'],2022,4,1,'nan_var',2022,6,24,'instrument burial!?',0)
                df=adjuster(site,df,['winddirection'],2022,6,30,'nan_var',2022,7,31,'instrument fail!?',0)
                df=adjuster(site,df,['windspeed'],2022,7,1,'nan_var',2022,7,31,'instrument fail!?',0)


    
            if site=='SWC':
                df=adjuster(site,df,['winddirection'],2020,8,9,'rotate',2022,8,3,'wind dir box 180deg off',180)

Some of these issues should be fixed with standard filters (applied at all sites) and some will need to have their entry in the flag files or adjustment files.

@patrickjwright
Copy link

In general, it would be nice to not have station-specific QC in pypromice. If I ever get around to implementing percentile-based QC, it will do great at flagging outliers specific to each station's data distribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants