CTD_Profiles_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--QC_INFORMATION-->\n",
    "<img align=\"left\" style=\"padding-right:10px;\" width=\"250\" src=\"_code_software/img/Marine_logo.jpg\"></br>\n",
    "<h1><center>Ocean Science - CTD Profiles Processing notebook</center></h1>\n",
    "<span style='color:Blue'>  \n",
    "\n",
    "### The following Notebook holds a self contained SOP throughout\n",
    "\n",
    "### There are two main types of Jupyter Notebook cells:\n",
    "- Markdown cells (like the one you are reading) display human readable text\n",
    "- Code cells contain blocks of Python code used for specific computing tasks    \n",
    "### This notebook is operated by running a series of code cells\n",
    "### Users are encouraged to run each code cell individually using 'Run' in the toolbar above\n",
    "### Actual code cells can be hidden (default) or displayed through the toggle that will appear when you run the first cell\n",
    "### See below for some hints and tips\n",
    "- No prior knowledge of Python is required to run these Notebooks\n",
    "- Running cells and running the notebook itself will quickly become famaliar to new users\n",
    "- Hidden or invisible code cells can be selected by clicking just underneath the preceeding markdown cell\n",
    "- Minimal adjusting of folder directory paths within the code is required on the first run only\n",
    "- When a cell has just ran, the position of the Notebook may jump up or down, scrolling to reorientate yourself will quickly become routine\n",
    "- When a cell is still running (some are slow) an egg-timer will replace the book icon on the very top bar of the browser\n",
    "    - Also, if a cell is running it will have a * inside the cell identifier to the top left of that cell: i.e. In [*]:\n",
    "    - if the cell has not yet run it will be empty: In [ ]:\n",
    "    - if the cell has just completed running, it will have the latest run number displayed, e.g. In [27]:\n",
    "### A small number of cells will require or accept user input through a dropdown interface\n",
    "### Note: most code cells, when run, will output relevant information underneath them\n",
    "        \n",
    "</span>\n",
    "\n",
    "\n",
    "## Checklist for CTD operator in preparation for, and during the survey:\n",
    "\n",
    "1. Get copies of the sensor calibration sheets for all sensors on the CTD for the duration of the survey.\n",
    "2. Check the master XMLCON file has been populated with the correct calibration information.\n",
    "3. Populate a CTD logsheet and water sampling logsheet for every cast.\n",
    "4. Follow the CTD data acquisition checklist/SOP during each cast.\n",
    "5. If deployed on the CTD, download the SBE35 after each cast. Include the CTD ID/number in the filename e.g. SBE35_{CRUISEID}_CTD{#}.asc.\n",
    "6. Add logsheet information to Excel template and name {CRUISEID}_Log.xlsx.\n",
    "7. At the end of the survey scan all CTD logsheets and bottle firing logsheets and save in the logsheets folder as a pdf.\n",
    "\n",
    "## Processing overview\n",
    "<img align=\"centre\" src=\"_code_software/img/overview.jpg\">\n",
    "\n",
    "   \n",
    "## The first cell carries out a number of tasks:\n",
    "- It loads all Python tookboxes needed by the Notebook as follows:\n",
    "\n",
    "    Toolbox versions that software was developed to use:\n",
    "    - json 2.0.9\n",
    "    - ipywidgets 8.0.4\n",
    "    - numpy 1.24.3\n",
    "    - pandas 1.5.3\n",
    "    - re 2.2.1\n",
    "    - seawater 3.3.4\n",
    "    - bokeh 3.1.0\n",
    "    - xlrd 2.0.1\n",
    "\n",
    " \n",
    "- It sets location of the master SeaBird .psa templates used to create the PSA files for the different SBE processing steps. This is hardcoded. Please do not change.\n",
    "- It provides a toggle switch to hide or display the Python code cells\n",
    "- Once run it requests user input as described below:\n",
    "   \n",
    "### Set cruise name\n",
    "Insert Cruise ID below. Cruise ID should be in the format CXxxxxx\n",
    "\n",
    "### Set variable \"proc_mode\" to indicate file conversion mode: \n",
    "    0 = process all files through Sea-Bird application; \n",
    "    1 = only process new files through Sea-Bird application.\n",
    "\n",
    "### Set variable \"heave_mode\" to indicate if heave function is required, heave function is strongly recommended for most cases\n",
    "    0 = Run Heave function [Default/Recommended]; \n",
    "    1 = Bypass the Heave function.\n",
    "    \n",
    "### Set default oxygen alignment value\n",
    "As the CTD package profiles the water column, a given parcel of water passes over the oxygen sensor slightly later than the temperature and conductivity sensors. Therefore values returned from the sensors are slightly out of step for the timestamp in the data file. Experience suggests for the MI CTD rig setup a value of 2 seconds aligns the oxygen sensor outputs with the temperature and conductivity.\n",
    "This will be visually checked during processing and can be adjusted at that stage, where appropriate.\n",
    "\n",
    "### Are there stations that have part-casts to be combined?\n",
    "\n",
    "\tIf so, provide a dictionary of the form:\n",
    "\tcombined = {'initial cast CNV filename' : {'CNV file to be merged','cast status','CNV file to be merged','cast status'} } See commented out example below. \n",
    "\n",
    "\tcombined = {'CE21003_CTD002.CNV': {'CE21003_CTD002B.CNV': 'D/U','CE21003_CTD002C.CNV': 'U'}}\n",
    "\n",
    "\tThe above dictionary drives the function combine_files2cast(data_all,combined) to rename the profile name for files 'CE21003_CTD002B.CNV' & 'CE21003_CTD002C.CNV' to 'CE21003_CTD002.CNV'.\n",
    "\n",
    "\tIf no files require merging leave line commented out (default).\n",
    "\n",
    "### When starting a new profiles Notebook, for example for a new survey not processed before, it is good practice to clear the memory first\n",
    "### Do this by clicking 'Kernel' on the toolbar at the top of the browser page, then click 'Restart & Clear Output'\n",
    "### To get started with the Notebook, run this first cell below by clicking anywhere on it and clicking 'Run' on the toolbar above\n",
    "- And best of luck! :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "                                                                        ### Cell 1 ###\n",
    "\n",
    "# Remove warnings being displayed when dashboard in production - comment out next 2 lines during dev and testing\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import json\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import re\n",
    "import sys\n",
    "import xml\n",
    "import bokeh\n",
    "import shutil\n",
    "import xml\n",
    "import yaml\n",
    "import gc\n",
    "import time\n",
    "\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "import chevron               ## If not installed open the Anaconda prompt and run >> conda install -c conda-forge chevron\n",
    "import ipywidgets as widgets ## If not installed using the Anaconda prompt run >> conda install -c conda-forge ipywidgets\n",
    "import seawater              ## If not installed using the Anaconda prompt run >> conda install -c conda-forge seawater\n",
    "import xlrd                  ## If not installed using the Anaconda prompt run >> conda install -c conda-forge xlrd\n",
    "\n",
    "# Import bokeh functions\n",
    "from bokeh.plotting import show\n",
    "from bokeh.io import output_notebook, push_notebook\n",
    "from bokeh.resources import INLINE\n",
    "\n",
    "# Import bespoke functions\n",
    "import scripts.ctd as ctd\n",
    "import scripts.ctd_bokeh as ctd_bokeh\n",
    "import scripts.sensor_configuration as sensor_configuration\n",
    "import scripts.data_processing as data_processing\n",
    "import scripts.bottle_processing as bottle_processing\n",
    "import scripts.calculations as calculations\n",
    "import scripts.seabird_processes as seabird_processes\n",
    "\n",
    "# Print toolbox versions to screen\n",
    "#print(\"Toolbox version:\")\n",
    "#print('\\n'.join(f'{m.__name__} {m.__version__}' for m in globals().values() if getattr(m, '__version__', None)))\n",
    "\n",
    "# Print PSA folder location to screen\n",
    "wd = os.getcwd()\n",
    "PSA_template_folder = os.path.normpath(os.path.join(wd,'psa_templates'))\n",
    "print(\"PSA master template folder is : %s\\n\" % PSA_template_folder)\n",
    "\n",
    "# Set drop down options\n",
    "proc_dict = {'Process all files through Sea-Bird application' : 0,\n",
    "             'Only process new files through Sea-Bird application' : 1}\n",
    "\n",
    "heave_dict = {'Run Heave function [Default/Recommended]' : 0,\n",
    "              'Bypass the Heave function': 1}\n",
    "\n",
    "# Define widgets\n",
    "# Cruise ID - string\n",
    "cruiseID_widget = widgets.Text(\n",
    "    placeholder='Type Cruise ID here',\n",
    "    description='Cruise ID:',\n",
    "    disabled=False,\n",
    "    style = {'description_width': '50%'}\n",
    ")\n",
    "\n",
    "# Processing Mode - int;\n",
    "proc_mode_widget = widgets.Dropdown(\n",
    "    options = list(proc_dict.keys()),\n",
    "    description = 'Proc Mode:',\n",
    "    disable = False,\n",
    "    style = {'description_width': '30%'},\n",
    "    layout = widgets.Layout(width='500px'))\n",
    "\n",
    "# Heave Mode - int;\n",
    "heave_mode_widget = widgets.Dropdown(\n",
    "    options = list(heave_dict.keys()),\n",
    "    description = 'Heave Mode:',\n",
    "    disable = False,\n",
    "    style = {'description_width': '30%'},\n",
    "    layout = widgets.Layout(width='500px'))\n",
    "\n",
    "# Oxygen Alignment - int\n",
    "# Any number between 0 and 10\n",
    "oxygen_alignment_widget = widgets.Dropdown(\n",
    "    options = list(str(x) for x in range(0,10)),\n",
    "    description = 'Oxygen Alignment:',\n",
    "    value = '2',\n",
    "    disable = False,\n",
    "    style = {'description_width': '50%'}\n",
    "\n",
    ")\n",
    "\n",
    "# Combined Widget - this allows a user to combine casts which have may been split into multiple files due to technical difficulties\n",
    "combined_widget = widgets.Text(\n",
    "    placeholder='Provide dictionary of files for combining here when cast logging split',\n",
    "    description='Files for combining:',\n",
    "    disabled=False,\n",
    "    style = {'description_width': '15%'},\n",
    "    layout = widgets.Layout(width='1000px')\n",
    ")\n",
    "\n",
    "# Bin Unit - metre or decibar\n",
    "bin_unit_widget = widgets.Dropdown(\n",
    "    options = ['metre','decibar'],\n",
    "    description = 'Binning unit:',\n",
    "    disable = False,\n",
    "    style = {'description_width': '50%'})\n",
    "\n",
    "# Combine all widgets and display them\n",
    "display(widgets.VBox([cruiseID_widget, proc_mode_widget, heave_mode_widget, oxygen_alignment_widget, combined_widget, bin_unit_widget]))\n",
    "\n",
    "# The following code will hide all code from the user \n",
    "# A button will appear under this cell that will allow you to toggle on and off the code\n",
    "# display(HTML(\"<style>.container { width:100% !important; }</style>\"))\n",
    "# HTML('''<script>\n",
    "# code_show=true;\n",
    "# function code_toggle() {\n",
    "# if (code_show){\n",
    "# $('div.input').hide();\n",
    "# } else {\n",
    "# $('div.input').show();\n",
    "# }\n",
    "# code_show = !code_show\n",
    "# }\n",
    "# $( document ).ready(code_toggle);\n",
    "# </script>\n",
    "# <form action=\"javascript:code_toggle()\"><input type=\"submit\" value=\"Click here to toggle on/off the raw code\"></form>\n",
    "# ''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run next cell to implement user inputs and to set working directories\n",
    "\n",
    "### Sets up folders within the working directory if they do not already exist and sets variables for folder paths\n",
    "Folder structure contents:\n",
    "<li> <b>psa</b> </li>\n",
    "- Contains the PSA files to configure the SBE processing steps. These files are generated by this script from the PSA templates in the master folder named previously.\n",
    "\n",
    "<li> <b>raw_files</b> </li>\n",
    "    - Copy of the SBE files here (.HEX, .XMLCON, .BL, .ROS, etc) for the casts to be processed. Exclude any deck test files.\n",
    "\n",
    "<li> <b>logsheets</b> </li>\n",
    "    - Where a log file has been created this folder should contain an Excel file with a digitised events log, CTD logsheets and bottle sampling logsheets. File should be named \"CRUISEID_Log.xlsx\".\n",
    "\n",
    "<li> <b>SBE35</b> </li>\n",
    "    - When a SBE35 temperature logger has been deployed on the CTD rig, one file per cast should be placed here. Filename should follow convention \"SBE35_CRUISEID_CTDxxx.txt\".\n",
    "\n",
    "<li> <b>cal_samples</b> </li>\n",
    "    - Water sample salinity and Winkler oxygen data files with Bedford Numbers to identify samples.\n",
    "\n",
    "<li> <b>output</b></li>- final binned and calibrated data files merged with metadata (datetime, position, station name, etc). Sub-folders:\n",
    "\n",
    "\n",
    "- bottle - contains SBE generated .BTL files and contatenated bottle firing details with metadata as CSVs.\n",
    "\n",
    "- screen_2Hz - contains SBE processed .CNV files at 2Hz resolution after DATACNV, FILTER, WILDEDIT, CELLTM and BINAVG routines\n",
    "\n",
    "- all_2Hz - contains notebook processed data saved as CSV files after each major processing step (SBE processing - sensor alignment and derived measurement recalculation - auto heave flagging)\n",
    "\n",
    "- plots - folder contains plots output after processing for cruise report section or use during the cruise.\n",
    "\n",
    "<span style='color:Blue'>  \n",
    "\n",
    "### On initial setup and first run, need to adjust the directory or 'filepath' within the following code cell\n",
    "### If running from a local machine (or network) update/enter the full directory address found between the single quotation marks on the first filepath = os.path.normpath... line\n",
    "### Using the '#' symbol to comment/uncomment out a line, ensure only the correct: filepath = os.path.... line is uncommented (without a # in front).\n",
    "\n",
    "### If running from one of the vessel's CTD PC machines, uncomment only the final filepath = os.path.... line\n",
    "</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 2 ###\n",
    "    # Set cruise name as variable here\n",
    "cruiseID = cruiseID_widget.value\n",
    "\n",
    "if cruiseID == '':\n",
    "    print(\"\\033[1;31m*** You have not provided a Cruise ID, please enter Cruise ID ***\\033[0m\")\n",
    "else:\n",
    "    print(\"cruiseID: %s\\n\" % cruiseID)\n",
    "\n",
    "    # Set variable to indicate conversion mode: \n",
    "    #    0 = process all files through Sea-Bird application; \n",
    "    #    1 = only process new files through Sea-Bird application.\n",
    "\n",
    "    proc_mode = proc_dict[proc_mode_widget.value]\n",
    "\n",
    "    if proc_mode == 0:\n",
    "        print(\"All profiles will be processed, even if previously converted by SBE processing.\")\n",
    "    elif proc_mode == 1:\n",
    "        print(\"Only profiles that haven't been previously processed will be run through the SBE software conversion steps.\")\n",
    "    else:\n",
    "        print(\"\\033[1;31mError: value for proc_mode is not valid. Please update to 0 or 1 as appropriate.\\033[0m\")\n",
    "\n",
    "    # Set variable to indicate if Heave function is required [Heave function is strongly recommended for almost all cases]: \n",
    "    #    0 = Run Heave function [Default/Recommended]; \n",
    "    #    1 = Bypass the Heave function.\n",
    "\n",
    "    heave_mode = heave_dict[heave_mode_widget.value]\n",
    "\n",
    "    if heave_mode == 0:\n",
    "        print(\"All profiles, under proc_mode selection, will undergo heave flagging.\")\n",
    "    elif heave_mode == 1:\n",
    "        print(\"Heave flagging turned off for this run.\")\n",
    "    else:\n",
    "        print(\"\\033[1;31mError: value for heave_mode is not valid. Please update to 0 or 1 as appropriate.\\033[0m\")\n",
    "\n",
    "    oxy_align_default = oxygen_alignment_widget.value\n",
    "\n",
    "    print(\"Oxygen sensor alignment default value: %s\" % oxy_align_default)\n",
    "\n",
    "    combined = combined_widget.value\n",
    "\n",
    "    if combined=='':\n",
    "        print(\"No files listed for merging.\")\n",
    "        combined = None\n",
    "    elif type(yaml.load(combined, Loader=yaml.Loader))==dict:\n",
    "        print(\"Files listed for merging: %s\" % combined)\n",
    "        combined = yaml.load(combined, Loader=yaml.Loader)\n",
    "    else:\n",
    "        print(\"Please update 'Files for combining field above. Either leave blank or provide a dictionary.'\")\n",
    "        combined = None\n",
    "    print(\"Data will in binned to 1 %s\\n\" % bin_unit_widget.value)\n",
    "\n",
    "    \n",
    "    # Set cruise working directory\n",
    "    pyear = '20'+cruiseID[2:4]\n",
    "    pvessel = cruiseID[0:2]\n",
    "    filepaths_dictionary = {'CE': os.path.join('C:/Users/Documents/mi-ctd-data-processing/', pvessel, pyear, cruiseID),\n",
    "                            'TC': os.path.join('C:/Users/Documents/mi-ctd-data-processing/', pyear, cruiseID)\n",
    "                           } ### Need to check/change these to appropriate addresses\n",
    "\n",
    "                        ### Change path to local Jupyter Notebook working drive directory\n",
    "    filepath = os.path.normpath(os.path.join('C:/Users/Documents/mi-ctd-data-processing/',pvessel,pyear,cruiseID))  ## local directory on a local computer [adjust as required]\n",
    "\n",
    "    print(\"Working directory is: %s\" % filepath)\n",
    "\n",
    "    backup_server_path = os.path.normpath(os.path.join('Z:/2.1 Oceanographic', cruiseID, f'{cruiseID}_processed_CTD_data_BACKUP'))\n",
    "\n",
    "    print('Backup directory is: %s' % backup_server_path)\n",
    "\n",
    "    # Set the directory filepaths as variables\n",
    "    directories = {'raw': 'raw_files', 'logs': 'logsheets', 'sbe35_raw': 'SBE35', 'cals': 'cal_samples', 'psa': 'psa', 'out': 'output'} \n",
    "    directories_out = {'bottle': 'bottle', 'screen_2Hz': 'screen_2Hz', 'all_2Hz': 'all_2Hz', 'plots': 'plots'}\n",
    "\n",
    "    def dir_path(path,name):\n",
    "        d = os.path.join(path,name)\n",
    "        if not os.path.exists(d):\n",
    "            os.mkdir(d)\n",
    "        return d\n",
    "\n",
    "    print(\"Processing directory folder locations:\")\n",
    "    for item in directories.keys():\n",
    "        globals()[item] = dir_path(filepath,directories[item])\n",
    "        print(globals()[item])\n",
    "    for item in directories_out.keys():\n",
    "        globals()[item] = dir_path(out,directories_out[item])\n",
    "        print(globals()[item])\n",
    "    \n",
    "    # Check if files present in the raw files working directory\n",
    "    if len(os.listdir(raw))>0:\n",
    "        count, countn = 0, 0\n",
    "        for filename in os.listdir(raw):\n",
    "            #print(filename)\n",
    "            if cruiseID in filename.upper():\n",
    "                count += 1\n",
    "            else:\n",
    "                countn += 1\n",
    "               \n",
    "        if countn != 0:\n",
    "            print('\\n\\033[1;31m*** Check filenames. Sea-Bird files present in the raw_files directory file do not follow filename convention <CRUISE>_CTD<NUMBER> ***\\n\\033[0m')\n",
    "        else:\n",
    "            print('\\nFiles present in raw_files directory: %s' % count)\n",
    "        \n",
    "    else:\n",
    "        print('\\n\\033[1;31m*** No Sea-Bird files present in the raw_files directory. If running the notebook for the first time for this cruise, before proceeding copy across the raw SBE files to the \"raw_files\" folder in the working directory. ***\\n\\033[0m')\n",
    "\n",
    "####################################################################################################################################        \n",
    "# Get HEX filenames\n",
    "####################################################################################################################################\n",
    "\n",
    "filelist = os.listdir(raw)\n",
    "hexfilelist = []\n",
    "\n",
    "for item in filelist:\n",
    "    if item.endswith(\".hex\"):\n",
    "        item = item.split('.')\n",
    "        hexfilelist.append(item[0].upper())\n",
    "print('\\tNumber of HEX files available in cruise folder: %s' % (len(hexfilelist)))\n",
    "        \n",
    "####################################################################################################################################        \n",
    "# Extract metadata from HDR files\n",
    "####################################################################################################################################\n",
    "print(\"\\nExtracting cast metadata from the header information for each cast for reference.\")\n",
    "df_NMEA = data_processing.get_NMEA_from_header(raw,'hdr')\n",
    "\n",
    "# Check all fields populated\n",
    "print('\\tNumber of HDR files available in cruise folder: %s' % (len(df_NMEA)))\n",
    "df_missingNMEA = df_NMEA.isnull().sum()\n",
    "\n",
    "for item in ['Lat','Long','Upload Time','UTC Time']:\n",
    "    if df_missingNMEA[item]!=0:\n",
    "        counts = df_missingNMEA[item]\n",
    "        print(\"\\033[1;31mACTION *** %s missing in %s HDR files *** Ensure %s entered into logsheet from paper logs for:\\033[0m\" % (item, counts, tem))\n",
    "        print(df_NMEA[df_NMEA[item].isnull()]['CTD number'].tolist())\n",
    "    else:\n",
    "        print(\"\\t%s present in all HDR files\" % (item))\n",
    "\n",
    "\n",
    "####################################################################################################################################\n",
    "# Check if a logsheet has been populated and provided for the cruise\n",
    "####################################################################################################################################\n",
    "\n",
    "logsheets = os.path.join(logs,'%s_Log.xls' % cruiseID)\n",
    "if os.path.exists(logsheets):\n",
    "    # Load CTD event metadata from logsheets\n",
    "    print('\\nLogsheet file saved in archive. Loading metadata from the logsheet file.')\n",
    "    ctd_log = pd.read_excel(logsheets, \n",
    "                               sheet_name='CTD logs',\n",
    "                               usecols = \"C,D,E,F,H,O,P,R\")\n",
    "    ctd_log.columns = ['Cruise','Event number','CTD Cast number','Standard Station Name','CTD number',                        \n",
    "                       'Latitude [degrees_north]','Longitude [degrees_east]','Bot. depth [m]']\n",
    "    ctd_log['Event number'] = ctd_log['Event number'].astype(int)\n",
    "    ctd_log['CTD number']  = ctd_log['CTD number'].str.upper()\n",
    "\n",
    "    print(\"\\tNumber of CTD events in logsheet: %s\" % len(ctd_log))\n",
    "    \n",
    "    # Check cruise matches ID provided for processing and is unique within the logsheet\n",
    "    log_cruise_values = ctd_log['Cruise'].unique().tolist()\n",
    "    if len(log_cruise_values)!=1:\n",
    "        print(\"\\033[1;31m\\nACTION *** Multiple cruises in the logsheet. Please check for typos or split logsheet by cruises.\\033[0m\")\n",
    "        print(\"\\tCruises listed in logsheet: %s\" % log_cruise_values)\n",
    "    else:\n",
    "        if log_cruise_values[0] != cruiseID:\n",
    "            print(\"\\033[1;31m\\nACTION *** Cruise recorded in the logsheet does not match the cruise ID provided for processing. Please correct the logsheet.\\033[0m\")\n",
    "            print(\"\\tCruise listed in logsheet: %s\" % log_cruise_values[0])\n",
    "    \n",
    "    # Check CNV filenames against filenames in logsheet\n",
    "    a = list(set(hexfilelist) - set(ctd_log['CTD number'].unique().tolist()))\n",
    "    b = list(set(ctd_log['CTD number'].unique().tolist()) - set(hexfilelist))\n",
    "    \n",
    "    if len(a) !=0 or len(b) != 0:\n",
    "        print('\\033[1;31m\\nACTION *** Discrepancy in available CTD metadata between the Log and the header files. ***\\033[0m')\n",
    "        print(\"\\tFilenames not in the CTD logsheet \\t\\t\\t\\t%s\" % a)\n",
    "        print(\"\\tCTD logsheet filenames without files in the raw data folder \\t%s\" % b)\n",
    "\n",
    "else:\n",
    "    print('Logsheet file not provided.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## This cell below generates a list of sensors, voltage channels and sensor calibration coefficients by cast from the configuration files\n",
    "\n",
    "### Before starting CTD processing review list of sensors:\n",
    "- Have sensors been switched between channels?\n",
    "- Are serial numbers consistant or show changes during the cruise?\n",
    "- Are all sensors you expect to see in the list?\n",
    "- Does this match what you expect to see?\n",
    "\n",
    "### Also perform a sensor calibration coefficients check:\n",
    "- Are the coefficients for each sensor consistent across the casts? \n",
    "- If they change does this match records of instrument changes from the CTD technician?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "                                                                                ### Cell 3 ###\n",
    "sensor_configuration_and_labels = sensor_configuration.sensor_config(raw, cruiseID)\n",
    "   \n",
    "df_cast_sensors = sensor_configuration_and_labels['cast_sensors']\n",
    "master_sensor_df = sensor_configuration_and_labels['cast_labels']\n",
    "\n",
    "display(master_sensor_df)\n",
    "\n",
    "master_sensor_table = os.path.join(out,'master_sensor_table.csv')\n",
    "master_sensor_df.to_csv(master_sensor_table)\n",
    "\n",
    "master_sensor_coeffs = {'OxygenSensor': ['Soc','offset','A','B','C','D0','D1','D2','E','Tau20','H1','H2','H3'],\n",
    "                        'TurbidityMeter': ['ScaleFactor','DarkVoltage'],\n",
    "                        'WET_LabsCStar': ['M','B','PathLength'],\n",
    "                        'TransWetlabAC3Sensor': ['Ch2o','Vh2o','VDark','X'],\n",
    "                        'FluoroWetlabWetstarSensor': ['ScaleFactor','Vblank'],\n",
    "                        'AltimeterSensor': ['ScaleFactor','Offset'],\n",
    "                        'FluoroWetlabECO_AFL_FL_Sensor': ['ScaleFactor','Vblank'],                      \n",
    "                       }\n",
    "\n",
    "sensor_counts = sensor_configuration.get_sensor_coefficients(master_sensor_coeffs = master_sensor_coeffs,\n",
    "                                                          df_cast_sensors=df_cast_sensors,\n",
    "                                                          raw_data_directory=raw,\n",
    "                                                          output_directory=out)          \n",
    "display(sensor_counts)\n",
    "print(\"Sensor coefficient summaries saved to: %s\" % out)\n",
    "\n",
    "### Cell 10 ###\n",
    "#check if psa file is present from previous run - may affect file list in driver.txt and running of Seabird software\n",
    "print(psa)\n",
    "psanums=[]\n",
    "for fname in os.listdir(psa):\n",
    "    if fname.endswith('.psa'):\n",
    "        psanums.append(fname)\n",
    "        os.remove(os.path.join(psa,fname))\n",
    "\n",
    "if len(psanums) > 0:\n",
    "    print('There were '+ str(len(psanums)) + ' psa files in the psa folder. These have now been removed prior to the fresh batch run.\\n') \n",
    "else:\n",
    "    print('No psa files in psa folder. All good to proceed.\\n')\n",
    "\n",
    "data = ctd.generate_psa_files(sensor_counts=sensor_counts,\n",
    "                              PSA_template_folder = PSA_template_folder,\n",
    "                              raw=raw,\n",
    "                              screen_2Hz=screen_2Hz,\n",
    "                              bottle=bottle,\n",
    "                              psa=psa,\n",
    "                              proc_mode=proc_mode)\n",
    "\n",
    "print(\"\\nIf hex or xmlcon files are missing further investigation and intervention required.\")\n",
    "print(\"\\n\\tCasts without .HEX file: %s\" % len(data.hexMissing))\n",
    "print(\"\\tCasts missing .HDR file only: %s\" % len(data.headerMissing))\n",
    "print(\"\\tCasts missing .BL file only or .BL file is empty: %s\" % len(data.blMissing))\n",
    "print(\"\\tCasts missing both .HDR and .BL file: %s\" % len(data.headerANDblMissing))\n",
    "print(\"\\tCasts with .HDR and .BL file: %s\\n\" % len(data.headerANDblPresent))\n",
    "print(\"\\tCasts with .CNV file: %s\\n\" % len(data.cnvPresent))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial data conversion and bottle file creation\n",
    "<img align=\"centre\" src=\"_code_software/img/part1.jpg\">\n",
    "\n",
    "\n",
    "### The following cell creates a batch file for running SeaBird Data Processing software:\n",
    "\n",
    "It runs the Data Conversion, Bottle Summary, WildEdit, Filter and CellTM routines from the SBE software, as called from the command line. In the last step BinAvg creates a 2Hz version of the data are created for provisional plotting to identify the surf soak and end of cast, as well as determine the alignment of the oxygen sensor (more details in the next step)\n",
    "The cell then calls on and runs the SeaBird data processing modules to batch process the data. These seabird GUI 'windows' might pop up on your screen or be seen as active on your taskbar\n",
    "- It is required that a relatively new version of Seabird Data Processing Software be installed on the machine running the Notebooks\n",
    "\n",
    "### Below is the Sea-Bird Data Processing PSA file default setup for MI Ocean Climate cruise CTDs\n",
    "#### Data Conversion\n",
    "This SBE Data Processing module converts the raw files (.HEX) to ASCII and applies calibrations as provided by the instrument configuration file (.XMLCON). Before running this stage:\n",
    "1. The XMLCON file must be checked for accuracy and updated as required.\n",
    "2. All sensor outputs should be converted to engineering units as recommended by the manufacturer.\n",
    "3. Output at full 24 Hz resolution.\n",
    "\n",
    "The setup for the Data Conversion module applied in this script is shown here:\n",
    "\n",
    "<img align=\"centre\" src=\"_code_software/img/datconv.jpg\">\n",
    "\n",
    "#### Bottle Summary\n",
    "This module calculates sensor values covering the period of bottle firing on the up cast for each bottle on the rosette.\n",
    "\n",
    "The setup for the Bottle Summary module applied in this script is shown here:\n",
    "\n",
    "<img align=\"centre\" src=\"_code_software/img/botsumm.jpg\">\n",
    "\n",
    "#### Wild Edit\n",
    "This module should be run to remove pressure spikes if present. Pressure spikes can be identified from the pressure vs time plots. Wild Edit must be run on pressure **only** and before Filter as pressure spikes can cause Filter to smooth data incorrectly.\n",
    "\n",
    "Note that if a data file is particularly corrupted, WildEdit might need to be run more than once, with different block sizes nd number of standard deviations.\n",
    "\n",
    "#### Filter\n",
    "Filter must be run on the pressure channel before any editing is carried out. Filter smoothes out response-time issues in the sensors, which may affect processing at later stages, such as for CellTM. The typical filter time constant is equal to fours times the scan rate. For the SBE911<i>plus</i> pressure sensor this is 0.15 seconds.\n",
    "\n",
    "#### Cell Thermal Mass\n",
    "Cell Thermal Mass filters conductivity cell thermal mass effect from the measured conductivity. The recommended Sea-Bird settings should be applied to both primary and secondary sensors.\n",
    "\n",
    "For SBE9<i>plus</i> with TC duct and 3000 rpm pump the recommendations are as follows:<br>\n",
    "alpha = 0.03<br>\n",
    "1/beta = 7\n",
    "\n",
    "The setup for respective modules applied in this script are shown here:\n",
    "<img align=\"centre\" src=\"_code_software/img/phase1.jpg\"> \n",
    "### Order of processing steps follow previous work carried out by BODC and associated scientists\n",
    "\n",
    "### This cell also carries out aditional steps as follows:\n",
    "- determine the down and up-cast components of a profile and collate all profiles into single CSV file,\n",
    "- merge multiple files from a cast where data loggin interupted resulting in multiple files for the cast.\n",
    "- determine the start of the down-cast (minimum depth/pressure of the down-cast after the pump has switched on)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 4 ###\n",
    "\n",
    "if len(os.listdir(psa))>0:    \n",
    "    # Set the batch processing driver file\n",
    "    driver1 = os.path.join(filepath,'driver1.txt')\n",
    "\n",
    "    # Add processing stages to the driver file\n",
    "    drv = open(driver1,'w')\n",
    "\n",
    "    # Check for each psa individually. Add processing stages to the driver file\n",
    "    for item in os.listdir(psa):\n",
    "        if any(x in item for x in('datcnv_headerANDblPresent','datcnv_headerANDblMissing','datcnv_headerMissing','datcnv_blMissing' )):        \n",
    "            drv.write(f\"datcnv /p{os.path.join(psa,item)}\\n\")\n",
    "    for item in os.listdir(psa):\n",
    "        if 'MI_botsum' in item:\n",
    "            drv.write(f\"bottlesum /p{os.path.join(psa,item)}\\n\")       \n",
    "    for item in os.listdir(psa):\n",
    "        if 'wildedit' in item:\n",
    "            drv.write(f\"wildedit /p{os.path.join(psa,item)}\\n\")\n",
    "    for item in os.listdir(psa):\n",
    "        if 'filter' in item:\n",
    "            drv.write(f\"filter /p{os.path.join(psa,item)}\\n\")\n",
    "    for item in os.listdir(psa):\n",
    "        if 'cellTM' in item:\n",
    "            drv.write(f\"cellTM /p{os.path.join(psa,item)}\\n\")\n",
    "    for item in os.listdir(psa):\n",
    "        if 'binavg2Hz' in item:\n",
    "            drv.write(f\"binavg /p{os.path.join(psa,item)}\\n\")\n",
    "    drv.close()\n",
    "\n",
    "    # Run SBE batch using driver file\n",
    "    os.system(\"sbebatch %s %s %s %s\" % (driver1,raw,bottle,screen_2Hz))\n",
    "\n",
    "    print(\"Check for successful completion of batch processing\")\n",
    "    \n",
    "else:\n",
    "    print(\"Processing mode set to new files only. No new files to be processed.\")\n",
    "\n",
    "# Concatenate profile data into a csv file and flag data as either down or up cast (adds column 'cast' to the dataframe and populates as 'D' or 'U' based on being before or after the maximum depth).\n",
    "files = os.listdir(screen_2Hz)\n",
    "profile_casts = [w.upper() for w in files]\n",
    "\n",
    "# Set output file for concatenated 2Hz data\n",
    "all_2Hz_csv = os.path.join(all_2Hz,'cruise_SBEproc_2Hz.csv')\n",
    "# determine if 2Hz data file already exists\n",
    "all_2Hz_file_exists = os.path.exists(all_2Hz_csv)\n",
    "\n",
    "if proc_mode == 0 or all_2Hz_file_exists==False:\n",
    "    data_all = data_processing.cnv2df(cruiseID,\n",
    "                                      files,\n",
    "                                      params=[],\n",
    "                                      raw_folder = raw,\n",
    "                                      directory = screen_2Hz,\n",
    "                                      ud_id = True,\n",
    "                                      z_cord = 'prDM',\n",
    "                                      )\n",
    "    print(\"Number of CTD events loaded from processed files: %s\" % len(data_all.profile.unique().tolist()))\n",
    "    print(\"Number of data rows loaded from processed files: %s\" % len(data_all))\n",
    "    \n",
    "    # Merge split files from single cast. Uses the dictionary \"combined\" set up at the start of the file.\n",
    "    data_all = data_processing.combine_files2cast(data_all,combined,3)\n",
    "\n",
    "    # Save aggregated DataFrame for archive\n",
    "    data_all.to_csv(all_2Hz_csv)\n",
    "    df_profile = data_all.copy(deep=True)\n",
    "       \n",
    "elif proc_mode == 1 and all_2Hz_file_exists==True:\n",
    "    all_2Hz_existing = pd.read_csv(all_2Hz_csv, index_col = 0)\n",
    "    all_2Hz_casts = all_2Hz_existing['profile'].unique().tolist()\n",
    "    #combined = {'CE21003_CTD002.CNV': [{'CE21003_CTD002B.CNV': 'U'}]}\n",
    "    if combined != None:\n",
    "        for item in combined.keys():\n",
    "            if item in all_2Hz_casts:\n",
    "                for xitem in combined[item].keys():\n",
    "                    all_2Hz_casts.append(xitem)\n",
    "    new_casts = list(set(profile_casts) - set(all_2Hz_casts))\n",
    "    \n",
    "    if len(new_casts)>0:\n",
    "        files_new = [w.lower() for w in new_casts]\n",
    "        data_add = data_processing.cnv2df(cruiseID,\n",
    "                                          files_new,\n",
    "                                          params=[],\n",
    "                                          raw_folder = raw,\n",
    "                                          directory = screen_2Hz,\n",
    "                                          ud_id = True,\n",
    "                                          z_cord = 'prDM',\n",
    "                                          )\n",
    "        print(\"Number of new CTD events loaded from processed files: %s\" % len(data_add.profile.unique().tolist()))\n",
    "        print(\"Number of new data rows loaded from processed files: %s\" % len(data_add))\n",
    "        data_all = pd.concat([all_2Hz_existing, data_add], ignore_index=False, sort=False).sort_index()\n",
    "        \n",
    "        # Merge split files from single cast. Uses the dictionary \"combined\" set up at the start of the file.\n",
    "        data_all = data_processing.combine_files2cast(data_all,combined,3)\n",
    "\n",
    "        # Save aggregated DataFrame for archive\n",
    "        data_all.to_csv(all_2Hz_csv)\n",
    "        df_profile = data_all.copy(deep=True)\n",
    "    else:\n",
    "        print(\"No newly processed casts for concatenation to existing data file.\")\n",
    "        df_profile = all_2Hz_existing.copy(deep=True)\n",
    "        \n",
    "######## determine the start of the down-cast (minimum depth/pressure of the down-cast after the pump has switched on).\n",
    "\n",
    "# Get minimum downcast depth after the pump switches on\n",
    "pump_csv = os.path.join(out,'pump_on_time.csv')\n",
    "# determine if surface soak file already exists\n",
    "pump_file_exists = os.path.exists(pump_csv)\n",
    "\n",
    "# Determine whether previously saved pump and surface soak data are to be over-written\n",
    "if proc_mode == 0 or pump_file_exists==False:\n",
    "    pumpdf = data_processing.start_dcast(df_profile,'profile','prDM')\n",
    "    pumpdf.to_csv(pump_csv)\n",
    "    print(\"\\nSurface soak details run for all processed casts and saved to: %s\" % pump_csv)\n",
    "elif proc_mode == 1 and pump_file_exists==True:\n",
    "    pumpdf_existing = pd.read_csv(pump_csv, index_col = 0)\n",
    "    pump_casts = pumpdf_existing['profile'].unique().tolist()\n",
    "    new_casts = list(set(profile_casts) - set(pump_casts))\n",
    "    if len(new_casts)>0:\n",
    "        pumpdf_add = data_processing.start_dcast(df_profile[df_profile['profile'].isin(new_casts)],'profile','prDM')\n",
    "        pumpdf_merged = pd.concat([pumpdf_existing, pumpdf_add], ignore_index=False, sort=False).sort_index()\n",
    "        pumpdf_merged.to_csv(pump_csv)\n",
    "        print(\"\\nSurface soak details for newly processed casts appended to existing records and saved to: %s\" % pump_csv)\n",
    "    else:\n",
    "        print(\"No newly processed casts for surface soak identification.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot data to check surface soak identification and interactively determine oxygen sensor alignment\n",
    "<img align=\"centre\" src=\"_code_software/img/part2.jpg\">\n",
    "\n",
    "## Run following cell to display surface soak and oxygen alignment dashboards\n",
    "- If required, adjust the point at the end of the surface soak (per cast) by selecting a single point on plot and clicking 'Set Cast Start'\n",
    "- If required, assess best oxygen alignment, but can only be applied per cruise and not on a cast by cast basis\n",
    "- For your info: on Marine Institute cruises, with their specific oxygen sensor setup, experience shows that a 2 second allignment works throughout for every cruise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 5 ###\n",
    "# Load 2Hz CTD data\n",
    "profile_data = os.path.join(all_2Hz,'cruise_SBEproc_2Hz.csv')\n",
    "df_profile = pd.read_csv(profile_data)\n",
    "df_profile = df_profile.rename(columns={'Unnamed: 0': 'Cycles'})\n",
    "\n",
    "# Load pump on time for each cast\n",
    "pumpdf = pd.read_csv(os.path.join(out,'pump_on_time.csv'))\n",
    "pumpdf = pumpdf.rename(columns={'Unnamed: 0': 'Cycles'})\n",
    "\n",
    "surface_soak_bokeh = ctd_bokeh.bokeh_layout(profile_data=df_profile,\n",
    "                                           pump_data=pumpdf,\n",
    "                                           output_path=out,\n",
    "                                           downcast_data=None)\n",
    "\n",
    "output_notebook(INLINE)\n",
    "show(surface_soak_bokeh.surface_soak_screening, notebook_handle=True)\n",
    "push_notebook()\n",
    "\n",
    "if 'sbeox0Mm/L' in surface_soak_bokeh.param_list:\n",
    "    # Set oxygen alignment value\n",
    "    oxygen1_align = widgets.Text(value = oxy_align_default)\n",
    "    print(\"Update oxygen sensor 1 alignment value (in seconds) here if appropriate:\")\n",
    "    display(oxygen1_align)\n",
    "if 'sbeox1Mm/L' in surface_soak_bokeh.param_list:\n",
    "    oxygen2_align = widgets.Text(value = oxy_align_default)\n",
    "    print(\"Update oxygen sensor 2 alignment value (in seconds) here if appropriate:\")\n",
    "    display(oxygen2_align)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complete CTD processing steps\n",
    "\n",
    "### The following cell performs the following tasks:\n",
    "- load data from archived SDB processed to 2 Hz CSV file into a dataframe,\n",
    "- if oxygen sensors are deployed, then advance oxygen voltage channel(s) by number of seconds determined from the interactive plots above (or the set defualt value),\n",
    "- at the end of the up-cast, flag the 'cast' where pressure is less than 2 dbar (~height of the rig) to indicate rig breaking surface (flag = 'S'),\n",
    "- load the down-cast start time (as seconds elapsed since data acquisition started) and flag the 'cast' to indicate the surface soak (flag = 'SS'),\n",
    "- drop channels labeled 'NotInUse','pumps','Start',\n",
    "- set temperatures outside range -5 to 40 degrees C as NaN,\n",
    "- recalculate practical salinities, potential temperature, sigma-theta and sound velocity (EOS-80 toolbox),\n",
    "- recalculate the oxygen concentration in umol/L and saturation using algorithms defined in SBE Application notes 64 and 64-3,\n",
    "- save data to CSV file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 6 ###\n",
    "# Load data from csv\n",
    "profile_data = os.path.join(all_2Hz,'cruise_SBEproc_2Hz.csv')\n",
    "\n",
    "df = pd.read_csv(profile_data)\n",
    "\n",
    "p_full_list = df.columns.tolist()\n",
    "\n",
    "# Where oxygen sensor(s) deployed on the CTD rig align sensors based on visual check results\n",
    "if 'sbeox0V' in p_full_list:\n",
    "    try:\n",
    "        oxygen1_align.value\n",
    "        oxy1_align_value = oxygen1_align.value\n",
    "    except NameError:\n",
    "        oxy1_align_value = oxy_align_default\n",
    "    # Update the oxygen voltage alignment\n",
    "    df['sbeox0V'] = df['sbeox0V'].shift(-2*int(oxy1_align_value))\n",
    "    # Add the oxygen sensor voltage change {Sets the oxygen voltage dV/dt for use later in the oxygen calculations}\n",
    "    df['oxy0dV/dt'] = df['sbeox0V'].diff()/df['timeS'].diff()\n",
    "\n",
    "    print(\"Oxygen sensor 1 voltage alignment value: %s seconds\" % oxy1_align_value)\n",
    "    \n",
    "if 'sbeox1V' in p_full_list:\n",
    "    try:\n",
    "        oxygen2_align.value\n",
    "        oxy2_align_value = oxygen2_align.value\n",
    "    except NameError:\n",
    "        oxy2_align_value = oxy_align_default\n",
    "    # Update the oxygen voltage alignment\n",
    "    df['sbeox1V'] = df['sbeox1V'].shift(-2*int(oxy2_align_value))\n",
    "    # Add the oxygen sensor voltage change {Sets the oxygen voltage dV/dt for use later in the oxygen calculations}\n",
    "    df['oxy1dV/dt'] = df['sbeox1V'].diff()/df['timeS'].diff()\n",
    "\n",
    "    print(\"Oxygen sensor 2 voltage alignment value: %s seconds\\n\" % oxy2_align_value)\n",
    "\n",
    "# Load profile start time from file\n",
    "pumpdf = pd.read_csv(os.path.join(out,'pump_on_time.csv'))\n",
    "\n",
    "pumpdf.columns=['Cycles','profile','Start','prDM']\n",
    "\n",
    "print(\"Number of CTDs with surface soak identified: %s\\n\" % len(pumpdf))\n",
    "\n",
    "# Add the start timeS for each profile from the manual inspection of the plot {Removes the surface soak and data where the frame is partially out of the water}\n",
    "print(\"Number of rows in profile dataframe prior to start/end merge: %s\" % len(df))\n",
    "df = pd.merge(df, pumpdf[['profile','Start']], on = 'profile', how = 'inner')\n",
    "df['profile'] = df['profile'].str.replace('.CNV','',regex=True)\n",
    "print(\"Number of rows in profile dataframe after start/end merge: %s\\n\" % len(df))\n",
    "\n",
    "# Set cast channel to indicate cycles where the time elapsed covers the surface soak ('SS') and the rig is close to the surface ('S').\n",
    "# In this script the is cst is flagged 'S' where depth is less than 2m (approx. height of the CTD frame) at the end of the upcast.\n",
    "surface_depth = 2\n",
    "mask = (df['depSM'] < surface_depth) & (df['cast']=='U')\n",
    "df.loc[mask,'cast'] = 'S'\n",
    "mask = df.timeS < df.Start\n",
    "df.loc[mask,'cast'] = 'SS'\n",
    "\n",
    "# Set the columns to be kept {analgous to SBE Strip routine}\n",
    "p_full_list = df.columns.tolist()\n",
    "\n",
    "keep_cols = ['profile','cast']\n",
    "for item in p_full_list:\n",
    "    if item not in ['profile','cast','NotInUse','NotInUse.1','NotInUse.2','NotInUse.3','NotInUse.4','NotInUse.5','NotInUse.6','NotInUse.7','pumps','Start','Unnamed: 0']:\n",
    "        keep_cols.append(item)\n",
    "\n",
    "df = df[keep_cols].copy(deep=True)\n",
    "df = df.rename(columns={'profile': 'CTD number'})\n",
    "\n",
    "# Set temperature values to nan for outside range -5 to 40 degrees Celsius\n",
    "df.loc[df['t090C']<-5,'t090C']=np.nan\n",
    "df.loc[df['t090C']>40,'t090C']=np.nan\n",
    "\n",
    "df.loc[df['t190C']<-5,'t190C']=np.nan\n",
    "df.loc[df['t190C']>40,'t190C']=np.nan\n",
    "\n",
    "## Rederive calculated channels\n",
    "# Recalculate salinities (practical)\n",
    "if 'c0S/m' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['sal00'] = seawater.eos80.salt(df['c0S/m'].div(4.2914),df['t090C'],df['prDM'])\n",
    "else:\n",
    "    print(\"Salinity (sal00) not calculate as one of c0S/m, t090C an prDM not in file.\")\n",
    "if 'c1S/m' and 't190C' and 'prDM' in p_full_list:\n",
    "    df['sal11'] = seawater.eos80.salt(df['c1S/m'].div(4.2914),df['t190C'],df['prDM'])\n",
    "else:\n",
    "    print(\"Salinity (sal11) not calculate as one of c1S/m, t190C an prDM not in file.\")\n",
    "\n",
    "# Generate potential temperature\n",
    "if 'sal00' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['potemp090C'] = seawater.eos80.ptmp(df['sal00'],df['t090C'],df['prDM'],pr=0)\n",
    "else:\n",
    "    print(\"Potential temperature (potemp090C) not calculate as one of sal00, t090C an prDM not in file.\")\n",
    "if 'sal11' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['potemp190C'] = seawater.eos80.ptmp(df['sal11'],df['t190C'],df['prDM'],pr=0)\n",
    "else:\n",
    "    print(\"Potential temperature (potemp190C) not calculate as one of sal11, t190C an prDM not in file.\")\n",
    "\n",
    "# Generate sigma-theta\n",
    "if 'sal00' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['sigma-theta00'] = seawater.eos80.pden(df['sal00'],df['t090C'],df['prDM'],pr=0) - 1000\n",
    "else:\n",
    "    print(\"Potential density (sigma-theta00) not calculate as one of sal00, t090C an prDM not in file.\")\n",
    "if 'sal11' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['sigma-theta11'] = seawater.eos80.pden(df['sal11'],df['t190C'],df['prDM'],pr=0) - 1000\n",
    "else:\n",
    "    print(\"Potential density (sigma-theta11) not calculate as one of sal11, t190C an prDM not in file.\")\n",
    "\n",
    "# Generate sound velocity\n",
    "if 'sal11' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['svel00'] = seawater.eos80.svel(df['sal00'],df['t090C'],df['prDM'])\n",
    "else:\n",
    "    print(\"Sound velocity (svel00) not calculate as one of sal00, t090C an prDM not in file.\")\n",
    "if 'sal11' and 't090C' and 'prDM' in p_full_list:\n",
    "    df['svel11'] = seawater.eos80.svel(df['sal11'],df['t190C'],df['prDM'])\n",
    "else:\n",
    "    print(\"Sound velocity (svel11) not calculate as one of sal11, t190C an prDM not in file.\")\n",
    "    \n",
    "# If oxygen sensor deployed on the CTD rig\n",
    "if 'sbeox0V' in df.columns.tolist():\n",
    "    # Recalculate oxygen concentration\n",
    "    # Set holding columns\n",
    "    df['sbeox0Mm/L'] = None\n",
    "    \n",
    "    # Read in oxygen sensor coefficients from \n",
    "    df_oxy0_coeffs = pd.read_csv(os.path.join(out,'sensor_coeffs_fulltable_OxygenSensor0.csv'))\n",
    "    df_oxy0_coeffs.rename(columns = {'Unnamed: 0':'CTD number'}, inplace = True)\n",
    "    df_oxy0_coeffs = df_oxy0_coeffs.set_index('CTD number')\n",
    "    \n",
    "    for index,row in df_oxy0_coeffs.iterrows():\n",
    "        coef = row.to_dict()\n",
    "        mask = df['CTD number']==index.upper()\n",
    "        df.loc[mask,'sbeox0Mm/L'] = calculations.sbe43_oxycalc(df.loc[mask,'sbeox0V'], df.loc[mask,'t090C'], df.loc[mask,'prDM'], df.loc[mask,'sal00'], coef, df.loc[mask,'oxy0dV/dt'], 'umol/L')\n",
    "    \n",
    "    # Generate oxygen saturation\n",
    "    df['sbeox0PS'] = df['sbeox0Mm/L'].div(calculations.oxysol(df['t090C'],df['sal00']).mul(44.66)).mul(100)\n",
    "\n",
    "    # If second oxygen sensor deployed on the CTD rig\n",
    "    if 'sbeox1V' in df.columns.tolist():\n",
    "        # Recalculate oxygen concentration\n",
    "        # Set holding columns\n",
    "        df['sbeox1Mm/L'] = None\n",
    "                                 \n",
    "        # Read in oxygen sensor coefficients from \n",
    "        df_oxy1_coeffs = pd.read_csv(os.path.join(out,'sensor_coeffs_fulltable_OxygenSensor1.csv'))\n",
    "        df_oxy1_coeffs.rename(columns = {'Unnamed: 0':'CTD number'}, inplace = True)\n",
    "        df_oxy1_coeffs = df_oxy1_coeffs.set_index('CTD number')\n",
    "\n",
    "        for index,row in df_oxy1_coeffs.iterrows():\n",
    "            coef = row.to_dict()\n",
    "            mask = df['CTD number']==index.upper()\n",
    "            df.loc[mask,'sbeox1Mm/L'] = calculations.sbe43_oxycalc(df.loc[mask,'sbeox1V'], df.loc[mask,'t190C'], df.loc[mask,'prDM'], df.loc[mask,'sal11'], coef, df.loc[mask,'oxy1dV/dt'], 'umol/L')\n",
    "        # Generate oxygen saturation\n",
    "        df['sbeox1PS'] = df['sbeox1Mm/L'].div(calculations.oxysol(df['t190C'],df['sal11']).mul(44.66)).mul(100)\n",
    "\n",
    "    elif 'sbeox1V' not in df.columns.tolist():\n",
    "        # Recalculate oxygen concentration\n",
    "        # Set holding columns\n",
    "        df['sbeox1Mm/L'] = None\n",
    "\n",
    "        for index,row in df_oxy0_coeffs.iterrows():\n",
    "            coef = row.to_dict()\n",
    "            mask = df['CTD number']==index.upper()\n",
    "            df.loc[mask,'sbeox1Mm/L'] = calculations.sbe43_oxycalc(df.loc[mask,'sbeox0V'], df.loc[mask,'t190C'], df.loc[mask,'prDM'], df.loc[mask,'sal11'], coef, df.loc[mask,'oxy0dV/dt'], 'umol/L')\n",
    "        # Generate oxygen saturation\n",
    "        df['sbeox1PS'] = df['sbeox1Mm/L'].div(calculations.oxysol(df['t190C'],df['sal11']).mul(44.66)).mul(100)\n",
    "\n",
    "# Save profiles ready for screening\n",
    "screen_file = os.path.join(all_2Hz,'cruise_screenready_2Hz.csv')\n",
    "df.to_csv(screen_file, index=False)\n",
    "\n",
    "print(\"Columns in data file:\")\n",
    "print(df.columns.tolist())\n",
    "\n",
    "print('\\nFile saved to: %s' % screen_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated screen for heave entrainment features\n",
    "Swell at the surface causes heave on the CTD wire while the rig is being lowered and raised through the water column. Sea-water trapped in the rig infrastructure from higher in the\n",
    "water column can flush past the sensors on the base of the rig each time the rig decelerates and raises back through the water column. This is called entrainment.\n",
    "\n",
    "Function \"heave_flag()\" iterates down the pressure channel (prDM) and flags:\n",
    "- down-cast rows where velocity is below a user defined threshold (default vel = 0.2 m/s),\n",
    "- a user defined window of cycles prior to the velocity threshold being reached (default window = 2),\n",
    "- cycles where pressure is less than the first cycle below the velocity threshold for each entrainment/heave feature.\n",
    "\n",
    "### The following input window allows users to enter preferred values, if differing from default\n",
    "- If sea-swell was considerable or if CTD rig was lowered slower than recommended, (e.g. < 0.6 m/s), then lowering the 'Velocity minimum threshold' to say 0.15 or 0.1, will allow more data to be retained\n",
    "- However, this must be balanced with preventing entrainment values from staying in the dataset. Entrainment readings are not valid and should be discarded in all but special cases.\n",
    "- For special cases, e.g. cruises that do not take a profile, but simply sample at the surface, it is suggested to run the Notebook with the Heave Function switched off (see first cell).\n",
    "- At present, there is not the ability to split the cruise, so having different thresholds per cast is not yet possible.\n",
    "\n",
    "Results are then saved to file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 7 ###\n",
    "\n",
    "velocity_widget = widgets.BoundedFloatText(value=0.20,\n",
    "                                           min=0,\n",
    "                                           max=1.0,\n",
    "                                           step=0.05,\n",
    "                                           description='Velocity minimum threshold (m/s):',\n",
    "                                           disabled=False,\n",
    "                                           style = {'description_width': '75%'}\n",
    "                                          )\n",
    "\n",
    "window_widget = widgets.BoundedIntText(value=2,\n",
    "                                       min=0,\n",
    "                                       max=10,\n",
    "                                       step=1,\n",
    "                                       description='Window of cycles before threshold:',\n",
    "                                       disabled=False,\n",
    "                                       style = {'description_width': '75%'}\n",
    "                                      )\n",
    "\n",
    "display(widgets.VBox([velocity_widget, window_widget]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visually display results of heave flagging function\n",
    "Plots are generated for showing the heave flagging (marked in red) and can be reviewed per CTD cast\n",
    "- If the velocity threshold is not sufficient, the previous cell (cell 7) can simply be re-run with a new threshold, followed by a rerun of the cell below to review the plots again \n",
    "\n",
    "N.B. The function iterates row by row, so will take some time depending on the number and depth of the casts being processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 8 ###\n",
    "if heave_mode == 0:\n",
    "    # User defined values for function\n",
    "    vel = velocity_widget.value\n",
    "    window = window_widget.value\n",
    "\n",
    "    print(\"Velocity threshold = %s\" % vel)\n",
    "    print(\"Window = %s\\n\" % window)\n",
    "\n",
    "    # Load data from file\n",
    "    file_in = os.path.join(all_2Hz,'cruise_screenready_2Hz.csv')\n",
    "    df = pd.read_csv(file_in)\n",
    "\n",
    "    df = seabird_processes.heave_flagging(df,vel,window)\n",
    "\n",
    "    file_out = os.path.join(all_2Hz,'cruise_heavescreened_2Hz.csv')\n",
    "    df.to_csv(file_out, index=False)\n",
    "\n",
    "    print(\"Heave flagging routine complete. File saved to: %s\" % file_out)\n",
    "    \n",
    "    # Load 2Hz CTD data\n",
    "    file_in = os.path.join(all_2Hz,'cruise_heavescreened_2Hz.csv')\n",
    "    df_profile = pd.read_csv(file_in)\n",
    "    df_dcasts = df_profile[df_profile['cast']=='D'][['CTD number','timeS','prDM','prDM_QC','t090C','sal00','CTDvel']]\n",
    "    df_dcasts = df_dcasts.rename(columns={'Unnamed: 0': 'Cycles'})\n",
    "    df_dcasts['prDM_QC'] = df_dcasts['prDM_QC'].astype(int).astype(str)\n",
    "    \n",
    "    heave_bokeh_layout = ctd_bokeh.bokeh_layout(profile_data=df_profile,\n",
    "                                                pump_data=pumpdf,\n",
    "                                                output_path=out,\n",
    "                                                downcast_data=df_dcasts)\n",
    "    \n",
    "    output_notebook(INLINE)\n",
    "    show(heave_bokeh_layout.heave_screening, notebook_handle=True)\n",
    "    push_notebook()\n",
    "    \n",
    "elif heave_mode == 1:\n",
    "    print(\"heave flagging turned off for this run\")\n",
    "    \n",
    "else:\n",
    "    print(\"\\033[1;31mError: value for heave_mode is not valid. Please update to 0 or 1 as appropriate.\\033[0m\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bin data excluding heave entrainment flagged rows \n",
    "\n",
    "This is the only step that erases data; all data after this point is retained and flagged where bad\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 9 ###\n",
    "print(\"Binning down cast to 1 %s\" % bin_unit_widget.value)\n",
    "    \n",
    "cast='D'\n",
    "bin_dict = {'metre': 'depSM', 'decibar': 'prDM'}\n",
    "zcord = bin_dict[bin_unit_widget.value]\n",
    "profile_id='CTD number'\n",
    "\n",
    "            ### ED removing second o2 sensor variables from end of following list 21/06/2021\n",
    "            ### RT re-added second O2 sensor variables 18/05/2022\n",
    "params_out =[]\n",
    "params_master = ['CTD number','depSM','prDM','t090C','c0S/m','t190C','c1S/m','sal00','sal11','potemp090C','potemp190C','sigma-theta00','sigma-theta11','svel00','svel11','sbeox0V','sbeox0Mm/L','sbeox0PS','sbeox1V','sbeox1Mm/L','sbeox1PS']\n",
    "metadata = ['CTD number']\n",
    "voltage_channels = ['FluoroWetlabWetstarSensor','TurbidityMeter','OxygenSensor','WET_LabsCStar','AltimeterSensor','FluoroWetlabECO_AFL_FL_Sensor']\n",
    "\n",
    "\n",
    "if heave_mode == 0:\n",
    "    # Load data from file\n",
    "    file_in = os.path.join(all_2Hz,'cruise_heavescreened_2Hz.csv')\n",
    "    df_profiles2Hz_screened = pd.read_csv(file_in)\n",
    "else:\n",
    "    file_in = os.path.join(all_2Hz,'cruise_screenready_2Hz.csv')\n",
    "    df_profiles2Hz_screened = pd.read_csv(file_in)\n",
    "    df_profiles2Hz_screened['prDM_QC'] = 0\n",
    "    \n",
    "for channel in voltage_channels:\n",
    "    i = 0\n",
    "    for column in df_profiles2Hz_screened.columns:\n",
    "        if channel in column:\n",
    "            new_label = channel+'_'+str(i)\n",
    "            df_profiles2Hz_screened.rename(columns = {column:new_label}, inplace = True)\n",
    "            params_master.append(new_label)\n",
    "            i = i + 1\n",
    "    \n",
    "for item in params_master:\n",
    "    if item in df_profiles2Hz_screened.columns:\n",
    "        params_out.append(item)\n",
    "\n",
    "df_binned = seabird_processes.bin_data(df_profiles2Hz_screened,cast,zcord,profile_id,params_out)\n",
    "\n",
    "print(params_out)    \n",
    "    \n",
    "# Convert oxygen values from umol/l to ml/l and umol/kg\n",
    "if 'sbeox0Mm/L' in params_out:\n",
    "    df_binned['sbeox0Mm/kg'] = df_binned['sbeox0Mm/L']/((df_binned['sigma-theta00']+1000)/1000)\n",
    "\n",
    "    df_binned['sbeox0ml/l'] = df_binned['sbeox0Mm/L']/44.66\n",
    "    \n",
    "if 'sbeox1Mm/L' in params_out:\n",
    "    df_binned['sbeox1Mm/kg'] = df_binned['sbeox1Mm/L']/((df_binned['sigma-theta11']+1000)/1000)\n",
    "\n",
    "    df_binned['sbeox1ml/l'] = df_binned['sbeox1Mm/L']/44.66\n",
    "\n",
    "uncal_file = os.path.join(out,'cruise_data_uncal_1mbinned.csv') \n",
    "df_binned.to_csv(uncal_file, index=False)\n",
    "\n",
    "print(\"Pre-calibration processing completed. File saved to: %s\" % uncal_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge processed profiles with metadata from an electronic logsheet\n",
    "\n",
    "<span style='color:Blue'> \n",
    "\n",
    "### A logsheet is not a requirement but only limited metadata will be read from headers and retained in output file if no logsheet is provided\n",
    "\n",
    "</span>\n",
    "\n",
    "## Collate bottle firing details with CTD metadata, Bedford numbers and SBE35 temperatures for the cruise\n",
    "\n",
    "This cell creates a bottle summary file, useful to summarise sensor values and metadata at time of bottle firing\n",
    "\n",
    "See file in folder ...\\output\\bottle\\..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 10 ###\n",
    "# Check if data frames already exist in notebook memeory and if so clear from memory to remove risk of inconsistencies   \n",
    "try:\n",
    "    ctd_events\n",
    "    del(ctd_events)\n",
    "    gc.collect()\n",
    "    #print(\"Deleting ctd_events dataframe that already exists to clear history for this cell\")\n",
    "except NameError:\n",
    "    #print(\"Dataframe ctd_events does not exist in notebook memory. OK to proceed.\")\n",
    "    True\n",
    "try:\n",
    "    ctd_log\n",
    "    del(ctd_log)\n",
    "    gc.collect()\n",
    "    #print(\"Deleting ctd_log dataframe that already exists to clear history for this cell.\")\n",
    "except NameError:\n",
    "    #print(\"Dataframe ctd_log does not exist in notebook memory. OK to proceed.\")\n",
    "    True\n",
    "# Check if logsheet file exists\n",
    "\n",
    "            ### ED change CTD_Log.xlsx file to .xls format before running when using newer versions of XLSXRD\n",
    "            ### And change file extension here to use it 21/06/2021\n",
    "\n",
    "# Load cast start seconds from the pump_on_time.csv file into a dataframe\n",
    "pumpdf = pd.read_csv(os.path.join(out,'pump_on_time.csv'))\n",
    "pumpdf.columns = ['Cycles','CTD number','Start','prDM']\n",
    "pumpdf['CTD number'] = pumpdf['CTD number'].str.replace('.CNV','',regex=True)\n",
    "pumpdf['Start'] = pd.to_timedelta(pumpdf['Start'], unit='seconds')\n",
    "pumpdf = pumpdf[['CTD number','Start']]\n",
    "\n",
    "ctd_events = data_processing.create_ctd_events(cruiseID=cruiseID,\n",
    "                                               raw_directory=raw,\n",
    "                                               logs=logs,\n",
    "                                               pumpdf=pumpdf)\n",
    "\n",
    "try:\n",
    "    display(ctd_events)\n",
    "    df = data_processing.merge_data_with_metadata(cruiseID,\n",
    "                                                  output_directory=out,\n",
    "                                                  ctd_events=ctd_events,\n",
    "                                                  logs=logs)\n",
    "    data_processing.create_output_csv_for_fisheries(df=df, output_directory=out)\n",
    "    bottle_processing.create_bottle_summary(bottle_directory=bottle,\n",
    "                                            cruiseID=cruiseID,\n",
    "                                            logs=logs,\n",
    "                                            ctd_events=ctd_events,\n",
    "                                            sbe35_raw=sbe35_raw,\n",
    "                                            raw_folder=raw)\n",
    "    \n",
    "except NameError as e:\n",
    "    print(\"\\033[1;31mMetadata merge not possible without a populated logsheet XLSX file.\\33[0m\")\n",
    "    print(e)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display the Visualisation dashboard:\n",
    "\n",
    "Successfull running of the following cell is a great indication that everything has worked correctly and the process is almost complete.\n",
    "\n",
    "The output display provides an opportunity to view the processed data for the first time, e.g. while still at sea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                                                ### Cell 11 ###\n",
    "# Load binned CTD data\n",
    "uncal_file = os.path.join(out,'%s_CTDprofiles_uncal_1mbinned_meta.csv' % cruiseID) \n",
    "df_plot = pd.read_csv(uncal_file, parse_dates = ['CTD_start'])\n",
    "\n",
    "profile_list = df_plot['CTD number'].unique().tolist()\n",
    "\n",
    "# Convert postion to northing and eastings for plotting    \n",
    "df_plot['Eastings'], df_plot['Northings'] = calculations.merc_from_arrays(df_plot['Latitude [degrees_north]'], df_plot['Longitude [degrees_east]'])\n",
    "\n",
    "binning_layout = ctd_bokeh.bokeh_layout(profile_data = df_plot,\n",
    "                                       pump_data=None,\n",
    "                                       output_path=out,\n",
    "                                       downcast_data=None)\n",
    "\n",
    "output_notebook(INLINE)\n",
    "show(binning_layout.bin_screen, notebook_handle=True)\n",
    "push_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## At this stage of the notebook processing of the profiles is complete\n",
    "\n",
    "### Now backup processed files on-board\n",
    "\n",
    "The following cell automates the backup of processed files onboard the MI research vessels.\n",
    "\n",
    "***Only run this cell if on an MI ship, or alternately, adjust the file directory path to use cell for your own automated backup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Python 3 shutil.copytree function does not allow you to overwrite an existing directory so need to delete the directory before copying it\n",
    "#  Python 3.8 and higher does have this functionality so if we upgrade should not remove the directoory but just overwrite it using dirs_exist_ok=True\n",
    "\n",
    "#time.sleep(2.5)\n",
    "\n",
    "#if not os.path.exists(os.path.normpath(os.path.join('Z:/2.1 Oceanographic', cruiseID))):\n",
    "#    os.mkdir(os.path.normpath(os.path.join('Z:/2.1 Oceanographic', cruiseID)))\n",
    "    \n",
    "#if not os.path.exists(backup_server_path):\n",
    "#    os.mkdir(backup_server_path)\n",
    "\n",
    "#if os.path.exists(backup_server_path):\n",
    "#    shutil.rmtree(backup_server_path)\n",
    "\n",
    "#shutil.copytree(os.path.join(filepath, 'output'), backup_server_path)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "ctd_env312_notebooks",
   "language": "python",
   "name": "ctd_env312_notebooks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}