Calibration-No-Normalization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calibration and Fusion\n",
    "\n",
    "Python script from https://gitlab.eurecom.fr/nautsch/pybosaris/tree/master. Python implementation of BOSARIS toolkit https://sites.google.com/site/bosaristoolkit/. The script is based on `linear_calibrate_scores.m`, `train_linear_calibration.m` and `train_binary_classifier.m` from the BOSARIS Matlab toolkit.\n",
    "\n",
    "**Ressources**\n",
    "\n",
    "* [The BOSARIS Toolkit: Theory, Algorithms and\n",
    "Code for Surviving the New DCF](https://arxiv.org/pdf/1304.2865.pdf)\n",
    "* [The BOSARIS Toolkit User Guide: Theory, Algorithms and Code for Binary Classifier Score Processing](https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxib3NhcmlzdG9vbGtpdHxneDozOTEwZjAzZmM3ZThmNjE0)\n",
    "\n",
    "**Notes**\n",
    "\n",
    "* la calibration ne change pas l'EER, mais elle change la valeur du seuil qu'il faut utiliser pour obtenir le point EER;\n",
    "* la calibration va changer le \"detection cost\", car il est évalué pour une valeur fixe du seuil;\n",
    "* par contre le \"minimum detection cost\" est celui qu'on obtient en choisissant un seuil qui minimise le coût de détection, donc la calibration ne changera pas le \"minimum detection cost\" non plus;\n",
    "* quand on soumet au NIST, on donne une liste de trials avec leurs scores, et le NIST trace la courbe DET, détermine le seuil pour l'EER, et calcule un \"detection cost\" avec un seuil qui est prédeterminé.\n",
    "* Dans le plan d'évaluation 2019, il y a en fait deux seuils, nommés log(\\beta_1) et log(\\beta_2), qui seront utilisés et la mesure principale, le C_primary, est la moyenne des deux coûts de détection correspondants.\n",
    "* en conclusion, le C_primary sera influencé par la calibration. En principe, une calibration idéale ferait qu'on obtiendrait la valeur minimale de C_primary  pour les seuils fixes log(\\beta_1) et log(\\beta_2) .\n",
    "\n",
    "**DET Curve**\n",
    "\n",
    "<img src=\"imgs/det-curve.jpg\" alt=\"IMAGE ALT TEXT HERE\" align=\"middle\" width=\"500\" border=\"1\" /></a>\n",
    "\n",
    "\n",
    "https://link.springer.com/referenceworkentry/10.1007%2F978-0-387-73003-5_643\n",
    "\n",
    "Detection Error Tradeoff curves are ROC (receiver operating characteristic) type curves showing the range of operating points of systems performing detection tasks as a threshold is varied to alter the miss and false alarm rates and plotted using a normal deviate scale for each axis. DET curves have the property that if the underlying score distributions for the two types of trials are normal, the curve becomes a straight line. They have been widely used to present the performance characteristics of speaker recognition systems.\n",
    "\n",
    "**DCF Plot**\n",
    "\n",
    "*Good Calibration*\n",
    "\n",
    "<img src=\"imgs/dcf-curve.jpg\" alt=\"IMAGE ALT TEXT HERE\" align=\"middle\" width=\"500\" border=\"1\" /></a>\n",
    "\n",
    "https://arxiv.org/pdf/1304.2865.pdf\n",
    "\n",
    "**The following description refers to the above plot.**\n",
    "\n",
    "The plot show curves for dev database used to train the calibration. The actual Bayes error-rate (red), and contributions, is shown by plain lines. The minimum Bayes error-rate (thick red), and contributions, is shown by dashed lines. \n",
    "\n",
    "BOSARIS can plot contributions of the misses and false alarms to both the minimum Bayes error-rate (respectively dashed cyan and dashed pink) and actual Bayes error-rate (respectively plain blue and plain greend). \n",
    "\n",
    "The operating points `logit(P_tar)` are shown on the plots by the vertical dashed magenta line at `logit(0.005)` and logit`(0.01)`. `dev FA DR30` refers to the point to the left of which there are fewer than 30 false-alarms. DR30 refer to Doddington’s Rule of 30. This rule suggests you need at least 30 false-alarms and at least 30 misses for meaningful evaluation. The toolkit can plot both the DR30 point for the misses (to the right of which the absolute number of misses drops below 30) and the one for the false alarms (to the left of which the absolute number of false-alarms drops below 30). These points are on the Emin curve, because we use the false-alarm count and miss count that result from the evaluator’s optimized threshold.\n",
    "\n",
    "*Bad Calibration*\n",
    "\n",
    "<img src=\"imgs/bad-dcf-curve.jpg\" alt=\"IMAGE ALT TEXT HERE\" align=\"middle\" width=\"500\" border=\"1\" /></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unittest import TestCase\n",
    "import os\n",
    "import sys\n",
    "lib_path = os.path.join(os.path.dirname('.'), \"pybosaris-master\")\n",
    "sys.path.append(lib_path)\n",
    "from pybosaris.calibration.linear_fuser import LinearFuser\n",
    "from pybosaris.calibration.objectives import evaluate_objective\n",
    "from pybosaris.calibration.training import train_binary_classifier\n",
    "from pybosaris.libperformance import cllr, min_cllr\n",
    "from operator import itemgetter\n",
    "from sklearn.metrics import roc_curve\n",
    "from scipy.optimize import brentq\n",
    "from scipy.interpolate import interp1d\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['figure.dpi']= 300\n",
    "%matplotlib inline\n",
    "# DATA_PATH_v1 = '/Volumes/dfs/gen/Misc/data19/patx/alamja/sre19works/fusion_calibration/all_scores_with_keys'\n",
    "# # DATA_PATH = '/Volumes/dfs/gen/Misc/data19/patx/monteijo/scores_sre2019/cosine/aspp_pre'\n",
    "# DATA_PATH = '/Volumes/dfs/gen/Misc/scratch05/patx/alamja/sre19works/fusion_stuffs/final_to_olda_6oct2019/v2mfcc_coral/lda200_sre1216_adapted_sre19trn_sre19_unlab'\n",
    "# DATA_PATH_2 = '/Volumes/dfs/gen/Misc/scratch05/patx/alamja/sre19works/fusion_stuffs/final_to_olda_6oct2019/v2sre04to18_train_sre19trn/lda200_train_combined_200k_sre19trn'\n",
    "KEY_DIR = '/Volumes/dfs/gen/Misc/scratch05/patx/alamja/sre19works/fusion_stuffs/scripts_for_calibration_fusion/keys'\n",
    "DATA_PATH = '/Volumes/dfs/gen/Misc/scratch05/patx/alamja/sre19works/fusion_stuffs/scripts_for_calibration_fusion/systems'\n",
    "SUBSYS = 'CRIM_30SEP_V2MFCC_LDA250_SRE1618'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preprocess training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(349685, 4)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>test</th>\n",
       "      <th>score</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1131_sre18</td>\n",
       "      <td>aaeeiknb_sre18</td>\n",
       "      <td>-25.22875</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1131_sre18</td>\n",
       "      <td>aaqclfew_sre18</td>\n",
       "      <td>-14.63305</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1131_sre18</td>\n",
       "      <td>abmnegny_sre18</td>\n",
       "      <td>-28.95438</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1131_sre18</td>\n",
       "      <td>aboerwai_sre18</td>\n",
       "      <td>-14.89049</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1131_sre18</td>\n",
       "      <td>abwyevvm_sre18</td>\n",
       "      <td>-37.73000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      speaker            test     score  label\n",
       "0  1131_sre18  aaeeiknb_sre18 -25.22875      0\n",
       "1  1131_sre18  aaqclfew_sre18 -14.63305      0\n",
       "2  1131_sre18  abmnegny_sre18 -28.95438      0\n",
       "3  1131_sre18  aboerwai_sre18 -14.89049      0\n",
       "4  1131_sre18  abwyevvm_sre18 -37.73000      0"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load initial scores with labels\n",
    "dev_scores_labels = pd.read_csv(os.path.join(KEY_DIR, 'sre18_cmn2_evl_tst'), sep=' ', header=None)\n",
    "dev_scores_labels.columns = ['speaker', 'test', 'label']\n",
    "dev_scores_labels = dev_scores_labels.replace({'imp': 0, 'tgt': 1})\n",
    "\n",
    "# Load actual scores without labels\n",
    "dev_scores = pd.read_csv(os.path.join(DATA_PATH, SUBSYS, 'sre18_cmn2_evl_tst.txt'), sep=' ', header=None)\n",
    "dev_scores.columns = ['speaker', 'test', 'score']\n",
    "\n",
    "# Labels are missing so we will join the labels from the initial scores\n",
    "scores = dev_scores.merge(dev_scores_labels[['speaker', 'test', 'label']], on=['speaker', 'test'])\n",
    "scores.shape\n",
    "scores.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data in target and non-target trials\n",
    "tar = np.array(scores[scores.label == 1]['score'])\n",
    "non = np.array(scores[scores.label == 0]['score'])\n",
    "\n",
    "# Instantiate a LinearFuser\n",
    "# Function handle for function that must be trained\n",
    "# Get a starting point for the calibration weights: 'w0',\n",
    "# which are zeros by default\n",
    "train_scores = np.hstack((tar, non))\n",
    "fuser = LinearFuser(scores=train_scores)\n",
    "\n",
    "# Create label vector\n",
    "# Let the trainer know which scores are target scores and which are non-target scores. \n",
    "ntar = tar.shape[0]\n",
    "nnon = non.shape[0]\n",
    "classf = np.hstack((np.ones(ntar), -np.ones(nnon)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Train linear classifier\n",
    "\n",
    "Do the training to get the calibration weights 'w'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-10-11 13:01:45.379 INFO     TR 0 (initial state): obj = 1, Delta = 10.1719\n",
      "2019-10-11 13:01:45.505 INFO     CG 0: curv=27345.8, converged inside trust region; radius = 0.00378367, residual=0.212558, model=0.195743\n",
      "2019-10-11 13:01:45.529 INFO     TR 1: obj=0.720254; rho=1.42914\n",
      "2019-10-11 13:01:45.635 INFO     CG 0: curv=1526.5, converged inside trust region; radius = 0.0115509, residual=0.41897, model=0.331304\n",
      "2019-10-11 13:01:45.657 INFO     TR 2: obj=0.299232; rho=1.2708\n",
      "2019-10-11 13:01:45.772 INFO     CG 0: curv=73.9269, converged inside trust region; radius = 0.00797785, residual=0.224558, model=0.0737261\n",
      "2019-10-11 13:01:45.794 INFO     TR 3: obj=0.207434; rho=1.24513\n",
      "2019-10-11 13:01:45.899 INFO     CG 0: curv=3.52994, converged inside trust region; radius = 0.00681839, residual=0.387285, model=0.0216939\n",
      "2019-10-11 13:01:45.917 INFO     TR 4: obj=0.181552; rho=1.19305\n",
      "2019-10-11 13:01:46.022 INFO     CG 0: curv=0.136314, radius = 0.00349469, residual=1.12016, model=0.00300768\n",
      "2019-10-11 13:01:46.107 INFO     CG 1: curv=2.63966E-06, converged inside trust region; radius = 0.255653, residual=1.11705e-12, model=0.0274613\n",
      "2019-10-11 13:01:46.128 INFO     contracting: Delta=2.54298\n",
      "2019-10-11 13:01:46.128 INFO     TR 5: obj=0.181552; backtracking; rho=-0.849028\n",
      "2019-10-11 13:01:46.129 INFO     CG 0: curv=-1.53222e-14, jump to trust region boundary\n",
      "2019-10-11 13:01:46.153 INFO     contracting: Delta=0.635744\n",
      "2019-10-11 13:01:46.154 INFO     TR 5: obj=0.181552; backtracking; rho=-1.71407e-11\n",
      "2019-10-11 13:01:46.255 INFO     CG 0: curv=1.72231E-27, converged inside trust region; radius = 6.16848e-15, residual=0.00339593, model=3.70641e-29\n",
      "2019-10-11 13:01:46.276 INFO     contracting: Delta=0.158936\n",
      "2019-10-11 13:01:46.277 INFO     TR 5: obj=0.181552; backtracking; rho=-1.79725e+13\n",
      "2019-10-11 13:01:46.372 INFO     CG 0: curv=2.98993e-37, radius = 5.56626e-15, residual=2.25593, model=2.83947e-33\n",
      "2019-10-11 13:01:46.463 INFO     CG 1: curv=1.01075E-35, converged inside trust region; radius = 5.56669e-15, residual=4.6578e-16, model=2.83969e-33\n",
      "2019-10-11 13:01:46.489 INFO     TR 5: converged with minimal model change\n",
      "2019-10-11 13:01:46.492 INFO     cxe = 0.181552, pen = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scaling: 0.3061653635786684, Offset: 0.0114315348042734\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.181552218067861,\n",
       " 0,\n",
       " {'Delta': 0.15893596319922618,\n",
       "  'y': 0.181552218067861,\n",
       "  'g': array([-1.92592994e-34, -2.28704181e-34]),\n",
       "  'hess': functools.partial(<bound method ReplaceHessian.hessian of <pybosaris.calibration.objectives.ReplaceHessian object at 0x11e23c978>>, dy=1, w=array([0.30616536, 0.01143153]))},\n",
       " True)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Taken from `/misc/scratch05/patx/alamja/sre19works/fusion_stuffs/scripts_for_calibration_fusion/fusion_sre19_cmn2_CRIM_v0.m`\n",
    "prior = 0.005\n",
    "maxiters = 25\n",
    "quiet = False\n",
    "objfun = None\n",
    "w0 = fuser.w\n",
    "w, train_cxe, w_pen, optimizerState, converged = train_binary_classifier(\n",
    "    classifier=fuser, classf=classf, w0=w0,\n",
    "    objective_function=objfun, prior=prior,\n",
    "    penalizer=None, penalizer_weight=0,\n",
    "    maxiters=maxiters, maxCG=100, optimizerState=None, quiet=quiet, cstepHessian=True)\n",
    "print(f'Scaling: {w[0]}, Offset: {w[1]}')\n",
    "train_cxe, w_pen, optimizerState, converged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Calibrate input scores\n",
    "\n",
    "Create a function handle that will calibrate input scores using the trained weights: 'w'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-10-11 13:01:52.545 INFO     train Cxe = 0.181552218067861, cllr: 0.17854079981480248, min: 0.12304418931467008\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pybosaris.calibration.linear_fuser.LinearFuser at 0x121cdb5f8>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "array([ 12.74475409,   1.75607016,   4.80392676, ...,  -6.83571801,\n",
       "       -10.0726605 ,  -6.42719851])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.17854079981480248"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.12304418931467008"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_fuser = LinearFuser(scores=train_scores, w=w)\n",
    "train_fused_scores = train_fuser.fusion()\n",
    "train_c = cllr(train_fused_scores[:ntar], train_fused_scores[ntar:])\n",
    "train_c_min = min_cllr(train_fused_scores[:ntar], train_fused_scores[ntar:])\n",
    "logging.info('train Cxe = {}, cllr: {}, min: {}'.format(train_cxe, train_c, train_c_min))\n",
    "train_fuser\n",
    "train_fused_scores\n",
    "train_c\n",
    "train_c_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_eer(y, y_score, pos):\n",
    "# y denotes groundtruth scores,\n",
    "# y_score denotes the prediction scores.\n",
    "\n",
    "    fpr, tpr, thresholds = roc_curve(y, y_score, pos_label=pos)\n",
    "    eer = brentq(lambda x : 1. - x - interp1d(fpr, tpr)(x), 0., 1.)\n",
    "    thresh = interp1d(fpr, thresholds)(eer)\n",
    "\n",
    "    return eer, thresh, fpr, tpr\n",
    "\n",
    "# Creates a list of false-negative rates, a list of false-positive rates\n",
    "# and a list of decision thresholds that give those error-rates.\n",
    "def ComputeErrorRates(scores, labels):\n",
    "\n",
    "    # Sort the scores from smallest to largest, and also get the corresponding\n",
    "    # indexes of the sorted scores.  We will treat the sorted scores as the\n",
    "    # thresholds at which the the error-rates are evaluated.\n",
    "    sorted_indexes, thresholds = zip(*sorted(\n",
    "        [(index, threshold) for index, threshold in enumerate(scores)],\n",
    "        key=itemgetter(1)))\n",
    "    sorted_labels = []\n",
    "    labels = [labels[i] for i in sorted_indexes]\n",
    "    fnrs = []\n",
    "    fprs = []\n",
    "\n",
    "    # At the end of this loop, fnrs[i] is the number of errors made by\n",
    "    # incorrectly rejecting scores less than thresholds[i]. And, fprs[i]\n",
    "    # is the total number of times that we have correctly accepted scores\n",
    "    # greater than thresholds[i].\n",
    "    for i in range(0, len(labels)):\n",
    "        if i == 0:\n",
    "            fnrs.append(labels[i])\n",
    "            fprs.append(1 - labels[i])\n",
    "        else:\n",
    "            fnrs.append(fnrs[i-1] + labels[i])\n",
    "            fprs.append(fprs[i-1] + 1 - labels[i])\n",
    "    fnrs_norm = sum(labels)\n",
    "    fprs_norm = len(labels) - fnrs_norm\n",
    "\n",
    "    # Now divide by the total number of false negative errors to\n",
    "    # obtain the false positive rates across all thresholds\n",
    "    fnrs = [x / float(fnrs_norm) for x in fnrs]\n",
    "\n",
    "    # Divide by the total number of corret positives to get the\n",
    "    # true positive rate.  Subtract these quantities from 1 to\n",
    "    # get the false positive rates.\n",
    "    fprs = [1 - x / float(fprs_norm) for x in fprs]\n",
    "    return fnrs, fprs, thresholds\n",
    "\n",
    "# Computes the minimum of the detection cost function.  The comments refer to\n",
    "# equations in Section 3 of the NIST 2016 Speaker Recognition Evaluation Plan.\n",
    "def ComputeMinDcf(fnrs, fprs, thresholds, p_target, c_miss, c_fa):\n",
    "    min_c_det = float(\"inf\")\n",
    "    min_c_det_threshold = thresholds[0]\n",
    "    for i in range(0, len(fnrs)):\n",
    "        # See Equation (2).  it is a weighted sum of false negative\n",
    "        # and false positive errors.\n",
    "        c_det = c_miss * fnrs[i] * p_target + c_fa * fprs[i] * (1 - p_target)\n",
    "        if c_det < min_c_det:\n",
    "            min_c_det = c_det\n",
    "            min_c_det_threshold = thresholds[i]\n",
    "    # See Equations (3) and (4).  Now we normalize the cost.\n",
    "    c_def = min(c_miss * p_target, c_fa * (1 - p_target))\n",
    "    min_dcf = min_c_det / c_def\n",
    "    return min_dcf, min_c_det_threshold\n",
    "\n",
    "def compute_actual_cost(scores, labels, p_target, c_miss=1, c_fa=1):\n",
    "    beta = c_fa * (1 - p_target) / (c_miss * p_target)\n",
    "    decisions = (scores >= np.log(beta)).astype('i')\n",
    "    num_targets = np.sum(labels)\n",
    "    fp = np.sum(decisions * (1 - labels))\n",
    "    num_nontargets = np.sum(1 - labels)\n",
    "    fn = np.sum((1 - decisions) * labels)\n",
    "    fpr = fp / num_nontargets if num_nontargets > 0 else np.nan\n",
    "    fnr = fn / num_targets if num_targets > 0 else np.nan\n",
    "    print(\"act_C : {0:.4f}, at threshold {1:.4f} (p-target={2}, c-miss={3}, \"\n",
    "    \"c-fa={4})\".format(fnr + beta * fpr, np.log(beta), p_target, c_miss, c_fa))\n",
    "    return fnr + beta * fpr, np.log(beta), fpr, fnr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "lib_path = '/Volumes/dfs/gen/Misc/data19/projets/multi/sre19_multimedia_asv/multimedia_scoring_software'\n",
    "sys.path.append(lib_path)\n",
    "import sre_scorer as sc\n",
    "import scoring_utils as st\n",
    "\n",
    "def compute_equalized_act_cost(scores, tar_nontar_labs,\n",
    "                               p_target, c_miss=1, c_fa=1):\n",
    "    act_c = 0.\n",
    "    for p_t in p_target:\n",
    "        beta = c_fa * (1 - p_t) / (c_miss * p_t)\n",
    "#        act_c_norm = np.zeros(len(partition_masks))\n",
    "        fpr, fnr = np.zeros((2, 1))\n",
    "        _, fpr, fnr = sc.compute_actual_cost(scores,\n",
    "                                             tar_nontar_labs, p_t,\n",
    "                                             c_miss, c_fa)\n",
    "#        act_c += act_c_norm.mean()\n",
    "        fnr_avg = 0. if np.all(np.isnan(fnr)) else np.nanmean(fnr)\n",
    "        fpr_avg = 0. if np.all(np.isnan(fpr)) else np.nanmean(fpr)\n",
    "        act_c += fnr_avg + beta * fpr_avg\n",
    "    return act_c / len(p_target)\n",
    "\n",
    "def compute_act_cost(scores, tar_nontar_labs, p_target,\n",
    "                                 c_miss=1, c_fa=1):\n",
    "    act_c = 0.\n",
    "    for p_t in p_target:\n",
    "        act_c += sc.compute_actual_cost(scores, tar_nontar_labs, p_t,\n",
    "                                        c_miss=1, c_fa=1)[0]\n",
    "    return act_c / len(p_target)\n",
    "\n",
    "def compute_min_cost(scores, labels, ptar, weights=None):\n",
    "    fnr, fpr = sc.compute_pmiss_pfa_rbst(scores, labels, weights)\n",
    "    eer = sc.compute_eer(fnr, fpr)\n",
    "    min_c = 0.\n",
    "    for pt in ptar:\n",
    "        min_c += sc.compute_c_norm(fnr, fpr, pt)\n",
    "    return eer, min_c / len(ptar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Calibrated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### vastMinDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vastMinDCF : 0.141142\n",
      "EER : 0.030019%\n"
     ]
    }
   ],
   "source": [
    "# c_miss = 1\n",
    "# c_fa = 1\n",
    "# p_target = 0.05\n",
    "classf[classf == -1] = 0\n",
    "\n",
    "# fnrs, fprs, thresholds = ComputeErrorRates(train_fused_scores, \n",
    "#                                            classf)\n",
    "# mindcf, threshold = ComputeMinDcf(fnrs, fprs, thresholds, p_target, c_miss, c_fa)\n",
    "\n",
    "# print(\"minDCF : {0:.4f}, at threshold {1:.4f} (p-target={2}, c-miss={3}, \"\n",
    "#     \"c-fa={4})\\n\".format(mindcf, threshold, p_target,c_miss, c_fa))\n",
    "\n",
    "vast_eer, vast_min_c = compute_min_cost(train_fused_scores, classf, [0.05])\n",
    "print(\"vastMinDCF : {0:.6f}\".format(vast_min_c))\n",
    "print('EER : %.6f%%'%(vast_eer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### vastActDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vast16ActDCF : 0.156545\n"
     ]
    }
   ],
   "source": [
    "vast_act_c = compute_act_cost(train_fused_scores, classf, [0.05])\n",
    "print(\"vast16ActDCF : {0:.6f}\".format(vast_act_c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### sre16MinDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sre16MinDCF : 0.250086\n",
      "EER : 0.030019%\n"
     ]
    }
   ],
   "source": [
    "eer, min_c = compute_min_cost(train_fused_scores, classf, [0.01, 0.005])\n",
    "print(\"sre16MinDCF : {0:.6f}\".format(min_c))\n",
    "print('EER : %.6f%%'%(eer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### sre16ActDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sre16ActDCF : 0.273762\n"
     ]
    }
   ],
   "source": [
    "# C_norm, thresh, fpr, fnr = compute_actual_cost(train_fused_scores, classf, 0.05)\n",
    "# C_norm1, thresh1, fpr, fnr = compute_actual_cost(train_fused_scores, classf, 0.01)\n",
    "# C_norm2, thresh2, fpr, fnr = compute_actual_cost(train_fused_scores, classf, 0.005)\n",
    "# C_primary = (C_norm1 + C_norm2) / 2\n",
    "# print(\"C_primary : {0:.4f}\\n\".format(C_primary))\n",
    "\n",
    "act_c = compute_act_cost(train_fused_scores, classf, [0.01, 0.005]) # Same as compute_equalized_act_cost\n",
    "act_c = compute_equalized_act_cost(train_fused_scores, classf, [0.01, 0.005])\n",
    "print(\"sre16ActDCF : {0:.6f}\".format(act_c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Uncalibrated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### vastMinDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vastMinDCF : 0.141142\n",
      "EER : 0.030019%\n"
     ]
    }
   ],
   "source": [
    "# c_miss = 1\n",
    "# c_fa = 1\n",
    "# p_target = 0.05\n",
    "classf[classf == -1] = 0\n",
    "\n",
    "# fnrs, fprs, thresholds = ComputeErrorRates(train_scores, \n",
    "#                                            classf)\n",
    "# mindcf, threshold_un = ComputeMinDcf(fnrs, fprs, thresholds, p_target, c_miss, c_fa)\n",
    "\n",
    "# print(\"minDCF : {0:.4f}, at threshold {1:.4f} (p-target={2}, c-miss={3}, \"\n",
    "#     \"c-fa={4})\\n\".format(mindcf, threshold, p_target,c_miss, c_fa))\n",
    "\n",
    "vast_eer, vast_min_c = compute_min_cost(train_scores, classf, [0.05])\n",
    "print(\"vastMinDCF : {0:.6f}\".format(vast_min_c))\n",
    "print('EER : %.6f%%'%(vast_eer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### vastActDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vast16ActDCF : 0.166403\n"
     ]
    }
   ],
   "source": [
    "vast_act_c = compute_act_cost(train_scores, classf, [0.05])\n",
    "print(\"vast16ActDCF : {0:.6f}\".format(vast_act_c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### sre16MinDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sre16MinDCF : 0.250086\n",
      "EER : 0.030019%\n"
     ]
    }
   ],
   "source": [
    "eer, min_c = compute_min_cost(train_scores, classf, [0.01, 0.005])\n",
    "print(\"sre16MinDCF : {0:.6f}\".format(min_c))\n",
    "print('EER : %.6f%%'%(eer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### sre16ActDCF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sre16ActDCF : 0.540947\n"
     ]
    }
   ],
   "source": [
    "# C_norm, thresh, fpr, fnr = compute_actual_cost(train_scores, classf, 0.05)\n",
    "# C_norm1, thresh1_un, fpr, fnr = compute_actual_cost(train_scores, classf, 0.01)\n",
    "# C_norm2, thresh1_un, fpr, fnr = compute_actual_cost(train_scores, classf, 0.005)\n",
    "# C_primary = (C_norm1 + C_norm2) / 2\n",
    "# print(\"C_primary : {0:.4f}\\n\".format(C_primary))\n",
    "\n",
    "act_c = compute_act_cost(train_scores, classf, [0.01, 0.005]) # Same as compute_equalized_act_cost\n",
    "act_c = compute_equalized_act_cost(train_scores, classf, [0.01, 0.005])\n",
    "print(\"sre16ActDCF : {0:.6f}\".format(act_c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14,10),)\n",
    "fig = plt.subplot(211)\n",
    "plt.hist(train_scores[:ntar], bins=200, alpha=0.5,)\n",
    "plt.hist(train_scores[ntar:], bins=200, alpha=0.5,)\n",
    "plt.text(threshold_un+0.25, 35000, 'minDCF', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh1_un+0.25, 30000, 'act_C (p-target=0.01)', c='green', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh2_un+0.25, 25000, 'act_C (p-target=0.005)', c='red', fontsize=14, verticalalignment='top')\n",
    "plt.axvline(x=threshold_un, c='black', linestyle='--', lw=0.75, label='minDCF: '+str(np.round(threshold,4)))\n",
    "plt.axvline(x=thresh1_un, c='green', linestyle='--', lw=0.75, label='act_C (p-target=0.01): '+str(np.round(thresh1,4)))\n",
    "plt.axvline(x=thresh2_un, c='red', linestyle='--', lw=0.75, label='act_C (p-target=0.005): '+str(np.round(thresh2,4)))\n",
    "plt.title('Uncalibrated Scores')\n",
    "plt.grid()\n",
    "# plt.xlim(-15,20);\n",
    "\n",
    "fig = plt.subplot(212)\n",
    "plt.hist(train_fused_scores[:ntar], bins=200, alpha=0.5,)\n",
    "plt.hist(train_fused_scores[ntar:], bins=200, alpha=0.5,)\n",
    "plt.text(threshold+0.25, 35000, 'minDCF', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh1+0.25, 30000, 'act_C (p-target=0.01)', c='green', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh2+0.25, 25000, 'act_C (p-target=0.005)', c='red', fontsize=14, verticalalignment='top')\n",
    "plt.axvline(x=threshold, c='black', linestyle='--', lw=0.75, label='minDCF: '+str(np.round(threshold,4)))\n",
    "plt.axvline(x=thresh1, c='green', linestyle='--', lw=0.75, label='act_C (p-target=0.01): '+str(np.round(thresh1,4)))\n",
    "plt.axvline(x=thresh2, c='red', linestyle='--', lw=0.75, label='act_C (p-target=0.005): '+str(np.round(thresh2,4)))\n",
    "plt.title('Calibrated Scores')\n",
    "plt.grid()\n",
    "plt.tight_layout();\n",
    "# plt.legend()\n",
    "# plt.xlim(-15,20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. System Fusing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Preprocess training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load initial scores with labels\n",
    "dev_scores_labels = pd.read_csv(os.path.join(DATA_PATH_v1, 'js4_scores_sre19_dev_test_cmn2_adapt.txt'), sep=' ', header=None)\n",
    "dev_scores_labels.columns = ['speaker', 'test', 'score', 'label']\n",
    "dev_scores_labels = dev_scores_labels.replace({'nontarget': 0, 'target': 1})\n",
    "\n",
    "# Load actual scores without labels\n",
    "dev_scores = pd.read_csv(os.path.join(DATA_PATH_2, 'scores_sre19_dev_test_cmn2_adapt'), sep=\" \", header=None)\n",
    "dev_scores.columns = ['speaker', 'test', 'score']\n",
    "\n",
    "# Labels are missing so we will join the labels from the initial scores\n",
    "scores = dev_scores.merge(dev_scores_labels[['speaker', 'test', 'label']], on=['speaker', 'test'])\n",
    "scores.shape\n",
    "scores.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data in target and non-target trials\n",
    "tar = np.array(scores[scores.label == 1]['score'])\n",
    "non = np.array(scores[scores.label == 0]['score'])\n",
    "\n",
    "# Instantiate a LinearFuser\n",
    "# Function handle for function that must be trained\n",
    "# Get a starting point for the calibration weights: 'w0',\n",
    "# which are zeros by default\n",
    "train_scores = np.hstack((tar, non))\n",
    "fuser = LinearFuser(scores=train_scores)\n",
    "\n",
    "# Create label vector\n",
    "# Let the trainer know which scores are target scores and which are non-target scores. \n",
    "ntar = tar.shape[0]\n",
    "nnon = non.shape[0]\n",
    "classf = np.hstack((np.ones(ntar), -np.ones(nnon)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Train linear classifier\n",
    "\n",
    "Do the training to get the calibration weights 'w'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Taken from `/misc/scratch05/patx/alamja/sre19works/fusion_stuffs/scripts_for_calibration_fusion/fusion_sre19_cmn2_CRIM_v0.m`\n",
    "prior = 0.005\n",
    "maxiters = 50\n",
    "quiet = False\n",
    "objfun = None\n",
    "w0 = fuser.w\n",
    "w, train_cxe, w_pen, optimizerState, converged = train_binary_classifier(\n",
    "    classifier=fuser, classf=classf, w0=w0,\n",
    "    objective_function=objfun, prior=prior,\n",
    "    penalizer=None, penalizer_weight=0,\n",
    "    maxiters=maxiters, maxCG=100, optimizerState=None, quiet=quiet, cstepHessian=True)\n",
    "print(f'Scaling: {w[0]}, Offset: {w[1]}')\n",
    "train_cxe, w_pen, optimizerState, converged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Calibrate input scores\n",
    "\n",
    "Create a function handle that will calibrate input scores using the trained weights: 'w'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_fuser = LinearFuser(scores=train_scores, w=w)\n",
    "train_fused_scores_2 = train_fuser.fusion()\n",
    "train_c = cllr(train_fused_scores_2[:ntar], train_fused_scores_2[ntar:])\n",
    "train_c_min = min_cllr(train_fused_scores_2[:ntar], train_fused_scores_2[ntar:])\n",
    "logging.info('train Cxe = {}, cllr: {}, min: {}'.format(train_cxe, train_c, train_c_min))\n",
    "train_fuser\n",
    "train_fused_scores_2\n",
    "train_c\n",
    "train_c_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eer, thresh, fpr, tpr = calculate_eer(classf, train_fused_scores_2, pos=1)\n",
    "print('EER : %.2f%%'%(eer*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c_miss = 1\n",
    "c_fa = 1\n",
    "p_target = 0.05\n",
    "classf[classf == -1] = 0\n",
    "\n",
    "fnrs, fprs, thresholds = ComputeErrorRates(train_fused_scores_2, \n",
    "                                           classf)\n",
    "mindcf, threshold = ComputeMinDcf(fnrs, fprs, thresholds, p_target, c_miss, c_fa)\n",
    "\n",
    "print(\"minDCF : {0:.4f}, at threshold {1:.4f} (p-target={2}, c-miss={3}, \"\n",
    "    \"c-fa={4})\".format(mindcf, threshold, p_target,c_miss, c_fa))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C_norm1, thresh1, fpr, fnr = compute_actual_cost(train_fused_scores_2, classf, 0.01)\n",
    "C_norm2, thresh2, fpr, fnr = compute_actual_cost(train_fused_scores_2, classf, 0.005)\n",
    "C_primary = (C_norm1 + C_norm2) / 2\n",
    "print(\"C_primary : {0:.4f}\".format(C_primary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5 Merge scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divide scores of each system to target and non-target\n",
    "tar = train_fused_scores[classf==1]\n",
    "non = train_fused_scores[classf==0]\n",
    "tar2 = train_fused_scores_2[classf==1]\n",
    "non2 = train_fused_scores_2[classf==0]\n",
    "\n",
    "# Merge target and non-target instances of each system\n",
    "# Merge to single scores array\n",
    "tar = np.vstack((tar, tar2))\n",
    "non = np.vstack((non, non2))\n",
    "merged_train_scores = np.vstack((tar.T, non.T)).T\n",
    "\n",
    "# Instantiate Fuser\n",
    "merged_fuser = LinearFuser(scores=merged_train_scores)\n",
    "\n",
    "# Labels\n",
    "ntar = tar.shape[1]\n",
    "nnon = non.shape[1]\n",
    "merged_classf = np.hstack((np.ones(ntar), -np.ones(nnon)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.6 Train Linear Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prior = 0.005\n",
    "maxiters = 50\n",
    "quiet = False\n",
    "objfun = None\n",
    "w0 = merged_fuser.w\n",
    "w, train_cxe, w_pen, optimizerState, converged = train_binary_classifier(\n",
    "    classifier=merged_fuser, classf=merged_classf, w0=w0,\n",
    "    objective_function=objfun, prior=prior,\n",
    "    penalizer=None, penalizer_weight=0,\n",
    "    maxiters=maxiters, maxCG=100, optimizerState=None, quiet=quiet, cstepHessian=True)\n",
    "# print(f'Scaling: {w[0]}, Offset: {w[1]}')\n",
    "train_cxe, w_pen, optimizerState, converged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.7 Calibrate input scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_train_fuser = LinearFuser(scores=merged_train_scores, w=w)\n",
    "merged_train_fused_scores = merged_train_fuser.fusion()\n",
    "train_c = cllr(merged_train_fused_scores[:ntar], merged_train_fused_scores[ntar:])\n",
    "train_c_min = min_cllr(merged_train_fused_scores[:ntar], merged_train_fused_scores[ntar:])\n",
    "logging.info('train Cxe = {}, cllr: {}, min: {}'.format(train_cxe, train_c, train_c_min))\n",
    "merged_train_fuser\n",
    "merged_train_fused_scores\n",
    "train_c\n",
    "train_c_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.8 Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eer, thresh, fpr, tpr = calculate_eer(merged_classf, merged_train_fused_scores, pos=1)\n",
    "print('EER : %.2f%%'%(eer*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c_miss = 1\n",
    "c_fa = 1\n",
    "p_target = 0.05\n",
    "merged_classf[merged_classf == -1] = 0\n",
    "\n",
    "fnrs, fprs, thresholds = ComputeErrorRates(merged_train_fused_scores, \n",
    "                                           merged_classf)\n",
    "mindcf, threshold = ComputeMinDcf(fnrs, fprs, thresholds, p_target, c_miss, c_fa)\n",
    "\n",
    "print(\"minDCF : {0:.4f}, at threshold {1:.4f} (p-target={2}, c-miss={3}, \"\n",
    "    \"c-fa={4})\".format(mindcf, threshold, p_target,c_miss, c_fa))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C_norm1, thresh1, fpr, fnr = compute_actual_cost(merged_train_fused_scores, merged_classf, 0.01)\n",
    "C_norm2, thresh2, fpr, fnr = compute_actual_cost(merged_train_fused_scores, merged_classf, 0.005)\n",
    "C_primary = (C_norm1 + C_norm2) / 2\n",
    "print(\"C_primary : {0:.4f}\".format(C_primary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14,10),)\n",
    "fig = plt.subplot(111)\n",
    "plt.hist(merged_train_fused_scores[:ntar], bins=200, alpha=0.5,)\n",
    "plt.hist(merged_train_fused_scores[ntar:], bins=200, alpha=0.5,)\n",
    "plt.text(threshold+0.25, 35000, 'minDCF', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh1+0.25, 30000, 'act_C (p-target=0.01)', c='green', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh2+0.25, 25000, 'act_C (p-target=0.005)', c='red', fontsize=14, verticalalignment='top')\n",
    "plt.axvline(x=threshold, c='black', linestyle='--', lw=0.75, label='minDCF: '+str(np.round(threshold,4)))\n",
    "plt.axvline(x=thresh1, c='green', linestyle='--', lw=0.75, label='act_C (p-target=0.01): '+str(np.round(thresh1,4)))\n",
    "plt.axvline(x=thresh2, c='red', linestyle='--', lw=0.75, label='act_C (p-target=0.005): '+str(np.round(thresh2,4)))\n",
    "plt.title('Calibrated Scores')\n",
    "plt.grid()\n",
    "plt.tight_layout();\n",
    "# plt.legend()\n",
    "plt.xlim(-15,20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.9 Transform eval set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# eval_scores = pd.read_csv(os.path.join(DATA_PATH, 'js4_scores_sre19_eval_test_cmn2_adapt.txt'), sep=' ', header=None)\n",
    "eval_scores = pd.read_csv(os.path.join(DATA_PATH, 'scores_sre19_eval_test_cmn2'), sep=' ', header=None)\n",
    "eval_scores.columns = ['speaker', 'test', 'score']\n",
    "eval_scores.shape\n",
    "eval_scores.head()\n",
    "\n",
    "eval_scores2 = pd.read_csv(os.path.join(DATA_PATH_2, 'scores_sre19_eval_test_cmn2_adapt'), sep=' ', header=None)\n",
    "eval_scores2.columns = ['speaker', 'test', 'score']\n",
    "eval_scores2.shape\n",
    "eval_scores2.head()\n",
    "\n",
    "merged_eval_scores = np.vstack((eval_scores.score, eval_scores2.score))\n",
    "eval_fuser = LinearFuser(scores=merged_eval_scores, w=w)\n",
    "merged_eval_fused_scores = eval_fuser.fusion()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14,10),)\n",
    "fig = plt.subplot(311)\n",
    "plt.hist(eval_scores['score'], bins=200, alpha=0.5,)\n",
    "plt.text(thresh1_un+0.25, 50000, 'act_C (p-target=0.01)', c='green', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh2_un+0.25, 40000, 'act_C (p-target=0.005)', c='red', fontsize=14, verticalalignment='top')\n",
    "plt.axvline(x=thresh1_un, c='green', linestyle='--', lw=0.75, label='act_C (p-target=0.01): '+str(np.round(thresh1,4)))\n",
    "plt.axvline(x=thresh2_un, c='red', linestyle='--', lw=0.75, label='act_C (p-target=0.005): '+str(np.round(thresh2,4)))\n",
    "plt.title('Uncalibrated Scores: '+DATA_PATH.split('/')[-1])\n",
    "plt.grid()\n",
    "# plt.xlim(-15,20);\n",
    "\n",
    "fig = plt.subplot(312)\n",
    "plt.hist(eval_scores2['score'], bins=200, alpha=0.5,)\n",
    "plt.text(thresh1_un+0.25, 50000, 'act_C (p-target=0.01)', c='green', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh2_un+0.25, 40000, 'act_C (p-target=0.005)', c='red', fontsize=14, verticalalignment='top')\n",
    "plt.axvline(x=thresh1_un, c='green', linestyle='--', lw=0.75, label='act_C (p-target=0.01): '+str(np.round(thresh1,4)))\n",
    "plt.axvline(x=thresh2_un, c='red', linestyle='--', lw=0.75, label='act_C (p-target=0.005): '+str(np.round(thresh2,4)))\n",
    "plt.title('Uncalibrated Scores: '+DATA_PATH_2.split('/')[-1])\n",
    "plt.grid()\n",
    "# plt.xlim(-15,20);\n",
    "\n",
    "fig = plt.subplot(313)\n",
    "plt.hist(merged_train_fused_scores, bins=200, alpha=0.5,)\n",
    "plt.text(thresh1+0.25, 50000, 'act_C (p-target=0.01)', c='green', fontsize=14, verticalalignment='top')\n",
    "plt.text(thresh2+0.25, 40000, 'act_C (p-target=0.005)', c='red', fontsize=14, verticalalignment='top')\n",
    "plt.axvline(x=thresh1, c='green', linestyle='--', lw=0.75, label='act_C (p-target=0.01): '+str(np.round(thresh1,4)))\n",
    "plt.axvline(x=thresh2, c='red', linestyle='--', lw=0.75, label='act_C (p-target=0.005): '+str(np.round(thresh2,4)))\n",
    "plt.title('Calibrated Scores')\n",
    "plt.grid()\n",
    "plt.tight_layout();\n",
    "# plt.legend()\n",
    "# plt.xlim(-15,20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.10 Save Outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_name = DATA_PATH.split('/')[-1]+'+'+DATA_PATH_2.split('/')[-1]\n",
    "eval_df = pd.DataFrame(np.vstack((eval_scores.speaker, eval_scores.test, merged_eval_fused_scores))).T\n",
    "eval_df.to_csv('/Volumes/dfs/gen/Misc/scratch05/tesd/noiseuce/NIST-SRE/'+save_name+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter nbconvert /Users/noiseuce/Documents/NIST-SRE/Calibration-No-Normalization.ipynb --to html --output /Volumes/dfs/gen/Misc/scratch05/tesd/noiseuce/NIST-SRE/$save_name-Calibration_No-Normalization.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/Volumes/dfs/gen/Misc/scratch05/tesd/noiseuce/NIST-SRE/'+save_name+'.txt', mode='w') as f:\n",
    "    f.write('train_cxe: {}\\nw_pen: {}\\noptimizerState: {}\\n'. format(train_cxe, w_pen, optimizerState))\n",
    "    f.write(\"minDCF : {0:.4f}, at threshold {1:.4f} (p-target={2}, c-miss={3}, \"\n",
    "    \"c-fa={4})\\n\".format(mindcf, threshold, p_target,c_miss, c_fa))\n",
    "    f.write('EER : %.2f%%\\n'%(eer*100))\n",
    "    f.write(\"C_primary : {0:.4f}\".format(C_primary))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (nist-sre)",
   "language": "python",
   "name": "nist-sre"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}