homework/Day7_homework_machinelearning_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业1\n",
    "\n",
    "**The future of employment** \n",
    "How susceptible are jobs to computerisation?\n",
    "\n",
    "We examine how susceptible jobs are to computerisation. To assess this, we begin by implementing a novel methodology to estimate the probability of computerisation for 702 detailed occupations, using a Gaussian process classifier. Based on these estimates, we examine expected impacts of future computerisation on **US labour market** outcomes, with the primary objective of analysing the number of jobs at risk and the relationship between an occupations probability of computerisation, wages and educational attainment.\n",
    "\n",
    "- C. Frey, M. Osborne The future of employment: How susceptible are jobs to computerisation? Technological Forecasting & Social Change 114 (2017) 254–280\n",
    "\n",
    "First, together with a group of ML researchers, we subjectively hand-labelled 70 occupations, assigning 1 if automatable, and 0 if not.\n",
    "\n",
    "Second, we use objective **O*NET** variables corresponding to the defined bottlenecks to computerisation. We are interested in variables describing the level of perception and manipulation, creativity, and social intelligence required to perform it. We identified **nine variables** of O*NET that describe these attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:09:30.125247Z",
     "start_time": "2022-07-21T06:09:27.389545Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pylab as plt\n",
    "import seaborn as sns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:09:31.123531Z",
     "start_time": "2022-07-21T06:09:31.083366Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>soc</th>\n",
       "      <th>Element Name</th>\n",
       "      <th>id</th>\n",
       "      <th>label</th>\n",
       "      <th>Data Value</th>\n",
       "      <th>computerization</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Assisting and Caring for Others</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>2.205</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Cramped Work Space, Awkward Positions</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1.415</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Fine Arts</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Finger Dexterity</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>2.000</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Manual Dexterity</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0      soc                           Element Name  id  label  \\\n",
       "0           0  11-1011        Assisting and Caring for Others  70      0   \n",
       "1           1  11-1011  Cramped Work Space, Awkward Positions  70      0   \n",
       "2           2  11-1011                              Fine Arts  70      0   \n",
       "3           3  11-1011                       Finger Dexterity  70      0   \n",
       "4           4  11-1011                       Manual Dexterity  70      0   \n",
       "\n",
       "   Data Value  computerization  \n",
       "0       2.205            0.015  \n",
       "1       1.415            0.015  \n",
       "2       0.915            0.015  \n",
       "3       2.000            0.015  \n",
       "4       0.000            0.015  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://github.com/SocratesAcademy/ccbook/tree/master/data/jobdata.csv\n",
    "df = pd.read_csv('jobdata.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:09:36.046218Z",
     "start_time": "2022-07-21T06:09:36.035180Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "585"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:09:49.303820Z",
     "start_time": "2022-07-21T06:09:49.288817Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_list=list(df['Data Value'])\n",
    "X=[]\n",
    "for i in range(0,585,9):\n",
    "    list1=data_list[i:i+9]\n",
    "    X.append(list1)\n",
    "X=np.array(X)\n",
    "\n",
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:10:08.910009Z",
     "start_time": "2022-07-21T06:10:08.900878Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.205, 1.415, 0.915, 2.   , 0.   , 3.935, 4.125, 4.44 , 3.935])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:10:15.407155Z",
     "start_time": "2022-07-21T06:10:15.390558Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0],\n",
       "       [0],\n",
       "       [0]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_list1=list(df['label'])\n",
    "Y=[]\n",
    "for i in range(0,585,9):\n",
    "    list1=data_list1[i]\n",
    "    Y.append(list1)\n",
    "Y=np.array(Y)\n",
    "Y=Y[:,np.newaxis]\n",
    "Y[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:10:37.404452Z",
     "start_time": "2022-07-21T06:10:36.623030Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn.metrics import mean_squared_error, r2_score, explained_variance_score, roc_auc_score, accuracy_score\n",
    "\n",
    "X1, X2, y1, y2 = train_test_split(X, Y, random_state=0,\n",
    "                                  train_size=0.6, test_size = 0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-07-21T06:10:41.952031Z",
     "start_time": "2022-07-21T06:10:41.929435Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.8846153846153846, 0.8846153846153847)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "bayes = GaussianNB()\n",
    "bayes.fit(X1, y1.flatten())\n",
    "y2_model = bayes.predict(X2)  \n",
    "accuracy_score(y2, y2_model), roc_auc_score(y2, y2_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:33:42.887787Z",
     "start_time": "2020-08-12T05:33:42.365540Z"
    }
   },
   "source": [
    "**任务**: 使用RandomForestClassifier训练并计算accuracy_score和roc_auc_score \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**任务**: 使用SVC算法训练并计算accuracy_score和roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:43:07.191849Z",
     "start_time": "2020-08-12T05:43:07.151756Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score =  0.8846153846153846\n",
      "ROC_AUC score = 0.8846153846153846\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:34:43.576060Z",
     "start_time": "2020-08-12T05:34:29.386789Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9142857142857143"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "def cross_validation(model):\n",
    "    roc_auc= cross_val_score(model, X, Y.flatten(), scoring=\"roc_auc\", cv = 5)\n",
    "    return roc_auc\n",
    "\n",
    "svc_linear = SVC(kernel='linear', C=1E10)\n",
    "np.mean(cross_validation(svc_linear))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**任务**： 换用rbf的kernel来做交叉验证\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:34:45.832881Z",
     "start_time": "2020-08-12T05:34:45.816294Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9142857142857143"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  GPy \n",
    "The Gaussian processes framework in Python. https://github.com/SheffieldML/GPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T04:47:27.100091Z",
     "start_time": "2020-08-12T04:47:21.498418Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting GPy\n",
      "  Using cached GPy-1.9.9-cp37-cp37m-macosx_10_9_x86_64.whl (1.5 MB)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.7 in /opt/anaconda3/lib/python3.7/site-packages (from GPy) (1.18.1)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.16 in /opt/anaconda3/lib/python3.7/site-packages (from GPy) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: six in /opt/anaconda3/lib/python3.7/site-packages (from GPy) (1.14.0)\n",
      "Processing /Users/datalab/Library/Caches/pip/wheels/c8/4a/0e/6e0dc85541825f991c431619e25b870d4b812c911214690cf8/paramz-0.9.5-cp37-none-any.whl\n",
      "Requirement already satisfied, skipping upgrade: decorator>=4.0.10 in /opt/anaconda3/lib/python3.7/site-packages (from paramz>=0.9.0->GPy) (4.4.1)\n",
      "Installing collected packages: paramz, GPy\n",
      "Successfully installed GPy-1.9.9 paramz-0.9.5\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade GPy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:00:55.813586Z",
     "start_time": "2020-08-12T05:00:55.770938Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Name : GP regression\n",
      "Objective : 28.130643010540453\n",
      "Number of Parameters : 3\n",
      "Number of Optimization Parameters : 3\n",
      "Updates : True\n",
      "Parameters:\n",
      "  \u001b[1mGP_regression.         \u001b[0;0m  |               value  |  constraints  |  priors\n",
      "  \u001b[1mrbf.variance           \u001b[0;0m  |  0.3113242734729479  |      +ve      |        \n",
      "  \u001b[1mrbf.lengthscale        \u001b[0;0m  |   3.933616340596464  |      +ve      |        \n",
      "  \u001b[1mGaussian_noise.variance\u001b[0;0m  |  0.0964434513555219  |      +ve      |        \n"
     ]
    }
   ],
   "source": [
    "import GPy\n",
    "\n",
    "kernel = GPy.kern.RBF(input_dim=9, variance=1., lengthscale=1.)\n",
    "m = GPy.models.GPRegression(X,Y,kernel)\n",
    "m.optimize(messages=False)\n",
    "\n",
    "print(m)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T04:59:44.355794Z",
     "start_time": "2020-08-12T04:59:44.311581Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score =  0.9230769230769231\n",
      "ROC_AUC score = 0.9230769230769231\n"
     ]
    }
   ],
   "source": [
    "X1, X2, y1, y2 = train_test_split(X, Y, random_state=0,\n",
    "                                  train_size=0.6, test_size = 0.4)\n",
    "m = GPy.models.GPRegression(X1,y1,kernel)#, normalizer = True)\n",
    "m.optimize(messages=False)\n",
    "y2_model = m.predict(X2)[0]\n",
    "y2_hat = [1 if i > 0.5 else 0  for i in y2_model ]\n",
    "print('Accuracy score = ', accuracy_score(y2, y2_hat))\n",
    "print('ROC_AUC score =', roc_auc_score(y2, y2_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业2\n",
    "\n",
    "\n",
    "**Fake News Detection**\n",
    "Develop a machine learning program to identify when an article might be fake news.\n",
    "\n",
    "- train.csv: A full training dataset with the following attributes:\n",
    "    - id: unique id for a news article\n",
    "    - title: the title of a news article\n",
    "    - author: author of the news article\n",
    "    - text: the text of the article; could be incomplete\n",
    "    - label: a label that marks the article as potentially unreliable\n",
    "        - 1: unreliable\n",
    "        - 0: reliable\n",
    "- test.csv: A testing training dataset with all the same attributes at train.csv without the label.\n",
    "\n",
    "- submit.csv: A sample submission that you can"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:04:38.771789Z",
     "start_time": "2020-08-12T05:04:37.996472Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>author</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>House Dem Aide: We Didn’t Even See Comey’s Let...</td>\n",
       "      <td>Darrell Lucus</td>\n",
       "      <td>House Dem Aide: We Didn’t Even See Comey’s Let...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>FLYNN: Hillary Clinton, Big Woman on Campus - ...</td>\n",
       "      <td>Daniel J. Flynn</td>\n",
       "      <td>Ever get the feeling your life circles the rou...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Why the Truth Might Get You Fired</td>\n",
       "      <td>Consortiumnews.com</td>\n",
       "      <td>Why the Truth Might Get You Fired October 29, ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>15 Civilians Killed In Single US Airstrike Hav...</td>\n",
       "      <td>Jessica Purkiss</td>\n",
       "      <td>Videos 15 Civilians Killed In Single US Airstr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Iranian woman jailed for fictional unpublished...</td>\n",
       "      <td>Howard Portnoy</td>\n",
       "      <td>Print \\nAn Iranian woman has been sentenced to...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                              title              author  \\\n",
       "0   0  House Dem Aide: We Didn’t Even See Comey’s Let...       Darrell Lucus   \n",
       "1   1  FLYNN: Hillary Clinton, Big Woman on Campus - ...     Daniel J. Flynn   \n",
       "2   2                  Why the Truth Might Get You Fired  Consortiumnews.com   \n",
       "3   3  15 Civilians Killed In Single US Airstrike Hav...     Jessica Purkiss   \n",
       "4   4  Iranian woman jailed for fictional unpublished...      Howard Portnoy   \n",
       "\n",
       "                                                text  label  \n",
       "0  House Dem Aide: We Didn’t Even See Comey’s Let...      1  \n",
       "1  Ever get the feeling your life circles the rou...      0  \n",
       "2  Why the Truth Might Get You Fired October 29, ...      1  \n",
       "3  Videos 15 Civilians Killed In Single US Airstr...      1  \n",
       "4  Print \\nAn Iranian woman has been sentenced to...      1  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('./Day7_kaggle_fakenews/train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:23:27.642534Z",
     "start_time": "2020-08-12T05:23:27.637678Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'House Dem Aide: We Didn’t Even See Comey’s Letter Until Jason Chaffetz Tweeted It'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0]['title']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:23:39.321540Z",
     "start_time": "2020-08-12T05:23:39.317362Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20800, 5)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:23:50.588273Z",
     "start_time": "2020-08-12T05:23:50.582536Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0   -- >  House Dem Aide: We Didn’t Even See Comey’s Letter Until Jason Chaffetz Tweeted It\n",
      "1   -- >  FLYNN: Hillary Clinton, Big Woman on Campus - Breitbart\n",
      "2   -- >  Why the Truth Might Get You Fired\n",
      "3   -- >  15 Civilians Killed In Single US Airstrike Have Been Identified\n",
      "4   -- >  Iranian woman jailed for fictional unpublished story about woman stoned to death for adultery\n"
     ]
    }
   ],
   "source": [
    "for k, i in enumerate(df['title'][:5]):\n",
    "    print(k,'  -- > ',  i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:23:59.982799Z",
     "start_time": "2020-08-12T05:23:59.977775Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    0\n",
       "2    1\n",
       "3    1\n",
       "4    1\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['label'][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:24:22.701877Z",
     "start_time": "2020-08-12T05:24:22.697796Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'House Dem Aide: We Didn’t Even See Comey’s Letter Until Jason Chaffetz Tweeted It By Darrell Lucus on October 30, 2016 Subscribe Jason Chaffetz on the stump in American Fork, Utah ( image courtesy Michael Jolley, available under a Creative Commons-BY license) \\nWith apologies to Keith Olbermann, ther'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][0][:300]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:24:52.008417Z",
     "start_time": "2020-08-12T05:24:40.438314Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer, CountVectorizer\n",
    "\n",
    "df=df.fillna(' ')\n",
    "df['total']=df['title']+' '+df['author']+df['text']\n",
    "transformer = TfidfTransformer(smooth_idf=False)\n",
    "count_vectorizer = CountVectorizer(ngram_range=(1,1), \n",
    "                                   stop_words = 'english',\n",
    "                                  max_features = 10000)\n",
    "counts = count_vectorizer.fit_transform(df['total'].values)\n",
    "tfidf = transformer.fit_transform(counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:24:58.286918Z",
     "start_time": "2020-08-12T05:24:58.276451Z"
    }
   },
   "outputs": [],
   "source": [
    "diction  = count_vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:25:13.609124Z",
     "start_time": "2020-08-12T05:25:11.322341Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0., 0., 0., ..., 0., 0., 0.]),\n",
       " array([0., 0., 0., ..., 0., 0., 0.]),\n",
       " array([0., 0., 0., ..., 0., 0., 0.])]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = [i.toarray()[0] for i in tfidf]\n",
    "tfidf[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:25:23.350757Z",
     "start_time": "2020-08-12T05:25:23.347944Z"
    }
   },
   "outputs": [],
   "source": [
    "y = df['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:25:52.882860Z",
     "start_time": "2020-08-12T05:25:52.872091Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(tfidf, y, \n",
    "                                                random_state=1, \n",
    "                                                train_size = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:26:48.241964Z",
     "start_time": "2020-08-12T05:26:48.236400Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9019230769230769"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "model = MultinomialNB() \n",
    "model.fit(Xtrain, ytrain)\n",
    "y_model = model.predict(Xtest)\n",
    "\n",
    "from sklearn.metrics import accuracy_score, roc_auc_score,  roc_curve, auc\n",
    "accuracy_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-12T05:26:58.163874Z",
     "start_time": "2020-08-12T05:26:58.156986Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9027959809796708"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**任务**: 换另外一种算法训练并计算accuracy_score和roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业3 （任选，可不选）\n",
    "\n",
    "### Predicting poverty and wealth from mobile phone metadata\n",
    "\n",
    "All other data and code, including all intermediate data needed to replicate these results and apply these methods in other contexts, are available through the Inter-university Consortium for Political and Social Research (http://doi.org/10.3886/E50592V2). https://www.openicpsr.org/openicpsr/project/100144/version/V5/view\n",
    "\n",
    "数据和代码说明见 https://github.com/SocratesAcademy/css/issues/11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}