Python46-文本数据处理的终极指南(python实现)(一).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ultimate guide to deal with Text Data (using Python) – for Data Scientists & Engineers](https://www.analyticsvidhya.com/blog/2018/02/the-different-methods-deal-text-data-predictive-python/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 简介\n",
    "实现任何程度或者级别的人工智能所必需的最大突破之一就是拥有可以处理文本数据的机器。值得庆幸的是，全世界文本数据的数量在最近几年已经实现指数级增长。这也迫切需要人们从文本数据中挖掘新知识、新观点。从社交媒体分析到风险管理和网络犯罪保护，处理文本数据已经变得前所未有的重要。\n",
    "![](https://i.loli.net/2018/03/09/5aa23d3780765.jpg)\n",
    "在这篇文章中，我们将要讨论不同的特征提取方法，从一些基本技巧逐步深入学习高级自然语言处理技术。我们也将会学习如何预处理文本数据，以便可以从“干净”数据中提取更好的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 目录\n",
    "1. 文本数据的基本体征提取\n",
    "  - 词汇数量\n",
    "  - 字符数量\n",
    "  - 平均字长\n",
    "  - 停用词数量\n",
    "  - 特殊字符数量\n",
    "  - 数字数量\n",
    "  - 大写字母数量\n",
    "2. 文本数据的基本预处理\n",
    "  - 小写转换\n",
    "  - 去除标点符号\n",
    "  - 去除停用词\n",
    "  - 去除频现词\n",
    "  - 去除稀疏词\n",
    "  - 拼写校正\n",
    "  - 分词(tokenization)\n",
    "  - 词干提取(stemming)\n",
    "  - 词形还原(lemmatization)\n",
    "3. 高级文本处理\n",
    "  - N-grams语言模型\n",
    "  - 词频\n",
    "  - 逆文档频率\n",
    "  - TF-IDF\n",
    "  - 词袋\n",
    "  - 情感分析\n",
    "  - 词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. 基本特征提取\n",
    "即使我们对NLP没有充足的知识储备，但是我们可以使用python来提取文本数据的几个基本特征。在开始之前，我们使用pandas将数据集加载进来，以便后面其他任务的使用，数据集是Twitter情感文本数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id  label                                              tweet\n",
      "0   1      0   @user when a father is dysfunctional and is s...\n",
      "1   2      0  @user @user thanks for #lyft credit i can't us...\n",
      "2   3      0                                bihday your majesty\n",
      "3   4      0  #model   i love u take with u all the time in ...\n",
      "4   5      0             factsguide: society now    #motivation\n",
      "5   6      0  [2/2] huge fan fare and big talking before the...\n",
      "6   7      0   @user camping tomorrow @user @user @user @use...\n",
      "7   8      0  the next school year is the year for exams.ð...\n",
      "8   9      0  we won!!! love the land!!! #allin #cavs #champ...\n",
      "9  10      0   @user @user welcome here !  i'm   it's so #gr...\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "train=pd.read_csv(\"files/data/python46-data/train_E6oV3lV.csv\")\n",
    "print(train.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 词汇数量\n",
    "对每一条推文，我们可以提取的最基本特征之一就是词语数量。这样做的初衷就是通常情况下，负面情绪评论含有词语数量比正面情绪评论多。\n",
    "\n",
    "我们可以简单地调用split函数，将句子切分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  word_count\n",
       "0   @user when a father is dysfunctional and is s...          21\n",
       "1  @user @user thanks for #lyft credit i can't us...          22\n",
       "2                                bihday your majesty           5\n",
       "3  #model   i love u take with u all the time in ...          17\n",
       "4             factsguide: society now    #motivation           8"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['word_count']=train['tweet'].apply(lambda x:len(str(x).split(\" \")))\n",
    "train[['tweet','word_count']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 字符数量\n",
    "选择字符数量作为特征的原因和前一个特征一样。在这里，我们直接通过字符串长度计算每条推文字符数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>char_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  char_count\n",
       "0   @user when a father is dysfunctional and is s...         102\n",
       "1  @user @user thanks for #lyft credit i can't us...         122\n",
       "2                                bihday your majesty          21\n",
       "3  #model   i love u take with u all the time in ...          86\n",
       "4             factsguide: society now    #motivation          39"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['char_count']=train['tweet'].str.len()\n",
    "train[['tweet','char_count']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**这里字符串的个数包含了推文中的空格个数，我们根据需要自行去除掉"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 平均词汇长度\n",
    "我们接下来将计算每条推文的平均词汇长度作为另一个特征，这个有可能帮助我们改善模型。将每条推文所有单词的长度然后除以每条推文单词的个数，即可作为平均词汇长度\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>avg_word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>4.555556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>5.315789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>5.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>4.928571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  avg_word\n",
       "0   @user when a father is dysfunctional and is s...  4.555556\n",
       "1  @user @user thanks for #lyft credit i can't us...  5.315789\n",
       "2                                bihday your majesty  5.666667\n",
       "3  #model   i love u take with u all the time in ...  4.928571\n",
       "4             factsguide: society now    #motivation  8.000000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def avg_word(sentence):\n",
    "    words=sentence.split()\n",
    "    return (sum(len(word) for word in words)/len(words))\n",
    "\n",
    "train['avg_word']=train['tweet'].apply(lambda x:avg_word(x))\n",
    "train[['tweet','avg_word']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 停用词的数量\n",
    "通常情况下，在解决NLP问题时，首要任务时去除停用词（stopword）。但是有时计算停用词的数量可以提供我们之前失去的额外信息。下面关于停用词的解释：\n",
    "\n",
    "为节省存储空间和提高搜索效率，搜索引擎在索引页面或处理搜索请求时会自动忽略某些字或词，这些字或词即被称为Stop Words(停用词)。通常意义上，Stop Words大致为如下两类：\n",
    "- 这些词应用十分广泛，在Internet上随处可见，比如“Web”一词几乎在每个网站上均会出现，对这样的词搜索引擎无 法保证能够给出真正相关的搜索结果，难以帮助缩小搜索范围，同时还会降低搜索的效率；\n",
    "- 这类就更多了，包括了语气助词、副词、介词、连接词等，通常自身 并无明确的意义，只有将其放入一个完整的句子中才有一定作用，如常见的“的”、“在”之类。\n",
    "\n",
    "在这里，我们导入NLTK库中的stopwors模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>stopwords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  stopwords\n",
       "0   @user when a father is dysfunctional and is s...         10\n",
       "1  @user @user thanks for #lyft credit i can't us...          4\n",
       "2                                bihday your majesty          1\n",
       "3  #model   i love u take with u all the time in ...          5\n",
       "4             factsguide: society now    #motivation          1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop=stopwords.words('english')\n",
    "train['stopwords']=train['tweet'].apply(lambda sen:len([x for x in sen.split() if x in stop]))\n",
    "train[['tweet','stopwords']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.5 特殊字符的数量\n",
    "一个比较有趣的特征就是我们可以从每个推文中提取“#”和“@”符号的数量。这也有利于我们从文本数据中提取更多信息\n",
    "\n",
    "这里我们使用`startswith`函数来处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>hashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  hashtags\n",
       "0   @user when a father is dysfunctional and is s...         1\n",
       "1  @user @user thanks for #lyft credit i can't us...         3\n",
       "2                                bihday your majesty         0\n",
       "3  #model   i love u take with u all the time in ...         1\n",
       "4             factsguide: society now    #motivation         1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['hashtags']=train['tweet'].apply(lambda sen:len([x for x in sen.split() if x.startswith(\"#\")]))\n",
    "train[['tweet','hashtags']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.6 数字的数量\n",
    "这个特征并不常用，但是在做相似任务时，数字数量是一个比较有用的特征\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>numerics</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  numerics\n",
       "0   @user when a father is dysfunctional and is s...         0\n",
       "1  @user @user thanks for #lyft credit i can't us...         0\n",
       "2                                bihday your majesty         0\n",
       "3  #model   i love u take with u all the time in ...         0\n",
       "4             factsguide: society now    #motivation         0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['numerics']=train['tweet'].apply(lambda sen:len([x for x in sen.split() if x.isdigit()]))\n",
    "train[['tweet','numerics']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.7 大写单词的数量\n",
    "“Anger”或者 “Rage”通常情况下使用大写来表述，所以有必要去识别出这些词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "      <th>upper</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@user when a father is dysfunctional and is s...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@user @user thanks for #lyft credit i can't us...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bihday your majesty</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>#model   i love u take with u all the time in ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>factsguide: society now    #motivation</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               tweet  upper\n",
       "0   @user when a father is dysfunctional and is s...      0\n",
       "1  @user @user thanks for #lyft credit i can't us...      0\n",
       "2                                bihday your majesty      0\n",
       "3  #model   i love u take with u all the time in ...      0\n",
       "4             factsguide: society now    #motivation      0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['upper']=train['tweet'].apply(lambda sen:len([x for x in sen.split() if x.isupper()]))\n",
    "train[['tweet','upper']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 文本数据的预处理\n",
    "到目前为止,我们已经学会了如何从文本数据中提取基本特征。深入文本和特征提取之前,我们的第一步应该是清洗数据,以获得更好的特性。\n",
    "\n",
    "我们将实现这一目标做一些基本的训练数据预处理步骤。\n",
    "## 2.1 小写转化\n",
    "预处理的第一步,我们要做的是把我们的推文变成小写。这避免了拥有相同的多个副本。例如,当我们计算字词汇数量时,“Analytics”和“analytics”将被视为不同的单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    @user when a father is dysfunctional and is so...\n",
       "1    @user @user thanks for #lyft credit i can't us...\n",
       "2                                  bihday your majesty\n",
       "3    #model i love u take with u all the time in ur...\n",
       "4                  factsguide: society now #motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['tweet']=train['tweet'].apply(lambda sen:\" \".join(x.lower() for x in sen.split()))\n",
    "train['tweet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 去除标点符号\n",
    "下一步是去除标点符号,因为它在文本数据中不添加任何额外的信息。因此删除的所有符号将帮助我们减少训练数据的大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    user when a father is dysfunctional and is so ...\n",
       "1    user user thanks for lyft credit i cant use ca...\n",
       "2                                  bihday your majesty\n",
       "3    model i love u take with u all the time in urð...\n",
       "4                    factsguide society now motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['tweet'] = train['tweet'].str.replace('[^\\w\\s]','')\n",
    "train['tweet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "正如你所看到的在上面的输出中,所有的标点符号,包括\"#\"和\"@\"已经从训练数据中去除"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 停用词去除\n",
    "正如我们前面所讨论的,停止词(或常见单词)应该从文本数据中删除。为了这个目的,我们可以创建一个列表stopwords作为自己停用词库或我们可以使用预定义的库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    user father dysfunctional selfish drags kids d...\n",
       "1    user user thanks lyft credit cant use cause do...\n",
       "2                                       bihday majesty\n",
       "3                model love u take u time urð ðððð ððð\n",
       "4                        factsguide society motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop=stopwords.words('english')\n",
    "train['tweet']=train['tweet'].apply(lambda sen:\" \".join(x for x in sen.split() if x not in stop))\n",
    "train['tweet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 常见词去除\n",
    "我们可以把常见的单词从文本数据首先,让我们来检查中最常出现的10个字文本数据然后再调用删除或保留。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user     17473\n",
       "love      2647\n",
       "ð         2511\n",
       "day       2199\n",
       "â         1797\n",
       "happy     1663\n",
       "amp       1582\n",
       "im        1139\n",
       "u         1136\n",
       "time      1110\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq=pd.Series(' '.join(train['tweet']).split()).value_counts()[:10]\n",
    "freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们把这些词去除掉，因为它们对我们文本数据分类没有任何作用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['user', 'love', 'ð', 'day', 'â', 'happy', 'amp', 'im', 'u', 'time']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq=list(freq.index)\n",
    "freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    father dysfunctional selfish drags kids dysfun...\n",
       "1    thanks lyft credit cant use cause dont offer w...\n",
       "2                                       bihday majesty\n",
       "3                              model take urð ðððð ððð\n",
       "4                        factsguide society motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['tweet']=train['tweet'].apply(lambda sen:' '.join(x for x in sen.split() if x not in freq))\n",
    "train['tweet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5 稀缺词去除\n",
    "同样,正如我们删除最常见的话说,这一次让我们从文本中删除很少出现的词。因为它们很稀有,它们之间的联系和其他词主要是噪音。可以替换罕见的单词更一般的形式,然后这将有更高的计数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "thankyoulordâðâ    1\n",
       "hogrider           1\n",
       "cancellations      1\n",
       "foiinãbranlable    1\n",
       "92230              1\n",
       "withxxx            1\n",
       "ludhiana           1\n",
       "croftso            1\n",
       "nothingness        1\n",
       "lifegroup          1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq = pd.Series(' '.join(train['tweet']).split()).value_counts()[-10:]\n",
    "freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    father dysfunctional selfish drags kids dysfun...\n",
       "1    thanks lyft credit cant use cause dont offer w...\n",
       "2                                       bihday majesty\n",
       "3                              model take urð ðððð ððð\n",
       "4                        factsguide society motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq = list(freq.index)\n",
    "train['tweet'] = train['tweet'].apply(lambda x: \" \".join(x for x in x.split() if x not in freq))\n",
    "train['tweet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "所有这些预处理步骤是必不可少的,帮助我们减少我们的词汇噪音,这样最终产生更有效的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.6 拼写校对\n",
    "我们都见过推文存在大量的拼写错误。我们再短时间内匆忙发送tweet,很难发现这些错误。在这方面,拼写校正是一个有用的预处理步骤,因为这也会帮助我们减少单词的多个副本。例如,“Analytics”和“analytcs”将被视为不同的单词,即使它们在同一意义上使用。\n",
    "\n",
    "为实现这一目标,我们将使用textblob库。\n",
    "\n",
    "TextBlob是一个用Python编写的开源的文本处理库。它可以用来执行很多自然语言处理的任务，比如，词性标注，名词性成分提取，情感分析，文本翻译，等等。你可以在[官方文档](http://textblob.readthedocs.io/en/dev/)阅读TextBlog的所有特性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    father dysfunctional selfish drags kiss dysfun...\n",
       "1    thanks left credit can use cause dont offer wh...\n",
       "2                                       midday majesty\n",
       "3                               model take or ðððð ððð\n",
       "4                        factsguide society motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "train['tweet'][:5].apply(lambda x: str(TextBlob(x).correct()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意,它会花费很多时间去做这些修正。因此,为了学习的目的,我只显示这种技术运用在前5行的效果。\n",
    "\n",
    "另外在使用这个技术之前，需要小心一些，因为如果推文中存在大量缩写，比如“your”缩写为“ur”，那么将修正为“or”\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.7 分词\n",
    "分词是指将文本划分为一系列的单词或词语。在我们的示例中,我们使用了textblob库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordList(['thanks', 'lyft', 'credit', 'cant', 'use', 'cause', 'dont', 'offer', 'wheelchair', 'vans', 'pdx', 'disapointed', 'getthanked'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(train['tweet'][1]).words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.8 词干提取\n",
    "词形还原（lemmatization），是把一个任何形式的语言词汇还原为一般形式（能表达完整语义），而词干提取\n",
    "（stemming）是抽取词的词干或词根形式（不一定能够表达完整语义）。词形还原和词干提取是词形规范化的两类重要方式，都能够达到有效归并词形的目的，二者既有联系也有区别。具体介绍请参考[词干提取（stemming）和词形还原（lemmatization）](http://blog.csdn.net/march_on/article/details/8935462)\n",
    "\n",
    "词干提取(stemming)是指通过基于规则的方法去除单词的后缀，比如“ing”,“ly”，“s”等等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        father dysfunct selfish drag kid dysfunct run\n",
       "1    thank lyft credit cant use caus dont offer whe...\n",
       "2                                       bihday majesti\n",
       "3                              model take urð ðððð ððð\n",
       "4                              factsguid societi motiv\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "st=PorterStemmer()\n",
    "train['tweet'][:5].apply(lambda x:\" \".join([st.stem(word) for word in x.split()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上面的输出中，“dysfunctional ”已经变为“dysfunct ”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.9 词性还原\n",
    "词形还原处理后获得的结果是具有一定意义的、完整的词，一般为词典中的有效词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    father dysfunctional selfish drag kid dysfunct...\n",
       "1    thanks lyft credit cant use cause dont offer w...\n",
       "2                                       bihday majesty\n",
       "3                              model take urð ðððð ððð\n",
       "4                        factsguide society motivation\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textblob import Word\n",
    "train['tweet']=train['tweet'].apply(lambda x:\" \".join([Word(word).lemmatize() for word in x.split()]))\n",
    "train['tweet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 高级文本处理\n",
    "到目前为止,我们已经做了所有的可以清洗我们数据的预处理基本步骤。现在,我们可以继续使用NLP技术提取特征。\n",
    "\n",
    "## 3.1 N-grams\n",
    "N-grams称为N元语言模型，是多个词语的组合，是一种统计语言模型，用来根据前(n-1)个item来预测第n个item。常见模型有一元语言模型(unigrams)、二元语言模型（bigrams ）、三元语言模型(trigrams )。\n",
    "\n",
    "Unigrams包含的信息通常情况下比bigrams和trigrams少，需要根据具体应用选择语言模型，因为如果n-grams太短，这时不能捕获重要信息。另一方面，如果n-grams太长，那么捕获的信息基本上是一样的，没有差异性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WordList(['father', 'dysfunctional']),\n",
       " WordList(['dysfunctional', 'selfish']),\n",
       " WordList(['selfish', 'drag']),\n",
       " WordList(['drag', 'kid']),\n",
       " WordList(['kid', 'dysfunction']),\n",
       " WordList(['dysfunction', 'run'])]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TextBlob(train['tweet'][0]).ngrams(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 词频\n",
    "词频(Term frequency)就是一个单词在一个句子出现的次数与这个句子单词个数的比例。\n",
    "\n",
    "** TF = (Number of times term T appears in the particular row) / (number of terms in that row)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>tf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cause</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>credit</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>offer</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>use</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>van</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>thanks</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>dont</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>pdx</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>lyft</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>getthanked</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>wheelchair</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>cant</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>disapointed</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          words  tf\n",
       "0         cause   1\n",
       "1        credit   1\n",
       "2         offer   1\n",
       "3           use   1\n",
       "4           van   1\n",
       "5        thanks   1\n",
       "6          dont   1\n",
       "7           pdx   1\n",
       "8          lyft   1\n",
       "9    getthanked   1\n",
       "10   wheelchair   1\n",
       "11         cant   1\n",
       "12  disapointed   1"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf1 = (train['tweet'][1:2]).apply(lambda x: pd.value_counts(x.split(\" \"))).sum(axis = 0).reset_index()\n",
    "tf1.columns = ['words','tf']\n",
    "tf1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 反转文档频率\n",
    "反转文档频率(Inverse Document Frequency)，简称为IDF，其原理可以简单理解为如果一个单词在所有文档都会出现，那么可能这个单词对我们没有那么重要。\n",
    "\n",
    "一个单词的IDF就是所有行数与出现该单词的行的个数的比例，最后对数。\n",
    "**IDF = log(N/n)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>tf</th>\n",
       "      <th>idf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cause</td>\n",
       "      <td>1</td>\n",
       "      <td>5.690172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>credit</td>\n",
       "      <td>1</td>\n",
       "      <td>7.327781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>offer</td>\n",
       "      <td>1</td>\n",
       "      <td>6.522155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>use</td>\n",
       "      <td>1</td>\n",
       "      <td>3.552287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>van</td>\n",
       "      <td>1</td>\n",
       "      <td>5.236505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>thanks</td>\n",
       "      <td>1</td>\n",
       "      <td>4.597751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>dont</td>\n",
       "      <td>1</td>\n",
       "      <td>3.745585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>pdx</td>\n",
       "      <td>1</td>\n",
       "      <td>8.762865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>lyft</td>\n",
       "      <td>1</td>\n",
       "      <td>8.762865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>getthanked</td>\n",
       "      <td>1</td>\n",
       "      <td>9.679156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>wheelchair</td>\n",
       "      <td>1</td>\n",
       "      <td>9.273691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>cant</td>\n",
       "      <td>1</td>\n",
       "      <td>3.538194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>disapointed</td>\n",
       "      <td>1</td>\n",
       "      <td>10.372303</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          words  tf        idf\n",
       "0         cause   1   5.690172\n",
       "1        credit   1   7.327781\n",
       "2         offer   1   6.522155\n",
       "3           use   1   3.552287\n",
       "4           van   1   5.236505\n",
       "5        thanks   1   4.597751\n",
       "6          dont   1   3.745585\n",
       "7           pdx   1   8.762865\n",
       "8          lyft   1   8.762865\n",
       "9    getthanked   1   9.679156\n",
       "10   wheelchair   1   9.273691\n",
       "11         cant   1   3.538194\n",
       "12  disapointed   1  10.372303"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "for i,word in enumerate(tf1['words']):\n",
    "    tf1.loc[i, 'idf'] =np.log(train.shape[0]/(len(train[train['tweet'].str.contains(word)])))\n",
    "tf1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 词频-反转文档频率(TF-IDF)\n",
    "TF-IDF=TF`*`IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>tf</th>\n",
       "      <th>idf</th>\n",
       "      <th>tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cause</td>\n",
       "      <td>1</td>\n",
       "      <td>5.690172</td>\n",
       "      <td>5.690172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>credit</td>\n",
       "      <td>1</td>\n",
       "      <td>7.327781</td>\n",
       "      <td>7.327781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>offer</td>\n",
       "      <td>1</td>\n",
       "      <td>6.522155</td>\n",
       "      <td>6.522155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>use</td>\n",
       "      <td>1</td>\n",
       "      <td>3.552287</td>\n",
       "      <td>3.552287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>van</td>\n",
       "      <td>1</td>\n",
       "      <td>5.236505</td>\n",
       "      <td>5.236505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>thanks</td>\n",
       "      <td>1</td>\n",
       "      <td>4.597751</td>\n",
       "      <td>4.597751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>dont</td>\n",
       "      <td>1</td>\n",
       "      <td>3.745585</td>\n",
       "      <td>3.745585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>pdx</td>\n",
       "      <td>1</td>\n",
       "      <td>8.762865</td>\n",
       "      <td>8.762865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>lyft</td>\n",
       "      <td>1</td>\n",
       "      <td>8.762865</td>\n",
       "      <td>8.762865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>getthanked</td>\n",
       "      <td>1</td>\n",
       "      <td>9.679156</td>\n",
       "      <td>9.679156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>wheelchair</td>\n",
       "      <td>1</td>\n",
       "      <td>9.273691</td>\n",
       "      <td>9.273691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>cant</td>\n",
       "      <td>1</td>\n",
       "      <td>3.538194</td>\n",
       "      <td>3.538194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>disapointed</td>\n",
       "      <td>1</td>\n",
       "      <td>10.372303</td>\n",
       "      <td>10.372303</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          words  tf        idf      tfidf\n",
       "0         cause   1   5.690172   5.690172\n",
       "1        credit   1   7.327781   7.327781\n",
       "2         offer   1   6.522155   6.522155\n",
       "3           use   1   3.552287   3.552287\n",
       "4           van   1   5.236505   5.236505\n",
       "5        thanks   1   4.597751   4.597751\n",
       "6          dont   1   3.745585   3.745585\n",
       "7           pdx   1   8.762865   8.762865\n",
       "8          lyft   1   8.762865   8.762865\n",
       "9    getthanked   1   9.679156   9.679156\n",
       "10   wheelchair   1   9.273691   9.273691\n",
       "11         cant   1   3.538194   3.538194\n",
       "12  disapointed   1  10.372303  10.372303"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf1['tfidf']=tf1['tf']*tf1['idf']\n",
    "tf1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到，TF-IDF已经“惩罚了”‘don’t’, ‘can’t’, 和‘use’,因为它们是通用词，tf-idf的值都比较低。\n",
    "\n",
    "另外可以通过sksklearn直接计算tf-idf值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<31962x1000 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 114053 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tfidf = TfidfVectorizer(max_features=1000, lowercase=True, analyzer='word',\n",
    " stop_words= 'english',ngram_range=(1,1))\n",
    "train_vect = tfidf.fit_transform(train['tweet'])\n",
    "\n",
    "train_vect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 词袋\n",
    "BOW，就是将文本/Query看作是一系列词的集合。由于词很多，所以咱们就用袋子把它们装起来，简称词袋。至于为什么用袋子而不用筐（basket）或者桶（bucket），这咱就不知道了。举个例子：\n",
    "\n",
    "                   文本1：苏宁易购/是/国内/著名/的/B2C/电商/之一\n",
    "\n",
    "这是一个短文本。“/”作为词与词之间的分割。从中我们可以看到这个文本包含“苏宁易购”，“B2C”，“电商”等词。换句话说，该文本的的词袋由“苏宁易购”，“电商”等词构成。\n",
    "\n",
    "详细请参考[词袋模型和词向量模型](http://blog.csdn.net/wangongxi/article/details/51591031)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<31962x1000 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 128396 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "bow = CountVectorizer(max_features=1000, lowercase=True, ngram_range=(1,1),analyzer = \"word\")\n",
    "train_bow = bow.fit_transform(train['tweet'])\n",
    "train_bow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6 情感分析\n",
    "我们最终需要解决的任务就是如何对推文进行情感分析，在使用ML/DL模型之前，我们可以使用textblob库去进行评测情感"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    (-0.3, 0.5354166666666667)\n",
       "1                    (0.2, 0.2)\n",
       "2                    (0.0, 0.0)\n",
       "3                    (0.0, 0.0)\n",
       "4                    (0.0, 0.0)\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['tweet'][:5].apply(lambda x:TextBlob(x).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用TextBlob情感分析的结果，以元组的方式进行返回，形式如(polarity, subjectivity). 其中polarity的分数是一个范围为 [-1.0 , 1.0 ] 浮点数, 正数表示积极，负数表示消极。subjectivity 是一个 范围为 [0.0 , 1.0 ] 的浮点数，其中 0.0 表示 客观，1.0表示主观的。 \n",
    "\n",
    "下面是一个简单实例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment(polarity=0.39166666666666666, subjectivity=0.4357142857142857)\n"
     ]
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "testimonial = TextBlob(\"Textblob is amazingly simple to use. What great fun!\")\n",
    "print(testimonial.sentiment) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>tweet</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>father dysfunctional selfish drag kid dysfunct...</td>\n",
       "      <td>-0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>thanks lyft credit cant use cause dont offer w...</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>bihday majesty</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>model take urð ðððð ððð</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>factsguide society motivation</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                              tweet  sentiment\n",
       "0   1  father dysfunctional selfish drag kid dysfunct...       -0.3\n",
       "1   2  thanks lyft credit cant use cause dont offer w...        0.2\n",
       "2   3                                     bihday majesty        0.0\n",
       "3   4                            model take urð ðððð ððð        0.0\n",
       "4   5                      factsguide society motivation        0.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['sentiment'] = train['tweet'].apply(lambda x: TextBlob(x).sentiment[0] )\n",
    "train[['id','tweet','sentiment']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.7 词嵌入\n",
    "词嵌入就是文本的向量化表示，潜在思想就是相似单词的向量之间的距离比较短。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "glove_input_file = 'glove.6B.100d.txt'\n",
    "word2vec_output_file = 'glove.6B.100d.txt.word2vec'\n",
    "glove2word2vec(glove_input_file, word2vec_output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 总结\n",
    "通过这篇文章，希望大家对文本数据处理步骤以及特征选择有了大致了解，推荐大家在这些基础之上，使用机器学习或者深度学习方法进行情感预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>label</th>\n",
       "      <th>tweet</th>\n",
       "      <th>word_count</th>\n",
       "      <th>char_count</th>\n",
       "      <th>avg_word</th>\n",
       "      <th>stopwords</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>numerics</th>\n",
       "      <th>upper</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>father dysfunctional selfish drag kid dysfunct...</td>\n",
       "      <td>21</td>\n",
       "      <td>102</td>\n",
       "      <td>4.555556</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>thanks lyft credit cant use cause dont offer w...</td>\n",
       "      <td>22</td>\n",
       "      <td>122</td>\n",
       "      <td>5.315789</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>bihday majesty</td>\n",
       "      <td>5</td>\n",
       "      <td>21</td>\n",
       "      <td>5.666667</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>model take urð ðððð ððð</td>\n",
       "      <td>17</td>\n",
       "      <td>86</td>\n",
       "      <td>4.928571</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>factsguide society motivation</td>\n",
       "      <td>8</td>\n",
       "      <td>39</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  label                                              tweet  word_count  \\\n",
       "0   1      0  father dysfunctional selfish drag kid dysfunct...          21   \n",
       "1   2      0  thanks lyft credit cant use cause dont offer w...          22   \n",
       "2   3      0                                     bihday majesty           5   \n",
       "3   4      0                            model take urð ðððð ððð          17   \n",
       "4   5      0                      factsguide society motivation           8   \n",
       "\n",
       "   char_count  avg_word  stopwords  hashtags  numerics  upper  sentiment  \n",
       "0         102  4.555556         10         1         0      0       -0.3  \n",
       "1         122  5.315789          4         3         0      0        0.2  \n",
       "2          21  5.666667          1         0         0      0        0.0  \n",
       "3          86  4.928571          5         1         0      0        0.0  \n",
       "4          39  8.000000          1         1         0      0        0.0  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 情感分析实例---推特情感分析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 基本方法-TextBlob\n",
    "（数据不进行预处理）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from textblob import TextBlob\n",
    "test=pd.read_csv('files/data/python46-data/test_tweets_anuFYb8.csv')\n",
    "test['label']=test['tweet'].apply(lambda x:1 if TextBlob(x).sentiment[0]<0 else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    0\n",
       "3    0\n",
       "4    0\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['label'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "test[['id','label']].to_csv(\"files/data/python46-data/test_predictions.csv\",index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提交之后的结果为：\t0.1693121693\n",
    "\n",
    "![结果1.png](https://i.loli.net/2018/03/10/5aa36a4761f80.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 数据预处理+TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from textblob import TextBlob,Word\n",
    "from nltk.corpus import stopwords\n",
    "test=pd.read_csv('files/data/python46-data/test_tweets_anuFYb8.csv')\n",
    "# 1.小写转换\n",
    "test['tweet']=test['tweet'].apply(lambda x:\" \".join([word.lower() for word in x.split()]))\n",
    "# 2.去除标点符号\n",
    "test['tweet']=test['tweet'].str.replace('[^\\w\\s]','')\n",
    "# 3.去除停用词\n",
    "stop=stopwords.words('english')\n",
    "test['tweet']=test['tweet'].apply(lambda x:\" \".join(word for word in x.split() if word not in stop))\n",
    "# 4.去除频现词\n",
    "freq=pd.Series(' '.join(test['tweet']).split()).value_counts()[:10]\n",
    "test['tweet']=test['tweet'].apply(lambda x:\" \".join(word for word in x .split() if word not in freq))\n",
    "# 5.去除稀缺词\n",
    "rare=pd.Series(' '.join(test['tweet']).split()).value_counts()[-10:]\n",
    "test['tweet']=test['tweet'].apply(lambda x:\" \".join(word for word in x.split() if word not in rare))\n",
    "# 6.词形还原(lemmatization)\n",
    "test['tweet']=test['tweet'].apply(lambda x:\" \".join([Word(word).lemmatize() for word in x.split()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    studiolife aislife requires passion dedication...\n",
       "1    user white supremacist want everyone see â bir...\n",
       "2     safe way heal acne altwaystoheal healthy healing\n",
       "3    hp cursed child book reservation already yes ð...\n",
       "4    3rd bihday amazing hilarious nephew eli ahmir ...\n",
       "Name: tweet, dtype: object"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['tweet'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>tweet</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>31963</td>\n",
       "      <td>studiolife aislife requires passion dedication...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>31964</td>\n",
       "      <td>user white supremacist want everyone see â bir...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>31965</td>\n",
       "      <td>safe way heal acne altwaystoheal healthy healing</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>31966</td>\n",
       "      <td>hp cursed child book reservation already yes ð...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>31967</td>\n",
       "      <td>3rd bihday amazing hilarious nephew eli ahmir ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>31968</td>\n",
       "      <td>choose momtips</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>31969</td>\n",
       "      <td>something inside dy ððâ eye ness smokeyeyes ti...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>31970</td>\n",
       "      <td>finishedtattooinkedinkloveitâï âïâïâïâï thanks...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>31971</td>\n",
       "      <td>user user user never understand dad left young...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>31972</td>\n",
       "      <td>delicious food lovelife capetown mannaepicure ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                                              tweet  label\n",
       "0  31963  studiolife aislife requires passion dedication...      0\n",
       "1  31964  user white supremacist want everyone see â bir...      0\n",
       "2  31965   safe way heal acne altwaystoheal healthy healing      0\n",
       "3  31966  hp cursed child book reservation already yes ð...      0\n",
       "4  31967  3rd bihday amazing hilarious nephew eli ahmir ...      0\n",
       "5  31968                                     choose momtips      0\n",
       "6  31969  something inside dy ððâ eye ness smokeyeyes ti...      1\n",
       "7  31970  finishedtattooinkedinkloveitâï âïâïâïâï thanks...      0\n",
       "8  31971  user user user never understand dad left young...      0\n",
       "9  31972  delicious food lovelife capetown mannaepicure ...      0"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['label']=test['tweet'].apply(lambda x:1 if TextBlob(x).sentiment[0]<0 else 0)\n",
    "test.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "test[['id','label']].to_csv(\"files/data/python46-data/test_predictions.csv\",index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提交之后的结果为：\t0.1739130435"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}