Python43-使用scikit-learn进行电影评论情感分类.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[【参考】](http://yphuang.github.io/blog/2016/04/21/Sentiment-Analysis-Using-sklearn/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 加载数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\yanqiang\\\\quincy-python\\\\files\\\\data\\\\python43-data\\\\txt_sentoken\\\\'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_dir_name='C:\\\\Users\\\\yanqiang\\\\quincy-python\\\\files\\\\data\\\\python43-data\\\\txt_sentoken\\\\'\n",
    "dataset_dir_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_files\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import metrics\n",
    "\n",
    "#加载数据集，分割为训练集和测试集\n",
    "movie_reviews=load_files(dataset_dir_name)\n",
    "\n",
    "#分割\n",
    "doc_terms_train,doc_terms_test,doc_class_train,doc_class_test=train_test_split(\n",
    "    movie_reviews.data,movie_reviews.target,test_size=0.2,random_state=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'sklearn.utils.Bunch'> b\"arnold schwarzenegger has been an icon for action enthusiasts , since the late 80's , but lately his films have been very sloppy and the one-liners are getting worse . \\nit's hard seeing arnold as mr . freeze in batman and robin , especially when he says tons of ice jokes , but hey he got 15 million , what's it matter to him ? \\nonce again arnold has signed to do another expensive blockbuster , that can't compare with the likes of the terminator series , true lies and even eraser . \\nin this so called dark thriller , the devil ( gabriel byrne ) has come upon earth , to impregnate a woman ( robin tunney ) which happens every 1000 years , and basically destroy the world , but apparently god has chosen one man , and that one man is jericho cane ( arnold himself ) . \\nwith the help of a trusty sidekick ( kevin pollack ) , they will stop at nothing to let the devil take over the world ! \\nparts of this are actually so absurd , that they would fit right in with dogma . \\nyes , the film is that weak , but it's better than the other blockbuster right now ( sleepy hollow ) , but it makes the world is not enough look like a 4 star film . \\nanyway , this definitely doesn't seem like an arnold movie . \\nit just wasn't the type of film you can see him doing . \\nsure he gave us a few chuckles with his well known one-liners , but he seemed confused as to where his character and the film was going . \\nit's understandable , especially when the ending had to be changed according to some sources . \\naside form that , he still walked through it , much like he has in the past few films . \\ni'm sorry to say this arnold but maybe these are the end of your action days . \\nspeaking of action , where was it in this film ? \\nthere was hardly any explosions or fights . \\nthe devil made a few places explode , but arnold wasn't kicking some devil butt . \\nthe ending was changed to make it more spiritual , which undoubtedly ruined the film . \\ni was at least hoping for a cool ending if nothing else occurred , but once again i was let down . \\ni also don't know why the film took so long and cost so much . \\nthere was really no super affects at all , unless you consider an invisible devil , who was in it for 5 minutes tops , worth the overpriced budget . \\nthe budget should have gone into a better script , where at least audiences could be somewhat entertained instead of facing boredom . \\nit's pitiful to see how scripts like these get bought and made into a movie . \\ndo they even read these things anymore ? \\nit sure doesn't seem like it . \\nthankfully gabriel's performance gave some light to this poor film . \\nwhen he walks down the street searching for robin tunney , you can't help but feel that he looked like a devil . \\nthe guy is creepy looking anyway ! \\nwhen it's all over , you're just glad it's the end of the movie . \\ndon't bother to see this , if you're expecting a solid action flick , because it's neither solid nor does it have action . \\nit's just another movie that we are suckered in to seeing , due to a strategic marketing campaign . \\nsave your money and see the world is not enough for an entertaining experience . \\n\"\n",
      "['neg', 'pos']\n"
     ]
    }
   ],
   "source": [
    "print(type(movie_reviews),movie_reviews.data[0])\n",
    "print(movie_reviews.target_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1600, 400, ['neg', 'pos'])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(doc_class_train),len(doc_class_test),(movie_reviews.target_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 创建vectorizer/classifier pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline=Pipeline([\n",
    "    ('vect',TfidfVectorizer(min_df=3,max_df=0.95)),\n",
    "    ('clf',LinearSVC(C=1000))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 优化参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mean: 0.81625, std: 0.01946, params: {'vect__ngram_range': (1, 1)}, mean: 0.85062, std: 0.00766, params: {'vect__ngram_range': (1, 2)}]\n"
     ]
    }
   ],
   "source": [
    "parameters={\n",
    "    'vect__ngram_range':[(1,1),(1,2)]\n",
    "}\n",
    "grid_search=GridSearchCV(pipeline,parameters,n_jobs=-1)\n",
    "grid_search.fit(doc_terms_train,doc_class_train)\n",
    "print(grid_search.grid_scores_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        neg       0.86      0.87      0.87       191\n",
      "        pos       0.88      0.87      0.88       209\n",
      "\n",
      "avg / total       0.87      0.87      0.87       400\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 模型预测结果苹果\n",
    "y_predicted=grid_search.predict(doc_terms_test)\n",
    "print(metrics.classification_report(doc_class_test,y_predicted,target_names=movie_reviews.target_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[167  24]\n",
      " [ 27 182]]\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "confusion_matrix=metrics.confusion_matrix(doc_class_test,y_predicted)\n",
    "print(confusion_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}