HW5_ABTesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CTtrdqGpCpIZ"
   },
   "source": [
    "# Homework 5 - A/B Testing\n",
    "\n",
    "This notebook provides some skeleton code to get you started on the homework. Add in your own code and markdown cells to answer the homework questions. If you want to submit the notebook as a PDF, make sure your code and markdowns are clear and concise to make grading easy for the TAs.\n",
    "This notebook can be opened in Colab Open In Colab\n",
    "\n",
    "Before starting, select \"Runtime->Factory reset runtime\" to start with your directories and environment in the base state.\n",
    "\n",
    "If you want to save changes to the notebook, select \"File->Save a copy in Drive\" from the top menu in Colab. This will save the notebook in your Google Drive.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zr4vM1Sd4RQd"
   },
   "source": [
    "# Clones, Installs, and Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xxAnuxWhC8Dx"
   },
   "source": [
    "## Clone GitHub Repository\n",
    "This will clone the repository to your machine. This includes the code and data files. Then change into the directory of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nu5VjT4uC81x"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Yu9ToTexV9fu"
   },
   "source": [
    "## Install Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "STuorg4AV_gh"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bhvV8ImiCpIa"
   },
   "source": [
    "## Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y3C8G8FfCpIa"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H92DV5YyCpIc"
   },
   "source": [
    "# Problem 1. (30 points) Tweet Sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DG9ifhLjCpIc"
   },
   "source": [
    "## 1. (2 points) Load data\n",
    "\n",
    "Load the tweets in the file `\"data/oreo_sentiment.csv\"` into a dataframe. Print out the head of this dataframe so we know you did it right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2bxI5u3vCpId"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w4PeV6XQd2Ki"
   },
   "source": [
    "## 2. (10 points) Cluster tweets\n",
    "\n",
    "Clean the text with the `clean_tweet` function and calculate the tf embedding.  Then calculate the two dimensional UMAP embedding of this embedding.  Finally, use k-means to cluster the UMAP embeddings into `k=4` clusters.  Make a count plot of the number of tweets in each cluster and a bar plot of the mean sentiment per cluster, with proper axis labels and title.  This question might look familiar to you (check a previous homework)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "q4bLvPmBiQIZ"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mlXQab8Qgzx_"
   },
   "source": [
    "## 3. (4 points) Plot mean sentiment vs mean retweet count per cluster\n",
    "\n",
    "Make a scatterplot of the mean sentiment  vs the mean retweet count of each k-means cluster.  Set the marker size to 1000.   Make sure to add axes labels and a title to the figure.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Df9d1XC4hUpC"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dCgePRNtjU5d"
   },
   "source": [
    "## 4. (4 points) Correlation of retweet count and sentiment\n",
    "\n",
    "Calculate the Pearson correlation coefficient of the mean sentiment and mean retweet count of the clusters.  Print the coefficient and p-value up to three decimal points.  Is the correlation significant at a 1% level?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zCmvgLqRh5s4"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5ND8bw3jlUBG"
   },
   "source": [
    "## 5. (10 points) T-test of cluster sentiment\n",
    "\n",
    "Find the clusters with the lowest and highest mean sentiment and print their cluster label so we know who you are checking.  Then use a t-test to determine if these mean sentiments are different at a 1% level.  Print your t-statistics, pvalues, and whether it is significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OxOgNJ4DiynK"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y-3xoHR1qWzR"
   },
   "source": [
    "# Problem 2. (34 points) Tweet Keywords and Retweet Count\n",
    "\n",
    "We will look at the retweet count as a function of the keyword in a tweet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4Lw0cA2N_-Wg"
   },
   "source": [
    "## 1. (2 points) Load data\n",
    "\n",
    "Load the tweets in the file `\"data/oreo_sentiment.csv\"` into a dataframe. Print out the head of this dataframe so we know you did it right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7zNxhJVq__lB"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rsbaRShfqeDE"
   },
   "source": [
    "## 2. (4 points) Plot retweet count vs. keyword\n",
    "\n",
    "We will look at the keywords `\"fordtrucks\"`, `\"thinsprotectionprogram\"`,`\"pokemon\"`, and `\"oreo\"`.  For each keyword, make a bar plot of the mean retweet count of tweets with the word and without the word.  Put labels on the axes and a title indicating which keyword you are plotting.  A `for` loop might be useful here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F-2l47GaosR5"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDfD_e5JuukK"
   },
   "source": [
    "## 3. (10 points) T-test of retweet count\n",
    "\n",
    "Do a t-test for the retweet count of tweets with and without the keyword for each keyword.  Print your t-statistics and p-values. A `for` loop might be useful here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mtgFuEx4vA7P"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BVfRO10fvKr0"
   },
   "source": [
    "## 4. (8 points) T-test of retweet count with Holm-Bonferroni Method\n",
    "\n",
    "Apply the Holm-Bonferroni method to the calculated p-values.  Print the pvalues and whether the retweet count difference is significant at a 1% level for each keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cFJVZT34tIV2"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3SBzASJ_w0Zq"
   },
   "source": [
    "## 5. (10 points) Oreo vs. Pokemon\n",
    "\n",
    "Apply a t-test to compare the mean retweet count of tweets that contain `\"pokemon\"` to those that contain `\"oreo\"`.  Print the mean retweet count of each group of tweets, the t-statistic and p-value.  Is the difference significant at a 1% level?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OavoiUWJw8wW"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CDZoWEgQ5nYY"
   },
   "source": [
    "# Problem 3. (22 points) Comparing Users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3Q8OWHz1EVkG"
   },
   "source": [
    "## 1. (2 points) Load data\n",
    "\n",
    "Load the tweets in the file `\"data/tweets_KingJamesStephenCurry30.csv\"` into a dataframe.  These are tweets for a few prominent Twitter users.  Print out the unique values in the column `screen_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uLs4uJMi55HO"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bXVSh1fz7pVg"
   },
   "source": [
    "## 2. (10 points) Fisher exact test for keyword usage\n",
    "\n",
    "Do a Fisher exact test to determine if users with screen names `\"KingJames\"` and `\"StephenCurry30\"` use the word `\"champion\"` with the same frequency.  Print the values in your contingency table (`a`,`b`,`c`,`d` defined in lecture).  Print the p-value and  whether the difference significant at a 1% level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HrF-54Xx67io"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "62J5cE_X7w2a"
   },
   "source": [
    "## 3. (10 point) Engagement\n",
    "\n",
    "Define engagement as the ratio of `favorite_count` to `views` count.  Do a t-test to determine if `\"KingJames\"` and `\"StephCurry30\"` have the same engagement.  Print the mean engagement of each user, the t-statistic, p-value, and if the difference significant at a 1% level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Dgx-pHH7h2aI"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G4ONoezD__n2"
   },
   "source": [
    "# Problem 4. (14 points) Tweet Day\n",
    "\n",
    "We will look at the effect the day of the week has on the retweet count for Lebron James."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HBzhnE5g768k"
   },
   "source": [
    "## 1. (4 points) Plot retweet count vs day\n",
    "\n",
    "From the tweets you loaded from the file `\"data/tweets_KingJamesStephenCurry30.csv\"`, get the tweets of the screen name `\"KingJames\"`.  Make a bar plot of the mean retweet count vs. day of the week (you can keep the time in UTC).  Make sure to label axes and put the screen name in the title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_XiAUCjUh4I6"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nkQDG08B-8ZW"
   },
   "source": [
    "## 2. (10 points) T-test of Monday vs. Friday retweet count\n",
    "\n",
    "Apply a t-test to compare the mean retweet count of tweets posted on Sunday vs. posted on Thursday.  Print the mean retweet count of each group of tweets, the t-statistic and p-value.  Is the difference significant at a 1% level?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dKwgeqKYhBQZ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}