00_course_info_definitions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is Data Science?\n",
    " \n",
    "_Authors: David Yakobovitch (NYC)\n",
    " \n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a id=\"learning-objectives\"></a>\n",
    "### Learning Objectives\n",
    "*After this lesson, you will be able to:*\n",
    "\n",
    "- Describe the roles and components of a successful development environment.\n",
    "- Define data science and the data science workflow.\n",
    "- Apply the data science workflow to solve a task.\n",
    "- Discuss common data science terminology and processes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lesson Guide\n",
    "- [Activity: Data Science in the Real World (5 min)](#ds-real-world)\n",
    "- [How to Ask a Question (10 min)](#question)\n",
    "- [Data Science Workflow Through Ames Data (20 min)](#dsfw)\n",
    "- [Summary (5 min)](#summary1)\n",
    "- [Common Machine Learning Definitions (15 min)](#common-ml-defs)\n",
    "- [Activity: Quiz or Group (15 min)](#ml-activity)\n",
    "- [Summary (5 min)](#summary2)\n",
    "- [Course & Project Structure (5 min)](#course-info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a id=\"ds-real-world\"> </a>\n",
    "# Activity: Data Science in the Real World\n",
    "\n",
    "- List five products or services that you think utilize data science.\n",
    "- **Examples**\n",
    "    - Providing movie recommendations on Netflix.\n",
    "    - Making product suggestions on Amazon.\n",
    "    - Offering election and sports coverage on the stats site FiveThirtyEight.\n",
    "    - Calculating daily bet predictions on the fantasy sports site DraftKings.\n",
    "    - Returning auto-translate and search results on Google."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"question\"> </a>\n",
    "# How to Ask Good Questions\n",
    "\n",
    "### How Do Data Scientists Solve Problems?\n",
    "\n",
    "Most practitioners apply a version of the scientific method in order to logically deconstruct and analyze an issue. We call this the data science workflow, which we've broken down into a series of steps.\n",
    "\n",
    "This problem-solving framework will help you produce results that are reliable (so that your findings will be more accurate) and reproducible (so that others can follow your steps and achieve the same results).\n",
    "\n",
    "Note that, depending on the problem, this process is not always linear. You may require lots of iteration and repetition before any conclusions can be drawn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"good_q\"></a>\n",
    "### Asking a Good Question\n",
    "\n",
    "Even though all data science projects have different general flows, they start in the same place: with a problem.  From this problem statement arises questions; questions we will ask the data in order to gain more information so we can attempt to find a solution to that problem.\n",
    "\n",
    "**Why do we need a good question?**\n",
    "\n",
    "_“A problem well stated is half solved.”_ — Charles Kettering\n",
    "\n",
    "A good question: \n",
    "\n",
    "- Sets you up for success as you begin analysis.\n",
    "- Establishes the basis for reproducibility.\n",
    "- Enables collaboration through clear goals.\n",
    "    - It's hard to collaborate without a vision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to approach formulating a question is through goal-setting via the SMART Goals Framework:\n",
    "\n",
    "- **Specific**: The data set and key variables are clearly defined.\n",
    "- **Measurable**: The type of analysis and major assumptions are articulated.\n",
    "- **Attainable**: The question you are asking is feasible for your data set and not likely to be biased.\n",
    "- **Reproducible**: Another person (or future you) can read and understand exactly how your analysis is performed.\n",
    "- **Time-bound**: You clearly state the time period and population to which this analysis pertains.\n",
    "\n",
    "#### Check for Understanding\n",
    "\n",
    "- Can you make a goal for this class with a partner? Use the SMART Goals Framework to make sure that it's specific, measurable, attainable, reproducible, and time-bound."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What Are Some Common Questions Asked in Data Science?\n",
    "\n",
    "**Machine learning more or less asks the following questions:**\n",
    "\n",
    "- Does X predict Y? (Where X is a set of data and y is an outcome.)\n",
    "- Are there any distinct groups in our data?\n",
    "- What are the key components of our data?\n",
    "- Is one of our observations “weird”?\n",
    "\n",
    "**From a business perspective, we can ask:**\n",
    "\n",
    "- What is the likelihood that a customer will buy this product?\n",
    "- Is this a good or bad review?\n",
    "- How much demand will there be for my service tomorrow?\n",
    "- Is this the cheapest way to deliver my goods?\n",
    "- Is there a better way to segment my marketing strategies?\n",
    "- What groups of products are customers purchasing together?\n",
    "- Can we automate this simple yes/no decision?\n",
    "\n",
    "_This list may seem limited, but we rewrite most questions to fit this form._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a id=\"dswf\"></a>\n",
    "## Introduction: The Data Science Workflow\n",
    "\n",
    "---\n",
    "\n",
    "Throughout this course and for our projects, we'll be following a general workflow. This workflow will help you produce *reliable* and *reproducible* results.\n",
    "\n",
    "- **Reliable**: Accurate findings.\n",
    "- **Reproducible**: Others can follow your steps and achieve the same results.\n",
    "\n",
    "### Steps in the Data Science Workflow\n",
    "\n",
    "- **Frame**: Develop a hypothesis-driven approach to your analysis.\n",
    "- **Prepare**: Select, import, explore, and clean your data.\n",
    "- **Analyze**: Structure, visualize, and complete your analysis.\n",
    "- **Interpret**: Derive recommendations and business decisions from your data.\n",
    "- **Communicate**: Present (edited) insights from your data to different audiences.\n",
    "\n",
    "![](./assets/Data-Framework-White-BG.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Notes about the Data Workflow\n",
    "\n",
    "_Remember, these steps are not hard-set rules; instead, think of them as problem-solving guidelines._\n",
    "\n",
    "\n",
    "- Some projects may not require every step.\n",
    "- These steps are iterative; it's normal to go back and repeat certain steps a few times in a row.\n",
    "- The process is cyclical; after completing the process, you may restart it on new findings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application: Data Science Workflow Through Ames Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frame\n",
    "\n",
    "---\n",
    "\n",
    "- Identify the business/product objectives.\n",
    "- Identify and hypothesize goals and criteria for success.\n",
    "- Create a set of questions to help you identify the correct data set.\n",
    "\n",
    "We work for a real estate company interested in using data science to determine the best properties to buy and resell. Specifically, your company would like to identify the characteristics of residential houses that estimate their sale price and the cost-effectiveness of doing renovations.\n",
    "\n",
    "#### Identify the Business/Product Objectives\n",
    "The customer tells us their business goals are to accurately predict prices for houses (so that they can sell them for as large a profit as possible) and to identify which kinds of features in the housing market would be more likely to lead to foreclosure and other abnormal sales (which could represent more profitable sales for the company).\n",
    "\n",
    "#### Identify and Hypothesize Goals and Criteria for Success\n",
    "\n",
    "Ultimately, the customer wants us to:\n",
    "* Deliver a presentation to the real estate team.\n",
    "* Write a business report discussing results, procedures used, and rationales.\n",
    "* Build an API that provides estimated returns.\n",
    "\n",
    "#### Create a Set of Questions to Help You Identify the Correct Data Set\n",
    "\n",
    "* Can you think of questions that would help this customer deliver on their business goals? \n",
    "* What sort of features or columns would you want to see in the data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Ideal Data vs. Available Data**  \n",
    "\n",
    "Oftentimes, we'll start by identifying the *ideal data* we would want for a project.\n",
    "\n",
    "Then, during the data acquisition phase, we'll learn about the limitations on the types of data actually available. We have to decide if these limitations will inhibit our ability to answer our question of interest or if we can work with what we have to find a reasonable and reliable answer.\n",
    "\n",
    "For example, we provide a set of housing data for Ames, Iowa, which [includes](./extra-materials/ames_data_documentation.txt):\n",
    "\n",
    "- 20 continuous variables indicating square footage.\n",
    "- 14 discrete variables indicating number of each room type.\n",
    "- 46 categorical variables containing 2–28 classes each, e.g., street type (gravel/paved) and neighborhood (city district name).\n",
    "\n",
    "---\n",
    "\n",
    "#### **Optional Check**\n",
    "\n",
    "Take a moment to look through the data description. How closely does the set match the ideal data that you envisioned? Would it be sufficient for our purposes? What limitations does it have?\n",
    "\n",
    "---\n",
    "\n",
    "This is possibly the hardest step in the data science workflow. At this stage, it's common to realize that the problem you're trying to solve may not be solvable with the information available. The data could be incomplete, non-existant, or unable to meet the criteria necessary to answer your question.  \n",
    "\n",
    "That said, you now have a better feel for the data that's available and the information they could contain. You can now identify a new, answerable question that ultimately helps you solve or better understand your problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data acquisition\n",
    "\n",
    "---\n",
    "\n",
    "- **What are some questions we should ask during the acquisition process?**\n",
    "\n",
    "- Our Ames data set contains the following information:\n",
    "    - [Ames Data Set Introduction PDF](./extra-materials/ames.pdf) (from the \"Journal of Statistics Education\")\n",
    "    - \"Data set contains information from the Ames Assessor’s Office used in computing assessed values for individual residential properties sold in Ames, IA from 2006 to 2010.\"\n",
    "\n",
    "### Data Quality\n",
    "\n",
    "---\n",
    "\n",
    "- **What are some questions we should ask when checking the data for quality?**\n",
    "  - [Ames Data Set Documentation](./extra-materials/ames_data_documentation.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "\n",
    "> - Where are the data coming from?\n",
    "> - How do the data fit together?\n",
    "> - Are there enough data?\n",
    "> - Do our data appropriately align with the question/problem statement?\n",
    "> - Can the data set be trusted? How was it collected?\n",
    "> - Is this data set aggregated? Can we use the aggregation, or do we need to obtain it pre-aggregation?\n",
    "> - What are necessary resources, requirements, assumptions, and constraints?\n",
    "> - Can we import data from the web (Google Analytics, HTML, XML)?\n",
    "> - Can we import data from a file (CSV, XML, TXT, JSON)?\n",
    "> - Can we import data from a pre-existing database (SQL)?\n",
    "> - Can we set up local or remote data structures?\n",
    "> - What are the most appropriate tools for working with the data?\n",
    "> - Do these tools align with the format and size of the data set?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Prepare\n",
    "\n",
    "---\n",
    "\n",
    "Often, we are given *secondary data*, or data that were collected previously. In these cases, we have to learn as much as possible about our data using tools like data dictionaries and source documentation to determine how the set was gathered.\n",
    "\n",
    "Here's an example of a data dictionary:\n",
    "\n",
    "Variable | Description | Type of Variable\n",
    "---| ---| ---\n",
    "Square Footage | Floating Point | Continuous\n",
    "Street Type | 1 - Gravel, 2 - Paved | Categorical\n",
    "Neighborhood | String, e.g., 'Tenderloin' | Categorical\n",
    "Number of Bedrooms | Integer | Discrete\n",
    "\n",
    "**Common considerations when preparing our data include:**  \n",
    "\n",
    "- **Ensuring data is clearly defined and structured**\n",
    "- **Check and clean data formatting as needed**\n",
    "\n",
    "**Common considerations for cleaning include**:\n",
    "\n",
    "- **Most data will **not** come perfectly clean and ready to use. Cleaning data is normally the most time-consuming task a data scientist faces.**\n",
    "\n",
    "---\n",
    "\n",
    "As you can see, the \"Prepare\" phase of the data science workflow encompasses several steps: the act of reviewing, indexing, and cleaning your data. This normally consumes a great deal of time!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "\n",
    "- Reading any documentation provided with the data (e.g., the data dictionary above).\n",
    "- Performing exploratory surface analysis via filtering, sorting, and simple visualizations.\n",
    "- Describing the data structure and the information being collected.\n",
    "- Exploring variables and data types via `SELECT` statements.\n",
    "- Assessing preliminary outliers and trends.\n",
    "- Verifying the quality of the data (feedback loop -> 1).\n",
    "- Sampling the data and determining sampling methodology.\n",
    "- Iterating and exploring outliers and null values via SELECT statements.\n",
    "- Assessing qualitative versus quantitative data.\n",
    "- Formatting and cleaning data in Python (e.g., dates, number signs, formatting).\n",
    "- Defining how to appropriately address missing values (cleaning).\n",
    "- Categorizing, manipulating, slicing, formating, and integrating data.\n",
    "- Formatting and combining different data points, separate columns, etc.\n",
    "- Determining most appropriate methods for aggregating, cleaning, etc.\n",
    "- Creating the necessary derived columns from the data (new data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Analyze\n",
    "\n",
    "---\n",
    "\n",
    "As an example of basic statistics, Data scientists often check the mean, standard deviation, or specific frequency counts of their data. Statistics that we might expect for the earlier housing variables include:\n",
    "\n",
    "Variable | Mean or Frequency (%)\n",
    "---| ---\n",
    "Square Footage | 2201.3\n",
    "Street Type - Gravel | 8%\n",
    "Street Type - Paved | 92%\n",
    "Number of Bedrooms | 1.8\n",
    "\n",
    "**What sort of questions do these types of statistics allow us to answer? Why would we do this?**\n",
    "\n",
    "### Creating a Predictive Model \n",
    "\n",
    "We generate predictive models based on the SMART goal we decided upon earlier. Typically, our interest is in predicting or guessing some sort of value we might be interested in (such as the housing price for a house given some set of fixed characteristics). \n",
    "\n",
    "**What are some other business goals we can support as data scientists for this realty company? What are some values we would like to guess?*\n",
    "\n",
    "**What do you think are the steps for model building?**\n",
    "\n",
    "_We'll be spending much of our time in this course on data analysis and predictive modeling._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "\n",
    "- Identifying trends and outliers.\n",
    "- Deciding how to deal with outliers — excluding, filtering, and communication.\n",
    "- Applying descriptive and inferential statistics.\n",
    "- Determining initial visualization techniques.\n",
    "- Documenting and capturing knowledge.\n",
    "- Choosing visualization techniques for different data types.\n",
    "- Transforming data.\n",
    "\n",
    "\n",
    "- Selecting the appropriate model.\n",
    "- Building the model.\n",
    "- Evaluating and refining the model.\n",
    "- Predicting outcomes and action items."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Interpret\n",
    "\n",
    "---\n",
    "\n",
    "### Develop Recommendations and Decisions\n",
    "\n",
    "**Now that you have a model, what are some things you should check?**\n",
    "\n",
    "**Now that you have a model, can you convert your model's finding into a conclusion or next step for your employer?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "\n",
    "- Did you reject or fail to reject your hypotheses?\n",
    "    - What does this mean for your project?\n",
    "    - What does this mean for your client?\n",
    "- Were your questions answered?\n",
    "    - Which ones?\n",
    "    - What do you need to do to answer the ones that weren't?\n",
    "- Do your findings support any business recommendations, actions, or decisions?\n",
    "    - Is there further supportive analysis?\n",
    "    - How do your data support these recommendations?\n",
    "    \n",
    "- Reaching a conclusion:\n",
    "    - Seek guidance/interaction with subject matter experts (SMEs).\n",
    "    - If those are not available, check with the data — are you coming to reasonable conclusions and predictions given what you've seen?\n",
    "    - Do the next steps that you envision have any dependencies or corollary steps?\n",
    "- What are some conclusions you can draw?\n",
    "    - Conclusion: \"Customers from large companies were twice as likely to place another order with Planet Express than customers from small companies.\"\n",
    "    - Recommendation:  \"We should target more large companies to use our delivery service.\"\n",
    "    - Conclusion: \"Other than size of company, I found no significant evidence that any other feature affected the odds of customers reusing our delivery service.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Communicate\n",
    "\n",
    "---\n",
    "\n",
    "#### Share the Results of Your Analysis  \n",
    "\n",
    "Presentations are a critical part of your analysis. It doesn't matter how brilliant your model is or how illuminating your findings are — without effective communication, your work will not be used.\n",
    "\n",
    "The most basic form of a data science presentation should include a simple sentence that describes your results:\n",
    "\n",
    "_\"Customers from large companies had twice (CI 1.9, 2.1) the odds for placing another order with Planet Express compared to customers from small companies.\"_\n",
    "\n",
    "Data science presentations can also be far more complex and exciting, like some of the [research presented by Nate Silver's FiveThirtyEight blog](http://fivethirtyeight.com/burrito/#brackets-view).\n",
    "\n",
    "When crafting a presentation, always consider your audience and make sure to practice your presentation beforehand. Consider the types of questions people might ask or — better yet — test your presentation on a few people and pay attention to their response. Clarify and refine your presentation accordingly.\n",
    "\n",
    "Make sure to consider your needs and goals, as well as those of your audience. A presentation created for your fellow data scientists will be vastly different than a presentation intended for executives trying to make a business decision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**A Note About Iteration**\n",
    "\n",
    "Iteration is an important part of *every step* in the data science workflow. At any given point in the process, you may find yourself repeating or going back and redoing steps in order to better understand your data, clarify your model, and refine your presentation.\n",
    "\n",
    "**What are some things you may want to redo or iterate over after presenting your findings?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "\n",
    "- Identifying follow-up problems and questions for future analysis.\n",
    "- Creating a visually effective summary or report.\n",
    "- Adding more features.\n",
    "- Considering the needs of different stakeholders and how your report might be changed for them.\n",
    "- Identifying the limitations of your analysis.\n",
    "- Identifying relationships between visualizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"summary1\"></a>\n",
    "# Summary\n",
    "\n",
    "---\n",
    "\n",
    "1) **Crafting good questions is key.** <br>\n",
    "  - Without a thoughtful, targeted, and SMART question, it can be difficult to create an effective model.\n",
    "2) **Use the data science workflow to iteratively develop solutions.** <br>\n",
    "  - **Frame**: Develop a hypothesis-driven approach to your analysis.\n",
    "  - **Prepare**: Select, import, explore, and clean your data.\n",
    "  - **Analyze**: Structure, visualize, and complete your analysis.\n",
    "  - **Interpret**: Derive recommendations and business decisions from your data.\n",
    "  - **Communicate**: Present (edited) insights from your data to different audiences.\n",
    "3) **Informed by your past work, continue to refine your findings and models.** <br>\n",
    "  - While the data science workflow may appear to be linear, we consistently return to past steps to implement new findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ML\"></a>\n",
    "\n",
    "## Machine Learning\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples of Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "- [GloVe Word Vectors](https://github.com/messiest/glove-vectors)\n",
    "- [Image Classification](https://github.com/messiest/pytorch-image-classifiers)\n",
    "- _Bonus_ [Reinforcement Learning](https://github.com/messiest/pytorch-cartpole)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"common-ml-defs\"> </a>\n",
    "## Common Machine Learning Definitions\n",
    "\n",
    "There are two main categories of machine learning: supervised learning and unsupervised learning.\n",
    "\n",
    "**Supervised learning (a.k.a., “predictive modeling”):**  \n",
    "_Classification and regression_\n",
    "- Predicts an outcome based on input data.\n",
    "    - Example: Predicts whether an email is spam or ham.\n",
    "- Attempts to generalize.\n",
    "- Requires past data on the element we want to predict (the target).\n",
    "\n",
    "**Unsupervised learning:**  \n",
    "_Clustering and dimensionality reduction_\n",
    "- Extracts structure from data.\n",
    "    - Example: Segmenting grocery store shoppers into “clusters” that exhibit similar behaviors.\n",
    "- Attempts to represent.\n",
    "- **Does not require** past data on the element we want to predict.\n",
    "\n",
    "Oftentimes, we may combine both types of machine learning in a project to reduce the cost of data collection by learning a better representation. This is referred to as transfer learning.\n",
    "\n",
    "Unsupervised learning tends to present more difficult problems because its goals are amorphous. Supervised learning has goals that are almost too clear and can lead people into the trap of optimizing metrics without considering business value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"supervised\"></a>\n",
    "### Supervised Learning\n",
    "\n",
    "Supervised learning tends to be the most frequent type of work that data scientists do and will be the main focus of this course. How does supervised learning work?\n",
    "\n",
    "1) We train a **machine learning model** (more on that shortly) using **labeled data** (the \"response\" label from earlier). <br>\n",
    "    - The “machine learning model” learns some kind of relationship between the features and the response.\n",
    "\n",
    "2) We make predictions on **new data** for which the response is unknown. <br>\n",
    "\n",
    "The primary goal of supervised learning is to build a model that “generalizes” — i.e., accurately predicts the **future** rather than the **past**!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practice: Classification vs. Regression\n",
    "\n",
    "There are two categories of supervised learning:\n",
    "\n",
    "**Regression**\n",
    "- The outcome we are trying to predict is a continuous value.\n",
    "    - **Can you think of anything we would want to predict like this?** \n",
    "\n",
    "**Classification**\n",
    "- The outcome we are trying to predict is categorical (i.e., it comes in one of a set number of classes).\n",
    "    - **Can you think of anything that we would want to predict like this?**\n",
    "\n",
    "The type of supervised learning problem has nothing to do with the features; only the response matters!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    ">Examples of regression targets include price, blood pressure, temperature, etc.\n",
    "\n",
    ">Examples of classification include spam/ham, cancer class of tissue sample, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised Learning\n",
    "\n",
    "#### Common Types of Unsupervised Learning\n",
    "\n",
    "- **Clustering:** Groups “similar” data points together.\n",
    "- **Dimensionality reduction:** Reduce the dimensionality of a data set by extracting features that capture most of the variance in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Steps for Clustering**\n",
    "\n",
    "Imagine that we had a bunch of coins we wanted to automatically split into groups. An unsupervised learning technique would involve the following steps:\n",
    "\n",
    "1) Clustering the coins based on “similarity\" — this could be through the size, the material, or the language on the coins. <br>\n",
    "2) Inspecting the grouping that the algorithm found. <br>\n",
    "\n",
    "Hopefully this would put the coins into sets of related groups.\n",
    "\n",
    "**Steps for Dimensionality Reduction**\n",
    "\n",
    "Imagine that we had a huge amount of features related to those coins — country of origin, size, weight, mass, density, condition, chemical makeup, etc. Moreover, say that we had thousands or (in some cases, millions) of different features. Not all of these features are helpful, however! Unsupervised learning can help us by grouping features together automatically. It would involve the following:\n",
    "\n",
    "1) The unsupervised learning technique groups or combines features that are similar or do the same thing into a smaller set, leading to a set of new features that's smaller in size. <br>\n",
    "\n",
    "Hopefully, the algorithm would recognize something like.\n",
    "\n",
    "$$\\dfrac {mass} {size} = density$$\n",
    "\n",
    "Here, density could take the place of two different features from before. \n",
    "\n",
    "Sometimes unsupervised learning is used as a “preprocessing” step for supervised learning. (Can you guess how?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    ">It can be used as a preprocessing step to provide a new set of features for a structured machine learning technique down the line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples\n",
    "\n",
    "**Supervised Learning: Coin Classifier**\n",
    "\n",
    "- **Observations:** Coins.\n",
    "- **Features:** Size and mass.\n",
    "- **Response or target variable:** Hand-labeled coin type.\n",
    "\n",
    "- Train a machine learning model using labeled data.\n",
    "    - The model learns the relationship between the features and the coin type.\n",
    "\n",
    "- Make predictions on new data for which the response is unknown.\n",
    "    - Give the model a new coin and it will predict the coin type automatically.\n",
    "    \n",
    "**Unsupervised Learning: Types of Customers at a Bar**\n",
    "\n",
    "- **Observations:** Customers.\n",
    "- **Features:** Drink purchases, people they interact with, etc.\n",
    "- **Response or target variable:** There isn’t one — instead, we group similar customers together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = 'algorithm'></a>\n",
    "\n",
    "## Algorithms\n",
    "\n",
    "Regardless of whether it's supervised or unsupervised, the underlying engine driving a machine learning model is an algorithm. These algorithms are used to help identify trends, represent said trends, and explain the overall variance of the data.   \n",
    "\n",
    "Let's say we are a real estate agent looking to price a house using only its square footage. We know there are other features that can highly influence this outcome, but we are only focusing on square footage for now. We know that, as square footage increases, so does price. At this point, you may be thinking that a simple algebra equation could be useful; one that helps us price the house by its square footage.  \n",
    "\n",
    "Recently, we sold a house whose square footage was 2,500 for about \\$285,000. If we apply this information to a normal linear equation — $ Y = mx + b$ — we can create a simple _algorithm_ to help us predict a house.\n",
    "\n",
    "$$285,000 = 2,500x + b$$\n",
    "\n",
    "$$ x = 114, b = 0 $$ \n",
    "\n",
    "_The Y intercept has been omitted for this example._\n",
    "\n",
    "#### Final Algorithm\n",
    "\n",
    "$$ Price = 114x $$\n",
    "\n",
    "This is an example of a model built with the intent of predicting price. The algorithm is simple and built off of limited information. Typically, our models will be more complex, and we'll consider a greater amount of prior data to help us develop a final algorithm.  \n",
    "\n",
    "#### Algorithm Training \n",
    "\n",
    "In our example, we used previously known information to find our coefficients. This action is also referred to as \"training.\" But, let's make something clear:\n",
    "\n",
    "- Model building would be the task of constructing an actual algorithm.\n",
    "    - This is the linear model of $ Y = mx + b $.\n",
    "- Training involves figuring out the coefficient and the Y intercept the model uses for _our intended purpose_.  \n",
    "    - The coefficients uncovered via training were $m= 114$ and $b=0$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ml-activity\"></a>\n",
    "\n",
    "## Partner Activity\n",
    "\n",
    "Partner with someone sitting next to you and tackle the following challenge! We'll spend a few minutes working together and then check in with one or two groups to see what they've come up with.\n",
    "\n",
    "You'll need the [Ames Data Set documentation](./extra-materials/ames_data_documentation.txt) and, optionally, the [Ames Data Set Introduction PDF](./extra-materials/ames.pdf):\n",
    "\n",
    "**Your Task**\n",
    "\n",
    "With a partner, sketch out answers to the following:\n",
    "\n",
    "- What is a potential target in your data for a regression model?\n",
    "- What is a potential target in your data for a classification model?\n",
    "- Could unsupervised learning be used within this data? How so?\n",
    "\n",
    "Once you've done so:\n",
    "\n",
    "- Together, pick one of your targets and sketch out what a data science workflow would look like for that question. Don't forget to identify what you think would be most important during each step:\n",
    "  - **Frame**: Develop a hypothesis-driven approach to your analysis. \n",
    "  - **Prepare**: Select, import, explore, and clean your data.\n",
    "  - **Analyze**: Structure, visualize, and complete your analysis.\n",
    "  - **Interpret**: Derive recommendations and business decisions from your data.\n",
    "  - **Communicate**: Present (edited) insights from your data to different audiences.\n",
    "  \n",
    "Feel free to avoid getting too specific — this exercise is meant to help you practice approaching a problem the way a data scientist would. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"conclusion2\"></a>\n",
    "## Conclusion\n",
    "\n",
    "---\n",
    "\n",
    "Check to see if you can answer the following questions easily:\n",
    "\n",
    "- What is data science?\n",
    "- What is the data science workflow?\n",
    "- What is the difference between supervised and unsupervised learning?\n",
    "- What is the difference between regression and classification? \n",
    "- What is an algorithm?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}