README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%",
  message = F,
  warning = F
)

ggplot2::theme_set(ggplot2::theme_minimal())

library(dplyr)
```


# peRspective <img src="man/figures/perspective.png" width="160px" align="right" />


```{r, echo = FALSE, results='asis', eval = T}

library(badger)

git_repo <- "favstats/peRspective"

cat(
  "[![](https://cranlogs.r-pkg.org/badges/grand-total/peRspective)](https://cran.rstudio.com/web/packages/peRspective/index.html)",
  # badge_custom("My 1st Package", "Ever", "blue"),
  badge_travis(git_repo),
  badge_cran_release("peRspective", "blue"),
  "[![Codecov test coverage](https://codecov.io/gh/favstats/peRspective/branch/master/graph/badge.svg)](https://codecov.io/gh/favstats/peRspective?branch=master)",
	badge_code_size(git_repo),
	badge_last_commit(git_repo)
)
```


Perspective is an API that uses machine learning models to score the perceived impact a comment might have on a conversation. [Website](http://www.perspectiveapi.com/).

`peRspective` provides access to the API using the R programming language.

For an excellent documentation of the Perspective API see [here](https://github.com/conversationai/perspectiveapi/blob/master/api_reference.md).

> This is a work-in-progress project and I welcome feedback and pull requests!

## Overview

+ [Setup](https://github.com/favstats/peRspective#setup)
+ [Models](https://github.com/favstats/peRspective#models)
+ [Usage](https://github.com/favstats/peRspective#usage)
    + [`prsp_score`](https://github.com/favstats/peRspective#prsp_score)
    + [`prsp_stream`](https://github.com/favstats/peRspective#prsp_stream)




## Setup

### Get an API key

1. Create a Google Cloud project in your [Google Cloud console](https://console.developers.google.com/)
2. Go to [Perspective API's overview page](https://console.developers.google.com/apis/api/commentanalyzer.googleapis.com/overview) and click **Enable**
3. Go to the [API credentials page](https://console.developers.google.com/apis/credentials), just click **Create credentials**, and choose "API Key".

Now you are ready to make a request to the Perspective API!

### Quota and character length Limits

Be sure to check your quota limit! You can learn more about Perspective API quota limit by visiting [your google cloud project's Perspective API page](https://console.cloud.google.com/apis/api/commentanalyzer.googleapis.com/quotas).

The maximum text size per request is 3000 bytes.

## Models

For detailed overview of the used models [see here](https://github.com/conversationai/perspectiveapi/blob/master/api_reference.md).

Here is a list of models currently supported by `peRspective`:

```{r, echo=F}

tibble::tribble(
  ~`Model Attribute Name`,  ~Version,                                  ~`Supported Languages`,   ~`Short Description`,
             "TOXICITY",     "Alpha",                                  "en, es, fr\\*, de\\*",    "rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.",  
      "SEVERE_TOXICITY",     "Alpha",                                  "en, es, fr\\*, de\\*",    "Same deep-CNN algorithm as TOXICITY, but is trained on 'very toxic' labels.",
      "IDENTITY_ATTACK",     "Experimental toxicity sub-attribute",    "en, fr\\*, de\\*, es\\*", "negative or hateful comments targeting someone because of their identity.",
               "INSULT",     "Experimental toxicity sub-attribute",    "en, fr\\*, de\\*, es\\*", "insulting, inflammatory, or negative comment towards a person or a group of people.",
            "PROFANITY",     "Experimental toxicity sub-attribute",    "en, fr\\*, de\\*, es\\*", "swear words, curse words, or other obscene or profane language.",
    "SEXUALLY_EXPLICIT",     "Experimental toxicity sub-attribute",    "en, fr\\*, de\\*, es\\*", "contains references to sexual acts, body parts, or other lewd content.",
               "THREAT",     "Experimental toxicity sub-attribute",    "en, fr\\*, de\\*, es\\*", "describes an intention to inflict pain, injury, or violence against an individual or group.",
           "FLIRTATION",     "Experimental toxicity sub-attribute",    "en, fr\\*, de\\*, es\\*", "pickup lines, complimenting appearance, subtle sexual innuendos, etc.",
     "ATTACK_ON_AUTHOR",     "NYT moderation models",                  "en",                      "Attack on the author of an article or post.",
  "ATTACK_ON_COMMENTER",    "NYT moderation models",                   "en",                      "Attack on fellow commenter.",
           "INCOHERENT",     "NYT moderation models",                  "en",                      "Difficult to understand, nonsensical.",
         "INFLAMMATORY",     "NYT moderation models",                  "en",                      "Intending to provoke or inflame.",
     "LIKELY_TO_REJECT",     "NYT moderation models",                  "en",                      "Overall measure of the likelihood for the comment to be rejected according to the NYT's moderation.",
              "OBSCENE",     "NYT moderation models",                  "en",                      "Obscene or vulgar language such as cursing.",
                 "SPAM",     "NYT moderation models",                  "en",                      "Irrelevant and unsolicited commercial content.",
        "UNSUBSTANTIAL",     "NYT moderation models",                  "en",                       "Trivial or short comments."
  ) %>% 
  knitr::kable()

```

**Note:** Languages that are annotated with "*" are only accessible in the `_EXPERIMENTAL` version of the models. In order to access them just add to the supplied model string like this: `TOXICITY_EXPERIMENTAL`.

A character vector that includes all supported models can be obtained like this:

```{r}
c(
  peRspective::prsp_models,
  peRspective::prsp_exp_models
)
```

## Usage

First, install package from GitHub:

```{r, eval = F}
devtools::install_github("favstats/peRspective")
```

Load package:

```{r}
library(peRspective)
```

Also the `tidyverse` for examples.

```{r, eval = F}
library(tidyverse)
```

```{r, eval = F, echo=F}
key <- readr::read_lines("../keys/prsp_simon.txt")

Sys.setenv(perspective_api_key = key)
```



**Define your key variable.**

`peRspective` functions will read the API key from environment variable `perspective_api_key`. In order to add your key to your environment file, you can use the function `edit_r_environ()` from the [`usethis` package](https://usethis.r-lib.org/).

```{r, eval = F}
usethis::edit_r_environ()
```


This will open your .Renviron file in your text editor. Now, you can add the following line to it:

```{r, eval = F}
perspective_api_key="YOUR_API_KEY"
```

Save the file and restart R for the changes to take effect.

Alternatively, you can provide an explicit definition of your API key with each function call using the `key` argument.

### `prsp_score`

Now you can use `prsp_score` to score your comments with various models provided by the Perspective API.

```{r, eval = T}

my_text <- "You wrote this? Wow. This is dumb and childish, please go f**** yourself."

text_scores <- prsp_score(
           text = my_text, 
           languages = "en",
           score_model = peRspective::prsp_models
           )

text_scores %>% 
  tidyr::gather() %>% 
  dplyr::mutate(key = forcats::fct_reorder(key, value)) %>% 
  ggplot2::ggplot(ggplot2::aes(key, value)) +
  ggplot2::geom_col() +
  ggplot2::coord_flip() +
  ggplot2::ylim(0, 1) +
  ggplot2::geom_hline(yintercept = 0.5, linetype = "dashed") +
  ggplot2::labs(x = "Model", y = "Probability", title = "Perspective API Results")
```

```{r, echo=F}
Sys.sleep(1)
```


A Trump Tweet:

```{r}
trump_tweet <- "The Fake News Media has NEVER been more Dishonest or Corrupt than it is right now. There has never been a time like this in American History. Very exciting but also, very sad! Fake News is the absolute Enemy of the People and our Country itself!"

text_scores <- prsp_score(
           trump_tweet, 
           score_sentences = F,
           score_model = peRspective::prsp_models
           )

text_scores %>% 
  tidyr::gather() %>% 
  dplyr::mutate(key = forcats::fct_reorder(key, value)) %>% 
  ggplot2::ggplot(ggplot2::aes(key, value)) +
  ggplot2::geom_col() +
  ggplot2::coord_flip() +
  ggplot2::ylim(0, 1) +
  ggplot2::geom_hline(yintercept = 0.5, linetype = "dashed") +
  ggplot2::labs(x = "Model", y = "Probability", title = "Perspective API Results")
```

```{r, echo=F}
Sys.sleep(1)
```

Instead of scoring just entire comments you can also score individual sentences with `score_sentences = T`. In this case the Perspective API will automatically split your text into reasonable sentences and score them in addition to an overall score.

```{r, eval = T, fig.width=12, fig.height=8}
trump_tweet <- "The Fake News Media has NEVER been more Dishonest or Corrupt than it is right now. There has never been a time like this in American History. Very exciting but also, very sad! Fake News is the absolute Enemy of the People and our Country itself!"

text_scores <- prsp_score(
           trump_tweet, 
           score_sentences = T,
           score_model = peRspective::prsp_models
           )

text_scores %>% 
  tidyr::unnest(sentence_scores) %>% 
  dplyr::select(type, score, sentences) %>% 
  tidyr::gather(value, key, -sentences, -score) %>% 
  dplyr::mutate(key = forcats::fct_reorder(key, score)) %>% 
  ggplot2::ggplot(ggplot2::aes(key, score)) +
  ggplot2::geom_col() +
  ggplot2::coord_flip() +
  ggplot2::facet_wrap(~sentences, ncol = 2) +
  ggplot2::geom_hline(yintercept = 0.5, linetype = "dashed") +
  ggplot2::labs(x = "Model", y = "Probability", title = "Perspective API Results")


```

```{r, echo=F}
Sys.sleep(1)
```

You can also use Spanish (`es`) for `TOXICITY`, `SEVERE_TOXICITY` and `_EXPERIMENTAL` models.

```{r}
spanish_text <- "gastan en cosas que de nada sirven-nunca tratan de saber la verdad del funcionalismo de nuestro sistema solar y origen del cosmos-falso por Kepler. LAS UNIVERSIDADES DEL MUNDO NO SABEN ANALIZAR VERDAD O MENTIRA-LO QUE DICE KEPLER"


text_scores <- prsp_score(
           text = spanish_text, 
           languages = "es",
           score_model = c("TOXICITY", "SEVERE_TOXICITY", "INSULT_EXPERIMENTAL")
           )

text_scores %>% 
  tidyr::gather() %>% 
  dplyr::mutate(key = forcats::fct_reorder(key, value)) %>% 
  ggplot2::ggplot(ggplot2::aes(key, value)) +
  ggplot2::geom_col() +
  ggplot2::coord_flip() +
  ggplot2::geom_hline(yintercept = 0.5, linetype = "dashed")  +
  ggplot2::labs(x = "Model", y = "Probability", title = "Perspective API Results")
```

**NOTE:** Your provided text will be stored by the Perspective API for future research. This option is the default. If the supplied texts are private or any of the authors of the texts are below 13 years old, `doNotStore` should be set to `TRUE.`

### `prsp_stream`

```{r, echo=F}
Sys.sleep(1)
```

So far we have only seen how to get individual comments or sentences scored. But what if you would like to run the function for an entire dataset with a text column? This is where `prsp_stream` comes in. At its core `prsp_stream` is a loop implemented within `purrr::map` to iterate over your text column. To use it let's first generate a mock tibble.

```{r}
text_sample <- tibble(
       ctext = c("You wrote this? Wow. This is dumb and childish, please go f**** yourself.",
                 "I don't know what to say about this but it's not good. The commenter is just an idiot",
                 "This goes even further!",
                 "What the hell is going on?",
                 "Please. I don't get it. Explain it again",
                 "Annoying and irrelevant! I'd rather watch the paint drying on the wall!"),
       textid = c("#efdcxct", "#ehfcsct", 
                  "#ekacxwt",  "#ewatxad", 
                  "#ekacswt",  "#ewftxwd")
       )
```


```{r, echo=F}
Sys.sleep(1)
```

`prsp_stream` requires a `text` and `text_id` column. It wraps `prsp_score` and takes all its arguments. Let's run the most basic version:

```{r, message=T}
text_sample %>%
  prsp_stream(text = ctext,
              text_id = textid,
              score_model = c("TOXICITY", "SEVERE_TOXICITY"))
```

You receive a `tibble` with your desired scorings including the `text_id` to match your score with your original dataframe.

Now, the problem is that sometimes the call might fail at some point. It is therefore suggested to set `safe_output = TRUE`. This will put the function into a `purrr::safely` environment to ensure that your function will keep running even if you encounter errors. 

Let's try it out with a new dataset that contains text that the Perspective API can't score

```{r, echo=F}
Sys.sleep(1)
```

```{r}
text_sample <- tibble(
       ctext = c("You wrote this? Wow. This is dumb and childish, please go f**** yourself.",
                 "I don't know what to say about this but it's not good. The commenter is just an idiot",
                 ## empty string
                 "",
                 "This goes even further!",
                 "What the hell is going on?",
                 "Please. I don't get it. Explain it again",
                 ## Gibberish
                 "kdlfkmgkdfmgkfmg",
                 "Annoying and irrelevant! I'd rather watch the paint drying on the wall!",
                 ## Gibberish
                 "Hippi Hoppo"),
       textid = c("#efdcxct", "#ehfcsct", 
                  "#ekacxwt",  "#ewatxad", 
                  "#ekacswt",  "#ewftxwd", 
                  "#eeadswt",  "#enfhxed",
                  "#efdmjd")
       )
```

And run the function with `safe_output = TRUE`.

```{r, echo=F}
Sys.sleep(1)
```

```{r, message=T}
text_sample %>%
  prsp_stream(text = ctext,
              text_id = textid,
              score_model = c("TOXICITY", "SEVERE_TOXICITY", "INSULT"),
              safe_output = T)
```

`safe_output = T` will also provide us with the error messages that occured so that we can check what went wrong!

Finally, there is one last argument: `verbose = TRUE`. Enable this argument and thanks to [`crayon`](https://github.com/r-lib/crayon) you will receive beautiful console output that guides you along the way, showing you errors and text scores as you go.

```{r, echo=F}
Sys.sleep(1)
```


```{r, eval = F}
text_sample %>%
  prsp_stream(text = ctext,
              text_id = textid,
              score_model = c("TOXICITY", "SEVERE_TOXICITY"),
              verbose = T,
              safe_output = T)
```

![](man/figures/prsp_stream_output.png)

Or the (not as pretty) output in Markdown

```{r, echo = F}
text_sample %>%
  prsp_stream(text = ctext,
              text_id = textid,
              score_model = c("TOXICITY", "SEVERE_TOXICITY"),
              verbose = T,
              safe_output = T)


```


## Citation

Thank you for using peRspective! Please consider citing:

Votta, Fabio. (2019). peRspective: A wrapper for the Perspective API. Source: https://github.com/favstats.


<div>Icons made by <a href="https://www.freepik.com/?__hstc=57440181.1504a979705d81fb44d4169a0ccdf2ae.1558036002089.1558036002089.1558036002089.1&__hssc=57440181.4.1558036002090&__hsfp=2902986854" title="Freepik">Freepik</a> from <a href="https://www.flaticon.com/" 			    title="Flaticon">www.flaticon.com</a> is licensed by <a href="http://creativecommons.org/licenses/by/3.0/" 			    title="Creative Commons BY 3.0" target="_blank">CC 3.0 BY</a></div>