Skip to content
This repository has been archived by the owner on Oct 18, 2019. It is now read-only.

Grammatical errors #10

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Grammatical errors #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
95984bd
Grammatical errors
ashleyvargas Oct 6, 2016
e799065
Few other grammatical errors
ashleyvargas Oct 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
View file
Open in desktop
Binary file not shown.
14 changes: 7 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Open nsfw model
This repo contains code for running Not Suitable for Work (NSFW) classification deep neural network Caffe models. Please refer our [blog](https://yahooeng.tumblr.com/post/151148689421/open-sourcing-a-deep-learning-solution-for) post which describes this work and experiments in more detail.
This repo contains code for running Not Suitable for Work (NSFW) classification deep neural network Caffe models. Please refer to our [blog](https://yahooeng.tumblr.com/post/151148689421/open-sourcing-a-deep-learning-solution-for) post which describes this work and experiments in more detail.

#### Not suitable for work classifier
Detecting offensive / adult images is an important problem which researchers have tackled for decades. With the evolution of computer vision and deep learning the algorithms have matured and we are now able to classify an image as not suitable for work with greater precision.
Detecting offensive / adult images is an important problem which researchers have tackled for decades. With the evolution of computer vision and deep learning the algorithms having matured, we are now able to classify an image as not suitable for work with greater precision.

Defining NSFW material is subjective and the task of identifying these images is non-trivial. Moreover, what may be objectionable in one context can be suitable in another. For this reason, the model we describe below focuses only on one type of NSFW content: pornographic images. The identification of NSFW sketches, cartoons, text, images of graphic violence, or other types of unsuitable content is not addressed with this model.

Expand All @@ -13,9 +13,9 @@ Since images and user generated content dominate the internet today, filtering n

#### Usage

* The network takes in an image and gives output a probability (score between 0-1) which can be used to filter not suitable for work images. Scores < 0.2 indicate that the image is likely to be safe with high probability. Scores > 0.8 indicate that the image is highly probable to be NSFW. Scores in middle range may be binned for different NSFW levels.
* Depending on the dataset, usecase and types of images, we advise developers to choose suitable thresholds. Due to difficult nature of problem, there will be errors, which depend on use-cases / definition / tolerance of NSFW. Ideally developers should create an evaluation set according to the definition of what is safe for their application, then fit a [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) curve to choose a suitable threshold if they are using the model as it is.
* ***Results can be improved by [fine-tuning](http://caffe.berkeleyvision.org/gathered/examples/finetune_flickr_style.html)*** the model for your dataset/ uscase / definition of NSFW. We do not provide any guarantees of accuracy of results. Please read the disclaimer below.
* The network takes in an image and outputs a probability (score between 0-1) which can be used to filter not suitable for work images. Scores < 0.2 indicate that the image is likely to be safe with high probability. Scores > 0.8 indicate that the image is highly probable to be NSFW. Scores in middle range may be binned for different NSFW levels.
* Depending on the dataset, usecase and types of images, we advise developers to choose suitable thresholds. Due to difficult nature of problem, there will be errors, which depend on usecases / definition / tolerance of NSFW. Ideally, developers should create an evaluation set according to the definition of what is safe for their application, then fit a [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) curve to choose a suitable threshold if they are using the model as it is.
* ***Results can be improved by [fine-tuning](http://caffe.berkeleyvision.org/gathered/examples/finetune_flickr_style.html)*** the model for your dataset/ usecase / definition of NSFW. We do not provide any guarantees of accuracy of results. Please read the disclaimer below.
* Using human moderation for edge cases in combination with the machine learned solution will help improve performance.

#### Description of model
Expand All @@ -24,7 +24,7 @@ We trained the model on the dataset with NSFW images as positive and SFW(suitabl
We use [CaffeOnSpark](https://github.com/yahoo/CaffeOnSpark) which is a wonderful framework for distributed learning that brings deep learning to Hadoop and Spark clusters for training models for our experiments. Big thanks to the CaffeOnSpark team!

The deep model was first pretrained on ImageNet 1000 class dataset. Then we finetuned the weights on the NSFW dataset.
We used the thin resnet 50 1by2 architecture as the pretrained network. The model was generated using [pynetbuilder](https://github.com/jay-mahadeokar/pynetbuilder) tool and replicates the [residual network](https://arxiv.org/pdf/1512.03385v1.pdf) paper's 50 layer network (with half number of filters in each layer). You can find more details on how the model was generated and trained [here](https://github.com/jay-mahadeokar/pynetbuilder/tree/master/models/imagenet)
We used the thin resnet 50 1by2 architecture as the pretrained network. The model was generated using [pynetbuilder](https://github.com/jay-mahadeokar/pynetbuilder) tool and replicates the [residual network](https://arxiv.org/pdf/1512.03385v1.pdf) paper's 50 layer network (with half number of filters in each layer). You can find more details on how the model was generated and trained [here](https://github.com/jay-mahadeokar/pynetbuilder/tree/master/models/imagenet).

Please note that deeper networks, or networks with more filters can improve accuracy. We train the model using a thin residual network architecture, since it provides good tradeoff in terms of accuracy, and the model is light-weight in terms of runtime (or flops) and memory (or number of parameters).

Expand Down Expand Up @@ -62,7 +62,7 @@ We will get the NSFW score returned:
NSFW score: 0.14057905972
```
#### Running the model
To run this model, please install [Caffe](https://github.com/BVLC/caffe) and its python extension and make sure pycaffe is available in your PYTHONPATH.
To run this model, please install [Caffe](https://github.com/BVLC/caffe) and its python extension. Make sure pycaffe is available in your PYTHONPATH.

We can use the [classify.py](https://github.com/BVLC/caffe/blob/master/python/classify.py) script to run the NSFW model. For convenience, we have provided the script in this repo as well, and it prints the NSFW score.

Expand Down