WebCam ImageNet Recognition

This is a simple web app to show how to use TensorFlow.js to perform image recognition based on a pre-trained NN and input from a WebCam.

The model used is MobileNet pre-trained on ImageNet (ILSVRC-2012-CLS) to recognize 1000 categories of images. FYI, a collection of sample images is shown on the right.

The demo loads the model converted to TensorFlow.js format, which is hosted at Google Cloud Storage (e.g. its weights_manifest.json can be obtained here).

The demo runs on reasonably new Chrome, Safari and Firefox (not on Edge nor IE). It runs on Mobile Safari and Android Chrome with good amount of device's GPU memory (i.e. iPhone 6 and later, Galaxy S7, Pixel 2 XL, etc.). It can take very long time to startup as the pre-trained model is quite large. It fails recognition when seemingly an out-of-GPU-memory situation (see Issues #2 and #3).

You can try it out at: https://tfjs-mobilenet-webcam.netlify.com/

Building and Running

The following commands will start a web server on localhost:8080 and open a browser page with the demo.

cd tfjs-mobilenet-webcam
yarn        # Installs dependencies.
yarn start  # Starts a web server and opens a page. Also watches for changes.

After yarn build, public directory holds the deployable files.

Notes

Although it supposed to distinguish 1000 categories, the categories that ImageNet classifies don't include that many usual everyday items. It may be a bit hard to find many recognizable things around you. Things like 'cellphone', 'water bottle' and 'remote control' usually work well.

Related Repos

If you are interested in knowing how to train NN, check out maru-labo/doodle.
If you are interested in replacing the pre-trained model, this README of a TF.js app included in the repo can be useful.