Skip to content
This repository has been archived by the owner on Mar 19, 2023. It is now read-only.

Commit

Permalink
Merge pull request #155 from robmarkcole/support-custom-model
Browse files Browse the repository at this point in the history 
Support custom model
  • Loading branch information
@robmarkcole
robmarkcole authored Dec 17, 2020
2 parents 224227b + bb64285 commit 0816256
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 55 deletions.
72 changes: 18 additions & 54 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,46 +1,15 @@
# HASS-Deepstack-object
[Home Assistant](https://www.home-assistant.io/) custom component for Deepstack object detection. [Deepstack](https://python.deepstack.cc/) is a service which runs in a docker container and exposes deep-learning models via a REST API. Deepstack [object detection](https://python.deepstack.cc/object-detection) uses [Yolo V3](https://pjreddie.com/darknet/yolo/) to identify 80 different kinds of objects (listed at bottom of this readme), including people (`person`) and animals. There is no cost for using Deepstack, although you will need a machine with 8 GB RAM. On your machine with docker, pull the latest image (approx. 2GB):
[Home Assistant](https://www.home-assistant.io/) custom component for Deepstack object detection. [Deepstack](https://deepstack.cc) is a service which runs in a docker container and exposes various computer vision models via a REST API. Deepstack object detection can identify 80 different kinds of objects (listed at bottom of this readme), including people (`person`), vehicles and animals. Alternatively a custom object detection model can be used. There is no cost for using Deepstack and it is [fully open source](https://github.com/johnolafenwa/DeepStack). To run Deepstack you will need a machine with 8 GB RAM, or an NVIDIA Jetson.

On your machine with docker, run Deepstack with the object detection service active on port `80`:
```
docker pull deepquestai/deepstack
```
OR, if you are using a legacy machine:
```
docker pull deepquestai/deepstack:noavx
```
**Recommended OS** Deepstack docker containers are optimised for Linux or Windows 10 Pro. Mac and regular windows users my experience performance issues. [You can also run deepstack on a Raspberry pi](https://python.deepstack.cc/raspberry-pi) if you own an Intel NCS (Movidius) stick (approx $70).

**GPU users** Note that if your machine has an Nvidia GPU you can get a 5 x 20 times performance boost by using the GPU.

**Legacy machine users** If you are using a machine that doesn't support avx or you are having issues with making requests, Deepstack has a specific build for these systems. Use `deepquestai/deepstack:noavx` instead of `deepquestai/deepstack` when you are installing or running Deepstack. I expect many users will be using noavx mode so I will use it in the examples below.

## Activating the Deepstack API
Before you get started, you will need to activate the Deepstack API. First, go to www.deepstack.cc and sign up for an account. Choose the basic plan which will give us unlimited access for one installation. You will then see an activation key in your portal.

On your machine with docker, run Deepstack (noavx mode) without any recognition so you can activate the API on port `5000`:
```
docker run -v localstorage:/datastore -p 5000:5000 deepquestai/deepstack:noavx
```

Now go to http://YOUR_SERVER_IP_ADDRESS:5000/ on another computer or the same one running Deepstack. Input your activation key from your portal into the text box below "Enter New Activation Key" and press enter. Now stop your docker container, and restart and run Deepstack (noavx mode) with the object detection service active on port `5000`:
```
docker run -e VISION-DETECTION=True -e API-KEY="mysecretkey" -v localstorage:/datastore -p 5000:5000 --name deepstack deepquestai/deepstack:noavx
```
You can test the endpoint is active using [curl](https://curl.haxx.se/), from within a directory containing an image `test.jpg`:
```
curl -X POST -F [email protected] 'http://localhost:5000/v1/vision/detection'
```
Which should return something like:
```json
{"success":true,
"predictions":[
{"confidence":0.99967474,"label":"person","y_min":25,"x_min":1184,"y_max":2692,"x_max":2277},
{"confidence":0.997645,"label":"person","y_min":148,"x_min":460,"y_max":2655,"x_max":1348},
{"confidence":0.9949693,"label":"tie","y_min":757,"x_min":1731,"y_max":1407,"x_max":1857}]}
docker run -e VISION-DETECTION=True -e API-KEY="mysecretkey" -v localstorage:/datastore -p 80:5000 deepquestai/deepstack
```

## Usage of this component
The `deepstack_object` component adds an `image_processing` entity where the state of the entity is the total count of target objects that are above a `confidence` threshold which has a default value of 80%. You can have a single target object class, or multiple. The time of the last detection of any target object is in the `last target detection` attribute. The type and number of objects (of any confidence) is listed in the `summary` attributes. Optionally a region of interest (ROI) can be configured, and only objects with their center (represented by a `x`) within the ROI will be included in the state count. The ROI will be displayed as a green box, and objects with their center in the ROI have a red box, whilst objects with their center outside the ROI have a yellow box. Also optionally the processed image can be saved to disk, with bounding boxes showing the location of detected objects. If `save_file_folder` is configured, an image with filename of format `deepstack_object_{source name}_latest.jpg` is over-written on each new detection of a target. Optionally this image can also be saved with a timestamp in the filename, if `save_timestamped_file` is configured as `True`. An event `deepstack.object_detected` is fired for each object detected. If you are a power user with advanced needs such as zoning detections or you want to track multiple object types, you will need to use the `deepstack.object_detected` events.
The `deepstack_object` component adds an `image_processing` entity where the state of the entity is the total count of target objects that are above a `confidence` threshold which has a default value of 80%. You can have a single target object class, or multiple. The time of the last detection of any target object is in the `last target detection` attribute. The type and number of objects (of any confidence) is listed in the `summary` attributes. Optionally a region of interest (ROI) can be configured, and only objects with their center (represented by a `x`) within the ROI will be included in the state count. The ROI will be displayed as a green box, and objects with their center in the ROI have a red box, whilst objects with their center outside the ROI have a yellow box.

Also optionally the processed image can be saved to disk, with bounding boxes showing the location of detected objects. If `save_file_folder` is configured, an image with filename of format `deepstack_object_{source name}_latest.jpg` is over-written on each new detection of a target. Optionally this image can also be saved with a timestamp in the filename, if `save_timestamped_file` is configured as `True`. An event `deepstack.object_detected` is fired for each object detected. If you are a power user with advanced needs such as zoning detections or you want to track multiple object types, you will need to use the `deepstack.object_detected` events.

**Note** that by default the component will **not** automatically scan images, but requires you to call the `image_processing.scan` service e.g. using an automation triggered by motion.

Expand All @@ -55,8 +24,9 @@ Add to your Home-Assistant config:
image_processing:
- platform: deepstack_object
ip_address: localhost
port: 5000
port: 80
api_key: mysecretkey
# custom_model: mask
save_file_folder: /config/snapshots/
save_timestamped_file: True
# roi_x_min: 0.35
Expand All @@ -66,6 +36,7 @@ image_processing:
targets:
- person
- car
- mask
source:
- entity_id: camera.local_file
```
Expand All @@ -75,6 +46,7 @@ Configuration variables:
- **port**: the port of your deepstack instance.
- **api_key**: (Optional) Any API key you have set.
- **timeout**: (Optional, default 10 seconds) The timeout for requests to deepstack.
- **custom_model**: (Optional) The name of a custom model if you are using one. Don't forget to add the targets from the custom model below
- **save_file_folder**: (Optional) The folder to save processed images to. Note that folder path should be added to [whitelist_external_dirs](https://www.home-assistant.io/docs/configuration/basic/)
- **save_timestamped_file**: (Optional, default `False`, requires `save_file_folder` to be configured) Save the processed image with the time of detection in the filename.
- **show_boxes**: (optional, default `True`), if `False` bounding boxes are not shown on saved images
Expand Down Expand Up @@ -132,7 +104,6 @@ The `box` coordinates and the box center (`centroid`) can be used to determine w
* The `box` is defined by the tuple `(y_min, x_min, y_max, x_max)` (equivalent to image top, left, bottom, right) where the coordinates are floats in the range `[0.0, 1.0]` and relative to the width and height of the image.
* The centroid is in `(x,y)` coordinates where `(0,0)` is the top left hand corner of the image and `(1,1)` is the bottom right corner of the image.


## Displaying the deepstack latest jpg file
It easy to display the `deepstack_object_{source name}_latest.jpg` image with a [local_file](https://www.home-assistant.io/components/local_file/) camera. An example configuration is:
```yaml
Expand Down Expand Up @@ -182,33 +153,26 @@ A1: Yes this is normal

------

Q2: Will Deepstack always be free, if so how do these guys make a living?

A2: Enterprise support pays the developers salaries.

------

Q3: What are the minimum hardware requirements for running Deepstack?
Q4: What are the minimum hardware requirements for running Deepstack?

A3. Based on my experience, I would allow 0.5 GB RAM per model.
A4. Based on my experience, I would allow 0.5 GB RAM per model.

------

Q4: Can object detection be configured to detect car/car colour?
Q5: Can object detection be configured to detect car/car colour?

A4: The list of detected object classes is at the end of the page [here](https://deepstackpython.readthedocs.io/en/latest/objectdetection.html). There is no support for detecting the colour of an object.
A5: The list of detected object classes is at the end of the page [here](https://deepstackpython.readthedocs.io/en/latest/objectdetection.html). There is no support for detecting the colour of an object.

------

Q5: I am getting an error from Home Assistant: `Platform error: image_processing - Integration deepstack_object not found`
Q6: I am getting an error from Home Assistant: `Platform error: image_processing - Integration deepstack_object not found`

A5: This can happen when you are running in Docker/Hassio, and indicates that one of the dependencies isn't installed. It is necessary to reboot your Hassio device, or rebuild your Docker container. Note that just restarting Home Assistant will not resolve this.
A6: This can happen when you are running in Docker/Hassio, and indicates that one of the dependencies isn't installed. It is necessary to reboot your Hassio device, or rebuild your Docker container. Note that just restarting Home Assistant will not resolve this.

------

## Objects list
The
following list is [from the deepstack docs](https://python.deepstack.cc/object-detection):
The following list is [from the deepstack docs](https://python.deepstack.cc/object-detection):
```
person, bicycle, car, motorcycle, airplane,
bus, train, truck, boat, traffic light, fire hydrant, stop_sign,
Expand All @@ -217,7 +181,7 @@ bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase,
frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove,
skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork,
knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot,
hot dog, pizza, donot, cake, chair, couch, potted plant, bed, dining table,
hot dog, pizza, donut, cake, chair, couch, potted plant, bed, dining table,
toilet, tv, laptop, mouse, remote, keyboard, cell phone, microwave,
oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear,
hair dryer, toothbrush.
Expand Down
12 changes: 11 additions & 1 deletion custom_components/deepstack_object/image_processing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
CONF_ROI_X_MIN = "roi_x_min"
CONF_ROI_Y_MAX = "roi_y_max"
CONF_ROI_X_MAX = "roi_x_max"
CONF_CUSTOM_MODEL = "custom_model"

DATETIME_FORMAT = "%Y-%m-%d_%H-%M-%S"
DEFAULT_API_KEY = ""
Expand Down Expand Up @@ -86,6 +87,7 @@
vol.Required(CONF_PORT): cv.port,
vol.Optional(CONF_API_KEY, default=DEFAULT_API_KEY): cv.string,
vol.Optional(CONF_TIMEOUT, default=DEFAULT_TIMEOUT): cv.positive_int,
vol.Optional(CONF_CUSTOM_MODEL, default=""): cv.string,
vol.Optional(CONF_TARGETS, default=DEFAULT_TARGETS): vol.All(
cv.ensure_list, [cv.string]
),
Expand Down Expand Up @@ -170,6 +172,7 @@ def setup_platform(hass, config, add_devices, discovery_info=None):
config.get(CONF_PORT),
config.get(CONF_API_KEY),
config.get(CONF_TIMEOUT),
config.get(CONF_CUSTOM_MODEL),
targets,
config.get(ATTR_CONFIDENCE),
config[CONF_ROI_Y_MIN],
Expand All @@ -195,6 +198,7 @@ def __init__(
port,
api_key,
timeout,
custom_model,
targets,
confidence,
roi_y_min,
Expand All @@ -209,7 +213,10 @@ def __init__(
):
"""Init with the API key and model id."""
super().__init__()
self._dsobject = ds.DeepstackObject(ip_address, port, api_key, timeout)
self._dsobject = ds.DeepstackObject(
ip_address, port, api_key, timeout, custom_model
)
self._custom_model = custom_model
self._targets = targets
self._confidence = confidence
self._camera = camera_entity
Expand Down Expand Up @@ -317,6 +324,9 @@ def device_state_attributes(self):
)
if self._last_detection:
attr["last_target_detection"] = self._last_detection
if self._custom_model:
attr["custom_model"] = self._custom_model
attr["targets"] = self._targets
attr["summary"] = self._summary
attr["objects"] = self._objects
return attr
Expand Down

0 comments on commit 0816256

Please sign in to comment.