diff --git a/docs/doc/en/sidebar.yaml b/docs/doc/en/sidebar.yaml index dabc8295..c7ddc47c 100644 --- a/docs/doc/en/sidebar.yaml +++ b/docs/doc/en/sidebar.yaml @@ -31,6 +31,8 @@ items: label: Image control - file: vision/find_blobs.md label: Finding color blocks +- file: vision/line_tracking.md + label: Line tracking - file: vision/qrcode.md label: QRcode identity - file: vision/apriltag.md diff --git a/docs/doc/en/vision/line_tracking.md b/docs/doc/en/vision/line_tracking.md new file mode 100644 index 00000000..6f38c51f --- /dev/null +++ b/docs/doc/en/vision/line_tracking.md @@ -0,0 +1,162 @@ +--- +title: MaixPy Line Tracking +update: + - date: 2024-05-09 + author: lxowalle + version: 1.0.0 + content: Initial document +--- + + +Before reading this article, make sure you already know how to develop MaixPy. For details, please read [MaixVision -- MaixPy Programming + Graphical Block Programming](../basic/maixvision.md). + +## Introduction + +In vision applications, the function of tracking line is often required in applications such as line-following robot. In this article, we will describe: + +- How to use MaixPy to tracking line. + +- How to tracking line using MaixCam's default application + + +## How to use MaixPy to tracking line + +The `maix.image.Image` module in MaixPy provides the `get_regression` method, which can conveniently tracking line. + +### Code example + +A simple example of finding and drawing a line. + +```python +from maix import camera, display, image + +cam = camera.Camera(320, 240) +disp = display.Display() + +# thresholds = [[0, 80, 40, 80, 10, 80]] # red +thresholds = [[0, 80, -120, -10, 0, 30]] # green +# thresholds = [[0, 80, 30, 100, -120, -60]] # blue + +while 1. + img = cam.read() + + lines = img.get_regression(thresholds, area_threshold = 100) + for a in lines. + img.draw_line(a.x1(), a.y1(), a.x2(), a.y2(), image.COLOR_GREEN, 2) + theta = a.theta() + if theta > 90. + theta = 270 - theta + if theta > 90: theta = 270 - theta + theta = 90 - theta + img.draw_string(0, 0, ‘theta: ’ + str(theta), image.COLOR_BLUE) + + disp.show(img) +``` + +Steps: + +1. import image, camera, display modules + + ```python + from maix import image, camera, display + ``` + +2. Initialize camera and display + + ```python + cam = camera.Camera(320, 240) # Initialise camera, output resolution 320x240 in RGB format. + disp = display.Display() + ``` + +3. Get the image from the camera and display it + + ```python + while 1. + img = cam.read() + disp.show(img) + ``` + +4. Call the `get_regression` method to find the straight line in the camera image and draw it to the screen + + ``` python + lines = img.get_regression(thresholds, pixels_threshold = 100) + for a in lines. + img.draw_line(a.x1(), a.y1(), a.x2(), a.y2(), image.COLOR_GREEN, 2) + theta = a.theta() + if theta > 90. + theta = 270 - theta + if theta > 90: theta = 270 - theta + theta = 90 - theta + img.draw_string(0, 0, ‘theta: ’ + str(theta), image.COLOR_BLUE) + ``` + + - `img` is the camera image read via `cam.read()`, when initialised as `cam = camera.Camera(320, 240)`, the `img` object is an RGB image with a resolution of 320x240. + - `img.get_regression` is used to find straight lines, `thresholds` is a list of colour thresholds, each element is a colour threshold, multiple thresholds are passed in if multiple thresholds are found at the same time, and each colour threshold has the format `[L_MIN, L_MAX, A_MIN, A_MAX, B_MIN, B_MAX]`, where ` L`, `A`, `B` are the three channels of `LAB` colour space, `L` channel is the luminance, `A` channel is the red-green channel, `B` channel is the blue-yellow channel. `pixels_threshold` is a pixel area threshold used to filter some unwanted straight lines. + - `for a in lines` is used to iterate through the returned `Line` objects, where `a` is the current `Line` object. Normally the `get_regression` function will only return one `Line` object, but if you need to find more than one line, try the `find_line` method. + - Use `img.draw_line` to draw the found line, `a.x1(), a.y1(), a.x2(), a.y2()` represent the coordinates of the ends of the line. + - Use `img.draw_string` to show the angle between the line and the x-axis in the upper left corner, and `a.theta()` is the angle between the line and the y-axis, which is converted to `theta` for easier understanding. + +5. Run the code through the maixvision, you can find the line, look at the effect! + + ![image-20240509110204007](../../../static/image/line_tracking_demo.jpg) + +### Common Parameter Explanations + +Here are explanations of commonly used parameters. If you cannot find parameters that can implement your application, you may need to consider using other algorithms or extending the required functionality based on the current algorithm's results. + +| Parameter | Description | Example | +| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| thresholds | Thresholds based on the LAB color space, thresholds=[[l_min, l_max, a_min, a_max, b_min, b_max]], representing:
Brightness range [l_min, l_max]
Green to red component range [a_min, a_max]
Blue to yellow component range [b_min, b_max]
Multiple thresholds can be set simultaneously | Set two thresholds to detect red and green
```img.find_blobs(thresholds=[[0, 80, 40, 80, 10, 80], [0, 80, -120, -10, 0, 30]])```
Red threshold is [0, 80, 40, 80, 10, 80]
Green threshold is [0, 80, -120, -10, 0, 30] | +| invert | Enable threshold inversion, when enabled, the passed thresholds are inverted. Default is False. | Enable threshold inversion
```img.find_blobs(invert=True)``` | +| roi | Set the rectangular region for the algorithm to compute, roi=[x, y, w, h], where x and y represent the coordinates of the top-left corner of the rectangle, and w and h represent the width and height of the rectangle, respectively. The default is the entire image. | Compute the region at (50, 50) with a width and height of 100
```img.find_blobs(roi=[50, 50, 100, 100])``` | +| area_threshold | Filter out blobs with a pixel area smaller than area_threshold, in units of pixels. The default is 10. This parameter can be used to filter out some useless small blobs. | Filter out blobs with an area smaller than 1000
```img.find_blobs(area_threshold=1000)``` | +| pixels_threshold | Filter out blobs with fewer valid pixels than pixels_threshold. The default is 10. This parameter can be used to filter out some useless small blobs. | Filter out blobs with fewer than 1000 valid pixels
```img.find_blobs(pixels_threshold=1000)``` | + +This article introduces commonly used methods. For more APIs, please see the [image](../../../api/maix/image.md) section of the API documentation. + +## How to tracking line using MaixCam's default application + +To quickly verify the line tracking functionality, you can use the `line_tracking` application provided by MaixCam to experience the line finding effect. + +### How to use it + +1. Select and open the `Line tracking` application. +2. Click on the line in the screen that needs to be identified and the colour of the line will be displayed on the left hand side +3. Click on the colour to be detected on the left (the colour below L A B in the screen) +4. The line will be identified and the coordinates and angle of the line will be output from the serial port. + +### Demo + + +### Advanced operations + +#### Manual adjustment of LAB threshold to tracking line + +The application provides manual setting of LAB threshold to tracking line accurately. + +Steps: + +1. `Click` the `options icon` in the bottom-left corner to enter configuration mode. +2. Point the `camera` at the `object` you need to `find`, `click` on the `target object` on the screen, and the `left side` will display a `rectangular frame` of the object's color and show the `LAB values` of that color. +3. Click on the bottom options `L Min`, `L Max`, `A Min`, `A Max`, `B Min`, `B Max`. After clicking, a slider will appear on the right side to set the value for that option. These values correspond to the minimum and maximum values of the L, A, and B channels in the LAB color format, respectively. +4. Referring to the `LAB values` of the object color calculated in step 2, adjust `L Min`, `L Max`, `A Min`, `A Max`, `B Min`, `B Max` to appropriate values to identify the corresponding color blobs. For example, if `LAB = (20, 50, 80)`, since `L=20`, to accommodate a certain range, set `L Min=10` and `L Max=30`. Similarly, since `A=50`, set `A Min=40` and `A Max=60`. Since `B=80`, set `B Min=70` and `B Max=90`. + +#### Getting Detection Data via Serial Protocol + +The line tracking application supports reporting detected straight line information via the serial port (default baud rate is 115200). + +Since only one report message is sent, we can illustrate the content of the report message with an example. + +For instance, if the report message is: + +```shell +AA CA AC BB 0E 00 00 00 00 E1 09 FC 01 01 00 E9 01 6F 01 57 00 C1 C6 +``` + +- `AA CA AC BB`: Protocol header, fixed content +- `0E 00 00 00`: Data length, the total length excluding the protocol header and data length, here means the length is 14. +- `E1`: Flag bit, used to identify the serial message flag +- `09`: Command type, for the line tracking application, this value is fixed at 0x09. + +- `FC 01 01 00 E9 01 6F 01 57 00`: The coordinates and angle information for both ends of line, with each value represented as a 2-byte value in little-end format. `FC 01` and `01 00` indicate that the coordinates of the first endpoint are (508, 1), `E9 01` and `6F 01` indicate that the coordinates of the second endpoint are (489, 367), and `57 00` indicates that the angle of the line to the x-axis is 87 degrees +- `C1 C6`: CRC checksum value, used to verify if the frame data has errors during transmission. \ No newline at end of file diff --git a/docs/doc/zh/sidebar.yaml b/docs/doc/zh/sidebar.yaml index 3c0a1185..def1a290 100644 --- a/docs/doc/zh/sidebar.yaml +++ b/docs/doc/zh/sidebar.yaml @@ -31,6 +31,8 @@ items: label: 基本图像操作 - file: vision/find_blobs.md label: 寻找色块 +- file: vision/line_tracking.md + label: 寻找直线 - file: vision/qrcode.md label: 二维码识别 - file: vision/apriltag.md diff --git a/docs/doc/zh/vision/line_tracking.md b/docs/doc/zh/vision/line_tracking.md new file mode 100644 index 00000000..342f2bd9 --- /dev/null +++ b/docs/doc/zh/vision/line_tracking.md @@ -0,0 +1,170 @@ +--- +title: MaixPy 寻找直线 +update: + - date: 2024-05-09 + author: lxowalle + version: 1.0.0 + content: 初版文档 +--- + +阅读本文前，确保已经知晓如何开发MaixPy，详情请阅读[MaixVision -- MaixPy 编程 + 图形化积木编程](../basic/maixvision.md) + +## 简介 + +在视觉应用中，在巡迹小车、巡线机器人等应用中经常需要寻找线条的功能。本文将介绍: + +- 如何使用MaixPy来寻找直线 + +- 如何使用MaixCam的默认应用程序寻找直线 + + +## 如何使用MaixPy来寻找直线 + +MaixPy的 `maix.image.Image`中提供了`get_regression`方法来寻找直线 + +### 代码示例 + +一个简单的示例，实现寻找并画出直线 + +```python +from maix import camera, display, image + +cam = camera.Camera(320, 240) +disp = display.Display() + +# thresholds = [[0, 80, 40, 80, 10, 80]] # red +thresholds = [[0, 80, -120, -10, 0, 30]] # green +# thresholds = [[0, 80, 30, 100, -120, -60]] # blue + +while 1: + img = cam.read() + + lines = img.get_regression(thresholds, area_threshold = 100) + for a in lines: + img.draw_line(a.x1(), a.y1(), a.x2(), a.y2(), image.COLOR_GREEN, 2) + theta = a.theta() + if theta > 90: + theta = 270 - theta + else: + theta = 90 - theta + img.draw_string(0, 0, "theta: " + str(theta), image.COLOR_BLUE) + + disp.show(img) + +``` + +步骤： + +1. 导入image、camera、display模块 + + ```python + from maix import image, camera, display + ``` + +2. 初始化摄像头和显示 + + ```python + cam = camera.Camera(320, 240) # 初始化摄像头，输出分辨率320x240 RGB格式 + disp = display.Display() + ``` + +3. 从摄像头获取图片并显示 + + ```python + while 1: + img = cam.read() + disp.show(img) + ``` + +4. 调用`get_regression`方法寻找摄像头图片中的直线，并画到屏幕上 + + ```python + lines = img.get_regression(thresholds, pixels_threshold = 100) + for a in lines: + img.draw_line(a.x1(), a.y1(), a.x2(), a.y2(), image.COLOR_GREEN, 2) + theta = a.theta() + if theta > 90: + theta = 270 - theta + else: + theta = 90 - theta + img.draw_string(0, 0, "theta: " + str(theta), image.COLOR_BLUE) + ``` + + - `img`是通过`cam.read()`读取到的摄像头图像，当初始化的方式为`cam = camera.Camera(320, 240)`时，`img`对象是一张分辨率为320x240的RGB图。 + - `img.get_regression`用来寻找直线， `thresholds` 是一个颜色阈值列表，每个元素是一个颜色阈值，同时找到多个阈值就传入多个，每个颜色阈值的格式为 `[L_MIN, L_MAX, A_MIN, A_MAX, B_MIN, B_MAX]`，这里的 `L`、`A`、`B` 是`LAB`颜色空间的三个通道，`L` 通道是亮度，`A` 通道是红绿通道，`B` 通道是蓝黄通道。`pixels_threshold`是一个像素面积的阈值，用来过滤一些不需要直线。 + - `for a in lines`用来遍历返回的`Line`对象， 其中`a`就是当前的`Line`对象。通常`get_regression`函数只会返回一个`Line`对象，如果需要寻找多条直线，可以尝试使用`find_line`方法 + - 使用`img.draw_line`来画出找到的线条，`a.x1(), a.y1(), a.x2(), a.y2()`分别代表直线两端的坐标 + - 使用`img.draw_string`在左上角显示直线与x轴的夹角， `a.theta()`是直线与y轴的夹角， 这里为了方便理解转换成直线与x轴的夹角`theta` + +5. 通过maixvision运行代码，就可以寻线啦，看看效果吧 + + ![image-20240509110204007](../../../static/image/line_tracking_demo.jpg) + +### 常用参数说明 + +列举常用参数说明，如果没有找到可以实现应用的参数，则需要考虑是否使用其他算法实现，或者基于目前算法的结果扩展所需的功能 + +| 参数 | 说明 | 示例 | +| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| thresholds | 基于lab颜色空间的阈值，threshold=[[l_min, l_max, a_min, a_max, b_min, b_max]]，分别表示：
亮度范围为[l_min, l_max]\|
绿色到红色的分量范围为[l_min, l_max]
蓝色到黄色的分量范围为[b_min, b_max]
可同时设置多个阈值 | 设置两个阈值来检测红色和绿色
```img.find_blobs(threshold=[[0, 80, 40, 80, 10, 80], [0, 80, -120, -10, 0, 30]])```
红色阈值为[0, 80, 40, 80, 10, 80]
绿色阈值为[0, 80, -120, -10, 0, 30] | +| invert | 使能阈值反转，使能后传入阈值与实际阈值相反，默认为False | 使能阈值反转
```img.find_blobs(invert=True)``` | +| roi | 设置算法计算的矩形区域，roi=[x, y, w, h]，x，y表示矩形区域左上角坐标，w，h表示矩形区域的宽度和高度，默认为整张图片 | 计算坐标为(50,50)，宽和高为100的区域
```img.find_blobs(roi=[50, 50, 100, 100])``` | +| area_threshold | 过滤像素面积小于area_threshold的直线，单位为像素点，默认为10。该参数可用于过滤一些无用的小直线 | 过滤面积小于1000的直线
```img.find_blobs(area_threshold=1000)``` | +| pixels_threshold | 过滤有效像素点小于pixels_threshold的直线，默认为10。该参数可用于过滤一些无用的小直线 | 过滤有效像素点小于1000的直线
```img.find_blobs(pixels_threshold=1000)``` | + +本文介绍常用方法，更多 API 请看 API 文档的 [image](../../../api/maix/image.md) 部分。 + +## 如何使用MaixCam的默认应用程序寻找直线 + +为了快速验证寻找直线的功能，可以先使用MaixCam提供的`line_tracking`应用程序来体验寻找直线的效果。 + +### 使用方法 +1. 选择并打开`Line tracking`应用 +2. 点击屏幕中需要识别的直线，左侧会显示该直线的颜色 +3. 点击左侧（界面中`L A B`下方的颜色）需要检测的颜色 +4. 此时就可以识别到对应的直线了，同时串口也会输出直线的坐标和角度信息。 + +### 演示 + + + +### 进阶操作 + +#### 手动设置LAB阈值寻找直线 + +APP提供手动设置LAB阈值来精确的寻找直线 + +操作方法： + +1. `点击`左下角`选项图标`，进入配置模式 + +2. 将`摄像头对准`需要`寻找的物体`，`点击`屏幕上的`目标直线`，此时界面中`L A B`下方会显示该物体对应颜色的`矩形框`，并显示该物体颜色的`LAB值`。 + +3. 点击下方选项`L Min，L Max，A Min，A Max，B Min，B Max`，点击后右侧会出现滑动条来设置该选项值。这些值分别对应LAB颜色格式的L通道、A通道和B通道的最小值和最大值 + +4. 参考步骤2计算的物体颜色的`LAB值`，将`L Min，L Max，A Min，A Max，B Min，B Max`调整到合适的值，即可识别到对应的直线。 + + 例如`LAB=(20, 50, 80)`，由于`L=20`，为了适配一定范围让`L Min=10`，`L Max=30`;同理，由于`A=50`，让`A Min=40`，`A Max=60`; 由于`B=80`，让`B Min=70`，`B Max=90`。 + +#### 通过串口协议获取检测数据 + +寻找直线应用支持通过串口（默认波特率为115200）上报检测到的直线信息。 + +由于上报信息只有一条，这里直接用示例来说明上报信息的内容。 + +例如上报信息为： + +```shell +AA CA AC BB 0E 00 00 00 E1 09 FC 01 01 00 E9 01 6F 01 57 00 C1 C6 +``` + +- `AA CA AC BB`：协议头部，内容固定 +- `0E 00 00 00`：数据长度，除了协议头部和数据长度外的总长度，这里表示长度为14 +- `E1`：标志位，用来标识串口消息标志 +- `09`：命令类型，对于寻找直线APP应用该值固定为0x09 +- `FC 01 01 00 E9 01 6F 01 57 00`：直线的两端坐标和角度信息，每个值用小端格式的2字节表示。`FC 01`和`01 00`表示第一个端点坐标为(508, 1)，`E9 01`和`6F 01`表示第二个端点坐标为(489, 367)，`57 00`表示直线与x轴的角度为87度 + +- ` C1 C6`：CRC 校验值，用以校验帧数据在传输过程中是否出错 + + + diff --git a/docs/static/image/line_tracking_demo.jpg b/docs/static/image/line_tracking_demo.jpg new file mode 100644 index 00000000..3bd95870 Binary files /dev/null and b/docs/static/image/line_tracking_demo.jpg differ diff --git a/docs/static/video/line_tracking_app.mp4 b/docs/static/video/line_tracking_app.mp4 new file mode 100644 index 00000000..09f1c51b Binary files /dev/null and b/docs/static/video/line_tracking_app.mp4 differ diff --git a/examples/vision/image_basic/line_tracking.py b/examples/vision/image_basic/line_tracking.py new file mode 100644 index 00000000..fbabae35 --- /dev/null +++ b/examples/vision/image_basic/line_tracking.py @@ -0,0 +1,23 @@ +from maix import camera, display, image + +cam = camera.Camera(320, 240) +disp = display.Display() + +# thresholds = [[0, 80, 40, 80, 10, 80]] # red +thresholds = [[0, 80, -120, -10, 0, 30]] # green +# thresholds = [[0, 80, 30, 100, -120, -60]] # blue + +while 1: + img = cam.read() + + lines = img.get_regression(thresholds, area_threshold = 100) + for a in lines: + img.draw_line(a.x1(), a.y1(), a.x2(), a.y2(), image.COLOR_GREEN, 2) + theta = a.theta() + if theta > 90: + theta = 270 - theta + else: + theta = 90 - theta + img.draw_string(0, 0, "theta: " + str(theta), image.COLOR_BLUE) + + disp.show(img)