diff --git a/docs/doc/assets/self_learn_classifier.jpg b/docs/doc/assets/self_learn_classifier.jpg new file mode 100644 index 00000000..a401b9ef Binary files /dev/null and b/docs/doc/assets/self_learn_classifier.jpg differ diff --git a/docs/doc/en/basic/maixpy_upgrade.md b/docs/doc/en/basic/maixpy_upgrade.md index c7ba3598..173b6f51 100644 --- a/docs/doc/en/basic/maixpy_upgrade.md +++ b/docs/doc/en/basic/maixpy_upgrade.md @@ -19,5 +19,16 @@ If you prefer not to update the system (since system changes are usually minimal * Set up WiFi in the settings to connect the system to the internet. * Click on `Update MaixPy` in the settings app to proceed with the update. + +You can also execute Python code to call system command to install: +```python +import os + +os.system("pip install MaixPy -U") +``` + > If you are comfortable using the terminal, you can also update MaixPy by using `pip install MaixPy -U` in the terminal. + + +And you can download `wheel` file (`.whl`format) manually, and send to device(transfer method see [MaixVision Usage](./maixvision.md)), then install by `pip install *****.whl` command. diff --git a/docs/doc/en/vision/image_ops.md b/docs/doc/en/vision/image_ops.md index 589b9d67..a00e89b7 100644 --- a/docs/doc/en/vision/image_ops.md +++ b/docs/doc/en/vision/image_ops.md @@ -64,6 +64,8 @@ MaixPy provides the `maix.image.load` method, which can read images from the fil from maix import image img = image.load("/root/image.jpg") +if img is None: + raise Exception(f"load image failed") print(img) ``` diff --git a/docs/doc/en/vision/self_learn_classifier.md b/docs/doc/en/vision/self_learn_classifier.md index 22e8daf9..fd4f5bc5 100644 --- a/docs/doc/en/vision/self_learn_classifier.md +++ b/docs/doc/en/vision/self_learn_classifier.md @@ -4,20 +4,24 @@ title: MaixPy Self-Learning Classifier ## Introduction to MaixPy Self-Learning Classifier -Typically, to recognize new categories, it is necessary to collect a new dataset and train on a computer, which can be cumbersome and complex. This method eliminates the need for computer-based training, allowing for immediate learning of new objects directly on the device, suitable for less complex scenarios. +Usually, to recognize new categories, we need to collect a dataset on a computer and retrain the model, which is a cumbersome and difficult process. Here, we provide a method that allows for instant learning of new objects directly on the device without the need for computer-side training, suitable for less complex scenarios. -For example, if there are a drink bottle and a mobile phone in front of you, take a photo of each to serve as the basis for two categories. Then, collect several photos from different angles of each item, extract their features and save them. During recognition, the image's features are compared with the saved feature values, and the closest match determines the classification. +For example, if there is a bottle and a phone in front of you, you can use the device to take a picture of each as the basis for two classifications. Then, you collect a few more pictures of them from different angles, extract their features and save them. During recognition, the feature values of the image are compared with the saved feature values, and the classification that is more similar to the saved features is considered the corresponding classification. ## Using the Self-Learning Classifier in MaixPy -Steps: +The default image comes with the [Self-Learning Classification APP](https://maixhub.com/app/30), which you can use directly to get familiar with the process. + +![](../../assets/self_learn_classifier.jpg) -* Collect n classification images. -* Collect n*m images, m images for each category, order does not matter. -* Start learning. -* Recognize images and output results. +Steps: +* Click the `+ Class` button to collect n classification (class) images. The object needs to be within the white frame on the screen while collecting the images. +* Click the `+ Sample` button to collect m sample images. Collect some images for each classification. The order does not matter, and the number is flexible. It's best to take pictures from different angles, but not too different. +* Click the `Learn` button to start learning. The device will automatically classify and learn based on the collected classification and sample images, obtaining the characteristics of the classifications. +* Align the object with the center of the screen, recognize the image, and output the result. The screen will show the classification it belongs to and the similarity distance to this classification. The closer the similarity distance, the more similar it is. +* The feature values ​​learned by this APP will be saved to `/root/my_classes.bin`, so the last one will be automatically loaded after exiting the application or restarting it. -Simplified version of the code, for the full version please refer to the complete code in the example. +Simplified version of the code, for the complete version, please refer to the [examples](https://github.com/sipeed/maixpy/tree/main/examples/vision/ai_vision) for the full code. ```python from maix import nn, image @@ -34,7 +38,6 @@ sample_4 = image.load("/root/sample_4.jpg") sample_5 = image.load("/root/sample_5.jpg") sample_6 = image.load("/root/sample_6.jpg") - classifier.add_class(img1) classifier.add_class(img2) classifier.add_class(img3) @@ -51,3 +54,20 @@ img = image.load("/root/test.jpg") max_idx, max_score = classifier.classify(img) print(max_idx, max_score) ``` + +## Storing and Loading Learned Feature Values + +Use the `save` function to store the learned feature values. This will generate a binary file containing the feature values of the objects. When you need to use it again, simply use the `load` function to load the feature values. + +```python +classifier.save("/root/my_classes.bin") +classifier.load("/root/my_classes.bin") +``` + +If you have named each classification and stored them in the `labels` variable, you can also use: + +```python +classifier.save("/root/my_classes.bin", labels=labels) +labels = classifier.load("/root/my_classes.bin") +``` + diff --git a/docs/doc/zh/basic/maixpy_upgrade.md b/docs/doc/zh/basic/maixpy_upgrade.md index 99913e52..1628a14f 100644 --- a/docs/doc/zh/basic/maixpy_upgrade.md +++ b/docs/doc/zh/basic/maixpy_upgrade.md @@ -20,6 +20,14 @@ title: 更新 MaixPy * 在设置中设置 WiFi, 让系统联网。 * 点击设置应用中的 `更新 MaixPy` 进行更新。 +也可以执行 Python 代码调用系统命令来更新: +```python +import os -> 如果你会使用终端, 也可以在终端中使用 `pip install MaixPy -U` 来更新 MaixPy。 +os.system("pip install MaixPy -U") +``` + +> 如果你会使用终端, 也可以直接在终端中使用 `pip install MaixPy -U` 来更新 MaixPy。 + +另外你也可以手动下载`wheel` 文件(`.whl`格式)传输到设备(传输方法见后文[MaixVision 使用](./maixvision.md))后通过 `pip install ******.whl` 命令来安装。 diff --git a/docs/doc/zh/vision/image_ops.md b/docs/doc/zh/vision/image_ops.md index 1b176075..4bc3869c 100644 --- a/docs/doc/zh/vision/image_ops.md +++ b/docs/doc/zh/vision/image_ops.md @@ -64,6 +64,8 @@ MaixPy 提供了`maix.image.load`方法,可以从文件系统读取图像: from maix import image img = image.load("/root/image.jpg") +if img is None: + raise Exception(f"load image failed") print(img) ``` diff --git a/docs/doc/zh/vision/self_learn_classifier.md b/docs/doc/zh/vision/self_learn_classifier.md index 87a9ba43..a5c0555f 100644 --- a/docs/doc/zh/vision/self_learn_classifier.md +++ b/docs/doc/zh/vision/self_learn_classifier.md @@ -12,14 +12,19 @@ title: MaixPy 自学习分类器 ## MaixPy 中使用自学习分类器 +默认镜像自带了 [自学习分类 APP](https://maixhub.com/app/30),可以直接尝试使用熟悉使用流程。 + +![](../../assets/self_learn_classifier.jpg) + 步骤: -* 采集 n 张分类图。 -* 采集 n*m 张图,每个分类采集 m 张,顺序无所谓。 -* 启动学习。 -* 识别图像输出结果。 +* 点击`+ Class` 按钮, 采集 n 张分类(class)图,采集图时物体需要在屏幕的白色框中。 +* 点击`+ Sample`按钮,采集 m 张样本图,每个分类都采集一些,顺序无所谓,张数也比较随意,最好是在各个角度拍一点,不要差距过大。 +* 点击`Learn`按钮,启动学习,会自动根据采集的分类图和样本图进行分类学习,得到分类的特征。 +* 屏幕中央对准物体,识别图像输出结果,可以看到屏幕显示了所属的分类,以及和这个分类的相似距离,相似距离越近则越相似。 +* 此 APP 学习后的特征值会存到`/root/my_classes.bin`,所以退出应用或者重启了仍然会自动加载上一次的。 +简洁版本代码,完整版本请看[例程](https://github.com/sipeed/maixpy/tree/main/examples/vision/ai_vision)里面的完整代码。 -简洁版本代码,完整版本请看例程里面的完整代码。 ```python from maix import nn, image @@ -53,5 +58,20 @@ max_idx, max_score = classifier.classify(img) print(maix_idx, max_score) ``` +## 储存和加载学习到的特征值 + +使用 `save` 函数进行储存,会得到一个二进制文件,里面存了物体的特征值。 +再使用时用`load`函数进行加载即可。 + +```python +classifier.save("/root/my_classes.bin") +classifier.load("/root/my_classes.bin") +``` + +如果你给每一个分类命名了,比如存到了`labels`变量,也可以使用: +```python +classifier.save("/root/my_classes.bin", labels = labels) +labels = classifier.load("/root/my_classes.bin") +``` diff --git a/examples/vision/ai_vision/nn_self_learn_classifier.py b/examples/vision/ai_vision/nn_self_learn_classifier.py new file mode 100644 index 00000000..f3f1f47e --- /dev/null +++ b/examples/vision/ai_vision/nn_self_learn_classifier.py @@ -0,0 +1,64 @@ +from maix import nn, image, display, app, time + +disp = display.Display() +classifier = nn.SelfLearnClassifier(model="/root/models/mobilenet_v2_no_top.mud") + +class_path = [ + "/root/1.jpg", + "/root/2.jpg", + "/root/3.jpg" +] + +samples_path = [ + "/root/sample_1.jpg", + "/root/sample_2.jpg" +] + + +class_images = [] +for path in class_path: + # load image from file + img = image.load(path) + if img is None: + raise Exception(f"load image {path} failed") + class_images.append(img) + # add new class + classifier.add_class(img) + +sample_images = [] +for path in samples_path: + # load image from file + img = image.load(path) + if img is None: + raise Exception(f"load image {path} failed") + sample_images.append(img) + # add new class + classifier.add_sample(img) + +if len(sample_images) > 0: + print("-- start learn") + classifier.learn() + print("-- learn complete") + +classifier.learn() + +img = image.load("/root/test.jpg") +result = classifier.classify(img) +print(f"distances: {result}") +print(f"min distance idx: {result[0][0]}, distance: {result[0][1]}") + +# show result +img_show = image.Image(disp.width(), disp.height()) +img = img.resize(disp.width() // 2, disp.height() // 2, image.Fit.FIT_CONTAIN) +img_show.draw_image(0, 0, img) +img_res = class_images[result[0][0]].resize(disp.width() // 2, disp.height() // 2, image.Fit.FIT_CONTAIN) +img_show.draw_image(disp.width() // 2, 0, img_res) +img_show.draw_string(2, disp.height() // 2 + 80, f"distance: {result[0][1]}", scale=1.5) +img_show.draw_string(2, disp.height() // 2 + 40, f"test.jpg", scale=1.5) +img_show.draw_string(disp.width() // 2 + 2, disp.height() // 2 + 40, class_path[result[0][0]], scale=1.5) + +disp.show(img_show) + + +while not app.need_exit(): + time.sleep_ms(100) diff --git a/examples/vision/ai_vision/nn_self_learn_classifier_cam.py b/examples/vision/ai_vision/nn_self_learn_classifier_cam.py new file mode 100644 index 00000000..19f5e886 --- /dev/null +++ b/examples/vision/ai_vision/nn_self_learn_classifier_cam.py @@ -0,0 +1,3 @@ + +# see https://github.com/sipeed/maixpy/tree/main/projects/app_self_learn_classifier + diff --git a/examples/vision/image_basic/binary.py b/examples/vision/image_basic/binary.py index 752fa712..3cd8de52 100644 --- a/examples/vision/image_basic/binary.py +++ b/examples/vision/image_basic/binary.py @@ -2,6 +2,8 @@ # 1. load image src_img = image.load("test.jpg") +if src_img is None: + raise Exception(f"load image {file_path} failed") # 2. binarize the image thresholds = ((0, 100, 20, 80, 10, 80)) diff --git a/examples/vision/image_basic/image_load.py b/examples/vision/image_basic/image_load.py index 952517bb..744d5e41 100644 --- a/examples/vision/image_basic/image_load.py +++ b/examples/vision/image_basic/image_load.py @@ -2,6 +2,8 @@ file_path = "/maixapp/share/icon/detector.png" img = image.load(file_path, format = image.Format.FMT_RGBA8888 if file_path.endswith(".png") else image.Format.FMT_RGB888) +if img is None: + raise Exception(f"load image {file_path} failed") disp = display.Display() disp.show(img, fit = image.Fit.FIT_CONTAIN) diff --git a/maix/maix_resize.py b/maix/maix_resize.py index e610c1ff..a842cf66 100644 --- a/maix/maix_resize.py +++ b/maix/maix_resize.py @@ -15,6 +15,8 @@ def main_cli(): from maix import image img = image.load(args.input) + if img is None: + raise Exception(f"load image {args.input} failed") fit = { 'fill': image.Fit.FIT_FILL, 'contain': image.Fit.FIT_CONTAIN, diff --git a/maix/v1/image.py b/maix/v1/image.py index 201d4cf6..cf198d55 100644 --- a/maix/v1/image.py +++ b/maix/v1/image.py @@ -42,6 +42,8 @@ def __init__(self, path=None, copy_to_fb=False, width=640, height=480, do_nothin self.__img = image.Image(width, height) else: self.__img = image.load(path) + if self.__img is None: + raise Exception(f"load image {path} failed") def get_priv_img(self): return self.__img diff --git a/maix/version.py b/maix/version.py index ad73408b..3674ece4 100644 --- a/maix/version.py +++ b/maix/version.py @@ -3,6 +3,6 @@ version_major = 4 version_minor = 3 -version_patch = 0 +version_patch = 2 __version__ = "{}.{}.{}".format(version_major, version_minor, version_patch) diff --git a/projects/app_self_learn_classifier/.gitignore b/projects/app_self_learn_classifier/.gitignore new file mode 100644 index 00000000..babf76a7 --- /dev/null +++ b/projects/app_self_learn_classifier/.gitignore @@ -0,0 +1,5 @@ + +build +dist +/CMakeLists.txt + diff --git a/projects/app_self_learn_classifier/app.png b/projects/app_self_learn_classifier/app.png new file mode 100644 index 00000000..0e506c27 Binary files /dev/null and b/projects/app_self_learn_classifier/app.png differ diff --git a/projects/app_self_learn_classifier/app.yaml b/projects/app_self_learn_classifier/app.yaml new file mode 100644 index 00000000..fe072818 --- /dev/null +++ b/projects/app_self_learn_classifier/app.yaml @@ -0,0 +1,10 @@ +id: self_learn_classifier +name: Self Learn Classifier +version: 1.0.1 +author: Sipeed Ltd +icon: app.png +desc: Learn anything on device, no PC training needed. +files: + - app.png + - app.yaml + - main.py diff --git a/projects/app_self_learn_classifier/main.py b/projects/app_self_learn_classifier/main.py new file mode 100644 index 00000000..f8798428 --- /dev/null +++ b/projects/app_self_learn_classifier/main.py @@ -0,0 +1,121 @@ +from maix import nn, image, camera, display, app, time, touchscreen +import os + +def get_learn_btn_rect(x, y, w, h, label): + s = image.string_size(label) + sc = image.string_size("a") + btn_w = s.width() + sc.width() * 2 + btn_h = s.height() + sc.height() * 2 + y = y - btn_h - 10 + return x, y, btn_w, btn_h, x + sc.width(), y + sc.height() + +def draw_btns(img: image.Image, btns): + for k, r in btns.items(): + img.draw_rect(r[0], r[1], r[2], r[3], image.COLOR_WHITE) + img.draw_string(r[4], r[5], k, image.COLOR_WHITE) + +def is_in_button(x, y, btn_pos): + return x > btn_pos[0] and x < btn_pos[0] + btn_pos[2] and y > btn_pos[1] and y < btn_pos[1] + btn_pos[3] + +def draw_string_center(img, msg): + img.draw_string((img.width() - image.string_size(msg, scale=2, thickness=6)[0]) // 2, img.height() // 2, msg, image.COLOR_WHITE, scale=2, thickness=6) + img.draw_string((img.width() - image.string_size(msg, scale=2, thickness=6)[0]) // 2, img.height() // 2, msg, image.COLOR_RED, scale=2, thickness=2) + +def main(disp): + cam = camera.Camera(disp.width(), disp.height()) + ts = touchscreen.TouchScreen() + + btns = {} + btns["Learn"] = get_learn_btn_rect(2, cam.height(), cam.width(), cam.height(), "Learn") + btns["< Exit"] = get_learn_btn_rect(2, btns["Learn"][3] + 10, cam.width(), cam.height(), "< Exit") + btns["+ Class"] = get_learn_btn_rect(2, cam.height() - btns["Learn"][3] - 50, cam.width(), cam.height(), "+ Class") + btns["+ Sample"] = get_learn_btn_rect(2, cam.height() - btns["Learn"][3] - btns["+ Class"][3] - 100, cam.width(), cam.height(), "+ Sample") + btns["Clear"] = get_learn_btn_rect(cam.width() - btns["Learn"][2], cam.height(), cam.width(), cam.height(), "Clear") + + + classifier = nn.SelfLearnClassifier(model="/root/models/mobilenet_v2_no_top.mud") + labels = [] + if os.path.exists("/root/my_classes.bin"): + labels = classifier.load("/root/my_classes.bin") + learn_times = 0 + show_msg = "" + show_msg_t = 0 + + res_max = None + last_pressed = False + while not app.need_exit(): + img = cam.read() + # recognize + crop = img.resize(classifier.input_width(), classifier.input_height(), image.Fit.FIT_COVER) + if classifier.class_num() > 0: + res_max = classifier.classify(crop)[0] + + # learn + x, y, pressed = ts.read() + if pressed: + if not last_pressed: + last_pressed = True + if is_in_button(x, y, btns["+ Class"]): + classifier.add_class(crop) + labels.append(f"Class {len(labels) + 1}") + classifier.save("/root/my_classes.bin", labels=labels) + elif is_in_button(x, y, btns["+ Sample"]): + classifier.add_sample(crop) + classifier.save("/root/my_classes.bin", labels=labels) + elif is_in_button(x, y, btns["< Exit"]): + app.set_exit_flag(True) + elif is_in_button(x, y, btns["Learn"]): + msg = "Learning ..." + draw_string_center(img, msg) + disp.show(img) + epoch = classifier.learn() + if epoch > 0: + learn_times += 1 + classifier.save("/root/my_classes.bin", labels=labels) + show_msg = "Learn complete" + show_msg_t = time.ticks_s() + else: + show_msg = "Already learned" + show_msg_t = time.ticks_s() + elif is_in_button(x, y, btns["Clear"]): + if os.path.exists("/root/my_classes.bin"): + os.remove("/root/my_classes.bin") + classifier.clear() + labels.clear() + res_max = None + learn_times = 0 + else: + last_pressed = False + + # show + min_len = min(img.width(), img.height()) + if img.width() == min_len: + x = 0 + y = (img.height() - min_len) // 2 + else: + x = (img.width() - min_len) // 2 + y = 0 + img.draw_rect(x, y, min_len, min_len, image.COLOR_WHITE, thickness=2) + img.draw_string(x, y + 2, f"class: {classifier.class_num()}, sample: {classifier.sample_num()}, learn: {learn_times}", image.COLOR_RED, scale=1.5) + img.draw_string(x, y + 4, f"class: {classifier.class_num()}, sample: {classifier.sample_num()}, learn: {learn_times}", image.COLOR_WHITE, scale=1.5) + if res_max: + img.draw_string(x, y + 30, f"{labels[res_max[0]]}", image.COLOR_WHITE, scale=2, thickness=6) + img.draw_string(x, y + 32, f"{labels[res_max[0]]}", image.COLOR_RED, scale=2, thickness=2) + img.draw_string(x, y + 62, f"similarity: {res_max[1]:.2f}", image.COLOR_WHITE, scale=1.5, thickness=4) + img.draw_string(x, y + 64, f"similarity: {res_max[1]:.2f}", image.COLOR_RED, scale=1.5, thickness=2) + draw_btns(img, btns) + if show_msg and time.ticks_s() - show_msg_t < 3: + draw_string_center(img, show_msg) + disp.show(img) + +disp = display.Display() +try: + main(disp) +except Exception: + import traceback + msg = traceback.format_exc() + img = image.Image(disp.width(), disp.height()) + img.draw_string(0, 0, msg, image.COLOR_WHITE) + disp.show(img) + while not app.need_exit(): + time.sleep_ms(100)