Add Hunter BTT (BLE) as plugin

[BasvanH]

Hi,

I'm reverse engineering the BLE protocol for controlling my Hunter BTT irrigation valve. As Im already using ESPEasy within my home ecosystem, I was thinking about writing a plugin for it.

Im using the NimBLE library to communicate with the Valve. I want to readout the battery status (done). I wish control open and close of the valve (done). And I wish to change it operating mode, schema / manual / disabled (WIP).

Plugin features:

• The plugin should support those features above.
• Additionally Im thinking about a BLE scan window so you can select your BTT as we need it's MAC address to communicate to. Manually enter the MAC could also be possible.

I currently own the single valve version, there's also a double valve version. I can add support but I then need to BLE traffic dumps.

My questions:

• Because the value is not really a sensor but a device, does this fit well within this ESPEasy project?
• Im a beginner in Arduino, Im able to build the parts for communication with the devices but not sure if I can write the whole plugin so perhaps some help is needed.

Regards,
Bastiaan

[TD-er]

Well it is just like a switch, so it would fit perfectly well in the ESPEasy architecture...

However (there is always one of those ;) ) we currently have not any Bluetooth related code integrated in ESPEasy so far.
Problem with Bluetooth on ESP32-xx is that it is using the same radio as WiFi.
So it is an exclusive-OR to use either Bluetooth or WiFi.
It can be done as some other projects already have shown, but it will be quite tricky to get the timings right between interacting with Bluetooth and WiFi or else either one may get a poor connection.

This is one of the reasons I have not yet looked into Bluetooth myself to integrate it into ESPEasy.

I am currently doing a complete rewrite of the WiFi/network related code, so I can keep this in mind.

N.B. not all ESP32's support the same Bluetooth. So that's another caveat for adding Bluetooth support.

Does your test setup use both WiFi and Bluetooth?
Is it loosing its WiFi connection, or acting 'sluggish' when communicating with Bluetooth?
You could try to let a ping run from any other computer in your network to the ESP and then see if it is still stable while interacting with Bluetooth.

[BasvanH]

Im currently testing/developing with the ESP32 connected with USB cable, so not via WiFi yet, I was not aware of this challenge. For this project it would be mandatory to use WiFi in combination with Bluetooth. I started this project to extend the very short Bluetooth range of this value over WiFi and off course add automation via ESPEasy.

These are my board details:

• PLATFORM: Espressif 32 (6.9.0) > Espressif ESP32 Dev Module
• HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
• Chip is ESP32-D0WD-V3 (revision v3.1)
• Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
• Crystal is 40MHz

I will include WiFi in my test to see what happens.

[BasvanH]

I did some testing. I enabled Wifi and Webserver and created a simple page that triggers some BLE commands. When BLE is connected, the webpage is very unstable, not usable. When BLE is disconnected, the page works fine. After some researching I got it working stable with these code parts:

#include <WiFi.h>
#include <WebServer.h>
#include <NimBLEDevice.h>

WebServer server(80);
// WebServer task for core 0
void WiFiTask(void * parameter) {
    for (;;) {
        server.handleClient();
        delay(10);
    }
}
void setup() {
    // Start webserver on core 0
    server.begin();
    xTaskCreatePinnedToCore(WiFiTask, "WiFiTask", 10000, NULL, 1, NULL, 0);
}

From what I understand it puts Webserver on core 0 where also wifi is serviced from. BLE is serviced from core 1. This works for me on a 250 lines sketch. Ping to ESP32 is fluctuating a lot, between 6 and 500ms.

[TD-er]

I expect the only way to get this stable is to let the WiFi negotiate a longer DTIM with the AP and keeping track of when a beacon interval is about to happen.
Inbetween those beacon intervals you can then handle Bluetooth.

But I think the only real stable option is to use Ethernet when using Bluetooth.

[BasvanH]

And what about using an HM10 Bluetooth module? It has support for BLE. You can't use an BLE library, but I can create the AT commands to reach the same goal. Do you see any problems with using that module?

[tonhuisman]

Do you see any problems with using that module?

It feels a bit backward to use an external BLE module on a unit that has BLE on-board (at least the ESP32 series), but it will probably be the simplest solution to communicate with Bluetooth devices.

You'd probably have to use the Serial Proxy (P087) or Serial Server (P020) plugins to control that module and receive/process the response in rules.

Not sure what the response speed would be though, but they seem to communicate at 115200 baud, so that should be fast enough for most types of sensors, once they are connected.

[BasvanH]

With BLE speed is not very essential. Just ordered some HM10's and will do some testing.

[uzi18]

@BasvanH do you have list of AT commands for this module?

[tonhuisman]

@uzi18 I found this page quite helpful: https://www.martyncurrey.com/hm-10-bluetooth-4ble-modules/#HM-10%20-%20AT%20commands

[uzi18]

@tonhuisman but BLE module is not a problem here.
To talk to Hunter BTT you need to know protocol ;)

Note.: The HM-10 is Bluetooth version 4.0 only. This means it cannot connect to Bluetooth 2/2.1 modules such as the HC-06 and HC-05.

[BasvanH]

@uzi18 I already have reversed engineered part of the Hunter BTT BLE protocol. Im able to manually control the valve by giving x minutes to run. And I can manually close it.

Open, run for 5 minutes: Service ID fcc0 / Characteristic fcd9

69 03 01 00 05

69 Manual control
03 Dont know
01 Open
00 05 5 minutes, max is 6 hours = 03 60

Close: Service ID fcc0 / Characteristic fcd9

69 03 00 00 00

69 Manual control
03 Dont know
01 Close
00 00 But doenst matter when closing

Im busy gatchering the data for switching between Disabled / Manual and Schedule.

If you want to do reverse engineering yourself, grab an android phone. Enable developer tools and enable HCI snooping. Reboot your phone. Then open Hunter BTT app and perform your actions, note down the time you execute those actions. Then pull the debug logs and drop them in Wireshark. There you can see the BLE packets and their service ID's and Characteristics.

//Edit
Hunter BTT = BLE = Bluetooth v4. so HM10 (with firmware v700 and higher) will work. Avoid the clone ones (no second crystal), as those you probably cant upgrade to latest firmware version v709 which you need to have BLE support.

[uzi18]

Nice, will compare these data with what have found on my own.

[BasvanH]

If you found any, please share it with the community.

[uzi18]

@BasvanH did you find any kind of ACK from BTT about new state?

[BasvanH]

No ACK, but it advertises the active state in service fcc0. When you WRITE for example ON with 5 minutes 69 03 01 00 05, you will see this value been published in that service. And, when a minute passes you will see it changing to 69 03 01 00 04. And when all minutes have passed it changes to 69 03 00 00 00.

[BasvanH]

I see there is a Write response.

This is the sequence:
Write Request

Frame 6892: 17 bytes on wire (136 bits), 17 bytes captured (136 bits)
    Encapsulation type: Bluetooth H4 with linux header (99)
    Arrival Time: Nov 11, 2024 10:51:31.499658000 West-Europa (standaardtijd)
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1731318691.499658000 seconds
    [Time delta from previous captured frame: 0.795861000 seconds]
    [Time delta from previous displayed frame: 0.795861000 seconds]
    [Time since reference or first frame: 50199.894685000 seconds]
    Frame Number: 6892
    Frame Length: 17 bytes (136 bits)
    Capture Length: 17 bytes (136 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    Point-to-Point Direction: Sent (0)
    [Protocols in frame: bluetooth:hci_h4:bthci_acl:btl2cap:btatt]
Bluetooth
    [Source: Google_a6:e7:d3 (74:74:46:a6:e7:d3)]
    [Destination: TexasIns_52:a5:e8 (30:45:11:52:a5:e8)]
Bluetooth HCI H4
    [Direction: Sent (0x00)]
    HCI Packet Type: ACL Data (0x02)
Bluetooth HCI ACL Packet
    .... 0000 0000 0001 = Connection Handle: 0x001
    ..00 .... .... .... = PB Flag: First Non-automatically Flushable Packet (0)
    00.. .... .... .... = BC Flag: Point-To-Point (0)
    Data Total Length: 12
    Data
    [Connect in frame: 6498]
    [Disconnect in frame: 7003]
    [Source BD_ADDR: Google_a6:e7:d3 (74:74:46:a6:e7:d3)]
    [Source Device Name: Pixel 7a]
    [Source Role: Unknown (0)]
    [Destination BD_ADDR: TexasIns_52:a5:e8 (30:45:11:52:a5:e8)]
    [Destination Device Name: BTT A5E8]
    [Destination Role: Unknown (0)]
    [Current Mode: Unknown (-1)]
Bluetooth L2CAP Protocol
    Length: 8
    CID: Attribute Protocol (0x0004)
Bluetooth Attribute Protocol
    Opcode: Write Request (0x12)
        0... .... = Authentication Signature: False
        .0.. .... = Command: False
        ..01 0010 = Method: Write Request (0x12)
    Handle: 0x004d (Unknown: Unknown)
        [Service UUID: Unknown (0xfcc0)]
        [UUID: Unknown (0xfcd9)]
    Value: 6903000005

Then you receive a Handle Value Notification:

Frame 6894: 15 bytes on wire (120 bits), 15 bytes captured (120 bits)
    Encapsulation type: Bluetooth H4 with linux header (99)
    Arrival Time: Nov 11, 2024 10:51:31.601932000 West-Europa (standaardtijd)
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1731318691.601932000 seconds
    [Time delta from previous captured frame: 0.058694000 seconds]
    [Time delta from previous displayed frame: 0.058694000 seconds]
    [Time since reference or first frame: 50199.996959000 seconds]
    Frame Number: 6894
    Frame Length: 15 bytes (120 bits)
    Capture Length: 15 bytes (120 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    Point-to-Point Direction: Received (1)
    [Protocols in frame: bluetooth:hci_h4:bthci_acl:btl2cap:btatt]
Bluetooth
    [Source: TexasIns_52:a5:e8 (30:45:11:52:a5:e8)]
    [Destination: Google_a6:e7:d3 (74:74:46:a6:e7:d3)]
Bluetooth HCI H4
    [Direction: Rcvd (0x01)]
    HCI Packet Type: ACL Data (0x02)
Bluetooth HCI ACL Packet
    .... 0000 0000 0001 = Connection Handle: 0x001
    ..10 .... .... .... = PB Flag: First Automatically Flushable Packet (2)
    00.. .... .... .... = BC Flag: Point-To-Point (0)
    Data Total Length: 10
    Data
    [Connect in frame: 6498]
    [Disconnect in frame: 7003]
    [Source BD_ADDR: TexasIns_52:a5:e8 (30:45:11:52:a5:e8)]
    [Source Device Name: BTT A5E8]
    [Source Role: Unknown (0)]
    [Destination BD_ADDR: Google_a6:e7:d3 (74:74:46:a6:e7:d3)]
    [Destination Device Name: Pixel 7a]
    [Destination Role: Unknown (0)]
    [Current Mode: Unknown (-1)]
Bluetooth L2CAP Protocol
    Length: 6
    CID: Attribute Protocol (0x0004)
Bluetooth Attribute Protocol
    Opcode: Handle Value Notification (0x1b)
        0... .... = Authentication Signature: False
        .0.. .... = Command: False
        ..01 1011 = Method: Handle Value Notification (0x1b)
    Handle: 0x0032 (Unknown: Unknown)
        [Service UUID: Unknown (0xfcc0)]
        [UUID: Unknown (0xfcd1)]
    Value: 610102

And when it's written, you receive a Write Response

Frame 6895: 10 bytes on wire (80 bits), 10 bytes captured (80 bits)
    Encapsulation type: Bluetooth H4 with linux header (99)
    Arrival Time: Nov 11, 2024 10:51:31.604472000 West-Europa (standaardtijd)
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1731318691.604472000 seconds
    [Time delta from previous captured frame: 0.002540000 seconds]
    [Time delta from previous displayed frame: 0.002540000 seconds]
    [Time since reference or first frame: 50199.999499000 seconds]
    Frame Number: 6895
    Frame Length: 10 bytes (80 bits)
    Capture Length: 10 bytes (80 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    Point-to-Point Direction: Received (1)
    [Protocols in frame: bluetooth:hci_h4:bthci_acl:btl2cap:btatt]
Bluetooth
    [Source: TexasIns_52:a5:e8 (30:45:11:52:a5:e8)]
    [Destination: Google_a6:e7:d3 (74:74:46:a6:e7:d3)]
Bluetooth HCI H4
    [Direction: Rcvd (0x01)]
    HCI Packet Type: ACL Data (0x02)
Bluetooth HCI ACL Packet
    .... 0000 0000 0001 = Connection Handle: 0x001
    ..10 .... .... .... = PB Flag: First Automatically Flushable Packet (2)
    00.. .... .... .... = BC Flag: Point-To-Point (0)
    Data Total Length: 5
    Data
    [Connect in frame: 6498]
    [Disconnect in frame: 7003]
    [Source BD_ADDR: TexasIns_52:a5:e8 (30:45:11:52:a5:e8)]
    [Source Device Name: BTT A5E8]
    [Source Role: Unknown (0)]
    [Destination BD_ADDR: Google_a6:e7:d3 (74:74:46:a6:e7:d3)]
    [Destination Device Name: Pixel 7a]
    [Destination Role: Unknown (0)]
    [Current Mode: Unknown (-1)]
Bluetooth L2CAP Protocol
    Length: 1
    CID: Attribute Protocol (0x0004)
Bluetooth Attribute Protocol
    Opcode: Write Response (0x13)
        0... .... = Authentication Signature: False
        .0.. .... = Command: False
        ..01 0011 = Method: Write Response (0x13)
    [Handle: 0x004d (Unknown: Unknown)]
        [Service UUID: Unknown (0xfcc0)]
        [UUID: Unknown (0xfcd9)]
    [Request in Frame: 6892]

[BasvanH]

Im able to control the Hunter BTT valve with Serial Server and a HM-10. However, when I send a command with serialsend or serialsendmix I do not see a return in the Log. This is the first time im using Serial Server, should it trigger an event when sending an command? Are events visable in the Log?

[TD-er]

Events are visible in the log, but I wonder where you're looking for the logs.

serialsend is a command for P020_Ser2Net, so you need to have a task enabled using this plugin.
The function handleSerialIn doesn't output a lot of logging and what it logs has log level debug. (and you need to have a build which has debug logs included)

So if you have debug log level included in the build, you can set the log level on the tools->Advanced page to this level.

[tonhuisman]

The plugin commands themselves will be reflected in the log, as said, but will/should not cause any events, unless of course the device sends some data back when the command is handled.

Serial Server waits for a packet of data received, or the time-out reached, before an event is generated, but also requires either a TCP client to be connected to the configured port, or the setting Process events without client checked.

Edit: And also Event processing should be set to Generic (the other options don't make sense for your device)

[BasvanH]

but I wonder where you're looking for the logs.

On the web interface / Tools / Log

I'm using the Tools / Command option to send commands. No rules or tasks yet.

The plugin commands themselves will be reflected in the log, as said, but will/should not cause any events, unless of course the device sends some data back when the command is handled.

I see my commands as an INFO event in the Log. The HM10 gives a result back after executing AT commands. This can take from a few ms until a few seconds, depening on the command. Those messages from th eHM10 are not shown as a INFO Log or so.

And also Event processing should be set to Generic

Yes, this I already figgered, this is set, still no Log items of the HM10 initiated messages.

These are some of the commands Im sending which the HM10 gives various resonses to.
Connect to BLE deivce / serialsend,AT+CON30451152A5E8
Readout Characteristics data / serialsend,AT+READDATA004D?
Open valve for 1 minute / serialsendmix,AT+SEND_DATAWR004D,0x69,0x03,0x01,0x00,0x01

What should I do to see those responses of the HM10? Enable Debug in my build?

Edit: I do see the response messages of the HM10 on the COMport of the ESP32.

[BasvanH]

Did some further testing. Enabled Single event with all values, and added rule:

On hm10#All Do
  LogEntry,'Received serial log all'
Endon

When I execute commands, the HM10 executes it, I see the result in the USB serial connection on the ESP32, but no log and no event in the Log of the web ui Logs.

[tonhuisman]

I have received a few HM10 devices, but did not get to playing lab-testing them yet. I would have expected some data received and in the logs, possibly you need to have the rule used like this:

On !serial Do
  LogEntry,'Received serial data: %eventvalue%'
Endon

The exclamation mark is needed for events that don't have numerical data in the first argument.

[BasvanH]

Yes, tried that one already, also no events.

Secondly, I notice when saving my rules the Serial Server stops working. I need to go into the device and Submit it again and then the AT commands work again. This is the sequence:

635206: WD : Uptime 11 ConnectFailures 0 FreeMem 229780 WiFiStatus: WL_CONNECTED 3 ESPeasy internal wifi status: Conn. IP Init
636468: HTTP: serialsend,AT           <= AT command works
644196: Upload: START, filename: rules1.txt    <= update a rule
644308: Upload: WRITE, Bytes: 804
644316: Upload: END, Size: 804
644590: ESPEasy console using ESPEasySerial
654551: HTTP: serialsend,AT+CON30451152A5E8    <= AT command no longer works, i do a Submit in the Serial Server device
663148: EVENT: Clock#Time=Tue,11:01
663581: Ser2Net: WiFi server closed
663642: ESPEasy console using ESPEasySerial
663687: Ser2Net: TaskIndex=1 port=4 rxPin=1 txPin=3 BAUDRATE=115200 SERVER_PORT=1234 SERIAL_PROCESSING=1
663713: Ser2Net: WiFi server started at port 1234
663938: EVENT: TaskExit#hm=1,1
664074: EVENT: TaskInit#hm=1,1
664078: EVENT: Rules#Timer=1,11
665206: WD : Uptime 11 ConnectFailures 0 FreeMem 227856 WiFiStatus: WL_CONNECTED 3 ESPeasy internal wifi status: Conn. IP Init
668161: HTTP: serialsend,AT+CON30451152A5E8     <= AT command works again

This happens everytime I save rule. And note the lacking of response AT events.