Add local dp_x handling based on known product ids

[fhempy]

Hi,
what do you think about adding known mappings directly to the project as iobroker does?
See https://github.com/Apollon77/ioBroker.tuya/blob/master/lib/schema.json

They use the product id to identify the dp mappings. So you don't need any connection to the cloud if you know your localkey and product id.

I used it in my project as well where I use a combination of tinytuya and localtuya (because of asyncio) code:
https://github.com/fhempy/fhempy/tree/master/FHEM/bindings/python/fhempy/lib/tuya

Would be great to make one mapping list in tinytuya where everybody could contribute.

[jasonacox]

I love the idea @fhempy ! Similar to the community driven Contrib extension for devices, we could add something like a "Products" extension. How would you like to see it work? My first thought...

from tinytuya import Products

product_id = 'MShdslm9Uw7Q59nN'

# Look up attributes for product
a = Products.attributes(product_id)
print("Device version = %d - name = %s - type = %s" % (a['version'], a['name'], a['type']))

# Look up data points for product
print("DPS:")
dps = Products.dps(product_id)
for point in dps:
    print("  %d = %s - %s", % (dps['id'], dps['name'], dps['type'])

# Advanced Device Connection
d = Products.connect(
       product_id,
       dev_id='DEVICE_ID_HERE',
       address='IP_ADDRESS_HERE',
       local_key='LOCAL_KEY_HERE')

d.turn_on()
if d.device() is 'led_switch':
    d.set_colour(d.GREEN)

I'm sure there is a lot more we could do here. We should also make sure scanner.py and wizard.py keep product_id information for each of the devices to help with easier mapping.

Is this something you would be willing to contribute to help get started? I recommend we start small/simple, perhaps with just the dictionary and some lookup functions (a subset of my "look up" examples above).

[fhempy]

Great to see that you are also interested in this functionality. I would just slightly adapt it:

from tinytuya import Products

product_id = 'MShdslm9Uw7Q59nN'

# Look up attributes for product
a = Products.attributes(product_id)
print("Device version = %d - name = %s - type = %s" % (a['version'], a['name'], a['type']))

I think the version shouldn't be part of the mappings, as it depends on the user if a device was updated or not. Therefore I would always detect the version instead of having it in the code.
We might add here:

• biz_type
• category (which could be used to map to the device type after connect)
• icon
• model
• product_name
  Those are the things we usually use to retrieve from tuya iot cloud and are static.

# Look up data points for product
print("DPS:")
dps = Products.dps(product_id)
for point in dps:
    print("  %d = %s - %s", % (dps['id'], dps['name'], dps['type'])

That's good. I would add:

• mode: rw (=function) or ro (=status)
• values like tuya is sending it (e.g. "unit": "s", "min": 0, "max": 86400, "scale": 0, "step": 1), depending on the type
• description

# Advanced Device Connection
d = Products.connect(
       product_id,
       dev_id='DEVICE_ID_HERE',
       address='IP_ADDRESS_HERE',
       local_key='LOCAL_KEY_HERE')

I would suggest to make the address optional and scan for the IP if it isn't provided. I already had some users where the local IP changed and they forgot to change it in their configuration, therefore I would like to get rid of the IP and just use the dev_id.

d.turn_on()
if d.device() is 'led_switch':
    d.set_colour(d.GREEN)

I assume you mean that it should work together with the device type extension.

I'll try to contribute a first (simple) version.

[jasonacox]

All of this makes sense.

On the Product.connect() proposal, the IP lookup could be optional, but I like the idea. I would add that @uzlonewolf is developing a new scanner.py that will make "IP" lookup ~10x faster which would make this even more viable.

Thanks for raising this idea @fhempy - Looking forward to seeing the PR! ❤️ Simple is gold (I'm a big fan of the MVP approach).

[uzlonewolf]

Unfortunately I'm not really feeling this one. In my experience the DP mapping is only half right with a significant number of device DPs not being in the list, the device not actually using quite a few list DPs, and a handful of DPs that are in both are listed with the wrong scale/step/min/max.

If we do add this I think it would be better for the wizard to pull it when retrieving the local keys so we always have the latest version (in case a firmware update adds more) and don't have to maintain or include a massive JSON blob.

@jasonacox The auto-IP on connect/init does not use anything from scanner.py, but making the IP detection function in core.py much, much quicker is a relatively minor change. I'll make a PR just for that so we can get that in now while I'm finishing up the scanner.py stuff.

[jasonacox]

I love this pivot! @uzlonewolf has a great point. If we include the massive JSON blog in the module, it sort of defeats the whole "Tiny" part of the project. 😜

I looked through the JSON file and it is essentially what we get from a wizard pull (but for products we don't have). I did a spot comparison and the DP mapping is as @uzlonewolf points out, mostly not very helpful. However, I can see some value in data like category , product_name, biz_type which TinyTuya users could leverage in their own projects.

Questions and thoughts on this pivot:

• What is the best place to store the additional device data if we proceed with having wizard pull this? My first inclination would be to put this in devices.json along with the name, id, key and mac. Currently this data is recorded tuya-raw.json as part of the wizard setup but perhaps we don't want all of it. What am I missing? Thoughts or concerns combining these two files?
• Is there value in a community curated collection of product_idmappings of DP data? The point would be to extend the Tuya Cloud DP data based on what the community discovers. Quality could be an issue as mentioned, but could still provide some help (thinking of how the DPS Table in the README has expanded over time). We could store that in the /docs section of the repo which would keep it out of the TinyTuya python library but still provide it as a reference for anyone who wants to use it. It could also be used to generate a markup table to potentially replace or augment https://github.com/jasonacox/tinytuya#tuya-data-points---dps-table.
• AProduct extension as mentioned above still makes sense to me (would love @uzlonewolf or others to provide candid feedback), even if the datasource it uses is devices.json or wherever we decide to store the extended Tuya Cloud data.

What else am I missing?

[uzlonewolf]

I say there's no such thing as too much data and would also include model and sub. sub is important for device scanning/polling as sub-devices are not directly connected to the network and as such cannot be queried directly; a network scan/poll will never find them. icon could also be useful for things like the server webpage.

[uzlonewolf]

While working on my test scripts for #188 I realized d = tinytuya.OutletDevice( '0123456789abcdef0123' ) works but d = tinytuya.OutletDevice( '0123456789abcdef0123', '1.2.3.4' ) does not since the version (3.3 for this device) is not stored in devices.json. So, that's something else to add to devices.json or wherever.

[uzlonewolf]

In the PR I already had open I added the storing of that additional data to devices.json. version and last_ip still need to be added somehow, probably via the scanner (to be run after retrieving the Cloud device list) and/or server.

I'm not seeing what a Product extension would do without either the official (mostly useless) DPS list or a community-curated one. Would it just be a way to pull the additional data out of devices.json? Just stuffing that data into a XenonDevice variable would be trivial since I already do that for the local key lookup.

[fhempy]

Hi,
thanks for the discussion! I would like to bring in my use case as it might make things more clear.

• I'm using tinytuya as a library, therefore any implementation in wizard doesn't make sense for me, as wizard is the interactive part of tinytuya which is not used by any code outside of tinytuya. If you don't agree, please let me know.
• devices.json is only used within the tinytuya project. I can't think of using devices.json from a library perspective as my code doesn't store anything on the filesystem. Therefore from a library point of view devices.json is never there.
• In my case I don't use any of the device type classes. I always focus on generic solutions for devices to prevent the need of device type specific implementations. I don't want to implement anything new in my project if a new tuya device type appears on the market. It should always be supported out of the box. Therefore I just use the set_dp() function of the localtuya project (https://github.com/fhempy/fhempy/blob/master/FHEM/bindings/python/fhempy/lib/tuya/pytuya/__init__.py#L513) which is independent of the data type for the specific DP.

As I mentioned in the last point I don't use device specific classes. I just provide all DPs to the users. Status DPs are just reported and function DPs are provided as a command on the user interface. As the users don't know what the certain DP is for or which values can be set via commands, I just need the name, type and possible values - I don't need the type of the device. Based on that information I can provide the human readable information to the users. Furthermore I can map DPs which are not provided by tuya iot cloud mappings.

I agree that putting the mappings blob to the code is something I don't like either, but I would like to provide the full functionality for users if they know their local key already. As soon as the users have the local key, there shouldn't be a need to connect to the tuya iot cloud as the DPs could be retrieved "locally".

So let's focus the discussion on the first step:

• Do we want to provide DP mappings by tinytuya to avoid the need to connect to tuya cloud if the local key is known?

[uzlonewolf]

@fhempy I agree with the theory and would love to see that, however where will the DP mappings come from? The "official" mappings are just straight up wrong, and trying to use them will lead to frustrated users when they do not work.

[uzlonewolf]

I keep thinking about adding this but keep getting stuck on how, exactly, it would work. How would tinytuya get the DPS mappings? If the user provides them then there really isn't much for tinytuya to do besides maybe scaling the set/returned values and enforcing limits when setting; as the user/calling code already has the human-readable names there is no real need for tinytuya to also keep track of them. To use the mostly-incorrect "official" mappings you need the Product Id which means either pulling it from the Cloud (requires internet access and an account) or being directly connected to the local network to receive the broadcasts (will not work if there is a router between you and the device). I have yet to find a way of directly querying a device for the Product Id. Even if you do have the Product Id you still have the issue of what to do with new/unknown devices.

In short I don't really understand what it is you want tinytuya to do. Providing the massive blob of mappings fails the "don't need to change anything when a new device is released" test and can easily be done today once v1.7.1 is released by your code with:

import tinytuya

did = 'efd01234df6510cbd6abcd'
dkey = 'sadfdsafdsafsad'

(dev_ip, dev_ver, dev_info) = tinytuya.find_device(did)

print(dev_info)

if dev_info and 'productKey' in dev_info:
    product_id = dev_info['productKey']
    dps_mappings = your_dps_lookup( product_id ) # i.e. { 1: {'name': 'Switch 1', 'type': 'bool' } } 
else:
    product_id = None
    dps_mappings = {}

d = tinytuya.OutletDevice( did, dev_ip, dkey, version=float(dev_ver) )

status = d.status()

if status and 'dps' in status:
    for dp in status['dps']:
        val = status['dps'][dp]
        dp = int(dp)
        if dp in dps_mappings:
            name = dps_mappings[dp]['name']
            if 'scale' in dps_mappings[dp] and dps_mappings[dp]['scale']:
                val *= dps_mappings[dp]['scale']
        else:
            name = 'Unknown DPS %d' % dp

        print( name, 'is now', val )


# user says "Set 'Switch 1' On":
for dp in dps_mappings:
    if dps_mappings[dp]['name'] == 'Switch 1':
        d.set_value( dp, True )
        break

# user says "Set 'Dimmer 4' to 80%":
for dp in dps_mappings:
    if dps_mappings[dp]['name'] == 'Dimmer 4':
        d.set_value( dp, 80 )
        break

Just provide that massive list with your software and refer to it when implementing your_dps_lookup() before calling set_value() (tinytuya's version of set_dp()) with the appropriate DP. Or am I missing something?

[fhempy]

You are right, it can be managed within the codes which use the tinytuya library. That's how I currently implement it.

I thought it would make sense that other projects benefit from it as well and therefore tinytuya might be the better place. It would allow us to work together (cross project) on a mapping list which we would maintain together.

The only real benefit for tinytuya would be that users don't need to setup the tuya iot project if they know their local key already and the device is already supported in the mappings blob. The best solution would be to have a REST API where users could request mappings and provide new ones via tinytuya.

Finally I'm also fine if I continue to maintain the mappings in my project.

[uzlonewolf]

As the "official" mapping list is both ginormous and wrong I do not wish to see it included. I am not opposed to a community-provided/verified list however. Either way, I think it would be a good idea to add a new device type that, when passed mappings obtained from wherever, maps and formats the returned values for you. Say,

import tinytuya
dps_data = {
    '2' : { 'name': 'mode', 'enum': ['auto', 'cool', 'heat', 'emergencyheat', 'off'] },
    '16': { 'name': 'temp_set', 'alt': 'setpoint_c', 'scale': 100 },
}

dev = tinytuya.MappedDevice( 'abcd1234', mapping=dps_data )
# and/or
dev.set_mapping(dps_data)

# equivalent
dev.set.mode = 'heat'
dev.set['mode'] = 2

status = dev.status()
if status and 'changed' in status and 'setpoint_c' in status['changed']:
    # setpoint_c changed
    # equivalent
    print( status['changed']['setpoint_c'] ) # i.e. "24.5"
    print( dev.get.setpoint_c ) # i.e. "24.5"
    print( dev.get['setpoint_c'] ) # i.e. "24.5"

[elad-bar]

there is an endpoint that can provide that information, but in order to get there, it should have login session cookie,
i took it from the UI of developer portal and didn't found corresponding in the open api:
https://{REGION}.iot.tuya.com/micro-app/cloud/api/v9/dp/info/list

const formData = {
   "devId": {DEVICE_ID},
   "region": {REGION}
};

 const headers = {
   'Content-Type': 'application/x-www-form-urlencoded',
   'csrf-token': "{developerPortalCSRFToken}",
   'Cookie': "{developerPortalRegionalCookie}"
};

since it require a manual process of login via browser (with captcha) and extract the cookie (if will be relevant, will post the process), we will need a new endpoint to inject the cookie string,
it can be done using a new endpoint in the server code to accept cookie as paramater for request,
it will run all over the devices connected and can extract the DPS from Tuya Developer Portal (same as the sync is working to extract list of users)

[uzlonewolf]

Actually I found that changing the Instruction Mode to "DP Instruction" unlocks the full list for the open API. #284 (comment) Finding this actually bumped this "Mapped Device" project to pretty high on my to-do list. Not sure when I'll be able to implement it, hopefully Soon(tm).

[elad-bar]

Right, also manage to do that and published it over my repo 2 months ago for HA tuya custom integration, but it's too complex and you need to do that for every device category, with that approach of accessing the dp list is much easier but requires more technical knowledge... will keep investigating of how to extract that without need for dev tools