[udf] Unlock JavaScript for user-defined functions

CHANGES.rst

@@ -12,6 +12,7 @@ in progress
   As per RFC 7568, SSLv3 has been deprecated in 2015 already.
 - Tests: Add more test cases to increase mqttwarn core coverage to ~100%
 - Improve example "Forward OwnTracks low-battery warnings to ntfy"
+- [udf] Unlock JavaScript for user-defined functions. Thanks, @extremeheat.
 
 
 2023-04-28 0.34.0

docs/configure/transformation.md

@@ -407,6 +407,66 @@ the invocation of your function, you will create an endless loop.
 :::
 
 
+(user-defined-functions-languages)=
+### Using other languages
+
+This section is about using languages other than Python within the mqttwarn transformation
+subsystem.
+
+#### Introduction
+Imagine you are running a large cloud infrastructure for [4IR] purposes, where hardware
+appliances or industrial machines submit telemetry data to concentrators or hubs, in order
+to converge them to a message bus like [Kafka], where, because you favoured a [FaaS]
+solution, your [serverless] function handler will consume the telemetry messages, in
+order to conduct further processing, like to filter, transform, and store the data.
+Chances are that you used the [Serverless Framework], built on [Node.js], so your application
+infrastructure is effectively based on a bunch of JavaScript code.
+
+Now, you know how the story ends. The infrastructure costs exploded, you had to fire your
+engineering team, and now you are sitting in front of a mixture of infrastructure setup
+recipes and actual application code, and you are not able to make any sense of that at all.
+After some  digging, you are able to identify the spot where the real application logic
+evaluates a set of transformation rules, in order to mangle inbound data appropriately
+into outbound data, written in JavaScript.
+
+You quickly realize that the same thing could effectively be solved using [MQTT] and Python,
+and operated on a single server machine, because you are doing only a few hundred requests
+per second anyway. However, you don't have the time, because, well, the customer quickly
+needs a fix because they added another telemetry data field waiting to be consumed and
+handled appropriately. Well, and it's Kafka anyway, how would one get rid of _that_?
+
+After a glimpse of desperation, you are reading the fine manual again, and catch up with
+the fact that telemetry data _is actually ingested using MQTT!_ That is useful, indeed!
+Remembering conversations about a thing called _mqttwarn_ at the Späti the other day,
+and that it gained the ability to run JavaScript code recently, in order to bring cloud
+computing techniques back to personal computing, you think it would be feasible to change
+that architecture of your system in the blink of an eye.
+
+There you go: You rip out the JavaScript transformation rules into a single-file version,
+export its main entry point symbol, configure mqttwarn to use `functions = mycloud.js`,
+and adjust its settings to use your MQTT broker endpoint at the beginning of the data
+pipeline, invoke mqttwarn, and turn off Kafka. It works!
+
+:::{note}
+Rest assured we are overexaggerating a bit, and [Kafka] can only be compared to [MQTT]
+if you are also willing to compare apples with oranges, but you will get the point that
+we believe simpler systems are more often than not able to solve problems equally well,
+if not more efficient, both at runtime, and on details of maintenance and operation.
+
+Other than this, every kind of system migration should be conducted with better planning
+than outlined in our rant above.
+:::
+
+#### JavaScript
+
+For running user-defined functions code written in JavaScript, mqttwarn uses the
+excellent [JSPyBridge] package.
+
+You can find an example implementation for a `filter` function written in JavaScript
+at [](#owntracks-ntfy-variants-udf).
+
+
+
 ## User-defined function examples
 
 In this section, you can explore a few example scenarios where user-defined
@@ -507,6 +567,12 @@ was received on, here `owntracks/jane/phone`. The name of the section will be
 the `section` argument, here `owntracks/#/phone`.
 :::
 
+:::{note}
+The recipe about how to use [](#owntracks-ntfy-variants-udf) demonstrates
+corresponding examples for writing a `filter` function.
+:::
+
+
 (decode-topic)=
 ### `alldata`: Decoding topic names
 
@@ -631,10 +697,18 @@ weather,topic=tasmota/temp/ds/1 temperature=19.7 1517525319000
 ```
 
 
+[4IR]: https://en.wikipedia.org/wiki/Fourth_Industrial_Revolution
+[FaaS]: https://en.wikipedia.org/wiki/Cloud_computing#Serverless_computing_or_Function-as-a-Service_(FaaS)
 [geo-fence]: https://en.wikipedia.org/wiki/Geo-fence
 [InfluxDB line format]: https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/
 [Jinja2 templates]: https://jinja.palletsprojects.com/templates/
+[JSPyBridge]: https://pypi.org/project/javascript/
+[Kafka]: https://en.wikipedia.org/wiki/Apache_Kafka
+[MQTT]: https://en.wikipedia.org/wiki/MQTT
+[Node.js]: https://en.wikipedia.org/wiki/Node.js
+[OwnTracks]: https://owntracks.org
 [Tasmota]: https://github.com/arendst/Tasmota
 [Tasmota JSON status response for a DS18B20 sensor]: https://tasmota.github.io/docs/JSON-Status-Responses/#ds18b20
-[OwnTracks]: https://owntracks.org
+[serverless]: https://en.wikipedia.org/wiki/Serverless_computing
+[Serverless Framework]: https://github.com/serverless/serverless
 [waypoint]: https://en.wikipedia.org/wiki/Waypoint

examples/owntracks-ntfy/mqttwarn-owntracks.js

@@ -0,0 +1,35 @@
+/**
+ *
+ * Forward OwnTracks low-battery warnings to ntfy.
+ * https://mqttwarn.readthedocs.io/en/latest/examples/owntracks-battery/readme.html
+ *
+ */
+
+// mqttwarn filter function, returning true if the message should be ignored.
+// In this case, ignore all battery level telemetry values above a certain threshold.
+function owntracks_batteryfilter(topic, message) {
+    let ignore = true;
+    let data;
+
+    // Decode inbound message.
+    try {
+        data = JSON.parse(message);
+    } catch {
+        data = null;
+    }
+
+    // Evaluate filtering rule.
+    if (data && "batt" in data && data.batt !== null) {
+        ignore = Number.parseFloat(data.batt) > 20;
+    }
+
+    return ignore;
+}
+
+// Status message.
+console.log("Loaded JavaScript module.");
+
+// Export symbols.
+module.exports = {
+    "owntracks_batteryfilter": owntracks_batteryfilter,
+};

examples/owntracks-ntfy/readme-variants.md

@@ -0,0 +1,71 @@
+---
+orphan: true
+---
+
+(owntracks-ntfy-variants)=
+
+# OwnTracks-to-ntfy setup variants
+
+
+## About
+
+This section informs you about additional configuration and operation variants of the
+[](#owntracks-ntfy-recipe) recipe. For example, you may want to use Docker or Podman
+to run both mqttwarn and ntfy, or you may want to use another language than Python to
+implement your filtering function.
+
+
+## Docker and Podman
+
+### Running mqttwarn as container
+This command will run mqttwarn in a container, using the `docker` command to launch it.
+Alternatively, `podman` can be used. It expects an MQTT broker to be running on `localhost`,
+so it uses the `--network=host` option. The command will mount the configuration file and
+the user-defined functions file correctly, and will invoke mqttwarn with the corresponding
+`--config-file` option.
+```shell
+docker run --rm -it --network=host --volume=$PWD:/etc/mqttwarn \
+  ghcr.io/jpmens/mqttwarn-standard \
+  mqttwarn --config-file=mqttwarn-owntracks.ini
+```
+
+### Running ntfy as container
+While this tutorial uses the ntfy service at ntfy.sh, it is possible to run your own
+instance. For example, use Docker or Podman.
+```shell
+docker run --name=ntfy --rm -it --publish=5555:80 \
+  binwiederhier/ntfy serve --base-url="http://localhost:5555"
+```
+In this case, please adjust the ntfy configuration section `[config:ntfy]` to use
+a different URL, and make sure to restart mqttwarn afterwards.
+```ini
+[config:ntfy]
+targets   = {'testdrive': 'http://localhost:5555/testdrive'}
+```
+
+
+(owntracks-ntfy-variants-udf)=
+
+## Alternative languages for user-defined functions
+
+### JavaScript
+
+In order to try that on the OwnTracks-to-ntfy example, use the alternative
+`mqttwarn-owntracks.js` implementation by adjusting the `functions` setting within the
+`[defaults]` section of your configuration file, and restart mqttwarn.
+```ini
+[defaults]
+functions = mqttwarn-owntracks.js
+```
+
+The JavaScript function `owntracks_batteryfilter()` implements the same rule as the
+previous one, which was written in Python.
+
+:::{literalinclude} mqttwarn-owntracks.js
+:language: javascript
+:::
+
+:::{attention}
+The feature to run JavaScript code is currently considered to be experimental.
+Please use it responsibly.
+:::

examples/owntracks-ntfy/readme.md

@@ -96,34 +96,13 @@ echo 'foobar' | mosquitto_pub -t 'owntracks/testdrive' -l
 
 ## Appendix
 
-This section demonstrates a few alternative methods for solving different aspects of this
-recipe, and also includes administrative information.
-
-### Running mqttwarn as container
-This command will run mqttwarn in a container, using the `docker` command to launch it.
-Alternatively, `podman` can be used. It expects an MQTT broker to be running on `localhost`,
-so it uses the `--network=host` option. The command will mount the configuration file and
-the user-defined functions file correctly, and will invoke mqttwarn with the corresponding
-`--config-file` option. 
-```shell
-docker run --rm -it --network=host --volume=$PWD:/etc/mqttwarn \
-  ghcr.io/jpmens/mqttwarn-standard \
-  mqttwarn --config-file=mqttwarn-owntracks.ini
-```
+### Configuration and operation variants
 
-### Running ntfy as container
-While this tutorial uses the ntfy service at ntfy.sh, it is possible to run your own
-instance. For example, use Docker or Podman.
-```shell
-docker run --name=ntfy --rm -it --publish=5555:80 \
-  binwiederhier/ntfy serve --base-url="http://localhost:5555"
-```
-In this case, please adjust the ntfy configuration section `[config:ntfy]` to use
-a different URL, and make sure to restart mqttwarn afterwards.
-```ini
-[config:ntfy]
-targets   = {'testdrive': 'http://localhost:5555/testdrive'}
-```
+There are different variants to configure and operate this setup. For example,
+you may want to use Docker or Podman to run both mqttwarn and ntfy, or you may
+want to use another language than Python to implement your filtering function.
+We summarized the available options on the [](#owntracks-ntfy-variants) page,
+together with corresponding guidelines how to use them.
 
 ### Backlog
 :::{todo}

mqttwarn/util.py

@@ -7,6 +7,7 @@
 import os
 import re
 import string
+import threading
 import types
 import typing as t
 from pathlib import Path
@@ -180,24 +181,31 @@ def import_module(name: str, path: t.Optional[t.List[str]] = None) -> types.Modu
         return import_module(remaining_names, path=list(module.__path__))
 
 
-def load_functions(filepath: t.Optional[str] = None) -> t.Optional[types.ModuleType]:
+def load_functions(filepath: t.Optional[t.Union[str, Path]] = None) -> t.Optional[types.ModuleType]:
 
     if not filepath:
         return None
 
+    filepath = str(filepath)
+
     if not os.path.isfile(filepath):
         raise IOError("'{}' not found".format(filepath))
 
     mod_name, file_ext = os.path.splitext(os.path.split(filepath)[-1])
 
+    logger.info(f"Loading functions module {mod_name} from {filepath}")
+
     if file_ext.lower() == ".py":
         py_mod = imp.load_source(mod_name, filepath)
 
     elif file_ext.lower() == ".pyc":
         py_mod = imp.load_compiled(mod_name, filepath)
 
+    elif file_ext.lower() in [".js", ".javascript"]:
+        py_mod = load_source_js(mod_name, filepath)
+
     else:
-        raise ValueError("'{}' does not have the .py or .pyc extension".format(filepath))
+        raise RuntimeError(f"Unable to interpret module file: {filepath}")
 
     return py_mod
 
@@ -251,3 +259,29 @@ def load_file(path: t.Union[str, Path], retry_tries=None, retry_interval=0.075,
         except:  # pragma: nocover
             pass
     return reader
+
+
+def module_factory(name, variables):
+    """
+    Create a synthetic Python module object.
+
+    Derived from:
+    https://www.oreilly.com/library/view/python-cookbook/0596001673/ch15s03.html
+    """
+    module = imp.new_module(name)
+    module.__dict__.update(variables)
+    module.__file__ = "<synthesized>"
+    return module
+
+
+def load_source_js(mod_name, filepath):
+    """
+    Load a JavaScript module, and import its exported symbols into a synthetic Python module.
+    """
+    import javascript
+
+    js_code = load_file(filepath, retry_tries=0).read().decode("utf-8")
+    module = {}
+    javascript.eval_js(js_code)
+    threading.Event().wait(0.01)
+    return module_factory(mod_name, module["exports"])

pyproject.toml

@@ -56,6 +56,7 @@ ignore = [
 extend-exclude = [
   # Always.
   ".venv*",
+  "example_*",
   "tests/etc/functions_bad.py",
   # Temporary.
   "examples",

setup.py

@@ -16,6 +16,7 @@
     "funcy<3",
     "future>=0.18.0,<1",
     "importlib-metadata; python_version<'3.8'",
+    "javascript==1!1.0.1; python_version>='3.7'",
     "jinja2<4",
     "paho-mqtt<2",
     "requests<3",
@@ -195,6 +196,8 @@
         "Operating System :: POSIX",
         "Operating System :: Unix",
         "Operating System :: MacOS",
+        "Operating System :: Microsoft :: Windows",
+        "Programming Language :: JavaScript",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
@@ -223,7 +226,7 @@
     author="Jan-Piet Mens, Ben Jones, Andreas Motl",
     author_email="[email protected]",
     url="https://github.com/mqtt-tools/mqttwarn",
-    keywords="mqtt notification plugins data acquisition push transformation engine mosquitto",
+    keywords="mqtt notification plugins data acquisition push transformation engine mosquitto python javascript",
     packages=find_packages(),
     include_package_data=True,
     package_data={