Expand documentation

docs/documentation/ForDevs.md

@@ -5,6 +5,20 @@ description: ~
 ---
 # Motivation and Rationale
 
+So, in this section, I'm going to just give an overall view and present the main reasons for some design decisions that might help whoever wants to contribute to or fork this project. I'm not going into much detail, because I'll assume that if you're doing so, you must have some python background.
+
+## Imports
+
+The first thing you must have notice is the local generation of messages, that is, the `pyimc_generated` folder. It was implemented this way so that you can work with any IMC version, without changing the main library. Of course, this will not hold in case a big change happens, such as a change in the header fields, because the main library assumes this fixed header. This "messes up" the imports, since the python interpreter will have to import the library during runtime. It is nothing critical, but makes type checking and static code analysis more difficult.
+
+## asyncio and multiprocessing
+
+Working so intensively on low-level tasks such as serializing and CRC calculations is not exactly suited for Python. I would say that going back and forth from Python objects to byte strings is a big overhead. An attempt to alleviate this issue was the usage of the multiprocessing module, which didn't gave great results. It was a little slower than the current approach with asyncio. You can find some "deprecated" multiprocessing code along the way, so if you ever try to optimize it, just keep in mind that this approach was attempted.
+
+## Message classes and descriptor protocol
+
+You may find weird the usage of the descriptor protocol and the usage of an `Attributes` attribute. Well, the descriptor protocol is indeed mostly deprecated. Initially, I intended to use it to make safer applications (by checking the types of the message fields) but it turns out, in Python, you, as a library developer, can't prevent an error from happening. You can't reliably force/enforce anything on the user on the type level. "We are all consenting adults", they say, don't they? An error/exception when the user assign `int` where `str` is expected, whether thrown by me (by using descriptors) or by the serializer, makes no difference. The program will crash either way. So instead, I'm using descriptors just to give a better error message, that is, my approach became rather "to teach/help the user". That's also why `Attributes`, which is a class attribute (low memory cost for its benefits) exists.
+
 # Updating the package
 
 - You might need:

docs/documentation/ForUsers.md

@@ -6,7 +6,7 @@ description: ~
 
 First and foremost, please use a decent IDE. You hearing, Armin?
 
-I tried to shove documentation wherever possible, so that the IDE can give some hints of the type and the meaning of the variables. To do that, just hover your mouse over the thing you want some information. It may not always be helpful, but at least I tried. Also, the code is almost all type annotated, which may also help.
+I tried to shove documentation wherever possible, so that the IDE can give some hints of the type and the meaning of the variables. To do that, just hover your mouse over the thing you want some information. Some IDEs also give autocomplete suggestions, which can be used to the same end. It may not always be helpful, but at least I tried. Also, the code is almost all type annotated, which may also help.
 
 # Package structure
 The installed package contains 4 submodules, that can be divided in two groups, plus a module that will be generated locally. However, normally, you should only need `pyimclsts.network` and the module generated by `pyimclsts.extract`.
@@ -39,8 +39,76 @@ Lastly, when the `pyimclsts.extract` is run with `python3 -m pyimclsts.extract`,
 
 # Publisher-Subscriber model
 
-There are three main elements in this model: a subscriber, a publisher and a message bus. In short, a publisher application writes messages to a shared interface, also known as the message bus, and a subscriber application register with the broker the messages it wants to consume. So, in order to comply with this model, we need a message broker to receive messages from the network, execute the subscribed functions, and also distribute the messages. Additionally, the subscriber functions should be able to have state, run independently (not creating deadlocks nor race conditions), and send and receive messages.
+There are three main elements in this model: a subscriber, a publisher and a message bus. In short, a publisher application writes messages to a shared interface, also known as the message bus, and a subscriber application register with the broker the messages it wants to consume. The message broker, therefore, gathers the messages sent by the publishers and distribute them to the subscribers.
 
 ## The `subscriber`
 
-## The subscriber functions
+In this implementation, the `subscriber` class provides objects that register the subscriber functions. To instantiate it, we give it an interface from which it will read the messages and then we can give it the callbacks for each message we desired. (In a sense, it actually works as a message broker, but we kept this name since from the point of view of the LSTS systems, the applications written with this tools would be subscribers and the vehicles would be the publishers. And I'm lazy and don't want to change it now.)
+
+## The subscriber methods
+
+Besides some private methods (the ones that start with `_`, Python convention), the main subscriber methods are presented below:
+
+```python
+class subscriber:
+    ...
+    def subscribe_async(self, 
+                        callback : Callable[[_core.IMC_message, Callable[[_core.IMC_message], None]], None], 
+                        msg_id : Optional[Union[int, _core.IMC_message, str, _types.ModuleType]] = None, *, 
+                        src : Optional[str] = None, 
+                        src_ent : Optional[str] = None):
+        ...
+
+    def periodic_async(self, 
+                      callback : Callable[[_core.IMC_message], None], 
+                      period : float):
+        ...
+
+    def subscribe_mp(self, 
+                    callback : Callable[[_core.IMC_message, Callable[[_core.IMC_message], None]], None], 
+                    msg_id : Optional[Union[int, _core.IMC_message, str, _types.ModuleType]] = None, *, 
+                    src : Optional[str] = None, 
+                    src_ent : Optional[str] = None):
+        ...
+
+    def call_once(self, 
+                  callback : Callable[[Callable[[_core.IMC_message], None]], None], 
+                  delay : Optional[float] = None) -> None:
+        ...
+
+    def print_information(self) -> None:
+        '''Looks for (and asks for, when applicable) the first Announce and EntityList messages and print them.
+        
+        Executes no other subscription.
+        '''
+        ...
+
+    def stop(self) -> None:
+        ...
+
+    def run(self) -> None:
+        ...
+```
+
+To subscribe a function, there are 3 (+ 1 not yet implemented) possible ways: `subscribe_async`, `periodic_async`, `call_once` and `subscribe_mp` (not implemented). As the names suggest, `call_once` can be used to call a function once, optionally after a delay; `periodic_async` executes a callback every `period` seconds. `subscribe_async` executes the callback for every received message in `msg_id` and filters according to `src` and `src_ent`, if given. `msg_id` can be an `int` (the message id), the message class (or its instance) or a `str` (camel case) or a Python module (the files/modules inside the `category` folder) to specify a category of messages. `src` and `src_ent` are strings that indicate the vehicle and the entity inside a vehicle, for example, "lauv-xplore-1" and "TemperatureSensor".
+
+The subscribed functions must receive as arguments 1. A `send_callback`, and 2. A message (when applicable). The `send_callback` is nothing more than a function object of the method bound to the instance of the internal message broker of the subscriber. Is this greek? Let me clarify: Internally, the subscriber uses the given IO interface (file or TCP, for now) and creates a `message_broker`, which is used to manage (send and receive) messages. By using a `message_broker` we can internally use the same interface for both files or TCP. So, finally, the `send_callback` is simply a reference to the `.send()` method of this `message_broker`. You can use it as a normal function. <mark>Normally, the `src`, `src_ent`, `dst` and `dst_ent` are inferred from the IO interface, but you can use this function to overwrite them.</mark> Simply pass them as named arguments (as `int`s), for example, `send_callback(msg, dst=31)`. For more information regarding the message, please check [IMC Message](IMCMsg.html#overview).
+
+`run` and `stop` start and stop the event loop. That is, once `run()` is called, the application will be blocked as the control of the program will now be given to and managed by `subscriber`. To stop the event loop, you may pass the `.stop` callback itself to the instance to the subscriber. For example:
+
+```python
+if __name__ == '__main__':
+    conn = p.tcp_interface('localhost', 6006)
+    vehicle = FollowRef_Vehicle('lauv-xplore-1')
+
+    sub = p.subscriber(conn)
+    def stop_subscriber(send_callback):
+        sub.stop()
+    # or more neatly:
+    stop_subscriber = lambda _ : sub.stop()
+
+    # Set a delay
+    sub.call_once(stop_subscriber, 60)
+```
+
+`print_information` is just an utility function, that I wrote mainly for file reading or usage during simulations. It saves the list of subscribed functions, starts the event loop in search of an Announce and an EntityList messages, prints them, stops the event loop and restores the list of subscribed functions.

docs/documentation/IMCMsg.md

@@ -10,6 +10,8 @@ IMC abstracts hardware and communication differences, providing a shared set of
 
 # Parts of the Message
 
+In this section, we'll briefly take a look at how an IMC message is structured.
+
 ## Header:
 
 The message header is tuple of
@@ -44,4 +46,58 @@ To ensure accurate transportation, some field types may require special treatmen
 `message` is serialized by prepending a value of type uint16_t, representing the identification number of the message, to the serialized message payload. The special identification number 65535 must be used when no message is present. On deserialization the prepended value is used to retrieve the correct message identification number.
 
 * message-list
-`message-list` is serialized by prepending a value of type uint16_t, representing the number of messages in the list, to the serialized message payload. On deserialization the prepended value is used to retrieve the correct number of messages.
+`message-list` is serialized by prepending a value of type uint16_t, representing the number of messages in the list, to the serialized message payload. On deserialization the prepended value is used to retrieve the correct number of messages.
+
+# The IMC Message Python class
+
+By using the extract module as an executable, you can generate Python classes that represent IMC messages and can be utilized with the functions provide in this package.
+
+```shell
+$ python3 -m pyimclsts.extract
+```
+
+All the classes inherit from `base_message` class, which provides the methods that serialize (`pack`), tests for equality (`__eq__`), and pretty prints. As utility, it also has a method that gets the timestamp (`get_timestamp`). There is also an `IMC_message` class, which is empty, and exists only for type checking and avoiding cyclic references.
+
+All message classes have an `Attributes` attribute (a named tuple) that contains the basic message definition, as provided by XML file. Additionally, they have the message fields as attributes, a `_header` and a `_footer`, which are private and not supposed to be used by the end user. In particular, regarding the header, only the `src`, `src_ent`, `dst` and `dst_ent` fields can be defined by the user. To do so, these values must be passed to the `.pack` method or the the `send_callback` that is given to the the subscribed function (see [The subscriber methods](ForUsers.html#the-subscriber-methods)). Normally, this is inferred by the interface in use. Lastly, should a message define an enumeration or a bitfield, they will be included in the message class as a nested class.
+
+In the following example, we can see most of these features. Also, note that if an enumeration or bitfield is locally of globally defined in the XML, it will be indicated in its docstring.
+
+```python
+class DevCalibrationControl(_base.base_message):
+    '''Operation to perform. Enumerated (Local).
+
+       This message class contains the following fields and their respective types:
+    op : uint8_t, unit: Enumerated (Local)'''
+
+    class OP(_enum.IntEnum):
+        '''Full name: Operation
+        Prefix: DCAL'''
+
+        START = 0
+        '''Name: Start'''
+
+        STOP = 1
+        '''Name: Stop'''
+
+        STEP_NEXT = 2
+        '''Name: Perform Next Calibration Step'''
+
+        STEP_PREVIOUS = 3
+        '''Name: Perform Previous Calibration Step'''
+
+
+    __slots__ = ['_Attributes', '_header', '_footer', '_op']
+    Attributes = _base.MessageAttributes(description = "This message controls the calibration procedure of a given device. The destination device is selected using the destination entity identification number.", source = "vehicle,ccu", abbrev = "DevCalibrationControl", name = "Device Calibration Control", flags = None, usedby = None, stable = None, fields = ('op',), category = "Core", id = 12)
+
+    op = _base.mutable_attr({'name': 'Operation', 'type': 'uint8_t', 'unit': 'Enumerated', 'prefix': 'DCAL'}, "Operation to perform. Enumerated (Local).")
+    '''Operation to perform. Enumerated (Local). Type: uint8_t'''
+
+    def __init__(self, op = None):
+        '''Class constructor
+
+        Operation to perform. Enumerated (Local).
+
+       This message class contains the following fields and their respective types:
+    op : uint8_t, unit: Enumerated (Local)'''
+        self._op = op
+```