added reviewer feedback

README.md

@@ -12,9 +12,11 @@ SPDX-License-Identifier: CC-BY-NC-ND-4.0
 
 ## Table of Content
 - [Temporal Behavior Trees: Robustness and Segmentation](#temporal-behavior-trees-robustness-and-segmentation)
-  - [Brief Summary of the Supported Operators](#brief-summary-of-the-supported-operators)
   - [Getting Started](#getting-started)
+  - [Folder Structure](#folder-structure)
+  - [Brief Summary of the Supported Operators](#brief-summary-of-the-supported-operators)
   - [Docker Environment](#docker-environment)
+  - [How to Interpret the Output Format](#how-to-interpret-the-output-format)
   - [Contributors](#contributors)
   - [Contributing](#contributing)
   - [Changes](#changes)
@@ -45,6 +47,57 @@ It is also useful to visualize the resulting segmentation, as shown below for th
 <img src="figs/Segmentation.png" width="400">
 </p>
 
+## Getting Started
+
+1. [Install Rust](https://www.rust-lang.org/)
+1. Specify a TBT, e.g., as done [here](src/tree/shipdeck_landing/lateral_maneuver.rs)
+1. [Provide a Trace by implementing ``get_trace``](src/tree/shipdeck_landing/get_trace_and_tree.rs)
+1. [Provide a Tree by implementing ``get_tree``](src/tree/shipdeck_landing/get_trace_and_tree.rs)
+1. [Replace the ``user_defined``-function by your own](src/main.rs)
+1. Call ``cargo build`` or ``cargo build --release`` 
+1. Call ``cargo run -- --help`` to get help on the command-line-usage
+1. Call ``cargo test`` to see if the tests are successful
+
+For instance:
+
+``cargo run --release -- -s -f .\res\logs_wind_front_Lateral\`` runs segmentation using subsampling on a provided logfile. 
+For this example, ``get_trace`` and ``get_tree`` is already provided.
+
+Using the [visualization script](scripts/visualize_ship_landing.py), we can easily plot a segmentation by, e.g., ``python visualize_ship_landing.py plot -b Lateral -s 5000 10000 20000 -e 0 -l ../res/logs_wind_front_Lateral/`` where ``5000, 10000, 20000`` represent beginning of segments (omitting 0), ``-b`` states the expected behavior and is used to plot the dotted lines, and ``-e`` represents the number of skipped entries due to subsampling.
+We can also replay the flight by, e.g.,  ``python visualize_ship_landing.py live -l ../res/logs_wind_front_Lateral/ -b Lateral -f 0.005 0.1 2.0``.
+
+For more information call ``python visualize_ship_landing.py --help``.
+
+## Folder Structure
+- [figs](figs) are resources used for this readme document
+- [res](res) contains the logfiles used in the HSCC paper
+  - The logfolder name specifies the wind direction (*front* or *side*) and the anticipated maneuver (*45Deg*, *Lateral*, *Oblique*, or *Straight*)
+  - Each flight consists of two csv-files: one for the ship and one for the aircraft
+  - The files contain the position, the velocity, and the angles for the ship and the aircraft
+- [scripts](scripts) provides auxiliary resources
+  - [Makefile](scripts/Makefile) is used for Rust profiling
+  - [clean.sh](scripts/clean.sh) is used for cleaning up the repository
+  - [infer_parameters_visualization.py](scripts/infer_parameters_visualization.py) is used for the [run.sh](scripts/run.sh) script to extract the segments from the produced output files
+  - [run.sh](scripts/run.sh) is a script that executes our segmentation tool on all the available [logfiles](res) and produces png files to further analyze ([infer_parameters_visualization.py](scripts/infer_parameters_visualization.py))
+  - [visualize_ship_landing.py](scripts/visualize_ship_landing.py) is a script that is used to produced the png files that show the flight and the computed segments
+- [src](src) contains the source code  
+    - [lib.rs](src/lib.rs) provides a package of TBTs functions that can be used by others 
+      - It requires a user to provide two functions [get_trace()](src/lib.rs) and [get_tree()](src/lib.rs)
+      - This repository provides these functions for the ship landing: [tree/](src/tree/)
+    - [main.rs](src/main.rs) is an example that uses [lib.rs](src/lib.rs) and the user-defined functions [tree/](src/tree/)
+    - [stl.rs](src/stl.rs) provides the syntax and semantics for STL formulas
+    - [behaviortree.rs](src/behaviortree.rs) provides the syntax and semantics for TBTs
+    - [command_line_parser.rs](src/command_line_parser.rs) is used to interface with the command line
+    - [csv_reader.rs](src/csv_reader.rs) represent auxiliary functions such as reading a csv-file
+    - [table.rs](src/table.rs) represents the main data structure for the dynamic programming
+    - [test.rs](src/tests.rs) contains multiple test cases that can be executed to test whether the compilation works
+    - [tree/](src/tree/) is an example implementation for the *UserProvidedFunctions* required by [lib.rs](src/lib.rs)
+      - [atomics/](src/tree/) are implemented function that take trace data (eg provided by reading a csv-file) and output a robustness verdict.
+      - [*_maneuver.rs](src/tree/) are instances of TBTs using [behaviortree.rs](src/behaviortree.rs)
+- [Dockerfile](Dockerfile) just c/p the whole repository and builds it to produce a docker container that then can run [run.sh](scripts/run.sh) to procude the HSCC artifacts
+
+> To use the TBT tool for a different use-case, a user needs to provide the *UserProvidedFunction* ([get_trace()](src/lib.rs) and [get_tree()](src/lib.rs)) similar to what has been done here for the ship landing ([tree/](src/tree/)). I.e., he/she needs to extract logdata into a *Trace* struct and needs to build the TBT.
+
 ## Brief Summary of the Supported Operators
 
 TBT ``T:=``
@@ -72,39 +125,73 @@ The TBT operators are defined [here](src/behaviortree.rs) and the STL operators
 
 For more details, we refer to the paper. TODO: add paper link here
 
-## Getting Started
-
-1. [Install Rust](https://www.rust-lang.org/)
-1. Specify a TBT, e.g., as done [here](src/tree/shipdeck_landing/lateral_maneuver.rs)
-1. [Provide a Trace by implementing ``get_trace``](src/tree/shipdeck_landing/get_trace_and_tree.rs)
-1. [Provide a Tree by implementing ``get_tree``](src/tree/shipdeck_landing/get_trace_and_tree.rs)
-1. [Replace the ``user_defined``-function by your own](src/main.rs)
-1. Call ``cargo build`` or ``cargo build --release`` 
-1. Call ``cargo run -- --help`` to get help on the command-line-usage
-1. Call ``cargo test`` to see if the tests are successful
-
-For instance:
-
-``cargo run --release -- -s -f .\res\logs_wind_front_Lateral\`` runs segmentation using subsampling on a provided logfile. 
-For this example, ``get_trace`` and ``get_tree`` is already provided.
-
-Using the [visualization script](scripts/visualize_ship_landing.py), we can easily plot a segmentation by, e.g., ``python visualize_ship_landing.py plot -b Lateral -s 5000 10000 20000 -e 0 -l ../res/logs_wind_front_Lateral/`` where ``5000, 10000, 20000`` represent beginning of segments (omitting 0), ``-b`` states the expected behavior and is used to plot the dotted lines, and ``-e`` represents the number of skipped entries due to subsampling.
-We can also replay the flight by, e.g.,  ``python visualize_ship_landing.py live -l ../res/logs_wind_front_Lateral/ -b Lateral -f 0.005 0.1 2.0``.
-
-For more information call ``python visualize_ship_landing.py --help``.
-
 ## Docker Environment
 1. Install [Docker](https://docs.docker.com/engine/install/)
 2. Builder Docker Image: ``docker build -t tbt .``
 3. Run Docker Container: ``docker run -it --rm --name tbt-container tbt bash``
 4. To test if the container is working reproduce paper results by (being in the docker bash):
-   - Run ``. scripts/run.sh`` that takes all logfiles and computes the segmentation
+   - Run ``. scripts/run.sh`` that takes all logfiles and computes the segmentation.    
+      - The script calls the tool as defined [here](#getting-started) for each logfolder that exists in [res](res).
+      - The results for each run are stored in the respective logfolder.
    - Check results of each logfiles that are located in the following subfolder: ``cd ./res/``
-   - Within you will find ``subsampling_result.txt`` and ``subsamplingAndLazy_result.txt``, as well `.png`-plots for each of them.
+   - The files in the folders [res/<folder\>](res) are called ``subsampling_result.txt`` and ``subsamplingAndLazy_result.txt``.
+   - Besides the result-files, for each segmentation, the script produces a `.png`-plot. Every `.png`-plot that has a name that ends with `aX`, where `X` is a number, represents an alternative segmentations where the number corresponds to the alternative in the result-file.  For instance, Figure 9 of the HSCC paper can then be found [here](res/logs_wind_front_Oblique/subsampling_result_a3.png).
    - (Optional) To display plots copy them from the docker container to your host machine; dont use the docker bash.
      -  ``docker cp <container_id>:<location_png/results> <location_to_be_stored>`` (copy whole folder or individual files), e.g., ``docker cp e7ba94d69e94:/app/res ./docker``
      -  to get container_id call ``docker ps``
 
+> Note that the Dockerfile uses multiple stages. The first stage builds the executable using rust/cargo and the second stage uses a debian environment to execute it. Therefore, there are no cargo-commands available in the container while running. Line 7 in run.sh will fail, but since it is prebuilded during the build stage, the consecutive command will work.
+
+
+## How to Interpret the Output Format
+Running e.g. `cargo run --release -- -l -c -s -f /<your-foulder>/TBT-Segmentation/res/logs_wind_front_Lateral/` produces an output that contains the following lines:
+
+> SETTING: <br>
+> Logfile: /root/canbedeleted/TBT-Segmentation/res/logs_wind_front_Lateral/ <br>
+
+represents the logfile name.
+
+> Approximations: (lazy evaluation= true, subsampling = true(delta: 100))
+
+show which approximations are enabled.
+In this case, lazy evauation and subsampling with a delta of 100 are enabled.
+
+> Trace length: 252
+
+provides that the length of the trace is 252 after subsampling. 
+I.e. the original file has >25.000 entries.
+
+> Temporal behavior tree: <br>
+  Sequence(22)[
+    Fallback(20)[ <...> ]]
+
+shows a pretty print of the used TBT with its node identifiers.
+Here, the root node has ID 22.
+
+> Created tree table with 733,194 entries. <br>
+Created formula table with 828,828 entries.
+
+are information on the table used for dynamic programming.
+
+> Statistics: Robustness value is 0.05925286 with 3,277,611 total tree lookups and 1,503,504 formula lookups
+
+provides information how effective dynamic programming was.
+
+> Get segmentation after 0 seconds.
+
+states that it took 0 seconds to compute a segmentation.
+
+> Approximate segmentation with robustness 0.05925286 and subsampling delta of 0.5564443 is:
+
+is the beginning of the segmentation. The following lines provide information on the segments.
+
+> lower:          0   upper:         78   value:      0.31250286  segment: Leaf(0 move_to_position_lateral)
+
+represent one segment.
+It states that the leaf node `move_to_position_lateral` was assigned to the segment that begins at index 0 and ends at index 78. 
+Further its robustness value is 0.31, i.e., the trace segment did satisfy this node.
+
+
 ## Contributors
 - Sebastian Schirmer
 

scripts/infer_parameters_visualization.py

@@ -11,6 +11,7 @@ def build_call(filename):
     with open(filename, 'r') as f:
         read_result = False
         skip = True
+        all_segments = []
         segments = []
         skipped_entries = 0
         for line in f:
@@ -29,7 +30,9 @@ def build_call(filename):
                         print(f"Something is of, expected {pattern} but was: {line}")
                         exit()
                 else:
-                   break
+                    if segments != []:
+                        all_segments.append(segments)
+                    segments = []
             else:
                 if line.strip().startswith('Is greedy:'):
                     pattern = r"(.*)\((\d+)\)(.*)"
@@ -39,10 +42,6 @@ def build_call(filename):
                 if line.startswith('Get segmentation after'):
                     read_result = True
                     skip = True
-        segment_str = []
-        for i, (_, upper) in enumerate(segments):
-            if i != len(segments)-1:
-                segment_str.append(str(upper))
         behavior = None
         if 'Lateral' in filename:
             behavior = 'Lateral'
@@ -54,16 +53,25 @@ def build_call(filename):
             behavior = 'Straight'
         else:
             print(f"Unexpected logfile, expected one of the bahviors in the filename but wasnt: {filename}")
-
-        output_location = filename[:-4]
-        directory = os.path.dirname(filename)
-        call = ["python3", "visualize_ship_landing.py", f'-l' ,directory, '-b' ,f'{behavior}', '-s']
-        for segment in segment_str:
-            call.append(segment)
-        call = call + ['-p', output_location, '-e', str(skipped_entries), 'plot']
-        return call
-
+
+        calls = []
+        for i, segments in enumerate(all_segments):
+            output_location = filename[:-4] + "_best"
+            if i != 0:
+                output_location = filename[:-4] + "_a" + str(i)
+            directory = os.path.dirname(filename)
+            call = ["python3", "visualize_ship_landing.py", f'-l' ,directory, '-b' ,f'{behavior}', '-s']
+            segment_str = []
+            for i, (_, upper) in enumerate(segments):
+                if i != len(segments)-1:
+                    segment_str.append(str(upper))
+            for segment in segment_str:
+                call.append(segment)
+            call = call + ['-p', output_location, '-e', str(skipped_entries), 'plot']
+            calls.append(call)
+        return calls
 
 if __name__ == "__main__":
-    call  = build_call(sys.argv[1])
-    subprocess.run(call)
+    calls  = build_call(sys.argv[1])
+    for call in calls:
+        subprocess.run(call)