Merge pull request #14 from gadorlhiac/MNT/names_and_imports

MNT Naming and Import Clarifications

docs/adrs/README.md

@@ -3,13 +3,13 @@
 - Please refer to the `madr_template.md` for creating new ADRs. This template was adapted from the [MADR template](https://adr.github.io/madr/) (MIT License).
 - A table of ADRs is provided below.
 
-| ADR No. | Record Date | Title                                                                     | Status       |
-|:-------:|:-----------:|:--------------------------------------------------------------------------|:------------:|
-| 1       | 2023-11-06  | All analysis `Task`s inherit from a base class                            | **Accepted** |
-| 2       | 2023-11-06  | Analysis `Task` submission and communication is performed via `Executor`s | **Accepted** |
-| 3       | 2023-11-06  | `Executor`s will run all `Task`s via subprocess                           | **Proposed** |
-| 4       | 2023-11-06  | Airflow `Operator`s and LUTE `Executor`s are separate entities.           | **Proposed** |
-| 5       | 2023-12-06  | Task-Executor IPC is Managed by Communicator Objects                      | **Proposed** |
-| 6       | 2024-02-12  | Third-party Config Files Managed by Templates Rendered by `BinaryTask`s   | **Proposed** |
-| 7       | 2024-02-12  | `Task` Configuration is Stored in a Database Managed by `Executor`s       | **Proposed** |
-|         |             |                                                                           |              |
+| ADR No. | Record Date | Title                                                                       | Status       |
+|:-------:|:-----------:|:----------------------------------------------------------------------------|:------------:|
+| 1       | 2023-11-06  | All analysis `Task`s inherit from a base class                              | **Accepted** |
+| 2       | 2023-11-06  | Analysis `Task` submission and communication is performed via `Executor`s   | **Accepted** |
+| 3       | 2023-11-06  | `Executor`s will run all `Task`s via subprocess                             | **Proposed** |
+| 4       | 2023-11-06  | Airflow `Operator`s and LUTE `Executor`s are separate entities.             | **Proposed** |
+| 5       | 2023-12-06  | Task-Executor IPC is Managed by Communicator Objects                        | **Proposed** |
+| 6       | 2024-02-12  | Third-party Config Files Managed by Templates Rendered by `ThirdPartyTask`s | **Proposed** |
+| 7       | 2024-02-12  | `Task` Configuration is Stored in a Database Managed by `Executor`s         | **Proposed** |
+|         |             |                                                                             |              |

docs/adrs/adr-6.md

@@ -1,4 +1,4 @@
-# [ADR-6] Third-party Config Files Managed by Templates Rendered by `BinaryTask`s
+# [ADR-6] Third-party Config Files Managed by Templates Rendered by `ThirdPartyTask`s
 
 **Date:** 2024-02-12
 
@@ -12,7 +12,7 @@
 - Ideally all aspects of configuraiton could be managed from the single LUTE configuration file.
 
 ## Decision
-Templates will be used for the third party configuration files. A generic interface to heterogenous templates will be provided through a combination of pydantic models and the `BinaryTask` implementation. The pydantic models will label extra arguments to `BinaryTask`s as being `ThirdPartyParameters`. I.e. any extra parameters are considered to be for a templated configuration file. The `BinaryTask` will find the necessary template and render it if any extra parameters are found. This puts the burden of correct parsing on the template definition itself.
+Templates will be used for the third party configuration files. A generic interface to heterogenous templates will be provided through a combination of pydantic models and the `ThirdPartyTask` implementation. The pydantic models will label extra arguments to `ThirdPartyTask`s as being `TemplateParameters`. I.e. any extra parameters are considered to be for a templated configuration file. The `ThirdPartyTask` will find the necessary template and render it if any extra parameters are found. This puts the burden of correct parsing on the template definition itself.
 
 ### Decision Drivers
 * Need to be able to configure the necessary files from within the LUTE framework.
@@ -21,9 +21,9 @@ Templates will be used for the third party configuration files. A generic interf
   * Text substiution provides a means to do this.
 
 ### Considered Options
-* Separate configuration `Task` to be run before the main `BinaryTask`.
+* Separate configuration `Task` to be run before the main `ThirdPartyTask`.
 * Generate the configuration file in its entirety from within the `Task`.
-  * This removes the simplicity in allowing all `BinaryTask`s to be run as instances of a single class.
+  * This removes the simplicity in allowing all `ThirdPartyTask`s to be run as instances of a single class.
 
 ## Consequences
 * Can configure and run third party tasks which require the use of a configuration file.

docs/tutorial/new_task.md

@@ -21,9 +21,9 @@ A brief overview of parameters objects will be provided below. The following inf
 
 **`Task`s and `TaskParameter`s**
 
-All `Task`s have a corresponding `TaskParameters` object. These objects are linked **exclusively** by a named relationship. For a `Task` named `MyBinaryTask`, the parameters object **must** be named `MyBinaryTaskParameters`. For third-party `Task`s there are a number of additional requirements:
+All `Task`s have a corresponding `TaskParameters` object. These objects are linked **exclusively** by a named relationship. For a `Task` named `MyThirdPartyTask`, the parameters object **must** be named `MyThirdPartyTaskParameters`. For third-party `Task`s there are a number of additional requirements:
 - The model must inherit from a base class called `BaseBinaryParameters`.
-- The model must have one field specified called `executable`. The presence of this field indicates that the `Task` is a third-party `Task` and the specified executable must be called. This allows all third-party `Task`s to be defined exclusively by their parameters model. A single `BinaryTask` class handles execution of **all** third-party `Task`s.
+- The model must have one field specified called `executable`. The presence of this field indicates that the `Task` is a third-party `Task` and the specified executable must be called. This allows all third-party `Task`s to be defined exclusively by their parameters model. A single `ThirdPartyTask` class handles execution of **all** third-party `Task`s.
 
 All models are stored in `lute/io/models`. For any given `Task`, a new model can be added to an existing module contained in this directory or to a new module. If creating a new module, make sure to add an import statement to `lute.io.models.__init__`.
 
@@ -294,10 +294,10 @@ Task2Runner.shell_source("/sdf/group/lcls/ds/tools/new_task_setup.sh") # Will so
 Some third-party executables will require their own configuration files. These are often separate JSON or YAML files, although they can also be bash or Python scripts which are intended to be edited. Since LUTE requires its own configuration YAML file, it attempts to handle these cases by using Jinja templates. When wrapping a third-party task a template can also be provided - with small modifications to the `Task`'s pydantic model, LUTE can process special types of parameters to render them in the template. LUTE offloads all the template rendering to Jinja, making the required additions to the pydantic model small. On the other hand, it does require understanding the Jinja syntax, and the provision of a well-formatted template, to properly parse parameters. Some basic examples of this syntax will be shown below; however, it is recommended that the `Task` implementer refer to the [official Jinja documentation](https://jinja.palletsprojects.com/en/3.1.x/) for more information.
 
 LUTE provides two additional base models which are used for template parsing in conjunction with the primary `Task` model. These are:
-- `ThirdPartyParameters` objects which hold parameters which will be used to render a portion of a template.
+- `TemplateParameters` objects which hold parameters which will be used to render a portion of a template.
 - `TemplateConfig` objects which hold two strings: the name of the template file to use and the full path (including filename) of where to output the rendered result.
 
-`Task` models which inherit from the `BaseBinaryParameters` model, as all third-party `Task`s should, allow for extra arguments. LUTE will parse any extra arguments provided in the configuration YAML as `ThirdPartyParameters` objects automatically, which means that they do not need to be explicitly added to the pydantic model (although they can be). As such the **only** requirement on the Python-side when adding template rendering functionality to the `Task` is the addition of one parameter - an instance of `TemplateConfig`. The instance **MUST** be called `lute_template_cfg`.
+`Task` models which inherit from the `BaseBinaryParameters` model, as all third-party `Task`s should, allow for extra arguments. LUTE will parse any extra arguments provided in the configuration YAML as `TemplateParameters` objects automatically, which means that they do not need to be explicitly added to the pydantic model (although they can be). As such the **only** requirement on the Python-side when adding template rendering functionality to the `Task` is the addition of one parameter - an instance of `TemplateConfig`. The instance **MUST** be called `lute_template_cfg`.
 
 ```py
 from pydantic import Field, validator
@@ -381,7 +381,7 @@ We save this file as `jsonuser.json` in `config/templates`. Next, we will update
 
 from pydantic import Field, validator
 
-from .base import TemplateConfig #, ThirdPartyParameters
+from .base import TemplateConfig #, TemplateParameters
 
 class RunJsonUserParameters:
     executable: str = Field(
@@ -399,12 +399,12 @@ class RunJsonUserParameters:
         ),
         description="Template rendering configuration",
     )
-    # We do not need to include these ThirdPartyParameters, they will be added
+    # We do not need to include these TemplateParameters, they will be added
     # automatically if provided in the YAML
-    #str_var: Optional[ThirdPartyParameters]
-    #int_var: Optional[ThirdPartyParameters]
-    #p3_b: Optional[ThirdPartyParameters]
-    #val: Optional[ThirdPartyParameters]
+    #str_var: Optional[TemplateParameters]
+    #int_var: Optional[TemplateParameters]
+    #p3_b: Optional[TemplateParameters]
+    #val: Optional[TemplateParameters]
 
 
     # Tell LUTE to write the rendered template to the location provided with
@@ -429,7 +429,7 @@ RunJsonUser:
     val: 2 # Will substitute for "param4": [2, 2, 3] in the JSON
 ```
 
-If on the other hand, a user were to have an already valid JSON file, it is possible to turn off the template rendering. (ALL) Template variables (`ThirdPartyParameters`) are simply excluded from the configuration YAML.
+If on the other hand, a user were to have an already valid JSON file, it is possible to turn off the template rendering. (ALL) Template variables (`TemplateParameters`) are simply excluded from the configuration YAML.
 
 ```yaml
 RunJsonUser:

lute/io/models/base.py

@@ -10,7 +10,7 @@
     BaseBinaryParameters(TaskParameters): Base class for Third-party, binary
         executable Tasks.
 
-    ThirdPartyParameters: Dataclass to represent parameters of binary
+    TemplateParameters: Dataclass to represent parameters of binary
         (third-party) Tasks which are used for additional config files.
 
     TemplateConfig(BaseModel): Class for holding information on where templates
@@ -21,7 +21,7 @@
     "TaskParameters",
     "AnalysisHeader",
     "TemplateConfig",
-    "ThirdPartyParameters",
+    "TemplateParameters",
     "BaseBinaryParameters",
 ]
 __author__ = "Gabriel Dorlhiac"
@@ -99,7 +99,7 @@ class Config:
 
 
 @dataclass
-class ThirdPartyParameters:
+class TemplateParameters:
     """Class for representing parameters for third party configuration files.
 
     These parameters can represent arbitrary data types and are used in
@@ -137,7 +137,7 @@ class Config(TaskParameters.Config):
     def extra_fields_to_thirdparty(cls, values):
         for key in values:
             if key not in cls.__fields__:
-                values[key] = ThirdPartyParameters(values[key])
+                values[key] = TemplateParameters(values[key])
 
         return values
 

lute/io/models/smd.py

@@ -144,18 +144,18 @@ def use_producer(
             lute_template_cfg.output_path = values["producer"]
         return lute_template_cfg
 
-    # detnames: ThirdPartyParameters = ThirdPartyParameters({})
-    # epicsPV: ThirdPartyParameters = ThirdPartyParameters({})
-    # ttCalib: ThirdPartyParameters = ThirdPartyParameters({})
-    # aioParams: ThirdPartyParameters = ThirdPartyParameters({})
-    # getROIs: ThirdPartyParameters = ThirdPartyParameters({})
-    # getAzIntParams: ThirdPartyParameters = ThirdPartyParameters({})
-    # getAzIntPyFAIParams: ThirdPartyParameters = ThirdPartyParameters({})
-    # getPhotonsParams: ThirdPartyParameters = ThirdPartyParameters({})
-    # getDropletParams: ThirdPartyParameters = ThirdPartyParameters({})
-    # getDroplet2Photons: ThirdPartyParameters = ThirdPartyParameters({})
-    # getSvdParams: ThirdPartyParameters = ThirdPartyParameters({})
-    # getAutocorrParams: ThirdPartyParameters = ThirdPartyParameters({})
+    # detnames: TemplateParameters = TemplateParameters({})
+    # epicsPV: TemplateParameters = TemplateParameters({})
+    # ttCalib: TemplateParameters = TemplateParameters({})
+    # aioParams: TemplateParameters = TemplateParameters({})
+    # getROIs: TemplateParameters = TemplateParameters({})
+    # getAzIntParams: TemplateParameters = TemplateParameters({})
+    # getAzIntPyFAIParams: TemplateParameters = TemplateParameters({})
+    # getPhotonsParams: TemplateParameters = TemplateParameters({})
+    # getDropletParams: TemplateParameters = TemplateParameters({})
+    # getDroplet2Photons: TemplateParameters = TemplateParameters({})
+    # getSvdParams: TemplateParameters = TemplateParameters({})
+    # getAutocorrParams: TemplateParameters = TemplateParameters({})
 
 
 class FindOverlapXSSParameters(TaskParameters):

lute/tasks/__init__.py

@@ -0,0 +1,63 @@
+"""LUTE Tasks
+
+Functions:
+    import_task(task_name: str) -> Type[Task]: Provides conditional import of
+       Task's. This prevents import conflicts as Task's may be intended to run
+       in different environments.
+Exceptions:
+    TaskNotFoundError: Raised if
+"""
+
+from typing import Type
+from .task import Task
+
+
+class TaskNotFoundError(Exception):
+    """Exception raised if an unrecognized Task is requested.
+
+    The Task could be invalid (e.g. misspelled, nonexistent) or it may not have
+    been registered with the `import_task` function below.
+    """
+
+    ...
+
+
+def import_task(task_name: str) -> Type[Task]:
+    """Conditionally imports Task's to prevent environment conflicts.
+
+    Args:
+        task_name (str): The name of the Task to import.
+
+    Returns:
+        TaskType (Type[Task]): The requested Task class.
+
+    Raises:
+        TaskNotFoundError: Raised if the requested Task is unrecognized.
+            If the Task exits it may not have been registered.
+    """
+    if task_name == "Test":
+        from .test import Test
+
+        return Test
+
+    if task_name == "TestSocket":
+        from .test import TestSocket
+
+        return TestSocket
+
+    if task_name == "TestReadOutput":
+        from .test import TestReadOutput
+
+        return TestReadOutput
+
+    if task_name == "TestWriteOutput":
+        from .test import TestWriteOutput
+
+        return TestWriteOutput
+
+    if task_name == "FindPeaksPyAlgos":
+        from .sfx_find_peaks import FindPeaksPyAlgos
+
+        return FindPeaksPyAlgos
+
+    raise TaskNotFoundError

lute/tasks/dataclasses.py

@@ -6,7 +6,8 @@
     TaskStatus: Enumeration of possible Task statuses (running, pending, failed,
         etc.).
 
-    BinaryTask: Class to run a third-party executable binary as a `Task`.
+    DescribedAnalysis: Executor's description of a `Task` run (results,
+        parameters, env).
 """
 
 __all__ = ["TaskResult", "TaskStatus", "DescribedAnalysis"]

lute/tasks/task.py

@@ -3,10 +3,10 @@
 Classes:
     Task: Abstract base class from which all analysis tasks are derived.
 
-    BinaryTask: Class to run a third-party executable binary as a `Task`.
+    ThirdPartyTask: Class to run a third-party executable binary as a `Task`.
 """
 
-__all__ = ["Task", "TaskResult", "TaskStatus", "DescribedAnalysis", "BinaryTask"]
+__all__ = ["Task", "TaskResult", "TaskStatus", "DescribedAnalysis", "ThirdPartyTask"]
 __author__ = "Gabriel Dorlhiac"
 
 import time
@@ -19,7 +19,7 @@
 
 from ..io.models.base import (
     TaskParameters,
-    ThirdPartyParameters,
+    TemplateParameters,
     TemplateConfig,
     AnalysisHeader,
 )
@@ -159,7 +159,7 @@ def clean_up_timeout(self) -> None:
         ...
 
 
-class BinaryTask(Task):
+class ThirdPartyTask(Task):
     """A `Task` interface to analysis with binary executables."""
 
     def __init__(self, *, params: TaskParameters) -> None:
@@ -199,7 +199,7 @@ def _add_to_jinja_context(self, param_name: str, value: Any) -> None:
         """
         context_update: Dict[str, Any] = {param_name: value}
         if __debug__:
-            msg: Message = Message(contents=f"ThirdPartyParameters: {context_update}")
+            msg: Message = Message(contents=f"TemplateParameters: {context_update}")
             self._report_to_executor(msg)
         self._template_context.update(context_update)
 
@@ -273,8 +273,8 @@ def _pre_run(self) -> None:
                 or isinstance(self._task_parameters.__dict__[param], AnalysisHeader)
             ):
                 continue
-            if isinstance(self._task_parameters.__dict__[param], ThirdPartyParameters):
-                # ThirdPartyParameters objects have a single parameter `params`
+            if isinstance(self._task_parameters.__dict__[param], TemplateParameters):
+                # TemplateParameters objects have a single parameter `params`
                 self._add_to_jinja_context(param_name=param, value=value.params)
                 continue