Updated Input REST Souce Stage to have pipeline support

[spatil44]

Updated REST Source stages to support Pipeline mode.
This will close Feature Request #1234

@mdemoret-nv I have tested these changes on the SI detection NLP pipeline with REST Source and Sink stage, and it's working totally fine. Let me know if there is anything else that needs to be done to merge this PR, Happy to contribute.

I was testing SI Model workflow with REST stage using pipeline CLI, able to make it work after the above changes, the thought of contributing :)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mdemoret-nv]

/ok to test

[mdemoret-nv]

Approved. Thanks for your contribution!

[mdemoret-nv]

Approving again now that its not in draft

[dagardner-nv]

I'm not sure this is needed, when register_stage is called and modes is None then all modes are added for the stage.

[mdemoret-nv]

@spatil44 Looking at this again I agree with @dagardner-nv that it should already be working since the default is to use all modes. Can you double check that you cant use to-http, to-http-server, from-http and from-http-server? You can check whether the commands are available with the Morpheus CLI help commands. For example:

$ morpheus run pipeline-other --help
Usage: morpheus run pipeline-other [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

  Configure and run the pipeline. To configure the pipeline, list the stages in the order that data should flow. The output of each stage will become the input for the next stage. For example, to
  read, classify and write to a file, the following stages could be used

  pipeline from-file --filename=my_dataset.json deserialize preprocess inf-triton --model_name=my_model --server_url=localhost:8001 filter --threshold=0.5 to-file --filename=classifications.json

  Pipelines must follow a few rules: 1. Data must originate in a source stage. Current options are `from-file` or `from-kafka` 2. A `deserialize` stage must be placed between the source stages and
  the rest of the pipeline 3. Only one inference stage can be used. Zero is also fine 4. The following stages must come after an inference stage: `add-class`, `filter`, `gen-viz`

Options:
  --model_fea_length INTEGER RANGE
                                  Number of features trained in the model  [default: 1; x>=1]
  --label TEXT                    Specify output labels. Ignored when --labels_file is specified
  --labels_file DATA FILE         Specifies a file to read labels from in order to convert class IDs into labels. A label file is a simple text file where each line corresponds to a label.
  --viz_file FILE                 Save a visualization of the pipeline at the specified location
  --viz_direction [BT|LR|RL|TB]   Set the direction for the Graphviz pipeline diagram, ignored unless --viz_file is also specified.  [default: LR]
  --help                          Show this message and exit.

Commands:
  add-class         Add detected classifications to each message.
  add-scores        Add probability scores to each message.
  buffer            (Deprecated) Buffer results.
  delay             (Deprecated) Delay results for a certain duration.
  deserialize       Messages are logically partitioned based on the pipeline config's `pipeline_batch_size` parameter.
  dropna            Drop null data entries from a DataFrame.
  filter            Filter message by a classification threshold.
  from-file         Load messages from a file.
  from-http         Source stage that starts an HTTP server and listens for incoming requests on a specified endpoint.
  from-http-client  Source stage that polls a remote HTTP server for incoming data.
  from-kafka        Load messages from a Kafka cluster.
  from-rss          Load RSS feed items into a pandas DataFrame.
  inf-identity      Perform inference for testing that performs a no-op.
  inf-pytorch       Perform inference with PyTorch.
  inf-triton        Perform inference with Triton Inference Server.
  mlflow-drift      Report model drift statistics to ML Flow.
  monitor           Display throughput numbers at a specific point in the pipeline.
  serialize         Includes & excludes columns from messages.
  to-elasticsearch  This class writes the messages as documents to Elasticsearch.
  to-file           Write all messages to a file.
  to-http           Write all messages to an HTTP endpoint.
  to-http-server    Sink stage that starts an HTTP server and listens for incoming requests on a specified endpoint.
  to-kafka          Write all messages to a Kafka cluster.
  trigger           Buffer data until the previous stage has completed.
  validate          Validate pipeline output for testing.

[spatil44]

@mdemoret-nv those commands work with Morpheus CLI, Please feel free to close this PR in that case.

[spatil44]

Closing this MR refer #1234 for more details