Add StreamCategoryFilter and stream_category to StreamDTO

[kingzacko1]

Some initial code implementation to allow for Stream categories in support of Graylog2/graylog-plugin-enterprise#7945 and specifically Graylog2/graylog-plugin-enterprise#7926.

Description

• Adds categories field to StreamDTO
• Adds StreamCategoryFilter
• Adds logic in StreamServiceImpl to map stream categories to stream IDs
• Adds logic in Query to treat a StreamCategoryFilter similar to having a StreamFilter set
• Remaps StreamCategoryFilters to the appropriate StreamFilters during search normalization

This works specifically on queries from the search bar hitting the Search API endpoints. Widgets will be handled in a subsequent PR. I am mostly looking for feedback on if this stream category -> stream ID mapping as part of search normalization is a viable option and, if so, anywhere that I may have missed accounting for stream categories.

Motivation and Context

Graylog2/graylog-plugin-enterprise#7926

How Has This Been Tested?

Testing from the API browser is a bit involved at the moment, but it looks something like this:

• Manually set a category on a stream(s): db.streams.updateOne({title:'Random'}, {$set:{categories: ['firewall']}})
• Create a search with a StreamCategoryFilter using the /views/search API endpoint. The following example is for 200 response codes from the random HTTP input:

{
  "queries": [
    {
      "query": {
        "type": "elasticsearch",
        "query_string": "http_response_code:200"
      },
      "timerange": {
        "type": "relative",
        "from": 300
      },
      "filter": {
        "type": "or",
        "filters": [
          {
            "type": "stream_category",
            "category": "firewall"
          }
        ]
      },
      "filters": [],
      "search_types": [
        {
          "name": "chart",
          "series": [
            {
              "id": "count()",
              "type": "count"
            }
          ],
          "rollup": true,
          "row_groups": [
            {
              "type": "time",
              "fields": [
                "timestamp"
              ],
              "interval": {
                "type": "auto",
                "scaling": 1
              }
            }
          ],
          "type": "pivot",
          "id": "da992275-2931-46db-8adc-7041ec1ab553",
          "filters": [],
          "column_groups": [],
          "sort": []
        },
        {
          "offset": 0,
          "decorators": [],
          "type": "messages",
          "id": "e343b2ac-1175-4f6a-8b79-96149ccf9c33",
          "limit": 150,
          "filters": [],
          "sort": [
            {
              "field": "timestamp",
              "order": "DESC"
            }
          ]
        }
      ]
    }
  ],
  "parameters": []
}

• From that response, grab the top level ID and the query ID and paste it into the /views/search/{id}/execute API endpoint. The top level ID will go into the {id} path parameter and the body will be:

{
  "global_override": {
    "keep_queries": [
      "UUID-of-the-query-intheresponse"
    ]
  },
  "parameter_bindings": {}
}

• From that response, grab the id and executing_node to be used in the /views/searchjobs/{nodeId}/{jobId}/status API endpoint. The id corresponds to the jobId and executing_node to nodeId. This call should return the search query containing the StreamCategoryFilter and search results from the stream with the category assigned in the first step.

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[ryan-carroll-graylog]

Looks like a solid start and tests successfully for me! Certainly touches quite a lot pieces of the application so nice job navigating that.

Left one comment on a thought I had, but definitely curious to hear feedback from those that have more experience with search.

[ryan-carroll-graylog]

Looks good to me and tests successfully resolving streams from stream categories and filtering out streams that a search user does not have access to.

I like this CommandFactory/PluggableSearchNormalization approach.

[kingzacko1]

@ryan-carroll-graylog I made one more change moving the CommandFactory changes up to the MessagesResource that has access to the SearchUser again to respect stream permissions.

[ryan-carroll-graylog]

@ryan-carroll-graylog I made one more change moving the CommandFactory changes up to the MessagesResource that has access to the SearchUser again to respect stream permissions.

Right on, tests successfully with the latest as well.

[ryan-carroll-graylog]

Ran through again with the latest and all looks good and is testing successfully for me.

Left a couple minor nit comments for consideration but otherwise this is looks solid from my perspective.

[danotorrey]

LGTM and tests successfully!

[luk-kaminski]

Thanks a lot!!!