This covers more advanced CLI use cases and workflows. To get started with the CLI, reference the README. Basic use of the CLI such as creating a rule or testing are referenced in the contribution guide.
CLI commands which are tied to Kibana and Elasticsearch are capable of parsing auth-related keyword args from a config file or environment variables.
If a value is set in multiple places, such as config file and environment variable, the order of precedence will be as follows:
- explicitly passed args (such as
--user joe) - environment variables
- config values
- prompt (this only applies to certain values)
In the root directory of this repo, create the file .detection-rules-cfg.json and add relevant values
Currently supported arguments:
- elasticsearch_url
- kibana_url
- cloud_id
- *_username (kibana and es)
- *_password (kibana and es)
Environment variables using the argument format: DR_<UPPERCASED_ARG_NAME> will be parsed in commands which expect it.
EX: DR_USER=joe
Using the environment variable DR_BYPASS_NOTE_VALIDATION_AND_PARSE will bypass the Detection Rules validation on the note field in toml files.
Using the environment variable DR_BYPASS_BBR_LOOKBACK_VALIDATION will bypass the Detection Rules lookback and interval validation
on the building block rules.
Using the environment variable DR_BYPASS_TAGS_VALIDATION will bypass the Detection Rules Unit Tests on the tags field in toml files.
You can import rules into the repo using the create-rule or import-rules commands. Both of these commands will
require that the rules are schema-compliant and able to pass full validation. The biggest benefit to using these
commands is that they will strip* additional fields** and prompt for missing required
fields.
Alternatively, you can manually place rule files in the directory and run tests to validate as well.
* Note: This is currently limited to flat fields and may not apply to nested values.
** Note: Additional fields are based on the current schema at the time the command is used.
Usage: detection_rules create-rule [OPTIONS] PATH
Create a detection rule.
Options:
-c, --config FILE Rule or config file
--required-only Only prompt for required fields
-t, --rule-type [machine_learning|query|threshold]
Type of rule to create
-h, --help Show this message and exit.This command will allow you to pass a rule file using the -c/--config parameter. This is limited to one rule at a time
and will accept any valid rule in the following formats:
- toml
- json
- yaml (yup)
- ndjson (as long as it contains only a single rule and has the extension
.ndjsonor.jsonl)
Usage: detection_rules import-rules [OPTIONS] [INPUT_FILE]...
Import rules from json, toml, or Kibana exported rule file(s).
Options:
-d, --directory DIRECTORY Load files from a directory
-h, --help Show this message and exit.The primary advantage of using this command is the ability to import multiple rules at once. Multiple rule paths can be
specified explicitly with unlimited arguments, recursively within a directory using -d/--directory*, or
a combination of both.
In addition to the formats mentioned using create-rule, this will also accept an .ndjson/jsonl file
containing multiple rules (as would be the case with a bulk export).
This will also strip additional fields and prompt for missing required fields.
* Note: This will attempt to parse ALL files recursively within a specified directory.
Commands which connect to Elasticsearch or Kibana are embedded under the subcommands:
- es
- kibana
These command groups will leverage their respective clients and will automatically use parsed config options if defined, otherwise arguments should be passed to the sub-command as:
python -m detection-rules kibana -u <username> -p <password> upload-rule <...>
python -m detection_rules es -h
Usage: detection_rules es [OPTIONS] COMMAND [ARGS]...
Commands for integrating with Elasticsearch.
Options:
-et, --timeout INTEGER Timeout for elasticsearch client
-ep, --es-password TEXT
-eu, --es-user TEXT
--cloud-id TEXT
-e, --elasticsearch-url TEXT
-h, --help Show this message and exit.
Commands:
collect-events Collect events from Elasticsearch.Providers are the name that Elastic Cloud uses to configure authentication in Kibana. When we create deployment, Elastic Cloud configures two providers by default: basic/cloud-basic and saml/cloud-saml (for SSO).
python -m detection_rules kibana -h
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
Usage: detection_rules kibana [OPTIONS] COMMAND [ARGS]...
Commands for integrating with Kibana.
Options:
--ignore-ssl-errors TEXT
--space TEXT Kibana space
--provider-name TEXT For cloud deployments, Elastic Cloud configures
two providers by default: cloud-basic and
cloud-saml (for SSO)
--provider-type TEXT For cloud deployments, Elastic Cloud configures
two providers by default: basic and saml (for
SSO)
-ku, --kibana-user TEXT
--kibana-url TEXT
-kp, --kibana-password TEXT
-kc, --kibana-cookie TEXT Cookie from an authed session
--cloud-id TEXT ID of the cloud instance. Defaults the cloud
provider to cloud-basic if this option is
supplied
-h, --help Show this message and exit.
Commands:
search-alerts Search detection engine alerts with KQL.
upload-rule Upload a list of rule .toml files to Kibana.Alerts stored in Kibana can be quickly be identified by searching with the search-alerts command.
python -m detection_rules kibana search-alerts -h
Kibana client:
Options:
--ignore-ssl-errors TEXT
--space TEXT Kibana space
--provider-name TEXT For cloud deployments, Elastic Cloud configures
two providers by default: cloud-basic and
cloud-saml (for SSO)
--provider-type TEXT For cloud deployments, Elastic Cloud configures
two providers by default: basic and saml (for
SSO)
-ku, --kibana-user TEXT
--kibana-url TEXT
-kp, --kibana-password TEXT
-kc, --kibana-cookie TEXT Cookie from an authed session
--cloud-id TEXT ID of the cloud instance. Defaults the cloud
provider to cloud-basic if this option is
supplied
Usage: detection_rules kibana search-alerts [OPTIONS] [QUERY]
Search detection engine alerts with KQL.
Options:
-d, --date-range <TEXT TEXT>...
Date range to scope search
-c, --columns TEXT Columns to display in table
-e, --extend If columns are specified, extend the
original columns
-h, --help Show this message and exit.Running the following command will print out a table showing any alerts that have been generated recently.
python3 -m detection_rules kibana --provider-name cloud-basic --kibana-url <url> --kibana-user <username> --kibana-password <password> search-alerts
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
===================================================================================================================================
host rule
hostname name @timestamp
===================================================================================================================================
stryker-malwares-MacBook-Pro.local Sudo Heap-Based Buffer Overflow Attempt 2022-06-21T14:08:34.288Z
stryker-malwares-MacBook-Pro.local Suspicious Automator Workflows Execution 2022-06-21T13:58:30.857Z
stryker-malwares-MacBook-Pro.local Privilege Escalation Enumeration via LinPEAS 2022-06-21T13:33:18.218Z
stryker-malwares-MacBook-Pro.local Privilege Escalation Enumeration via LinPEAS 2022-06-21T13:28:14.685Z
stryker-malwares-MacBook-Pro.local Potential Reverse Shell Activity via Terminal 2022-06-21T12:53:00.234Z
stryker-malwares-MacBook-Pro.local Potential Reverse Shell Activity via Terminal 2022-06-21T12:53:00.237Z
stryker-malwares-MacBook-Pro.local Potential Kerberos Attack via Bifrost 2022-06-20T20:33:53.810Z
stryker-malwares-MacBook-Pro.local Potential Kerberos Attack via Bifrost 2022-06-20T20:33:53.813Z
stryker-malwares-MacBook-Pro.local Potential Privilege Escalation via Root Crontab File Modification 2022-06-20T20:23:50.557Z
stryker-malwares-MacBook-Pro.local Download and Execution of JavaScript Payload 2022-06-20T20:18:46.211Z
===================================================================================================================================
Toml formatted rule files can be uploaded as custom rules using the kibana upload-rule command. To upload more than one
file, specify multiple files at a time as individual args. This command is meant to support uploading and testing of
rules and is not intended for production use in its current state.
python -m detection_rules kibana upload-rule -h
Kibana client:
Options:
--space TEXT Kibana space
-kp, --kibana-password TEXT
-ku, --kibana-user TEXT
--cloud-id TEXT
-k, --kibana-url TEXT
Usage: detection_rules kibana upload-rule [OPTIONS]
Upload a list of rule .toml files to Kibana.
Options:
-f, --rule-file FILE
-d, --directory DIRECTORY Recursively export rules from a directory
-id, --rule-id TEXT
-r, --replace-id Replace rule IDs with new IDs before export
-h, --help Show this message and exit.
(detection-rules-build) (base) ➜ detection-rules git:(rule-loader) ✗Alternatively, rules can be exported into a consolidated ndjson file which can be imported in the Kibana security app directly.
Usage: detection_rules export-rules [OPTIONS]
Export rule(s) into an importable ndjson file.
Options:
-f, --rule-file FILE
-d, --directory DIRECTORY Recursively export rules from a directory
-id, --rule-id TEXT
-o, --outfile FILE Name of file for exported rules
-r, --replace-id Replace rule IDs with new IDs before export
--stack-version [7.8|7.9|7.10|7.11|7.12]
Downgrade a rule version to be compatible
with older instances of Kibana
-s, --skip-unsupported If `--stack-version` is passed, skip rule
types which are unsupported (an error will
be raised otherwise)
-h, --help Show this message and exit.*To load a custom rule, the proper index must be setup first. The simplest way to do this is to click
the Load prebuilt detection rules and timeline templates button on the detections page in the Kibana security app.
Importing rules will convert from any supported format to toml. Additionally, the
command view-rule will also allow you to view a converted rule without importing it by specifying the --rule-format flag.
To view a rule in JSON format, you can also use the view-rule command with the --api-format flag, which is the default.
(See the note on the JSON formatted rules and versioning)
The rule toml files exist slightly different than they do in their final state as a JSON file in Kibana. The files are
white space stripped, normalized, sorted, and indented, prior to their json conversion. Everything within the metadata
table is also stripped out, as this is meant to be used only in the context of this repository and not in Kibana..
Additionally, the version of the rule is added to the file prior to exporting it. This is done to restrict version bumps
to occur intentionally right before we create a release. Versions are auto-incremented based on detected changes in
rules. This is based on the hash of the rule in the following format:
- sorted json
- serialized
- b64 encoded
- sha256 hash
As a result, all cases where rules are shown or converted to JSON are not just simple conversions from TOML.
Most of the CLI errors will print a concise, user friendly error. To enable debug mode and see full error stacktraces,
you can define "debug": true in your config file, or run python -m detection-rules -d <commands...>.
Precedence goes to the flag over the config file, so if debug is enabled in your config and you run
python -m detection-rules --no-debug, debugging will be disabled.
A transform is any data that will be incorporated into existing rule fields at build time, from within the
TOMLRuleContents.to_dict method. How to process each transform should be defined within the Transform class as a
method specific to the transform type.
This applies to osquery and insights for the moment but could expand in the future.
(venv38) ➜ detection-rules-fork git:(2597-validate-osquery-insights) python -m detection_rules dev transforms -h
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
Usage: detection_rules dev transforms [OPTIONS] COMMAND [ARGS]...
Commands for managing TOML [transform].
Options:
-h, --help Show this message and exit.
Commands:
guide-plugin-convert Convert investigation guide plugin format to toml
guide-plugin-to-rule Convert investigation guide plugin format to toml
guide-plugin-convert will print out the formatted toml.
(venv38) ➜ detection-rules-fork git:(2597-validate-osquery-insights) python -m detection_rules dev transforms guide-plugin-convert
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
Enter plugin contents []: !{osquery{"query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services\nWHERE NOT (user_account LIKE \"%LocalSystem\" OR user_account LIKE \"%LocalService\" OR user_account LIKE \"%NetworkService\" OR user_account == null)","label":"label2","ecs_mapping":{"labels":{"field":"description"},"agent.build.original":{"value":"fast"}}}}
[transform]
[[transform.osquery]]
query = "SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services\nWHERE NOT (user_account LIKE \"%LocalSystem\" OR user_account LIKE \"%LocalService\" OR user_account LIKE \"%NetworkService\" OR user_account == null)"
label = "label2"
[transform.osquery.ecs_mapping]
[transform.osquery.ecs_mapping.labels]
field = "description"
[transform.osquery.ecs_mapping."agent.build.original"]
value = "fast"
The easiest way to update a rule with existing transform entries is to use guide-plugin-convert and manually add it
to the rule.