Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

security-service-integrations: retain event.original value upon pipeline errors #12067

Open
efd6 opened this issue Dec 10, 2024 · 1 comment
Open

security-service-integrations: retain event.original value upon pipeline errors #12067

efd6 opened this issue Dec 10, 2024 · 1 comment
Assignees
@efd6
Labels
enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]

Comments

@efd6
Copy link
Contributor

efd6 commented Dec 10, 2024 •
edited
Loading

This is the security-service-integrations sub issue for #12045.

  • add preserve_original_event in primary on_failure handles ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12046

  • ensure no remove event.original processors remain ssi_all: do not remove event.original in main ingest pipeline #12076

     for f in $(
 	(
 		for p in $(
 			yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
 			| grep -v -- ---
 		); do
 			find packages/$p -name default.yml
 		done
 	)|sort|uniq
 ); do
 	yq -o=json $f|quatch -l $f -p '{"processors":{"remove":{"field":["event.original"]}}}'
 done

    quatch

    • packages/bitdefender/data_stream/push_notifications/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/collection/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/group/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/member/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/policy/elasticsearch/ingest_pipeline/default.yml
    • packages/cisco_meraki/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/darktrace/data_stream/ai_analyst_alert/elasticsearch/ingest_pipeline/default.yml
    • packages/darktrace/data_stream/model_breach_alert/elasticsearch/ingest_pipeline/default.yml
    • packages/darktrace/data_stream/system_status_alert/elasticsearch/ingest_pipeline/default.yml
    • packages/eset_protect/data_stream/detection/elasticsearch/ingest_pipeline/default.yml
    • packages/eset_protect/data_stream/device_task/elasticsearch/ingest_pipeline/default.yml
    • packages/eset_protect/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/f5/data_stream/bigipafm/elasticsearch/ingest_pipeline/default.yml
    • packages/f5/data_stream/bigipapm/elasticsearch/ingest_pipeline/default.yml
    • packages/forcepoint_web/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_access/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_authentication/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_config/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_core/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_access/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_authentication/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_config/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_core/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_sync/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/code_scanning/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/dependabot/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/issues/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/secret_scanning/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/pages/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/production/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/sidekiq/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/finding/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/source/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/access_transparency/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/admin/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/context_aware_access/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/device/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/drive/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/gcp/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/group_enterprise/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/groups/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/login/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/rules/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/saml/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/token/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/user_accounts/elasticsearch/ingest_pipeline/default.yml
    • packages/imperva_cloud_waf/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_bloxone_ddi/data_stream/dhcp_lease/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_bloxone_ddi/data_stream/dns_config/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_bloxone_ddi/data_stream/dns_data/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_nios/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/telemetry/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/telemetry_legacy/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/web_threat_events/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/web_traffic_events/elasticsearch/ingest_pipeline/default.yml
    • packages/jumpcloud/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/keycloak/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/lastpass/data_stream/detailed_shared_folder/elasticsearch/ingest_pipeline/default.yml
    • packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml
    • packages/lastpass/data_stream/user/elasticsearch/ingest_pipeline/default.yml
    • packages/lyve_cloud/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/mattermost/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/menlo/data_stream/dlp/elasticsearch/ingest_pipeline/default.yml
    • packages/menlo/data_stream/web/elasticsearch/ingest_pipeline/default.yml
    • packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/microsoft_exchange_online_message_trace/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/archive_search_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/dlp_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/message_release_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/threat_intel_malware_customer/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/threat_intel_malware_grid/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/ttp_ap_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/ttp_ip_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/ttp_url_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/netskope/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/netskope/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/okta/data_stream/system/elasticsearch/ingest_pipeline/default.yml
    • packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/panw_cortex_xdr/data_stream/incidents/elasticsearch/ingest_pipeline/default.yml
    • packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/pps/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_access/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/host/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/host_profile/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/incident_audit/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/clicks_blocked/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/clicks_permitted/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/message_blocked/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/message_delivered/elasticsearch/ingest_pipeline/default.yml
    • packages/pulse_connect_secure/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/qualys_vmdr/data_stream/user_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/rapid7_insightvm/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/rapid7_insightvm/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/santa/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one_cloud_funnel/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/activity/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/agent/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/group/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/slack/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/issues/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/vulnerabilities/elasticsearch/ingest_pipeline/default.yml
    • packages/sophos_central/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/sophos_central/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_edr_cloud/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_endpoint_security/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_endpoint_security/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/action_history/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/client_status/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/discover/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/endpoint_config/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/reporting/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/threat_response/elasticsearch/ingest_pipeline/default.yml
    • packages/teleport/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/plugin/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_sc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_sc/data_stream/plugin/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_sc/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/thycotic_ss/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/malwarebazaar/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/malware/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/threatfox/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/url/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_anomali/data_stream/threatstream/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_crowdstrike/data_stream/intel/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_crowdstrike/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_cybersixgill/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/apt/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/botnet/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/cc/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/domains/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/files/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/ip/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/url/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_maltiverse/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_misp/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/tines/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/tines/data_stream/time_saved/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_otx/data_stream/pulses_subscribed/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_otx/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_rapid7_threat_command/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_rapid7_threat_command/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_rapid7_threat_command/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_recordedfuture/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_threatconnect/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_edr_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_epo_cloud/data_stream/device/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_epo_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_epo_cloud/data_stream/group/elasticsearch/ingest_pipeline/default.yml
    • packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml
    • packages/trend_micro_vision_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/trend_micro_vision_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/trend_micro_vision_one/data_stream/detection/elasticsearch/ingest_pipeline/default.yml
    • packages/vectra_detect/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/wiz/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/wiz/data_stream/issue/elasticsearch/ingest_pipeline/default.yml
    • packages/zerofox/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/zeronetworks/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/app_connector_status/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/browser_access/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/user_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/user_status/elasticsearch/ingest_pipeline/default.yml
@efd6 efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Dec 10, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6 efd6 mentioned this issue Dec 10, 2024
Merged
5 tasks
@efd6 efd6 self-assigned this Dec 10, 2024
@andrewkroh andrewkroh mentioned this issue Dec 10, 2024
Open
@efd6 efd6 mentioned this issue Dec 11, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@efd6 efd6

Labels
enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@elasticmachine @efd6