[jamf_pro]: Integration is only sporadically importing device inventory #12021

dkdavlar · 2024-12-06T09:23:54Z

Integration Name

Jamf Pro [jamf_pro]

Dataset Name

inventory

Integration Version

0.1.3

Agent Version

8.15.2

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.2

OS Version and Architecture

Windows Server 2022 Datacenter 21H2 - Build 20348.2849

Software/API Version

Jamf Pro 11.10.2-t1729874551

Error Message

Logs from the agent https://pastebin.com/yvTsuWeL

Event Original

Data does not get imported into Elastic so there is no event.

What did you do?

{
  "package": {
    "name": "jamf_pro",
    "version": "0.1.3",
    "experimental_data_stream_features": []
  },
  "name": "jamf_pro-1",
  "namespace": "",
  "description": "",
  "policy_ids": [
    "fleet-first-agent-policy"
  ],
  "vars": {},
  "inputs": {
    "general_settings-cel": {
      "enabled": true,
      "vars": {
        "api_host": "https://SERVERNAME.com",
        "client_id": "CLIENTID",
        "client_secret": {
          "id": "CLIETSECRET",
          "isSecretRef": true
        }
      },
      "streams": {
        "jamf_pro.inventory": {
          "enabled": true,
          "vars": {
            "interval": "24h",
            "page_size": "50",
            "preserve_original_event": false,
            "enable_request_tracer": true,
            "tags": [
              "forwarded"
            ],
            "enable_section_general": false,
            "enable_section_hardware": true,
            "enable_section_operating_system": false,
            "enable_section_user_and_location": false,
            "enable_section_disk_encryption": false,
            "enable_section_purchasing": false,
            "enable_section_applications": false,
            "enable_section_storage": false,
            "enable_section_configuration_profiles": false,
            "enable_section_printers": false,
            "enable_section_security": false,
            "enable_section_services": false,
            "enable_section_local_user_accounts": false,
            "enable_section_certificates": false,
            "enable_section_attachments": false,
            "enable_section_plugins": false,
            "enable_section_package_receipts": false,
            "enable_section_fonts": false,
            "enable_section_licensed_software": false,
            "enable_section_ibeacons": false,
            "enable_section_software_updates": false,
            "enable_section_extension_attributes": false,
            "CONTENT_CACHING": false,
            "enable_section_group_memberships": false
          }
        }
      }
    },
    "general_settings-http_endpoint": {
      "enabled": false,
      "vars": {
        "ssl": "REDACTED"
      },
      "streams": {
        "jamf_pro.events": {
          "enabled": false,
          "vars": {
            "listen_address": "0.0.0.0",
            "listen_port": 9202,
            "url": "/jamf-pro-events",
            "tags": [
              "forwarded",
              "jamf_pro-events"
            ],
            "preserve_original_event": false,
            "preserve_duplicate_custom_fields": false
          }
        }
      }
    }
  }
}

What did you see?

I see the agent retrieving a bearer token from the Jamf server and using it to request the device inventory. The first page of the inventory is returned, but then it just stops. On some runs, it imports a few inventory records to Elastic.

Debugging logs from the agent https://pastebin.com/yvTsuWeL

What did you expect to see?

I expected it to use pagination to download and import all the devices. We have 3000+ devices, but it only requests the first page, and then nothing more happens in the log.

Anything else?

I have tried to test by enabling and disabling different sections in the inventory section without making a difference. I also tried to tweak the page size.
I can see in the log that the first page is successfully retrieved, confirming that the API client info is correct and has the correct permissions.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2024-12-06T09:56:00Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2024-12-08T23:20:34Z

Can I clarify some things?

There are 1000 entries in the response, but you say that there are no inventory records ingested. Do you not see any of the records in your ES index that are in the list you show in the request trace log you linked?
In the agent logs when logging at debug level, the agent will log the state that it is in after each API request. These have a message "response state". Can you get those lines from debug logs and see whether it contains "want_more":true? Note that the logging there contains sensitive information, so do not post it publicly.

dkdavlar · 2024-12-12T06:22:27Z

Hi @efd6,

I will say beforehand that I am not very familiar with Elastic (My knowledge is Jamf), so I'm sorry if this is not what you asked for.

I finally figured out where to look in Elastic and it seems that almost every time it runs it adds a document with this error:
@timestampNov 10, 2024 @ 00:11:03.685agent.REDACTED.snapshotfalseelastic_agent.version8.15.2error.message[ failed eval: ERROR: <input>:19:67: no such key: sections | bytes(resp.Body).decode_json().as(body, (resp.StatusCode != 200) ? | ..................................................................^, Processor rename with tag in pipeline logs-jamf_pro.inventory-0.1.1 failed with message: field [message] doesn't exist ]event.agent_id_statusverifiedevent.datasetjamf_pro.inventoryevent.ingestedNov 10, 2024 @ 00:11:13.000event.kindpipeline_errorevent.modulejamf_proinput.typeceltagsforwarded_idd5wyE5MBNKieqmMaJfee_ignored - _index.ds-logs-jamf_pro.inventory-default-2024.10.11-000001_score1

Regarding your questions, I already posted everything that was in the debug log in the original post.

efd6 · 2024-12-12T07:50:36Z

Thanks @dkdavlar. That's not what I asked for, but it does help. The key part is where is says "no such key: sections". This is referring to this part of the collector in the agent.

integrations/packages/jamf_pro/data_stream/inventory/agent/stream/cel.yml.hbs

Line 37 in 6ee9c1a

"section": state.sections,

It's not finding the sections field that it expects. I have a pretty good idea why this is happening, but it would be really helpful if you could check something (this is following my second bullet point — with instructions).

In kibana, navigate to the fleet page via the nav menu (at bottom)

Then to the agent that is running your Jamf policy (in my case its elastic-agent-40384). There are three tabs, "Agent details", "Logs" and "Diagnostics". In the "Logs" set the logging level to "debug" (bottom of page), let the agent run for a bit (at least collection period) and then in the "Diagnostics" tab

request diagnostics and look at the logs in the /logs/data directory in the zip

You should be able to find lines like what I've posted below (they will have "message":"request state" or "message":"response state"). You can see in the "message":"request state" line here that there is \"sections\":[\"OPERATING_SYSTEM\"]. Can you see that same thing in yours? (It should also appear in the "message":"response state" line (as it does below).

{"log.level":"debug","@timestamp":"2024-12-12T07:28:43.450Z","message":"request state","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"log.origin":{"file.line":250,"file.name":"cel/input.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1"},"service.name":"filebeat","input_source":"http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory","input_url":"http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory","log.logger":"input.cel","id":"cel-jamf_pro.inventory-07a67b0d-f2d7-47fc-b888-03059c5bc091","cel":{"ecs.version":"1.6.0","state":"{\"cursor\":{\"last_report_date\":\"2024-06-19T15:54:37.692Z\"},\"page_size\":1,\"preserve_original_event\":false,\"sections\":[\"OPERATING_SYSTEM\"],\"url\":\"http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory\",\"want_more\":true}"},"ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2024-12-12T07:28:43.454Z","message":"response state","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"input_source":"http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory","log.logger":"input.cel","log.origin":{"file.line":254,"file.name":"cel/input.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1"},"service.name":"filebeat","id":"cel-jamf_pro.inventory-07a67b0d-f2d7-47fc-b888-03059c5bc091","input_url":"http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory","cel":{"ecs.version":"1.6.0","state":"{\"cursor\":{\"last_report_date\":\"2024-06-19T15:54:37.692Z\"},\"events\":[{\"message\":{\"applications\":null,\"attachments\":null,\"certificates\":null,\"configurationProfiles\":null,\"contentCaching\":null,\"diskEncryption\":null,\"extensionAttributes\":null,\"fonts\":null,\"general\":{\"barcode1\":\"null\",\"declarativeDeviceManagementEnabled\":false,\"enrolledViaAutomatedDeviceEnrollment\":false,\"extensionAttributes\":[],\"initialEntryDate\":\"2024-06-19\",\"itunesStoreAccountActive\":false,\"jamfBinaryVersion\":\"11.4.1-t1712591696\",\"lastContactTime\":\"2024-04-18T14:26:51.514Z\",\"lastEnrolledDate\":\"2023-02-22T10:46:17.199Z\",\"lastIpAddress\":\"10.122.26.87\",\"lastReportedIp\":\"10.122.26.87\",\"managementId\":\"1a59c510-b3a9-41cb-8afa-3d4187ac60d0\",\"mdmCapable\":{\"capable\":false,\"capableUsers\":[]},\"name\":\"acme-C07DM3AZQ6NV\",\"platform\":\"Mac\",\"remoteManagement\":{\"managed\":true},\"reportDate\":\"2024-06-19T15:54:37.692Z\",\"site\":{\"id\":\"-1\",\"name\":\"None\"},\"supervised\":false,\"userApprovedMdm\":false},\"groupMemberships\":null,\"hardware\":null,\"ibeacons\":null,\"id\":\"3\",\"licensedSoftware\":null,\"localUserAccounts\":null,\"operatingSystem\":null,\"packageReceipts\":null,\"plugins\":null,\"printers\":null,\"purchasing\":null,\"security\":null,\"services\":null,\"softwareUpdates\":null,\"storage\":null,\"udid\":\"5982CE36-4526-580B-B4B9-ECC6782535BA\",\"userAndLocation\":null}}],\"page_size\":1,\"preserve_original_event\":false,\"sections\":[\"OPERATING_SYSTEM\"],\"url\":\"http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory\",\"want_more\":false}"},"ecs.version":"1.6.0"}

The second thing that I'd like to check is what the configuration looks like. In the diagnostic zip you will see the pre-config.yml file.

There should be a stanza in the YAML that looks like this (don't post it here, it has secrets — this one is a test instance)

    - data_stream:
        namespace: "22416"
      id: cel-general_settings-07a67b0d-f2d7-47fc-b888-03059c5bc091
      meta:
        package:
            name: jamf_pro
            version: 0.1.3
      name: jamf_pro-inventory-22416
      package_policy_id: 07a67b0d-f2d7-47fc-b888-03059c5bc091
      revision: 1
      streams:
        - auth:
            oauth2:
                client:
                    id: test_client_id
                    secret: test_client_secret
                endpoint_params:
                    grant_type: client_credentials
                token_url: http://svc-jamf_pro_api_stub:8080/api/oauth/token
          config_version: 2
          data_stream:
            dataset: jamf_pro.inventory
            type: logs
          id: cel-jamf_pro.inventory-07a67b0d-f2d7-47fc-b888-03059c5bc091
          interval: 24h
          keep_null: true
          max_executions: null
          program: |-
            request(
            	"GET",
            	state.url.trim_right("/") + "?" + {
            		"section": state.sections,
            		"page-size": [string(state.page_size)],
            		"sort": ["general.reportDate:asc"],
            		?"filter": (has(state.?cursor.last_report_date) && state.?cursor.last_report_date.orValue("") != "") ?
            			optional.of(["general.reportDate>=\"" + state.cursor.last_report_date + "\""])
            		:
            			optional.none(),
            	}.format_query()
            ).with(
            	{
            		"Header": {
            			"Content-Type": ["application/json"],
            		},
            	}
            ).do_request().as(resp,
            	bytes(resp.Body).decode_json().as(body, (resp.StatusCode != 200) ?
            		{
            			"events": [
            				{
            					"error": {"message": "response: " + string(resp.StatusCode)},
            					"event": {"original": resp.Body},
            				},
            			],
            		}
            	:
            		state.with(
            			{
            				"events": body.results.map(e,
            					{
            						"message": e,
            						?"event.original": state.?preserve_original_event.orValue(false) ? optional.of(e.encode_json()) : optional.none(),
            					}
            				),
            				"want_more": body.totalCount > size(body.results),
            				"cursor": {
            					"last_report_date": (size(body.results) == 0 || size(body.results) == body.totalCount) ?
            						state.?cursor.last_report_date.orValue("")
            					:
            						string(body.results[size(body.results) - 1].general.reportDate),
            				},
            			}
            		)
            	)
            )
          publisher_pipeline.disable_host: true
          resource:
            timeout: null
            url: http://svc-jamf_pro_api_stub:8080/api/v1/computers-inventory
          state:
            page_size: 1
            preserve_original_event: false
            sections:
                - OPERATING_SYSTEM
          tags:
            - forwarded
      type: cel
      use_output: default

At the bottom you should see the sections: field array. Does your configuration have this?

dkdavlar · 2024-12-12T09:39:39Z

Thanks for the guide :D

For the first log file I see
"state": "{\"cursor\":{\"last_report_date\":\"2024-12-11T10:03:08.644Z\"},\"page_size\":50,\"preserve_original_event\":false,\"sections\":[\"GENERAL\"],\"url\":\"https://SERVERNAME.com/api/v1/computers-inventory\",\"want_more\":true}"

And from the pre-config:
sections: - GENERAL

I hope this was what you were looking for.

Just adding something I found odd in the debug logs from the agent: No matter how many computers the API call made, it only ever requested the first page. Infact looking in the logs I never saw the "page" paramter used, only the "page-size". But maybe this is just the product of it failing before it starts page 2.

efd6 · 2024-12-13T02:42:05Z

Do you see any log lines where the state is present, but sections is absent in the state?

The page thing is something that I raised with our partners. The design that's used here doesn't use page, instead using the filter. There are some issues with this (obviously). They suggested two possible alternative approaches that we could take to address this.

dkdavlar · 2024-12-13T08:12:26Z

No, in all the places where I can find the state, the General section is present:
"sections\":[\"GENERAL\"]

efd6 · 2024-12-13T08:17:52Z

So there are no errors like what you show above in the logs in the diagnostic zip?

dkdavlar · 2024-12-13T08:28:58Z

Ah yes, there are a few of these:

    "log.level": "error",
    "@timestamp": "2024-12-12T08:47:53.105Z",
    "message": "failed evaluation",
    "component": {
        "binary": "filebeat",
        "dataset": "elastic_agent.filebeat",
        "id": "cel-default",
        "type": "cel"
    },
    "log": {
        "source": "cel-default"
    },
    "input_url": "https://SERVERNAME.com/api/v1/computers-inventory",
    "error": {
        "message": "failed eval: ERROR: <input>:19:67: no such key: reportDate\n |  bytes(resp.Body).decode_json().as(body, (resp.StatusCode != 200) ?\n | ..................................................................^"
    },
    "log.origin": {
        "file.line": 260,
        "file.name": "cel/input.go",
        "function": "github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1"
    },
    "log.logger": "input.cel",
    "service.name": "filebeat",
    "id": "cel-jamf_pro.inventory-22ac6087-a403-4e1c-888c-e6769ba2b948",
    "input_source": "https://SERVERNAME.com/api/v1/computers-inventory",
    "ecs.version": "1.6.0",
    "ecs.version": "1.6.0"
}

dkdavlar added the needs:triage label Dec 6, 2024

andrewkroh added Integration:jamf_pro Jamf Pro Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Dec 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jamf_pro]: Integration is only sporadically importing device inventory #12021

[jamf_pro]: Integration is only sporadically importing device inventory #12021

dkdavlar commented Dec 6, 2024

elasticmachine commented Dec 6, 2024

efd6 commented Dec 8, 2024

dkdavlar commented Dec 12, 2024

efd6 commented Dec 12, 2024

dkdavlar commented Dec 12, 2024

efd6 commented Dec 13, 2024 •

edited

Loading

dkdavlar commented Dec 13, 2024

efd6 commented Dec 13, 2024

dkdavlar commented Dec 13, 2024

[jamf_pro]: Integration is only sporadically importing device inventory #12021

[jamf_pro]: Integration is only sporadically importing device inventory #12021

Comments

dkdavlar commented Dec 6, 2024

Integration Name

Dataset Name

Integration Version

Agent Version

Agent Output Type

Elasticsearch Version

OS Version and Architecture

Software/API Version

Error Message

Event Original

What did you do?

What did you see?

What did you expect to see?

Anything else?

elasticmachine commented Dec 6, 2024

efd6 commented Dec 8, 2024

dkdavlar commented Dec 12, 2024

efd6 commented Dec 12, 2024

dkdavlar commented Dec 12, 2024

efd6 commented Dec 13, 2024 • edited Loading

dkdavlar commented Dec 13, 2024

efd6 commented Dec 13, 2024

dkdavlar commented Dec 13, 2024

efd6 commented Dec 13, 2024 •

edited

Loading