User Behavior Insights

[jzonthemtn]

Per [RFC] User Behavior Insights, implement the initial User Behavior Insights functionality to provide a schema for client-side events and queries, along with the ability to capture and index queries.

[jzonthemtn]

Pull request is #13546

[andrross]

[Triage - attendees 1 2 3 4]
@jzonthemtn Thanks for filing. Looking forward to seeing progress on the PR. Can you create a UBI meta issue for all the open issues planned for this feature? It will help folks with being able to follow the progress.

[jzonthemtn]

[Triage - attendees 1 2 3 4] @jzonthemtn Thanks for filing. Looking forward to seeing progress on the PR. Can you create a UBI meta issue for all the open issues planned for this feature? It will help folks with being able to follow the progress.

Sure, not a problem -- #13605. Thanks for the issue triaging!

[MShah890]

Hi @jzonthemtn

On this Issue - #12084 you mentioned the following

Thanks for proposing this. I would like to treat events as metadata. In our use case, we have metadata information in rest API request header. Metadata information like client_id, data object type, etc. We really need raw opensearch request body and response body logged in AWS cloudWatch or s3 for query pattern analysis. We did plugin explore and we found hard blocker in opensearch security plugin. The reason is opensearch security plugin already did getRestHandlerWrapper override. We cannot override this in our plugin to do our own logging and this restHandlerWrapper is only way for us to do this as far as I know. I hope we can prioritize this RFC for community. Or if anyone can guide me to generate log in alternative way will be appreciated.

Thanks @shikeli, and thanks for the pointer on the getRestHandlerWrapper override. You are looking to capture the raw queries and their results? Would being able to export the captured metadata to a file format like Parquet work for your purposes?

Thanks for quick response. What do you mean by metadata, is it raw request and raw response? If you can export raw request and response to a file, that should be able to solve our problem.

@shikeli Yes, the export would be the search requests/responses along with the events generated by the client-side. Our desire is to capture the raw requests/responses, but I'm not yet entirely sure what technical impediments we might encounter (like your getRestHandlerWrapper problem) but raw is our goal.

Can you please give me a link to a tracking ticket for this feature ?
If this feature is already implemented can you please point to documentation ?
Will this feature be available for OS 2.11 ?

I went through this user-behavior-insights but not sure if I was able to find anything on capturing raw requests/responses to and from OS

[jzonthemtn]

Hi @MShah890, We still don't capture the raw requests and responses. We capture the query entered by the user and the query that gets sent to OpenSearch. We determined during development that the raw information was not needed for our purposes. We just needed the query itself, and the search results to that query.

[jzonthemtn]

The development of UBI was changed to be a plugin instead of a module and this issue was inadvertently not closed. The UBI plugin repository is at https://github.com/opensearch-project/user-behavior-insights. I am going to close this issue for clarity but please feel free to comment/inquire on UBI. :)