Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Delete blocks #144

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,9 @@
"suggestEdit": true,
"raiseIssue": true
},
"seo": {
"indexHiddenPages": false
},
"redirects": [
{
"source": "/index",
Expand Down
70 changes: 70 additions & 0 deletions restapi/delete-blocks.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
title: 'Delete blocks of data'
tothmano marked this conversation as resolved.
Show resolved Hide resolved
description: 'This page explains how to delete blocks of data.'
sidebarTitle: 'Delete blocks'
---

After data ingestion, Axiom stores events in blocks of tens of thousands of events. You cannot delete individual events after you send them to Axiom, but you can delete entire blocks of data. With the `/v2/datasets/_apl/delete` endpoint, you can specify an APL query and delete all blocks that contain matching events.

<Warning>
The `/v2/datasets/_apl/delete` endpoint deletes all blocks containing events that match your query. For example, even if a block only contains one event that matches your query, the endpoint removes all 65,536 events in the block. For this reason, using this endpoint may remove more data than your query specifies.

Deleting data using this endpoint is slow and expensive. Only use this endpoint when you have sent sensitive data to Axiom by mistake and you need to immediately delete it for regulatory reasons.

Use the block deletion endpoint for deleting less than one million rows in one request. To delete more than one million rows, send a number of requests and delete a limited number of rows in each request. If you delete too many rows in one request, the request might fail unexpectedly.
</Warning>

## Delete blocks

To delete all blocks containing events that match an APL query, send a `POST` request to `/v2/datasets/_apl/delete`.

To authenticate your request, use an API token with update permissions for datasets. For more information, see [API tokens](/reference/tokens).

In the request body, include a JSON object with the following fields:

- `apl`: An APL query that specifies the events to be deleted. You can only use the operators [`where`](/apl/tabular-operators/where-operator) and [`search`](/apl/tabular-operators/search-operator).
- `startTime`: The start of the time period over which you wantto run the query.
- `endTime`: The end of the time period over which you want to run the query.
- `commit`: A Boolean that specifies whether to actually perform the deletion. Set `commit` to `false` to preview the impact of the deletion request without actually deleting any events.

A valid request receives a 200 response with the following fields present in the response body:

- `dryRun`: Returns `true` if the value of `commit` specified in the request was `false`.
- `rowsMatched`: The number of matching events.
- `rowsDeleted`: The number of deleted events. This is the number of all events in all the blocks that contain events matching your query. The number of deleted events is higher than the number of matching events.
- `firstMatchedEvent`: The timestamp of the first event matching the query.
- `lastMatchedEvent`: The timestamp of the last event matching the query.

When you make a successful request with `commit` set to `true`, blocks containing matching events are queued for deletion. The action cannot be reversed. A block queued for deletion is typically deleted within 5 minutes, but it may take longer if your organization makes a large number of deletion requests.

## Best practices

- Start with a dry run. Set `commit` to `false` to safely review which events are to be deleted by your request.
- Deletions are slow, resource-intensive, and costly operations. Use them rarely.
- After deleting data, [vacuum the fields of the dataset](/reference/datasets#vacuum-fields) to clean up the dataset completely.

## Example request

```bash
curl -X 'POST' 'https://api.axiom.co/v2/datasets/_apl/delete' \
-H 'Authorization: Bearer API_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
"apl": "[\"otel-demo-traces\"] | where trace_id == \"b630e23cfc3a6ac9e57fee13542b3e50\"",
"startTime": "now-6h",
"endTime": "now",
"commit": false
}'
```

## Example response

```bash
{
"dryRun": true,
"firstMatchedEvent": "2024-08-25 00:18:19 +0000 UTC",
"lastMatchedEvent": "2024-08-25 00:18:19 +0000 UTC",
"rowsDeleted": 10249,
"rowsMatched": 1
}
```