Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: measure object upload bytes #280

Merged
merged 2 commits into from
Jul 11, 2023
Merged

feat: measure object upload bytes #280

merged 2 commits into from
Jul 11, 2023

Conversation

jeqo
Copy link
Contributor

@jeqo jeqo commented Jun 14, 2023

Changes:

  • To enable metrics to capture the number of bytes uploaded to storage backends (e.g. after transforms), upload is expanded to return the number of bytes uploaded.
  • Metrics to measure object uploads per suffix type are added.

@jeqo jeqo requested a review from a team as a code owner June 14, 2023 07:58
@ivanyu ivanyu self-assigned this Jun 14, 2023
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch 2 times, most recently from e49ca4a to 2d28abc Compare June 14, 2023 12:52
@jeqo jeqo requested a review from ivanyu June 14, 2023 12:52
@jeqo jeqo marked this pull request as draft June 14, 2023 13:51
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch 3 times, most recently from 374ce3f to 5cf07cc Compare June 14, 2023 14:10
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch from 5cf07cc to 268fa38 Compare June 30, 2023 07:56
@jeqo jeqo marked this pull request as ready for review June 30, 2023 08:07
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch 2 times, most recently from 10d035f to 4c8a116 Compare July 3, 2023 10:35
@jeqo jeqo changed the title feat(storage): upload to return size feat: measure object upload bytes Jul 3, 2023
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch from 4c8a116 to 0ae6fb3 Compare July 3, 2023 10:41
@jeqo jeqo requested a review from ivanyu July 3, 2023 10:42
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch 2 times, most recently from 6ee4f15 to f41a1d7 Compare July 7, 2023 16:18
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch from f41a1d7 to 21dea2b Compare July 11, 2023 10:38
@jeqo jeqo requested a review from giuseppelillo July 11, 2023 10:43
@@ -238,17 +238,20 @@ private void uploadSegmentLog(final RemoteLogSegmentMetadata remoteLogSegmentMet
throws IOException, StorageBackendException {
final String fileKey = objectKey.key(remoteLogSegmentMetadata, ObjectKey.Suffix.LOG);
try (final var sis = transformFinisher.toInputStream()) {
uploader.upload(sis, fileKey);
final var bytes = uploader.upload(sis, fileKey);
metrics.recordObjectUpload(ObjectKey.Suffix.LOG, bytes);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a way to get the aggregated number of bytes. It seems to me, one of:

  1. Add a separate aggregated sensor.
  2. Make the suffix a tag instead of a part of the sensor name (so we can sum by this tag).
  3. Don't bother with separating by tag, the log files will anyway be 99%.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can follow a similar approach as with topic vs topic-partition, and have an aggregated metric on INFO level, and by object type in DEBUG, wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you feel like it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, giving a try now. Pushed latest changes.

@jeqo jeqo requested a review from ivanyu July 11, 2023 12:38
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch from 21dea2b to ae541e9 Compare July 11, 2023 13:22
Copy link
Contributor

@ivanyu ivanyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conflicts

@jeqo jeqo force-pushed the jeqo/storage-upload-size branch from ae541e9 to 569a1e1 Compare July 11, 2023 15:06
@jeqo jeqo force-pushed the jeqo/storage-upload-size branch from 569a1e1 to 75b4b43 Compare July 11, 2023 15:11
@jeqo jeqo requested a review from ivanyu July 11, 2023 15:13
@ivanyu ivanyu merged commit ca2d78a into main Jul 11, 2023
@ivanyu ivanyu deleted the jeqo/storage-upload-size branch July 11, 2023 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants