Metrics on document sizes

[VannTen]

Problem statement

As a Thoth operator/developer, I'd like to have metrics regarding the size of
the file storage.

High-level Goals

• Being able to consult various aggregate on the object storages state
  (total, average, etc).

Proposal description

Implements two metrics, total size / number of objects.
Optionally, partition by object types.

Avoid too much partition since it would make the cardinality blow up.

Alternatives

Use object storage builtin metrics (Ceph might have somehthing like this)

Additional context

Prompted by thoth-station/storages#2691 (comment)

Acceptance Criteria

• metrics are available
• grafana dashboard exploiting them

/sig observability
/priority backlog
/good-first-issue

[VannTen]

/priority backlog
/good-first-issue
/sig observability

[sesheta]

@VannTen:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/priority backlog
/good-first-issue
/sig observability

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[VannTen]

We might want to consider:

https://github.com/ribbybibby/s3_exporter
https://github.com/jamotion/s3-exporter
https://github.com/blemmenes/radosgw_usage_exporter

Could also be relevant for #821