chore: rebase merge branch migrate-metrics-dev into master #1758
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mpl (part 1) (apache#1374) This PR is to migrate replica-level metrics of `pegasus_server_impl` to new framework (apache#1333). Since there are many replica-level metrics in `pegasus_server_impl`, this PR is part 1 for this migration and other metrics (all of which are gauges) will be migrated in the later PRs.
…mpl (part 2) (apache#1386) This PR is to migrate replica-level metrics of pegasus_server_impl to new framework, 2nd part, for apache#1333. This PR focuses on migrating all rocksdb-related metrics for each replica, including total number and size of sst files, estimated number of keys, memory usage and hit rate, write/read amplification, negatives/positives of bloom filters.
…lculator (apache#1387) apache#1334 Migrate the metrics in capacity_unit_calculator to new framework, including read/write capacity units and the number of bytes consumed by get, multi_get, batch_get, scan, put, multi_put, check_and_set, check_and_mutate and backup requests.
…pache#1397) apache#1342 Migrate replica-level metrics in `replica` class to new framework, including the size of private log, the number of backup requests, the number of read/write/backup requests that are delayed/rejected due to throttling, the number of read/write requests that are rejected due to splitting, the number of write requests that are rejected due to bulk load ingestion, the number of non-idempotent write requests that are rejected due to duplication. Note that the old metrics for the number of backup requests, the number of delayed/rejected backup requests due to throttling and the number of rejected non-idempotent write requests are table-level, which have been changed to replica-level in new metrics, since they could be aggregated to table-level if necessary.
…stener (apache#1407) apache#1343 Migrate replica-level metrics in pegasus_event_listener class to new framework, all of which are rocksdb-related, including the number of completed flushes/compactions, the size of flush output in bytes, the size of compaction input/output in bytes. Note that in old perf counters there are just 2 replica-level metrics for pegasus_event_listener while all of others are server-level. Migrated to new framework all of the metrics have become replica-level; once server-level metrics are needed, just aggregate on replica-level ones.
…_duplicator (apache#1413) apache#1412 Migrate replica-level metrics in pegasus_mutation_duplicator class to new framework, including the numbers of successful/failed DUPLICATE requests sent from mutation duplicator.
apache#1414 Add server-level metric entity, including: - Define server-level entity prototype in metrics.cpp; - Implement getter to acquire the (only) instance of server entity; - Provide convenient macros to simplify operations for server entity.
apache#1321 Migrate built-in server-level metrics to new framework, including the total amount of virtual/physical memory usage in MB. Both metrics are wrapped in `builtin_metrics` and updated periodically by a timer, which is started or stopped along with meta/replica servers. Since `dsn_perf_counter_test` involves both metrics which have been removed from perf-counters, just disable it.
apache#1329 Migrate server-level metrics to new framework for both server/client sides of nfs, including the data size in bytes that are requested by client or read from local file in server, the number of nfs copy requests that fail for client or server, the data size in bytes that are written to local file in client, the number of failed writes to local file in client. The old type in perf counters of all these metrics are volatile counter, while all of them become accumulated counter for new metrics.
…l metrics for fs_manager (apache#1427) apache#1425 In perf counters, all metrics of `fs_manager` are server-level. For example, the total capacity and the available capacity of all disks where there are data of pegasus. However, sometimes the capacity and the available capacity of each disk seem more important: no space left on the disk will lead to serious problems. Therefore, after being migrated to new framework, the server-level metrics of perf counters become disk-level, including the capacity and the available capacity of a disk. As for another disk-level metric -- the available percentage of each disk used by a replica server, just use division operator. Once server-level metrics are needed, just aggregate on the disk-level ones. To compute another 2 server-level metrics -- the minimal/maximal available percentage among all disks used by a replica server in a node, for example, just use min/max operators over disk-level ones for Prometheus. To implement disk-level metrics, disk-level metric entity are also added.
…vel metrics for server_state of meta (apache#1431) apache#1331 In perf counters, all metrics of server_state are server-level, for example, the number of healthy partitions among all tables of a pegasus cluster. However, sometimes this is not enough. For example, the metric shows that there are 4 unwritable partitions: the 4 unwritable partitions might belong to different tables; or, they might belong to one table. Therefore, these server-level metrics could be changed to table-level. This will provide us with the status of each table. On the other hand, once server-level metrics is needed, just aggregate on table-level ones. The metrics of server_state that are migrated and changed to table-level include: The number of dead, unreadable, unwritable, writable-ill, and healthy partitions among all partitions of a table, the number of times the configuration has been changed and the number of times the status of partition has been changed to unwritable or writable for a table. To implement table-level metrics, table-level metric entity is also added.
…ition-level metrics for greedy_load_balancer of meta (apache#1435) apache#1331 In perf counters, all metrics of greedy_load_balancer are server-level, for example, the number of each kind of operations by greedy balancer, including moving primaries, copying primaries and copying secondaries. For new metrics, it is hoped that they are fine-grained, since sometimes we want to know which primaries are moved. Also, it is convenient to calculate table-level or server-level metrics by just aggregate on partition-level ones. The metrics of greedy_load_balancer that are changed to partition-level and migrated to new framework include: the number of balance operations by greedy balancer that are recently needed to be executed, move primaries, copy primaries, and copy secondaries. In addition to the metrics of greedy_load_balancer, we also change some metrics of server_state again to partition-level which have been migrated to table-level in apache#1431, for the reason that partition-level is considered more appropriate for them than table-level. The metrics changed to partition-level include the number of times the configuration has been changed and the number of times the status of partition has been changed to unwritable or writable for a partition. To implement table-level metrics, partition-level metric entity is also added.
…che#1437) apache#1331 Migrate metrics to new framework for meta_service, including the number of disconnections with replica servers, and the number of unalive and alive replica servers. All of these metrics are server-level, maintained in meta server. The old type in perf counters of the number of disconnections is volatile counter, which would be changed to non-volatile, while another 2 metrics would keep the type of gauge.
…backup-policy-level metrics for meta_backup_service (apache#1438) apache#1331 In perf counters, there's only one metric for meta_backup_service, namely the recent backup duration for each policy, which means this metric is policy-level. Therefore policy-level entity would also be implemented in new metrics.
…dian (apache#1440) apache#1331 In perf counters, there's only one metric for partition_guardian, namely the number of operations that fail to choose the primary replica, which is server-level. It would be changed to partition-level in new metrics since this could give which partitions fail to choose primaries and how frequency those happen. Still, to compute table-level or server-level metrics just aggregate on partition-level ones.
…ompact_service (apache#1443) apache#1441 In perf counters, all of the 2 metrics of `pegasus_manual_compact_service` are about the numbers of tasks of rocksdb manual compaction: one is the number of current queued tasks, while another is the number of current running tasks. Both metrics are server-level. They would become replica-level after migrating to the new metrics, based on which server-level ones could also be achieved. A convenient class `auto_count` is also provided to increment gauge that will be decremented automatically at the end of the scope.
…1455) apache#1454 This is the 1st part of migrating metrics of `replica_stub` to new framework. After migrating to new framework, the 3 metrics, including the total number of replicas, the number of opening/closing replicas, are still kept server-level. Another metric, the number of committed requests, is changed to replica-level. The naming of metric variable would lead to duplication with class member (such as `_opening_replicas` in `replica_stub` class). Therefore, a macro `METRIC_VAR_NAME` is introduced to manage the new naming, which is prefixed with `_metric_` to avoid duplication. Also, generated metric function names are also managed by related macros.
…1459) apache#1454 This is the 2nd part of migrating metrics of `replica_stub` to new framework, all of which are learn-related. During this migration, there are 3 metrics still keeping server-level, including the number of learning replicas, the max duration and the max size of files that are copied from learnee among all learning replicas. Another 11 metrics are changed from server-level to replica-level, since they should be observed for each replica. All of them are observed from the view of learners, namely the potential secondary replica. The learnee is the primary replica. These metrics include: the number of learns launched by learner, the number of learn rounds launched by learner (during a learn there might be multiple rounds, the number of files that are copied from learnee, the size of files that are copied from learnee, the size of data that are copied from learnee's buffer, the number of learn responses of `LT_CACHE`, `LT_APP` and `LT_LOG` type decided by learner with each learn response related to an `RPC_LEARN` request, the number of times learner resets its local state (since its local state is newer than learnee's) with each reset related to an learn response of an `RPC_LEARN` request, the number of failed and successful learns launched by learner.
…che#1461) Co-authored-by: neverchanje <[email protected]>
…1462) apache#1454 This is the 3rd part of migrating metrics of replica_stub to new framework. During this migration, there are 3 metrics which are changed from server-level to replica-level, including the number of failed RPC_PREPARE requests, the number of failed RPC_GROUP_CHECK requests launched by primary replicas, the number of triggered emergency checkpoints. Another 7 metrics are still kept server-level, the number of replicas whose dirs are moved as error or garbage, the number of removed replica dirs, error replica dirs (*.err), garbage replica dirs (*.gar), tmp replica dirs (*.tmp) and origin replica dirs (*.ori) for disk migration. There are 2 metrics removed, since both are shared-log-related.
…1463) apache#1454 This is the 4th part of migrating metrics of replica_stub to new framework. During this migration, there are 3 metrics which are changed from server-level to replica-level, including the number of confirmed/pending mutations for dup and the number of write requests whose size exceeds threshold. Another 5 metrics are still kept server-level, including the memory bytes that are released by tcmalloc recently, the number of failed read/write requests and the number of busy read/write requests.
…1469) apache#1454 This is the 5th part of migrating metrics of replica_stub to new framework, most of which are cold-backup-related. During this migration, there are 10 metrics which are changed from server-level to replica-level, including the number of current running backups, the max backup duration among backups, the max size of uploaded files among backups, the number of started/failed/successful/cancelled backups, the number of failed/successful file uploads for backups, the total size of uploaded files for backups. The metric tracking the number of paused cold backup has been removed, since it's not in use. `tcmalloc_released_bytes` is changed from gauge to counter, since the counter would be a more accurate reflection of recently released memory by tcmalloc.
…1474) apache#1454 This is the 6th part of migrating metrics of replica_stub to new framework, all of which are bulk-load-related. During this migration, there are 7 metrics which are changed from server-level to replica-level, including the number of downloading/ingesting bulk loads, the number of successful/failed bulk loads, the number of files that have been downloaded successfully or have failed to be downloaded for bulk loads, the size of files that have been downloaded successfully for bulk loads. Another 3 metrics are still kept server-level, including the number of current running bulk loads, the max duration of ingestions or the whole bulk loads.
…1475) apache#1454 This is the 7th part of migrating metrics of replica_stub to new framework, all of which are partition-splitting-related. This is also the last part for replica_stub. During this migration, there are 6 metrics which are changed from server-level to replica-level, including the number of started splittings, the number and the size of files copied for splitting, the number of mutations copied for splitting, the number of failed/successful splittings. Another 4 metrics are still kept server-level, including the number of current splitting replicas, the max duration among all splitting replicas, the max duration among all splitting replicas for async learns, the max size of copied files among all splitting replicas.
…#1482) apache#1481 Some duplication-related classes, including `ship_mutation`, `mutation_buffer`, `load_from_private_log`, are migrated to the new framework. During this migration, there are 6 metrics which are changed from server-level to replica-level, all of which are duplication-related, including the shipped size of private log, the number of times private log files have failed to be loaded, the bytes of mutations that have been skipped due to failed loadings of private log files, the size read from private log, the number of mutations read from private log, the number of lost mutations recently. Another 2 metrics, the numbers of failed/successful DUPLICATE requests sent from client, are renamed according to the new style of naming for duplication.
apache#1483 All of the 3 metrics of task queue are migrated to the new framework, including the length of task queue, the accumulative number of delayed/rejected tasks by throttling before enqueue. All of these metrics are attached to the queue-level entity, which is also newly introduced in this PR. Furthermore, this PR solves some other problems: - The circular including of header files between `common/json_helper.h` and `utils/metrics.h` leads to compilation error, resolved by dropping a redundant header file `runtime/tool_api.h` in the including chains. - Undefined reference of `http_service::register_handler` while compiling on macOs, is resolved by linking `dsn_http` to static library of `dsn_utils`. - Github actions run out of disk space while building releases, resolved by removing directories and tarballs after packing server and tools to spare more space, which is also tracked by another issue: apache#1485 - Add some hint messages for errors while getting entity pointers.
…units (apache#1491) apache#1490 For the enum classes, such as metric type and unit, in addition to the declaration of all enumerators, extra code is needed to implement `enum_to_string` for each enumerator. However, the "extra code" tended to be forgotten, just as what has been said in the above issue apache#1490. Therefore, we should find a way that could share all the enumerators between the declaration and `enum_to_string`. Each enumerator would be written only once, and there is no need to remember to add `enum_to_string` for each enumerator. We could implement the "sharing" by function-like macros. They have the names of all the enumerators, and accept an argument to perform custom actions over the names to implement the "sharing". Both metric types and units would be refactored in this way. While building ASAN Github actions failed due to running out of disk space. This problem is resolved by dropping all directories of `CMakeFiles`, which is also tracked another issue: apache#1497
apache#1501 The only server-level metric of failure detector is migrated to the new framework, i.e. the number of failed beacons sent by failure detector.
apache#1503 The only server-level gauge of network is split into 2 gauges, and migrated to the new framework. The only gauge is a metric about TCP connections: from client side, the network could connect to different remote servers; while from server side, it could receive connections from remote clients. In the past, the only gauge has been used to measure either the number of different remote servers from client side, or the number of different remote clients from server side, which might result in confusion. Therefore, it's better to split the only gauge into 2 individual ones.
apache#1505 All of the server-level metrics of rocksdb are migrated to the new framework, including 2 metrics: the memory usage of block cache, and the through bytes per second that go through the rate limiter which takes control of the write rate of flush and compaction. This migration are somewhat different from previous ones: - firstly, both metrics are declared as static variables; - then, both variables need to be defined globally; - also, both metrics have to be initialized by assignment operator, rather than in the member initializer lists of the constructor of user class. To support all of these new features, some new macros are introduced.
Related issue: apache#1206 This patch collects all raw metric data from pegasus, aggregate them into table/server/cluster level and sinks them into prometheus. If the type of the metric is replica partition, it will be aggregated into cluster level table metric. If the type of the metric is table, it will be aggregated into cluster level table metric, it also will be aggregated into server level table metric. If the type of the metric is server, it will be aggregated into cluster level table metric, it also will be sinked into prometheus directly. Currently, this patch supports 3 type of prometheus metric types: counter, gauge, summary.
apache#1523 Profiler-level metric entity is introduced and 12 profiler-related metrics are migrate to the new framework, including the number of tasks in all queues, the number of tasks that have been executed, the number of cancelled tasks, the latency it takes for each task to wait in each queue before beginning to be executed, the latency it takes for each task to be executed, the latency from enqueue point to reply point on the server side for each RPC task, the non-timeout latency from call point to enqueue point on the client side for each RPC task, the body length of request received or response replied on the server side for each RPC task, the accumulative number of dropped RPC tasks on the server side, the accumulative number of timeout RPC tasks on the client side. All these metrics are configurable thus might be NULL. `METRIC_DEFINE_*_NOTNULL` macros are introduced to help defining the member functions that check if the metric variables are NULL pointers. Some macros are refactored to make it more convenient to define the member functions that increment/decrement/set/get the value of the metric variables.
…pache#1528) apache#1527 Maintained in an array that includes all of the profilers, each profiler is indexed and accessed by its task code. Therefore, `TASK_CODE_INVALID`, albeit meaningless, should also be pushed into this array; otherwise, all of the profiled tasks would be indexed by the wrong task code, then also measured by the wrong metrics.
…ica (apache#1531) apache#1327 Remove all table-level metrics that are measured by perf-counters. Later, table-level metrics would be aggregated by Go Collector, if necessary.
…ared log (apache#1533) apache#1532 In apache#1462, all metrics of shared log have been removed. However, there are still some perf-counters that are used in shared log. All of them should be removed.
apache#1536 The only metric of latency tracer, namely the duration between two points (stages), is migrated to the new framework: - latency tracer entity is introduced as the new metric entity to which the metric is attached; - create new class that bind each entity and manage the metric; - create new class that manage all entity instances, with a read-write lock that helps improving the performance of search for the target entity; - solve the problem of "No space left on device" for Build ASAN temporarily.
…apache#1542) apache#1541 After rebasing from branch master onto migrate-metrics-dev, there are some problems found for metrics total_capacity_mb/total_available_mb of fs_manager: - both are referencing the value of perf-counters which have been changed to new framework and cannot be compiled successfully; - setting/getting both metrics are not atomic. This PR is to solve both problems.
apache#1539 In XiaoMi/rdsn#349 we've supported getting value of counter by http request. Now since perf counter will be replaced by new metrics, it can also be removed.
apache#1547 Since perf counters would be removed, `pegasus_counter_reporter` would not be used to collect metrics to target monitoring system, thus could be removed.
empiredan
changed the title
…iles for new metrics
…ts and write_size_exceed_threshold_requests
…branch migrate-metrics-dev into master such as including absl/strings/string_view.h
empiredan
force-pushed
the
rebase-migrate-metrics-dev-onto-master
branch
from
ae58c64
to
8b9221a
Compare
|
Compilation error for Go collector would be tracked and fixed by: #1761 |
|
I've reviewed these patches in branch |
acelyc111
approved these changes
Dec 11, 2023
GehaFearless
approved these changes
Dec 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.