Pyroscope api support #438

akvlad · 2024-01-24T11:20:34Z

support some of pyroscope API:

profileTypes,
labelNames
labelValues
selectMergeStacktraces
selectSeries

Signed-off-by: Tomer Shafir <[email protected]>

Update scripts.js

# Conflicts: # lib/db/clickhouse.js # lib/db/maintain/index.js # qryn_bun.mjs

socket-security · 2024-01-24T11:30:49Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@grpc/[email protected]	environment, filesystem, network	`+31`	6.82 MB	murgatroid99
npm/[email protected]	None	`0`	820 kB	dibenede

View full report↗︎

lmangani · 2024-01-24T11:43:20Z

Could you provide the full API specs for documentation? The same details as the other APIs would be terrific

github-actions · 2024-01-24T14:14:39Z

Workflow Telemetry - QRYN CI / build (16.x)

Workflow telemetry for commit 037eff5217ac5730f4d98d03aa230ad4f2b66f0f
You can access workflow job details here

Step Trace

gantt
	title build (16.x)
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1706105615000, 1706105616000
	Initialize containers : 1706105616000, 1706105626000
	Run actions/checkout@v4 : 1706105627000, 1706105627000
	Use Node.js 16.x : 1706105627000, 1706105629000
	Run npm install : 1706105630000, 1706105641000
	Run npm run postinstall : 1706105641000, 1706105641000
	Run git submodule init : 1706105641000, 1706105641000
	Run git submodule update : 1706105641000, 1706105642000
	Workflow Telemetry : 1706105642000, 1706105642000

CPU Metrics

Memory Metrics

IO Metrics

	Read	Write
Network I/O
Disk I/O

Process Trace

Top 100 processes with highest duration

gantt
	title build (16.x)
	dateFormat x
	axisFormat %H:%M:%S
	jest : 1706105643129, 1706105672701
	node : 1706105643129, 1706105672701
	node : 1706105643321, 1706105643361
	find : 1706105643638, 1706105643639
	find : 1706105643649, 1706105644074
	node : 1706105644104, 1706105644340
	node : 1706105644115, 1706105644333
	node : 1706105644134, 1706105644341
	node : 1706105644377, 1706105672672
	node : 1706105644389, 1706105672659
	node : 1706105644413, 1706105672645
	node : 1706105644602, 1706105644714
	node : 1706105644650, 1706105644763
	node : 1706105644683, 1706105644798

github-actions · 2024-01-24T14:14:46Z

Workflow Telemetry - QRYN CI / build (16.x)

Workflow telemetry for commit 037eff5217ac5730f4d98d03aa230ad4f2b66f0f
You can access workflow job details here

Step Trace

gantt
	title build (16.x)
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1706105616000, 1706105617000
	Run actions/checkout@v4 : 1706105618000, 1706105618000
	Use Node.js 16.x : 1706105618000, 1706105620000
	Run npm install : 1706105620000, 1706105630000
	Run npm run postinstall : 1706105631000, 1706105631000
	Run git submodule init : 1706105631000, 1706105631000
	Run git submodule update : 1706105631000, 1706105632000
	Run docker-compose -f docker/e2e/docker-compose-cluster.yaml up -d : 1706105632000, 1706105644000
	Run sleep 5 : 1706105644000, 1706105649000
	Workflow Telemetry : 1706105649000, 1706105649000
	Run node qryn.mjs >/dev/stdout & : 1706105649000, 1706105654000
	Run npm run test --forceExit : 1706105655000, 1706105684000

CPU Metrics

Memory Metrics

IO Metrics

	Read	Write
Network I/O
Disk I/O

Process Trace

Top 100 processes with highest duration

gantt
	title build (16.x)
	dateFormat x
	axisFormat %H:%M:%S
	bash : 1706105654691, 1706105684582
	npm : 1706105654692, 1706105684578
	node : 1706105654692, 1706105684578
	jest : 1706105654952, 1706105684574
	node : 1706105654952, 1706105684574
	node : 1706105655118, 1706105655150
	find : 1706105655369, 1706105655370
	find : 1706105655377, 1706105655592
	node : 1706105655616, 1706105655726
	node : 1706105655622, 1706105655728
	node : 1706105655627, 1706105655728
	node : 1706105655751, 1706105684541
	node : 1706105655756, 1706105684527
	node : 1706105655761, 1706105684548
	node : 1706105655857, 1706105655902
	node : 1706105655861, 1706105655906
	node : 1706105655868, 1706105655901

github-actions · 2024-01-24T14:14:46Z

Workflow Telemetry - QRYN CI / build (20)

Workflow telemetry for commit 037eff5217ac5730f4d98d03aa230ad4f2b66f0f
You can access workflow job details here

Step Trace

gantt
	title build (20)
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1706105615000, 1706105616000
	Run actions/checkout@v4 : 1706105616000, 1706105617000
	Use Node.js 20 : 1706105617000, 1706105622000
	Run npm install : 1706105622000, 1706105633000
	Run npm run postinstall : 1706105633000, 1706105633000
	Run git submodule init : 1706105633000, 1706105633000
	Run git submodule update : 1706105634000, 1706105634000
	Run docker-compose -f docker/e2e/docker-compose-cluster.yaml up -d : 1706105634000, 1706105645000
	Run sleep 5 : 1706105645000, 1706105650000
	Workflow Telemetry : 1706105650000, 1706105650000
	Run node qryn.mjs >/dev/stdout & : 1706105650000, 1706105655000
	Run npm run test --forceExit : 1706105656000, 1706105685000

CPU Metrics

Memory Metrics

IO Metrics

	Read	Write
Network I/O
Disk I/O

Process Trace

Top 100 processes with highest duration

gantt
	title build (20)
	dateFormat x
	axisFormat %H:%M:%S
	bash : 1706105650663, 1706105650664
	bash : 1706105655681, 1706105685170
	npm : 1706105655682, 1706105685162
	node : 1706105655682, 1706105685162
	jest : 1706105655946, 1706105685162
	node : 1706105655946, 1706105685162
	node : 1706105656096, 1706105656128
	find : 1706105656327, 1706105656328
	find : 1706105656333, 1706105656547
	node : 1706105656565, 1706105656670
	node : 1706105656569, 1706105656665
	node : 1706105656572, 1706105656669
	node : 1706105656686, 1706105685118
	node : 1706105656689, 1706105685116
	node : 1706105656692, 1706105685116
	node : 1706105656769, 1706105656806
	node : 1706105656794, 1706105656860
	node : 1706105656802, 1706105656852

github-actions · 2024-01-24T14:14:47Z

Workflow Telemetry - QRYN CI / build (18)

Workflow telemetry for commit 037eff5217ac5730f4d98d03aa230ad4f2b66f0f
You can access workflow job details here

Step Trace

gantt
	title build (18)
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1706105615000, 1706105616000
	Initialize containers : 1706105616000, 1706105627000
	Run actions/checkout@v4 : 1706105627000, 1706105628000
	Use Node.js 18 : 1706105628000, 1706105635000
	Run npm install : 1706105636000, 1706105650000
	Run npm run postinstall : 1706105650000, 1706105651000
	Run git submodule init : 1706105651000, 1706105651000
	Run git submodule update : 1706105651000, 1706105652000
	Workflow Telemetry : 1706105652000, 1706105652000

CPU Metrics

Memory Metrics

IO Metrics

	Read	Write
Network I/O
Disk I/O

Process Trace

Top 100 processes with highest duration

gantt
	title build (18)
	dateFormat x
	axisFormat %H:%M:%S
	bash : 1706105652226, 1706105681240
	npm : 1706105652227, 1706105681235
	node : 1706105652227, 1706105681235
	jest : 1706105652552, 1706105681231
	node : 1706105652552, 1706105681231
	node : 1706105652731, 1706105652777
	find : 1706105653028, 1706105653033
	find : 1706105653042, 1706105653405
	node : 1706105653433, 1706105653555
	node : 1706105653438, 1706105653556
	node : 1706105653441, 1706105653553
	node : 1706105653573, 1706105681155
	node : 1706105653575, 1706105681162
	node : 1706105653579, 1706105681173
	node : 1706105653697, 1706105653746
	node : 1706105653714, 1706105653749
	node : 1706105653720, 1706105653758

github-actions · 2024-01-24T14:14:48Z

Workflow Telemetry - QRYN CI / build (18)

Workflow telemetry for commit 037eff5217ac5730f4d98d03aa230ad4f2b66f0f
You can access workflow job details here

Step Trace

gantt
	title build (18)
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1706105616000, 1706105617000
	Run actions/checkout@v4 : 1706105617000, 1706105618000
	Use Node.js 18 : 1706105618000, 1706105620000
	Run npm install : 1706105620000, 1706105633000
	Run npm run postinstall : 1706105634000, 1706105634000
	Run git submodule init : 1706105634000, 1706105634000
	Run git submodule update : 1706105634000, 1706105635000
	Run docker-compose -f docker/e2e/docker-compose-cluster.yaml up -d : 1706105635000, 1706105646000
	Run sleep 5 : 1706105646000, 1706105651000
	Workflow Telemetry : 1706105651000, 1706105651000
	Run node qryn.mjs >/dev/stdout & : 1706105651000, 1706105656000

CPU Metrics

Memory Metrics

IO Metrics

	Read	Write
Network I/O
Disk I/O

Process Trace

Top 100 processes with highest duration

gantt
	title build (18)
	dateFormat x
	axisFormat %H:%M:%S
	bash : 1706105651229, 1706105651230
	bash : 1706105656247, 1706105686567
	npm : 1706105656249, 1706105686563
	node : 1706105656249, 1706105686563
	jest : 1706105656534, 1706105686557
	node : 1706105656534, 1706105686557
	node : 1706105656710, 1706105656742
	find : 1706105656972, 1706105656973
	find : 1706105656978, 1706105657199
	node : 1706105657220, 1706105657311
	node : 1706105657225, 1706105657312
	node : 1706105657228, 1706105657316
	node : 1706105657336, 1706105686497
	node : 1706105657338, 1706105686510
	node : 1706105657341, 1706105686520
	node : 1706105657438, 1706105657484
	node : 1706105657456, 1706105657498
	node : 1706105657467, 1706105657520

github-actions · 2024-01-24T14:14:55Z

Workflow Telemetry - QRYN CI / build (20)

Workflow telemetry for commit 037eff5217ac5730f4d98d03aa230ad4f2b66f0f
You can access workflow job details here

Step Trace

gantt
	title build (20)
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1706105615000, 1706105617000
	Initialize containers : 1706105617000, 1706105628000
	Run actions/checkout@v4 : 1706105628000, 1706105629000
	Use Node.js 20 : 1706105629000, 1706105645000
	Run npm install : 1706105645000, 1706105657000
	Run npm run postinstall : 1706105657000, 1706105657000
	Run git submodule init : 1706105657000, 1706105657000
	Run git submodule update : 1706105658000, 1706105659000
	Workflow Telemetry : 1706105659000, 1706105659000

CPU Metrics

Memory Metrics

IO Metrics

	Read	Write
Network I/O
Disk I/O

Process Trace

Top 100 processes with highest duration

gantt
	title build (20)
	dateFormat x
	axisFormat %H:%M:%S
	bash : 1706105659267, 1706105687626
	npm : 1706105659269, 1706105687625
	node : 1706105659269, 1706105687625
	jest : 1706105659540, 1706105687617
	node : 1706105659540, 1706105687617
	node : 1706105659695, 1706105659729
	find : 1706105659955, 1706105659956
	find : 1706105659962, 1706105660351
	node : 1706105660423, 1706105660559
	node : 1706105660428, 1706105660555
	node : 1706105660431, 1706105660541
	node : 1706105660574, 1706105687569
	node : 1706105660576, 1706105687574
	node : 1706105660579, 1706105687566
	node : 1706105660696, 1706105660746
	node : 1706105660697, 1706105660793
	node : 1706105660724, 1706105660782

akvlad · 2024-01-24T14:28:11Z

@lmangani

Supported api

The api is a strange mix of grpc and HTTP 1.1.
We have some HTTP 1.1 POST requests with a protobuf message inside the body.
The response is a protobuf message as well. Here is the table of the supported requests:

Route	PB Description	RPC Description
POST /querier.v1.QuerierService/ProfileTypes	https://github.com/metrico/qryn/blob/master/pyroscope/proto/querier.proto#L10	ProfileType returns a list of the existing profile types.
POST /querier.v1.QuerierService/LabelValues	https://github.com/metrico/qryn/blob/master/pyroscope/proto/querier.proto#L12	LabelValues returns the existing label values for the provided label names.
POST /querier.v1.QuerierService/LabelNames	https://github.com/metrico/qryn/blob/master/pyroscope/proto/querier.proto#L14	LabelNames returns a list of the existing label names.
POST /querier.v1.QuerierService/SelectSeries	https://github.com/metrico/qryn/blob/master/pyroscope/proto/querier.proto#L24	SelectSeries returns a time series for the total sum of the requested profiles.
POST /querier.v1.QuerierService/SelectMergeStacktraces	https://github.com/metrico/qryn/blob/master/pyroscope/proto/querier.proto#L18	SelectMergeStacktraces returns matching profiles aggregated in a flamegraph format. It will combine samples from within the same callstack, with each element being grouped by its function name.

akvlad and others added 15 commits November 15, 2023 18:39

schema init

de97276

upd: type_id, codecs

210161e

fix: distributed tables

f4c8073

Update scripts.js

c73ae9c

Signed-off-by: Tomer Shafir <[email protected]>

Merge pull request #407 from tomershafir/patch-1

30eba6d

Update scripts.js

fix: pyroscope series inconsistencies

d6bc8e1

drop profile_id

bf95d2d

selectMergeStacktraces rust impl

0286f27

fix: configurable timespan

9fb65ba

selectSeries handler

de55696

profiling api support init

fbf6828

pyroscope api init entry point

20cd680

cherry-pick the pyroscope folder from the pyroscope branch

8f52a86

Merge branch 'master' into pyroscope-2

bffac94

# Conflicts: # lib/db/clickhouse.js # lib/db/maintain/index.js # qryn_bun.mjs

fix conflicts

e081cb0

akvlad added 2 commits January 24, 2024 13:46

debug

af0d7ac

support ADVANCED_PROFILES_MERGE_LIMIT limit

8129c1a

metrico deleted a comment from github-actions bot Jan 24, 2024

akvlad added 3 commits January 24, 2024 13:59

support ADVANCED_PROFILES_MERGE_LIMIT to limit selectMergeStacktraces

0954584

support _dist table for clusters; fix init

4730177

debug: distributed tables reading; maintain scripts

037eff5

metrico deleted a comment from github-actions bot Jan 24, 2024

akvlad merged commit b0c8178 into master Jan 24, 2024
10 checks passed

akvlad deleted the pyroscope-2 branch January 24, 2024 14:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pyroscope api support #438

Pyroscope api support #438

akvlad commented Jan 24, 2024

socket-security bot commented Jan 24, 2024 •

edited

Loading

lmangani commented Jan 24, 2024 •

edited

Loading

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

akvlad commented Jan 24, 2024 •

edited

Loading

Pyroscope api support #438

Pyroscope api support #438

Conversation

akvlad commented Jan 24, 2024

socket-security bot commented Jan 24, 2024 • edited Loading

lmangani commented Jan 24, 2024 • edited Loading

github-actions bot commented Jan 24, 2024

Workflow Telemetry - QRYN CI / build (16.x)

Step Trace

CPU Metrics

Memory Metrics

IO Metrics

Process Trace

Top 100 processes with highest duration

github-actions bot commented Jan 24, 2024

Workflow Telemetry - QRYN CI / build (16.x)

Step Trace

CPU Metrics

Memory Metrics

IO Metrics

Process Trace

Top 100 processes with highest duration

github-actions bot commented Jan 24, 2024

Workflow Telemetry - QRYN CI / build (20)

Step Trace

CPU Metrics

Memory Metrics

IO Metrics

Process Trace

Top 100 processes with highest duration

github-actions bot commented Jan 24, 2024

Workflow Telemetry - QRYN CI / build (18)

Step Trace

CPU Metrics

Memory Metrics

IO Metrics

Process Trace

Top 100 processes with highest duration

github-actions bot commented Jan 24, 2024

Workflow Telemetry - QRYN CI / build (18)

Step Trace

CPU Metrics

Memory Metrics

IO Metrics

Process Trace

Top 100 processes with highest duration

github-actions bot commented Jan 24, 2024

Workflow Telemetry - QRYN CI / build (20)

Step Trace

CPU Metrics

Memory Metrics

IO Metrics

Process Trace

Top 100 processes with highest duration

akvlad commented Jan 24, 2024 • edited Loading

Supported api

socket-security bot commented Jan 24, 2024 •

edited

Loading

lmangani commented Jan 24, 2024 •

edited

Loading

akvlad commented Jan 24, 2024 •

edited

Loading