Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate JSON responses inside the database #21

Merged
merged 1 commit into from
Dec 6, 2023

Conversation

waldiTM
Copy link
Contributor

@waldiTM waldiTM commented Dec 6, 2023

What this PR does / why we need it:
PostgreSQL can actually generate JSON objects from scratch and aggregate existing JSON fields as well into lists. While it might not be faster, it frees the currently single threaded web server.

Also we need to use JSON instead of JSONB. The queries run about 50% faster and we don't manipulate JSON any further in the database anyway.

PostgreSQL can actually generate JSON objects from scratch and aggregate
existing JSON fields as well into lists.  While it might not be faster,
it frees the currently single threaded web server.

Also we need to use JSON instead of JSONB.  The queries run about 50%
faster and we don't manipulate JSON any further in the database anyway.
@waldiTM waldiTM marked this pull request as draft December 6, 2023 10:36
@waldiTM
Copy link
Contributor Author

waldiTM commented Dec 6, 2023

Benchmark:

Assembling JSON in web service: 510ms
Document Path:          /rest/json/cves/2.0+deb?virtualMatchString=cpe:2.3:o:debian:debian_linux:*:*:*:*:*:*:*:*
Document Length:        14515234 bytes
Concurrency Level:      1
Time taken for tests:   51.024 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      1451538800 bytes
HTML transferred:       1451523400 bytes
Requests per second:    1.96 [#/sec] (mean)
Time per request:       510.237 [ms] (mean)
Time per request:       510.237 [ms] (mean, across all concurrent requests)
Transfer rate:          27781.58 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   405  510  31.6    510     590
Waiting:      389  496  31.7    495     574
Total:        405  510  31.6    510     590
Percentage of the requests served within a certain time (ms)
  50%    510
  66%    522
  75%    530
  80%    536
  90%    551
  95%    564
  98%    579
  99%    590
 100%    590 (longest request)
Assembling JSON in database: 142ms
Document Path:          /rest/json/cves/2.0+deb?virtualMatchString=cpe:2.3:o:debian:debian_linux:*:*:*:*:*:*:*:*
Document Length:        15263315 bytes
Concurrency Level:      1
Time taken for tests:   14.195 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      1526347700 bytes
HTML transferred:       1526331500 bytes
Requests per second:    7.04 [#/sec] (mean)
Time per request:       141.947 [ms] (mean)
Time per request:       141.947 [ms] (mean, across all concurrent requests)
Transfer rate:          105008.83 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   124  142   9.4    140     191
Waiting:      110  126   9.0    124     170
Total:        124  142   9.4    140     191
Percentage of the requests served within a certain time (ms)
  50%    140
  66%    141
  75%    142
  80%    143
  90%    150
  95%    163
  98%    180
  99%    191
 100%    191 (longest request)

Interestingly the output get's quite a bit larger.

This query is also the worst case, returning almost 25k entries at once.

@waldiTM waldiTM marked this pull request as ready for review December 6, 2023 10:47
@waldiTM waldiTM merged commit 01e3839 into main Dec 6, 2023
1 check passed
@waldiTM waldiTM deleted the aggregate-json-database branch December 6, 2023 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants