Keep track of freestyle changes #1

renormalist · 2015-08-24T13:17:49Z

This is an experimental PR to track the changes of which I know are forked from the original reality.

...to correctly handle explicit LICENSE file.

Basically all =over's were unclosed, missing =back. Fixed that.

The items seemed to look like a numbered list but in fact meant the two hashes to be provided. Reowrd this to use '1st hash' and '2nd hash' making it less confusing.

...so we can automatically correlate the correct file.

...so they get available using File::ShareDir, like for creating the DB with cmdline tools.

Basically copy'n'pasted from ::Query::mysql, and replaced NOW() with CURRENT_TIMESTAMP. More changes are expected to come while reverse engineering how SQLite differs from MySQL... :-)

...in order to find the bug on SQLite: no such column: bav2.bench_additional_value

- Initial release - CPAN DAY - 2015-08-16 - based on the released code from early 2014 - This is a library to provide a schema for storing benchmark values in an SQL database - Primarily targets mysql, SQLite support is basically so far by cutting away known trouble maker SQL statement parts - Read more about the schema on http://benchmarkanything.org/

...to allow ultimate precision.

This avoids the error DBD::SQLite::db prepare failed: no such column: bav2.bench_additional_value at .../Tapper/Benchmark/Query.pm probably by correctly using the SQLite pragmas full_column_names short_column_names which at least can be configured to reproduce the error in plain sqlite3 (though I don't have proof that this is the ultimate solution to that problem).

- store benchmark VALUEs as VARCHAR to avoid rounding and allow non-number values en passant - dependency++ to DBD::SQLite 1.48 to avoid error 'no such column: bav2.bench_additional_value"

For key=value pairs where the key is undefined we skip the whole pair. I am not sure it makes sense but I need to avoid db errors resulting from that.

When we insert similar entries it used to insert similar fields and failed. Trying to fix this with simple INSERT IGNORE statements, for mysql and SQLite respectively.

For some reason it seems to Just Work(tm) in SQLite. Let's see if explicitely updating the last inserted id in mysql does the Right Thing.

- handling of duplicates and undefined values

Quite some code duplication but we are in stupid extension mode, refactoring later this week.

- get list of benchmark NAMEs

I am not exactly sure if this could be nested but anyway we push/pop the old values so at least that part works nested.

Currently this only covers the cleanup of the queued and already processed raw_bench_bundle rows, but other cleanup can be handled here, too.

- queuing mode for high-throughput incoming results * part 1) adds raw results into separate table * part 2) process one of those values, likely called in bundles or from multiple workers * part 3) garbage collect successfully processed results

When run with the same dbh and the queuing code running with transactions, it seems the garbage collection also needs transactions.

- enable transactions for garbage collector

Nested transactions don't work anyway, so the use of a queue with push and pop was over-engineered. Drop it to just setting a single old value.

Be crisp about UNSIGNED and use BLOB to never hit any column limits, just in case...

...when all raw_bench_bundles are processed.

- DB tuning/tweaking: * avoid deadlocks due to gap locking * fix missing finish_transaction * be crisp about UNSIGNED and use BLOB for raw queue * simplify resetting of AutoCommit mode

...and replace given/when with classic if/elsif/else cascades.

- drop feature 'switch' - replace given/when with classic if/else

...to save cpu cycles and let this handle the database.

- let the db take care of compression in raw queue

...into a ::common.pm class.

- refactor common methods between db backends

renormalist added 30 commits August 3, 2015 18:04

experiments to get an SQLite'ish create schema

343bb06

dist.ini: explicit github location

842ae5a

.gitignore

a436903

dzil: use @tapper bundle

8814e4c

...to correctly handle explicit LICENSE file.

docs: fix POD glitches

b8a4cad

Basically all =over's were unclosed, missing =back. Fixed that.

docs: fix confusing enumeration

e6d83f8

The items seemed to look like a numbered list but in fact meant the two hashes to be provided. Reowrd this to use '1st hash' and '2nd hash' making it less confusing.

docs: provide podweaver ABSTRACTs

87757d9

rename DB files according to DBD driver

1ef1bcc

...so we can automatically correlate the correct file.

mv schema DB files to share/

0582ccd

...so they get available using File::ShareDir, like for creating the DB with cmdline tools.

dzil: disable POD coverage tests

d647693

query backend: SQLite

2255b9b

Basically copy'n'pasted from ::Query::mysql, and replaced NOW() with CURRENT_TIMESTAMP. More changes are expected to come while reverse engineering how SQLite differs from MySQL... :-)

docs: fix order_by examples (argument order)

6bef415

fix: experiment with quoting

24a0f68

...in order to find the bug on SQLite: no such column: bav2.bench_additional_value

changelog++

6b60ebb

store bench_values as VARCHAR

0e04157

...to allow ultimate precision.

dzil: +Test::Compile

0803605

changelog++

bdc8134

improve error message

0fb0718

changelog++

a501dab

v0.002

5a34d32

- store benchmark VALUEs as VARCHAR to avoid rounding and allow non-number values en passant - dependency++ to DBD::SQLite 1.48 to avoid error 'no such column: bav2.bench_additional_value"

skip undefined values

47de833

For key=value pairs where the key is undefined we skip the whole pair. I am not sure it makes sense but I need to avoid db errors resulting from that.

ignore duplicate inserts

229ce79

When we insert similar entries it used to insert similar fields and failed. Trying to fix this with simple INSERT IGNORE statements, for mysql and SQLite respectively.

more duplicate handling

5695a61

For some reason it seems to Just Work(tm) in SQLite. Let's see if explicitely updating the last inserted id in mysql does the Right Thing.

changelog++

3940af0

v0.003

3bf9ece

- handling of duplicates and undefined values

get list of benchmark NAMEs

e181cec

Quite some code duplication but we are in stupid extension mode, refactoring later this week.

changelog++

e9d6299

v0.004

0ab0a17

- get list of benchmark NAMEs

renormalist added 30 commits September 8, 2015 10:56

code hygiene

503eac5

transaction handling - restore autocommit mode

5a3f467

I am not exactly sure if this could be nested but anyway we push/pop the old values so at least that part works nested.

garbage collector

80f5cd4

Currently this only covers the cleanup of the queued and already processed raw_bench_bundle rows, but other cleanup can be handled here, too.

fix syntax error

507cd5b

check POD for T::Benchmark now, but skip others

c8bd6d1

docs++

833875f

changelog++

62a0436

v0.007

010096c

- queuing mode for high-throughput incoming results * part 1) adds raw results into separate table * part 2) process one of those values, likely called in bundles or from multiple workers * part 3) garbage collect successfully processed results

transactions for gc()

fe0c010

When run with the same dbh and the queuing code running with transactions, it seems the garbage collection also needs transactions.

changelog++

8dfd310

v0.008

b57ca98

- enable transactions for garbage collector

simplify reset of AutoCommit mode

073613f

Nested transactions don't work anyway, so the use of a queue with push and pop was over-engineered. Drop it to just setting a single old value.

db schema type tweaks

0ff1710

Be crisp about UNSIGNED and use BLOB to never hit any column limits, just in case...

fix missing finish_transaction

366f7ed

...when all raw_bench_bundles are processed.

tuning: avoid deadlocks due to gap locking (under mysql)

98e098b

fix missing LIMIT 1 for queue workers

15c3ba0

changelog++

a69edb6

v0.009

1a56241

- DB tuning/tweaking: * avoid deadlocks due to gap locking * fix missing finish_transaction * be crisp about UNSIGNED and use BLOB for raw queue * simplify resetting of AutoCommit mode

drop feature 'switch'

d083df0

...and replace given/when with classic if/elsif/else cascades.

changelog++

6d68de4

v0.010

d6c5cab

- drop feature 'switch' - replace given/when with classic if/else

fix getting db driver name

f938266

do not compress in application side

aabcb77

...to save cpu cycles and let this handle the database.

mysql: enable compression in raw queue

6cad86f

changelog++

f3598fe

v0.011

793f143

- let the db take care of compression in raw queue

doc typo

412db2a

Refactor common query methods

d74f274

...into a ::common.pm class.

changelog++

52797e2

v0.012

dd639af

- refactor common methods between db backends

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keep track of freestyle changes #1

Keep track of freestyle changes #1

renormalist commented Aug 24, 2015

Keep track of freestyle changes #1

Are you sure you want to change the base?

Keep track of freestyle changes #1

Conversation

renormalist commented Aug 24, 2015