Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AB-357: Evaluate both MB database access methods to decide which one is more efficient #401

Draft
wants to merge 129 commits into
base: master
Choose a base branch
from
Draft

AB-357: Evaluate both MB database access methods to decide which one is more efficient #401

wants to merge 129 commits into from

Conversation

amCap1712
Copy link
Member

rsh7 added 30 commits August 5, 2018 16:24
@rsh7 @paramsingh
AB-334: Setup to directly access Musicbrainz Database
e6c1eab
@rsh7 @paramsingh
Database name as in brainzutils and update requirements.txt
cf9bb35
@rsh7 @paramsingh
AB-334: Update documentation for setting up MusicBrainz database
42e3802
@rsh7 @paramsingh
Convert documentation to markdown syntax
4b3b3d5
@rsh7 @paramsingh
Add recent BrainzUtils version
3a3e9c6
@rsh7 @paramsingh
AB-338: Create MusicBrainz schema in AB and add tables
6c0bfb9
@rsh7 @paramsingh
AB:338: Add sql commands for MB tables in db testing script and fix t…
d84c898 
…est errors
@rsh7 @paramsingh
AB-338: Add primary and foreign keys for MusicBrainz tables
3a979cd
@rsh7 @paramsingh
Create indexes for columns of MB tables and add update script for sch…
67ff744 
…ema change
@rsh7 @paramsingh
Add a condition to create schema command
fe1186a
@rsh7 @paramsingh
Add separate sql files for musicbrainz schema, table, indexes, primar…
ec7e8ca 
…y & foreign keys
@rsh7 @paramsingh
Add a function to initiate MB db and a command to initialize & create…
ae72050 
… table structure
@rsh7 @paramsingh
Correct the create musicbrainz foreign keys file
dd80e11
@rsh7 @paramsingh
Add more indexes to musicbrainz indexes file and add an update file
98ef544
@rsh7 @paramsingh
Update the musicbrainz schema update sql file and delete the separate…
8d9b43a 
… one
@rsh7 @paramsingh
AB-340: Import MusicBrainz data into separate schema tables in AB dat…
dc1c999 
…abase
@rsh7 @paramsingh
Add commands to start the import in readme
41e0937
@rsh7 @paramsingh
Get data in batches, 10000 recordings data at a time and make code mo…
131b4c4 
…dular
@rsh7 @paramsingh
Get data for FK referenced tables, write docstrings & comments for ev…
c0931ca 
…ery function
@rsh7 @paramsingh
Remove repeated MB db initialization command
5b38820
@rsh7 @paramsingh
Write docstrings in imperative mood
5f4b1b3
@rsh7 @paramsingh
AB-344: Add a force recreate musicbrainz schema option
4853ebc
@rsh7 @paramsingh
Add a line to print a message of dropping the schema and capitalize f…
038f204 
…irst letters of musicbrainz
@rsh7 @paramsingh
AB-343: Add number of recording per batch in config.py
ddaa587
@rsh7 @paramsingh
AB-342: Improve logging during MusicBrainz data import
1552006
@rsh7 @paramsingh
AB-346: Import MusicBrainz data for every new recording added to AB db
ac856e1
@rsh7 @paramsingh
Include table name in - No Data Found - message
4154ff1
@rsh7 @paramsingh
Add new musicbrainz_importer module with a musicbrainz_importer scrip…
5bba175 
…t and modify a message
@rsh7 @paramsingh
Import recordings in batches and pick the limit from config file
39dd8c9
@rsh7 @paramsingh
AB-345: Log the amount of time taken to import MB data in AB
e0c644f
rsh7 and others added 16 commits August 12, 2018 19:43
@rsh7
Remove a useless print statement
9f190aa
@rsh7
Move the queries to the db module
4b615bc
@rsh7
Move update and write replication sequence queries to db module
0a43c08
@rsh7
Close the tar file
0566d77
@rsh7
Passing the variables in query and their values in a dictionary
acae8b7
@rsh7
Add docstrings to get sequence functions in data.py
302ab50
@rsh7
Add proper formatting in query functions
8a07943
@paramsingh
Merge pull request #282 from rsh7/apply_replication
0253dfa 
AB-350: Update musicbrainz schema whenever actual MusicBrainz database is updated
@rsh7
Evaluate both db access methods by simply getting data from lowlevel …
13ada58 
…and recording
@rsh7
Add a limit to the number of recordings fetched
8884214
@rsh7
Add logging instead of print and change file name in manage.py
e58f15a
@rsh7
Move queries inside the db module
5adf1a5
@rsh7
Add docstrings for the functions moved to db module
bf6873a
@rsh7
Evaluate a functionality of mbid redirects for both db access methods
f7fa076
@rsh7
Add comments stating the explainations of some lines of code
6c42c4a
@amCap1712
Merge pull request #292 from rsh7/evaluation
a815672 
AB-357: Evaluate both MB database access methods to decide which one is more efficient.
@pep8speaks
Copy link

pep8speaks commented Jun 7, 2021 •
edited
Loading

Hello @amCap1712! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 1111:49: W291 trailing whitespace
Line 1121:1: E302 expected 2 blank lines, found 1
Line 1131:65: W291 trailing whitespace
Line 1132:27: W291 trailing whitespace
Line 1133:54: W291 trailing whitespace
Line 1141:1: E302 expected 2 blank lines, found 1
Line 1147:65: W291 trailing whitespace
Line 1148:27: W291 trailing whitespace
Line 1149:54: W291 trailing whitespace

Line 8:25: E261 at least two spaces before inline comment
Line 89:9: E124 closing bracket does not match visual indentation
Line 96:131: E501 line too long (148 > 130 characters)
Line 224:28: E222 multiple spaces after operator
Line 258:26: E222 multiple spaces after operator
Line 1196:13: E203 whitespace before ':'
Line 1197:15: E203 whitespace before ':'
Line 1198:23: E203 whitespace before ':'
Line 1199:20: E203 whitespace before ':'
Line 1200:18: E203 whitespace before ':'
Line 1221:15: E203 whitespace before ':'
Line 1222:17: E203 whitespace before ':'
Line 1223:22: E203 whitespace before ':'
Line 1224:22: E203 whitespace before ':'
Line 1225:14: E203 whitespace before ':'
Line 1319:11: E225 missing whitespace around operator
Line 1554:1: E302 expected 2 blank lines, found 1
Line 1718:9: E128 continuation line under-indented for visual indent
Line 1719:9: E128 continuation line under-indented for visual indent
Line 1720:5: E124 closing bracket does not match visual indentation
Line 1983:131: E501 line too long (139 > 130 characters)
Line 2068:131: E501 line too long (149 > 130 characters)
Line 2075:131: E501 line too long (179 > 130 characters)
Line 2218:69: E222 multiple spaces after operator

Line 793:5: E303 too many blank lines (2)

Line 19:5: E303 too many blank lines (2)
Line 23:131: E501 line too long (191 > 130 characters)
Line 24:13: E128 continuation line under-indented for visual indent
Line 24:131: E501 line too long (160 > 130 characters)
Line 33:13: E128 continuation line under-indented for visual indent
Line 33:131: E501 line too long (172 > 130 characters)
Line 34:13: E128 continuation line under-indented for visual indent
Line 35:13: E128 continuation line under-indented for visual indent
Line 35:131: E501 line too long (172 > 130 characters)
Line 37:13: E128 continuation line under-indented for visual indent
Line 37:131: E501 line too long (172 > 130 characters)
Line 45:5: E303 too many blank lines (2)
Line 51:131: E501 line too long (191 > 130 characters)
Line 52:13: E128 continuation line under-indented for visual indent
Line 52:131: E501 line too long (160 > 130 characters)
Line 61:13: E128 continuation line under-indented for visual indent
Line 61:131: E501 line too long (172 > 130 characters)
Line 62:13: E128 continuation line under-indented for visual indent
Line 63:13: E128 continuation line under-indented for visual indent
Line 63:131: E501 line too long (172 > 130 characters)
Line 65:13: E128 continuation line under-indented for visual indent
Line 65:131: E501 line too long (172 > 130 characters)
Line 74:13: E128 continuation line under-indented for visual indent
Line 83:13: E128 continuation line under-indented for visual indent
Line 91:131: E501 line too long (145 > 130 characters)
Line 92:13: E128 continuation line under-indented for visual indent
Line 92:131: E501 line too long (141 > 130 characters)
Line 93:9: E124 closing bracket does not match visual indentation
Line 100:5: E303 too many blank lines (2)
Line 106:131: E501 line too long (191 > 130 characters)
Line 107:13: E128 continuation line under-indented for visual indent
Line 107:131: E501 line too long (160 > 130 characters)
Line 116:13: E128 continuation line under-indented for visual indent
Line 116:131: E501 line too long (172 > 130 characters)
Line 117:13: E128 continuation line under-indented for visual indent
Line 118:13: E128 continuation line under-indented for visual indent
Line 118:131: E501 line too long (172 > 130 characters)
Line 120:13: E128 continuation line under-indented for visual indent
Line 120:131: E501 line too long (172 > 130 characters)
Line 129:13: E128 continuation line under-indented for visual indent
Line 138:13: E128 continuation line under-indented for visual indent
Line 146:131: E501 line too long (145 > 130 characters)
Line 147:13: E128 continuation line under-indented for visual indent
Line 147:131: E501 line too long (141 > 130 characters)
Line 148:9: E124 closing bracket does not match visual indentation
Line 155:131: E501 line too long (186 > 130 characters)
Line 162:5: E303 too many blank lines (2)
Line 166:131: E501 line too long (166 > 130 characters)
Line 167:13: E128 continuation line under-indented for visual indent
Line 167:131: E501 line too long (160 > 130 characters)
Line 168:131: E501 line too long (170 > 130 characters)
Line 169:131: E501 line too long (168 > 130 characters)
Line 170:131: E501 line too long (152 > 130 characters)
Line 171:131: E501 line too long (154 > 130 characters)
Line 179:5: E303 too many blank lines (2)
Line 185:131: E501 line too long (166 > 130 characters)
Line 186:13: E128 continuation line under-indented for visual indent
Line 186:131: E501 line too long (160 > 130 characters)
Line 187:131: E501 line too long (170 > 130 characters)
Line 188:131: E501 line too long (168 > 130 characters)
Line 189:131: E501 line too long (152 > 130 characters)
Line 190:131: E501 line too long (154 > 130 characters)
Line 198:131: E501 line too long (191 > 130 characters)
Line 199:13: E128 continuation line under-indented for visual indent
Line 199:131: E501 line too long (160 > 130 characters)
Line 208:13: E128 continuation line under-indented for visual indent
Line 208:131: E501 line too long (172 > 130 characters)
Line 209:13: E128 continuation line under-indented for visual indent
Line 210:13: E128 continuation line under-indented for visual indent
Line 210:131: E501 line too long (172 > 130 characters)
Line 212:13: E128 continuation line under-indented for visual indent
Line 212:131: E501 line too long (172 > 130 characters)
Line 221:13: E128 continuation line under-indented for visual indent
Line 230:13: E128 continuation line under-indented for visual indent
Line 238:131: E501 line too long (145 > 130 characters)
Line 239:13: E128 continuation line under-indented for visual indent
Line 239:131: E501 line too long (141 > 130 characters)
Line 240:9: E124 closing bracket does not match visual indentation
Line 254:5: E303 too many blank lines (2)
Line 258:131: E501 line too long (166 > 130 characters)
Line 259:13: E128 continuation line under-indented for visual indent
Line 259:131: E501 line too long (160 > 130 characters)
Line 260:131: E501 line too long (170 > 130 characters)
Line 261:131: E501 line too long (168 > 130 characters)
Line 262:131: E501 line too long (152 > 130 characters)
Line 263:131: E501 line too long (154 > 130 characters)
Line 271:131: E501 line too long (173 > 130 characters)
Line 272:13: E128 continuation line under-indented for visual indent
Line 280:5: E303 too many blank lines (2)
Line 286:131: E501 line too long (166 > 130 characters)
Line 287:13: E128 continuation line under-indented for visual indent
Line 287:131: E501 line too long (160 > 130 characters)
Line 288:131: E501 line too long (170 > 130 characters)
Line 289:131: E501 line too long (168 > 130 characters)
Line 290:131: E501 line too long (152 > 130 characters)
Line 291:131: E501 line too long (154 > 130 characters)
Line 299:131: E501 line too long (173 > 130 characters)
Line 300:13: E128 continuation line under-indented for visual indent
Line 308:131: E501 line too long (189 > 130 characters)
Line 315:5: E303 too many blank lines (2)
Line 321:131: E501 line too long (166 > 130 characters)
Line 322:13: E128 continuation line under-indented for visual indent
Line 322:131: E501 line too long (160 > 130 characters)
Line 323:131: E501 line too long (170 > 130 characters)
Line 324:131: E501 line too long (168 > 130 characters)
Line 325:131: E501 line too long (152 > 130 characters)
Line 326:131: E501 line too long (154 > 130 characters)
Line 342:13: E128 continuation line under-indented for visual indent
Line 343:13: E128 continuation line under-indented for visual indent
Line 343:131: E501 line too long (142 > 130 characters)
Line 344:13: E128 continuation line under-indented for visual indent
Line 352:5: E303 too many blank lines (2)
Line 358:131: E501 line too long (166 > 130 characters)
Line 359:13: E128 continuation line under-indented for visual indent
Line 359:131: E501 line too long (160 > 130 characters)
Line 360:131: E501 line too long (170 > 130 characters)
Line 361:131: E501 line too long (168 > 130 characters)
Line 362:131: E501 line too long (152 > 130 characters)
Line 363:131: E501 line too long (154 > 130 characters)
Line 379:13: E128 continuation line under-indented for visual indent
Line 380:13: E128 continuation line under-indented for visual indent
Line 380:131: E501 line too long (142 > 130 characters)
Line 381:13: E128 continuation line under-indented for visual indent
Line 389:131: E501 line too long (188 > 130 characters)
Line 396:5: E303 too many blank lines (2)
Line 403:131: E501 line too long (166 > 130 characters)
Line 404:13: E128 continuation line under-indented for visual indent
Line 404:131: E501 line too long (160 > 130 characters)
Line 405:131: E501 line too long (170 > 130 characters)
Line 406:131: E501 line too long (168 > 130 characters)
Line 407:131: E501 line too long (152 > 130 characters)
Line 408:131: E501 line too long (154 > 130 characters)
Line 424:13: E128 continuation line under-indented for visual indent
Line 425:13: E128 continuation line under-indented for visual indent
Line 425:131: E501 line too long (142 > 130 characters)
Line 426:13: E128 continuation line under-indented for visual indent
Line 441:131: E501 line too long (165 > 130 characters)
Line 442:13: E128 continuation line under-indented for visual indent
Line 450:13: E128 continuation line under-indented for visual indent
Line 451:9: E124 closing bracket does not match visual indentation
Line 465:131: E501 line too long (159 > 130 characters)
Line 466:13: E128 continuation line under-indented for visual indent
Line 467:9: E124 closing bracket does not match visual indentation
Line 475:13: E128 continuation line under-indented for visual indent
Line 483:5: E303 too many blank lines (2)
Line 490:131: E501 line too long (166 > 130 characters)
Line 491:13: E128 continuation line under-indented for visual indent
Line 491:131: E501 line too long (160 > 130 characters)
Line 492:131: E501 line too long (170 > 130 characters)
Line 493:131: E501 line too long (168 > 130 characters)
Line 494:131: E501 line too long (152 > 130 characters)
Line 495:131: E501 line too long (154 > 130 characters)
Line 511:13: E128 continuation line under-indented for visual indent
Line 512:13: E128 continuation line under-indented for visual indent
Line 512:131: E501 line too long (142 > 130 characters)
Line 513:13: E128 continuation line under-indented for visual indent
Line 528:131: E501 line too long (165 > 130 characters)
Line 529:13: E128 continuation line under-indented for visual indent
Line 537:13: E128 continuation line under-indented for visual indent
Line 538:9: E124 closing bracket does not match visual indentation
Line 552:131: E501 line too long (159 > 130 characters)
Line 553:13: E128 continuation line under-indented for visual indent
Line 554:9: E124 closing bracket does not match visual indentation
Line 561:5: E303 too many blank lines (2)
Line 575:131: E501 line too long (166 > 130 characters)
Line 576:13: E128 continuation line under-indented for visual indent
Line 576:131: E501 line too long (160 > 130 characters)
Line 577:131: E501 line too long (170 > 130 characters)
Line 578:131: E501 line too long (168 > 130 characters)
Line 579:131: E501 line too long (152 > 130 characters)
Line 580:131: E501 line too long (154 > 130 characters)
Line 596:13: E128 continuation line under-indented for visual indent
Line 597:13: E128 continuation line under-indented for visual indent
Line 597:131: E501 line too long (142 > 130 characters)
Line 598:13: E128 continuation line under-indented for visual indent
Line 613:131: E501 line too long (165 > 130 characters)
Line 614:13: E128 continuation line under-indented for visual indent
Line 622:13: E128 continuation line under-indented for visual indent
Line 623:9: E124 closing bracket does not match visual indentation
Line 637:131: E501 line too long (159 > 130 characters)
Line 638:13: E128 continuation line under-indented for visual indent
Line 639:9: E124 closing bracket does not match visual indentation
Line 646:131: E501 line too long (180 > 130 characters)
Line 653:5: E303 too many blank lines (2)
Line 660:131: E501 line too long (166 > 130 characters)
Line 661:13: E128 continuation line under-indented for visual indent
Line 661:131: E501 line too long (160 > 130 characters)
Line 662:131: E501 line too long (170 > 130 characters)
Line 663:131: E501 line too long (168 > 130 characters)
Line 664:131: E501 line too long (152 > 130 characters)
Line 665:131: E501 line too long (154 > 130 characters)
Line 673:131: E501 line too long (173 > 130 characters)
Line 674:13: E128 continuation line under-indented for visual indent
Line 697:13: E128 continuation line under-indented for visual indent
Line 698:13: E128 continuation line under-indented for visual indent
Line 698:131: E501 line too long (142 > 130 characters)
Line 699:13: E128 continuation line under-indented for visual indent
Line 714:131: E501 line too long (165 > 130 characters)
Line 715:13: E128 continuation line under-indented for visual indent
Line 723:13: E128 continuation line under-indented for visual indent
Line 724:9: E124 closing bracket does not match visual indentation
Line 738:131: E501 line too long (159 > 130 characters)
Line 739:13: E128 continuation line under-indented for visual indent
Line 740:9: E124 closing bracket does not match visual indentation
Line 747:131: E501 line too long (180 > 130 characters)
Line 754:131: E501 line too long (163 > 130 characters)
Line 755:13: E128 continuation line under-indented for visual indent
Line 755:131: E501 line too long (150 > 130 characters)
Line 762:5: E303 too many blank lines (2)
Line 769:131: E501 line too long (166 > 130 characters)
Line 770:13: E128 continuation line under-indented for visual indent
Line 770:131: E501 line too long (160 > 130 characters)
Line 771:131: E501 line too long (170 > 130 characters)
Line 772:131: E501 line too long (168 > 130 characters)
Line 773:131: E501 line too long (152 > 130 characters)
Line 774:131: E501 line too long (154 > 130 characters)
Line 782:131: E501 line too long (173 > 130 characters)
Line 783:13: E128 continuation line under-indented for visual indent
Line 806:13: E128 continuation line under-indented for visual indent
Line 807:13: E128 continuation line under-indented for visual indent
Line 807:131: E501 line too long (142 > 130 characters)
Line 808:13: E128 continuation line under-indented for visual indent
Line 823:131: E501 line too long (165 > 130 characters)
Line 824:13: E128 continuation line under-indented for visual indent
Line 832:13: E128 continuation line under-indented for visual indent
Line 833:9: E124 closing bracket does not match visual indentation
Line 847:131: E501 line too long (159 > 130 characters)
Line 848:13: E128 continuation line under-indented for visual indent
Line 849:9: E124 closing bracket does not match visual indentation
Line 856:131: E501 line too long (180 > 130 characters)
Line 863:131: E501 line too long (163 > 130 characters)
Line 864:13: E128 continuation line under-indented for visual indent
Line 864:131: E501 line too long (150 > 130 characters)
Line 871:131: E501 line too long (189 > 130 characters)

Line 132:13: E124 closing bracket does not match visual indentation

Line 347:1: E302 expected 2 blank lines, found 1

Line 41:131: E501 line too long (133 > 130 characters)
Line 41:133: E502 the backslash is redundant between brackets
Line 42:131: E501 line too long (139 > 130 characters)
Line 42:139: E502 the backslash is redundant between brackets
Line 43:131: E501 line too long (136 > 130 characters)
Line 43:136: E502 the backslash is redundant between brackets
Line 49:1: E302 expected 2 blank lines, found 1
Line 230:22: E211 whitespace before '('
Line 255:131: E501 line too long (145 > 130 characters)
Line 263:42: E211 whitespace before '('
Line 269:38: E211 whitespace before '('
Line 271:30: E211 whitespace before '('
Line 286:10: E211 whitespace before '('
Line 293:131: E501 line too long (139 > 130 characters)
Line 296:18: E211 whitespace before '('
Line 319:10: E211 whitespace before '('
Line 356:14: E211 whitespace before '('
Line 359:18: E211 whitespace before '('
Line 364:14: E211 whitespace before '('

Line 8:25: E261 at least two spaces before inline comment

Line 50:81: E222 multiple spaces after operator
Line 58:99: E222 multiple spaces after operator

Line 29:1: E101 indentation contains mixed spaces and tabs
Line 29:1: W191 indentation contains tabs
Line 30:1: W191 indentation contains tabs

Comment last updated at 2021-06-16 06:51:35 UTC

@amCap1712 amCap1712 force-pushed the musicbrainz-integration-gsoc branch from 4365129 to f9c80c9 Compare June 9, 2021 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@amCap1712 @pep8speaks @rsh7 @paramsingh