multiple changes to freezing, bugfixes, code cleanup

marian-code · Oct 17, 2019 · 83fd0cf · 83fd0cf
1 parent 12aa871
commit 83fd0cf
Show file tree

Hide file tree

Showing 65 changed files with 1,477 additions and 1,178 deletions.
diff --git a/.gitignore b/.gitignore
@@ -16,10 +16,12 @@ build/
 dist/
 
 # build dir
-setup/build/
-setup/*dist/
-setup/wiki_music.spec
-setup/unnecessary_dependencies.txt
+freeze/build/
+freeze/*dist/
+freeze/upx/
+freeze/wiki_music.spec
+freeze/unnecessary_dependencies.txt
+freeze/numpy-1.16.5+vanilla-cp37-cp37m-win_amd64.whl
 
 # dir for file output, profiling, logs
 wiki_music/logs/
@@ -42,4 +44,7 @@ deprecated/
 # built version of documentation
 docs/_build/
 docs/_static/
-docs/_templates/
+docs/_templates/
+
+# PyPi tokens
+.pypirc
diff --git a/README.md b/README.md
@@ -37,7 +37,7 @@ For more detailes refer to Documentation which can be found at:
 
 Anyone is welcome to use it or contribute. All of the dependencies are fairly
 common so you shouldn't encounter any problems. Curentlly supported versions of
-python are **3.6** and **3.7**.
+python are **3.6** - **3.8**.
 
 ## Bugs & Features
 

diff --git a/code_changelog.md b/code_changelog.md
@@ -1,16 +1,14 @@
 # To-Do
 
 ### Main problems ordered by targeted release
-- 0.4a0
-  - make docs 'about' page and open right url from GUI
-  - test frozen gui app
-  - create a github release
 - 0.5a0
+  - fix extraction for endless forms most beautiful
   - fix as many bugs as possible
+  - make parallel freezing, and package for release
+  - fix gui startup and show file speed, too slow, mybe big cover art?
   - preload needs a complete rewrite the logic is horribly complex, too many
     classes manipulate preload related variables
 - 0.xb0
-  - use underscores to mark private things
   - fix gui scaling and elements moving around
   - convert constants to re patterns for better matching and use more re for better extraction
   - try to setup some CI system
@@ -22,7 +20,8 @@
   - support more music formats
 
 ### Freezing problems
-- try compression options - upx
+- upx probably messes some dll, PIXmap does not work, pictures are blank
+- pyinstaller now does not include wiki_music in frozen app
 
 ### Ideas
 - parser probably should have its own lock? - access to its mutable variables should be guarded  see 13.1.2019 entry in changelog
@@ -36,8 +35,8 @@
 - research PIL interface to PyQt, and what about Pyside?
 - cover art search could anounce new downloaded images by signals if we were using QThreads
 - use custom widgets to simplify GUI https://www.learnpyqt.com/courses/qt-creator/embed-pyqtgraph-custom-widgets-qt-app/ e.g. tableWiew
-- implement main GUI progressbar
 - parser locks could be implemented easilly by getattr and set attr only for public attributes.
+- cells with dropdowns for subtracks
 
 ###  Individual problem cases 
 - load guests as in https://en.wikipedia.org/wiki/Emerald_Forest_and_the_Blackbird
@@ -47,6 +46,25 @@
 
 # Change Log
 
+### 16.10.2019 0.4a0
+- selenium dependency was caused by high download limit for
+  google_images_download, limit is now set to 100
+- fixed premature preload start
+- experiment with UPX
+- cleanup in parser.in_out
+- some dlls must be excluded from UPX compression otherwise they are messed up
+  and the executable is not working
+- we got ~37% reduction in size of GUI app and 25% for CLI app
+
+### 15.10.2019 - 0.3a4
+- fixed some gui scaling problems
+- parser get methods now return so they can be used directly
+- fixed marking of private methods in parser
+- added GUI progressbar
+- implemeted theadpool progressbar
+- added selenium dependency for google_images_download
+- added pypiwin32 dependency for building frozen app
+
 ### 14.10.2019 - 0.3a4
 - completelly reworked logging
 - finally readthedocs build is passing at long last, had to install with pip

diff --git a/docs/api.rst b/docs/api.rst
@@ -1,16 +1,20 @@
 API reference
 =============
 
+.. warning:: 
+    Documentation is stil under construction some things might not be up to
+    date.
 
 This file describes wiki_music API reference. Beware, some things might not be
 up to date and all is subject to change since we are still in early
 development phase.
 
-We often use throughout the documentation notation same as python 
-`typing <https://docs.python.org/3/library/typing.html>`_. 
-module to mark variable types as it is richer and preserves more information.
-e.g. List[str] obviously means list of strings. More on the matter can be
-read in the typing module documentation.
+.. note::
+    We often use throughout the documentation notation same as python 
+    `typing <https://docs.python.org/3/library/typing.html>`_. 
+    module to mark variable types as it is richer and preserves more
+    information. e.g. List[str] obviously means list of strings. More on the
+    matter can be read in the typing module documentation.
 
 .. toctree::
    :maxdepth: 2

diff --git a/docs/api_constants.rst b/docs/api_constants.rst
@@ -2,9 +2,8 @@ wiki_music.constants module
 ===========================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 constants
 ---------

diff --git a/docs/api_external_libraries.rst b/docs/api_external_libraries.rst
@@ -2,9 +2,8 @@ wiki_music.external_libraries module
 ====================================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 Here the quality of the documentation is not guaranted since we rely only on
 the docstrings provided by the autors of respectable packages, which are

diff --git a/docs/api_gui.rst b/docs/api_gui.rst
@@ -2,9 +2,8 @@ wiki_music.gui_lib module
 =========================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 gui_lib.qt_importer
 -------------------

diff --git a/docs/api_library_lyrics.rst b/docs/api_library_lyrics.rst
@@ -2,9 +2,8 @@ wiki_music.library.lyrics module
 ================================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 library.lyrics
 --------------

diff --git a/docs/api_library_parser.rst b/docs/api_library_parser.rst
@@ -2,9 +2,8 @@ wiki_music.library.parser module
 ================================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 library.parser.WikipediaRunner
 ------------------------------

diff --git a/docs/api_library_tags.rst b/docs/api_library_tags.rst
@@ -2,9 +2,8 @@ wiki_music.library.tags_handler and tags_io modules
 ===================================================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 library.tags_io
 ---------------

diff --git a/docs/api_utilities.rst b/docs/api_utilities.rst
@@ -2,9 +2,8 @@ wiki_music.utilities module
 ===========================
 
 .. warning:: 
-    This file describes wiki_music API reference. Beware, some things might not
-    be up to date and all is subject to change since we are still in early
-    development stage.
+    Documentation is stil under construction some things might not be up to
+    date.
 
 utilities.gui_utils
 -------------------

diff --git a/docs/contributing.rst b/docs/contributing.rst
@@ -1,6 +1,10 @@
 Contributing Guide
 ==================
 
+.. warning:: 
+    Documentation is stil under construction some things might not be up to
+    date.
+
 Running the tests
 -----------------
 

diff --git a/docs/index.rst b/docs/index.rst
@@ -6,7 +6,9 @@
 Welcome to Wiki music's documentation!
 ======================================
 
-Documentation is still under construction !!! but most parts are commplete.
+.. warning:: 
+    Documentation is stil under construction some things might not be up to
+    date.
 
 Projects home directory is:
 `wikipedia-music-tags <https://github.com/marian-code/wikipedia-music-tags>`_. 

diff --git a/docs/instalation.rst b/docs/instalation.rst
@@ -17,8 +17,15 @@ for GUI and one for CLI version.
 
 .. code-block:: bash
 
-    wiki_music_gui(.exe)
-    wiki_music_cli(.exe)
+    wiki-music-gui(.exe)
+    wiki-music-cli(.exe)
+
+There are also binary releases available on
+`Github Releases <https://github.com/marian-code/wikipedia-music-tags/releases>`_.
+
+.. warning::
+    Binary releases are still in very early development stage and may not work
+    properly.
 
 For Developers
 --------------
@@ -36,7 +43,7 @@ With only some minor modifications it should be able to run on Linux and
 Os X too. Problems concern mainly default paths and interaction with
 clipboard in GUI.
 
-Multiple Qt backends are supported with the help of QtPy. So you can substitute 
+Multiple Qt backends are supported thanks to QtPy. So you can substitute 
 PyQt5 for: PyQt4, PySide2 or PySide. However, for now only compatibility with 
 PyQt5 is tested, so naturally it is also recomended.
 
@@ -103,8 +110,8 @@ Building frozen app
     you can use option ``--exclude-module=<module_name>`` in freeze.py to
     exclude unwanted libraries. See section `Creating virtual environment`_
 
-There is one optional optimization which you can do before building frozen app.
-You can use 'vanilla' numpy to further reduce size of freezed app. Vanilla
+There are few optional optimization which you can do before building frozen
+app. You can use 'vanilla' numpy to further reduce size of freezed app. Vanilla
 numpy build can be downloaded from here:
 `numpy vanilla <https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy>`_.
 The problem with regular numpy is building against OPENBLAS (pip version ~40MB)
@@ -115,6 +122,15 @@ for your python version, install it by:
 
     pip install <package-name>.whl
 
+Other than that you can use `UPX <https://upx.github.io>`_ to compress the app
+to a smaller size. It prooves to be quite effective reducing app size..
+If you want to use it go to the provided link and download apropriate
+version for your system. Then unpack it in upx folder under wiki_music/freeze.
+
+.. warning::
+    This is not recomended in debugging stage as it adds another layer of
+    complexity.
+
 Now you are ready to go:
 
 .. code-block:: bash
@@ -127,17 +143,19 @@ To build the CLI app:
 
     python freeze.py cli
 
-When building in virtual env the frozen app should have ~ 75MB.
-Without vanilla numpy ~105MB.
+When building in virtual env the frozen app should have ~75MB.
+With UPX compression and vanilla Numpy ~56MB
+With OPENBLAS numpy and UPX compression ~105MB.
 
 To build the GUI app:
 
 .. code-block:: bash
 
     python freeze.py gui
 
-When building in virtual env the frozen app should have ~ 120MB.
-Without vanilla numpy ~150MB.
+When building in virtual env the frozen app should have ~120MB.
+With UPX compression and vanilla Numpy ~75MB
+With OPENBLAS numpy and UPX compression ~150MB.
 
 This will generate list three directories under setup/ folder: gdist/ cdist/
 and build/. Build contains just pyinstaller help files and **(g/c)dist/wiki_music**

diff --git a/docs/usage.rst b/docs/usage.rst
@@ -1,6 +1,10 @@
 Usage instructions
 ==================
 
+.. warning:: 
+    Documentation is stil under construction some things might not be up to
+    date.
+
 As an application
 -----------------
 
@@ -9,13 +13,13 @@ use two scripts provided by the package:
 
 .. code-block:: bash
 
-    wiki_music_cli(.exe)
+    wiki-music-cli(.exe)
 
 which runs the CLI app, and
 
 .. code-block:: bash
 
-    wiki_music_gui(.exe)
+    wiki-music-gui(.exe)
 
 which runs the GUI app. Both can be run with -h or --help to list addinional
 command line parameters.
@@ -84,16 +88,10 @@ You can initialize the parser and call its methods:
     'https://en.wikipedia.org/wiki/When_a_Shadow_Is_Forced_into_the_Light'
     >>> parser.cook_soup()
     >>> parser.get_contents()
-    >>> parser.contents
-    ['Track listing', 'Personnel', 'Charts', 'References']
-    >>> parser.get_contents()
-    >>> parser.contents
     ['Track listing', 'Personnel', 'Charts', 'References']
     >>> parser.get_genres()
-    >>> parser.genres
     ['Post-metal', 'gothic metal', 'black metal', 'doom metal']
     >>> parser.get_personnel()
-    >>> parser.personnel
     ['Mikko Kotamäki',  'Juho Räihä', 'Juha Raivio', 'Jaani Peuhu',
      'Matti Honkonen', 'Juuso Raatikainen']
 

diff --git a/setup/custom_hooks/__init__.py → freeze/custom_hooks/__init__.py b/setup/custom_hooks/__init__.py → freeze/custom_hooks/__init__.py
diff --git a/setup/custom_hooks/hook-PyQt5.py → freeze/custom_hooks/hook-PyQt5.py b/setup/custom_hooks/hook-PyQt5.py → freeze/custom_hooks/hook-PyQt5.py
diff --git a/setup/custom_hooks/hook-nltk.py → freeze/custom_hooks/hook-nltk.py b/setup/custom_hooks/hook-nltk.py → freeze/custom_hooks/hook-nltk.py