Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates pymorphy2 to pymorphy3 #20

Merged
merged 1 commit into from
Oct 30, 2024

Conversation

ThHuberResearch
Copy link
Contributor

The lexical_density function does not run in Python versions more recent than 3.10. This is because of an issue in the pymorphy2 package, which is used here. See this Stack Overflow answer for details - it seems like it is using a function that has been deprecated since Python 3.0, but which was only removed in Python 3.11.

Additionally pymorphy2 is abandoned: See here.
This pull request updates the code to use pymorphy3, which is the succcessor, still being maintained (as far as I can tell) and which fixes the issue of the lexical_density function from linguaf to not work under Python 3.11 (and most likely more recent versions as well, though I only tried it on 3.11).
The tests run without any issues.

Example demonstrating the issue (using Python 3.11):

from linguaf import lexical_diversity as ld


docs = ['This is document 1.', 'This is document 2.']
print(ld.lexical_density(docs))

With pymorphy2 the above code produces the following error:

Traceback (most recent call last):
  File "/home/thomas/PycharmProjects/LinguaF/debug.py", line 5, in <module>
    print(ld.lexical_density(docs))
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/linguaf/lexical_diversity.py", line 22, in lexical_density
    lex_items = get_lexical_items(documents=documents, remove_stopwords=remove_stopwords, lang=lang)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/linguaf/descriptive_statistics.py", line 363, in get_lexical_items
    morph = pymorphy2.MorphAnalyzer()
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/.venvLocal/lib/python3.11/site-packages/pymorphy2/analyzer.py", line 224, in __init__
    self._init_units(units)
  File "/home/thomas/PycharmProjects/LinguaF/.venvLocal/lib/python3.11/site-packages/pymorphy2/analyzer.py", line 235, in _init_units
    self._units.append((self._bound_unit(unit), False))
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/.venvLocal/lib/python3.11/site-packages/pymorphy2/analyzer.py", line 246, in _bound_unit
    unit = unit.clone()
           ^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/.venvLocal/lib/python3.11/site-packages/pymorphy2/units/base.py", line 35, in clone
    return self.__class__(**self._get_params())
                            ^^^^^^^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/.venvLocal/lib/python3.11/site-packages/pymorphy2/units/base.py", line 76, in _get_params
    (key, getattr(self, key, None)) for key in self._get_param_names()
                                               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/PycharmProjects/LinguaF/.venvLocal/lib/python3.11/site-packages/pymorphy2/units/base.py", line 70, in _get_param_names
    args, varargs, kw, default = inspect.getargspec(cls.__init__)
                                 ^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?

The same code, but using pymorphy3:

/home/thomas/PycharmProjects/LinguaF/.venvLocal/bin/python /home/thomas/PycharmProjects/LinguaF/debug.py 
50.0

Process finished with exit code 0

@Perevalov
Copy link
Member

Thanks @ThHuberSG I'll test it soon and approve if everything goes well

Copy link
Member

@Perevalov Perevalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Tested it locally.

@Perevalov Perevalov merged commit b1bd77d into WSE-research:main Oct 30, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants