Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added language support #6

Merged
merged 7 commits into from
Apr 29, 2024
Merged

Added language support #6

merged 7 commits into from
Apr 29, 2024

Conversation

nbruder
Copy link
Collaborator

@nbruder nbruder commented Apr 25, 2024

To add the possibility to stem and lemmatize different languages, the following changes were made:

  • adding an argument language to the constructor
  • creating a large sys.data file containing all the dictionaries for different languages. Used for lemmatization
  • expanding the auxiliary replaceSpecialCharacters function, such that language specific characters are not removed
  • added a function replace_special_characters, which replaces special characters in column names by their most similar latin character
  • added test data sets for each language

@nbruder nbruder requested a review from max-pilz April 25, 2024 09:13
nbruder and others added 2 commits April 25, 2024 17:30
@nbruder
removed swedish, portuguese and italian
e73527e
@max-pilz
merge main into language_support
e711098 
Merge branch 'main' into language_support

# Conflicts:
#	NAMESPACE
@max-pilz max-pilz merged commit 5af47cc into main Apr 29, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@max-pilz max-pilz max-pilz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nbruder @max-pilz