arwiki_parser is a small python script for extracting plain text articles from Arabic Wikipedia dumps. Grab the fresh dump from here, extract it, and give arwiki_parser a try.
The scripts should run on any unix-like system with Python 2.7 installed along with pip or easy_install to install additional packages.
The following third-party libraries neead to be installed as well:
- Beautiful Soup 4:
$ pip install beautifulsoup4 - mwlib:
$ pip install mwlib mwlib.xhtml - lxml:
$ pip install lxml
$ python arwiki_parser.py path/to/dump.xml path/to/output/dir/
The script will extract each article into separate files. To make dealing with the files easier on a window manager, the files are distributed across 256 directories with hexadecimal names 00-ff. All articles are stored as <article_id>.txt. The first line in each file is always the title of the article.
arwiki_parser - Extract plain text from Arabic Wikipedia dumps. Copyright (C) 2014 Ossama W. Obeid
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.