Examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples Chaining search\n",
    "This notebook contains a number of examples of chaining linguistic resources: corpora, lexica and treebanks. Try the examples, or copy the code and customize the examples in the [Sandbox](Sandbox.ipynb).\n",
    " * For a tutorial, refer to our [Quickstart](Quickstart.pdf).\n",
    " * Reference of our library *chaininglib*, described in the documentation ([local](doc/_build/html/index.html) or [online](https://chaining-search.readthedocs.io/en/latest/)).\n",
    " * If you encounter any bugs or errors, please let us know via our [GitHub issue tracker](https://github.com/INL/chaining-search/issues) or send an e-mail to servicedesk@ivdnt.org.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List of examples\n",
    "### Corpora\n",
    " * [Corpus search](#corpus-search)\n",
    " * [Frequency of *zeker*+verb and *vast*+verb compared in a historical corpus](#freq-puur-zuiver)\n",
    " * [Frequency of *leuk*+verb and *fijn*+verb compared in a modern corpus](#freq-leuk-fijn)\n",
    " * [Search in corpus and filter on metadata](#corpus-filter-metadata)\n",
    " * [Visualizing h-dropping](#visualizing-h-dropping)\n",
    " * [Generate a lexicon from a corpus](#lexicon-from-corpus)\n",
    " * [Train a POS tagger on an annotated historical corpus](#pos-tagger)\n",
    "\n",
    "### Lexica\n",
    " * [Lexicon search](#lexicon-search)\n",
    "\n",
    "### Corpus + lexicon\n",
    " * [Build a frequency list of the lemma of some corpus output](#freq-lemma-corpus)\n",
    " * [Find occurences of attributive adjectives not ending with -e, even though they are preceeded by a definite article](#adjective-e)\n",
    " * [Look up inflected forms and spelling variants for a given lemma in a corpus](#inflected-spelling-corpus)\n",
    " * [Corpus frequency list of lemmata from lexicon with given lemma](#corpus-frequency-lemma-pos)\n",
    " * [Build a frequency table of some corpus, based on lemmata of a given lexicon](#freqtable-lemmalist)\n",
    " * [Search corpus for wordforms of lemma not included in lexicon](#corpus-wordforms-not-lexicon)\n",
    " \n",
    "### Treebanks\n",
    " * [Treebank search](#treebank-search)\n",
    " * [Which objects of verb *geven* occur?](#treebank-objects-geven)\n",
    " ---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T10:51:07.839335Z",
     "start_time": "2019-03-04T10:51:07.835114Z"
    }
   },
   "source": [
    "## Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus search <a class=\"anchor\" id=\"corpus-search\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Run the cell below to show the UI, and fill in your search query\n",
    "* Choose one of the corpora:\n",
    "  * **zeebrieven**: The Brieven als Buit (Letters as Loot) corpus, consisting of 17th and 18th century letters from Dutch sailors\n",
    "  * **gysseling**: Corpus Gysseling, 13th century Dutch\n",
    "  * **openchn**: Externally accessible part of the Corpus Hedendaags Nederlands. Corpus of contemporary Dutch from the Dutch Antilles and Suriname, retrieved from newspapers and websites.\n",
    "  * **opus**: OPUS corpus of Dutch subtitles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-22T16:18:40.333209Z",
     "start_time": "2019-08-22T16:18:39.971563Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "930596b0d7574798bac848a0d649b2a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='[lemma=\"boek\"]', description='<b>CQL query:</b>'), Dropdown(description='<b>Corpus:…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from chaininglib.ui.search import create_corpus_ui\n",
    "from chaininglib.ui.dfui import display_df\n",
    "\n",
    "# Create corpus UI, creates references to field contents\n",
    "corpusQueryField, corpusField = create_corpus_ui()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * Click the cell below and press Run to perform the given query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-22T16:18:43.626160Z",
     "start_time": "2019-08-22T16:18:43.201709Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[F                                                                    "
     ]
    },
    {
     "data": {
      "text/html": [
       "<b>Results</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1020d17cb6a4fb8a6c4498b5dfaeb93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='Results.csv'), Button(button_style='warning',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from chaininglib.search.CorpusQuery import *\n",
    "\n",
    "query= corpusQueryField.value\n",
    "corpus_name = corpusField.value\n",
    "\n",
    "# BEWARE: we limit the results to 500 records here; not limiting may cause a search to take quite some time\n",
    "df_corpus = create_corpus(corpus_name).pattern(query).max_results(500).search().kwic()\n",
    "display_df(df_corpus, labels=\"Results\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency of *zeker*+verb and *vast*+verb compared <a class=\"anchor\" id=\"freq-puur-zuiver\"></a>\n",
    "* Below cell searches for *zeker*+verb and for *vast*+verb in the Letters as Loot (zeebrieven) corpus\n",
    "* Compare frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-20T09:38:11.274739Z",
     "start_time": "2019-08-20T09:38:10.443198Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[F                                                                    "
     ]
    },
    {
     "data": {
      "text/html": [
       "<b>zeker</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>left context</th>\n",
       "      <th>lemma 0</th>\n",
       "      <th>pos 0</th>\n",
       "      <th>word 0</th>\n",
       "      <th>lemma 1</th>\n",
       "      <th>pos 1</th>\n",
       "      <th>word 1</th>\n",
       "      <th>right context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>onder weeg is UE sal</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADV</td>\n",
       "      <td>seeker</td>\n",
       "      <td>kunnen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>kunne</td>\n",
       "      <td>sien hoi haer ge stael</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>maken soo salt maer alte</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADV</td>\n",
       "      <td>seckr</td>\n",
       "      <td>voortgaan</td>\n",
       "      <td>VRB</td>\n",
       "      <td>voort</td>\n",
       "      <td>gaen daerom meene ick het</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>dit zoo zijnde, dat niet</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>zeker</td>\n",
       "      <td>weten</td>\n",
       "      <td>VRB</td>\n",
       "      <td>weet,</td>\n",
       "      <td>kom in t geval van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>oope dat Ick daer van</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>seeker</td>\n",
       "      <td>zullen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>sal</td>\n",
       "      <td>weese maer het sal noch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>doch ick en kan nietmendal</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>seekers</td>\n",
       "      <td>schrijven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>schrijuen,</td>\n",
       "      <td>ende verhoope UE in corte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>niet en kan men niet</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADV</td>\n",
       "      <td>seecker</td>\n",
       "      <td>schrijven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>Schrijven</td>\n",
       "      <td>God gun en geeft ons</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>Nes UWEGb genoegen, hie sal</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADV</td>\n",
       "      <td>seeker</td>\n",
       "      <td>melden</td>\n",
       "      <td>VRB</td>\n",
       "      <td>gemeld</td>\n",
       "      <td>hebben de doot van suster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>kan ik uw ook niets</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADV</td>\n",
       "      <td>seeker</td>\n",
       "      <td>melden</td>\n",
       "      <td>VRB</td>\n",
       "      <td>melden,</td>\n",
       "      <td>want die producten dat wij</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>mijn vertrek kan ik niets</td>\n",
       "      <td>zeker</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>zeeker</td>\n",
       "      <td>melden</td>\n",
       "      <td>VRB</td>\n",
       "      <td>melde</td>\n",
       "      <td>denke in ’t Laast van</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  left context lemma 0 pos 0   word 0    lemma 1 pos 1  \\\n",
       "0         onder weeg is UE sal   zeker   ADV   seeker     kunnen   VRB   \n",
       "1     maken soo salt maer alte   zeker   ADV    seckr  voortgaan   VRB   \n",
       "2     dit zoo zijnde, dat niet   zeker   ADJ    zeker      weten   VRB   \n",
       "3        oope dat Ick daer van   zeker   ADJ   seeker     zullen   VRB   \n",
       "4   doch ick en kan nietmendal   zeker   ADJ  seekers  schrijven   VRB   \n",
       "5         niet en kan men niet   zeker   ADV  seecker  schrijven   VRB   \n",
       "6  Nes UWEGb genoegen, hie sal   zeker   ADV   seeker     melden   VRB   \n",
       "7          kan ik uw ook niets   zeker   ADV   seeker     melden   VRB   \n",
       "8    mijn vertrek kan ik niets   zeker   ADJ   zeeker     melden   VRB   \n",
       "\n",
       "       word 1               right context  \n",
       "0       kunne      sien hoi haer ge stael  \n",
       "1       voort   gaen daerom meene ick het  \n",
       "2       weet,          kom in t geval van  \n",
       "3         sal     weese maer het sal noch  \n",
       "4  schrijuen,   ende verhoope UE in corte  \n",
       "5   Schrijven        God gun en geeft ons  \n",
       "6      gemeld   hebben de doot van suster  \n",
       "7     melden,  want die producten dat wij  \n",
       "8       melde       denke in ’t Laast van  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff8b633d5b894fa48a7572d9e8d7ce3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='zeker.csv'), Button(button_style='warning', d…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[F                                                                    "
     ]
    },
    {
     "data": {
      "text/html": [
       "<b>vast</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>left context</th>\n",
       "      <th>lemma 0</th>\n",
       "      <th>pos 0</th>\n",
       "      <th>word 0</th>\n",
       "      <th>lemma 1</th>\n",
       "      <th>pos 1</th>\n",
       "      <th>word 1</th>\n",
       "      <th>right context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>M.Dalij, die heeft mij voor</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>beloven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>beloofd,</td>\n",
       "      <td>de helfte van sijne Reekening</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>tog kan ik het niet</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>schrijven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>schryven</td>\n",
       "      <td>want daris somtys gen stat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>ghe sicht dat wij voor</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>vertrouwen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>vertrouden</td>\n",
       "      <td>dat altemael turcken waeren wij</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>bessten koomen dan gij kend</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>geloven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>gelooven</td>\n",
       "      <td>dat gij mij niet half</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>ik kan het niet voor</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>schrijven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>schryve</td>\n",
       "      <td>Dog hoop ik dat Got</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>Vertrek Kan ik nog niet</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>fast</td>\n",
       "      <td>melden</td>\n",
       "      <td>VRB</td>\n",
       "      <td>Melden</td>\n",
       "      <td>want Het is Hier Zeer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>godt vartrout dij heft soo</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>bouwen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>gebout</td>\n",
       "      <td>hijer mede wens jck mij</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>en dan sal wy wel</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vas</td>\n",
       "      <td>afdanken</td>\n",
       "      <td>VRB</td>\n",
       "      <td>afgedak</td>\n",
       "      <td>worden moeder doet de groetenis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>daar zyn E myn soo</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>beloven</td>\n",
       "      <td>VRB</td>\n",
       "      <td>belooft</td>\n",
       "      <td>heeft geen schip te laten</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>en dat agter ’t schip</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>zijn</td>\n",
       "      <td>VRB</td>\n",
       "      <td>was</td>\n",
       "      <td>hebben wy verlooren, een groote</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>[…] waer op u l</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>mogen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>moocht</td>\n",
       "      <td>vertrouwen wij sijn noch altesamen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>aff scheijden &amp; met die</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>vaste</td>\n",
       "      <td>hopen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>hoope,</td>\n",
       "      <td>in onse loop baan wandelen,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>hoop Ick dat het nu</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>zullen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>sal</td>\n",
       "      <td>gaen daerom versuijmt nu geen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>in leyt want komt hy</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>terechtkomen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>te</td>\n",
       "      <td>regt want de heer en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>beste van Niet hebbe het</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>voornemen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>voorgenoomen</td>\n",
       "      <td>Zoo godt wel om bij</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>zyn, dat ik in gedagte</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>zijn</td>\n",
       "      <td>VRB</td>\n",
       "      <td>ben</td>\n",
       "      <td>geweest, hier geleegentheyd was dezelve</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>want ’t zal zyn leeven</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>verlengen</td>\n",
       "      <td>VRB</td>\n",
       "      <td>verlengen,</td>\n",
       "      <td>en buyten dien, zyn de</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>is van consept ik hadt</td>\n",
       "      <td>vast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>vast</td>\n",
       "      <td>denken</td>\n",
       "      <td>VRB</td>\n",
       "      <td>gedagt</td>\n",
       "      <td>dat uwed= gerippatriejeerdt soude hebbe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   left context lemma 0 pos 0 word 0       lemma 1 pos 1  \\\n",
       "0   M.Dalij, die heeft mij voor    vast   ADV   vast       beloven   VRB   \n",
       "1           tog kan ik het niet    vast   ADV   vast     schrijven   VRB   \n",
       "2        ghe sicht dat wij voor    vast   ADV   vast    vertrouwen   VRB   \n",
       "3   bessten koomen dan gij kend    vast   ADV   vast       geloven   VRB   \n",
       "4          ik kan het niet voor    vast   ADV   vast     schrijven   VRB   \n",
       "5       Vertrek Kan ik nog niet    vast   ADV   fast        melden   VRB   \n",
       "6    godt vartrout dij heft soo    vast   ADV   vast        bouwen   VRB   \n",
       "7             en dan sal wy wel    vast   ADV    vas      afdanken   VRB   \n",
       "8            daar zyn E myn soo    vast   ADV   vast       beloven   VRB   \n",
       "9         en dat agter ’t schip    vast   ADV   vast          zijn   VRB   \n",
       "10              […] waer op u l    vast   ADV   vast         mogen   VRB   \n",
       "11      aff scheijden & met die    vast   ADJ  vaste         hopen   VRB   \n",
       "12          hoop Ick dat het nu    vast   ADV   vast        zullen   VRB   \n",
       "13         in leyt want komt hy    vast   ADV   vast  terechtkomen   VRB   \n",
       "14     beste van Niet hebbe het    vast   ADV   vast     voornemen   VRB   \n",
       "15       zyn, dat ik in gedagte    vast   ADV   vast          zijn   VRB   \n",
       "16       want ’t zal zyn leeven    vast   ADV   vast     verlengen   VRB   \n",
       "17       is van consept ik hadt    vast   ADV   vast        denken   VRB   \n",
       "\n",
       "          word 1                            right context  \n",
       "0       beloofd,            de helfte van sijne Reekening  \n",
       "1       schryven               want daris somtys gen stat  \n",
       "2     vertrouden          dat altemael turcken waeren wij  \n",
       "3       gelooven                    dat gij mij niet half  \n",
       "4        schryve                      Dog hoop ik dat Got  \n",
       "5         Melden                    want Het is Hier Zeer  \n",
       "6         gebout                  hijer mede wens jck mij  \n",
       "7        afgedak          worden moeder doet de groetenis  \n",
       "8        belooft                heeft geen schip te laten  \n",
       "9            was          hebben wy verlooren, een groote  \n",
       "10        moocht       vertrouwen wij sijn noch altesamen  \n",
       "11        hoope,              in onse loop baan wandelen,  \n",
       "12           sal            gaen daerom versuijmt nu geen  \n",
       "13            te                     regt want de heer en  \n",
       "14  voorgenoomen                      Zoo godt wel om bij  \n",
       "15           ben  geweest, hier geleegentheyd was dezelve  \n",
       "16    verlengen,                   en buyten dien, zyn de  \n",
       "17        gedagt  dat uwed= gerippatriejeerdt soude hebbe  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce2c9d2fd0214adab6e22bc45faef390",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='vast.csv'), Button(button_style='warning', de…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Werkwoorden na <b>zeker</b>, maar niet na <b>vast</b>: voortgaan, weten, kunnen"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Werkwoorden na <b>vast</b>, maar niet na <b>zeker</b>: vertrouwen, mogen, hopen, zijn, beloven, bouwen, terechtkomen, denken, voornemen, geloven, verlengen, afdanken"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Werkwoorden zowel na <b>zeker</b> als na <b>vast</b>: schrijven, zullen, melden"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#from chaininglib import search\n",
    "from IPython.core.display import display, HTML\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.utils.dfops import column_difference\n",
    "\n",
    "corpus_name = \"zeebrieven\"\n",
    "\n",
    "# Word 1: puur\n",
    "word1= \"zeker\"\n",
    "cq1 = create_corpus(corpus_name).pattern(r'[lemma=\"' + word1 + r'\"][pos=\"VRB.*\"]')\n",
    "df_corpus1 = cq1.search().kwic()\n",
    "display_df(df_corpus1, word1)\n",
    "\n",
    "# Word 2: zuiver\n",
    "word2 = \"vast\"\n",
    "cq2 = create_corpus(corpus_name).pattern(r'[lemma=\"' + word2 + r'\"][pos=\"VRB.*\"]')\n",
    "df_corpus2 = cq2.search().kwic()\n",
    "display_df(df_corpus2, word2)\n",
    "\n",
    "# Compute difference\n",
    "diff_left, diff_right, intersec = column_difference(df_corpus1[\"lemma 1\"], df_corpus2[\"lemma 1\"])\n",
    "# Elements of 1 that are not in 2\n",
    "display(HTML('Werkwoorden na <b>' + word1 + '</b>, maar niet na <b>' + word2 + '</b>: ' + \", \".join(diff_left)))\n",
    "# Elements of 2 that are not in 1\n",
    "display(HTML('Werkwoorden na <b>' + word2 + '</b>, maar niet na <b>' + word1 + '</b>: ' + \", \".join(diff_right)))\n",
    "# Elements both in 1 and 2\n",
    "display(HTML('Werkwoorden zowel na <b>' + word1 + '</b> als na <b>' + word2 + '</b>: ' + \", \".join(intersec)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency of *leuk*+verb and *fijn*+verb compared <a class=\"anchor\" id=\"freq-leuk-fijn\"></a>\n",
    "* Same as above, but now in modern corpus, with modern words\n",
    "* This case study gathers much more data, so it may take more time to compute results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-20T09:46:43.499600Z",
     "start_time": "2019-08-20T09:38:17.603361Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[F                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    "
     ]
    },
    {
     "data": {
      "text/html": [
       "<b>leuk</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>left context</th>\n",
       "      <th>lemma 0</th>\n",
       "      <th>pos 0</th>\n",
       "      <th>word 0</th>\n",
       "      <th>lemma 1</th>\n",
       "      <th>pos 1</th>\n",
       "      <th>word 1</th>\n",
       "      <th>right context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>en twee keer voor ja</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>Leuk</td>\n",
       "      <td>horen</td>\n",
       "      <td>VRB(finiteness=fin,mood=imp|ind,tense=pres,number=sg)</td>\n",
       "      <td>hoor</td>\n",
       "      <td>Ik ken die man Welke</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>precies boven ons Dit kan</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>worden</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>worden</td>\n",
       "      <td>Ik denk eerder juist heel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Geef terug Willen jullie iets</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=postnom,case=gen,formal=infl-s)</td>\n",
       "      <td>leuks</td>\n",
       "      <td>zien</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>zien</td>\n",
       "      <td>Hij vindt t klote Gaaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>je je zal ze niet</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>vinden</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>vinden</td>\n",
       "      <td>Wanneer komt Sherry hier Vijf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>je je zal ze niet</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>vinden</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>vinden</td>\n",
       "      <td>Wanneer komt Sherry hier Vijf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9995</td>\n",
       "      <td>het rondneuzen Ze vindt je</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>weten</td>\n",
       "      <td>VRB(finiteness=fin,mood=ind,tense=past,number=sg)</td>\n",
       "      <td>wist</td>\n",
       "      <td>je dat Pardon Iona Kom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9996</td>\n",
       "      <td>mevrouw U zult het hier</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>gaan</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>gaan</td>\n",
       "      <td>vinden Ik had gedacht dat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9997</td>\n",
       "      <td>ik weet dat je dit</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>vinden</td>\n",
       "      <td>VRB(finiteness=fin,mood=ind,tense=pres,number=sg,person=2|3,formal=infl-t)</td>\n",
       "      <td>vindt</td>\n",
       "      <td>Uw briefje was zorgelijk Voelt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9998</td>\n",
       "      <td>weet dat het niet erg</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>zijn</td>\n",
       "      <td>VRB(finiteness=fin,mood=imp|ind,tense=pres,number=sg)</td>\n",
       "      <td>is</td>\n",
       "      <td>Dat weet ik wel Maar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9999</td>\n",
       "      <td>zijn Dat zou ik wel</td>\n",
       "      <td>leuk</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>leuk</td>\n",
       "      <td>vinden</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>vinden</td>\n",
       "      <td>Heb je meer pizza besteld</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       left context lemma 0  \\\n",
       "0              en twee keer voor ja    leuk   \n",
       "1         precies boven ons Dit kan    leuk   \n",
       "2     Geef terug Willen jullie iets    leuk   \n",
       "3                 je je zal ze niet    leuk   \n",
       "4                 je je zal ze niet    leuk   \n",
       "...                             ...     ...   \n",
       "9995     het rondneuzen Ze vindt je    leuk   \n",
       "9996        mevrouw U zult het hier    leuk   \n",
       "9997             ik weet dat je dit    leuk   \n",
       "9998          weet dat het niet erg    leuk   \n",
       "9999            zijn Dat zou ik wel    leuk   \n",
       "\n",
       "                                                       pos 0 word 0 lemma 1  \\\n",
       "0                           AA(degree=pos,position=adv|pred)   Leuk   horen   \n",
       "1                           AA(degree=pos,position=adv|pred)   leuk  worden   \n",
       "2     AA(degree=pos,position=postnom,case=gen,formal=infl-s)  leuks    zien   \n",
       "3                           AA(degree=pos,position=adv|pred)   leuk  vinden   \n",
       "4                           AA(degree=pos,position=adv|pred)   leuk  vinden   \n",
       "...                                                      ...    ...     ...   \n",
       "9995                        AA(degree=pos,position=adv|pred)   leuk   weten   \n",
       "9996                        AA(degree=pos,position=adv|pred)   leuk    gaan   \n",
       "9997                        AA(degree=pos,position=adv|pred)   leuk  vinden   \n",
       "9998                        AA(degree=pos,position=adv|pred)   leuk    zijn   \n",
       "9999                        AA(degree=pos,position=adv|pred)   leuk  vinden   \n",
       "\n",
       "                                                                           pos 1  \\\n",
       "0                          VRB(finiteness=fin,mood=imp|ind,tense=pres,number=sg)   \n",
       "1                                                        VRB(finiteness=ger|inf)   \n",
       "2                                                        VRB(finiteness=ger|inf)   \n",
       "3                                                        VRB(finiteness=ger|inf)   \n",
       "4                                                        VRB(finiteness=ger|inf)   \n",
       "...                                                                          ...   \n",
       "9995                           VRB(finiteness=fin,mood=ind,tense=past,number=sg)   \n",
       "9996                                                     VRB(finiteness=ger|inf)   \n",
       "9997  VRB(finiteness=fin,mood=ind,tense=pres,number=sg,person=2|3,formal=infl-t)   \n",
       "9998                       VRB(finiteness=fin,mood=imp|ind,tense=pres,number=sg)   \n",
       "9999                                                     VRB(finiteness=ger|inf)   \n",
       "\n",
       "      word 1                   right context  \n",
       "0       hoor            Ik ken die man Welke  \n",
       "1     worden       Ik denk eerder juist heel  \n",
       "2       zien          Hij vindt t klote Gaaf  \n",
       "3     vinden   Wanneer komt Sherry hier Vijf  \n",
       "4     vinden   Wanneer komt Sherry hier Vijf  \n",
       "...      ...                             ...  \n",
       "9995    wist          je dat Pardon Iona Kom  \n",
       "9996    gaan       vinden Ik had gedacht dat  \n",
       "9997   vindt  Uw briefje was zorgelijk Voelt  \n",
       "9998      is            Dat weet ik wel Maar  \n",
       "9999  vinden       Heb je meer pizza besteld  \n",
       "\n",
       "[10000 rows x 8 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b79ab7c59b04a3780c89b3dd77f8304",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='leuk.csv'), Button(button_style='warning', de…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[F                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        "
     ]
    },
    {
     "data": {
      "text/html": [
       "<b>fijn</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>left context</th>\n",
       "      <th>lemma 0</th>\n",
       "      <th>pos 0</th>\n",
       "      <th>word 0</th>\n",
       "      <th>lemma 1</th>\n",
       "      <th>pos 1</th>\n",
       "      <th>word 1</th>\n",
       "      <th>right context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>oudste zoon Ik zou het</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>vinden</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>vinden</td>\n",
       "      <td>als jij wat met me</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Ik zal even achter kijken</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>Fijn</td>\n",
       "      <td>bedanken</td>\n",
       "      <td>VRB(finiteness=part,tense=past)</td>\n",
       "      <td>bedankt</td>\n",
       "      <td>Een moment alstublieft Ik kan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Ik zal even achter kijken</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>Fijn</td>\n",
       "      <td>bedanken</td>\n",
       "      <td>VRB(finiteness=part,tense=past)</td>\n",
       "      <td>bedankt</td>\n",
       "      <td>Een moment alstublieft Ik kan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Ik zal even achter kijken</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>Fijn</td>\n",
       "      <td>bedanken</td>\n",
       "      <td>VRB(finiteness=part,tense=past)</td>\n",
       "      <td>bedankt</td>\n",
       "      <td>Een moment alstublieft Ik kan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>komen vele broeders Het zou</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>zijn</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>zijn</td>\n",
       "      <td>als je zou huilen Het</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9995</td>\n",
       "      <td>rij zelf mee Het zou</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>zijn</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>zijn</td>\n",
       "      <td>als we je naam wisten</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9996</td>\n",
       "      <td>rij zelf mee Het zou</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>zijn</td>\n",
       "      <td>VRB(finiteness=ger|inf)</td>\n",
       "      <td>zijn</td>\n",
       "      <td>als we je naam wisten</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9997</td>\n",
       "      <td>mormel Weet je wat ik</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>vinden</td>\n",
       "      <td>VRB(finiteness=fin,mood=imp|ind,tense=pres,number=sg)</td>\n",
       "      <td>vind</td>\n",
       "      <td>aan vrouwen Perfectie Lange benen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9998</td>\n",
       "      <td>zou ik hem niet zo</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>hebben</td>\n",
       "      <td>VRB(finiteness=fin,mood=ind,tense=pres,number=pl)</td>\n",
       "      <td>hebben</td>\n",
       "      <td>kunnen beledigen O ja en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9999</td>\n",
       "      <td>zou ik hem niet zo</td>\n",
       "      <td>fijn</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>fijn</td>\n",
       "      <td>hebben</td>\n",
       "      <td>VRB(finiteness=fin,mood=ind,tense=pres,number=pl)</td>\n",
       "      <td>hebben</td>\n",
       "      <td>kunnen beledigen O ja en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     left context lemma 0                             pos 0  \\\n",
       "0          oudste zoon Ik zou het    fijn  AA(degree=pos,position=adv|pred)   \n",
       "1       Ik zal even achter kijken    fijn  AA(degree=pos,position=adv|pred)   \n",
       "2       Ik zal even achter kijken    fijn  AA(degree=pos,position=adv|pred)   \n",
       "3       Ik zal even achter kijken    fijn  AA(degree=pos,position=adv|pred)   \n",
       "4     komen vele broeders Het zou    fijn  AA(degree=pos,position=adv|pred)   \n",
       "...                           ...     ...                               ...   \n",
       "9995         rij zelf mee Het zou    fijn  AA(degree=pos,position=adv|pred)   \n",
       "9996         rij zelf mee Het zou    fijn  AA(degree=pos,position=adv|pred)   \n",
       "9997        mormel Weet je wat ik    fijn  AA(degree=pos,position=adv|pred)   \n",
       "9998           zou ik hem niet zo    fijn  AA(degree=pos,position=adv|pred)   \n",
       "9999           zou ik hem niet zo    fijn  AA(degree=pos,position=adv|pred)   \n",
       "\n",
       "     word 0   lemma 1                                                  pos 1  \\\n",
       "0      fijn    vinden                                VRB(finiteness=ger|inf)   \n",
       "1      Fijn  bedanken                        VRB(finiteness=part,tense=past)   \n",
       "2      Fijn  bedanken                        VRB(finiteness=part,tense=past)   \n",
       "3      Fijn  bedanken                        VRB(finiteness=part,tense=past)   \n",
       "4      fijn      zijn                                VRB(finiteness=ger|inf)   \n",
       "...     ...       ...                                                    ...   \n",
       "9995   fijn      zijn                                VRB(finiteness=ger|inf)   \n",
       "9996   fijn      zijn                                VRB(finiteness=ger|inf)   \n",
       "9997   fijn    vinden  VRB(finiteness=fin,mood=imp|ind,tense=pres,number=sg)   \n",
       "9998   fijn    hebben      VRB(finiteness=fin,mood=ind,tense=pres,number=pl)   \n",
       "9999   fijn    hebben      VRB(finiteness=fin,mood=ind,tense=pres,number=pl)   \n",
       "\n",
       "       word 1                      right context  \n",
       "0      vinden                 als jij wat met me  \n",
       "1     bedankt      Een moment alstublieft Ik kan  \n",
       "2     bedankt      Een moment alstublieft Ik kan  \n",
       "3     bedankt      Een moment alstublieft Ik kan  \n",
       "4        zijn              als je zou huilen Het  \n",
       "...       ...                                ...  \n",
       "9995     zijn              als we je naam wisten  \n",
       "9996     zijn              als we je naam wisten  \n",
       "9997     vind  aan vrouwen Perfectie Lange benen  \n",
       "9998   hebben           kunnen beledigen O ja en  \n",
       "9999   hebben           kunnen beledigen O ja en  \n",
       "\n",
       "[10000 rows x 8 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a77a714708246c0b5028ba1b8f537e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='fijn.csv'), Button(button_style='warning', de…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Werkwoorden na <b>leuk</b> niet in <b>fijn</b>: besteden, oppeppen, deelnemen, woreen, lannen, inrichten, zappen, vermaken, acteren, bowlen, noemen, motiveren, sparen, verdienen, optreden, apporteren, inkopen, schijnen, lachen, bluffen, veronderstellen, opschieten, verkleden, voorkomen, klapperen, opmaken, gewinnen, iïmproviseren, passen, azijn, probeerten, scoren, combineren, gebruiken, ontmoeten, bedenken, opvangen, opheffen, toeren, vragen, ondernemen, aantrekken, herschrijven, hoeven, veroorzaken, verzinnen, improviseren, opsteken, schelden, horrorscrippen, bedoelen, kleden, spartelen, aankleden, uitkiezen, patiencen, verpakken, alsjeen, parkeren, uitkienen"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Werkwoorden na <b>fijn</b> niet in <b>leuk</b>: vertoeven, halen, weven, obsederen, televisiekijken, kenmerken, vallen, aanmoedigen, vreten, behandelen, schilderen, knikken, Mamen, huilen, opnemen, beschutten, rennen, helpen, tennissen, polijsten, vrijzellenfeesten, vertakken, bewerken, wandelen, brunchen, baden, draaien, klussen, aanraken, afronden, samenvatten, wroeten, begroeten, zwemmen, lichten, slaan, versnijden, studeren, waarderen, beschermen, vreemdgaan, voorverwarmen, klaarkomen, beroven, drukken, verloven, kalmeren, pakken, kiezen, aanvoelen, pletten, verdelen, trainen, analyseren, aaien, genieten, persen, ciseleren, vissen, feestvieren, gevoelen, vrijen, zwijnen, ontzien, lunchen, verhalen, strelen, meebewegen, aflopen, kokkerellen, sluiten, uitgiden, vinben, prakken, opeten, ophoepelen, kamperen, doorbabbelen, voegen, slingeren, uitdrukken, knijpen, meewerken, besparen, opmerken, missen, samenwerken, vliegen, slapen, oefenen, begluren, lezen, gluren, picknicken, uitbalanceren, wegwezen, schieten, melden, ruiken, wijzen, masseren, voorzien, gedogen, vermoorden, stoppen, misten, verzorgen, bellen, uitleggen, afwassen, tanken, doorzoeken, paen, datu, oomzwinden, halloween, dineren, uitjouwen, optutten, verkopen, ziften, barbecuen, spinnen, zingen, arresteren, krijgen, rusten, shoppen, dankjiden, jagen, gooien, wezen, achterblijven, verwennen, knippen, afstemmen, amuseren, schatten, tegenwerken, ontspannen, hakken, timen, liggen, voorjiden, zetten, snijden, trekken, wiegen, wachten, belazeren, bekijken, lopen, verzieken, reizen, dromen, opsluiten, worstelen, schrijven, wegslepen, fluiten, stampen, volgen, afstellen, toekijken, installeren"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Werkwoorden zowel voor <b>leuk</b> als voor <b>fijn</b>: aanpakken, denken, gaan, betalen, kletsen, zijn, danken, zeggen, winkelen, hebben, vinden, uitzien, snappen, vertellen, laten, leren, vergeten, lijken, zien, geven, werken, spreken, opknappen, meedoen, beginnen, weten, eten, maken, thuiskomen, leven, trouwen, doen, uitgaan, proberen, wonen, regelen, horen, brengen, komen, klinken, zitten, nemen, worden, stellen, babbelen, rijden, blijven, mogen, houden, praten, kijken, dansen, zullen, spelen, geloven, willen, moeten, begrijpen, kennismaken, bedanken, staan, voelen, verpesten, kunnen, zakendoen, eindigen"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#from chaininglib import search\n",
    "from IPython.core.display import display, HTML\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.utils.dfops import column_difference\n",
    "\n",
    "corpus_name = \"opus\"\n",
    "# Word 1: puur\n",
    "word1= \"leuk\"\n",
    "cq1 = create_corpus(corpus_name).max_results(10000).pattern(r'[lemma=\"' + word1 + r'\"][pos=\"VRB.*\"] within <s/>')\n",
    "df_corpus1 = cq1.search().kwic()\n",
    "display_df(df_corpus1, word1)\n",
    "# Word 2: zuiver\n",
    "word2 = \"fijn\"\n",
    "cq2 = create_corpus(corpus_name).max_results(10000).pattern(r'[lemma=\"' + word2 + r'\"][pos=\"VRB.*\"] within <s/>')\n",
    "df_corpus2 = cq2.search().kwic()\n",
    "display_df(df_corpus2, word2)\n",
    "# Compute difference\n",
    "diff_left, diff_right, intersec = column_difference(df_corpus1[\"lemma 1\"], df_corpus2[\"lemma 1\"])\n",
    "# Elements of 1 that are not in 2\n",
    "display(HTML('Werkwoorden na <b>' + word1 + '</b> niet in <b>' + word2 + '</b>: ' + \", \".join(diff_left)))\n",
    "# Elements of 2 that are not in 1\n",
    "display(HTML('Werkwoorden na <b>' + word2 + '</b> niet in <b>' + word1 + '</b>: ' + \", \".join(diff_right)))\n",
    "# Elements both in 1 and 2\n",
    "display(HTML('Werkwoorden zowel voor <b>' + word1 + '</b> als voor <b>' + word2 + '</b>: ' + \", \".join(intersec)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a POS tagger on an annotated historical corpus and tag a historical sentence<a class=\"anchor\" id=\"pos-tagger\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T10:31:40.134049Z",
     "start_time": "2019-03-21T10:31:38.212645Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.process.corpus import get_tagger\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.search.LexiconQuery import *\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "# gather some pattern including our word, out of annotated corpora\n",
    "\n",
    "dfs_all_corpora = pd.DataFrame()\n",
    "some_lemma = \"lopen\"\n",
    "\n",
    "for one_corpus in ['zeebrieven', 'gysseling']:\n",
    "    print('querying '+one_corpus+'...')\n",
    "    c = create_corpus(one_corpus).lemma(some_lemma).detailed_context(True).search()\n",
    "    df_corpus = c.kwic() \n",
    "    \n",
    "    # store the results\n",
    "    dfs_all_corpora = pd.concat( [dfs_all_corpora, df_corpus] )\n",
    "\n",
    "\n",
    "# get a tagger trained with our corpus data\n",
    "tagger = get_tagger(dfs_all_corpora, pos_key = 'pos') \n",
    "\n",
    "# Use the trained tagger to tag unknown sentences\n",
    "# The input must be like: tagger.tag(['today','is','a','beautiful','day'])\n",
    "\n",
    "sentence = 'De menschen lopen wel haesteliken op enen bome ghelike een ape'\n",
    "tagged_sentence = tagger.tag( sentence.split() )\n",
    "\n",
    "print(tagged_sentence)\n",
    "\n",
    "\n",
    "# Know we can lemmatize each occurence of our lemma in the new sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search in corpus and filter on metadata <a class=\"anchor\" id=\"corpus-filter-metadata\"></a>\n",
    "First, we request all available metadata fields of the corpus. Then, we issue a search query, and request all metadata fields for the result. Finally, we filter on metadata values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T10:50:27.492793Z",
     "start_time": "2019-03-21T10:50:26.879452Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.search.metadata import get_available_metadata\n",
    "from chaininglib.utils.dfops import df_filter, property_freq\n",
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "\n",
    "\n",
    "corpus_name=\"zeebrieven\"\n",
    "query=r'[lemma=\"boek\"]'\n",
    "# Request all metadata fields from corpus\n",
    "fields = get_available_metadata(corpus_name)\n",
    "# Perform query and ask all metadata\n",
    "c = create_corpus(corpus_name).pattern(query).extra_fields_doc(fields[\"document\"]).search()\n",
    "df_corpus = c.kwic()\n",
    "\n",
    "# Filter on year: > 1700\n",
    "df_filter_year = df_corpus[df_corpus[\"witnessYear_from\"].astype('int32') > 1700] \n",
    "display_df(df_filter_year, labels=\"After 1700\")\n",
    "\n",
    "# Filter on sender birth place Amsterdam\n",
    "condition = df_filter(df_corpus[\"afz_geb_plaats\"], pattern=\"Amsterdam\")\n",
    "df_filter_place = df_corpus[ condition ]\n",
    "display_df(df_filter_place, labels=\"Sender born in Amsterdam\")\n",
    "\n",
    "\n",
    "# Group by birth place\n",
    "df = property_freq(df_corpus,\"afz_loc_plaats\")\n",
    "display_df(df, labels=\"Most frequent sender locations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing h-dropping  <a class=\"anchor\" id=\"visualizing-h-dropping\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-20T09:47:19.106412Z",
     "start_time": "2019-08-20T09:46:43.501082Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get both normal and h-dropping language data\n",
      "\u001b[F                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    "
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    245\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;34m\"witnessYear_from\"\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 246\u001b[1;33m                 \u001b[0mdf\u001b[0m\u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[1;34m\"witnessYear_from\"\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;34m\"int32\"\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    247\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;34m\"witnessYear_to\"\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\generic.py\u001b[0m in \u001b[0;36mastype\u001b[1;34m(self, dtype, copy, errors, **kwargs)\u001b[0m\n\u001b[0;32m   5864\u001b[0m                         col.astype(\n\u001b[1;32m-> 5865\u001b[1;33m                             \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mcol_name\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   5866\u001b[0m                         )\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\generic.py\u001b[0m in \u001b[0;36mastype\u001b[1;34m(self, dtype, copy, errors, **kwargs)\u001b[0m\n\u001b[0;32m   5881\u001b[0m             new_data = self._data.astype(\n\u001b[1;32m-> 5882\u001b[1;33m                 \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   5883\u001b[0m             )\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\internals\\managers.py\u001b[0m in \u001b[0;36mastype\u001b[1;34m(self, dtype, **kwargs)\u001b[0m\n\u001b[0;32m    580\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mastype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 581\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mapply\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"astype\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    582\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\internals\\managers.py\u001b[0m in \u001b[0;36mapply\u001b[1;34m(self, f, axes, filter, do_integrity_check, consolidate, **kwargs)\u001b[0m\n\u001b[0;32m    437\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 438\u001b[1;33m             \u001b[0mapplied\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    439\u001b[0m             \u001b[0mresult_blocks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_extend_blocks\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mapplied\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresult_blocks\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\internals\\blocks.py\u001b[0m in \u001b[0;36mastype\u001b[1;34m(self, dtype, copy, errors, values, **kwargs)\u001b[0m\n\u001b[0;32m    558\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mastype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"raise\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalues\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 559\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_astype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalues\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    560\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\internals\\blocks.py\u001b[0m in \u001b[0;36m_astype\u001b[1;34m(self, dtype, copy, errors, values, **kwargs)\u001b[0m\n\u001b[0;32m    642\u001b[0m                     \u001b[0mvals1d\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mvalues\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 643\u001b[1;33m                     \u001b[0mvalues\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mastype_nansafe\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvals1d\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    644\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\peter\\chaining-search-master\\env\\lib\\site-packages\\pandas\\core\\dtypes\\cast.py\u001b[0m in \u001b[0;36mastype_nansafe\u001b[1;34m(arr, dtype, copy, skipna)\u001b[0m\n\u001b[0;32m    706\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0missubdtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minteger\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 707\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mlib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mastype_intsafe\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    708\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mpandas\\_libs\\lib.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.astype_intsafe\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    227\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 228\u001b[1;33m                 \u001b[0mdf_more\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    229\u001b[0m                 \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_more\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-6-a8813802d25b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Get both normal and h-dropping language data'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0mdf_corpus1\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcreate_corpus\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcorpus_to_search\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34mr'[lemma=\"h[aeo].*\" & word=\"h[aeo].*\"]'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextra_fields_doc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfields\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"document\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     14\u001b[0m \u001b[0mdf1_filtered\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf_corpus1\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mdf_corpus1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mapply\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mgroup_by_column\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[0mdf_corpus2\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcreate_corpus\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcorpus_to_search\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34mr'[lemma=\"h[aeo].*\" & word=\"[aeo].*\"]'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextra_fields_doc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfields\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"document\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkwic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\chaining-search-master\\chaininglib\\search\\CorpusQuery.py\u001b[0m in \u001b[0;36msearch\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    257\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    258\u001b[0m             \u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremove_wait_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 259\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"An error occured when searching corpus \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_resource\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\": \"\u001b[0m\u001b[1;33m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    260\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    261\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: An error occured when searching corpus zeebrieven: invalid literal for int() with base 10: '\\n        '"
     ]
    }
   ],
   "source": [
    "\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.search.metadata import get_available_metadata\n",
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.utils.dfops import join_df\n",
    " \n",
    "corpus_to_search=\"zeebrieven\"\n",
    "group_by_column = 'afz_geb_plaats'\n",
    "\n",
    "fields = get_available_metadata(corpus_to_search)\n",
    "\n",
    "print('Get both normal and h-dropping language data')\n",
    "\n",
    "df_corpus1 = create_corpus(corpus_to_search).pattern(r'[lemma=\"h[aeo].*\" & word=\"h[aeo].*\"]').extra_fields_doc(fields[\"document\"]).search().kwic()\n",
    "df1_filtered = df_corpus1[df_corpus1.apply(lambda x: len(x[group_by_column]) > 0, axis=1)]\n",
    "df_corpus2 = create_corpus(corpus_to_search).pattern(r'[lemma=\"h[aeo].*\" & word=\"[aeo].*\"]').extra_fields_doc(fields[\"document\"]).search().kwic()\n",
    "df2_filtered = df_corpus2[df_corpus2.apply(lambda x: len(x[group_by_column]) > 0, axis=1)]\n",
    "\n",
    "print('Get neutral language data')\n",
    "\n",
    "df_corpus1and2 = create_corpus(corpus_to_search).pattern(r'[lemma=\"h[aeo].*\"]').extra_fields_doc(fields[\"document\"]).search().kwic()\n",
    "df_corpus1and2.rename(columns={'lemma 0':'total'}, inplace=True)\n",
    "df1and2_filtered = df_corpus1and2[df_corpus1and2.apply(lambda x: len(x[group_by_column]) > 0, axis=1)]\n",
    "\n",
    "print('Counting...')\n",
    "\n",
    "group1 = df1_filtered[['lemma 0', group_by_column]].groupby(group_by_column).count()\n",
    "group2 = df2_filtered[['lemma 0', group_by_column]].groupby(group_by_column).count()\n",
    "group1and2 = df1and2_filtered[['total', group_by_column]].groupby(group_by_column).count() \n",
    "\n",
    "# compute relative counts\n",
    "df1 = join_df( [ group1, group1and2['total'] ] )\n",
    "df2 = join_df( [ group2, group1and2['total'] ] )\n",
    "df1['lemma 0'] = df1['lemma 0'] / df1['total']\n",
    "df2['lemma 0'] = df2['lemma 0'] / df2['total']\n",
    "\n",
    "# clean up data\n",
    "df1 = df1.dropna()\n",
    "df2 = df2.dropna()\n",
    "df1 = df1.drop(['total'], axis=1)\n",
    "df2 = df2.drop(['total'], axis=1)\n",
    "\n",
    "print('Draw charts showing geographic differences between normal language and h-dropping')\n",
    "\n",
    "display_df( df1.sort_values(ascending=False,by=['lemma 0']).head(25), labels=\"normal\", mode='chart') \n",
    "display_df( df2.sort_values(ascending=False,by=['lemma 0']).head(25), labels=\"h-dropping\", mode='chart')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a lexicon from a corpus <a class=\"anchor\" id=\"lexicon-from-corpus\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T13:25:41.896444Z",
     "start_time": "2019-03-22T13:24:44.095470Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.process.corpus import extract_lexicon\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "\n",
    "corpus_name = \"zeebrieven\"\n",
    "\n",
    "print('Querying '+corpus_name+'...')\n",
    "c = create_corpus(corpus_name).pos(\"NOU\").detailed_context(True).search()\n",
    "df_corpus = c.kwic() \n",
    "    \n",
    "# extract lexicon and show the result\n",
    "extracted_lexicon = extract_lexicon(df_corpus, posColumnName=\"pos\")\n",
    "display(extracted_lexicon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lexicon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lexicon search <a class=\"anchor\" id=\"lexicon-search\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Run the cell below to show the UI, and fill in your search query in the UI\n",
    "* Choose one of the lexica:\n",
    "  * **anw**: Algemeen Nederlands Woordenboek, dictionary of contemporary Dutch\n",
    "  * **celex**: Lexical database of the Dutch language\n",
    "  * **duelme**\n",
    "  * **molex**: A lexicon of modern Dutch, containing spelling variants per lemma\n",
    "  * **mnwlex**: Lexicon service access to Middelnederlands Woordenboek, a lexicon of Middle Dutch (ca. 1250 - 1550)\n",
    "  * **lexicon_service_db**: Lexicon service access to a lexicon of early modern and modern Dutch (1500 - 1976)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-21T14:46:31.732208Z",
     "start_time": "2019-08-21T14:46:31.710217Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8cb2220d89fd4340b0b2796f292a1373",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='boek', description='<b>Word:</b>'), Dropdown(description='<b>Lexicon:</b>', options…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from chaininglib.ui.search import create_lexicon_ui\n",
    "\n",
    "#from chaininglib import ui\n",
    "searchWordField, lexiconField = create_lexicon_ui()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * Click the cell below and press Run to perform the given query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-21T09:41:03.505158Z",
     "start_time": "2019-08-21T09:41:03.093427Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...Searching molex...\r",
      "\u001b[F...Querying molex at offset 0...\r",
      "\u001b[F                                                                    \r",
      "\u001b[F                                                                    \r",
      "\u001b[F"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemEntryId</th>\n",
       "      <th>lemma</th>\n",
       "      <th>lemPos</th>\n",
       "      <th>wordformId</th>\n",
       "      <th>wordform</th>\n",
       "      <th>hyphenation</th>\n",
       "      <th>wordformPos</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Number</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/10919</td>\n",
       "      <td>boek</td>\n",
       "      <td>http://universaldependencies.org/u/pos/NOUN</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/179523</td>\n",
       "      <td>boek</td>\n",
       "      <td>boek</td>\n",
       "      <td>http://universaldependencies.org/u/pos/NOUN</td>\n",
       "      <td>http://universaldependencies.org/u/feat/Gender.html#Neut</td>\n",
       "      <td>http://universaldependencies.org/u/feat/Number.html#Sing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/10919</td>\n",
       "      <td>boek</td>\n",
       "      <td>http://universaldependencies.org/u/pos/NOUN</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/39435</td>\n",
       "      <td>boeken</td>\n",
       "      <td>boe|ken</td>\n",
       "      <td>http://universaldependencies.org/u/pos/NOUN</td>\n",
       "      <td>http://universaldependencies.org/u/feat/Gender.html#Neut</td>\n",
       "      <td>http://universaldependencies.org/u/feat/Number.html#Plur</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/104054</td>\n",
       "      <td>boeken</td>\n",
       "      <td>http://universaldependencies.org/u/pos/VERB</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/365579</td>\n",
       "      <td>boek</td>\n",
       "      <td>boek</td>\n",
       "      <td>http://universaldependencies.org/u/pos/VERB</td>\n",
       "      <td></td>\n",
       "      <td>http://universaldependencies.org/u/feat/Number.html#Sing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/104054</td>\n",
       "      <td>boeken</td>\n",
       "      <td>http://universaldependencies.org/u/pos/VERB</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/803734</td>\n",
       "      <td>boek</td>\n",
       "      <td>boek</td>\n",
       "      <td>http://universaldependencies.org/u/pos/VERB</td>\n",
       "      <td></td>\n",
       "      <td>http://universaldependencies.org/u/feat/Number.html#Sing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               lemEntryId   lemma  \\\n",
       "0   http://rdf.ivdnt.org/lexica/diamant/entry/molex/10919    boek   \n",
       "1   http://rdf.ivdnt.org/lexica/diamant/entry/molex/10919    boek   \n",
       "2  http://rdf.ivdnt.org/lexica/diamant/entry/molex/104054  boeken   \n",
       "3  http://rdf.ivdnt.org/lexica/diamant/entry/molex/104054  boeken   \n",
       "\n",
       "                                        lemPos  \\\n",
       "0  http://universaldependencies.org/u/pos/NOUN   \n",
       "1  http://universaldependencies.org/u/pos/NOUN   \n",
       "2  http://universaldependencies.org/u/pos/VERB   \n",
       "3  http://universaldependencies.org/u/pos/VERB   \n",
       "\n",
       "                                                  wordformId wordform  \\\n",
       "0  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/179523     boek   \n",
       "1   http://rdf.ivdnt.org/lexica/diamant/wordform/molex/39435   boeken   \n",
       "2  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/365579     boek   \n",
       "3  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/803734     boek   \n",
       "\n",
       "  hyphenation                                  wordformPos  \\\n",
       "0        boek  http://universaldependencies.org/u/pos/NOUN   \n",
       "1     boe|ken  http://universaldependencies.org/u/pos/NOUN   \n",
       "2        boek  http://universaldependencies.org/u/pos/VERB   \n",
       "3        boek  http://universaldependencies.org/u/pos/VERB   \n",
       "\n",
       "                                                     Gender  \\\n",
       "0  http://universaldependencies.org/u/feat/Gender.html#Neut   \n",
       "1  http://universaldependencies.org/u/feat/Gender.html#Neut   \n",
       "2                                                             \n",
       "3                                                             \n",
       "\n",
       "                                                     Number  \n",
       "0  http://universaldependencies.org/u/feat/Number.html#Sing  \n",
       "1  http://universaldependencies.org/u/feat/Number.html#Plur  \n",
       "2  http://universaldependencies.org/u/feat/Number.html#Sing  \n",
       "3  http://universaldependencies.org/u/feat/Number.html#Sing  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "281e8116a61f44c78717ca365dc56fc0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='mijn_resultaten.csv'), Button(button_style='w…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from chaininglib.search.LexiconQuery import *\n",
    "from chaininglib.ui.dfui import display_df\n",
    "\n",
    "search_word = searchWordField.value\n",
    "lexicon_name = lexiconField.value\n",
    "# USER: can replace this by own custom query\n",
    "lex = create_lexicon(lexicon_name).lemma(search_word).search()\n",
    "df_lexicon = lex.kwic()\n",
    "display_df(df_lexicon)\n",
    "#df_columns_list = list(df_lexicon.columns.values)\n",
    "#df_lexicon_in_columns = df_lexicon[df_columns_list]\n",
    "#display(df_lexicon_in_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpus + lexicon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Build a frequency list of the lemma of some corpus output <a class=\"anchor\" id=\"freq-lemma-corpus\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T13:43:07.164023Z",
     "start_time": "2019-03-22T13:28:45.719971Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.process.corpus import *\n",
    "from chaininglib.ui.dfui import *\n",
    "\n",
    "# max number of records to retrieve (speed!)\n",
    "max_nr_of_records = 50000\n",
    "\n",
    "# do some corpus search\n",
    "print('This can take a while... please wait!')\n",
    "\n",
    "corpus_to_search=\"openchn\"\n",
    "df_corpus = create_corpus(corpus_to_search).detailed_context(True).pos(\"NOU.*\").max_results(max_nr_of_records).search().kwic()\n",
    "\n",
    "# compute and display a table of the frequencies of the lemmata\n",
    "freq_df = get_frequency_list(df_corpus)\n",
    "display_df(freq_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-16T15:46:20.519833Z",
     "start_time": "2019-01-16T15:46:20.516208Z"
    }
   },
   "source": [
    "### Find occurences of attributive adjectives not ending with -e, even though they are preceeded by a definite article <a class=\"anchor\" id=\"adjective-e\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T11:15:32.231218Z",
     "start_time": "2019-03-21T11:14:56.094365Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get occurences of attributive adjectives not ending with -e\n",
      "\u001b[F                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        "
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>left context</th>\n",
       "      <th>lemma 0</th>\n",
       "      <th>pos 0</th>\n",
       "      <th>word 0</th>\n",
       "      <th>lemma 1</th>\n",
       "      <th>pos 1</th>\n",
       "      <th>word 1</th>\n",
       "      <th>lemma 2</th>\n",
       "      <th>pos 2</th>\n",
       "      <th>word 2</th>\n",
       "      <th>right context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>van Duijneveldt met The Unborn</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>het</td>\n",
       "      <td>gouden</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>gouden</td>\n",
       "      <td>ei</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>ei</td>\n",
       "      <td>symbool van hoop en belofte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>s get wild edition heet</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=pers,person=3,gender=n,number=sg,position=pron)</td>\n",
       "      <td>het</td>\n",
       "      <td>gratis</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>gratis</td>\n",
       "      <td>flip</td>\n",
       "      <td>NOU-P(part-of-multiword=true)</td>\n",
       "      <td>Flip</td>\n",
       "      <td>Flop-muziekevenement dat op zondag 9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>lekker snoepen van alles wat</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>goedheilig</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>goedheilig</td>\n",
       "      <td>man</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>man</td>\n",
       "      <td>uitdeelt Oh ja en dan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Freeport een geldbedrag uitlooft voor</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gouden</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>gouden</td>\n",
       "      <td>tip</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>tip</td>\n",
       "      <td>zo vertelt eigenaar Mahesh Mukti</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>plaatste vervolgens een foto van</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>het</td>\n",
       "      <td>vinden</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gevonden</td>\n",
       "      <td>toestel</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>toestel</td>\n",
       "      <td>op Twitter in de hoop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9553</td>\n",
       "      <td>onderzoekers een kijkje nemen bij</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gezonken</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gezonken</td>\n",
       "      <td>titanic</td>\n",
       "      <td>NOU-P(number=sg)</td>\n",
       "      <td>Titanic</td>\n",
       "      <td>Tijdens de twintig dagen durende</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9554</td>\n",
       "      <td>het uitzoeken en registreren van</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gegeven</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gegeven</td>\n",
       "      <td>collectie</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>collectie</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9555</td>\n",
       "      <td>van een kampleider en tegen</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gedwongen</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gedwongen</td>\n",
       "      <td>deportatie</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>deportatie</td>\n",
       "      <td>naar LaosErkend vluchtelingDe Amerikaanse regering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9556</td>\n",
       "      <td>en verrassende voorbeelden zoals van</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gouden</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>gouden</td>\n",
       "      <td>koets</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>koets</td>\n",
       "      <td>en een haast honderdjarige boom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9557</td>\n",
       "      <td>doorgemaakt en zitten nu met</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gebakken</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gebakken</td>\n",
       "      <td>peer</td>\n",
       "      <td>NOU-C(number=pl)</td>\n",
       "      <td>peren</td>\n",
       "      <td>De ex-voorzitter is er twintig</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9558 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                               left context lemma 0  \\\n",
       "0            van Duijneveldt met The Unborn     het   \n",
       "1                   s get wild edition heet     het   \n",
       "2              lekker snoepen van alles wat      de   \n",
       "3     Freeport een geldbedrag uitlooft voor      de   \n",
       "4          plaatste vervolgens een foto van     het   \n",
       "...                                     ...     ...   \n",
       "9553      onderzoekers een kijkje nemen bij      de   \n",
       "9554       het uitzoeken en registreren van      de   \n",
       "9555            van een kampleider en tegen      de   \n",
       "9556   en verrassende voorbeelden zoals van      de   \n",
       "9557           doorgemaakt en zitten nu met      de   \n",
       "\n",
       "                                                        pos 0 word 0  \\\n",
       "0                                PD(type=d-p,subtype=art-def)    het   \n",
       "1     PD(type=pers,person=3,gender=n,number=sg,position=pron)    het   \n",
       "2                                PD(type=d-p,subtype=art-def)     de   \n",
       "3                                PD(type=d-p,subtype=art-def)     de   \n",
       "4                                PD(type=d-p,subtype=art-def)    het   \n",
       "...                                                       ...    ...   \n",
       "9553                             PD(type=d-p,subtype=art-def)     de   \n",
       "9554                             PD(type=d-p,subtype=art-def)     de   \n",
       "9555                             PD(type=d-p,subtype=art-def)     de   \n",
       "9556                             PD(type=d-p,subtype=art-def)     de   \n",
       "9557                             PD(type=d-p,subtype=art-def)     de   \n",
       "\n",
       "         lemma 1                             pos 1      word 1     lemma 2  \\\n",
       "0         gouden    AA(degree=pos,position=prenom)      gouden          ei   \n",
       "1         gratis  AA(degree=pos,position=adv|pred)      gratis        flip   \n",
       "2     goedheilig    AA(degree=pos,position=prenom)  goedheilig         man   \n",
       "3         gouden    AA(degree=pos,position=prenom)      gouden         tip   \n",
       "4         vinden    AA(position=prenom,degree=pos)    gevonden     toestel   \n",
       "...          ...                               ...         ...         ...   \n",
       "9553    gezonken    AA(position=prenom,degree=pos)    gezonken     titanic   \n",
       "9554     gegeven    AA(position=prenom,degree=pos)     gegeven   collectie   \n",
       "9555   gedwongen    AA(position=prenom,degree=pos)   gedwongen  deportatie   \n",
       "9556      gouden    AA(degree=pos,position=prenom)      gouden       koets   \n",
       "9557    gebakken    AA(position=prenom,degree=pos)    gebakken        peer   \n",
       "\n",
       "                              pos 2      word 2  \\\n",
       "0         NOU-C(gender=n,number=sg)          ei   \n",
       "1     NOU-P(part-of-multiword=true)        Flip   \n",
       "2       NOU-C(gender=f|m,number=sg)         man   \n",
       "3       NOU-C(gender=f|m,number=sg)         tip   \n",
       "4         NOU-C(gender=n,number=sg)     toestel   \n",
       "...                             ...         ...   \n",
       "9553               NOU-P(number=sg)     Titanic   \n",
       "9554    NOU-C(gender=f|m,number=sg)   collectie   \n",
       "9555    NOU-C(gender=f|m,number=sg)  deportatie   \n",
       "9556    NOU-C(gender=f|m,number=sg)       koets   \n",
       "9557               NOU-C(number=pl)       peren   \n",
       "\n",
       "                                           right context  \n",
       "0                            symbool van hoop en belofte  \n",
       "1                   Flop-muziekevenement dat op zondag 9  \n",
       "2                                  uitdeelt Oh ja en dan  \n",
       "3                       zo vertelt eigenaar Mahesh Mukti  \n",
       "4                                  op Twitter in de hoop  \n",
       "...                                                  ...  \n",
       "9553                    Tijdens de twintig dagen durende  \n",
       "9554                                                      \n",
       "9555  naar LaosErkend vluchtelingDe Amerikaanse regering  \n",
       "9556                     en een haast honderdjarige boom  \n",
       "9557                      De ex-voorzitter is er twintig  \n",
       "\n",
       "[9558 rows x 11 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[F...Querying molex at offset 0...                                                                                                                                                                                                                                                "
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemEntryId</th>\n",
       "      <th>lemma</th>\n",
       "      <th>lemPos</th>\n",
       "      <th>wordformId</th>\n",
       "      <th>wordform</th>\n",
       "      <th>hyphenation</th>\n",
       "      <th>wordformPos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/27460</td>\n",
       "      <td>gewafeld</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/102144</td>\n",
       "      <td>gewafelde</td>\n",
       "      <td>ge|wa|fel|de</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/28905</td>\n",
       "      <td>granaten</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/102404</td>\n",
       "      <td>granaten</td>\n",
       "      <td>gra|na|ten</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/26787</td>\n",
       "      <td>gereglementeerd</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/102662</td>\n",
       "      <td>gereglementeerde</td>\n",
       "      <td>ge|re|gle|men|teer|de</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/28966</td>\n",
       "      <td>grasrijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/104989</td>\n",
       "      <td>grasrijke</td>\n",
       "      <td>gras|rij|ke</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/25753</td>\n",
       "      <td>geil</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/105604</td>\n",
       "      <td>geilst</td>\n",
       "      <td>geilst</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7942</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376</td>\n",
       "      <td>grondrechtelijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870801</td>\n",
       "      <td>grondrechtelijks</td>\n",
       "      <td>grond|rech|te|lijks</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7943</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376</td>\n",
       "      <td>grondrechtelijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870802</td>\n",
       "      <td>grondrechtelijke</td>\n",
       "      <td>grond|rech|te|lij|ke</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7944</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376</td>\n",
       "      <td>grondrechtelijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870803</td>\n",
       "      <td>grondrechtelijk</td>\n",
       "      <td>grond|rech|te|lijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7945</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376</td>\n",
       "      <td>grondrechtelijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870804</td>\n",
       "      <td>grondrechtelijkste</td>\n",
       "      <td>grond|rech|te|lijk|ste</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7946</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376</td>\n",
       "      <td>grondrechtelijk</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "      <td>http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870879</td>\n",
       "      <td>grondrechtelijkst</td>\n",
       "      <td>grond|rech|te|lijkst</td>\n",
       "      <td>http://universaldependencies.org/u/pos/ADJ</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7947 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  lemEntryId            lemma  \\\n",
       "0      http://rdf.ivdnt.org/lexica/diamant/entry/molex/27460         gewafeld   \n",
       "1      http://rdf.ivdnt.org/lexica/diamant/entry/molex/28905         granaten   \n",
       "2      http://rdf.ivdnt.org/lexica/diamant/entry/molex/26787  gereglementeerd   \n",
       "3      http://rdf.ivdnt.org/lexica/diamant/entry/molex/28966         grasrijk   \n",
       "4      http://rdf.ivdnt.org/lexica/diamant/entry/molex/25753             geil   \n",
       "...                                                      ...              ...   \n",
       "7942  http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376  grondrechtelijk   \n",
       "7943  http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376  grondrechtelijk   \n",
       "7944  http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376  grondrechtelijk   \n",
       "7945  http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376  grondrechtelijk   \n",
       "7946  http://rdf.ivdnt.org/lexica/diamant/entry/molex/846376  grondrechtelijk   \n",
       "\n",
       "                                          lemPos  \\\n",
       "0     http://universaldependencies.org/u/pos/ADJ   \n",
       "1     http://universaldependencies.org/u/pos/ADJ   \n",
       "2     http://universaldependencies.org/u/pos/ADJ   \n",
       "3     http://universaldependencies.org/u/pos/ADJ   \n",
       "4     http://universaldependencies.org/u/pos/ADJ   \n",
       "...                                          ...   \n",
       "7942  http://universaldependencies.org/u/pos/ADJ   \n",
       "7943  http://universaldependencies.org/u/pos/ADJ   \n",
       "7944  http://universaldependencies.org/u/pos/ADJ   \n",
       "7945  http://universaldependencies.org/u/pos/ADJ   \n",
       "7946  http://universaldependencies.org/u/pos/ADJ   \n",
       "\n",
       "                                                     wordformId  \\\n",
       "0     http://rdf.ivdnt.org/lexica/diamant/wordform/molex/102144   \n",
       "1     http://rdf.ivdnt.org/lexica/diamant/wordform/molex/102404   \n",
       "2     http://rdf.ivdnt.org/lexica/diamant/wordform/molex/102662   \n",
       "3     http://rdf.ivdnt.org/lexica/diamant/wordform/molex/104989   \n",
       "4     http://rdf.ivdnt.org/lexica/diamant/wordform/molex/105604   \n",
       "...                                                         ...   \n",
       "7942  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870801   \n",
       "7943  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870802   \n",
       "7944  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870803   \n",
       "7945  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870804   \n",
       "7946  http://rdf.ivdnt.org/lexica/diamant/wordform/molex/870879   \n",
       "\n",
       "                wordform             hyphenation  \\\n",
       "0              gewafelde            ge|wa|fel|de   \n",
       "1               granaten              gra|na|ten   \n",
       "2       gereglementeerde   ge|re|gle|men|teer|de   \n",
       "3              grasrijke             gras|rij|ke   \n",
       "4                 geilst                  geilst   \n",
       "...                  ...                     ...   \n",
       "7942    grondrechtelijks     grond|rech|te|lijks   \n",
       "7943    grondrechtelijke    grond|rech|te|lij|ke   \n",
       "7944     grondrechtelijk      grond|rech|te|lijk   \n",
       "7945  grondrechtelijkste  grond|rech|te|lijk|ste   \n",
       "7946   grondrechtelijkst    grond|rech|te|lijkst   \n",
       "\n",
       "                                     wordformPos  \n",
       "0     http://universaldependencies.org/u/pos/ADJ  \n",
       "1     http://universaldependencies.org/u/pos/ADJ  \n",
       "2     http://universaldependencies.org/u/pos/ADJ  \n",
       "3     http://universaldependencies.org/u/pos/ADJ  \n",
       "4     http://universaldependencies.org/u/pos/ADJ  \n",
       "...                                          ...  \n",
       "7942  http://universaldependencies.org/u/pos/ADJ  \n",
       "7943  http://universaldependencies.org/u/pos/ADJ  \n",
       "7944  http://universaldependencies.org/u/pos/ADJ  \n",
       "7945  http://universaldependencies.org/u/pos/ADJ  \n",
       "7946  http://universaldependencies.org/u/pos/ADJ  \n",
       "\n",
       "[7947 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filtering lexicon results\n",
      "List of attributive adjectives not ending with -e even though they should have a final -e:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>left context</th>\n",
       "      <th>lemma 0</th>\n",
       "      <th>pos 0</th>\n",
       "      <th>word 0</th>\n",
       "      <th>lemma 1</th>\n",
       "      <th>pos 1</th>\n",
       "      <th>word 1</th>\n",
       "      <th>lemma 2</th>\n",
       "      <th>pos 2</th>\n",
       "      <th>word 2</th>\n",
       "      <th>right context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>lekker snoepen van alles wat</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>goedheilig</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>goedheilig</td>\n",
       "      <td>man</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>man</td>\n",
       "      <td>uitdeelt Oh ja en dan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>onvoorwaardelijk gesteund door hulpverleners in</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>het</td>\n",
       "      <td>gesloten</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gesloten</td>\n",
       "      <td>instituut</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>instituut</td>\n",
       "      <td>voor jongeren met gedragsproblemen waarin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>vier spirituele wetten zijn vergiffenis</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>het</td>\n",
       "      <td>goddelijk</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>goddelijk</td>\n",
       "      <td>doel</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>doel</td>\n",
       "      <td>doelen stellen in het leven</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>het museum ter ere van</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>geestelijk</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>geestelijk</td>\n",
       "      <td>vader</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>vader</td>\n",
       "      <td>van de fictieve indiaan Winnetou</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>van de jouvers die daarmee</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gebroken</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gebroken</td>\n",
       "      <td>nacht</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>nacht</td>\n",
       "      <td>hadden overleefd Tegen acht uur</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9542</td>\n",
       "      <td>hadden gezworen Zij hadden met</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>het</td>\n",
       "      <td>gewelddadig</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>gewelddadig</td>\n",
       "      <td>omverwerpen</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>omverwerpen</td>\n",
       "      <td>van de rechtsorde en krijgstucht</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9548</td>\n",
       "      <td>Helemaal geen probleem vinden zij</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>Het</td>\n",
       "      <td>gezond</td>\n",
       "      <td>AA(degree=pos,position=adv|pred)</td>\n",
       "      <td>gezond</td>\n",
       "      <td>houden</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>houden</td>\n",
       "      <td>van mensen in het binnenland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9549</td>\n",
       "      <td>tot stand wordt gebracht zal</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=pers,person=3,gender=n,number=sg,position=pron)</td>\n",
       "      <td>het</td>\n",
       "      <td>gewoon</td>\n",
       "      <td>AA(degree=pos,position=prenom)</td>\n",
       "      <td>gewoon</td>\n",
       "      <td>onderdeel</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>onderdeel</td>\n",
       "      <td>worden van het nationaal zekerheidsstelsel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9550</td>\n",
       "      <td>op 29 oktober in NIS</td>\n",
       "      <td>het</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>Het</td>\n",
       "      <td>gevarieerd</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gevarieerd</td>\n",
       "      <td>programma</td>\n",
       "      <td>NOU-C(gender=n,number=sg)</td>\n",
       "      <td>programma</td>\n",
       "      <td>zal uit zang bestaan van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9555</td>\n",
       "      <td>van een kampleider en tegen</td>\n",
       "      <td>de</td>\n",
       "      <td>PD(type=d-p,subtype=art-def)</td>\n",
       "      <td>de</td>\n",
       "      <td>gedwongen</td>\n",
       "      <td>AA(position=prenom,degree=pos)</td>\n",
       "      <td>gedwongen</td>\n",
       "      <td>deportatie</td>\n",
       "      <td>NOU-C(gender=f|m,number=sg)</td>\n",
       "      <td>deportatie</td>\n",
       "      <td>naar LaosErkend vluchtelingDe Amerikaanse regering</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3205 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         left context lemma 0  \\\n",
       "2                        lekker snoepen van alles wat      de   \n",
       "10    onvoorwaardelijk gesteund door hulpverleners in     het   \n",
       "17            vier spirituele wetten zijn vergiffenis     het   \n",
       "18                             het museum ter ere van      de   \n",
       "26                         van de jouvers die daarmee      de   \n",
       "...                                               ...     ...   \n",
       "9542                   hadden gezworen Zij hadden met     het   \n",
       "9548                Helemaal geen probleem vinden zij     het   \n",
       "9549                     tot stand wordt gebracht zal     het   \n",
       "9550                             op 29 oktober in NIS     het   \n",
       "9555                      van een kampleider en tegen      de   \n",
       "\n",
       "                                                        pos 0 word 0  \\\n",
       "2                                PD(type=d-p,subtype=art-def)     de   \n",
       "10                               PD(type=d-p,subtype=art-def)    het   \n",
       "17                               PD(type=d-p,subtype=art-def)    het   \n",
       "18                               PD(type=d-p,subtype=art-def)     de   \n",
       "26                               PD(type=d-p,subtype=art-def)     de   \n",
       "...                                                       ...    ...   \n",
       "9542                             PD(type=d-p,subtype=art-def)    het   \n",
       "9548                             PD(type=d-p,subtype=art-def)    Het   \n",
       "9549  PD(type=pers,person=3,gender=n,number=sg,position=pron)    het   \n",
       "9550                             PD(type=d-p,subtype=art-def)    Het   \n",
       "9555                             PD(type=d-p,subtype=art-def)     de   \n",
       "\n",
       "          lemma 1                             pos 1       word 1      lemma 2  \\\n",
       "2      goedheilig    AA(degree=pos,position=prenom)   goedheilig          man   \n",
       "10       gesloten    AA(position=prenom,degree=pos)     gesloten    instituut   \n",
       "17      goddelijk    AA(degree=pos,position=prenom)    goddelijk         doel   \n",
       "18     geestelijk    AA(degree=pos,position=prenom)   geestelijk        vader   \n",
       "26       gebroken    AA(position=prenom,degree=pos)     gebroken        nacht   \n",
       "...           ...                               ...          ...          ...   \n",
       "9542  gewelddadig    AA(degree=pos,position=prenom)  gewelddadig  omverwerpen   \n",
       "9548       gezond  AA(degree=pos,position=adv|pred)       gezond       houden   \n",
       "9549       gewoon    AA(degree=pos,position=prenom)       gewoon    onderdeel   \n",
       "9550   gevarieerd    AA(position=prenom,degree=pos)   gevarieerd    programma   \n",
       "9555    gedwongen    AA(position=prenom,degree=pos)    gedwongen   deportatie   \n",
       "\n",
       "                            pos 2       word 2  \\\n",
       "2     NOU-C(gender=f|m,number=sg)          man   \n",
       "10      NOU-C(gender=n,number=sg)    instituut   \n",
       "17      NOU-C(gender=n,number=sg)         doel   \n",
       "18    NOU-C(gender=f|m,number=sg)        vader   \n",
       "26    NOU-C(gender=f|m,number=sg)        nacht   \n",
       "...                           ...          ...   \n",
       "9542    NOU-C(gender=n,number=sg)  omverwerpen   \n",
       "9548    NOU-C(gender=n,number=sg)       houden   \n",
       "9549    NOU-C(gender=n,number=sg)    onderdeel   \n",
       "9550    NOU-C(gender=n,number=sg)    programma   \n",
       "9555  NOU-C(gender=f|m,number=sg)   deportatie   \n",
       "\n",
       "                                           right context  \n",
       "2                                  uitdeelt Oh ja en dan  \n",
       "10             voor jongeren met gedragsproblemen waarin  \n",
       "17                           doelen stellen in het leven  \n",
       "18                      van de fictieve indiaan Winnetou  \n",
       "26                       hadden overleefd Tegen acht uur  \n",
       "...                                                  ...  \n",
       "9542                    van de rechtsorde en krijgstucht  \n",
       "9548                        van mensen in het binnenland  \n",
       "9549          worden van het nationaal zekerheidsstelsel  \n",
       "9550                            zal uit zang bestaan van  \n",
       "9555  naar LaosErkend vluchtelingDe Amerikaanse regering  \n",
       "\n",
       "[3205 rows x 11 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb30652d1af94839bdcc4569d3032618",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Label(value='Sla uw resultaten op:'), Text(value='mijn_resultaten.csv'), Button(button_style='w…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.search.LexiconQuery import *\n",
    "from chaininglib.utils.dfops import df_filter\n",
    "from chaininglib.ui.dfui import display_df\n",
    "\n",
    "corpus_to_search=\"openchn\"\n",
    "lexicon_to_search=\"molex\"\n",
    "\n",
    "# CORPUS: get [article + attributive adjective + nouns] combinations in which the adjective does not end with -e\n",
    "print('Get occurences of attributive adjectives not ending with -e')\n",
    "cq = create_corpus(corpus_to_search).pattern(r'[lemma=\"de|het\"][word=\"^g(.+)[^e]$\" & pos=\"AA.*degree=pos.*\"][pos=\"NOU.*\"]')\n",
    "df_corpus = cq.search().kwic()\n",
    "display(df_corpus)\n",
    "\n",
    "# LEXICON: get adjectives the lemma of which does not end with -e \n",
    "#lq = create_lexicon(lexicon_to_search).lemma('^g(.+)[^e]$').pos('ADJ').search()\n",
    "lq = create_lexicon(lexicon_to_search).lemma('^g(.+)[^e]$').pos('ADJ(degree=pos)').search()\n",
    "df_lexicon = lq.search().kwic()\n",
    "\n",
    "display(df_lexicon)\n",
    "\n",
    "# LEXICON: get adjectives having a final -e in definite attributive use\n",
    "print('Filtering lexicon results')\n",
    "final_e_condition = df_filter(df_lexicon[\"wordform\"], 'e$')\n",
    "df_lexicon_form_e = df_lexicon[ final_e_condition ]\n",
    "\n",
    "# RESULT: get the records out of our first list in which the -e-less-adjectives match the lemma form of our last list\n",
    "print('List of attributive adjectives not ending with -e even though they should have a final -e:')\n",
    "e_forms = list(df_lexicon_form_e.lemma)\n",
    "no_final_e_condition = df_filter(df_corpus[\"word 1\"], pattern=set(e_forms), method=\"isin\")\n",
    "result_df = df_corpus[ no_final_e_condition ]\n",
    "display_df( result_df )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look up inflected forms and spelling variants for a given lemma in a corpus <a class=\"anchor\" id=\"inflected-spelling-corpus\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T11:15:32.702930Z",
     "start_time": "2019-03-21T11:15:32.232593Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.ui.dfui import display_df\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.search.LexiconQuery import *\n",
    "\n",
    "# Corpus Gysseling and lexicon mnwlex are from same period: 1250-1550\n",
    "lexicon_to_search=\"mnwlex\"\n",
    "corpus_to_search=\"gysseling\"\n",
    "\n",
    "##############################################\n",
    "# TODO  zelfde met meerdere lemmata en gegroepeerd \n",
    "##############################################\n",
    "\n",
    "lemma_to_look_for=\"denken\"\n",
    "\n",
    "# LEXICON: Search for the inflected forms of a lemma in a morphosyntactic lexicon\n",
    "lq = create_lexicon(lexicon_to_search).lemma(lemma_to_look_for).search()\n",
    "df_lexicon = lq.kwic()\n",
    "display_df(df_lexicon)\n",
    "\n",
    "# Put all inflected forms into a list\n",
    "inflected_wordforms = list(df_lexicon.wordform)\n",
    "\n",
    "# CORPUS: Look up the inflected forms in a (possibly unannotated) corpus\n",
    "# beware: If the corpus is not annotated, all we can do is searching for the inflected words\n",
    "#         But if the corpus is lemmatized, we have to make sure we're retrieving correct data by specifying the lemma as well\n",
    "annotated_corpus = True\n",
    "query = r'[lemma=\"'+lemma_to_look_for+r'\" & word=\"'+r\"|\".join(inflected_wordforms)+r'\"]' if annotated_corpus else r'[word=\"'+r\"|\".join(inflected_wordforms)+r'\"]'\n",
    "cq = create_corpus(corpus_to_search).pattern(query).search()\n",
    "df_corpus = cq.kwic() \n",
    "display_df(df_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus frequency list of lemmata from lexicon with given lemma <a class=\"anchor\" id=\"corpus-frequency-lemma-pos\"></a>\n",
    "Build a function with which we can gather all lemmata of a lexicon, and build a frequency list of those lemmata in a corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T11:15:49.920956Z",
     "start_time": "2019-03-21T11:15:32.705184Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.search.LexiconQuery import *\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.process.corpus import get_frequency_list\n",
    "from chaininglib.ui.dfui import display_df\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# build a function as required. We will run it afterwards\n",
    "\n",
    "def get_frequency_list_given_a_corpus(lexicon, pos, corpus):\n",
    "    \n",
    "    # LEXICON: get a lemmata list to work with\n",
    "\n",
    "    # query the lexicon for lemma with a given part-of-speech\n",
    "    lq = create_lexicon(lexicon).pos(pos).search()\n",
    "    df_lexicon = lq.kwic()\n",
    "\n",
    "    # Put the results into an array, so we can loop through the found lemmata\n",
    "    lexicon_lemmata_arr = [w.lower() for w in df_lexicon[\"writtenForm\"]][-200:]\n",
    "    # Instantiate a DataFrame, in which we will gather all single lemmata occurences\n",
    "    df_full_list = pd.DataFrame()\n",
    "\n",
    "\n",
    "    # CORPUS: loop through the lemmata list, query the corpus with each lemma, and count the results\n",
    "\n",
    "    # It's a good idea to query more than one lemma at at the time,\n",
    "    # but not too many, otherwise the server will get overloaded!\n",
    "    nr_of_lemmata_to_query_atonce = 100\n",
    "\n",
    "    # loop over lemma list \n",
    "    for i in range(0, len(lexicon_lemmata_arr), nr_of_lemmata_to_query_atonce):\n",
    "        \n",
    "        print('Lemmata processed: '+str(i)+'/'+str(len(lexicon_lemmata_arr)))\n",
    "        \n",
    "        # slice to small array of lemmata to query at once\n",
    "        small_lemmata_arr = lexicon_lemmata_arr[i : i+nr_of_lemmata_to_query_atonce] \n",
    "\n",
    "        # join set of lemmas to send them in a query all at once\n",
    "        # beware: single quotes need escaping\n",
    "        lemmata_list = \"|\".join(small_lemmata_arr).replace(\"'\", \"\\\\\\\\'\")\n",
    "        cq = create_corpus(corpus).pattern(r'[lemma=\"' + lemmata_list + r'\"]').search()\n",
    "        df_corpus = cq.kwic()\n",
    "        \n",
    "        # add the results to the full list\n",
    "        if \"lemma 0\" in df_corpus.columns:\n",
    "            df_full_list = pd.concat( [df_full_list, df_corpus[\"lemma 0\"]] )     \n",
    "        \n",
    "\n",
    "    # make sure the columnswith that contains the lemmata is same as given to get_frequency_list function\n",
    "    column_name=\"lemma\"\n",
    "    df_full_list.columns = [column_name]\n",
    "\n",
    "    # we're done with querying, build the frequency list now\n",
    "    print('Done.')\n",
    "    freq_df = get_frequency_list(df_full_list, column_name=column_name)\n",
    "    \n",
    "\n",
    "    return freq_df\n",
    "\n",
    "    \n",
    "# run it!\n",
    "lexicon=\"molex\"\n",
    "# TODO: Maybe too much too ask all nouns? Maybe take random sample?\n",
    "corpus_to_search=\"openchn\"\n",
    "pos=\"NOU.*\"\n",
    "\n",
    "freq_df = get_frequency_list_given_a_corpus(lexicon, pos, corpus_to_search)\n",
    "\n",
    "display_df(freq_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a frequency table of some corpus, based on lemma list of a given lexicon <a class=\"anchor\" id=\"freqtable-lemmalist\"></a>\n",
    "In this case study, we compare lemma frequencies for corpora from different time periods: CHN extern (contemporary Dutch Antilles & Suriname) and Letters as Loot (sailors' letters, 17th and 18th century).\n",
    "\n",
    "*For this case study, you need to run the previous case study first, because it generates a function we need here.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T11:16:57.899540Z",
     "start_time": "2019-03-21T11:15:49.922813Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.utils.dfops import get_rank_diff\n",
    "from chaininglib.ui.dfui import display_df\n",
    "\n",
    "# For this case study, you need to run the previous case study first, because it generates a function we need here\n",
    "\n",
    "# Use lexica and corpora from same period\n",
    "base_lexicon1=\"molex\"\n",
    "corpus_to_search1=\"openchn\"\n",
    "\n",
    "base_lexicon2=\"molex\"\n",
    "corpus_to_search2=\"zeebrieven\"\n",
    "\n",
    "# ADJ gives interesting comparison\n",
    "\n",
    "pos=\"ADJ.*\"\n",
    "\n",
    "# build frequency tables of two corpora\n",
    "\n",
    "df_frequency_list1 = get_frequency_list_given_a_corpus(base_lexicon1, pos, corpus_to_search1)\n",
    "# sort and display\n",
    "df_top25_descending = df_frequency_list1.sort_values(ascending=False,by=['token count']).head(25)\n",
    "df_top25_ascending =  df_frequency_list1.sort_values(ascending=True, by=['rank']).head(25)\n",
    "display_df( df_top25_descending[['lemmas', 'token count']].set_index('lemmas'), labels='df1 chart '+corpus_to_search1, mode='chart' )\n",
    "\n",
    "df_frequency_list2 = get_frequency_list_given_a_corpus(base_lexicon2, pos, corpus_to_search2)\n",
    "# sort and display\n",
    "df_top25_descending = df_frequency_list2.sort_values(ascending=False,by=['token count']).head(25)\n",
    "df_top25_ascending =  df_frequency_list2.sort_values(ascending=True, by=['rank']).head(25)\n",
    "display_df( df_top25_descending[['lemmas', 'token count']].set_index('lemmas'), labels='df2 chart '+corpus_to_search2, mode='chart' )\n",
    "\n",
    "\n",
    "# TODO: lemmata tonen die in 1 of 2 ontbreken\n",
    "\n",
    "# compute the rank diff of lemmata in frequency tables\n",
    "\n",
    "# sort and display\n",
    "df_rankdiffs = get_rank_diff(df_frequency_list1, df_frequency_list2, index='lemmas')\n",
    "\n",
    "display_df(df_rankdiffs.sort_values(by=['rank_diff']).head(25), labels='Differences in ranks')\n",
    "\n",
    "df_top25_descending = df_rankdiffs.sort_values(ascending=False, by=['rank_diff']).head(25)\n",
    "display_df( df_top25_descending['rank_diff'], labels='chart large diff', mode='chart' )\n",
    "\n",
    "df_top25_ascending = df_rankdiffs.sort_values(ascending=True, by=['rank_diff']).head(25)\n",
    "display_df( df_top25_ascending['rank_diff'], labels='chart small diff', mode='chart' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search corpus for wordforms of lemma not included in lexicon <a class=\"anchor\" id=\"corpus-wordforms-not-lexicon\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T11:17:09.521210Z",
     "start_time": "2019-03-21T11:16:57.901052Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.search.LexiconQuery import *\n",
    "from chaininglib.search.CorpusQuery import *\n",
    "from chaininglib.ui.dfui import display_df\n",
    "\n",
    "# Let's build a function to do the job:\n",
    "# The function will require a lexicon name and a part-of-speech to limit the search to, and the name of a corpus to be searched.\n",
    "# It will return a Pandas DataFrame associating lemmata to their paradigms ('known_wordforms' column) and\n",
    "# missing wordforms found in the corpus ('unknown_wordforms' column).\n",
    "\n",
    "def get_missing_wordforms(lexicon_name, lexicon_postag, corpus, corpus_postag):    \n",
    "    \n",
    "    print('Finding missing wordforms in a lexicon can take some time...');\n",
    "    \n",
    "    # LEXICON: \n",
    "    # get a lemmata list having a given part-of-speech\n",
    "    # MoLex is a convenient source to get a list of lemmata\n",
    "    \n",
    "    lq = create_lexicon(\"molex\").pos(lexicon_postag).search()\n",
    "    df_lexicon = lq.kwic()\n",
    "    \n",
    "    # Put the results into an array, so we can loop through the list of lemmata\n",
    "    lexicon_lemmata_arr = [w.lower() for w in df_lexicon[\"writtenForm\"]][-50:]\n",
    "    \n",
    "    # Test array, instead of querying Molex\n",
    "    #lexicon_lemmata_arr = [\"denken\", \"doen\", \"hebben\", \"maken\"]\n",
    "    \n",
    "    # Prepare the output:\n",
    "    # instantiate a DataFrame for storing lemmata and mssing wordforms\n",
    "    df_enriched_lexicon = pd.DataFrame(index=lexicon_lemmata_arr, columns=['lemma', 'pos', 'known_wordforms', 'unknown_wordforms'])\n",
    "    df_enriched_lexicon.index.name = 'lemmata'\n",
    "    \n",
    "    # CORPUS: \n",
    "    # loop through the lemmata list, query the corpus for each lemma, \n",
    "    # and compute paradigms differences between both\n",
    "\n",
    "    \n",
    "    # loop through the lemmata list\n",
    "    # and query the corpus for occurances of the lemmata\n",
    "    \n",
    "    # It's a good idea to work with more than one lemma at the time (speed)!\n",
    "    nr_of_lemmata_to_query_atonce = 100\n",
    "    \n",
    "    for i in range(0, len(lexicon_lemmata_arr), nr_of_lemmata_to_query_atonce):\n",
    "        \n",
    "        # slice to small array of lemmata to query at once\n",
    "        small_lemmata_arr = lexicon_lemmata_arr[i : i+nr_of_lemmata_to_query_atonce]\n",
    "        \n",
    "        # join set of lemmata to send them in a query all at once\n",
    "        # beware: single quotes need escaping\n",
    "        lemmata_list = \"|\".join(small_lemmata_arr).replace(\"'\", \"\\\\\\\\'\")\n",
    "        print(\"Querying lemmata %i-%i of %i from corpus.\" % (i, i+nr_of_lemmata_to_query_atonce, len(lexicon_lemmata_arr) ))\n",
    "        cq = create_corpus(corpus).pattern(r'[lemma=\"' + lemmata_list + r'\" & pos=\"'+corpus_postag+'\"]').search()\n",
    "        df_corpus = cq.kwic()\n",
    "        \n",
    "        # if the corpus gave results,\n",
    "        # query the lexicon for the same lemmata\n",
    "        # and compare the paradigms!\n",
    "        \n",
    "        if (len(df_corpus)>0):\n",
    "            small_lemmata_set = set(small_lemmata_arr)\n",
    "            for one_lemma in small_lemmata_set: \n",
    "                \n",
    "                # look up the known wordforms in the lexicon\n",
    "                ql = create_lexicon(lexicon_name).lemma(one_lemma).search()\n",
    "                df_known_wordforms = ql.kwic()\n",
    "                \n",
    "                # we have a lexicon paradigm to compare, do the job now\n",
    "                if (len(df_known_wordforms) != 0):\n",
    "                    \n",
    "                    # gather the lexicon wordforms in a set\n",
    "                    known_wordforms = set( df_known_wordforms['wordform'].str.lower() )\n",
    "                    \n",
    "                    # gather the corpus wordforms (of the same lemma) in a set too\n",
    "                    corpus_lemma_filter = (df_corpus['lemma 0'] == one_lemma)\n",
    "                    corpus_wordforms = set( (df_corpus[ corpus_lemma_filter ])['word 0'].str.lower() )\n",
    "                    \n",
    "                    # Now compute the differences:\n",
    "                    # gather in a set all the corpus wordforms that cannot be found in the lexicon wordforms \n",
    "                    unknown_wordforms = corpus_wordforms.difference(known_wordforms)\n",
    "\n",
    "                    # If we found some missing wordforms, add the results to the output!\n",
    "                    \n",
    "                    if (len(unknown_wordforms) !=0):                        \n",
    "                        # The index of our results will be a key consisting of lemma + part-of-speech\n",
    "                        # Part-of-speech is needed to distinguish homonyms with different grammatical categories.\n",
    "                        # Of course, we need to take glosses into account too to do a truely correct job\n",
    "                        # But we didn't do it here\n",
    "                        key = one_lemma + lexicon_postag\n",
    "                        df_enriched_lexicon.at[key, 'lemma'] = one_lemma\n",
    "                        df_enriched_lexicon.at[key, 'pos'] = lexicon_postag\n",
    "                        df_enriched_lexicon.at[key, 'known_wordforms'] = known_wordforms\n",
    "                        df_enriched_lexicon.at[key, 'unknown_wordforms'] = unknown_wordforms\n",
    "                \n",
    "    # return non-empty results, t.i. cases in which we found some wordforms\n",
    "    return df_enriched_lexicon[ df_enriched_lexicon['unknown_wordforms'].notnull() ]\n",
    "\n",
    "\n",
    "# Run the function!\n",
    "#\n",
    "# ask the lexicon which wordforms it knows, and try to find new unknown wordforms in the corpus\n",
    "\n",
    "lexicon_name=\"mnwlex\"\n",
    "corpus_to_search=\"zeebrieven\"\n",
    "\n",
    "# beware: lexicon and corpus may have different parts-of-speech sets in use\n",
    "df = get_missing_wordforms(lexicon_name, \"VERB\", corpus_to_search, \"VRB\")\n",
    "\n",
    "# After such a heavy process, it's a good idea to save the results\n",
    "\n",
    "df.to_csv( \"missing_wordforms.csv\", index=False)\n",
    "\n",
    "display_df(df, labels='Missing wordforms')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Treebanks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Treebank search <a class=\"anchor\" id=\"treebank-search\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-08-21T10:23:19.353216Z",
     "start_time": "2019-08-21T10:23:17.946754Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.search.TreebankQuery import *\n",
    "\n",
    "\n",
    "print (\"search...\")\n",
    "tbq = create_treebank(\"cgn\").pattern(\"//node[@cat='pp' and node[@cat='ap' and node[@cat='np']]]\").search()\n",
    "\n",
    "print (\"get XML...\")\n",
    "\n",
    "xml = tbq.xml()\n",
    "print(xml)\n",
    "\n",
    "print (\"get trees and their string representations...\")\n",
    "\n",
    "trees = tbq.trees()\n",
    "\n",
    "for tree in trees:\n",
    "    display(tree.toString())\n",
    "\n",
    "df = tbq.kwic()\n",
    "    \n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which kind of nouns are used in a prepositional complement of the verb *geven* ? <a class=\"anchor\" id=\"treebank-objects-geven\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-21T11:17:43.637212Z",
     "start_time": "2019-03-21T11:17:29.618818Z"
    }
   },
   "outputs": [],
   "source": [
    "from chaininglib.search.TreebankQuery import *\n",
    "\n",
    "\n",
    "print (\"search...\")\n",
    "\n",
    "tbq = create_treebank(\"cgn\").pattern('//node[node[@rel=\"hd\" and @pt=\"ww\" and @root=\"geven\"] and node[@rel=\"obj1\" and @pt=\"n\"]]').search()\n",
    "\n",
    "\n",
    "print (\"get list of nouns which are part of an PP, as argument of predicate 'geven'...\")\n",
    "\n",
    "trees = tbq.trees()\n",
    "\n",
    "list_of_nouns = []\n",
    "for tree in trees:\n",
    "    nouns = tree.extract(['pp', 'np'])\n",
    "    list_of_nouns = list_of_nouns + nouns\n",
    "    \n",
    "\n",
    "display(list_of_nouns)\n",
    "    \n",
    "df = tbq.kwic(align_lemma='geven')\n",
    "display(df)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}