01_Geoparser_Basics_Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "61d59df1-22a9-42c6-9846-48348b916585",
   "metadata": {},
   "source": [
    "# Irchel Geoparser Workshop: Getting Started with Geoparser\n",
    "\n",
    "This notebook is designed to help you explore and interact with the Geoparser library. By the end of this tutorial, you will have a solid understanding of how to use Geoparser for basic geoparsing tasks.\n",
    "\n",
    "**Objectives:**\n",
    "\n",
    "- Understand how to initialize and use Geoparser.\n",
    "- Perform geoparsing on text strings and documents.\n",
    "- Access and analyze geoparsing results.\n",
    "- Visualize locations on a map.\n",
    "\n",
    "---\n",
    "\n",
    "## 📖 **Documentation**\n",
    "\n",
    "This tutorial requires you to consult the Geoparser documentation: [docs.geoparser.app](https://docs.geoparser.app). Knowing where to find information about the library is crucial for working with it effectively, especially as it is likely that future updates may bring changes to functionality. The documentation will always be the definitive reference for how to use the package.\n",
    "\n",
    "---\n",
    "\n",
    "## 1. Geoparsing Text Strings <a name=\"geoparsing-text-strings\"></a>\n",
    "\n",
    "Let's begin by geoparsing some simple text strings directly within this notebook.\n",
    "\n",
    "### 1.1 Initialize Geoparser <a name=\"task-initialize-geoparser\"></a>\n",
    "\n",
    "**Objective:** Initialize the Geoparser so that it's ready to parse text.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Import the necessary class to use Geoparser.\n",
    "- Initialize an instance of the `Geoparser` class.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f23b3d8-6ea5-4bda-8314-401c1482b807",
   "metadata": {},
   "outputs": [],
   "source": [
    "from geoparser import Geoparser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ac7a309-9bab-49de-89d2-340aa0532b17",
   "metadata": {},
   "outputs": [],
   "source": [
    "geoparser = Geoparser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a794ea-b96c-44e5-accf-346f9f1055eb",
   "metadata": {},
   "source": [
    "### 1.2 Geoparse a List of Strings <a name=\"task-geoparse-a-list-of-strings\"></a>\n",
    "\n",
    "**Objective:** Use the Geoparser to parse a list of text strings.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use the example texts provided below.\n",
    "- Use the initialized Geoparser to parse these texts.\n",
    "- Store the results in a variable for further analysis.\n",
    "\n",
    "**Example Texts:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1be02052-bbe5-4b44-adc8-febe36de03be",
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = [\n",
    "    \"The University of Zurich, located in the heart of Zurich, Switzerland, is the largest university in Switzerland. It was founded in 1833 and has since become a leading institution in Europe.\",\n",
    "    \"Researchers at the University of Zurich frequently collaborate with institutions in Germany, Belgium, Japan and Australia.\",\n",
    "    \"The Botanical Garden of the University of Zurich is a popular attraction, featuring plants from regions like the Amazon rainforest and the Himalayas.\",\n",
    "    \"Notable alumni include Albert Einstein, who later worked at the Institute for Advanced Study in Princeton, New Jersey.\",\n",
    "    \"Students at the University of Zurich have the opportunity to study abroad in places like Tokyo, Sydney, and Cape Town.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4922c5a-832b-4300-9fe1-598a48ed1080",
   "metadata": {},
   "source": [
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a50e3802-bb62-4fd5-8b4c-eca1dd60c3ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = geoparser.parse(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cd0fa24-84d2-4dad-b0ef-bcb03bc9a355",
   "metadata": {},
   "source": [
    "### 1.3 Exploring Geoparsing Results <a name=\"exploring-geoparsing-results\"></a>\n",
    "\n",
    "**Objective:** Understand and explore the structure of the results.\n",
    "\n",
    "Now that you've parsed the texts, let's explore the results. Geoparsed documents are stored in `GeoDoc` objects, which contain `GeoSpan` objects for each recognized toponym. Each `GeoSpan` represents a toponym and provides access to resolved location information.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Iterate over the parsed documents:\n",
    "    - For each document:\n",
    "      - Print the original text.\n",
    "      - For each toponym found in the document:\n",
    "        - Print the toponym text.\n",
    "        - Access the resolved location information.\n",
    "        - Print some attributes of the resolved location.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- Use `for` loops to iterate over the documents and toponyms.\n",
    "- Check if the `location` attribute of a toponym is valid (i.e. not `None`) before accessing its attributes.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75fe404e-85fd-4ea3-a36c-3b575c98b9fd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for doc in docs:\n",
    "    print(f\"Document: {doc.text}\")\n",
    "    for toponym in doc.toponyms:\n",
    "        print(f\"- Toponym: {toponym.text}\")\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            print(f\"  Resolved Location: {location['name']}, {location['country_name']}\")\n",
    "            print(f\"  Feature Type: {location['feature_type']}\")\n",
    "            print(f\"  Coordinates: ({location['latitude']}, {location['longitude']})\")\n",
    "        else:\n",
    "            print(\"  Location could not be resolved.\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e84c8b57-0ae8-449f-9ec1-84a68673febb",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Geoparsing Text Documents <a name=\"geoparsing-text-documents\"></a>\n",
    "\n",
    "In this section, we'll work with actual text documents.\n",
    "\n",
    "### 2.1 Loading Text Files <a name=\"loading-text-files\"></a>\n",
    "\n",
    "**Objective:** Load text files from your `data` folder.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Prepare some text files that you would like to geoparse.\n",
    "  - You can use your own documents or, for example, download text files from [this site](http://textfiles.com/wdirectory.html).\n",
    "- Store the `.txt` files in a dedicated `data` folder in your working directory.\n",
    "- Use the provided function `load_texts_from_directory` to load all `.txt` files from the `data` directory.\n",
    "\n",
    "**Prepared Code:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e1e2fa1-d9a0-4332-b09b-63e17a1990bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "\n",
    "def load_texts_from_directory(directory):\n",
    "    texts = []\n",
    "    filenames = []\n",
    "    for filename in os.listdir(directory):\n",
    "        if filename.endswith('.txt'):\n",
    "            with open(os.path.join(directory, filename), 'r', encoding='utf-8') as file:\n",
    "                text = file.read()\n",
    "                # Replace newlines followed by a non-capitalized letter with a space\n",
    "                text = re.sub(r'\\n([a-z])', r' \\1', text)\n",
    "                texts.append(text)\n",
    "                filenames.append(filename)\n",
    "    return texts, filenames\n",
    "\n",
    "# Load the texts\n",
    "texts, filenames = load_texts_from_directory('data')\n",
    "\n",
    "# Check loaded texts\n",
    "for filename, text in zip(filenames, texts):\n",
    "    print(f\"{filename}: {len(text)} characters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb41418b-3d13-4176-beac-7746ab124d1a",
   "metadata": {},
   "source": [
    "### 2.2 Geoparse Loaded Documents <a name=\"task-geoparse-loaded-documents\"></a>\n",
    "\n",
    "**Objective:** Use Geoparser to parse the loaded text documents.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use the Geoparser to parse the list of texts you just loaded.\n",
    "- Store the results in a variable.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38618a91-84c5-4ce0-be70-606bf0edaea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = geoparser.parse(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b573dd57-6d57-4c01-a30a-0caf14c5c91b",
   "metadata": {},
   "source": [
    "### 2.3 Analyzing Results <a name=\"analyzing-results\"></a>\n",
    "\n",
    "**Objective:** Analyze the geoparsing results qualitatively.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- For each document:\n",
    "  - Print the filename.\n",
    "  - Print the number of toponyms found.\n",
    "  - List the toponyms and their resolved locations.\n",
    "  - For each resolved toponym, also print the similarity score.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "  - You can retrieve the sentence the toponym appears in using the `GeoSpan.sent.text` property.\n",
    "  - Use `for doc, filename in zip(docs, filenames)` to iterate through docs and filenames in parallel.\n",
    "\n",
    "**Analyze the Results:**\n",
    "\n",
    "- Qualitatively assess the results regarding toponym recognition and resolution.\n",
    "- Consider how the similarity score reflects the quality of the resolution.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5b70ad6-d792-4aa0-819e-90211fd3bcb6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for doc, filename in zip(docs, filenames):\n",
    "    print(f\"Filename: {filename}\")\n",
    "    print(f\"Number of toponyms found: {len(doc.toponyms)}\")\n",
    "    print()\n",
    "    for toponym in doc.toponyms:\n",
    "        print(f\"- Toponym: {toponym.text}\")\n",
    "        print(f\"  Sentence: {toponym.sent.text.strip()}\")\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            print(f\"  Resolved Location: {location['name']}, {location['admin1_name']}, {location['country_name']}\")\n",
    "            print(f\"  Score: {toponym.score}\")\n",
    "        else:\n",
    "            print(\"  Location could not be resolved.\")\n",
    "        print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "996eb53e-aadb-4564-8ff9-38c78995d6f1",
   "metadata": {},
   "source": [
    "### 2.4 Exploring Different Models <a name=\"task-exploring-different-models\"></a>\n",
    "\n",
    "**Objective:** Explore how using different spaCy and transformer models affect the geoparsing results in terms of toponym recognition and resolution.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Reinitialize Geoparser with different configurations.\n",
    "- Experiment with a different spaCy model (e.g. `en_core_web_trf`).\n",
    "- Experiment with a different transformer model (e.g. `dguzh/geo-all-distilroberta-v1`).\n",
    "- Compare the results (briefly).\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- You may need to install additional spaCy models if they are not already installed.\n",
    "\n",
    "**Note:** The suggested models are expected to be more accurate, but can also significantly increase runtimes when used for large collections of text.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8879490a-f27d-48a6-90ca-c3850cbbd483",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "geoparser = Geoparser(\n",
    "    spacy_model=\"en_core_web_trf\",\n",
    "    transformer_model=\"dguzh/geo-all-distilroberta-v1\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aac7f98-5786-4cae-8086-bb764761032c",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = geoparser.parse(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e90fb264-938b-4c46-82ab-3fa99f784a85",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for doc, filename in zip(docs, filenames):\n",
    "    print(f\"Filename: {filename}\")\n",
    "    print(f\"Number of toponyms found: {len(doc.toponyms)}\")\n",
    "    print()\n",
    "    for toponym in doc.toponyms:\n",
    "        print(f\"- Toponym: {toponym.text}\")\n",
    "        print(f\"  Sentence: {toponym.sent.text.strip()}\")\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            print(f\"  Resolved Location: {location['name']}, {location['admin1_name']}, {location['country_name']}\")\n",
    "            print(f\"  Score: {toponym.score}\")\n",
    "        else:\n",
    "            print(\"  Location could not be resolved.\")\n",
    "        print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877b2d5e-171c-4454-95fe-2b76e7e79b80",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Visualizing Locations on a Map <a name=\"visualizing-locations-on-a-map\"></a>\n",
    "\n",
    "Visualizing the extracted locations on a map can provide valuable insights.\n",
    "\n",
    "### 3.1 Preparing Data for Mapping <a name=\"preparing-data-for-mapping\"></a>\n",
    "\n",
    "**Objective:** Extract coordinates of resolved locations and prepare them for mapping.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Extract the coordinates of all resolved locations from the parsed documents.\n",
    "- Prepare a list of tuples containing the latitude and longitude of each location.\n",
    "  - The expected structure is a list of tuples: `[(latitude1, longitude1), (latitude2, longitude2), ...]`\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18f1241a-5500-4c7b-af5a-fa712b7ea574",
   "metadata": {},
   "outputs": [],
   "source": [
    "coordinates = []\n",
    "\n",
    "for doc in docs:\n",
    "    for toponym in doc.toponyms:\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            coordinates.append((location['latitude'], location['longitude']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d00a4f1-e42a-44c9-bea9-48132f19c6bf",
   "metadata": {},
   "source": [
    "### 3.2 Map the Geoparsed Locations <a name=\"task-map-the-geoparsed-locations\"></a>\n",
    "\n",
    "**Objective:** Create an interactive map displaying the resolved locations.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use the provided function `create_map` to generate a map with markers for each location.\n",
    "\n",
    "**Prepared Code:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09e7a8d3-33bf-4919-819c-dda7fa418b51",
   "metadata": {},
   "outputs": [],
   "source": [
    "import folium\n",
    "\n",
    "def create_map(coordinates):\n",
    "    # Initialize map\n",
    "    m = folium.Map()\n",
    "\n",
    "    # Add markers to the map\n",
    "    for coord in coordinates:\n",
    "        folium.Marker(\n",
    "            location=[coord[0], coord[1]],\n",
    "        ).add_to(m)\n",
    "\n",
    "    return m\n",
    "\n",
    "# Create and display the map\n",
    "m = create_map(coordinates)  # Replace 'coordinates' with your variable\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1075e7-5ee6-4ed9-badc-9e7220aa3f6c",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 4. Additional Exercises <a name=\"additional-exercises\"></a>\n",
    "\n",
    "### 4.1 Handling Ambiguous Toponyms <a name=\"handling-ambiguous-toponyms\"></a>\n",
    "\n",
    "**Objective:** Explore how contextual clues affect toponym disambiguation.\n",
    "\n",
    "Ambiguous toponyms like \"Paris\" can refer to multiple places around the world. Geoparser uses contextual clues to disambiguate such toponyms.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Parse the provided example texts.\n",
    "\n",
    "**Analyze the Results:**\n",
    "\n",
    "- Qualitatively assess how Geoparser resolves the toponym \"Paris\" in each case.\n",
    "- How does the context influence the resolution?\n",
    "- What limitations can you observe?\n",
    "\n",
    "**Example Texts:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9051d504-61fc-4b90-9697-e0ecd5671b19",
   "metadata": {},
   "outputs": [],
   "source": [
    "ambiguous_texts = [\n",
    "    \"The Eiffel Tower is one of the most famous landmarks in Paris.\",\n",
    "    \"After passing through Paris, we drove all the way to Dallas.\",\n",
    "    \"I have friends living in Paris, Ontario, who love to kayak on the Grand River.\",\n",
    "    \"The Governor of Texas visited Paris for the 2024 Summer Olympics.\",\n",
    "    \"There are 34 places named Paris in the United States. Examples include Paris, Arkansas, Paris, Texas and Paris, Wisconsin.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83aad817-d308-4a1f-bc84-992d288aa75f",
   "metadata": {},
   "source": [
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c720a083-a249-4eb5-a2da-11d091ac7a74",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "docs = geoparser.parse(ambiguous_texts)\n",
    "\n",
    "for doc in docs:\n",
    "    print(f\"Document: {doc.text}\")\n",
    "    for toponym in doc.toponyms:\n",
    "        print(f\"- Toponym: {toponym.text}\")\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            print(f\"  Resolved Location: {location['name']}, {location['admin1_name']}, {location['country_name']}\")\n",
    "            print(f\"  Score: {toponym.score}\")\n",
    "        else:\n",
    "            print(\"  Location could not be resolved.\")\n",
    "        print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623a869f-9122-4598-8583-8cfc3388f5f0",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.2 Filtering Candidate Locations <a name=\"filtering-candidate-locations\"></a>\n",
    "\n",
    "**Objective:** Use filters to restrict candidate locations during parsing.\n",
    "\n",
    "In some cases, you might want to restrict the candidate locations considered during geoparsing. This can be done by applying filters during candidate generation.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use the ambiguous texts from the previous exercise.\n",
    "- Apply a filter during parsing so that it only considers locations in France.\n",
    "\n",
    "**Note:** In practice, filters should only be used if you are absolutely sure about the scope of location references. Generally, using no filter is the recommended way to go.\n",
    "\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9df2030-dfa8-4814-811e-56671017d670",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "filter={'country_name': ['France']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0bd9ea6-de0f-4dc1-8e55-6a0335be0b26",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = geoparser.parse(ambiguous_texts, filter=filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf51cab7-f0bf-4cae-afde-fc6ec35c82cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in docs:\n",
    "    print(f\"Document: {doc.text}\")\n",
    "    for toponym in doc.toponyms:\n",
    "        print(f\"- Toponym: {toponym.text}\")\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            print(f\"  Resolved Location: {location['name']}, {location['admin1_name']}, {location['country_name']}\")\n",
    "            print(f\"  Score: {toponym.score}\")\n",
    "        else:\n",
    "            print(\"  Location could not be resolved.\")\n",
    "        print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "310a2a0d-9427-4f8f-ba85-1e65543f4830",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5. Bonus Exercise: Integrating Geoparser into a Workflow <a name=\"bonus-exercise\"></a>\n",
    "\n",
    "In this bonus exercise, we'll see how Geoparser can be integrated into a real-world workflow. We'll fetch news article headlines based on a keyword, parse them with Geoparser, and visualize the extracted locations on a map.\n",
    "\n",
    "### 5.1 Fetching News Articles <a name=\"fetching-news-articles\"></a>\n",
    "\n",
    "**Objective:** Fetch news article headlines from Google News RSS feed based on a keyword.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Choose a keyword to search for news articles (e.g., \"Technology\", \"Climate Change\", etc.).\n",
    "- Use the `fetch_news_articles` function to get the article titles.\n",
    "\n",
    "**Prepared Code:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ebbc8ff-a8ff-47f9-b17f-9f2b7e96d999",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import xml.etree.ElementTree as ET\n",
    "\n",
    "def fetch_news_articles(keyword):\n",
    "    url = f\"https://news.google.com/rss/search?q={keyword}\"\n",
    "    response = requests.get(url)\n",
    "\n",
    "    # Parse the RSS feed\n",
    "    root = ET.fromstring(response.content)\n",
    "\n",
    "    # Extract article titles\n",
    "    articles = []\n",
    "    for item in root.findall(\".//item\"):\n",
    "        title = item.find(\"title\").text\n",
    "        articles.append(title)\n",
    "\n",
    "    return articles\n",
    "\n",
    "# Choose a keyword\n",
    "keyword = \"Climate Change\"  # Replace with your chosen keyword\n",
    "\n",
    "# Fetch articles\n",
    "articles = fetch_news_articles(keyword)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d4c041-4a14-44fe-8365-5a11fd5c34b2",
   "metadata": {},
   "source": [
    "### 5.2 Parse the Articles with Geoparser <a name=\"task-parse-articles\"></a>\n",
    "\n",
    "**Objective:** Use Geoparser to parse the list of article titles.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use Geoparser to parse the `articles`.\n",
    "- Store the parsed documents for further analysis.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5abff439-4d74-4704-a1aa-a3589817839b",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = geoparser.parse(articles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa6e37e5-ce28-4132-94f8-53d7faf27dd3",
   "metadata": {},
   "source": [
    "### 5.3 Collecting and Analyzing Toponym Data <a name=\"collecting-toponym-data\"></a>\n",
    "\n",
    "**Objective:** Collect toponym data from the parsed article headlines.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Iterate over the parsed documents.\n",
    "- For each toponym in a document:\n",
    "  - Access the resolved location information.\n",
    "  - Group the results by location and format the data in a structure suitable for mapping.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- We want to create a data structure `toponym_data` that is a dictionary where:\n",
    "  - The key is a tuple `(location_name, latitude, longitude)`, representing a referenced location.\n",
    "  - The value is another dictionary with:\n",
    "    - `\"count\"`: number of times the location is referenced.\n",
    "    - `\"sentences\"`: list of sentences (article titles) where the location is referenced.\n",
    "- We have initialized a `defaultdict` named `toponym_data` in the code cell below. You can add data to this `defaultdict` like this:\n",
    "  - To up the counter for a specific location:\n",
    "    - `toponym_data[(\"Zurich\", 47.3769, 8.5417)][\"count\"] += 1`\n",
    "  - To add a sentence:\n",
    "    - `toponym_data[(\"Zurich\", 47.3769, 8.5417)][\"sentences\"].append(\"Zurich is a financial hub.\")`\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f401033-2aec-457a-a2d7-bc27a493df97",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "# Initialize toponym_data\n",
    "toponym_data = defaultdict(lambda: {\"count\": 0, \"sentences\": []})\n",
    "\n",
    "# Collect toponym data\n",
    "for doc in docs:\n",
    "    for toponym in doc.toponyms:\n",
    "        location = toponym.location\n",
    "        if location:\n",
    "            toponym_data[(location[\"name\"], location[\"latitude\"], location[\"longitude\"])][\"count\"] += 1\n",
    "            toponym_data[(location[\"name\"], location[\"latitude\"], location[\"longitude\"])][\"sentences\"].append(toponym.sent.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a9b910-78f3-4355-b54b-067490536292",
   "metadata": {},
   "source": [
    "### 5.4 Visualizing the Locations on a Map <a name=\"visualizing-news-locations\"></a>\n",
    "\n",
    "**Objective:** Create a map with the collected location data.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use the `create_news_map` function to create a map using your `toponym_data`.\n",
    "\n",
    "**Prepared Code:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1ede677-a089-479a-97a7-68d7fa743524",
   "metadata": {},
   "outputs": [],
   "source": [
    "import folium\n",
    "\n",
    "def create_news_map(toponym_data):\n",
    "    # Initialize map centered at global coordinates\n",
    "    m = folium.Map()\n",
    "\n",
    "    for (name, lat, lon), data in toponym_data.items():\n",
    "        # Prepare popup content\n",
    "        sentences_html = \"<br><br>\".join(data[\"sentences\"])\n",
    "        popup_html = f\"\"\"\n",
    "        <strong>{name}</strong><br><br>{sentences_html}\n",
    "        \"\"\"\n",
    "\n",
    "        folium.CircleMarker(\n",
    "            location=[lat, lon],\n",
    "            radius=5 + data[\"count\"] * 2,  # Size scales with the number of occurrences\n",
    "            popup=folium.Popup(popup_html, max_width=300),\n",
    "            color='crimson',\n",
    "            fill=True,\n",
    "            fill_color='crimson'\n",
    "        ).add_to(m)\n",
    "    return m\n",
    "\n",
    "# Create and display the map\n",
    "m = create_news_map(toponym_data)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b4426fe-f40a-4aa6-91bb-40332de546e8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}