cwpk-53-intro-other-tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#53: Intro to Other Tools\n",
    "=======================================\n",
    "\n",
    "Shifting Our Focus to How to Use the Knowledge Graph\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "Today's installment marks a kind of a turning point in our [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series. Now that our procedures for building, extracting, and managing the knowledge graph itself are in place, we can shift gears to explore how we may use this knowledge artifact. In today's installment, I introduce a number of category of tools for doing so, and point to their more formal treatment in ensuing installments.\n",
    "\n",
    "Some of our tools treat the knowledge graph as an object unto itself, providing statistics, logging, and traversals.  Some tools allow us to publish these interactive Notebooks or enable external access to the knowlege graph. These measures are largely independent of the specific *content* in [KBpedia](https://kbpedia.org/). Some tools aid visualizations, sometimes dynamic. Some of the tools enable us to do advanced analysis or machine learning. We can also do much with natural language processing and understanding. We can also find novel representations of our knowledge graph -- as a graph, as documents, as relations, or as terms -- that we can *embed* in our learners.\n",
    "\n",
    "These are the topics of most of our remaining installations in this **CWPK** series. Consider this installment as an introduction, then, to the remainder of this series. I present these tool clusters in approximate order of treament."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stats\n",
    "There is a wealth of counts and distributions of various resources within the KBpedia knowledge graph (or any graph, for that matter). Some of these statistics are automatically provided when using something like the [Protégé](https://en.wikipedia.org/wiki/Prot%C3%A9g%C3%A9_(software)) editor. We will accept these statistics as given and concentrate on other counts and statistics not provided by Protégé that we may calculate directly from the knowledge graph with Python. We address stats in the next **CWPK** installment.\n",
    "\n",
    "\n",
    "### Logging\n",
    "Python comes with a very capable <code>logging</code> module that is more useful than <code>print</code> statements sprinkled throughout the code. We will explore this module in some depth, and point to other useful logging extensions in the next **CWPK** installment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Charting\n",
    "There are many wonderful charting packages available in Python, some of which also are designed to work nicely with interactive Notebook pages. We'll survey these options and present some charting utilities as part of *cowpoke* in **CWPK #55**. We will use some of the stats calculated in the prior installment to provide the data for these charts.\n",
    "\n",
    "### Graphing and Graph Extraction\n",
    "A knowledge graph, duh, has a graph or network structure. Many properties (edges) connect multiple nodes (classes or reference concepts in the case of KBpedia). It is difficult to visualize these structures in their entirety, and sometimes computationally intense to render them. It is also possible to extract out local portions of the graph, and presenting a simpler sub-graph representation.\n",
    "\n",
    "Graph visualization and extract has many fewer options than charting, and ease-of-use and performance can also be challenges. We will inspect what is available with Python and make some visualization selections in **CWPK #56**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Publishing Stuff\n",
    "The progress of covering these topics means that sufficient criteria for eventually going live with the Cooking with Python and KBpedia series have been met and it is time to start the public release of the series. In the process, we will need to set up remote instances of *cowpoke* and KBpedia, establish and endpoint for querying it, and begin to publish our Notebook series. We want to publish those Notebooks such that they retain their interactivity. \n",
    "\n",
    "Researching, deciding upon, and then implementing the choices for these tools occupies four installments from **CWPK #57** to **#60**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Natural Language Processing\n",
    "In our **Part VI** that concludes the new coding and substantive portions of this series (**CWPK #61** to **CWPK #71**) we have much occasion to explore some additional specialty topics in natural language processing and understanding. These examples will tie into some of the most important Python packages available in NLP.\n",
    "\n",
    "### Embedding Models\n",
    "One major use of knowledge graqhs is providing the grist to various embedding models. Because of KBpedia's mappings to full-length articles in Wikipedia, we also have the option of employing document or word-rich embedding models. We will tee up the embedding approach in **CWPK #63** and then develop specific models in **CWPK #64**. We will stage and test a number of different embedding models, ranging from single term or concept ones to ones that leverage virtually all structural aspects of the knowledge graph.\n",
    "\n",
    "### Machine Learning\n",
    "The biggest chunk of installments in the **CWPK** series involves machine learning in a variety of forms. These examples undertake some aggressive uses of the knowledge graph, and then concludes the series with summaries of operating procedures and steps. We first look at 'standard' machine learning, with an emphasis on selecting models, creating training sets, setting parameters, and tuning performance. We then devote four installments (**CWPK #67** to **CWPK #70**) to so-called 'deep learning' models. We are then able to assemble up and compare results from all classifier learning activities in **CWPK #71**.\n",
    "\n",
    "The concluding last installments are all narrative in nature and wrap-up the series and provide summaries of use steps and general guidance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site. The <em>cowpoke</em> Python <a href=\"https://github.com/Cognonto/cowpoke\">code listing covering the series</a> is also available from GitHub.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}