cwpk-38-stub-starting-files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#38: Stubs and Starting Files\n",
    "=======================================\n",
    "\n",
    "This Installment Defines KBpedia's 'Bootstraps'\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "We begin the build process with this installment in the [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series. We do so by creating the '[bootstraps](https://en.wikipedia.org/wiki/Bootstrapping)' of 'core' [ontologies](https://en.wikipedia.org/wiki/Ontology_(information_science)) that are the targets as we ingest new classes, properties, and annotations for [KBpedia](https://kbpedia.org/). The general process we outline herein is appropriate to building any large knowledge graph. You may swap out your own starting ontology and semantic scaffoldings to apply this process to different knowledge graphs.\n",
    "\n",
    "The idea of a 'bootstrap' in computer science means a core set of rudimentary instructions that is called at immediate initialization of a program. This bootstrapped core provides all of the parent instructions that are called by the subsequent applications that actually do the desired computer tasks. The bootstrap is the way those applications can perform basic binary operations like allocating registers, creating files, pushing or popping instructions to the stack, and other low-level functions.\n",
    "\n",
    "In the case of KBpedia and our approach to the build process for knowledge graphs, the 'bootstrap' is the basic calls to the semantic languages such as [RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework) or [OWL](https://en.wikipedia.org/wiki/Web_Ontology_Language) and the creation of a top-level parental set of classes and properties to which we connect the subsequent knowledge graph content. We call these starting bootstraps 'stubs'.\n",
    "\n",
    "These 'stubs' are created outside of the build process, generally using an ontology [IDE](https://en.wikipedia.org/wiki/Integrated_development_environment) like [Protégé](https://en.wikipedia.org/wiki/Prot%C3%A9g%C3%A9_(software)). In our case, we have already created the 'stubs' used in the various KBpedia build processes. As we create new versions, we must make some minor modifications to these 'stubs'. However, in general, the stubs are rather static in nature and may only rarely need to be changed in a material manner. As you will see from inspection, these stubs are minimal in structure and rather easy to create on your own with your own favorite ontology editor.\n",
    "\n",
    "The KBpedia build processes use one core ontology stub, the KBpedia Knowledge Ontology ([KKO](https://kbpedia.org/docs/kko-upper-structure/)) and two supporting stubs for use in building the full KBpedia knowledge graph or individual typologies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview of the Build Process\n",
    "We set up a new directory structure with appropriate starting files as the first activity. The build first starts with a pre-ingest step of checking out input files for proper encoding and other 'cleaning' tests. Upon passing these checks, we are ready to continue with the build.\n",
    "\n",
    "The build process begins by loading the stub. This loaded stub then becomes the target for all subsequent ingest steps.\n",
    "\n",
    "The ingest process has two phases. In the first phase we ingest build files that specify the structural nature of the knowledge graph, in this case, KBpedia. This structural scaffolding consists of, first, class statements, and then object property or data property 'is-a' statements. In the case of classes, the binding predicate is the <code>rdfs:subClassOf</code> property. In the case of properties, it is the <code>rdfs:subPropertyOf</code> property.\n",
    "\n",
    "This phase sets the ***structure*** over which we can reason and infer with the knowledge graph. Thus, we also have the optional steps in this phase to check whether our ingests have been consistent and satisfiable. If the structural scaffolding meets these tests, we are ready for the second phase.\n",
    "\n",
    "The second phase is to bring in the many annotations that we have gathered for the classes and properties. A description and preferred label are requirements for each item. These are best supplemented with alternative labels (synonyms in the broadest sense) and other properties. We can then load either mapping or additional annotation properties should we desire them.\n",
    "\n",
    "These steps are not inviolate. Files that we know are clean can skip the pre-clean steps, for example. Or, we may already have a completed and vetted knowledge graph to which we only want to supplement some information. In other words, the build routines can also be used in different orders and with only partial input sets once we have a working system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps to Prep\n",
    "We will assume that you have already done your offline work to add to or modify your build input files. (As we proceed installment-by-installment during this build discussion we will provide a listing of required files as appropriate.) Depending on the given project, working on these offline build files may actually represent the bulk of your overall efforts. You might be querying outside sources to add to annotations, or changing or adding to your knowledge graph's structure, or trying new top-level ontologies, etc., etc.\n",
    "\n",
    "Once you deem this offline work to be complete, you need to do some prep to support the new build process (which in the simplest case are the extraction files we just discussed in this **CWPK** series). Your first task is to create a new skeletal directory structure under a new version parent, similar to what is shown in *Figure 2* in the prior [**CWPK #37**](https://www.mkbergman.com/2376/cwpk-37-organizing-the-code-base/) installment. One way to avoid typing in all new directory names is to copy a prior version directory, copy it to the new version location, and then delete irrelevant files. (Further, if you know you may do this multiple times, you may then copy this shell structure for later use for subsequent versions.)\n",
    "\n",
    "You then need to copy over all of the prior stub files from the prior version to the new 'stub' directory. Depending on what you have been doing locally, you may need to make further changes to mirror your needed work preferences.\n",
    "\n",
    "Each stub file then needs to be brought into an ontology editor (Protégé, of course, in our case) and updated for new version number, as this diagram indicates:\n",
    "  \n",
    "<div class=\"center-image\">\n",
    "\n",
    "<img src=\"files/stub-files-1.png\" title=\"Making Version Changes to KKO\" width=\"800\" alt=\"Making Version Changes to KKO\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 1: Making Version Changes to KKO\n",
    "\n",
    "</div> \n",
    "\n",
    "Note that every ontology has a base [IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier), and you should update the reference or version number (<code>http://kbpedia.org/kbpedia/v250</code> in our case) (**1**) in the ontology URI field. You then need to copy the text under your current <code>owl:versionInfo</code> annotation, and paste it into a new <code>owl:priorVersion</code> (**2**) annotation. You may need to make some minor editing changes to reflect past tense for the prior version. Then, last, you need to update the <code>owl:versionInfo</code> (**3**) annotation. \n",
    "\n",
    "You may, of course, make other ontology metadata changes at this time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KKO: The Core Stub\n",
    "The KKO stub is the core one for the build process. It represents its own standalone ontology, but also is the top-level ontology used by KBpedia.\n",
    "\n",
    "KKO is also the most likely of the three stubs to need modication before a new run. Recall that KKO is organized under three main branches corresponding to the [universal categories](https://en.wikipedia.org/wiki/Categories_(Peirce)) of [Charles Sanders Peirce](https://en.wikipedia.org/wiki/Charles_Sanders_Peirce). Two of the branches, <code>Monads</code> and <code>Particulars</code>, do not participate in a KBpedia build. (Though future version releases of KKO may affect these branches, in which case the KKO stub should be updated.) But the third branch, <code>Generals</code>, is very much involved in a KBpedia build. All roots (parents) of KBpedia's typologies tie-in under the <code>Generals</code> branch.\n",
    "\n",
    "You will need, then, to make changes to the <code>Generals</code> of KKO prior to starting a build if any of these conditions is met:\n",
    "\n",
    "1. You are dropping or removing any typologies or <code>SuperTypes</code>\n",
    "1. You are adding any typologies or <code>SuperTypes</code>.\n",
    "\n",
    "If you are only modifying a typology, you need not change KKO. Loading the modified typology during the full build process will accomplish this modification.\n",
    "\n",
    "Like the other two stubs, you also need to make sure you have updated your version references. As distributed with *cowpoke* as part of these **CWPK** installments, here is the KKO stub as used in this project (remember, to see the file chose Run from the notebook menu or press <code>shift+enter</code> when highlighting the cell:\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Note:</strong> You may obtain the three 'stub' files used in this installment from <code>https://github.com/Cognonto/CWPK/tree/master/sandbox/builds/stubs</code>. Make sure and use the ones with the <code>*.owl</code> extension.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r'C:\\1-PythonProjects\\kbpedia\\v300\\build_ins\\stubs\\kko.owl', 'r', encoding='utf8') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The KBpedia Stub\n",
    "The KBpedia stub is the 'umbrella' above the entire project. It incorporates the KKO stub, plus is the general target for all subsequent build steps in the full-build process. When looked at in code view, as the file below shows, this 'umbrella' is rather sparse. However, if you are to look at it in, say, Protégé, then you will also see all of KKO due to its being imported. \n",
    "\n",
    "Again, the KBpedia stub should have its version updated prior to a new version build:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r'C:\\1-PythonProjects\\kbpedia\\v300\\build_ins\\stubs\\kbpedia_rc_stub.owl', 'r', encoding='utf8') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Typology Stub\n",
    "The typology stub is the simplest of the three. Its use is merely to provide a 'header' sufficient for loading an individual typology into an editor such as Protégé.\n",
    "\n",
    "However, despite being listed last, it is the typology stub we will first work with in developing our build routines, because it is our simplest possible starting point. Again, assuming you have made your version updates, here is the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r'C:\\1-PythonProjects\\kbpedia\\v300\\build_ins\\stubs\\typology_stub.owl', 'r', encoding='utf8') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so our stubs are now updated and set up. We are ready to begin some ingest coding . . . .\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}