cwpk-56-graph-visualization-extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#56: Graph Visualization and Extraction\n",
    "=======================================\n",
    "\n",
    "Working Examples are Not so Easy to Come By\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "Today's installment in our [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series is a great example of how impressive uses of [Python](https://en.wikipedia.org/wiki/Python_(programming_language)) can be matched with frustrations over how we get there and whether our hoped-for desires can be met. The case study we tackle in this installment is visualization of the large-scale [KBpedia](https://kbpedia.org/) graph. With nearly 60,000 nodes and about a quarter of a million edge configurations, our KBpedia graph is hardly a toy example. Though there are certainly larger graphs out there, once we pass about 10,000 nodes we enter difficult territory for Python as a performant language. We'll cover this topic and more in this installment.\n",
    "\n",
    "Normally, what one might encounter online regarding graph visualization with Python begins with a simple example, which then goes on to discuss how to stage the input data and then make modifications to the visualization outputs. This approach does not work so well, however, when our use case scales up to the size of KBpedia. Initial toy examples do not provide good insight how the various Python visualization packages may operate at larger scales. So, while we can look at example visualizations and example code showing how to expose options, in the end whether we can get the package to perform requires us to install and test it. Since our review time is limited, and we have to in the end produce working code, we need a pretty efficient process of identifying, screening, and then selecting options.\n",
    "\n",
    "Our desires for a visualization package thus begin with the ability to handle large graphs, including graph analytic components in addition to visualization, compatibility with our Jupyter Notebook interactive environment, ease-of-learning and -implementation, and hopefully attractive rendering of the final graph. From a graph visualizatioin standpoint, some of our desires include:\n",
    "\n",
    "- Attractive outputs\n",
    "- Ability to handle a large graph with acceptable rendering speed\n",
    "- Color coding of nodes by SuperType\n",
    "- Varying node sizes depending on the importance (in-degree) of the node\n",
    "- Control over the graphical elements of the display (edge and node styles)\n",
    "- Perhaps some interactivity such as panning and zooming and tooltips when hovering over nodes, and\n",
    "- A choice of a variety of graph layout options to gauge which best displays the graph.\n",
    "\n",
    "Preferably, whatever packages work best for these criteria also have robust supporting capabilities within the Python data science ecosystem. To test these criteria, of course, it is first necessary to stage our graph in an input form that can be rendered by the visualization package. This staging of the graph data is thus where we obviously begin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation\n",
    "\n",
    "Given the visualization criteria above, we know that we want to produce an input file for a directed graph that consists of individual rows for each 'edge' (connection between two nodes) consisting of a source node (subclass), a target node to which it points (the parent node), a SuperType for the parent node (and possibly its matching rendering color), and a size for the parent node as measured by its number of direct subclasses. This should give us a tabular graph definition file with rows corresponding to the individual edges (subclasses of each parent) by sometime like these columns:\n",
    "\n",
    "<pre>\n",
    "  RC1(source)     RC2(target)     No Subclasses(weight)     SuperType     ST color  \n",
    "</pre>\n",
    "\n",
    "Different visualization packages may want this information in slightly different order, but that may be readily accomplished by shifting the order of written output.\n",
    "\n",
    "Another thing I wanted to do was to order the SuperTypes according to the order of the universal categories as shown by <code>kko-demo.n3</code>. This will tend to keep the color ordering more akin to the ordering of the universal categories (see further [**CWPK #8**](https://www.mkbergman.com/2335/cwpk-8-getting-familiar-with-the-kbpedia-structure/) for a description of these universal categories).\n",
    "\n",
    "It is pretty straightforward to generate a listing of hex color values from an existing large-scale bokeh color palette, as we used in the last **CWPK** installment. First, we count the number of categories in our use case (72 for the STs). Second, we pick one of the large (256) [bokeh palettes](https://docs.bokeh.org/en/latest/docs/reference/palettes.html). We then generate a listing of 72 hex colors from the palette, which we can then relate to the ST categories: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.palettes import Plasma256, linear_palette\n",
    "\n",
    "linear_palette(Plasma256,72)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reverse the order to go from lighter to darker, and then correlate the hex values to the SuperTypes listed in universal category order. Our resulting custom color dictionary then becomes:\n",
    "\n",
    "<pre>\n",
    "cmap = {'Constituents'           : '#EFF821',\n",
    "        'NaturalPhenomena'       : '#F2F126',\n",
    "        'TimeTypes'              : '#F4EA26',\n",
    "        'Times'                  : '#F6E525',\n",
    "        'EventTypes'             : '#F8DF24',\n",
    "        'SpaceTypes'             : '#F9D924',\n",
    "        'Shapes'                 : '#FBD324',\n",
    "        'Places'                 : '#FCCC25',\n",
    "        'AreaRegion'             : '#FCC726',\n",
    "        'LocationPlace'          : '#FDC128',\n",
    "        'Forms'                  : '#FDBC2A',\n",
    "        'Predications'           : '#FDB62D',\n",
    "        'AttributeTypes'         : '#FDB030',\n",
    "        'IntrinsicAttributes'    : '#FCAC32',\n",
    "        'AdjunctualAttributes'   : '#FCA635',\n",
    "        'ContextualAttributes'   : '#FBA238',\n",
    "        'RelationTypes'          : '#FA9C3B',\n",
    "        'DirectRelations'        : '#F8963F',\n",
    "        'CopulativeRelations'    : '#F79241',\n",
    "        'ActionTypes'            : '#F58D45',\n",
    "        'MediativeRelations'     : '#F48947',\n",
    "        'SituationTypes'         : '#F2844B',\n",
    "        'RepresentationTypes'    : '#EF7E4E',\n",
    "        'Denotatives'            : '#ED7B51',\n",
    "        'Indexes'                : '#EB7654',\n",
    "        'Associatives'           : '#E97257',\n",
    "        'Manifestations'         : '#E66D5A',\n",
    "        'NaturalMatter'          : '#E46A5D',\n",
    "        'AtomsElements'          : '#E16560',\n",
    "        'NaturalSubstances'      : '#DE6064',\n",
    "        'Chemistry'              : '#DC5D66',\n",
    "        'OrganicMatter'          : '#D8586A',\n",
    "        'OrganicChemistry'       : '#D6556D',\n",
    "        'BiologicalProcesses'    : '#D25070',\n",
    "        'LivingThings'           : '#CF4B74',\n",
    "        'Prokaryotes'            : '#CC4876',\n",
    "        'Eukaryotes'             : '#C8447A',\n",
    "        'ProtistsFungus'         : '#C5407D',\n",
    "        'Plants'                 : '#C13C80',\n",
    "        'Animals'                : '#BD3784',\n",
    "        'Diseases'               : '#BA3487',\n",
    "        'Agents'                 : '#B62F8B',\n",
    "        'Persons'                : '#B22C8E',\n",
    "        'Organizations'          : '#AE2791',\n",
    "        'Geopolitical'           : '#A92395',\n",
    "        'Symbolic'               : '#A51F97',\n",
    "        'Information'            : '#A01B9B',\n",
    "        'AVInfo'                 : '#9D189D',\n",
    "        'AudioInfo'              : '#9713A0',\n",
    "        'VisualInfo'             : '#9310A1',\n",
    "        'WrittenInfo'            : '#8E0CA4',\n",
    "        'StructuredInfo'         : '#8807A5',\n",
    "        'Artifacts'              : '#8405A6',\n",
    "        'FoodDrink'              : '#7E03A7',\n",
    "        'Drugs'                  : '#7901A8',\n",
    "        'Products'               : '#7300A8',\n",
    "        'PrimarySectorProduct'   : '#6D00A8',\n",
    "        'SecondarySectorProduct' : '#6800A7',\n",
    "        'TertiarySectorService'  : '#6200A6',\n",
    "        'Facilities'             : '#5E00A5',\n",
    "        'Systems'                : '#5701A4',\n",
    "        'ConceptualSystems'      : '#5101A2',\n",
    "        'Concepts'               : '#4C02A1',\n",
    "        'TopicsCategories'       : '#45039E',\n",
    "        'LearningProcesses'      : '#40039C',\n",
    "        'SocialSystems'          : '#3A049A',\n",
    "        'Society'                : '#330497',\n",
    "        'EconomicSystems'        : '#2D0494',\n",
    "        'Methodeutic'            : '#250591',\n",
    "        'InquiryMethods'         : '#1F058E',\n",
    "        'KnowledgeDomains'       : '#15068A',\n",
    "        'EmergentKnowledge'      : '#0C0786',\n",
    "        }\n",
    "</pre>        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have all of the input pieces to complete our graph dataset. Fortunately, we had already developed a routine in [**CWPK #49**](https://www.mkbergman.com/2395/cwpk-49-mapping-external-sources/) for generating an output listing from our [owlready2](https://owlready2.readthedocs.io/en/latest/intro.html) representation of KBpedia. We begin by loading up our necessary packages for working with this information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cowpoke.__main__ import *\n",
    "from cowpoke.config import *\n",
    "from owlready2 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we follow the same configuration setup approach that we have developed for prior extractions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "### KEY CONFIG SETTINGS (see build_deck in config.py) ###             \n",
    "# 'kb_src'        : 'standard'                                        # Set in master_deck\n",
    "# 'loop_list'     : kko_order_dict.values(),                          # Note 1   \n",
    "# 'base'          : 'C:/1-PythonProjects/kbpedia/v300/build_ins/mappings/',              \n",
    "# 'ext'           : '.csv',                                         \n",
    "# 'out_file'      : 'C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.csv',\n",
    "\n",
    "def graph_extractor(**extract_deck):\n",
    "    print('Beginning graph structure extraction . . .')\n",
    "    loop_list = extract_deck.get('loop_list')\n",
    "    loop = extract_deck.get('loop')\n",
    "    class_loop = extract_deck.get('class_loop')\n",
    "    base = extract_deck.get('base')\n",
    "    ext = extract_deck.get('ext')\n",
    "    \n",
    "    # Note 2\n",
    "    parent_set = ['kko.SocialSystems','kko.Products','kko.Methodeutic','kko.Eukaryotes',\n",
    "              'kko.ConceptualSystems','kko.AVInfo','kko.Systems','kko.Places',\n",
    "              'kko.OrganicChemistry','kko.MediativeRelations','kko.LivingThings',\n",
    "              'kko.Information','kko.CopulativeRelations','kko.Artifacts','kko.Agents',\n",
    "              'kko.TimeTypes','kko.Symbolic','kko.SpaceTypes','kko.RepresentationTypes',\n",
    "              'kko.RelationTypes','kko.OrganicMatter','kko.NaturalMatter',\n",
    "              'kko.AttributeTypes','kko.Predications','kko.Manifestations',\n",
    "              'kko.Constituents']\n",
    "\n",
    "    if loop is not 'class_loop':\n",
    "        print(\"Needs to be a 'class_loop'; returning program.\")\n",
    "        return\n",
    "    header = ['target', 'source', 'weight', 'SuperType']\n",
    "    out_file = extract_deck.get('out_file')\n",
    "    cur_list = []\n",
    "    with open(out_file, mode='w', encoding='utf8', newline='') as output:                                           \n",
    "        csv_out = csv.writer(output)\n",
    "        csv_out.writerow(header)    \n",
    "        for value in loop_list:\n",
    "            print('   . . . processing', value)\n",
    "            s_set = []\n",
    "            root = eval(value)\n",
    "            s_set = root.descendants()\n",
    "            frag = value.replace('kko.','')\n",
    "            for s_item in s_set:\n",
    "                child_set = list(s_item.subclasses())\n",
    "                count = len(list(child_set))\n",
    "                \n",
    "# Note 3                \n",
    "                if value not in parent_set:\n",
    "                    for child_item in child_set:\n",
    "                        s_rc = str(s_item)\n",
    "                        child = str(child_item)\n",
    "                        new_pair = s_rc + child\n",
    "                        new_pair = str(new_pair)\n",
    "                        cur_list.append(new_pair)\n",
    "                        s_rc = s_rc.replace('rc.','')\n",
    "                        child = child.replace('rc.','')\n",
    "                        row_out = (s_rc,child,count,frag)\n",
    "                        csv_out.writerow(row_out)\n",
    "                elif value in parent_set:\n",
    "                    for child_item in child_set:\n",
    "                        s_rc = str(s_item)\n",
    "                        child = str(child_item)\n",
    "                        new_pair = s_rc + child\n",
    "                        new_pair = str(new_pair)\n",
    "                        if new_pair not in cur_list:\n",
    "                            cur_list.append(new_pair)\n",
    "                            s_rc = s_rc.replace('rc.','')\n",
    "                            child = child.replace('rc.','')\n",
    "                            row_out = (s_rc,child,count,frag)\n",
    "                            csv_out.writerow(row_out)\n",
    "                        elif new_pair in cur_list:\n",
    "                            continue\n",
    "        output.close()         \n",
    "        print('Processing is complete . . .')\n",
    "graph_extractor(**extract_deck)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This routine is pretty consistent with the prior version except for a few changes. First, the order of the STs in the input dictionary has changed **(1)**, consistent with the order of the universal categories and with lower categories processed first. Since source-target pairs are only processed once, this ordering means duplicate assignments are always placed at their lowest point in the KBpedia hierarchy. Second, to help enforce this ordering, parental STs are separately noted **(2)** and then processed to skip source-target pairs that had been previously processed **(3)**.\n",
    "\n",
    "To see the output from this routine (without hex colors yet begin assigned by ST), run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.csv')\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation of Large-scale Graph Visualization Options\n",
    "In my prior work with large-scale graph visualizations, I have used [Cytoscape](https://cytoscape.org/) and [written about it](https://www.mkbergman.com/?p=415) (2008), as well as [Gephi](https://gephi.org/) and [written about it](https://www.mkbergman.com/968/a-new-best-friend-gephi-for-large-scale-networks/) (2011). Though my most recent efforts have preferred Gephi, neither is written in Python and both are rather cumbersome to set up for a given visualization.\n",
    "\n",
    "My interest here is either a pure Python option or one that has a ready Python wrapper. The Python visualization project, [PyViz](https://pyviz.org/tools.html#graphs-networks) provides a great listing of the options available. Since some are less capable than others, I found Timothy Lin's [benchmark comparisons of network packages](https://www.timlrx.com/2020/05/10/benchmark-of-popular-graph-network-packages-v2/) to be particularly valuable, and I have limited my evaluation to the packages he lists.\n",
    "\n",
    "The first package is [NetworkX](https://networkx.github.io/), which is written solely in Python and is the granddaddy of network analysis packages in the language. We will use it as our starting baseline.\n",
    "\n",
    "Lin also compares [SNAP](http://snap.stanford.edu/), [NetworkKit](https://networkit.github.io/), [igraph](https://igraph.org/), [graph-tool](https://graph-tool.skewed.de/), and [Lightgraphs](https://github.com/JuliaGraphs/LightGraphs.jl). I looked in detail at all of these packages except for Lightgraphs, which is written in [Julia](https://en.wikipedia.org/wiki/Julia_(programming_language)) and has no Python wrapper.\n",
    "\n",
    "Lin's comparisons showed NetworkX to be, by far, the slowest and least performant of all of the packages tested. However, NetworkX has a rich ecosystem around it and much use and documentation. As such, it appears to be a proper baseline for the testing. \n",
    "\n",
    "All of the remaining candidates implement their core algorithms in C or C++ for performance reasons, though Python wrappers are provided. Based on Lin's benchmark results and visualization examples online, my initial preference was for graph-tool, followed possibly by NetworkKit. SNAP had only recently been updated by Stanford, and igraph initially appeared as more oriented to R than Python. \n",
    "\n",
    "So, my plan was to first test NetworkX, and then try to implement one or more of the others if not satisfied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First NetworkX Visualizations\n",
    "With our data structure now in place for the entire KBpedia, it was time to attempt some visualizations using NetworkX. Though primarily an analysis package, NetworkX does support some graph visualizations, principally through [graphViz](https://graphviz.org/) or [matplotlib](https://matplotlib.org/). In this instance, we use the matplotlib option, using the spring layout. \n",
    "\n",
    "Note in the routine below, which is fairly straightforward in nature, I inserted a print statement to separate out the initial graph construction step from graph rendering. The graph construction takes mere seconds, while rendering the graph took multiple hours.\n",
    "\n",
    "<div class=\"boxPinkDotted\"><span style=\"font-weight: bold;\">WARNING!:</span> The cell below takes tens of minutes to hours to run. Please do not execute unless you are able to let this run in the background.</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df = pd.read_csv('C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.csv')\n",
    "Graphtype = nx.DiGraph()\n",
    "G = nx.from_pandas_edgelist(df, edge_attr='weight', create_using=Graphtype)\n",
    "print('Graph construction complete.')\n",
    "pos = nx.spring_layout(G,scale=1)\n",
    "\n",
    "nx.draw(G,pos, with_labels=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center-image\">\n",
    "\n",
    "<img src=\"files/large-scale-graph-1.png\" title=\"Baseline KBpedia Visualization with Labels\" height=\"800\" alt=\"Baseline KBpedia Visualization with Labels\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 1: Baseline KBpedia Visualization with Labels\n",
    "\n",
    "</div>\n",
    "\n",
    "Since the labels render this uninterpretable, we tried the same approach without labels.\n",
    "\n",
    "<div class=\"boxPinkDotted\"><span style=\"font-weight: bold;\">WARNING!:</span> The cell below takes tens of minutes to hours to run. Please do not execute unless you are able to let this run in the background.</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,6))\n",
    "nx.draw(G,pos, with_labels=False)\n",
    "plt.show() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center-image\">\n",
    "\n",
    "<img src=\"files/large-scale-graph-2.png\" title=\"Baseline KBpedia Visualization without Labels\" height=\"800\" alt=\"Baseline KBpedia Visualization without Labels\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 2: Baseline KBpedia Visualization without Labels\n",
    "\n",
    "</div>\n",
    "\n",
    "This view is hardly any better.\n",
    "\n",
    "Given the lengthy times it took to generate these visualizations, I decided to return to our candidate list and try other packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More Diligence on NetworkX Alternatives\n",
    "If you recall, my first preferred option was [graph-tool](https://graph-tool.skewed.de/) because of its reported speed and its wide variety of graph algorithms and layouts. The problem with graph-tool, as with the other alternatives, is that a C++ compiler is required, along with other dependencies. After extensive research online, I was unable to find an example of a Windows instance that was able to install graph-tool and its dependencies successfully.\n",
    "\n",
    "I turned next to [NetworKit](https://networkit.github.io/). Though visualization choices are limited in comparison to the other C++ alternatives, this package has clearly been designed for network analysis and has a strong basis in data science. This package *does* offer a [Windows 10 installation path](https://networkit.github.io/get_started.html), but one that suggests adding a virtual [Linux subsystem layer](https://docs.microsoft.com/en-us/windows/wsl/about) to Windows. Again, I deemed this to be more complexity than a single visualization component warranted.\n",
    "\n",
    "With [igraph](https://igraph.org/), I went through the steps of attempting an install, but clearly was also missing dependencies and using it kept killing the kernel in Jupyter Notebook. Again, perhaps with more research and time, I could have gotten this package to work, but it seemed to impose too much effort for a Windows environment for the possible reward.\n",
    "\n",
    "Lastly, given these difficulties, and the fact that [SNAP](http://snap.stanford.edu/) had been under less active development in recent years, I chose not to pursue this option further.\n",
    "\n",
    "In the end, I think with some work I could have figured out how to get igraph to install, and perhaps NetworKit as well. However, as a demo candidate being chosen for Python newbies, it struck me that no reader of this series would want to spend the time jumping through such complicated hoops in order to get a C++ option running. Perhaps in a production environment these configuration efforts may be warranted. However, for our teaching purposes, I judged trying to get a C++ installation on Windows as not worth the effort. I do believe this leaves an opening for one or more developers of these packages to figure out a better installation process for Windows. But that is a matter for the developers, not for a newbie Python user such as me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Faster Testing of NetworkX with the Upper KBpedia\n",
    "So these realizations left me with the NetworkX alternative as the prime option. Given the time it took to render the full KBpedia, I decided to use the smaller upper structure of KBpedia to work out the display and rendering options before applying it to the full KBpedia.\n",
    "\n",
    "I thus created offline a smaller graph dataset that consisted of the 72 SuperTypes and all of their direct resource concept (RC) children. You can inspect this dataset (<code>df_kko</code>) in a similar matter to the snippet noted above for the full KBpedia (<code>df</code>).\n",
    "\n",
    "Also, to overcome some of the display limitations of the standard NetworkX renderers, I recalled that the [HoloViews](http://holoviews.org/) package used in the last installment also had an optional component, [hvPlot](https://hvplot.holoviz.org/), designed specifically to work with NetworkX graph layouts and datasets. The advantage of this approach is that we would gain interactivity and some of the tooltips when hovering over nodes on the graph.\n",
    "\n",
    "I literally spent days trying to get all of these components to work together in terms of my desired visualizations where SuperType nodes (and their RCs) would be colored differently and the size of the nodes would be dependent on the number of subclasses. Alas, I was unable to get these desired options to work. In part, I think this is because of the immaturity of the complete ecosystem. In part, it is also due to my lack of Python skills and the fact that the entire chain of NetworkX &rarr; bokeh &rarr; HoloViews &rarr; hvPlot each provides its own syntax and optional functions for making visualization tweaks. It is hard to know what governs what and how to get all of the parts to work together nicely.\n",
    "\n",
    "Fortunately, with the smaller input graph set, it is nearly instantaneous to make and see changes in real time. Despite the number of tests applied, the resulting code is fairly small and straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import holoviews as hv\n",
    "import networkx as nx\n",
    "import hvplot.networkx as hvnx\n",
    "from holoviews import opts\n",
    "from bokeh.models import HoverTool\n",
    "\n",
    "hv.extension('bokeh')\n",
    "\n",
    "# Load the data\n",
    "# on MyBinder: https://github.com/Cognonto/CWPK/blob/master/sandbox/extracts/data/kko_graph_specs.csv\n",
    "df_kko = pd.read_csv('C:/1-PythonProjects/kbpedia/v300/extractions/data/kko_graph_specs.csv')\n",
    "\n",
    "# Define the graph\n",
    "G_kko = nx.from_pandas_edgelist(df_kko, 'source', 'target', ['Subs', 'SuperType', 'Color'], create_using=nx.DiGraph())\n",
    "\n",
    "pos = nx.spring_layout(G_kko, k=0.4, iterations=70)\n",
    "\n",
    "hvnx.draw(G_kko, pos, node_color='#D6556D', alpha=0.65).opts(node_size=10, width=950, height=950, \n",
    "                           edge_line_width=0.2, tools=['hover'], inspection_policy='edges')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final Large-scale Visualization with NetworkX\n",
    "With these tests of the smaller graph complete, we are now ready to produce the final visualization of the full KBpedia graph. Though the modified code is presented below, and does run, we actually use a captured figure below the code listing to keep this page size manageable.\n",
    "\n",
    "<div class=\"boxPinkDotted\"><span style=\"font-weight: bold;\">WARNING!:</span> The cell below takes more than three hours to run on our standard laptop and creates a page file of 36 MB. Please do not execute unless you are able to let this run in the background.</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import holoviews as hv\n",
    "import networkx as nx\n",
    "from holoviews import opts\n",
    "import hvplot.networkx as hvnx\n",
    "#from bokeh.models import HoverTool\n",
    "\n",
    "hv.extension('bokeh')\n",
    "\n",
    "df = pd.read_csv('C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.csv')\n",
    "Graphtype = nx.DiGraph()\n",
    "G = nx.from_pandas_edgelist(df, edge_attr='weight', create_using=Graphtype)\n",
    "print('Graph construction complete.')\n",
    "\n",
    "pos = nx.spring_layout(G, k=0.4, iterations=70)\n",
    "\n",
    "hvnx.draw(G, pos, node_color='#FCCC25', alpha=0.65).opts(node_size=5, width=950, height=950, \n",
    "                           edge_line_width=0.1)\n",
    "\n",
    "#, tools=['hover'], inspection_policy='edges'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center-image\">\n",
    "\n",
    "<img src=\"files/full-kbpedia-graph.png\" title=\"Full-sized KBpedia Graph\" width=\"940\" alt=\"Full-sized KBpedia Graph\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 3: Full-sized KBpedia Graph\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Graphing Options\n",
    "I admit I am disappointed and frustrated with available Python options to capture the full scale of KBpedia. The pure Python options are unacceptably slow. Options that promise better performance and a wider choice of layouts and visualizations are difficult, if not impossible, to install on Windows. Of all of the options directly tested, none allowed me (at least with my limited Python skill level) to vary node colors or node sizes by in-degrees. \n",
    "\n",
    "On the other hand, we began to learn some of the robust NetworkX package and will have occasion to investigate it further in relation to network analysis (**CWPK #61**). Further, as a venerable package, NetworkX offers a wide spectrum of graph data formats that it can read and write. We can export our graph specifications to a number of forms that perhaps will provide better visualization choices. As examples, here are ways to specify two of NetworkX's formats, both of which may be used as inputs to the Gephi package. (For more about Gephi and Cytoscape as options here, see the initial links at the beginning of this installment.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import networkx as nx\n",
    "\n",
    "df = pd.read_csv('C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.csv')\n",
    "Graphtype = nx.DiGraph()\n",
    "G = nx.from_pandas_edgelist(df, edge_attr='weight', create_using=Graphtype)\n",
    "\n",
    "nx.write_gexf(G, 'C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.gexf')\n",
    "\n",
    "print('Gephi file complete.')\n",
    "\n",
    "nx.write_gml(G, 'C:/1-PythonProjects/kbpedia/v300/extractions/data/graph_specs.gml')\n",
    "\n",
    "print('GML file complete.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "This section presents the significant amount of material reviewed in order to make the choices for use in this present **CWPK** installment. \n",
    "\n",
    "First, it is possible to get online help for most options to be tested. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hv.help(hvnx.draw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, here are some links related to options investigated for this installment, some tested, some not:\n",
    "\n",
    "#### NetworkX\n",
    "- https://networkx.github.io/\n",
    "  - - https://networkx.github.io/documentation/stable/\n",
    "- https://networkx.github.io/documentation/stable/tutorial.html (start here). Specific sub-topics of key relevance include:\n",
    "    - https://networkx.github.io/documentation/stable/reference/algorithms/index.html\n",
    "    - https://networkx.github.io/documentation/stable/reference/drawing.html\n",
    "    - https://networkx.github.io/documentation/stable/reference/readwrite/index.html\n",
    "- http://holoviews.org/user_guide/Network_Graphs.html    \n",
    "- https://docs.bokeh.org/en/latest/docs/user_guide/tools.html#hovertool    \n",
    "- https://networkx.github.io/documentation/stable/_downloads/networkx_reference.pdf (impressive 700+ pages in documentation).\n",
    "\n",
    "These same references above are also provided for the 'latest' version.\n",
    "\n",
    "#### graph-tool\n",
    "- https://graph-tool.skewed.de/static/doc/index.html\n",
    "  - https://graph-tool.skewed.de/static/doc/draw.html\n",
    "- https://git.skewed.de/count0/graph-tool/-/wikis/installation-instructions\n",
    "- http://ryancompton.net/2014/10/05/graph-tools-visualization-is-pretty-good/.\n",
    "\n",
    "#### NetworKit\n",
    "- https://networkit.github.io/\n",
    "  - https://github.com/networkit/networkit/blob/Dev/notebooks/User-Guide.ipynb (works in Notebook)\n",
    "- https://stackoverflow.com/questions/tagged/networkit.\n",
    "\n",
    "#### deepgraph\n",
    "- https://deepgraph.readthedocs.io/en/latest/\n",
    "- https://arxiv.org/pdf/1604.00971.pdf\n",
    "- https://github.com/deepgraph/deepgraph.\n",
    "\n",
    "#### nxviz\n",
    "- visualization package for NetworkX\n",
    "- https://github.com/ericmjl/nxviz\n",
    "- https://nxviz.readthedocs.io/en/latest/\n",
    "    - https://nxviz.readthedocs.io/en/latest/modules.html\n",
    "- https://nxviz.readthedocs.io/en/latest/usage.html\n",
    "- Perhaps limited plot types.\n",
    "\n",
    "#### Netwulf\n",
    "- https://github.com/benmaier/netwulf/\n",
    "- https://netwulf.readthedocs.io/en/latest/\n",
    "- depends on NetworkX\n",
    "- save as PDF\n",
    "- appears to be d3-force layout only\n",
    "- interactive styling and filtering.\n",
    "\n",
    "#### igraph\n",
    "- https://igraph.org/redirect.html\n",
    "- https://igraph.org/2020/02/14/igraph-0.8.0-python.html \n",
    "- https://github.com/igraph/python-igraph\n",
    "   - https://igraph.org/python/\n",
    "   - https://igraph.org/python/doc/igraph-module.html\n",
    "   - https://igraph.org/python/doc/python-igraph.pdf\n",
    "- https://igraph.org/python/doc/tutorial/tutorial.html\n",
    "- seems a bit more oriented to R.\n",
    "\n",
    "#### SNAP\n",
    "- http://snap.stanford.edu/\n",
    "- https://snap.stanford.edu/snap/\n",
    "- http://snap.stanford.edu/snappy/index.html.\n",
    "\n",
    "\n",
    "#### pygraphistry\n",
    "- https://github.com/graphistry/pygraphistry\n",
    "  - free API, then costs https://www.graphistry.com/.\n",
    "  \n",
    "#### ipycytoscape\n",
    "- https://blog.jupyter.org/interactive-graph-visualization-in-jupyter-with-ipycytoscape-a8828a54ab63\n",
    "- https://github.com/QuantStack/ipycytoscape/blob/master/examples/DataFrame%20interaction.ipynb\n",
    "- too new, immature\n",
    "- https://github.com/cytoscape/cytoscape-automation/blob/master/for-scripters/Python/advanced-cancer-networks-and-data.ipynb\n",
    "- https://py2cytoscape.readthedocs.io/en/latest/ (py2cytocape depends on python-igraph).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site. The <em>cowpoke</em> Python <a href=\"https://github.com/Cognonto/cowpoke\">code listing covering the series</a> is also available from GitHub.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}