example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "This example demonstrates how to scan query history from a data warehouse and save it in the data lineage app. The app automatically parses and extracts data lineage from the queries.\n",
    "\n",
    "The example consists of the following sequence of operations:\n",
    "\n",
    "* Start docker containers containing a demo. Refer to [docs](https://tokern.io/docs/data-lineage/installation) for detailed instructions on installing demo-wikimedia.\n",
    "* Scan and send queries from query history to data lineage app.\n",
    "* Visualize the graph by visiting Tokern UI.\n",
    "* Analyze the graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installation\n",
    "\n",
    "This demo requires wikimedia demo to be running. Start the demo using the following instructions:\n",
    "\n",
    "    # in a new directory run\n",
    "    wget https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/wikimedia-demo.yml\n",
    "    # or run\n",
    "    curl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/wikimedia-demo.yml -o docker-compose.yml\n",
    "\n",
    "\n",
    "Run docker-compose\n",
    "\n",
    "\n",
    "    docker-compose up -d\n",
    "\n",
    "\n",
    "Verify container are running\n",
    "\n",
    "\n",
    "    docker container ls | grep tokern\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Required configuration for API and wikimedia database network address\n",
    "\n",
    "docker_address = \"http://127.0.0.1:8000\"\n",
    "wikimedia_db = {\n",
    "  \"username\": \"etldev\",\n",
    "  \"password\": \"3tld3v\",\n",
    "  \"uri\": \"tokern-demo-wikimedia\",\n",
    "  \"port\": \"5432\",\n",
    "  \"database\": \"wikimedia\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "# Setup a connection to catalog using the SDK.\n",
    "from data_lineage import Catalog, Scan\n",
    "\n",
    "catalog = Catalog(docker_address)\n",
    "\n",
    "# Register wikimedia datawarehouse with data-lineage app.\n",
    "\n",
    "source = catalog.add_source(name=\"wikimedia\", source_type=\"postgresql\", **wikimedia_db)\n",
    "\n",
    "# Scan the wikimedia data warehouse and register all schemata, tables and columns.\n",
    "scan = Scan(docker_address)\n",
    "job = scan.start(source)\n",
    "\n",
    "# Wait for scan to complete\n",
    "\n",
    "status = \"\"\n",
    "\n",
    "while (status != \"finished\" and status != \"failed\"):\n",
    "    time.sleep(5)\n",
    "    status = scan.get(job[\"id\"])[\"status\"]\n",
    "    print(\"Status is {}\".format(status))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open(\"test/queries.json\", \"r\") as file:\n",
    "    queries = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from data_lineage import Analyze\n",
    "\n",
    "analyze = Analyze(docker_address)\n",
    "\n",
    "for query in queries:\n",
    "    print(query)\n",
    "    analyze.analyze(**query, source=source, start_time=datetime.now(), end_time=datetime.now())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visit [Kedro UI](http://localhost:8000/)\n",
    "\n",
    "![One Task Graph](./full_graph.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}