examples/tutorials/6_apps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. DSMS Apps and Pipelines\n",
    "\n",
    "In this tutorial we see how to create apps and run them manually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1: Setting up\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsms import DSMS, KItem, AppConfig\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now source the environmental variables from an `.env` file and start the DSMS-session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsms = DSMS(env=\"../../.env\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1. Investigating Available Apps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can investigate which apps are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[AppConfig(name=ckan-fetch, specification={'metadata': {'generateName': 'ckan-resource-request-'}}),\n",
       " AppConfig(name=csv_tensile_test, specification={'metadata': {'generateName': 'data2rdf-'}}),\n",
       " AppConfig(name=csv_tensile_test_f2, specification={'metadata': {'generateName': 'data2rdf-'}}),\n",
       " AppConfig(name=excel_notched_tensile_test, specification={'metadata': {'generateName': 'data2rdf-'}}),\n",
       " AppConfig(name=excel_shear_test, specification={'metadata': {'generateName': 'data2rdf-'}}),\n",
       " AppConfig(name=excel_tensile_test, specification={'metadata': {'generateName': 'data2rdf-'}}),\n",
       " AppConfig(name=ternary-plot, specification={'metadata': {'generateName': 'ckan-tenary-app-'}})]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dsms.app_configs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Create a new app config and apply it to a KItem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.1 Arbitrary python code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be defined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.2 - Data2RDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2.2.1 Prepare app and its config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we would like to upload some csv with some arbitrary data and describe it through an RDF. This will give us the opportunity to harmonize the entities of the data file through ontological concepts and allow us to convert values of the data frame columns in to any compatible unit we would like to have.\n",
    "\n",
    "Fist of all, let us define the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = \"\"\"A,B,C\n",
    "1.2,1.3,1.5\n",
    "1.7,1.8,1.9\n",
    "2.0,2.1,2.3\n",
    "2.5,2.6,2.8\n",
    "3.0,3.2,3.4\n",
    "3.6,3.7,3.9\n",
    "4.1,4.3,4.4\n",
    "4.7,4.8,5.0\n",
    "5.2,5.3,5.5\n",
    "5.8,6.0,6.1\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also give the config a defined name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "configname = \"testapp2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a next step, we want to create a new app specification. The specification is following the definition of an [**Argo Workflow**](https://argo-workflows.readthedocs.io/en/latest/). The workflow shall trigger a pipeline from a docker image with the  [**Data2RDF package**](https://data2rdf.readthedocs.io/en/latest/). \n",
    "\n",
    "The image has already been deployed on the k8s cluster of the DSMS and the workflow template with the name `dsms-data2rdf` has been implemented previously. Hence we only need to configure our pipeline for our data shown above, which we would like to upload and describe through an RDF.\n",
    "\n",
    "For more details about the data2rdf package, please refer to the documentation of Data2RDF mentioned above.\n",
    "\n",
    "The parameters of the app config are defining the inputs for our Data2RDF pipeline. This e.g. are: \n",
    "\n",
    "* the parser kind (`csv` here)\n",
    "* the time series header length (`1` here)\n",
    "* the metadata length (`0` here)\n",
    "* the time series separator (`,` here)\n",
    "* the log level (`DEBUG` here)\n",
    "* the mapping \n",
    "    * `A` is the test time and has a unit in seconds\n",
    "    * `B` is the standard force in kilonewtons\n",
    "    * `C` is the absolut cross head travel in millimeters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = [\n",
    "    {\"name\": \"parser\", \"value\": \"csv\"},\n",
    "    {\"name\": \"time_series_header_length\", \"value\": 1},\n",
    "    {\"name\": \"metadata_length\", \"value\": 0},\n",
    "    {\"name\": \"time_series_sep\", \"value\": \",\"},\n",
    "    {\"name\": \"log_level\", \"value\": \"DEBUG\"},\n",
    "    {\n",
    "        \"name\": \"mapping\",\n",
    "        \"value\": \"\"\"\n",
    "            [\n",
    "                {\n",
    "                    \"key\": \"A\",\n",
    "                    \"iri\": \"https://w3id.org/steel/ProcessOntology/TestTime\",\n",
    "                    \"unit\": \"s\"\n",
    "                },\n",
    "                {\n",
    "                    \"key\": \"B\",\n",
    "                    \"iri\": \"https://w3id.org/steel/ProcessOntology/StandardForce\",\n",
    "                    \"unit\": \"kN\"\n",
    "                },\n",
    "                {\n",
    "                    \"key\": \"C\",\n",
    "                    \"iri\": \"https://w3id.org/steel/ProcessOntology/AbsoluteCrossheadTravel\",\n",
    "                    \"unit\": \"mm\"\n",
    "                }\n",
    "            ]\n",
    "            \"\"\",\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we add the parameters to our app specification. We assign a prefix `datardf-` which shall generate a new name with some random characters as suffix. The workflow template with the Docker image we want to run is called `dsms-data2rdf` and its `entrypoint` is `execute_pipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define app specification\n",
    "specification = {\n",
    "    \"apiVersion\": \"argoproj.io/v1alpha1\",\n",
    "    \"kind\": \"Workflow\",\n",
    "    \"metadata\": {\"generateName\": \"data2rdf-\"},\n",
    "    \"spec\": {\n",
    "        \"entrypoint\": \"execute_pipeline\",\n",
    "        \"workflowTemplateRef\": {\"name\": \"dsms-data2rdf\"},\n",
    "        \"arguments\": {\"parameters\": parameters},\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we instanciate the new app config:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "appspec = AppConfig(\n",
    "    name=configname,\n",
    "    specification=specification,  # this can also be a file path to a yaml file instead of a dict\n",
    "    expose_sdk_config=True,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that the `expose_sdk_config` is important here, since the pipeline needs to be aware of the settings of our platform and our current SDK-session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We commit the new app config:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsms.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we would like to apply the app config to a KItem. The set the `triggerUponUpload` must be set to `True` so that the app is triggered automatically when we upload an attachment.\n",
    "\n",
    "Additionally, we must tell the file extension for which the upload shall be triggered. Here it is `.csv`.\n",
    "\n",
    "We also want to generate a qr code as avatar for the KItem with `avatar={\"include_qr\": True}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "item = KItem(\n",
    "    name=\"my tensile test experiment\",\n",
    "    ktype_id=dsms.ktypes.Dataset,\n",
    "    kitem_apps=[\n",
    "        {\n",
    "            \"executable\": appspec.name,\n",
    "            \"title\": \"data2rdf\",\n",
    "            \"additional_properties\": {\n",
    "                \"triggerUponUpload\": True,\n",
    "                \"triggerUponUploadFileExtensions\": [\".csv\"],\n",
    "            },\n",
    "        }\n",
    "    ],\n",
    "    avatar={\"include_qr\": True},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We commit the KItem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsms.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we add our data with our attachment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "item.attachments = [{\"name\": \"dummy_data.csv\", \"content\": data}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we commit again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsms.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2.2.2 Get results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can verify that the data extraction was successful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KItem(\n",
      "\n",
      "\tname = my tensile test experiment, \n",
      "\n",
      "\tid = 58683a2d-7823-4638-bc8e-91b461afa593, \n",
      "\n",
      "\tktype_id = dataset, \n",
      "\n",
      "\tin_backend = True, \n",
      "\n",
      "\tslug = mytensiletestexperiment-58683a2d, \n",
      "\n",
      "\tannotations = [], \n",
      "\n",
      "\tattachments = [\n",
      "\t\t{\n",
      "\t\t\tname: dummy_data.csv\n",
      "\t\t}\n",
      "\t], \n",
      "\n",
      "\tlinked_kitems = [], \n",
      "\n",
      "\taffiliations = [], \n",
      "\n",
      "\tauthors = [\n",
      "\t\t{\n",
      "\t\t\tuser_id: 7f0e5a37-353b-4bbc-b1f1-b6ad575f562d\n",
      "\t\t}\n",
      "\t], \n",
      "\n",
      "\tavatar_exists = True, \n",
      "\n",
      "\tcontacts = [], \n",
      "\n",
      "\tcreated_at = 2024-08-20 08:04:25.756982, \n",
      "\n",
      "\tupdated_at = 2024-08-20 08:04:25.756982, \n",
      "\n",
      "\texternal_links = [], \n",
      "\n",
      "\tkitem_apps = [\n",
      "\t\t{\n",
      "\t\t\tkitem_app_id: 22,\n",
      "\t\t\texecutable: testapp2,\n",
      "\t\t\ttitle: data2rdf,\n",
      "\t\t\tdescription: None,\n",
      "\t\t\ttags: None,\n",
      "\t\t\tadditional_properties: {triggerUponUpload: True, triggerUponUploadFileExtensions: ['.csv']}\n",
      "\t\t}\n",
      "\t], \n",
      "\n",
      "\tsummary = None, \n",
      "\n",
      "\tuser_groups = [], \n",
      "\n",
      "\tcustom_properties = None, \n",
      "\n",
      "\tdataframe = [\n",
      "\t\t{\n",
      "\t\t\tcolumn_id: 0,\n",
      "\t\t\tname: TestTime\n",
      "\t\t}, \n",
      "\t\t{\n",
      "\t\t\tcolumn_id: 1,\n",
      "\t\t\tname: StandardForce\n",
      "\t\t}, \n",
      "\t\t{\n",
      "\t\t\tcolumn_id: 2,\n",
      "\t\t\tname: AbsoluteCrossheadTravel\n",
      "\t\t}\n",
      "\t], \n",
      "\n",
      "\trdf_exists = True\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And also that the RDF generation was successful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@prefix csvw: <http://www.w3.org/ns/csvw#> .\n",
      "@prefix dcat: <http://www.w3.org/ns/dcat#> .\n",
      "@prefix dcterms: <http://purl.org/dc/terms/> .\n",
      "@prefix ns1: <http://qudt.org/schema/qudt/> .\n",
      "@prefix ns2: <http://xmlns.com/foaf/spec/> .\n",
      "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
      "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
      "\n",
      "<https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/dataset> a dcat:Dataset ;\n",
      "    dcterms:hasPart <https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/tableGroup> ;\n",
      "    dcat:distribution [ a dcat:Distribution ;\n",
      "            dcat:accessURL \"https://bue.materials-data.space/api/knowledge/data_api/58683a2d-7823-4638-bc8e-91b461afa593\"^^xsd:anyURI ;\n",
      "            dcat:mediaType \"http://www.iana.org/assignments/media-types/text/csv\"^^xsd:anyURI ] .\n",
      "\n",
      "<https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/AbsoluteCrossheadTravel> a <https://w3id.org/steel/ProcessOntology/AbsoluteCrossheadTravel> ;\n",
      "    ns1:hasUnit \"http://qudt.org/vocab/unit/MilliM\"^^xsd:anyURI .\n",
      "\n",
      "<https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/StandardForce> a <https://w3id.org/steel/ProcessOntology/StandardForce> ;\n",
      "    ns1:hasUnit \"http://qudt.org/vocab/unit/KiloN\"^^xsd:anyURI .\n",
      "\n",
      "<https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/TestTime> a <https://w3id.org/steel/ProcessOntology/TestTime> ;\n",
      "    ns1:hasUnit \"http://qudt.org/vocab/unit/SEC\"^^xsd:anyURI .\n",
      "\n",
      "<https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/tableGroup> a csvw:TableGroup ;\n",
      "    csvw:table [ a csvw:Table ;\n",
      "            rdfs:label \"Time series data\" ;\n",
      "            csvw:tableSchema [ a csvw:Schema ;\n",
      "                    csvw:column [ a csvw:Column ;\n",
      "                            ns1:quantity <https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/AbsoluteCrossheadTravel> ;\n",
      "                            csvw:titles \"C\"^^xsd:string ;\n",
      "                            ns2:page [ a ns2:Document ;\n",
      "                                    dcterms:format \"https://www.iana.org/assignments/media-types/application/json\"^^xsd:anyURI ;\n",
      "                                    dcterms:identifier \"https://bue.materials-data.space/api/knowledge/data_api/column-2\"^^xsd:anyURI ;\n",
      "                                    dcterms:type \"http://purl.org/dc/terms/Dataset\"^^xsd:anyURI ] ],\n",
      "                        [ a csvw:Column ;\n",
      "                            ns1:quantity <https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/StandardForce> ;\n",
      "                            csvw:titles \"B\"^^xsd:string ;\n",
      "                            ns2:page [ a ns2:Document ;\n",
      "                                    dcterms:format \"https://www.iana.org/assignments/media-types/application/json\"^^xsd:anyURI ;\n",
      "                                    dcterms:identifier \"https://bue.materials-data.space/api/knowledge/data_api/column-1\"^^xsd:anyURI ;\n",
      "                                    dcterms:type \"http://purl.org/dc/terms/Dataset\"^^xsd:anyURI ] ],\n",
      "                        [ a csvw:Column ;\n",
      "                            ns1:quantity <https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593/TestTime> ;\n",
      "                            csvw:titles \"A\"^^xsd:string ;\n",
      "                            ns2:page [ a ns2:Document ;\n",
      "                                    dcterms:format \"https://www.iana.org/assignments/media-types/application/json\"^^xsd:anyURI ;\n",
      "                                    dcterms:identifier \"https://bue.materials-data.space/api/knowledge/data_api/column-0\"^^xsd:anyURI ;\n",
      "                                    dcterms:type \"http://purl.org/dc/terms/Dataset\"^^xsd:anyURI ] ] ] ] .\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(item.subgraph.serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we are able to convert our data into any compatiable unit we want. For the `StandardForce`, it was previously `kN`, but we want to have it in `N` now:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1300.0,\n",
       " 1800.0,\n",
       " 2100.0,\n",
       " 2600.0,\n",
       " 3200.0,\n",
       " 3700.0,\n",
       " 4300.0,\n",
       " 4800.0,\n",
       " 5300.0,\n",
       " 6000.0]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.dataframe.StandardForce.convert_to(\"N\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2.2.3 Manipulate dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are able to retrieve the dataframe as pd.DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TestTime</th>\n",
       "      <th>StandardForce</th>\n",
       "      <th>AbsoluteCrossheadTravel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.7</td>\n",
       "      <td>1.8</td>\n",
       "      <td>1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.5</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.2</td>\n",
       "      <td>3.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3.6</td>\n",
       "      <td>3.7</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4.1</td>\n",
       "      <td>4.3</td>\n",
       "      <td>4.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>4.7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5.2</td>\n",
       "      <td>5.3</td>\n",
       "      <td>5.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5.8</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TestTime  StandardForce  AbsoluteCrossheadTravel\n",
       "0       1.2            1.3                      1.5\n",
       "1       1.7            1.8                      1.9\n",
       "2       2.0            2.1                      2.3\n",
       "3       2.5            2.6                      2.8\n",
       "4       3.0            3.2                      3.4\n",
       "5       3.6            3.7                      3.9\n",
       "6       4.1            4.3                      4.4\n",
       "7       4.7            4.8                      5.0\n",
       "8       5.2            5.3                      5.5\n",
       "9       5.8            6.0                      6.1"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item.dataframe.to_df()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are able to overwrite the dataframe with new data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "item.dataframe = {\n",
    "    \"TestTime\": list(range(100)),\n",
    "    \"StandardForce\": list(range(1,101)),\n",
    "    \"AbsoluteCrossheadTravel\": list(range(2,102))\n",
    "}\n",
    "dsms.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are able to retrieve the data colum-wise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "column: TestTime ,\n",
      " data: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]\n",
      "column: StandardForce ,\n",
      " data: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]\n",
      "column: AbsoluteCrossheadTravel ,\n",
      " data: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101]\n"
     ]
    }
   ],
   "source": [
    "for column in item.dataframe:\n",
    "    print(\"column:\", column.name, \",\\n\", \"data:\", column.get())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and also to modify the dataframe directly as we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df = item.dataframe.to_df().drop(['TestTime'], axis=1)\n",
    "item.dataframe = new_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2.2.4 Run app on demand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are able to run the app on demand, not being triggered automatically during the upload of an attachment every time. For this purpose, we just need to refer to the name of the app we assigned during the KItem creation ( here it is simply `data2rdf`).\n",
    "\n",
    "Additionally, we need to tell the `attachment_name` and hand over the access token and host url to the app by explicitly setting `expose_sdk_config` to `True`. This is basically telling the SDK that the app is using the SDK also internally and that the app should receive its parameters from the current SDK session.\n",
    "\n",
    "The app is running synchronously, hence the `job` is created when the pipeline run finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = item.kitem_apps.by_title[\"data2rdf\"].run(\n",
    "    attachment_name=item.attachments[0].name,\n",
    "    expose_sdk_config=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are able to retrieve the job status:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "JobStatus(phase='Succeeded', estimated_duration=None, finished_at='08/20/2024, 08:05:35', started_at='08/20/2024, 08:05:15', message=None, progress='1/1')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "job.status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and the job logs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"[2024-08-20 08:05:23,481 - dsms_data2rdf.main - INFO]: Fetch KItem: \\n KItem(\\n\\n\\tname = my tensile test experiment, \\n\\n\\tid = 58683a2d-7823-4638-bc8e-91b461afa593, \\n\\n\\tktype_id = dataset, \\n\\n\\tin_backend = True, \\n\\n\\tslug = mytensiletestexperiment-58683a2d, \\n\\n\\tannotations = [], \\n\\n\\tattachments = [\\n\\t\\t{\\n\\t\\t\\tname: dummy_data.csv\\n\\t\\t}\\n\\t], \\n\\n\\tlinked_kitems = [], \\n\\n\\taffiliations = [], \\n\\n\\tauthors = [\\n\\t\\t{\\n\\t\\t\\tuser_id: 7f0e5a37-353b-4bbc-b1f1-b6ad575f562d\\n\\t\\t}\\n\\t], \\n\\n\\tavatar_exists = True, \\n\\n\\tcontacts = [], \\n\\n\\tcreated_at = 2024-08-20 08:04:25.756982, \\n\\n\\tupdated_at = 2024-08-20 08:04:25.756982, \\n\\n\\texternal_links = [], \\n\\n\\tkitem_apps = [\\n\\t\\t{\\n\\t\\t\\tkitem_app_id: 22,\\n\\t\\t\\texecutable: testapp2,\\n\\t\\t\\ttitle: data2rdf,\\n\\t\\t\\tdescription: None,\\n\\t\\t\\ttags: None,\\n\\t\\t\\tadditional_properties: {triggerUponUpload: True, triggerUponUploadFileExtensions: ['.csv']}\\n\\t\\t}\\n\\t], \\n\\n\\tsummary = None, \\n\\n\\tuser_groups = [], \\n\\n\\tcustom_properties = None, \\n\\n\\tdataframe = [\\n\\t\\t{\\n\\t\\t\\tcolumn_id: 0,\\n\\t\\t\\tname: TestTime\\n\\t\\t}, \\n\\t\\t{\\n\\t\\t\\tcolumn_id: 1,\\n\\t\\t\\tname: StandardForce\\n\\t\\t}, \\n\\t\\t{\\n\\t\\t\\tcolumn_id: 2,\\n\\t\\t\\tname: AbsoluteCrossheadTravel\\n\\t\\t}\\n\\t], \\n\\n\\trdf_exists = True\\n)\\n[2024-08-20 08:05:23,494 - dsms_data2rdf.main - INFO]: Run pipeline with the following parser arguments: {'metadata_sep': ',', 'metadata_length': '0', 'time_series_sep': ',', 'time_series_header_length': '1', 'drop_na': 'false', 'fillna': ''}\\n[2024-08-20 08:05:23,494 - dsms_data2rdf.main - INFO]: Run pipeline with the following parser: Parser.csv\\n[2024-08-20 08:05:23,494 - dsms_data2rdf.main - INFO]: Run pipeline with the following config: {'base_iri': 'https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593', 'data_download_uri': 'https://bue.materials-data.space/api/knowledge/data_api/58683a2d-7823-4638-bc8e-91b461afa593', 'graph_identifier': 'https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593', 'separator': '/', 'encoding': 'utf-8'}\\n[2024-08-20 08:05:28,419 - dsms_data2rdf.main - INFO]: Pipeline did detect any metadata. Will not make annotations for KItem\\n[2024-08-20 08:05:28,810 - dsms_data2rdf.main - INFO]: Checking that dataframe is up to date.\\n[2024-08-20 08:05:28,810 - dsms_data2rdf.main - INFO]: Dataframe upload was successful after 0 retries.\\n[2024-08-20 08:05:28,810 - dsms_data2rdf.main - INFO]: Done!\\n\"\n"
     ]
    }
   ],
   "source": [
    "print(job.logs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case we would like to run the job in the background, we simply add a `wait=False`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = item.kitem_apps.by_title[\"data2rdf\"].run(\n",
    "    attachment_name=item.attachments[0].name,\n",
    "    expose_sdk_config=True,\n",
    "    wait=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are able to monitor the job status and logs asynchronously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Running' estimated_duration=None finished_at=None started_at='08/20/2024, 08:05:39' message=None progress='0/1'\n",
      "\n",
      " Current logs:\n",
      "\"\"\n",
      "\n",
      " Current status:\n",
      "phase='Succeeded' estimated_duration=None finished_at='08/20/2024, 08:05:59' started_at='08/20/2024, 08:05:39' message=None progress='1/1'\n",
      "\n",
      " Current logs:\n",
      "\"[2024-08-20 08:05:45,472 - dsms_data2rdf.main - INFO]: Fetch KItem: \\n KItem(\\n\\n\\tname = my tensile test experiment, \\n\\n\\tid = 58683a2d-7823-4638-bc8e-91b461afa593, \\n\\n\\tktype_id = dataset, \\n\\n\\tin_backend = True, \\n\\n\\tslug = mytensiletestexperiment-58683a2d, \\n\\n\\tannotations = [], \\n\\n\\tattachments = [\\n\\t\\t{\\n\\t\\t\\tname: dummy_data.csv\\n\\t\\t}\\n\\t], \\n\\n\\tlinked_kitems = [], \\n\\n\\taffiliations = [], \\n\\n\\tauthors = [\\n\\t\\t{\\n\\t\\t\\tuser_id: 7f0e5a37-353b-4bbc-b1f1-b6ad575f562d\\n\\t\\t}\\n\\t], \\n\\n\\tavatar_exists = True, \\n\\n\\tcontacts = [], \\n\\n\\tcreated_at = 2024-08-20 08:04:25.756982, \\n\\n\\tupdated_at = 2024-08-20 08:04:25.756982, \\n\\n\\texternal_links = [], \\n\\n\\tkitem_apps = [\\n\\t\\t{\\n\\t\\t\\tkitem_app_id: 22,\\n\\t\\t\\texecutable: testapp2,\\n\\t\\t\\ttitle: data2rdf,\\n\\t\\t\\tdescription: None,\\n\\t\\t\\ttags: None,\\n\\t\\t\\tadditional_properties: {triggerUponUpload: True, triggerUponUploadFileExtensions: ['.csv']}\\n\\t\\t}\\n\\t], \\n\\n\\tsummary = None, \\n\\n\\tuser_groups = [], \\n\\n\\tcustom_properties = None, \\n\\n\\tdataframe = [\\n\\t\\t{\\n\\t\\t\\tcolumn_id: 0,\\n\\t\\t\\tname: TestTime\\n\\t\\t}, \\n\\t\\t{\\n\\t\\t\\tcolumn_id: 1,\\n\\t\\t\\tname: StandardForce\\n\\t\\t}, \\n\\t\\t{\\n\\t\\t\\tcolumn_id: 2,\\n\\t\\t\\tname: AbsoluteCrossheadTravel\\n\\t\\t}\\n\\t], \\n\\n\\trdf_exists = True\\n)\\n[2024-08-20 08:05:45,485 - dsms_data2rdf.main - INFO]: Run pipeline with the following parser arguments: {'metadata_sep': ',', 'metadata_length': '0', 'time_series_sep': ',', 'time_series_header_length': '1', 'drop_na': 'false', 'fillna': ''}\\n[2024-08-20 08:05:45,485 - dsms_data2rdf.main - INFO]: Run pipeline with the following parser: Parser.csv\\n[2024-08-20 08:05:45,485 - dsms_data2rdf.main - INFO]: Run pipeline with the following config: {'base_iri': 'https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593', 'data_download_uri': 'https://bue.materials-data.space/api/knowledge/data_api/58683a2d-7823-4638-bc8e-91b461afa593', 'graph_identifier': 'https://bue.materials-data.space/58683a2d-7823-4638-bc8e-91b461afa593', 'separator': '/', 'encoding': 'utf-8'}\\n[2024-08-20 08:05:50,422 - dsms_data2rdf.main - INFO]: Pipeline did detect any metadata. Will not make annotations for KItem\\n[2024-08-20 08:05:50,816 - dsms_data2rdf.main - INFO]: Checking that dataframe is up to date.\\n[2024-08-20 08:05:50,816 - dsms_data2rdf.main - INFO]: Dataframe upload was successful after 0 retries.\\n[2024-08-20 08:05:50,816 - dsms_data2rdf.main - INFO]: Done!\\n\"\n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    time.sleep(1)\n",
    "    print(\"\\n Current status:\")\n",
    "    print(job.status)\n",
    "    print(\"\\n Current logs:\")\n",
    "    print(job.logs)\n",
    "    if job.status.phase != \"Running\":\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**: When job has run asychronously (in the background), we need to manually refresh the KItem afterwards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "item.refresh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up the DSMS from the tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "del dsms[item]\n",
    "del dsms[appspec]\n",
    "dsms.commit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sdk",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}