examples/sum/hardware/generate-schema.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate an input schema for Fletchgen\n",
    "\n",
    "We first import pyarrow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyarrow as pa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start constructing the schema.\n",
    "For this simple example, the schema will describe the types of a very simple table.\n",
    "The table will only contain a single column with some numbers, called \"num\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new field named \"number\" of type int64 that is not nullable.\n",
    "number_field = pa.field('number', pa.int64(), nullable=False)\n",
    "\n",
    "# Create a list of fields for pa.schema()\n",
    "schema_fields = [number_field]\n",
    "\n",
    "# Create a new schema from the fields.\n",
    "schema = pa.schema(schema_fields)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fletchgen would be able to process this schema already.\n",
    "However, we will take a look at how we can pass some additional options to Fletchgen to make sure it generates the type of hardware infrastructure that we want.\n",
    "\n",
    "### Schema mode\n",
    "One important option is the access mode of the RecordBatch that is described by this schema. The access mode can be either ...\n",
    "* ```'read'```: when you want the FPGA kernel to \"read\" from the RecordBatch in memory, or \n",
    "* ```'write'```: when you want the FPGA kernel to \"write\" to the RecordBatch in memory.\n",
    "\n",
    "... and is to be set in the metadata of the Arrow schema with the key `'fletcher_mode'`.\n",
    "\n",
    "Note that in the programming model of Arrow, RecordBatches are immutable once constructed. Therefore, a schema mode cannot be both `'read'` and `'write'` at the same time.\n",
    "\n",
    "### Schema name\n",
    "As Fletchgen can create kernels that operate on multiple input and/or output RecordBatches, we need a way of telling which RecordBatch is which, in case they have fields of the same name. Therefore, we must name each input schema using the metadata key 'fletcher_name'.\n",
    "\n",
    "We'll now go ahead and define that we'd like to read from the RecordBatch that this Schema can describe, and that its name should be 'ExampleBatch'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number: int64 not null\n",
      "metadata\n",
      "--------\n",
      "OrderedDict([(b'fletcher_mode', b'read'), (b'fletcher_name', b'ExampleBatch')])\n"
     ]
    }
   ],
   "source": [
    "# Construct some metadata to explain Fletchgen that it \n",
    "# should allow the FPGA kernel to read from this schema.\n",
    "metadata = {b'fletcher_mode': b'read',\n",
    "            b'fletcher_name': b'ExampleBatch'}\n",
    "\n",
    "# Add the metadata to the schema\n",
    "schema = schema.add_metadata(metadata)\n",
    "\n",
    "# Show the schema\n",
    "print(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now created a schema and added the appropriate metadata for Fletchgen to do it's job. All we have to do now is save it to a file so we can pass it to Fletchgen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Serialize the schema itself into an Arrow buffer.\n",
    "serialized_schema = schema.serialize()\n",
    "\n",
    "# Write the buffer to a file output stream.\n",
    "pa.output_stream('schema.as').write(serialized_schema);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In your project folder, you should now find a file that contains the schema. We will use this file as input for Fletchgen.\n",
    "\n",
    "[Return to the Sum tutorial](../README.md#generate-a-recordbatch)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}