1.9_snapshot_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "487b4777-9981-4de5-a440-916ee4af0513",
   "metadata": {},
   "source": [
    "# Working with Snapshot Files\n",
    "\n",
    "This notebook shows how to read and use content from Snapshot files, either from a local folder or from a specific file.\n",
    "\n",
    "In this notebook...\n",
    "* [Dependencies and Initialisation](#Dependencies-and-Initialisation)\n",
    "* [Supported File Formats]()\n",
    "* [Load a single AVRO file to a Pandas DataFrame]()\n",
    "* [Load all AVRO files in a folder to a Pandas DataFrame]()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd55c3bc-ffc6-4634-923f-aa8aee26d7d0",
   "metadata": {},
   "source": [
    "## Dependencies and Initialisation\n",
    "Import statements and environment initialisation using the package `dotenv`. More details in the [Configuration notebook](0.2_configuration.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7535aef1-faff-4eb1-b798-4d453468d498",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n"
     ]
    }
   ],
   "source": [
    "from factiva.news import SnapshotFiles\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44ecaa78-e0cd-4d37-a872-436686f316c1",
   "metadata": {},
   "source": [
    "## Supported File Formats\n",
    "\n",
    "At the moment, `factiva-*` packages only support `AVRO` files. The support for additional formats is currently under development."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bf7ac0-0a75-411c-9809-781bdca9aa9b",
   "metadata": {},
   "source": [
    "## Load a single AVRO file to a Pandas DataFrame\n",
    "\n",
    "This operations uses the function `<SnapshotFiles>.read_file`.\n",
    "\n",
    "* **`filepath`**: Relative or absolute file path\n",
    "* **`only_stats`**: _Optional_. Specifies if only file metadata is loaded (True), or if the full article content is loaded (False). On average, only_stats loads about 1/10 and is recommended for quick metadata-based analysis. (Default is False)\n",
    "* **`merge_body`**: _Optional_. Specifies if the body field should be merged with the snippet and this last column being dropped. (default is False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a495fac7-5096-43d6-9d0b-4d083cf02409",
   "metadata": {},
   "outputs": [],
   "source": [
    "sf = SnapshotFiles()\n",
    "articles = sf.read_file('./data/part-000000000000.avro', only_stats=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eec74feb-1ca6-424c-9efd-4ddd91098169",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['an', 'company_codes', 'company_codes_about', 'company_codes_occur',\n",
       "       'industry_codes', 'ingestion_datetime', 'language_code',\n",
       "       'modification_datetime', 'publication_datetime', 'publisher_name',\n",
       "       'region_codes', 'region_of_origin', 'source_code', 'source_name',\n",
       "       'subject_codes', 'title', 'word_count'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "32185020-a950-4c5e-a539-09c63293c230",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>an</th>\n",
       "      <th>publication_datetime</th>\n",
       "      <th>title</th>\n",
       "      <th>industry_codes</th>\n",
       "      <th>language_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>T000000020170922ed9m0006v</td>\n",
       "      <td>2017-09-22 00:00:00.000</td>\n",
       "      <td>Battery business plugs into electric car market</td>\n",
       "      <td>,i3432,i35104,i353,i351,iaut,iindele,iindstrls...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>DJDN000020160805ec8500186</td>\n",
       "      <td>2016-08-05 09:00:00.000</td>\n",
       "      <td>Press Release: Magna Announces Record Second Q...</td>\n",
       "      <td>,i3432,i353,iaut,iindele,iindstrls,itech,</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CHNDLY0020210630eh6u00006</td>\n",
       "      <td>2021-06-30 00:00:00.000</td>\n",
       "      <td>CATL prospects brighten on Tesla deal</td>\n",
       "      <td>,i3432,i35104,i351,iaut,iindele,iindstrls,itech,</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RTDJGE0020161130ecbu000ig</td>\n",
       "      <td>2016-11-30 15:35:43.463</td>\n",
       "      <td>Batteriehersteller/Varta verschiebt geplanten ...</td>\n",
       "      <td>,i3432,iindele,iindstrls,itech,</td>\n",
       "      <td>de</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DJDN000020200923eg9n0020e</td>\n",
       "      <td>2020-09-23 14:38:27.054</td>\n",
       "      <td>Global Energy Roundup: Market Talk</td>\n",
       "      <td>,i3432,i1,i25,i342,i35101,i35104,iindstrls,i35...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          an    publication_datetime  \\\n",
       "0  T000000020170922ed9m0006v 2017-09-22 00:00:00.000   \n",
       "1  DJDN000020160805ec8500186 2016-08-05 09:00:00.000   \n",
       "2  CHNDLY0020210630eh6u00006 2021-06-30 00:00:00.000   \n",
       "3  RTDJGE0020161130ecbu000ig 2016-11-30 15:35:43.463   \n",
       "4  DJDN000020200923eg9n0020e 2020-09-23 14:38:27.054   \n",
       "\n",
       "                                               title  \\\n",
       "0    Battery business plugs into electric car market   \n",
       "1  Press Release: Magna Announces Record Second Q...   \n",
       "2              CATL prospects brighten on Tesla deal   \n",
       "3  Batteriehersteller/Varta verschiebt geplanten ...   \n",
       "4                 Global Energy Roundup: Market Talk   \n",
       "\n",
       "                                      industry_codes language_code  \n",
       "0  ,i3432,i35104,i353,i351,iaut,iindele,iindstrls...            en  \n",
       "1          ,i3432,i353,iaut,iindele,iindstrls,itech,            en  \n",
       "2   ,i3432,i35104,i351,iaut,iindele,iindstrls,itech,            en  \n",
       "3                    ,i3432,iindele,iindstrls,itech,            de  \n",
       "4  ,i3432,i1,i25,i342,i35101,i35104,iindstrls,i35...            en  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles[['an', 'publication_datetime', 'title', 'industry_codes', 'language_code']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c726d7-4f38-4507-bec0-7aac4c3e21bb",
   "metadata": {},
   "source": [
    "## Load all files within a folder\n",
    "\n",
    "This operations loads all files with the same extention into a single Pandas DataFrame.\n",
    "\n",
    "* **`folderpath`**: Relative or absolute folder path\n",
    "* **`file_format`**: `AVRO` is the only format available at the moment\n",
    "* **`only_stats`**: _Optional_. Specifies if only file metadata is loaded (True), or if the full article content is loaded (False). On average, only_stats loads about 1/10 and is recommended for quick metadata-based analysis. (Default is False)\n",
    "* **`merge_body`**: _Optional_. Specifies if the body field should be merged with the snippet and this last column being dropped. (default is False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8179fc2b-7d2d-4c23-8f96-4a11c9b97af7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from factiva.news import SnapshotFiles\n",
    "sf = SnapshotFiles()\n",
    "articles = sf.read_folder(folderpath='./data/ztj2gkbldt', only_stats=False, merge_body=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b3b45fce-12f6-4a2b-a21d-4de74c88e237",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['copyright', 'subject_codes', 'modification_datetime', 'body',\n",
       "       'company_codes_occur_ticker_exchange', 'company_codes_occur',\n",
       "       'company_codes_about', 'company_codes_lineage',\n",
       "       'company_codes_ticker_exchange',\n",
       "       'company_codes_relevance_ticker_exchange', 'market_index_codes',\n",
       "       'section', 'company_codes_association_ticker_exchange',\n",
       "       'currency_codes', 'company_codes_about_ticker_exchange',\n",
       "       'region_of_origin', 'company_codes_lineage_ticker_exchange',\n",
       "       'ingestion_datetime', 'modification_date', 'source_name',\n",
       "       'language_code', 'region_codes', 'company_codes_association',\n",
       "       'person_codes', 'byline', 'dateline', 'company_codes_relevance',\n",
       "       'source_code', 'an', 'word_count', 'company_codes', 'industry_codes',\n",
       "       'title', 'publication_datetime', 'publisher_name', 'action'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1c91f9d0-941b-41d6-b0b3-0675973778a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>an</th>\n",
       "      <th>publication_datetime</th>\n",
       "      <th>title</th>\n",
       "      <th>industry_codes</th>\n",
       "      <th>language_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>T000000020170922ed9m0006v</td>\n",
       "      <td>2017-09-22 00:00:00.000</td>\n",
       "      <td>Battery business plugs into electric car market</td>\n",
       "      <td>,i3432,i35104,i353,i351,iaut,iindele,iindstrls...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>DJDN000020160805ec8500186</td>\n",
       "      <td>2016-08-05 09:00:00.000</td>\n",
       "      <td>Press Release: Magna Announces Record Second Q...</td>\n",
       "      <td>,i3432,i353,iaut,iindele,iindstrls,itech,</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CHNDLY0020210630eh6u00006</td>\n",
       "      <td>2021-06-30 00:00:00.000</td>\n",
       "      <td>CATL prospects brighten on Tesla deal</td>\n",
       "      <td>,i3432,i35104,i351,iaut,iindele,iindstrls,itech,</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RTDJGE0020161130ecbu000ig</td>\n",
       "      <td>2016-11-30 15:35:43.463</td>\n",
       "      <td>Batteriehersteller/Varta verschiebt geplanten ...</td>\n",
       "      <td>,i3432,iindele,iindstrls,itech,</td>\n",
       "      <td>de</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DJDN000020200923eg9n0020e</td>\n",
       "      <td>2020-09-23 14:38:27.054</td>\n",
       "      <td>Global Energy Roundup: Market Talk</td>\n",
       "      <td>,i3432,i1,i25,i342,i35101,i35104,iindstrls,i35...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          an    publication_datetime  \\\n",
       "0  T000000020170922ed9m0006v 2017-09-22 00:00:00.000   \n",
       "1  DJDN000020160805ec8500186 2016-08-05 09:00:00.000   \n",
       "2  CHNDLY0020210630eh6u00006 2021-06-30 00:00:00.000   \n",
       "3  RTDJGE0020161130ecbu000ig 2016-11-30 15:35:43.463   \n",
       "4  DJDN000020200923eg9n0020e 2020-09-23 14:38:27.054   \n",
       "\n",
       "                                               title  \\\n",
       "0    Battery business plugs into electric car market   \n",
       "1  Press Release: Magna Announces Record Second Q...   \n",
       "2              CATL prospects brighten on Tesla deal   \n",
       "3  Batteriehersteller/Varta verschiebt geplanten ...   \n",
       "4                 Global Energy Roundup: Market Talk   \n",
       "\n",
       "                                      industry_codes language_code  \n",
       "0  ,i3432,i35104,i353,i351,iaut,iindele,iindstrls...            en  \n",
       "1          ,i3432,i353,iaut,iindele,iindstrls,itech,            en  \n",
       "2   ,i3432,i35104,i351,iaut,iindele,iindstrls,itech,            en  \n",
       "3                    ,i3432,iindele,iindstrls,itech,            de  \n",
       "4  ,i3432,i1,i25,i342,i35101,i35104,iindstrls,i35...            en  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles[['an', 'publication_datetime', 'title', 'industry_codes', 'language_code']].head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}