1.6_snapshot_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "487b4777-9981-4de5-a440-916ee4af0513",
   "metadata": {},
   "source": [
    "# Snapshot Extraction\n",
    "\n",
    "This notebook shows how to run a Snapshot Explain operation with the minimal steps and a simple query.\n",
    "\n",
    "In this notebook...\n",
    "* [Dependencies and Initialisation](#dependencies-and-initialisation)\n",
    "* [The Where Statement](#the-where-statement)\n",
    "* [Extraction Query Options](#extraction-query-options)\n",
    "* [Running the Extraction Operation](#running-the-extraction-operation-decremental)\n",
    "* [Next Steps](#Next-Steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd55c3bc-ffc6-4634-923f-aa8aee26d7d0",
   "metadata": {},
   "source": [
    "## Dependencies and Initialisation\n",
    "Import statements and environment initialisation using the package `dotenv`. More details in the [Configuration notebook](0.2_configuration.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7535aef1-faff-4eb1-b798-4d453468d498",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n"
     ]
    }
   ],
   "source": [
    "from factiva.news import Snapshot\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0273c8a9-6bb6-447b-abaf-b81a87f6511d",
   "metadata": {},
   "source": [
    "## The Where Statement\n",
    "\n",
    "This notebook uses a simple query for illustration purposes. For more tips about queries, or guidance on how to build complex or large queries, checkout the [query reference](2.1_complex_large_queries.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1816e3a2-9258-4d4d-a737-a1fd1e45fd31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Industry i3432 is for Batteries\n",
    "where_statement = (\n",
    "    r\" publication_datetime >= '2016-01-01 00:00:00' \"\n",
    "    r\" AND LOWER(language_code) IN ('en', 'de', 'fr') \"\n",
    "    r\" AND REGEXP_CONTAINS(industry_codes, r'(?i)(^|,)(i3432)($|,)') \"\n",
    ")\n",
    "\n",
    "s = Snapshot(query=where_statement)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44ecaa78-e0cd-4d37-a872-436686f316c1",
   "metadata": {},
   "source": [
    "## Extraction Query Options\n",
    "\n",
    "An extraction query can use more parameters:\n",
    "\n",
    "* **`file_format`**: _Optional_, _Default: `'avro'`_. File format to be used for Extractions. Possible values are `'avro'`, `'csv'` or `'json'`. Used only by the Extraction operation.\n",
    "* **`limit`**: _Optional_, _Default: `0` (No limit)_. Positive integer that limits the amount of documents to extract. Used only by the Extraction operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "47f8c7b5-a1f5-48be-9f5c-1fcfa9b2c30a",
   "metadata": {},
   "outputs": [],
   "source": [
    "s.query.file_format = 'avro'\n",
    "# s.query.limit = 1000     # Uncomment this line to set a max number of extracted documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bf7ac0-0a75-411c-9809-781bdca9aa9b",
   "metadata": {},
   "source": [
    "## Running the Extraction Operation   `**(decremental)**`\n",
    "\n",
    "This operation builds a collection of files containing the articles selected according to query conditions.\n",
    "\n",
    "**This operation will decrement 1 extraction from your allowance, and can take several minutes to complete**\n",
    "\n",
    "An Extraction job decrements the account's allowance, and therefore, it's highly recommended to be executed after verifying the same query using [Explain](1.4_snapshot_explain.ipynb) and/or [Analytics](1.5_snapshot_analytics.ipynb) jobs return values in line with the expected volumes.\n",
    "\n",
    "The `<Snapshot>.process_extraction()` function directly submits, monitors the job and download the content. If a more manual process is required (send job, monitor job, get results), please see the [detailed package documentation](https://factiva-news-python.readthedocs.io/).\n",
    "\n",
    "To review the **Snapshot History**, please see the notebook [User Statistics](1.1_user_statistics.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a495fac7-5096-43d6-9d0b-4d083cf02409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n",
      "CPU times: user 4.72 s, sys: 1.9 s, total: 6.61 s\n",
      "Wall time: 7min 19s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "s.process_extraction(download_path='./data/')\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c726d7-4f38-4507-bec0-7aac4c3e21bb",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "* Work with the downloaded files as described in the [snapshot files](1.9_snapshot_files.ipynb) notebook\n",
    "* Create a new Stream"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}