Working-with-PDF-Text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with PDF Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pip install PyPDF2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note the capitalization\n",
    "import PyPDF2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading PDFs\n",
    "\n",
    "First we open a pdf, then create a reader object for it. Notice how we use the binary method of reading , 'rb', instead of just 'r'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notice we read it as a binary with 'rb'\n",
    "f = open('US_Declaration.pdf','rb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_reader = PyPDF2.PdfFileReader(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_reader.numPages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_one = pdf_reader.getPage(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then extract the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_one_text = page_one.extractText()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Declaration of IndependenceIN CONGRESS, July 4, 1776. The unanimous Declaration of the thirteen united States of America, When in the Course of human events, it becomes necessary for one people to dissolve the\\npolitical bands which have connected them with another, and to assume among the powers of the\\nearth, the separate and equal station to which the Laws of Nature and of Nature's God entitle\\n\\nthem, a decent respect to the opinions of mankind requires that they should declare the causes\\n\\nwhich impel them to the separation. \\nWe hold these truths to be self-evident, that all men are created equal, that they are endowed by\\n\\ntheir Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit\\nof Happiness.ŠThat to secure these rights, Governments are instituted among Men, deriving\\n\\ntheir just powers from the consent of the governed,ŠThat whenever any Form of Government\\nbecomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to\\ninstitute new Government, laying its foundation on such principles and organizing its powers in\\nsuch form, as to them shall seem most likely to effect their Safety and Happiness. Prudence,\\n\\nindeed, will dictate that Governments long established should not be changed for light and\\ntransient causes; and accordingly all experience hath shewn, that mankind are more disposed to\\nsuffer, while evils are sufferable, than to right themselves by abolishing the forms to which they\\n\\nare accustomed. But when a long train of abuses and usurpations, pursuing invariably the same\\nObject evinces a design to reduce them under absolute Despotism, it is their right, it is their duty,\\nto throw off such Government, and to provide new Guards for their future security.ŠSuch has\\nbeen the patient sufferance of these Colonies; and such is now the necessity which constrains\\n\\nthem to alter their former Systems of Government. The history of the present King of Great\\n\\nBritain is a history of repeated injuries and usurpations, all having in direct object the\\nestablishment of an absolute Tyranny over these States. To prove this, let Facts be submitted to a\\ncandid world. He has refused his Assent to Laws, the most wholesome and necessary for the\\npublic good.\\nHe has forbidden his Governors to pass Laws of immediate and pressing\\nimportance, unless suspended in their operation till his Assent should be obtained;\\nand when so suspended, he has utterly neglected to attend to them.\\n\\nHe has refused to pass other Laws for the accommodation of large districts of\\npeople, unless those people would relinquish the right of Representation in the\\nLegislature, a right inestimable to them and formidable to tyrants only. \\n\\nHe has called together legislative bodies at places unusual, uncomfortable, and distant\\nfrom the depository of their public Records, for the sole purpose of fatiguing them into\\ncompliance with his measures.\\n\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page_one_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding to PDFs\n",
    "\n",
    "We can not write to PDFs using Python because of the differences between the single string type of Python, and the variety of fonts, placements, and other parameters that a PDF could have.\n",
    "\n",
    "What we *can* do is copy pages and append pages to the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('US_Declaration.pdf','rb')\n",
    "pdf_reader = PyPDF2.PdfFileReader(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_page = pdf_reader.getPage(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_writer = PyPDF2.PdfFileWriter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_writer.addPage(first_page)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_output = open(\"Some_New_Doc.pdf\",\"wb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_writer.write(pdf_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_output.close()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have copied a page and added it to another new document!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Example\n",
    "\n",
    "Let's try to grab all the text from this PDF file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('US_Declaration.pdf','rb')\n",
    "\n",
    "# List of every page's text.\n",
    "# The index will correspond to the page number.\n",
    "pdf_text = [0]  # zero is a placehoder to make page 1 = index 1\n",
    "\n",
    "pdf_reader = PyPDF2.PdfFileReader(f)\n",
    "\n",
    "for p in range(pdf_reader.numPages):\n",
    "    \n",
    "    page = pdf_reader.getPage(p)\n",
    "    \n",
    "    pdf_text.append(page.extractText())\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0,\n",
       " \"Declaration of IndependenceIN CONGRESS, July 4, 1776. The unanimous Declaration of the thirteen united States of America, When in the Course of human events, it becomes necessary for one people to dissolve the\\npolitical bands which have connected them with another, and to assume among the powers of the\\nearth, the separate and equal station to which the Laws of Nature and of Nature's God entitle\\n\\nthem, a decent respect to the opinions of mankind requires that they should declare the causes\\n\\nwhich impel them to the separation. \\nWe hold these truths to be self-evident, that all men are created equal, that they are endowed by\\n\\ntheir Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit\\nof Happiness.ŠThat to secure these rights, Governments are instituted among Men, deriving\\n\\ntheir just powers from the consent of the governed,ŠThat whenever any Form of Government\\nbecomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to\\ninstitute new Government, laying its foundation on such principles and organizing its powers in\\nsuch form, as to them shall seem most likely to effect their Safety and Happiness. Prudence,\\n\\nindeed, will dictate that Governments long established should not be changed for light and\\ntransient causes; and accordingly all experience hath shewn, that mankind are more disposed to\\nsuffer, while evils are sufferable, than to right themselves by abolishing the forms to which they\\n\\nare accustomed. But when a long train of abuses and usurpations, pursuing invariably the same\\nObject evinces a design to reduce them under absolute Despotism, it is their right, it is their duty,\\nto throw off such Government, and to provide new Guards for their future security.ŠSuch has\\nbeen the patient sufferance of these Colonies; and such is now the necessity which constrains\\n\\nthem to alter their former Systems of Government. The history of the present King of Great\\n\\nBritain is a history of repeated injuries and usurpations, all having in direct object the\\nestablishment of an absolute Tyranny over these States. To prove this, let Facts be submitted to a\\ncandid world. He has refused his Assent to Laws, the most wholesome and necessary for the\\npublic good.\\nHe has forbidden his Governors to pass Laws of immediate and pressing\\nimportance, unless suspended in their operation till his Assent should be obtained;\\nand when so suspended, he has utterly neglected to attend to them.\\n\\nHe has refused to pass other Laws for the accommodation of large districts of\\npeople, unless those people would relinquish the right of Representation in the\\nLegislature, a right inestimable to them and formidable to tyrants only. \\n\\nHe has called together legislative bodies at places unusual, uncomfortable, and distant\\nfrom the depository of their public Records, for the sole purpose of fatiguing them into\\ncompliance with his measures.\\n\",\n",
       " 'He has dissolved Representative Houses repeatedly, for opposing with manlyfirmness his invasions on the rights of the people.He has refused for a long time, after such dissolutions, to cause others to beelected; whereby the Legislative powers, incapable of Annihilation, have returnedto the People at large for their exercise; the State remaining in the mean timeexposed to all the dangers of invasion from without, and convulsions within.He has endeavoured to prevent the population of these States; for that purposeobstructing the Laws for Naturalization of Foreigners; refusing to pass others toencourage their migrations hither, and raising the conditions of newAppropriations of Lands.He has obstructed the Administration of Justice, by refusing his Assent to Lawsfor establishing Judiciary powers.He has made Judges dependent on his Will alone, for the tenure of their offices,and the amount and payment of their salaries.He has erected a multitude of New Offices, and sent hither swarms of Officers toharrass our people, and eat out their substance.He has kept among us, in times of peace, Standing Armies without the Consent ofour legislatures.He has affected to render the Military independent of and superior to the Civil power.He has combined with others to subject us to a jurisdiction foreign to ourconstitution, and unacknowledged by our laws; giving his Assent to their Acts ofpretended Legislation:For Quartering large bodies of armed troops among us:For protecting them, by a mock Trial, from punishment for any Murders whichthey should commit on the Inhabitants of these States:For cutting off our Trade with all parts of the world:For imposing Taxes on us without our Consent: For depriving us in many cases,of the benefits of Trial by Jury:For transporting us beyond Seas to be tried for pretended offencesFor abolishing the free System of English Laws in a neighbouring Province,establishing therein an Arbitrary government, and enlarging its Boundaries so as',\n",
       " 'to render it at once an example and fit instrument for introducing the sameabsolute rule into these Colonies:For taking away our Charters, abolishing our most valuable Laws, and alteringfundamentally the Forms of our Governments:For suspending our own Legislatures, and declaring themselves invested withpower to legislate for us in all cases whatsoever.He has abdicated Government here, by declaring us out of his Protection andwaging War against us.He has plundered our seas, ravaged our Coasts, burnt our towns, and destroyed thelives of our people.He is at this time transporting large Armies of foreign Mercenaries to compleatthe works of death, desolation and tyranny, already begun with circumstances ofCruelty & perfidy scarcely paralleled in the most barbarous ages, and totallyunworthy of the Head of a civilized nation.He has constrained our fellow Citizens taken Captive on the high Seas to bearArms against their Country, to become the executioners of their friends and\\nBrethren, or to fall themselves by their Hands.He has excited domestic insurrections amongst us, and has endeavoured to bringon the inhabitants of our frontiers, the merciless Indian Savages, whose known\\nrule of warfare, is an undistinguished destruction of all ages, sexes and conditions. In every stage of these Oppressions We have Petitioned for Redress in the most humble terms:Our repeated Petitions have been answered only by repeated injury. A Prince whose character isthus marked by every act which may define a Tyrant, is unfit to be the ruler of a free people. Nor have We been wanting in attentions to our Brittish brethren. We have warned them fromtime to time of attempts by their legislature to extend an unwarrantable jurisdiction over us. Wehave reminded them of the circumstances of our emigration and settlement here. We haveappealed to their native justice and magnanimity, and we have conjured them by the ties of ourcommon kindred to disavow these usurpations, which, would inevitably interrupt ourconnections and correspondence. They too have been deaf to the voice of justice and ofconsanguinity. We must, therefore, acquiesce in the necessity, which denounces our Separation,and hold them, as we hold the rest of mankind, Enemies in War, in Peace Friends. We, therefore, the Representatives of the united States of America, in General Congress,Assembled, appealing to the Supreme Judge of the world for the rectitude of our intentions, do,in the Name, and by Authority of the good People of these Colonies, solemnly publish anddeclare, That these United Colonies are, and of Right ought to be Free and Independent States;that they are Absolved from all Allegiance to the British Crown, and that all political connection',\n",
       " 'between them and the State of Great Britain, is and ought to be totally dissolved; and that as Free\\n\\nand Independent States, they have full Power to levy War, conclude Peace, contract Alliances,\\nestablish Commerce, and to do all other Acts and Things which Independent States may of right\\n\\ndo. And for the support of this Declaration, with a firm reliance on the protection of divine\\nProvidence, we mutually pledge to each other our Lives, our Fortunes and our sacred Honor.\\n\\n[The 56 signatures on the Declaration were arranged in six columns:\\n] [Column 1]\\n Georgia:   Button Gwinnett\\n   Lyman \\nHall   George Walton \\n[Column 2]\\n North Carolina:   William Hooper\\n   Joseph Hewes\\n   John Penn\\n South Carolina:   Edward Ru\\ntledge   Thomas Heyward, Jr.\\n  Thomas Lynch, Jr.\\n  Arthur Middleton \\n[Column 3]\\n Massachusetts:   John Hancock\\n\\n Maryland:   Samuel Chase\\n\\n   William Paca\\n\\n   Thomas Stone\\n\\n   Charles Carroll of Carrollton\\n\\n Virginia:   George Wythe\\n\\n   Richard Henry Lee\\n\\n   Thomas Jefferson\\n\\n   Benjamin Harrison\\n\\n   Thomas Nelson, Jr.\\n\\n   Francis Lightfoot Lee\\n\\n   Carter Braxton \\n\\n[Column 4]\\n Pennsylvania:  Robert Morris\\n\\n   Benjamin Rush\\n   Benjamin Fran\\nklin   John Morton\\n',\n",
       " '   George Clymer\\n   James Smith\\n   George Taylor\\n   James Wilson\\n   George Ross\\n Delaware:   Caesar Rodney\\n   George Read\\n   Thomas McKean \\n[Column 5]\\n New York:   Wi\\nlliam Floyd   Philip Livingston\\n   Francis L\\newis   Lewis Morris\\n New Jersey:   Richard Stockton\\n   John Witherspoon\\n   Francis Hopkinson\\n   John Hart\\n   Abraham Clark \\n[Column 6]\\n New Hampshire:   Josiah Bartlett\\n   William Whipple\\n Massachusetts:   Samuel Adams\\n   John Adams\\n   Robert Treat Paine\\n   Elbridge Gerry\\n Rhode Island:   Stephen Hopkins\\n   William Ellery\\n Connecticut:   Roger Sherman\\n   Samuel Huntington\\n   William Williams\\n   Oliver Wolcott\\n New Hampshire: Matthew Thornton\\n ']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "He has dissolved Representative Houses repeatedly, for opposing with manlyfirmness his invasions on the rights of the people.He has refused for a long time, after such dissolutions, to cause others to beelected; whereby the Legislative powers, incapable of Annihilation, have returnedto the People at large for their exercise; the State remaining in the mean timeexposed to all the dangers of invasion from without, and convulsions within.He has endeavoured to prevent the population of these States; for that purposeobstructing the Laws for Naturalization of Foreigners; refusing to pass others toencourage their migrations hither, and raising the conditions of newAppropriations of Lands.He has obstructed the Administration of Justice, by refusing his Assent to Lawsfor establishing Judiciary powers.He has made Judges dependent on his Will alone, for the tenure of their offices,and the amount and payment of their salaries.He has erected a multitude of New Offices, and sent hither swarms of Officers toharrass our people, and eat out their substance.He has kept among us, in times of peace, Standing Armies without the Consent ofour legislatures.He has affected to render the Military independent of and superior to the Civil power.He has combined with others to subject us to a jurisdiction foreign to ourconstitution, and unacknowledged by our laws; giving his Assent to their Acts ofpretended Legislation:For Quartering large bodies of armed troops among us:For protecting them, by a mock Trial, from punishment for any Murders whichthey should commit on the Inhabitants of these States:For cutting off our Trade with all parts of the world:For imposing Taxes on us without our Consent: For depriving us in many cases,of the benefits of Trial by Jury:For transporting us beyond Seas to be tried for pretended offencesFor abolishing the free System of English Laws in a neighbouring Province,establishing therein an Arbitrary government, and enlarging its Boundaries so as\n"
     ]
    }
   ],
   "source": [
    "print(pdf_text[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excellent work! \n",
    "That is all for PyPDF2 for now, remember that this won't work with every PDF file and is limited in its scope to only the text of PDFs.\n",
    "## Next up: Regular Expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}