010_Grep_command.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grep command "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print lines matching a pattern  (THE per-line filter!!!)\n",
    "SINTAX: grep “STRING\" [file_pattern]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/dsc/Data/shell\n",
      "     1\tTHIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.\r",
      "\r\n",
      "     2\tthis line is the 1st lower case line in this file.\r",
      "\r\n",
      "     3\tThis Line Has All Its First Character Of The Word With Upper Case.\r",
      "\r\n",
      "     4\t\r",
      "\r\n",
      "     5\t\r",
      "\r\n",
      "     6\tTwo lines above this are empty.\r",
      "\r\n",
      "     7\tAnd this is the last line.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "% cd ~/Data/shell\n",
    "\n",
    "! cat -n Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Print lines containing \"the\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this line is the 1st lower case line in this file.\r",
      "\r\n",
      "And this is the last line.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep the Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -v : Invert the sense of matching, to select non-matching lines. Let's print lines without \"the\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.\r",
      "\r\n",
      "This Line Has All Its First Character Of The Word With Upper Case.\r",
      "\r\n",
      "\r",
      "\r\n",
      "\r",
      "\r\n",
      "Two lines above this are empty.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep -v the Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -i: case insensitive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.\r",
      "\r\n",
      "this line is the 1st lower case line in this file.\r",
      "\r\n",
      "This Line Has All Its First Character Of The Word With Upper Case.\r",
      "\r\n",
      "And this is the last line.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep -i tHe Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -n : Prefix each line of output with the 1-based line number within its input file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2:this line is the 1st lower case line in this file.\r",
      "\r\n",
      "7:And this is the last line.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep -n the Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -c : print  count of matching lines for each input file.  With the -v option it counts non-matching lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\r\n"
     ]
    }
   ],
   "source": [
    "! grep -c the Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* They can be combined. Count lines NOT containing \"the\" case insensitive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\r\n"
     ]
    }
   ],
   "source": [
    "! grep -cvi \"tHe\" Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -w :  Select  only  those  lines  containing matches that form whole words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "an\n",
      "ananas\n",
      " Whole words: \n",
      "an\n"
     ]
    }
   ],
   "source": [
    "! echo \"an\\nananas\" | grep an \n",
    "\n",
    "! echo \" Whole words: \"\n",
    "! echo \"an\\nananas\" | grep -w an \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [A/B/C] +N = Displaying lines after/before/around the match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\r\n",
      "Two lines above this are empty.\r",
      "\r\n",
      "And this is the last line.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep -C 1 Two Text_example.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1-THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.\r",
      "\r\n",
      "2:this line is the 1st lower case line in this file.\r",
      "\r\n",
      "3-This Line Has All Its First Character Of The Word With Upper Case.\r",
      "\r\n",
      "4-\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep -n -B 1 -A 2 lower Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -H : Print the file name for each match. Useful to grep with several files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text_example.txt:2:this line is the 1st lower case line in this file.\r",
      "\r\n",
      "lower_file.txt:1:lower\r\n",
      "lower_file.txt:3:lower again\r\n"
     ]
    }
   ],
   "source": [
    "! rm -f lower_file.txt\n",
    "! echo \"lower\\nupper\\nlower again\" > lower_file.txt\n",
    "! grep -Hn lower Text_example.txt lower_file.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -E : enable regular expression (WORKS with regular expressions!!!) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "! rm -f mares.txt\n",
    "! echo \"Maria\\nMario\\nPaco\\nMarisa\\nMarimar\" > mares.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Words starting with \"mar\", case insensitive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Maria\r\n",
      "Mario\r\n",
      "Marisa\r\n",
      "Marimar\r\n"
     ]
    }
   ],
   "source": [
    "! grep -Ei \"^mar\" mares.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Words ending with \"mar\", case insensitive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Marimar\r\n"
     ]
    }
   ],
   "source": [
    "! grep -Ei \"mar$\" mares.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Find something startning with a number from 1 to 5 and then a '2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\r\n",
      "125\r\n",
      "220\r\n",
      "225\r\n"
     ]
    }
   ],
   "source": [
    "! seq 5 5 300 | grep -E \"[1-5]{2}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -o : show just the pattern matched "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text_example.txt:1:TH\r\n",
      "Text_example.txt:2:th\r\n",
      "Text_example.txt:3:Th\r\n"
     ]
    }
   ],
   "source": [
    "! grep -EionH \"^th\" Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -b : show the byte  offset in the whole file of the starting point of output "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52:this line is the 1st lower case line in this file.\r",
      "\r\n",
      "176:Two lines above this are empty.\r",
      "\r\n",
      "209:And this is the last line.\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "! grep -b line Text_example.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to ~/Data/opentraveldata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/dsc/Data/opentraveldata\n",
      "iata_code^manufacturer^model^iata_group^iata_category^icao_code^nb_engines^aircraft_type\r\n"
     ]
    }
   ],
   "source": [
    "% cd ~/Data/opentraveldata\n",
    "! head -n 1 optd_aircraft.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use grep to extract all 7x7 (where x can be any number) airplane models from optd_aircraft.csv. Hint: model is in column 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "707-300\r\n",
      "707/720\r\n",
      "707 Freighter\r\n",
      "707 Combi\r\n",
      "717\r\n",
      "727-100\r\n",
      "727-200\r\n",
      "727\r\n",
      "727-200 Advanced\r\n",
      "727-100 Mixed\r\n",
      "727-200 Mixed\r\n",
      "727 Freighter (-100/200)\r\n",
      "727 Combi\r\n",
      "727-200 Adv.\r\n",
      "727-200 (winglets)\r\n",
      "727-100 Freighter\r\n",
      "727-200 Freighter\r\n",
      "737-100\r\n",
      "737-200\r\n",
      "737-300\r\n",
      "737-400\r\n",
      "737-500\r\n",
      "737-600\r\n",
      "737\r\n",
      "737-800\r\n",
      "737-900\r\n",
      "737-200 Advanced\r\n",
      "737-300 (winglets)\r\n",
      "737-500 (winglets)\r\n",
      "737 Freighter\r\n",
      "737-700\r\n",
      "737-800 (winglets)\r\n",
      "737-900 (winglets)\r\n",
      "737-200 Mixed Configuration\r\n",
      "737-200 Combi\r\n",
      "737-300 Mixed Configuration\r\n",
      "737-400 Freighter\r\n",
      "737-400 Mixed Configuration\r\n",
      "737-700 Mixed Configuration\r\n",
      "737-200 Advanced\r\n",
      "737-700 (winglets)\r\n",
      "737-200 Freighter\r\n",
      "737-300 Freighter\r\n",
      "737-400 Freighter\r\n",
      "747-100\r\n",
      "747-200\r\n",
      "747-300\r\n",
      "747-400\r\n",
      "747\r\n",
      "747-400 Swingtail Freighter\r\n",
      "747-200 Combi\r\n",
      "747-300 Combi\r\n",
      "747-400 Combi\r\n",
      "747 Freighter\r\n",
      "747-8i\r\n",
      "747-400 (Domestic)\r\n",
      "747SP\r\n",
      "747 Combi\r\n",
      "747-8F\r\n",
      "747SR\r\n",
      "747-100 Freighter\r\n",
      "747-300 / 747-200 SUD Freighter\r\n",
      "747SR Freighter\r\n",
      "747-200 Freighter\r\n",
      "747-400 Freighter\r\n",
      "757-200\r\n",
      "757-300\r\n",
      "757\r\n",
      "757 Freighter\r\n",
      "757 Mixed\r\n",
      "757-300 (winglets)\r\n",
      "757-200 (winglets)\r\n",
      "767-200\r\n",
      "767-300\r\n",
      "767-400\r\n",
      "767\r\n",
      "767 Freighter\r\n",
      "767-300 (winglets) Freighter\r\n",
      "767-300 (winglets)\r\n",
      "767-200 Freighter\r\n",
      "767-300 Freighter\r\n",
      "777-200\r\n",
      "777-300\r\n",
      "777\r\n",
      "777 Freighter\r\n",
      "777-200LR\r\n",
      "777-300ER\r\n",
      "777-200F Freighter\r\n",
      "787-3\r\n",
      "787\r\n",
      "787-8\r\n",
      "787-9\r\n"
     ]
    }
   ],
   "source": [
    "! cut -d \"^\" -f 3 optd_aircraft.csv | grep -E \"7[0-9]7\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use grep to extract all 3xx (where x can be any number) airplane models from optd_aircraft.csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "146-300\r\n",
      "146 Freighter (-100/200/300QT & QC)\r\n",
      "146 Freighter (-300QT & QC)\r\n",
      "A319\r\n",
      "A310\r\n",
      "A310-200\r\n",
      "A310-300\r\n",
      "A318\r\n",
      "A319\r\n",
      "A310F\r\n",
      "A319 (Sharklets)\r\n",
      "A310-200F\r\n",
      "A310-300F\r\n",
      "A320-100/200\r\n",
      "A321-100/200\r\n",
      "A320 (sharklets)\r\n",
      "A321 (sharklets)\r\n",
      "A318 (sharklets)\r\n",
      "A319 (sharklets)\r\n",
      "A320 Freighter\r\n",
      "A321 Freighter\r\n",
      "A330\r\n",
      "A330-200\r\n",
      "A330-300\r\n",
      "A330 Freighter\r\n",
      "A330-200 Freighter\r\n",
      "A340\r\n",
      "A340-200\r\n",
      "A340-300\r\n",
      "A340-500\r\n",
      "A340-600\r\n",
      "A350\r\n",
      "A350-1000\r\n",
      "A350-800\r\n",
      "A350-900\r\n",
      "A380\r\n",
      "A380\r\n",
      "A380F\r\n",
      "707-300\r\n",
      "737-300\r\n",
      "737-300 (winglets)\r\n",
      "737-300 Mixed Configuration\r\n",
      "737-300 Freighter\r\n",
      "747-300\r\n",
      "747-300 Combi\r\n",
      "747-300 / 747-200 SUD Freighter\r\n",
      "757-300\r\n",
      "757-300 (winglets)\r\n",
      "767-300\r\n",
      "767-300 (winglets) Freighter\r\n",
      "767-300 (winglets)\r\n",
      "767-300 Freighter\r\n",
      "777-300\r\n",
      "777-300ER\r\n",
      "A300B2\r\n",
      "A300\r\n",
      "A300B2/B4/C4\r\n",
      "A300-600\r\n",
      "A300-600ST Beluga\r\n",
      "A300F\r\n",
      "A300-600 Mixed Configuration\r\n",
      "A300C4/F4 Freighter\r\n",
      "A300-600F\r\n",
      "SA330 Puma / AS332 Super Puma\r\n",
      "ATR 42-300 / 320\r\n",
      "One Eleven 300\r\n",
      "Challenger 300\r\n",
      "CS300\r\n",
      "Do 328\r\n",
      "DHC-8-300 Dash 8 / 8Q\r\n",
      "EMB-505 Phenom 300\r\n",
      "F28 Fellowship 3000\r\n",
      "328JET\r\n",
      "328JET\r\n",
      "H-250 Courier / H-295 / 385 Super Courie\r\n",
      "IL-96-300\r\n",
      "IL-96-300 Freighter\r\n",
      "L-182 / 282 / 382 (L-100) Hercules\r\n",
      "AS350 Ecureuil / AS355 Ecureuil 2\r\n",
      "SA365C / SA365N Dauphin 2\r\n",
      "Hawker 390 Premier 1/1A\r\n",
      "SF340A/B\r\n",
      "SF340B\r\n",
      "SF340 Freighter\r\n",
      "SD.330\r\n",
      "SD.360\r\n",
      "Tu-334\r\n"
     ]
    }
   ],
   "source": [
    "! cut -d \"^\" -f 3 optd_aircraft.csv | grep -E \"3[0-9][0-9]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use grep to obtain the number of airlines with prefix “aero” (case insensitive) in their name from optd_airlines.csv . Hint: Name is in column 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pk^env_id^validity_from^validity_to^3char_code^2char_code^num_code^name^name2^alliance_code^alliance_status^type^wiki_link^flt_freq\n",
      " \n",
      "29\n"
     ]
    }
   ],
   "source": [
    "! head -n 1 optd_airlines.csv\n",
    "! echo \" \"\n",
    "\n",
    "! cut -d \"^\" -f 8 optd_airlines.csv | grep -Ei \"^aero\" | sort | uniq | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How many optd_por_public.csv columns have “name” as part of their name? What are their numerical positions? (hint: use seqand paste)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4:is_geonames\r\n",
      "5:geoname_id\r\n",
      "7:name\r\n",
      "8:asciiname\r\n",
      "19:country_name\r\n",
      "20:continent_name\r\n",
      "22:adm1_name_utf\r\n",
      "23:adm1_name_ascii\r\n",
      "25:adm2_name_utf\r\n",
      "26:adm2_name_ascii\r\n",
      "38:city_name_list\r\n",
      "44:alt_name_section\r\n",
      "46:wac_name\r\n"
     ]
    }
   ],
   "source": [
    "! head optd_por_public.csv -n 1 |  tr \"^\" \"\\n\" | grep -nE \"name\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Find all files with txt extension inside home directory (including all sub directories) that have word“Science” (case insensitive) inside the content. Print file path and the line containing the (S/s)cienceword. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/dsc/Data/shell/Finn.txt:5499:Science of Phrenology” at such and such a place, on the blank day of\n",
      "/home/dsc/.config/google-chrome/Subresource Filter/Unindexed Rules/9.1/LICENSE.txt:109:       architecture or science; a performance; a broadcast; a phonogram; a\n",
      "/home/dsc/anaconda3/pkgs/scipy-1.0.0-py36hbf646e7_0/lib/python3.6/site-packages/scipy/THANKS.txt:1:SciPy is an open source library of routines for science and engineering\n",
      "/home/dsc/anaconda3/pkgs/scipy-1.0.0-py36hbf646e7_0/lib/python3.6/site-packages/scipy/spatial/qhull/COPYING.txt:8:       The National Science and Technology Research Center for\n",
      "/home/dsc/anaconda3/lib/python3.6/site-packages/scipy/THANKS.txt:1:SciPy is an open source library of routines for science and engineering\n",
      "/home/dsc/anaconda3/lib/python3.6/site-packages/scipy/spatial/qhull/COPYING.txt:8:       The National Science and Technology Research Center for\n"
     ]
    }
   ],
   "source": [
    "! find ~ -name \"*.txt\" -exec grep -iwHn \"Science\" {} \\;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:right\">\n",
    "Juan Luis García López (@huanlui)\n",
    "<a href=\"https://github.com/huanlui\" class=\"fa fa-github\"> Github </a>\n",
    "<a href=\"https://twitter.com/huanlui\" class=\"fa fa-twitter\"> Twitter </a>\n",
    "<a href=\"https://www.linkedin.com/in/juan-luis-garcía-lópez-99057138\" class=\"fa fa-linkedin\"> Linkedin </a>\n",
    "<div>\n",
    "   \n",
    "<div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}