NER pipeline with Spacy & HuggingFace transformers.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"NER pipeline with Spacy & HuggingFace transformers.ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyMI5FFXqHifO5w2P/MlUfBq"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"QbQk4kWIAtNb"},"source":["## Named-entity recognition pipeline with ready-made solutions out of the box."]},{"cell_type":"code","metadata":{"id":"FEv_kErp0WPw","executionInfo":{"status":"ok","timestamp":1638746170701,"user_tz":-180,"elapsed":514,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}}},"source":["# Install necessary libraries for NER\n","# !pip install spacy==3.2     \n","# !pip install spacy_transformers"],"execution_count":14,"outputs":[]},{"cell_type":"code","metadata":{"id":"-2zCuHMVkqlP","executionInfo":{"status":"ok","timestamp":1638746196273,"user_tz":-180,"elapsed":4618,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}}},"source":["import pandas as pd\n","import numpy as np\n","\n","import warnings\n","warnings.filterwarnings('ignore')\n","\n","import spacy\n","from spacy import displacy"],"execution_count":1,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":35},"id":"P8jVUfCKn5P5","executionInfo":{"status":"ok","timestamp":1638747665154,"user_tz":-180,"elapsed":277,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}},"outputId":"dbf853ac-cd61-4bb5-cc49-c8dc7838254a"},"source":["# Check spacy version\n","spacy.__version__"],"execution_count":39,"outputs":[{"output_type":"execute_result","data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["'3.2.0'"]},"metadata":{},"execution_count":39}]},{"cell_type":"markdown","metadata":{"id":"8PDAivC20xYU"},"source":["#### Download latest transformer-based pipeline from scapy library\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"3GSXzWTOl_KA","executionInfo":{"status":"ok","timestamp":1638748155438,"user_tz":-180,"elapsed":3795,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}},"outputId":"4fc62fa8-10bd-4646-e8ce-2270aaac10c7"},"source":["# English transformer pipeline (roBERTa-base).\n","spacy.load(\"en_core_web_trf\")"],"execution_count":51,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<spacy.lang.en.English at 0x7fce1d2a79d0>"]},"metadata":{},"execution_count":51}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":752},"id":"h_v0ndFLlJ0n","executionInfo":{"status":"ok","timestamp":1638748607934,"user_tz":-180,"elapsed":1610,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}},"outputId":"cc8cc20a-0af7-4813-9423-f816df5cb9ec"},"source":["email_body = \"\"\"\n","HI Mr.Anderson,\n","\n","Hope the email finds you well! I am sending this again just in case my last email was missed.\n","\n","I am looking for ML tech leads/lead engineers/senior engineers for Meta's rapid growth. I'd love to tell you a bit about our ML plans here specifically in our London office. If relocation is required, Meta will sponsor work visa and full relo support for your family to start anytime in 2022.\n","Our projects include working with some of the largest deployments of ML globally across our advertising teams, giving you license to experiment with our traffic to make improvements that could effect what our 2+ billion users see on Facebook.\n","We also have teams that are utilizing ML/NLP/CV to combat fake profiles, false information, spam, online abuse/bullying and hate speech. They run ML algorithms on top of a system which is the largest Haskell based system in production globally. We use this system to identifying users at risk of harming themselves or others based on their online behavior.\n","\n","Furthermore, we have open source projects Pytorch, Papers with Code and Facebook Shops (our ecommerce platform) teams expanding as well. :)\n","It's a ML focus role containing ~50% applied research and 50% engineering/building ML products that can impact billions of people and push the state of the art AI. You can find more about our ML work here: https://www.facebook.com/careers/life/machine-learning-at-facebook\n","\n","Please let me know if you are interested, happy to give you more details!\n","\n","Thanks, John Doe\n","Facebook\n","\"\"\"\n","\n","doc = nlp(email_body)\n","\n","# Selecting only person entities\n","org_results = [ent.text for ent in doc.ents if ent.label_ == \"ORG\"]\n","person_results = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n","\n","# Displaying result\n","print(\"Found Names:\", set(person_results))\n","print(\"Found Organizations:\", set(org_results))\n","\n","# Entity visualization\n","displacy.render(doc,style=\"ent\",jupyter=True)"],"execution_count":58,"outputs":[{"output_type":"stream","name":"stdout","text":["Found Names: {'John Doe', 'Anderson'}\n","Found Organizations: {'Facebook', 'Meta'}\n"]},{"output_type":"display_data","data":{"text/html":["<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\"></br>HI Mr.\n","<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Anderson\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n","</mark>\n",",</br></br>Hope the email finds you well! I am sending this again just in case my last email was missed.</br></br>I am looking for ML tech leads/lead engineers/senior engineers for \n","<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Meta\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n","</mark>\n","'s rapid growth. I'd love to tell you a bit about our ML plans here specifically in our \n","<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    London\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n","</mark>\n"," office. If relocation is required, \n","<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Meta\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n","</mark>\n"," will sponsor work visa and full relo support for your family to start anytime in \n","<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    2022\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n","</mark>\n",".</br>Our projects include working with some of the largest deployments of ML globally across our advertising teams, giving you license to experiment with our traffic to make improvements that could effect what our \n","<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    2+ billion\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n","</mark>\n"," users see on \n","<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Facebook\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n","</mark>\n",".</br>We also have teams that are utilizing ML/NLP/CV to combat fake profiles, false information, spam, online abuse/bullying and hate speech. They run ML algorithms on top of a system which is the largest Haskell based system in production globally. We use this system to identifying users at risk of harming themselves or others based on their online behavior.</br></br>Furthermore, we have open source projects \n","<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Pytorch\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n","</mark>\n",", \n","<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Papers with Code\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n","</mark>\n"," and Facebook Shops (our ecommerce platform) teams expanding as well. :)</br>It's a ML focus role containing \n","<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    ~50%\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERCENT</span>\n","</mark>\n"," applied research and \n","<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    50%\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERCENT</span>\n","</mark>\n"," engineering/building ML products that can impact \n","<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    billions\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n","</mark>\n"," of people and push the state of the art AI. You can find more about our ML work here: https://www.facebook.com/careers/life/machine-learning-at-facebook</br></br>Please let me know if you are interested, happy to give you more details!</br></br>Thanks, \n","<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    John Doe\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n","</mark>\n","</br>\n","<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n","    Facebook\n","    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n","</mark>\n","</br></div></span>"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{}}]},{"cell_type":"markdown","metadata":{"id":"pVx7mk1zlJ8W"},"source":["#### BERT Transformer application for QnA (based on SQuAd question-answering dataset)"]},{"cell_type":"code","metadata":{"id":"Jd223H7alKCl","executionInfo":{"status":"ok","timestamp":1638748644245,"user_tz":-180,"elapsed":10019,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}}},"source":["from transformers import pipeline\n","\n","# Instatiate the model from checkpoint\n","model_checkpoint = \"bert-large-uncased-whole-word-masking-finetuned-squad\"\n","model = pipeline(\n","    'question-answering',\n","    model=model_checkpoint,\n","    tokenizer=model_checkpoint\n",")"],"execution_count":59,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nsp3i6UuDXn0"},"source":["#### Let's check how BERT transformer-based model can handle with our questions"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"s2SJtH0dlKIn","executionInfo":{"status":"ok","timestamp":1638749150450,"user_tz":-180,"elapsed":21421,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}},"outputId":"1007e652-89e6-4a99-e167-771813ff0e21"},"source":["# Our questions\n","questions = [\n","   \"What is the recruiter name?\",\n","   \"Who is recipient?\",\n","   \"What is the company?\",\n","   \"What is the position?\",          \n","   \"What is location for this position?\"          \n","]\n","\n","\n","answers = model(\n","    context=email_body,\n","    question=questions,\n","    topk=1 # Gives 1 answer per question with highest score\n",")\n","\n","answers"],"execution_count":68,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[{'answer': 'John Doe',\n","  'end': 1512,\n","  'score': 0.6521063446998596,\n","  'start': 1504},\n"," {'answer': 'Mr.Anderson',\n","  'end': 15,\n","  'score': 0.25034067034721375,\n","  'start': 4},\n"," {'answer': 'Meta', 'end': 184, 'score': 0.41864168643951416, 'start': 180},\n"," {'answer': 'ML tech leads/lead engineers/senior engineers',\n","  'end': 175,\n","  'score': 0.1773664504289627,\n","  'start': 130},\n"," {'answer': 'London', 'end': 278, 'score': 0.6946215629577637, 'start': 272}]"]},"metadata":{},"execution_count":68}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"dVupf7u9E9cr","executionInfo":{"status":"ok","timestamp":1638749893325,"user_tz":-180,"elapsed":277,"user":{"displayName":"Pavel Kondratev","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjUt4MocKFp8rsDN-iR9pLUIH0uQ6XRSzUwte-h1w=s64","userId":"16072367302414325382"}},"outputId":"e4c6c20a-8f1d-4d8c-a94e-6c3fed10ac68"},"source":["# Summarize answers for our questions\n","result = pd.DataFrame()\n","result['Questions'] = questions\n","result['Answers'] = [d['answer'] for d in answers if 'answer' in d]\n","result['Relevance score'] =  [d['score'] for d in answers if 'score' in d]\n","result"],"execution_count":86,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Questions</th>\n","      <th>Answers</th>\n","      <th>Relevance score</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>What is the recruiter name?</td>\n","      <td>John Doe</td>\n","      <td>0.652106</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Who is recipient?</td>\n","      <td>Mr.Anderson</td>\n","      <td>0.250341</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>What is the company?</td>\n","      <td>Meta</td>\n","      <td>0.418642</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>What is the position?</td>\n","      <td>ML tech leads/lead engineers/senior engineers</td>\n","      <td>0.177366</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>What is location for this position?</td>\n","      <td>London</td>\n","      <td>0.694622</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                             Questions  ... Relevance score\n","0          What is the recruiter name?  ...        0.652106\n","1                    Who is recipient?  ...        0.250341\n","2                 What is the company?  ...        0.418642\n","3                What is the position?  ...        0.177366\n","4  What is location for this position?  ...        0.694622\n","\n","[5 rows x 3 columns]"]},"metadata":{},"execution_count":86}]}]}