produce_json_from_pdf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e496c8fc-49b7-49ff-945a-d5d09884355a",
   "metadata": {},
   "source": [
    "# Machine Readable Text of the Kenya 2010 Constitution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ea06882-8edd-4567-8ac4-33e914fa025c",
   "metadata": {},
   "source": [
    "In this notebook, we're going to produce a machine-readable version of the Kenya 2010 constitution. We'll use a PDF version of the constitution from the [Kenya Law][0] website as the source.\n",
    "\n",
    "This version was last revised in 2022 and published by the Government Printer, so we can have a high degree of confidence on its integrity.\n",
    "\n",
    "[0]: https://kenyalaw.org/kl/fileadmin/pdfdownloads/TheConstitutionOfKenya.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7f6089a-2fcc-4690-bc16-f7e795b323ea",
   "metadata": {},
   "source": [
    "First step is to download it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fe5c2ba5-a723-4e68-8405-c996822987a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"TheConstitutionOfKenya.pdf\"\n",
    "!mkdir -p pdf\n",
    "![ ! -f \"pdf/{file_name}\" ] && wget \"https://kenyalaw.org/kl/fileadmin/pdfdownloads/TheConstitutionOfKenya.pdf\" -O \"pdf/{file_name}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3577e777-8e1b-45f1-a404-06f984fcebfe",
   "metadata": {},
   "source": [
    "We'll use [pypdf][3] to extract the text. We install it and load the pdf:\n",
    "\n",
    "[3]: https://pypi.org/project/pypdf/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bd2401bc-dd22-4227-9225-2fd642d97474",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --quiet pypdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "45b1e666-ffc6-48ff-8a29-d58a8394dc94",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pypdf import PdfReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b116b506-1f38-4ee6-8508-672e28e65f1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = PdfReader('pdf/TheConstitutionOfKenya.pdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5e35d85b-3692-4148-8f58-f53260c07105",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "166"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(reader.pages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f11559-3395-4eb3-8135-8a96ffb38f1d",
   "metadata": {},
   "source": [
    "The pdf file has 166 pages. We can print a sample of extracted text from page 23:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3e98269c-de9b-464a-9a21-aaf2538028ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Rev. 2022] Constitution of Kenya 23\n",
      "Human dignity.\n",
      "28.  Every person has inherent dignity and the right to have that dignity respected\n",
      "and protected.\n",
      "Freedom and security of the person.\n",
      "29.  Every person has the right to freedom and security of the person, which\n",
      "includes the right not to be—\n",
      "(a) deprived of freedom arbitrarily or without just cause;\n",
      "(b) detained without trial, except during a state of emergency, in which\n",
      "case the detention is subject to Article 58;\n",
      "(c) subjected to any form of violence from either public or private sources;\n",
      "(d) subjected to torture in any manner, whether physical or psychological;\n",
      "(e) subjected to corporal punishment; or\n",
      "(f)treated or punished in a cruel, inhuman or degrading manner.\n",
      "Slavery, servitude and forced labour.\n",
      "30.   (1)  A person shall not be held in slavery or servitude.\n",
      "(2)  A person shall not be required to perform forced labour.\n",
      "Privacy.\n",
      "31.  Every person has the right to privacy, which includes the right not to have—\n",
      "(a) their person, home or property searched;\n",
      "(b) their possessions seized;\n",
      "(c) information relating to their family or private affairs unnecessarily\n",
      "required or revealed; or\n",
      "(d) the privacy of their communications infringed.\n",
      "Freedom of conscience, religion, belief and opinion.\n",
      "32.   (1)  Every person has the right to freedom of conscience, religion, thought,\n",
      "belief and opinion.\n",
      "(2)  Every person has the right, either individually or in community with others,\n",
      "in public or in private, to manifest any religion or belief through worship, practice,\n",
      "teaching or observance, including observance of a day of worship.\n",
      "(3)  A person may not be denied access to any institution, employment or facility,\n",
      "or the enjoyment of any right, because of the person’s belief or religion.\n",
      "(4)  A person shall not be compelled to act, or engage in any act, that is contrary\n",
      "to the person’s belief or religion.\n",
      "Freedom of expression.\n",
      "33.   (1)  Every person has the right to freedom of expression, which includes—\n",
      "(a) freedom to seek, receive or impart information or ideas;\n",
      "(b) freedom of artistic creativity; and\n",
      "(c) academic freedom and freedom of scientiﬁc research.\n",
      "(2)  The right to freedom of expression does not extend to—\n",
      "(a) propaganda for war;\n"
     ]
    }
   ],
   "source": [
    "sample = reader.pages[22]\n",
    "\n",
    "print(sample.extract_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "317ad112-8419-490e-9a4f-27a99be71b3c",
   "metadata": {},
   "source": [
    "We extract text from all the pages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a144a55f-f642-418b-aa94-63146d6257d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "page_texts = [p.extract_text() for p in reader.pages]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c058551-7f93-466b-9d46-75f16c473bbb",
   "metadata": {},
   "source": [
    "The constitution is divided into chapters, under which we have the specific articles. The articles are 264 in total. These are in pages 14 to 123. So we narrow down pages to this range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e8a3e3b9-63f0-48a0-8816-0605cb85f6df",
   "metadata": {},
   "outputs": [],
   "source": [
    "relevant_text = \"\\n\".join(page_texts[13:123])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9500126b-f93b-48b5-95a0-5eedf3b58c6d",
   "metadata": {},
   "source": [
    "We the break down the text into lines as extracted from the pdf file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d526bdc8-bead-4280-bf1e-93ac643bceb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "\n",
    "mem_file = io.StringIO(relevant_text)\n",
    "all_lines = [l for l in mem_file]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a6994ae5-b741-4276-8ca5-cdcbccac0982",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4657"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a33c733-b61c-44c9-a2b0-c0ea1fe3a3b1",
   "metadata": {},
   "source": [
    "Some exploration indicates that the articles each start from a new line with a pattern like `31.  Every person has the right to privacy...`; that is, a number followed by a period followed by some whitespace, then non-whitespace characters. We create a regular expression for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "50a65ab2-035f-41ee-896b-5cf8cba8497d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "article_num_pattern = re.compile(r'(\\d+)\\.\\s+\\S+')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df1c035d-7e2b-4916-975d-1f092b035f6d",
   "metadata": {},
   "source": [
    "We can extract the text like below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1ef93a12-0dae-4387-abb6-6e7472a6c56a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'31'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article_num_pattern.match('31.  Every person has the right to privacy').group(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6bc600f-de47-4b5e-ba16-a6c3d271f835",
   "metadata": {},
   "source": [
    "Running the pattern through all the lines will identify 264 of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eebff64c-3892-4f08-aff0-b83be88f3815",
   "metadata": {},
   "outputs": [],
   "source": [
    "article_nums = []\n",
    "\n",
    "for line in all_lines:\n",
    "    line = line.strip()\n",
    "    match = article_num_pattern.match(line)\n",
    "    if match:\n",
    "        article_nums.append(int(match.group(1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c9328d19-8711-45ba-87d4-608db09ee824",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "264"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(article_nums)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04353526-2924-48c3-9ef4-843b7b2735a4",
   "metadata": {},
   "source": [
    "Validating that numbers range from 1 to 264:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bb9c2596-1fd2-4155-aea2-818ea62a026e",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert [i+1 for i in range(264)] == article_nums"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daea7d35-07f6-417c-bdc6-9bd5d37c5003",
   "metadata": {},
   "source": [
    "Further observation shows that the title of each article seems to precede the line with the number. To validate this, we'll pair each article number with the text of the previous line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5ae60d1f-6bde-4c49-909d-e4518a34e5c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = []\n",
    "\n",
    "for i, line in enumerate(all_lines):\n",
    "    line = line.strip()\n",
    "    match = article_num_pattern.match(line)\n",
    "    if match:\n",
    "        articles.append((int(match.group(1)), all_lines[i-1].strip()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f071cde0-e56a-45d4-8fd3-4f42921db422",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 'Sovereignty of the people.'),\n",
       " (2, 'Supremacy of this Constitution.'),\n",
       " (3, 'Defence of this Constitution.'),\n",
       " (4, 'Declaration of the Republic.'),\n",
       " (5, 'Territory of Kenya.'),\n",
       " (6, 'Devolution and access to services.'),\n",
       " (7, 'National, ofﬁcial and other languages.'),\n",
       " (8, 'State and religion.'),\n",
       " (9, 'National symbols and national days.'),\n",
       " (10, 'National values and principles of governance.'),\n",
       " (11, 'Culture.'),\n",
       " (12, 'Entitlements of citizens.'),\n",
       " (13, 'Retention and acquisition of citizenship.'),\n",
       " (14, 'Citizenship by birth.'),\n",
       " (15, 'Citizenship by registration.'),\n",
       " (16, 'Dual citizenship.'),\n",
       " (17, 'Revocation of citizenship.'),\n",
       " (18, 'Legislation on citizenship.'),\n",
       " (19, 'Rights and fundamental freedoms.'),\n",
       " (20, 'Application of Bill of Rights.'),\n",
       " (21, 'Implementation of rights and fundamental freedoms.'),\n",
       " (22, 'Enforcement of Bill of Rights.'),\n",
       " (23, 'Authority of courts to uphold and enforce the Bill of Rights.'),\n",
       " (24, 'Limitation of rights and fundamental freedoms.'),\n",
       " (25, 'Fundamental Rights and freedoms that may not be limited.'),\n",
       " (26, 'Right to life.'),\n",
       " (27, 'Equality and freedom from discrimination.'),\n",
       " (28, 'Human dignity.'),\n",
       " (29, 'Freedom and security of the person.'),\n",
       " (30, 'Slavery, servitude and forced labour.'),\n",
       " (31, 'Privacy.'),\n",
       " (32, 'Freedom of conscience, religion, belief and opinion.'),\n",
       " (33, 'Freedom of expression.'),\n",
       " (34, 'Freedom of the media.'),\n",
       " (35, 'Access to information.'),\n",
       " (36, 'Freedom of association.'),\n",
       " (37, 'Assembly, demonstration, picketing and petition.'),\n",
       " (38, 'Political rights.'),\n",
       " (39, 'Freedom of movement and residence.'),\n",
       " (40, 'Protection of right to property.'),\n",
       " (41, 'Labour relations.'),\n",
       " (42, 'Environment.'),\n",
       " (43, 'Economic and social rights.'),\n",
       " (44, 'Language and culture.'),\n",
       " (45, 'Family.'),\n",
       " (46, 'Consumer rights.'),\n",
       " (47, 'Fair administrative action.'),\n",
       " (48, 'Access to justice.'),\n",
       " (49, 'Rights of arrested persons.'),\n",
       " (50, 'Fair hearing.'),\n",
       " (51, 'Rights of persons detained, held in custody or imprisoned.'),\n",
       " (52, 'Interpretation of this Part.'),\n",
       " (53, 'Children.'),\n",
       " (54, 'Persons with disabilities.'),\n",
       " (55, 'Youth.'),\n",
       " (56, 'Minorities and marginalised groups.'),\n",
       " (57, 'Older members of society.'),\n",
       " (58, 'State of emergency.'),\n",
       " (59, 'Kenya National Human Rights and Equality Commission.'),\n",
       " (60, 'Principles of land policy.'),\n",
       " (61, 'Classiﬁcation of land.'),\n",
       " (62, 'Public land.'),\n",
       " (63, 'Community land.'),\n",
       " (64, 'Private land.'),\n",
       " (65, 'Landholding by non-citizens.'),\n",
       " (66, 'Regulation of land use and property.'),\n",
       " (67, 'National Land Commission.'),\n",
       " (68, 'Legislation on land.'),\n",
       " (69, 'Obligations in respect of the environment.'),\n",
       " (70, 'Enforcement of environmental rights.'),\n",
       " (71, 'Agreements relating to natural resources.'),\n",
       " (72, 'Legislation relating to the environment.'),\n",
       " (73, 'Responsibilities of leadership.'),\n",
       " (74, 'Oath of ofﬁce of State ofﬁcers.'),\n",
       " (75, 'Conduct of State ofﬁcers.'),\n",
       " (76, 'Financial probity of State ofﬁcers.'),\n",
       " (77, 'Restriction on activities of State ofﬁcers.'),\n",
       " (78, 'Citizenship and leadership.'),\n",
       " (79, 'Legislation to establish the ethics and anti-corruption commission.'),\n",
       " (80, 'Legislation on leadership.'),\n",
       " (81, 'General principles for the electoral system.'),\n",
       " (82, 'Legislation on elections.'),\n",
       " (83, 'Registration as a voter.'),\n",
       " (84,\n",
       "  'Candidates for election and political parties to comply with code of conduct.'),\n",
       " (85, 'Eligibility to stand as an independent candidate.'),\n",
       " (86, 'Voting.'),\n",
       " (87, 'Electoral disputes.'),\n",
       " (88, 'Independent Electoral and Boundaries Commission.'),\n",
       " (89, 'Delimitation of electoral units.'),\n",
       " (90, 'Allocation of party list seats.'),\n",
       " (91, 'Basic requirements for political parties.'),\n",
       " (92, 'Legislation on political parties.'),\n",
       " (93, 'Establishment of Parliament.'),\n",
       " (94, 'Role of Parliament.'),\n",
       " (95, 'Role of the National Assembly.'),\n",
       " (96, 'Role of the Senate.'),\n",
       " (97, 'Membership of the National Assembly.'),\n",
       " (98, 'Membership of the Senate.'),\n",
       " (99,\n",
       "  'Qualiﬁcations and disqualiﬁcations for election as member of Parliament.'),\n",
       " (100, 'Promotion of representation of marginalised groups.'),\n",
       " (101, 'Election of members of Parliament.'),\n",
       " (102, 'Term of Parliament.'),\n",
       " (103, 'Vacation of ofﬁce of member of Parliament.'),\n",
       " (104, 'Right of recall.'),\n",
       " (105, 'Determination of questions of membership.'),\n",
       " (106, 'Speakers and Deputy Speakers of Parliament.'),\n",
       " (107, 'Presiding in Parliament.'),\n",
       " (108, 'Party leaders.'),\n",
       " (109, 'Exercise of legislative powers.'),\n",
       " (110, 'Bills concerning county government.'),\n",
       " (111, 'Special Bills concerning county governments.'),\n",
       " (112, 'Ordinary Bills concerning county governments.'),\n",
       " (113, 'Mediation committees.'),\n",
       " (114, 'Money Bills.'),\n",
       " (115, 'Presidential assent and referral.'),\n",
       " (116, 'Coming into force of laws.'),\n",
       " (117, 'Powers, privileges and immunities.'),\n",
       " (118, 'Public access and participation.'),\n",
       " (119, 'Right to petition Parliament.'),\n",
       " (120, 'Ofﬁcial languages of Parliament.'),\n",
       " (121, 'Quorum.'),\n",
       " (122, 'Voting in Parliament.'),\n",
       " (123, 'Decisions of Senate.'),\n",
       " (124, 'Committees and Standing Orders.'),\n",
       " (125, 'Power to call for evidence.'),\n",
       " (126, 'Location of sittings of Parliament.'),\n",
       " (127, 'Parliamentary Service Commission.'),\n",
       " (128, 'Clerks and staff of Parliament.'),\n",
       " (129, 'Principles of executive authority.'),\n",
       " (130, 'The National Executive.'),\n",
       " (131, 'Authority of the President.'),\n",
       " (132, 'Functions of the President.'),\n",
       " (133, 'Power of mercy.'),\n",
       " (134, 'Exercise of presidential powers during temporary incumbency.'),\n",
       " (135, 'Decisions of the President.'),\n",
       " (136, 'Election of the President.'),\n",
       " (137, 'Qualiﬁcations and disqualiﬁcations for election as President.'),\n",
       " (138, 'Procedure at presidential election.'),\n",
       " (139, 'Death before assuming ofﬁce.'),\n",
       " (140, 'Questions as to validity of presidential election.'),\n",
       " (141, 'Assumption of ofﬁce of President.'),\n",
       " (142, 'Term of ofﬁce of President.'),\n",
       " (143, 'Protection from legal proceedings.'),\n",
       " (144, 'Removal of President on grounds of incapacity.'),\n",
       " (145, 'Removal of President by impeachment.'),\n",
       " (146, 'Vacancy in the ofﬁce of President.'),\n",
       " (147, 'Functions of the Deputy President.'),\n",
       " (148, 'Election and swearing-in of Deputy President.'),\n",
       " (149, 'Vacancy in the ofﬁce of Deputy President.'),\n",
       " (150, 'Removal of Deputy President.'),\n",
       " (151, 'Remuneration and beneﬁts of President and Deputy President.'),\n",
       " (152, 'Cabinet.'),\n",
       " (153, 'Decisions, responsibility and accountability of the Cabinet.'),\n",
       " (154, 'Secretary to the Cabinet.'),\n",
       " (155, 'Principal Secretaries.'),\n",
       " (156, 'Attorney-General.'),\n",
       " (157, 'Director of Public Prosecutions.'),\n",
       " (158, 'Removal and resignation of Director of Public Prosecutions.'),\n",
       " (159, 'Judicial authority.'),\n",
       " (160, 'Independence of the Judiciary.'),\n",
       " (161, 'Judicial ofﬁces and ofﬁcers.'),\n",
       " (162, 'System of courts.'),\n",
       " (163, 'Supreme Court.'),\n",
       " (164, 'Court of Appeal.'),\n",
       " (165, 'High Court.'),\n",
       " (166, 'Appointment of Chief Justice, Deputy Chief Justice and other judges.'),\n",
       " (167, 'Tenure of ofﬁce of the Chief Justice and other judges.'),\n",
       " (168, 'Removal from ofﬁce.'),\n",
       " (169, 'Subordinate courts.'),\n",
       " (170, 'Kadhis’ courts.'),\n",
       " (171, 'Establishment of the Judicial Service Commission.'),\n",
       " (172, 'Functions of the Judicial Service Commission.'),\n",
       " (173, 'Judiciary Fund.'),\n",
       " (174, 'Objects of devolution.'),\n",
       " (175, 'Principles of devolved government.'),\n",
       " (176, 'County governments.'),\n",
       " (177, 'Membership of county assembly.'),\n",
       " (178, 'Speaker of a county assembly.'),\n",
       " (179, 'County executive committees.'),\n",
       " (180, 'Election of county governor and deputy county governor.'),\n",
       " (181, 'Removal of a county governor.'),\n",
       " (182, 'Vacancy in the ofﬁce of county governor.'),\n",
       " (183, 'Functions of county executive committees.'),\n",
       " (184, 'Urban areas and cities.'),\n",
       " (185, 'Legislative authority of county assemblies.'),\n",
       " (186, 'Respective functions and powers of national and county governments.'),\n",
       " (187, 'Transfer of functions and powers between levels of government.'),\n",
       " (188, 'Boundaries of counties.'),\n",
       " (189, 'Cooperation between national and county governments.'),\n",
       " (190, 'Support for county governments.'),\n",
       " (191, 'Conﬂict of laws.'),\n",
       " (192, 'Suspension of a county government.'),\n",
       " (193, 'Qualiﬁcations for election as member of county assembly.'),\n",
       " (194, 'Vacation of ofﬁce of member of county assembly.'),\n",
       " (195, 'County assembly power to summon witnesses.'),\n",
       " (196,\n",
       "  'Public participation and county assembly powers, privileges and immunities.'),\n",
       " (197, 'County assembly gender balance and diversity.'),\n",
       " (198, 'County government during transition.'),\n",
       " (199, 'Publication of county legislation.'),\n",
       " (200, 'Legislation on Chapter.'),\n",
       " (201, 'Principles of public ﬁnance.'),\n",
       " (202, 'Equitable sharing of national revenue.'),\n",
       " (203, 'Equitable share and other ﬁnancial laws.'),\n",
       " (204, 'Equalisation Fund.'),\n",
       " (205, 'Consultation on ﬁnancial legislation affecting counties.'),\n",
       " (206, 'Consolidated Fund and other public funds.'),\n",
       " (207, 'Revenue Funds for county governments.'),\n",
       " (208, 'Contingencies Fund.'),\n",
       " (209, 'Power to impose taxes and charges.'),\n",
       " (210, 'Imposition of tax.'),\n",
       " (211, 'Borrowing by national government.'),\n",
       " (212, 'Borrowing by counties.'),\n",
       " (213, 'Loan guarantees by national government.'),\n",
       " (214, 'Public debt.'),\n",
       " (215, 'Commission on Revenue Allocation.'),\n",
       " (216, 'Functions of the Commission on Revenue Allocation.'),\n",
       " (217, 'Division of revenue.'),\n",
       " (218, 'Annual Division and Allocation of Revenue Bills.'),\n",
       " (219, 'Transfer of equitable share.'),\n",
       " (220, 'Form, content and timing of budgets.'),\n",
       " (221, 'Budget estimates and annual Appropriation Bill.'),\n",
       " (222, 'Expenditure before annual budget is passed.'),\n",
       " (223, 'Supplementary appropriation.'),\n",
       " (224, 'County appropriation Bills.'),\n",
       " (225, 'Financial control.'),\n",
       " (226, 'Accounts and audit of public entities.'),\n",
       " (227, 'Procurement of public goods and services.'),\n",
       " (228, 'Controller of Budget.'),\n",
       " (229, 'Auditor-General.'),\n",
       " (230, 'Salaries and Remuneration Commission.'),\n",
       " (231, 'Central Bank of Kenya.'),\n",
       " (232, 'Values and principles of public service.'),\n",
       " (233, 'The Public Service Commission.'),\n",
       " (234, 'Functions and powers of the Public Service Commission.'),\n",
       " (235, 'Stafﬁng of county governments.'),\n",
       " (236, 'Protection of public ofﬁcers.'),\n",
       " (237, 'Teachers Service Commission.'),\n",
       " (238, 'Principles of national security.'),\n",
       " (239, 'National security organs.'),\n",
       " (240, 'Establishment of the National Security Council.'),\n",
       " (241, 'Establishment of Kenya Defence Forces and Defence Council.'),\n",
       " (242, 'Establishment of National Intelligence Service.'),\n",
       " (243, 'Establishment of the National Police Service.'),\n",
       " (244, 'Objects and functions of the National Police Service.'),\n",
       " (245, 'Command of the National Police Service.'),\n",
       " (246, 'National Police Service Commission.'),\n",
       " (247, 'Other police services.'),\n",
       " (248, 'Application of Chapter.'),\n",
       " (249,\n",
       "  'Objects, authority and funding of commissions and independent ofﬁces.'),\n",
       " (250, 'Composition, appointment and terms of ofﬁce.'),\n",
       " (251, 'Removal from ofﬁce.'),\n",
       " (252, 'General functions and powers.'),\n",
       " (253, 'Incorporation of commissions and independent ofﬁces.'),\n",
       " (254, 'Reporting by commissions and independent ofﬁces.'),\n",
       " (255, 'Amendment of this Constitution.'),\n",
       " (256, 'Amendment by parliamentary initiative.'),\n",
       " (257, 'Amendment by popular initiative.'),\n",
       " (258, 'Enforcement of this Constitution.'),\n",
       " (259, 'Construing this Constitution.'),\n",
       " (260, 'Interpretation.'),\n",
       " (261, 'Consequential legislation.'),\n",
       " (262, 'Transitional and consequential provisions.'),\n",
       " (263, 'Effective Date.'),\n",
       " (264, 'Repeal of previous Constitution.')]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "005d8b08-a677-4a25-9802-3aa327fda6db",
   "metadata": {},
   "source": [
    "Cross checking with the pdf version, it looks like the articles and their titles were correctly identified."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73a0a7de-2957-4e62-a47c-9ec5a3ddcb17",
   "metadata": {},
   "source": [
    "The chapters all seem to start a new line in the form: `CHAPTER ONE`, `CHAPTER TWO` and so on, with the title in the succeeding line. We extract these and check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "753dfe96-35d6-425e-bb7d-48db508fa756",
   "metadata": {},
   "outputs": [],
   "source": [
    "chapter_pattern = re.compile(r'CHAPTER\\s+(\\w+)')\n",
    "\n",
    "chapters = []\n",
    "\n",
    "for i, line in enumerate(all_lines):\n",
    "    line = line.strip()\n",
    "    if chapter_pattern.match(line):\n",
    "        chapters.append((line, all_lines[i+1].strip()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2a893c37-46cc-44d2-b9ce-9fdb62c70af2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('CHAPTER ONE',\n",
       "  'SOVEREIGNTY OF THE PEOPLE AND SUPREMACY OF THIS CONSTITUTION'),\n",
       " ('CHAPTER TWO', 'THE REPUBLIC'),\n",
       " ('CHAPTER THREE', 'CITIZENSHIP'),\n",
       " ('CHAPTER FOUR', 'THE BILL OF RIGHTS'),\n",
       " ('CHAPTER FIVE', 'LAND AND ENVIRONMENT'),\n",
       " ('CHAPTER SIX', 'LEADERSHIP AND INTEGRITY'),\n",
       " ('CHAPTER SEVEN', 'REPRESENTATION OF THE PEOPLE'),\n",
       " ('CHAPTER EIGHT', 'THE LEGISLATURE'),\n",
       " ('CHAPTER NINE', 'THE EXECUTIVE'),\n",
       " ('CHAPTER TEN', 'JUDICIARY'),\n",
       " ('CHAPTER ELEVEN', 'DEVOLVED GOVERNMENT'),\n",
       " ('CHAPTER TWELVE', 'PUBLIC FINANCE'),\n",
       " ('CHAPTER THIRTEEN', 'THE PUBLIC SERVICE'),\n",
       " ('CHAPTER FOURTEEN', 'NATIONAL SECURITY'),\n",
       " ('CHAPTER FIFTEEN', 'COMMISSIONS AND INDEPENDENT OFFICES'),\n",
       " ('CHAPTER SIXTEEN', 'AMENDMENT OF THIS CONSTITUTION'),\n",
       " ('CHAPTER SEVENTEEN', 'GENERAL PROVISIONS'),\n",
       " ('CHAPTER EIGHTEEN', 'TRANSITIONAL AND CONSEQUENTIAL PROVISIONS')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chapters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691e013d-ffa8-4620-b2bf-843cf71bd83b",
   "metadata": {},
   "source": [
    "That seems to have correctly identified each chapter with its title."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c6f36e8-02e4-4018-bddc-88028f66c5c1",
   "metadata": {},
   "source": [
    "Some chapters are divided into parts. They are represented as lines of the form `PART 1 – ESTABLISHMENT AND ROLE OF PARLIAMENT`. We create a regex for this as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "85d42fa5-5ef7-484e-a434-ff7b249f09dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PART 1 – GENERAL PROVISIONS TO THE BILL OF RIGHTS\n",
      "PART 2 – RIGHTS AND FUNDAMENTAL FREEDOMS\n",
      "PART 3 – SPECIFIC APPLICATION OF RIGHTS\n",
      "PART 4 – STATE OF EMERGENCY\n",
      "PART 5 – KENYA NATIONAL HUMAN RIGHTS AND EQUALITY COMMISSION\n",
      "PART 1 – LAND\n",
      "PART 2 – ENVIRONMENT AND NATURAL RESOURCES\n",
      "PART 1 – ELECTORAL SYSTEM AND PROCESS\n",
      "PART 2 – INDEPENDENT ELECTORAL AND BOUNDARIES\n",
      "PART 3 – POLITICAL PARTIES\n",
      "PART 1 – ESTABLISHMENT AND ROLE OF PARLIAMENT\n",
      "PART 2 – COMPOSITION AND MEMBERSHIP OF PARLIAMENT\n",
      "PART 3 – OFFICES OF PARLIAMENT\n",
      "PART 4 – PROCEDURES FOR ENACTING LEGISLATION\n",
      "PART 5 – PARLIAMENT'S GENERAL PROCEDURES AND RULES\n",
      "PART 6 – MISCELLANEOUS\n",
      "PART 1 – PRINCIPLES AND STRUCTURE OF THE NATIONAL EXECUTIVE\n",
      "PART 2 – THE PRESIDENT AND DEPUTY PRESIDENT\n",
      "PART 3 – THE CABINET\n",
      "PART 4 – OTHER OFFICES\n",
      "PART 1 – JUDICIAL AUTHORITY AND LEGAL SYSTEM\n",
      "PART 2 – SUPERIOR COURTS\n",
      "PART 3 – SUBORDINATE COURTS\n",
      "PART 4 – JUDICIAL SERVICE COMMISSION\n",
      "PART 1 – OBJECTS AND PRINCIPLES OF DEVOLVED GOVERNMENT\n",
      "PART 2 – COUNTY GOVERNMENTS\n",
      "PART 3 – FUNCTIONS AND POWERS OF COUNTY GOVERNMENTS\n",
      "PART 4 – THE BOUNDARIES OF COUNTIES\n",
      "PART 5 – RELATIONSHIP BETWEEN GOVERNMENTS\n",
      "PART 6 – SUSPENSION OF COUNTY GOVERNMENTS\n",
      "PART 7 – GENERAL\n",
      "PART 2 – OTHER PUBLIC FUNDS\n",
      "PART 3 – REVENUE-RAISING AND THE PUBLIC DEBT\n",
      "PART 4 – REVENUE ALLOCATION\n",
      "PART 5 – BUDGETS AND SPENDING\n",
      "PART 6 – CONTROL OF PUBLIC MONEY\n",
      "PART 7 – FINANCIAL OFFICERS AND INSTITUTIONS\n",
      "PART 1 – VALUES AND PRINCIPLES OF PUBLIC SERVICE\n",
      "PART 2 – THE PUBLIC SERVICE COMMISSION\n",
      "PART 3 – TEACHERS SERVICE COMMISSION\n",
      "PART 1 – NATIONAL SECURITY ORGANS\n",
      "PART 2 – THE KENYA DEFENCE FORCES\n",
      "PART 3 – THE NATIONAL INTELLIGENCE SERVICE\n",
      "PART 4 – THE NATIONAL POLICE SERVICE\n"
     ]
    }
   ],
   "source": [
    "part_pattern = re.compile(r'PART\\s+(\\d+)\\s+–\\s+([\\s\\w]+)')\n",
    "\n",
    "for line in all_lines:\n",
    "    line = line.strip()\n",
    "    if part_pattern.match(line):\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e552f3d6-cbcc-462b-bd68-e24461ee2a92",
   "metadata": {},
   "source": [
    "At this point we've identified all the important sections. However, some lines represent the header of each page in the original doc. The headers include the pattern `[Rev. 2022]`, so we create a regexp for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f108eaaa-31e7-4075-84e5-c5e33cdd5068",
   "metadata": {},
   "outputs": [],
   "source": [
    "header_pattern = re.compile(r'\\[Rev\\.\\s+2022\\]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4c2966c8-3d55-4aca-a593-0dadc8100afb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 15       16 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 17       18 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 19       20 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 21       22 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 23       24 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 25       26 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 27       28 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 29       30 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 31       32 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 33       34 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 35       36 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 37       38 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 39       40 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 41       42 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 43       44 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 45       46 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 47       48 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 49       50 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 51       52 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 53       54 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 55       56 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 57       58 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 59       60 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 61       62 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 63       64 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 65       66 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 67       68 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 69       70 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 71       72 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 73       74 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 75       76 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 77       78 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 79       80 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 81       82 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 83       84 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 85       86 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 87       88 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 89       90 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 91       92 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 93       94 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 95       96 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 97       98 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 99       100 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 101       102 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 103       104 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 105       106 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 107       108 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 109       110 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 111       112 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 113       114 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 115       116 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 117       118 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 119       120 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 121       122 Constitution of Kenya [Rev. 2022]       [Rev. 2022] Constitution of Kenya 123       "
     ]
    }
   ],
   "source": [
    "for line in all_lines:\n",
    "    line = line.strip()\n",
    "    if header_pattern.search(line):\n",
    "        print(line, end=\"       \")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d34f1a9-5000-4eb2-85aa-01eebcdbfd7b",
   "metadata": {},
   "source": [
    "We can now exclude these lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8ece0839-8810-44f6-8eb6-33a87cdc1737",
   "metadata": {},
   "outputs": [],
   "source": [
    "relevant_lines = [l for l in all_lines if not header_pattern.search(l)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ee194d5a-649d-48de-b95a-2aafa2a5e2a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4547"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(relevant_lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "57e20b1e-7d3e-45e0-83cb-8dae66ed6172",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['CHAPTER ONE\\n',\n",
       " 'SOVEREIGNTY OF THE PEOPLE AND SUPREMACY OF THIS CONSTITUTION\\n',\n",
       " 'Sovereignty of the people.\\n',\n",
       " '1. \\xa0 (1)\\xa0\\xa0All sovereign power belongs to the people of Kenya and shall be exercised\\n',\n",
       " 'only in accordance with this Constitution.\\n',\n",
       " '(2)\\xa0\\xa0The people may exercise their sovereign power either directly or through their\\n',\n",
       " 'democratically elected representatives.\\n',\n",
       " '(3)\\xa0\\xa0Sovereign power under this Constitution is delegated to the following State\\n',\n",
       " 'organs, which shall perform their functions in accordance with this Constitution—\\n',\n",
       " '(a) Parliament and the legislative assemblies in the county governments;\\n',\n",
       " '(b) the national executive and the executive structures in the county\\n',\n",
       " 'governments; and\\n',\n",
       " '(c) the Judiciary and independent tribunals.\\n',\n",
       " '(4)\\xa0\\xa0The sovereign power of the people is exercised at—\\n',\n",
       " '(a) the national level; and\\n',\n",
       " '(b) the county level.\\n',\n",
       " 'Supremacy of this Constitution.\\n',\n",
       " '2. \\xa0 (1)\\xa0\\xa0This Constitution is the supreme law of the Republic and binds all persons\\n',\n",
       " 'and all State organs at both levels of government.\\n',\n",
       " '(2)\\xa0\\xa0No person may claim or exercise State authority except as authorised under\\n',\n",
       " 'this Constitution.\\n',\n",
       " '(3)\\xa0\\xa0The validity or legality of this Constitution is not subject to challenge by or\\n',\n",
       " 'before any court or other State organ.\\n',\n",
       " '(4)\\xa0\\xa0Any law, including customary law, that is inconsistent with this Constitution\\n',\n",
       " 'is void to the extent of the inconsistency, and any act or omission in contravention\\n',\n",
       " 'of this Constitution is invalid.\\n',\n",
       " '(5)\\xa0\\xa0The general rules of international law shall form part of the law of Kenya.\\n',\n",
       " '(6)\\xa0\\xa0Any treaty or convention ratiﬁed by Kenya shall form part of the law of Kenya\\n',\n",
       " 'under this Constitution.\\n',\n",
       " 'Defence of this Constitution.\\n']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relevant_lines[:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e6f5134-662d-4455-9e81-cfa4cf2d5552",
   "metadata": {},
   "source": [
    "Looking at the first few lines, we notice the non-breaking space character `\\xa0` in some lines. These can be normalized to regular whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "64c26fbe-aee7-4bdc-a0e3-390b31da3f03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('1.   (1)  All sovereign power belongs to the people of Kenya and shall be exercised\\n',\n",
       " '1. \\xa0 (1)\\xa0\\xa0All sovereign power belongs to the people of Kenya and shall be exercised\\n')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import unicodedata\n",
    "\n",
    "l = relevant_lines[3]\n",
    "unicodedata.normalize('NFKC', l), l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e3462c-577f-4ed9-a446-66e62ae2c1b9",
   "metadata": {},
   "source": [
    "We do the same for all the lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a8cefaa8-9f23-444c-9bac-2f27395bbb48",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['CHAPTER ONE\\n',\n",
       " 'SOVEREIGNTY OF THE PEOPLE AND SUPREMACY OF THIS CONSTITUTION\\n',\n",
       " 'Sovereignty of the people.\\n',\n",
       " '1.   (1)  All sovereign power belongs to the people of Kenya and shall be exercised\\n',\n",
       " 'only in accordance with this Constitution.\\n',\n",
       " '(2)  The people may exercise their sovereign power either directly or through their\\n',\n",
       " 'democratically elected representatives.\\n',\n",
       " '(3)  Sovereign power under this Constitution is delegated to the following State\\n',\n",
       " 'organs, which shall perform their functions in accordance with this Constitution—\\n',\n",
       " '(a) Parliament and the legislative assemblies in the county governments;\\n',\n",
       " '(b) the national executive and the executive structures in the county\\n',\n",
       " 'governments; and\\n',\n",
       " '(c) the Judiciary and independent tribunals.\\n',\n",
       " '(4)  The sovereign power of the people is exercised at—\\n',\n",
       " '(a) the national level; and\\n',\n",
       " '(b) the county level.\\n',\n",
       " 'Supremacy of this Constitution.\\n',\n",
       " '2.   (1)  This Constitution is the supreme law of the Republic and binds all persons\\n',\n",
       " 'and all State organs at both levels of government.\\n',\n",
       " '(2)  No person may claim or exercise State authority except as authorised under\\n',\n",
       " 'this Constitution.\\n',\n",
       " '(3)  The validity or legality of this Constitution is not subject to challenge by or\\n',\n",
       " 'before any court or other State organ.\\n',\n",
       " '(4)  Any law, including customary law, that is inconsistent with this Constitution\\n',\n",
       " 'is void to the extent of the inconsistency, and any act or omission in contravention\\n',\n",
       " 'of this Constitution is invalid.\\n',\n",
       " '(5)  The general rules of international law shall form part of the law of Kenya.\\n',\n",
       " '(6)  Any treaty or convention ratified by Kenya shall form part of the law of Kenya\\n',\n",
       " 'under this Constitution.\\n',\n",
       " 'Defence of this Constitution.\\n']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relevant_lines = [unicodedata.normalize('NFKC', l) for l in relevant_lines]\n",
    "relevant_lines[:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "747f6773-fddb-4a34-8312-8de192aa9342",
   "metadata": {},
   "source": [
    "At this point we can put all the ingredients together. We'll extract each of the 264 articles as a list of all the relevant lines. For each article, we include the chapter number and title it belongs to. For chapters that are divided into parts, we include the part number and title the article belongs to; otherwise this is set to null."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "43936319-d9e9-4b28-906b-dd50635824d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "number_words = [\n",
    "    \"one\", \"two\", \"three\", \"four\", \"five\", \"six\", \"seven\", \"eight\", \"nine\", \"ten\",\n",
    "    \"eleven\", \"twelve\", \"thirteen\", \"fourteen\", \"fifteen\", \"sixteen\", \"seventeen\", \"eighteen\"\n",
    "]\n",
    "\n",
    "current_chapter, current_part, current_article = None, None, None\n",
    "article_list = []\n",
    "index = 0\n",
    "\n",
    "def to_int(word):\n",
    "    \"\"\"Convert string word to integer number\"\"\"\n",
    "    word = word.lower()\n",
    "    return number_words.index(word) + 1\n",
    "\n",
    "def is_article_title(index):\n",
    "    next_index = index + 1\n",
    "    if next_index < len(relevant_lines):\n",
    "        next_line = relevant_lines[next_index].strip()\n",
    "        return article_num_pattern.match(next_line) is not None\n",
    "    return True  \n",
    "\n",
    "\n",
    "while index < len(relevant_lines):\n",
    "    line = relevant_lines[index].strip()\n",
    "    if match := chapter_pattern.match(line):\n",
    "        # reset part, article\n",
    "        current_part = None\n",
    "\n",
    "        if current_article:\n",
    "            # append to list before reseting\n",
    "            article_list.append(current_article)\n",
    "        current_article = None\n",
    "\n",
    "        # Current line is the chapter num, next line is the title\n",
    "        num_word = match.group(1)\n",
    "        current_chapter = (to_int(num_word), relevant_lines[index+1].strip())\n",
    "        index += 1 # swallow the title line\n",
    "    elif match := part_pattern.match(line):\n",
    "        part_number, part_title = int(match.group(1)), match.group(2)\n",
    "        current_part = (part_number, part_title)\n",
    "    elif match := article_num_pattern.match(line):\n",
    "        \n",
    "        if current_article:\n",
    "            # append previous article before starting new one\n",
    "            article_list.append(current_article)\n",
    "\n",
    "        current_article = dict(\n",
    "            number=int(match.group(1)),\n",
    "            title=relevant_lines[index-1].strip(),\n",
    "            lines=[re.sub(r'\\d+\\.\\s+', '', relevant_lines[index], count=1)],  # strip out article number\n",
    "            part=current_part,\n",
    "            chapter=current_chapter,\n",
    "        )\n",
    "    else:\n",
    "        # not the beginning of any section\n",
    "        \n",
    "        # if we have a current_article, then this\n",
    "        # is likely an additional line of the article.\n",
    "         \n",
    "        # However, since a new article's title precedes\n",
    "        # the article number, check that possibility first\n",
    "        if current_article and not is_article_title(index):\n",
    "            current_article['lines'].append(relevant_lines[index])\n",
    "    index += 1\n",
    "\n",
    "# append last article\n",
    "article_list.append(current_article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "fa5b7bdd-ce28-4892-996d-1fa7e6d14f88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "264"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(article_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e803071f-0280-4cd2-8446-40ad658e5764",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'chapter': (1, 'SOVEREIGNTY OF THE PEOPLE AND SUPREMACY OF THIS CONSTITUTION'),\n",
      " 'lines': ['(1)  All sovereign power belongs to the people of Kenya and shall '\n",
      "           'be exercised\\n',\n",
      "           'only in accordance with this Constitution.\\n',\n",
      "           '(2)  The people may exercise their sovereign power either directly '\n",
      "           'or through their\\n',\n",
      "           'democratically elected representatives.\\n',\n",
      "           '(3)  Sovereign power under this Constitution is delegated to the '\n",
      "           'following State\\n',\n",
      "           'organs, which shall perform their functions in accordance with '\n",
      "           'this Constitution—\\n',\n",
      "           '(a) Parliament and the legislative assemblies in the county '\n",
      "           'governments;\\n',\n",
      "           '(b) the national executive and the executive structures in the '\n",
      "           'county\\n',\n",
      "           'governments; and\\n',\n",
      "           '(c) the Judiciary and independent tribunals.\\n',\n",
      "           '(4)  The sovereign power of the people is exercised at—\\n',\n",
      "           '(a) the national level; and\\n',\n",
      "           '(b) the county level.\\n'],\n",
      " 'number': 1,\n",
      " 'part': None,\n",
      " 'title': 'Sovereignty of the people.'}\n",
      "***\n",
      "{'chapter': (1, 'SOVEREIGNTY OF THE PEOPLE AND SUPREMACY OF THIS CONSTITUTION'),\n",
      " 'lines': ['(1)  This Constitution is the supreme law of the Republic and '\n",
      "           'binds all persons\\n',\n",
      "           'and all State organs at both levels of government.\\n',\n",
      "           '(2)  No person may claim or exercise State authority except as '\n",
      "           'authorised under\\n',\n",
      "           'this Constitution.\\n',\n",
      "           '(3)  The validity or legality of this Constitution is not subject '\n",
      "           'to challenge by or\\n',\n",
      "           'before any court or other State organ.\\n',\n",
      "           '(4)  Any law, including customary law, that is inconsistent with '\n",
      "           'this Constitution\\n',\n",
      "           'is void to the extent of the inconsistency, and any act or '\n",
      "           'omission in contravention\\n',\n",
      "           'of this Constitution is invalid.\\n',\n",
      "           '(5)  The general rules of international law shall form part of the '\n",
      "           'law of Kenya.\\n',\n",
      "           '(6)  Any treaty or convention ratified by Kenya shall form part of '\n",
      "           'the law of Kenya\\n',\n",
      "           'under this Constitution.\\n'],\n",
      " 'number': 2,\n",
      " 'part': None,\n",
      " 'title': 'Supremacy of this Constitution.'}\n",
      "***\n",
      "{'chapter': (1, 'SOVEREIGNTY OF THE PEOPLE AND SUPREMACY OF THIS CONSTITUTION'),\n",
      " 'lines': ['(1)  Every person has an obligation to respect, uphold and defend '\n",
      "           'this\\n',\n",
      "           'Constitution.\\n',\n",
      "           '(2)  Any attempt to establish a government otherwise than in '\n",
      "           'compliance with\\n',\n",
      "           'this Constitution is unlawful.\\n'],\n",
      " 'number': 3,\n",
      " 'part': None,\n",
      " 'title': 'Defence of this Constitution.'}\n",
      "***\n",
      "{'chapter': (2, 'THE REPUBLIC'),\n",
      " 'lines': ['(1)  Kenya is a sovereign Republic.\\n',\n",
      "           '(2)  The Republic of Kenya shall be a multi-party democratic State '\n",
      "           'founded on\\n',\n",
      "           'the national values and principles of governance referred to in '\n",
      "           'Article 10.\\n'],\n",
      " 'number': 4,\n",
      " 'part': None,\n",
      " 'title': 'Declaration of the Republic.'}\n",
      "***\n",
      "{'chapter': (2, 'THE REPUBLIC'),\n",
      " 'lines': ['Kenya consists of the territory and territorial waters comprising '\n",
      "           'Kenya on the\\n',\n",
      "           'effective date, and any additional territory and territorial '\n",
      "           'waters as defined by an\\n',\n",
      "           'Act of Parliament.\\n'],\n",
      " 'number': 5,\n",
      " 'part': None,\n",
      " 'title': 'Territory of Kenya.'}\n",
      "***\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "for article in article_list[:5]:\n",
    "    pprint(article)\n",
    "    print('***')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "495ee489-5f2f-4a54-81c6-05853cf84b07",
   "metadata": {},
   "source": [
    "We can check that all articles were extracted in order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "38881263-f3f7-4644-aea6-fb9c20dcfc8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(article_list)):\n",
    "    article_num = article_list[i]['number']\n",
    "    assert article_num == i+1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e96bb8c4-4770-49bd-b324-7e7035487b45",
   "metadata": {},
   "source": [
    "Now we have all articles of the constitution as a list of dictionaries. Each dictionary has the following attributes:\n",
    "- `number`: the article number\n",
    "- `title`: the article's title\n",
    "- `lines`: the lines that consist the article, including all clauses and sub-clauses.\n",
    "- `chapter`: a tuple of chapter number and title the article belongs to\n",
    "- `part`: a tuple of the part number and title the article belongs to if available, otherwise this will be `None`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3d8aae-28af-4d5d-8043-665c2de1fa40",
   "metadata": {},
   "source": [
    "We can now save this in JSON:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "db1179a1-ea3a-4f7a-a02f-68e4d13fdd2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p json\n",
    "\n",
    "import json\n",
    "\n",
    "with open('json/ConstitutionKenya2010.json', 'wt') as f:\n",
    "    json.dump(article_list, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3304f8e-dce6-4ba3-81b4-1da53d2ec747",
   "metadata": {},
   "source": [
    "We now have a JSON version of the 2010 constitution's articles that can be easily be ingested by programs without having to do complicated parsing.\n",
    "\n",
    "Kudos to the folks at Govt Printer for producing a PDF version that had consistent formatting and was easy to parse without ugly hacks. At least in this case, Kenyans can be happy their tax money was used properly :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}