Semantic_Relation_Extraction/Relation_extraction_semeval2010.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Relation Extraction using DNN\n",
    "\n",
    "In this notebook, relations between the entities in a sequence is being predicted. The model is given the tokenized sequence which is padded or truncated to max_sent_len and the position of entities in the sequence. The entity positions are given by making a vector of zeros of size max_sent_len and masking (replace 0 with 1) the corresponding positions of entities in the sequence. \n",
    "\n",
    "The model uses CNN and GRU to get the word level information. The features from this layers are than passsed through Mask Max Pooling Layer, which pulls the information pertaining to the word features corresponding to entity words in the sentence. The word level information is also attended using a self attention layer. The globally max pooled word level features, mask max pooled entity level features and the attended features are all appended and passed to dense layers to be classified into one of the relations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('2.2.4',\n",
       " '1.11.0',\n",
       " '3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras, tensorflow, sys\n",
    "keras.__version__, tensorflow.__version__, sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the necessary packages\n",
    "\n",
    "from keras.models import Model, Input\n",
    "\n",
    "from keras.layers import Dense, LSTM, Dropout, Embedding,  concatenate, Flatten, Permute\n",
    "from keras.layers import GlobalMaxPooling1D, Convolution1D, CuDNNGRU, Activation, Lambda\n",
    "from keras.layers import GlobalAveragePooling1D, Concatenate, SpatialDropout1D, Bidirectional\n",
    "\n",
    "from keras.optimizers import Adam, RMSprop\n",
    "\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "from keras.engine.topology import Layer\n",
    "from keras import initializers as initializers, regularizers, optimizers\n",
    "\n",
    "from keras import backend as K\n",
    "\n",
    "from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report, confusion_matrix\n",
    "\n",
    "import regex as re\n",
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "from nltk.tokenize import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_sent_len = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get mask for entity words\n",
    "\n",
    "Get a mask vector for sentence by putting 1 for entity words or else 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_mask_entities(x, word_index):\n",
    "    ''' 1 for entity words, 0 otherwise '''\n",
    "    \n",
    "    ret = np.zeros_like(x)\n",
    "    for i in range(x.shape[0]): \n",
    "        e1 = [0, 0]\n",
    "        e2 = [0, 0]\n",
    "        for j in range(x.shape[1]):\n",
    "            if x[i][j] == word_index[\"e1_start\"]:\n",
    "                e1[0] = j\n",
    "            elif x[i][j] == word_index[\"e1_end\"]: \n",
    "                e1[1] = j\n",
    "            elif x[i][j] == word_index[\"e2_start\"]: \n",
    "                e2[0] = j\n",
    "            elif x[i][j] == word_index[\"e2_end\"]: \n",
    "                e2[1] = j\n",
    "                break\n",
    "        for j in range(e1[0]+1, e1[1]): \n",
    "            ret[i][j] = 1\n",
    "        for j in range(e2[0]+1, e2[1]): \n",
    "            ret[i][j] = 1\n",
    "    \n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Glove Embedding\n",
    "\n",
    "Load the embedding file 'glove.840B.300d.txt' and find the mean and standard deviation vectors of the word vectors. Than for all the words in the vocab initialize the corresponding word vector from the loaded embedded file. For the words for which wordvecs cannot be found in the embedding file, initialize them with a random normal distribution with the above found mean and standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_glove(word_index):\n",
    "    EMBEDDING_FILE = '../../embeddings/glove.840B.300d/glove.840B.300d.txt'\n",
    "    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n",
    "    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\"))\n",
    "\n",
    "    all_embs = np.stack(embeddings_index.values())\n",
    "    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n",
    "    embed_size = all_embs.shape[1]\n",
    "\n",
    "    num_words = len(word_index)\n",
    "    embedding_matrix = np.random.normal(emb_mean, emb_std, (num_words, embed_size))\n",
    "    for word, i in word_index.items():\n",
    "        if i >= num_words: continue\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n",
    "            \n",
    "    return embedding_matrix \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and Process data file.\n",
    "\n",
    "SemEval2010_task8 : In this task of SemEval 2010 challenge, the puspose was to find the one out of the 19 possible relations between the entities in a sentence. The entities in the sentences are marked by <ei>, </ei> and the corresponding relation of the entities are given in the same line as sentence, seperated by a tab.\n",
    "\n",
    "The data files is uploaded along with the code. The data is loaded and preprocesssing steps are performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT\") as f:\n",
    "    train_file = f.readlines()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the sentences and relations from the lines in the text file and replace the <ei>, </ei> tags with Ei_Start and Ei_End words, so that the tokenizer dosen't misunderstand '<' , '>' symbols as punctuations. Than word tokenize the sequences using NLTK word_tokenize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of setences and relations, i.e. no. of samples: 8000\n"
     ]
    }
   ],
   "source": [
    "       \n",
    "lines = [line.strip() for line in train_file]\n",
    "sentences, relations = [], []\n",
    "for idx in range(0, len(lines), 4):\n",
    "    sentence = lines[idx].split(\"\\t\")[1][1:-1]\n",
    "    label = lines[idx+1]\n",
    "\n",
    "    sentence = sentence.replace(\"<e1>\", \" E1_START \").replace(\"</e1>\", \" E1_END \")\n",
    "    sentence = sentence.replace(\"<e2>\", \" E2_START \").replace(\"</e2>\", \" E2_END \")\n",
    "\n",
    "    tokens = word_tokenize(sentence)        \n",
    "    \n",
    "    sentences.append(tokens)\n",
    "    relations.append(label)\n",
    "\n",
    "print(\"Number of setences and relations, i.e. no. of samples:\", len(sentences))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample text after pre - processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['The', 'system', 'as', 'described', 'above', 'has', 'its', 'greatest', 'application', 'in', 'an', 'arrayed', 'E1_START', 'configuration', 'E1_END', 'of', 'antenna', 'E2_START', 'elements', 'E2_END', '.']\n"
     ]
    }
   ],
   "source": [
    "# Sample text\n",
    "print(sentences[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize and pad the text sequences\n",
    "\n",
    "Tokenize -> change the word to there integer ids\n",
    "\n",
    "Pad -> Trim or pad with zeros to make all sentences of same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size: 19938\n"
     ]
    }
   ],
   "source": [
    "# Tokenize the sentence and than pad/ truncate the sentences  \n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(list(sentences))\n",
    "print(\"Vocab size:\", len(tokenizer.word_counts))\n",
    "\n",
    "sentences_idx = tokenizer.texts_to_sequences(sentences)\n",
    "sentences_idx = pad_sequences(sentences_idx, maxlen=max_sent_len, padding=\"post\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get dictionary of word indexes.\n",
    "word_index = tokenizer.word_index\n",
    "\n",
    "# Get masked sentence, i.e. only entity words are 1, rest all the words are 0.\n",
    "mask_entites = get_mask_entities(sentences_idx, word_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "# Sample mask\n",
    "print(mask_entites[0])\n",
    "\n",
    "# Notice the two ones here | &  here | , representing the postition of the entity words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedding matrix shape: (19938, 300)\n"
     ]
    }
   ],
   "source": [
    "# Load Embedding for all the words in the vocab.\n",
    "embedding = load_glove(word_index)\n",
    "print(\"Embedding matrix shape:\", embedding.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different Relations\n",
    "\n",
    "Label encode the relations, and show different relations possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Number of relations: 19\n",
      "Relations:\n",
      " ['Cause-Effect(e1,e2)' 'Cause-Effect(e2,e1)' 'Component-Whole(e1,e2)'\n",
      " 'Component-Whole(e2,e1)' 'Content-Container(e1,e2)'\n",
      " 'Content-Container(e2,e1)' 'Entity-Destination(e1,e2)'\n",
      " 'Entity-Destination(e2,e1)' 'Entity-Origin(e1,e2)' 'Entity-Origin(e2,e1)'\n",
      " 'Instrument-Agency(e1,e2)' 'Instrument-Agency(e2,e1)'\n",
      " 'Member-Collection(e1,e2)' 'Member-Collection(e2,e1)'\n",
      " 'Message-Topic(e1,e2)' 'Message-Topic(e2,e1)' 'Other'\n",
      " 'Product-Producer(e1,e2)' 'Product-Producer(e2,e1)']\n"
     ]
    }
   ],
   "source": [
    "# Encoding the relations to indexes.\n",
    "label_encoder = LabelEncoder()\n",
    "relations_idx = label_encoder.fit_transform(relations)\n",
    "print(\"Total Number of relations:\", len(label_encoder.classes_))\n",
    "print(\"Relations:\\n\", label_encoder.classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the model\n",
    "\n",
    "Class MaskMaxPoolingLayer is used to perform max pooling for the masked entity words from the feature vector of the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "class MaskMaxPoolingLayer(Layer):\n",
    "    \n",
    "    def __init__(self, **kwargs):\n",
    "        super(MaskMaxPoolingLayer, self).__init__(**kwargs)\n",
    "    \n",
    "    def build(self, input_shape):\n",
    "        super(MaskMaxPoolingLayer, self).build(input_shape)\n",
    "\n",
    "    def call(self, x):\n",
    "        x_1_float32 = K.cast(x[1], dtype='float32')\n",
    "        x_0 = K.permute_dimensions(x[0], pattern=[2, 0, 1])\n",
    "        x_0 = tf.multiply(x_0, x_1_float32)\n",
    "        x_0 = K.permute_dimensions(x_0, pattern=[1, 2, 0])\n",
    "        x_0 = K.max(x_0, axis=-2)\n",
    "        return x_0\n",
    "    \n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return (input_shape[0][0], input_shape[0][-1])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using preloaded glove vectors as embedding weights for the model.\n",
    "\n",
    "Embedded word vectors are first passed to 1D convolution and than passed to bidirectional GRU. GRU takes care of the sequential information, while CNN improves the embeddings by emphasizing on neighbor information. \n",
    "\n",
    "Global max pool layer pools 1 feature from each of the feature vector, unlike maxpool where we determine how many values is to be pooled.\n",
    "\n",
    "Mask maxpool layers pool the feature vectors corresponding to the entity words. \n",
    "\n",
    "Global max pooled, mask max pooled and Self-attended features of the RNN output are all concatenated and passed to the dense layers.\n",
    "\n",
    "Finally multiple fully-connected layers are used to classify the incoming query into one of the possible relations.\n",
    "\n",
    "Adam optimizer and sparse categorical crossentropy loss are used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Program_Files\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel_launcher.py:11: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"tanh\", padding=\"same\", strides=1, filters=256, kernel_size=3)`\n",
      "  # This is added back by InteractiveShellApp.init_path()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 100, 300)     5981400     input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 100, 300)     0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, 100, 256)     230656      dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100, 256)     0           conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional) (None, 100, 128)     123648      dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, 100, 128)     0           bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100, 1)       129         activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "permute_1 (Permute)             (None, 1, 100)       0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "attn_softmax (Activation)       (None, 1, 100)       0           permute_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lambda_1 (Lambda)               (None, 1, 128)       0           attn_softmax[0][0]               \n",
      "                                                                 activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 128)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "mask_max_pooling_layer_1 (MaskM (None, 128)          0           activation_1[0][0]               \n",
      "                                                                 input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 128)          0           lambda_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 384)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 mask_max_pooling_layer_1[0][0]   \n",
      "                                                                 flatten_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 384)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          115500      dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 19)           5719        dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, 19)           0           dense_3[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 6,457,052\n",
      "Trainable params: 6,457,052\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Model\n",
    "\n",
    "words_input = Input(shape=(max_sent_len,), dtype='int32')\n",
    "words_input_mask = Input(shape=(max_sent_len,), dtype='int32')\n",
    "\n",
    "words = Embedding(input_dim=embedding.shape[0], output_dim=embedding.shape[1], weights=[embedding], trainable=True,\n",
    "                  embeddings_regularizer=regularizers.l2(0.00001))(words_input)\n",
    "\n",
    "words = Dropout(rate=0.5)(words)\n",
    "\n",
    "output = Convolution1D(nb_filter=256, filter_length=3, activation=\"tanh\", padding='same', strides=1)(words)\n",
    "output = Dropout(rate=0.3)(output)\n",
    "\n",
    "output = Bidirectional(CuDNNGRU(units=64, return_sequences=True, recurrent_regularizer=regularizers.l2(0.00001)),\n",
    "                       merge_mode='concat') (output)\n",
    "\n",
    "output_h = Activation('tanh')(output)\n",
    "\n",
    "output1 = GlobalMaxPooling1D()(output_h) \n",
    "\n",
    "output2 = MaskMaxPoolingLayer()([output_h, words_input_mask]) \n",
    "\n",
    "# Applying attention to RNN output\n",
    "output = Dense(units=1, kernel_regularizer=regularizers.l2(0.00001))(output_h)\n",
    "output = Permute((2, 1))(output)\n",
    "output = Activation('softmax', name=\"attn_softmax\")(output)\n",
    "output = Lambda(lambda x: tf.matmul(x[0], x[1]))([output, output_h])\n",
    "output3 = Flatten()(output)\n",
    "\n",
    "# Concatenating maxpooled, mask maxpooled and self attended features.\n",
    "output = Concatenate()([output1, output2, output3])\n",
    "\n",
    "output = Dropout(rate=0.3)(output)\n",
    "\n",
    "output = Dense(units=300, kernel_regularizer=regularizers.l2(0.00001), activation='tanh')(output)\n",
    "\n",
    "output = Dense(units=len(label_encoder.classes_), kernel_regularizer=regularizers.l2(0.00001))(output)\n",
    "output = Activation('softmax')(output)\n",
    "\n",
    "model = Model(inputs=[words_input, words_input_mask], outputs=[output])\n",
    "\n",
    "model.compile(loss='sparse_categorical_crossentropy', optimizer= optimizers.Adadelta(lr=1.0, decay=0.0),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x179f02ae828>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([sentences_idx, mask_entites], relations_idx, shuffle=True, batch_size=128, epochs=100, verbose=0) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Test data and evaluate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of setences and relations, i.e. no. of samples: 2717\n"
     ]
    }
   ],
   "source": [
    "with open(\"SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT\") as f:\n",
    "    test_file = f.readlines()\n",
    "\n",
    "test = [line.strip() for line in test_file]\n",
    "test_x, test_y = [], []\n",
    "for idx in range(0, len(test), 4):\n",
    "    sentence = test[idx].split(\"\\t\")[1][1:-1]\n",
    "    label = test[idx+1]\n",
    "\n",
    "    sentence = sentence.replace(\"<e1>\", \" E1_START \").replace(\"</e1>\", \" E1_END \")\n",
    "    sentence = sentence.replace(\"<e2>\", \" E2_START \").replace(\"</e2>\", \" E2_END \")\n",
    "\n",
    "    tokens = word_tokenize(sentence)     \n",
    "\n",
    "    test_x.append(tokens)\n",
    "    test_y.append(label)\n",
    "    \n",
    "print(\"Number of setences and relations, i.e. no. of samples:\", len(test_x))\n",
    "\n",
    "test_x_idx = tokenizer.texts_to_sequences(test_x)\n",
    "test_x_idx = pad_sequences(test_x_idx, maxlen=max_sent_len)\n",
    "\n",
    "test_mask_entites = get_mask_entities(test_x_idx, word_index)\n",
    "\n",
    "test_y_idx = label_encoder.transform(test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict([test_x_idx, test_mask_entites])\n",
    "pred = np.argmax(pred, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1_score: 0.7349170060047774 0.7670224512329775\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Program_Files\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\sklearn\\metrics\\classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print(\"f1_score:\",f1_score(test_y_idx, pred, average=\"macro\"), accuracy_score(test_y_idx, pred))    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}