mvlstm.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "mvlstm.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1kHP4Vmy7YMb"
      },
      "source": [
        "# mvlstm算法实战"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3stRbT0S8fk5"
      },
      "source": [
        "## 数据准备\r\n",
        "数据来源于https://github.com/terrifyzhao/text_matching， 里面是中文的匹配数据，有sentence1，sentence2和对应的label，0代表不匹配，1代表匹配。可以直接在GitHub中引用出来"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O1AsR2BFQ_9m",
        "outputId": "ad9c0867-06f1-43ee-9d6a-86123ef05c12"
      },
      "source": [
        "!git clone https://github.com/terrifyzhao/text_matching.git\r\n",
        "!cp -r /content/text_matching/input ./"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'text_matching'...\n",
            "remote: Enumerating objects: 334, done.\u001b[K\n",
            "remote: Total 334 (delta 0), reused 0 (delta 0), pack-reused 334\u001b[K\n",
            "Receiving objects: 100% (334/334), 24.77 MiB | 31.59 MiB/s, done.\n",
            "Resolving deltas: 100% (212/212), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a8YVW8lK9aIJ",
        "outputId": "a996ddee-2922-420d-ffae-cc39870a4c04"
      },
      "source": [
        "!ls ./input/"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dev.csv  test.csv  train.csv  vocab.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IYFHCK4r9cFn",
        "outputId": "9e658b07-9317-4f87-c650-c1ada0dbebad"
      },
      "source": [
        "!head -5 ./input/train.csv"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "﻿sentence1,sentence2,label\n",
            "用微信都6年，微信没有微粒贷功能,4。  号码来微粒贷,0\n",
            "微信消费算吗,还有多少钱没还,0\n",
            "交易密码忘记了找回密码绑定的手机卡也掉了,怎么最近安全老是要改密码呢好麻烦,0\n",
            "你好 我昨天晚上申请的没有打电话给我 今天之内一定会打吗？,什么时候可以到账,0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QGr62DLFIWka"
      },
      "source": [
        "# 首先设置一下本模型中要用到的具体的参数\r\n",
        "# 设置基本的参数\r\n",
        "base_params = {\r\n",
        "    'num_classes':2,               # 类别   \r\n",
        "    'max_features':1700,             # 嵌入层的最大词数\r\n",
        "    'embed_size':200,              # 嵌入维度\r\n",
        "    'filters':300,\r\n",
        "    'kernel_size':3,\r\n",
        "    'strides':1,\r\n",
        "    'padding':'same',\r\n",
        "    'conv_activation_func':'relu',\r\n",
        "    'embedding_matrix':[],\r\n",
        "    'w_initializer':'random_uniform',\r\n",
        "    'b_initializer':'zeros',\r\n",
        "    'dropout_rate':0.2,\r\n",
        "    'mlp_activation_func':'relu',\r\n",
        "    'mlp_num_layers':1,\r\n",
        "    'mlp_num_units':128,\r\n",
        "    'mlp_num_fan_out':128,\r\n",
        "    'input_shapes':[(64,),(64,)],         # 每句话填充后的最大长度\r\n",
        "    'task':'Classification',\r\n",
        "}\r\n",
        "\r\n",
        "mvlstm_params = base_params\r\n",
        "mvlstm_params['lstm_units'] = 64\r\n",
        "mvlstm_params['top_k'] = 50\r\n",
        "mvlstm_params['mlp_num_units'] = 128\r\n",
        "mvlstm_params['mlp_num_fan_out'] = 128\r\n",
        "mvlstm_params['dropout_rate'] = 0.3\r\n",
        "mvlstm_params['embed_size'] = 100\r\n",
        "base_params = mvlstm_params"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_2hCIGv7j1h"
      },
      "source": [
        "# 数据输入与转换\r\n",
        "由于mvlstm算法对于中文要求的输入是字向量，因为字相比于词，数量大大减少"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nFGtdcHeRdLt"
      },
      "source": [
        "#首先把文本中所有的字统计出来，制作出字表\r\n",
        "import pandas as pd\r\n",
        "\r\n",
        "df = pd.read_csv('input/train.csv')\r\n",
        "p = df['sentence1'].values\r\n",
        "h = df['sentence2'].values\r\n",
        "p_seg = list(map(lambda x: list(x.replace(\" \",\"\")), p))\r\n",
        "h_seg = list(map(lambda x: list(x.replace(\" \",\"\")), h))\r\n",
        "common_texts = []\r\n",
        "common_texts.extend(p_seg)\r\n",
        "common_texts.extend(h_seg)\r\n",
        "\r\n",
        "\r\n",
        "\r\n",
        "df = pd.read_csv('input/dev.csv')\r\n",
        "p = df['sentence1'].values\r\n",
        "h = df['sentence2'].values\r\n",
        "p_seg = list(map(lambda x: list(x.replace(\" \",\"\")), p))\r\n",
        "h_seg = list(map(lambda x: list(x.replace(\" \",\"\")), h))\r\n",
        "common_texts.extend(p_seg)\r\n",
        "common_texts.extend(h_seg)\r\n",
        "\r\n",
        "df = pd.read_csv('input/test.csv')\r\n",
        "p = df['sentence1'].values\r\n",
        "h = df['sentence2'].values\r\n",
        "p_seg = list(map(lambda x: list(x.replace(\" \",\"\")), p))\r\n",
        "h_seg = list(map(lambda x: list(x.replace(\" \",\"\")), h))\r\n",
        "common_texts.extend(p_seg)\r\n",
        "common_texts.extend(h_seg)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tsti40c-TCCs",
        "outputId": "91b65b39-a07d-40d0-dffe-b39b4ff686c6"
      },
      "source": [
        "common_texts[0]"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['用',\n",
              " '微',\n",
              " '信',\n",
              " '都',\n",
              " '6',\n",
              " '年',\n",
              " '，',\n",
              " '微',\n",
              " '信',\n",
              " '没',\n",
              " '有',\n",
              " '微',\n",
              " '粒',\n",
              " '贷',\n",
              " '功',\n",
              " '能']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3YhdaWqLTGDK"
      },
      "source": [
        "#使用set来统计不同的字\r\n",
        "char_set = set()\r\n",
        "for sample in common_texts:\r\n",
        "    for char in sample:\r\n",
        "        char_set.add(char)\r\n",
        "with open('input/char_vocab.txt','w',encoding='utf8') as f:\r\n",
        "    f.write(\"\\n\".join(sorted(list(char_set),reverse=True)))"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JR6tcyF-_WX3"
      },
      "source": [
        "#首先读取训练集的数据\r\n",
        "df = pd.read_csv('input/train.csv')\r\n",
        "p = df['sentence1'].values\r\n",
        "h = df['sentence2'].values\r\n",
        "label = df['label'].values"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-0ZqWsoi_8oW"
      },
      "source": [
        "#写一个shuffle函数，将里面的数据随机打乱\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def shuffle(*arrs):\r\n",
        "    \"\"\" shuffle\r\n",
        "\r\n",
        "    Shuffle 数据\r\n",
        "\r\n",
        "    Arguments:\r\n",
        "        *arrs: 数组数据\r\n",
        "\r\n",
        "    Returns:\r\n",
        "        shuffle后的数据\r\n",
        "\r\n",
        "    \"\"\"\r\n",
        "    arrs = list(arrs)\r\n",
        "    for i, arr in enumerate(arrs):\r\n",
        "        assert len(arrs[0]) == len(arrs[i])\r\n",
        "        arrs[i] = np.array(arr)\r\n",
        "    p = np.random.permutation(len(arrs[0]))\r\n",
        "    return tuple(arr[p] for arr in arrs)\r\n",
        "\r\n",
        "p, h, label = shuffle(p, h, label)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1rBZEU0wATTt"
      },
      "source": [
        "#接下来要将字转换成对应的数字，所以需要先对字表中的字编码\r\n",
        "# 加载字典\r\n",
        "def load_char_vocab():\r\n",
        "    vocab = [line.strip() for line in open('./input/char_vocab.txt', encoding='utf-8').readlines()]\r\n",
        "    word2idx = {word: index for index, word in enumerate(vocab,start=1)}\r\n",
        "    idx2word = {index: word for index, word in enumerate(vocab,start=1)}\r\n",
        "    return word2idx, idx2word\r\n",
        "\r\n",
        "word2idx, idx2word = load_char_vocab()"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RvPQOivdBOyd"
      },
      "source": [
        "#利用字典进行转换\r\n",
        "p_list, h_list = [], []\r\n",
        "for p_sentence, h_sentence in zip(p, h):\r\n",
        "    p = [word2idx[word.lower()] for word in p_sentence if len(word.strip()) > 0 and word.lower() in word2idx.keys()]\r\n",
        "    h = [word2idx[word.lower()] for word in h_sentence if len(word.strip()) > 0 and word.lower() in word2idx.keys()]\r\n",
        "\r\n",
        "    p_list.append(p)\r\n",
        "    h_list.append(h)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ETvIrVg5Bp3L",
        "outputId": "2a1e9e9c-92f8-472a-99f8-0704c9e601e2"
      },
      "source": [
        "#看一下编码后的句子\r\n",
        "p_list[0]"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1246, 1552, 918, 388, 1464, 578, 197, 1552, 1255]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1hhKZdCRByf2"
      },
      "source": [
        "#接下来需要写一个pad_sequences函数，将每句话都填充到相同长度，不足的补零\r\n",
        "def pad_sequences(sequences, maxlen=None, dtype='int32', padding='post',\r\n",
        "                  truncating='post', value=0.):\r\n",
        "    \"\"\" pad_sequences\r\n",
        "\r\n",
        "    把序列长度转变为一样长的，如果设置了maxlen则长度统一为maxlen，如果没有设置则默认取\r\n",
        "    最大的长度。填充和截取包括两种方法，post与pre，post指从尾部开始处理，pre指从头部\r\n",
        "    开始处理，默认都是从尾部开始。\r\n",
        "\r\n",
        "    Arguments:\r\n",
        "        sequences: 序列\r\n",
        "        maxlen: int 最大长度\r\n",
        "        dtype: 转变后的数据类型\r\n",
        "        padding: 填充方法'pre' or 'post'\r\n",
        "        truncating: 截取方法'pre' or 'post'\r\n",
        "        value: float 填充的值\r\n",
        "\r\n",
        "    Returns:\r\n",
        "        x: numpy array 填充后的序列维度为 (number_of_sequences, maxlen)\r\n",
        "\r\n",
        "    \"\"\"\r\n",
        "    lengths = [len(s) for s in sequences]\r\n",
        "\r\n",
        "    nb_samples = len(sequences)\r\n",
        "    if maxlen is None:\r\n",
        "        maxlen = np.max(lengths)\r\n",
        "\r\n",
        "    x = (np.ones((nb_samples, maxlen)) * value).astype(dtype)\r\n",
        "    for idx, s in enumerate(sequences):\r\n",
        "        if len(s) == 0:\r\n",
        "            continue  # empty list was found\r\n",
        "        if truncating == 'pre':\r\n",
        "            trunc = s[-maxlen:]\r\n",
        "        elif truncating == 'post':\r\n",
        "            trunc = s[:maxlen]\r\n",
        "        else:\r\n",
        "            raise ValueError(\"Truncating type '%s' not understood\" % padding)\r\n",
        "\r\n",
        "        if padding == 'post':\r\n",
        "            x[idx, :len(trunc)] = trunc\r\n",
        "        elif padding == 'pre':\r\n",
        "            x[idx, -len(trunc):] = trunc\r\n",
        "        else:\r\n",
        "            raise ValueError(\"Padding type '%s' not understood\" % padding)\r\n",
        "    return x\r\n",
        "\r\n",
        "p_list = pad_sequences(p_list, maxlen=base_params['input_shapes'][0][0])\r\n",
        "h_list = pad_sequences(h_list, maxlen=base_params['input_shapes'][0][0])"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Chfb7uH5CbGu",
        "outputId": "868bdab1-ae79-417d-ce73-544a2b63a434"
      },
      "source": [
        "#看一下编码后的句子\r\n",
        "p_list[0]"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1246, 1552,  918,  388, 1464,  578,  197, 1552, 1255,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
              "          0,    0,    0,    0,    0,    0,    0,    0,    0], dtype=int32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KE1ELQ2zCeHE"
      },
      "source": [
        "#将训练集数据制作成特征和标签\r\n",
        "import keras\r\n",
        "p = p_list\r\n",
        "h = h_list\r\n",
        "y = label\r\n",
        "x = [p,h]\r\n",
        "y = keras.utils.to_categorical(y,num_classes=2)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vj_UQWmcDows"
      },
      "source": [
        "#对验证集进行相同的操作\r\n",
        "df = pd.read_csv('input/dev.csv')\r\n",
        "p = df['sentence1'].values\r\n",
        "h = df['sentence2'].values\r\n",
        "label = df['label'].values\r\n",
        "\r\n",
        "p, h, label = shuffle(p, h, label)\r\n",
        "word2idx, idx2word = load_char_vocab()\r\n",
        "\r\n",
        "#利用字典进行转换\r\n",
        "p_list, h_list = [], []\r\n",
        "for p_sentence, h_sentence in zip(p, h):\r\n",
        "    p = [word2idx[word.lower()] for word in p_sentence if len(word.strip()) > 0 and word.lower() in word2idx.keys()]\r\n",
        "    h = [word2idx[word.lower()] for word in h_sentence if len(word.strip()) > 0 and word.lower() in word2idx.keys()]\r\n",
        "\r\n",
        "    p_list.append(p)\r\n",
        "    h_list.append(h)\r\n",
        "\r\n",
        "p_list = pad_sequences(p_list, maxlen=base_params['input_shapes'][0][0])\r\n",
        "h_list = pad_sequences(h_list, maxlen=base_params['input_shapes'][0][0])\r\n",
        "\r\n",
        "p_eval = p_list\r\n",
        "h_eval = h_list\r\n",
        "y_eval = label\r\n",
        "x_eval = [p_eval, h_eval]\r\n",
        "y_eval = keras.utils.to_categorical(y_eval,num_classes=2)"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FPI-TsrHEi3E"
      },
      "source": [
        "#对测试集进行相同的操作,注意这里的测试集是有标签的，便于比较算法效果，测试集并不参与训练过程\r\n",
        "df = pd.read_csv('input/test.csv')\r\n",
        "p = df['sentence1'].values\r\n",
        "h = df['sentence2'].values\r\n",
        "label = df['label'].values\r\n",
        "\r\n",
        "p, h, label = shuffle(p, h, label)\r\n",
        "word2idx, idx2word = load_char_vocab()\r\n",
        "\r\n",
        "#利用字典进行转换\r\n",
        "p_list, h_list = [], []\r\n",
        "for p_sentence, h_sentence in zip(p, h):\r\n",
        "    p = [word2idx[word.lower()] for word in p_sentence if len(word.strip()) > 0 and word.lower() in word2idx.keys()]\r\n",
        "    h = [word2idx[word.lower()] for word in h_sentence if len(word.strip()) > 0 and word.lower() in word2idx.keys()]\r\n",
        "\r\n",
        "    p_list.append(p)\r\n",
        "    h_list.append(h)\r\n",
        "\r\n",
        "p_list = pad_sequences(p_list, maxlen=base_params['input_shapes'][0][0])\r\n",
        "h_list = pad_sequences(h_list, maxlen=base_params['input_shapes'][0][0])\r\n",
        "\r\n",
        "p_test = p_list\r\n",
        "h_test = h_list\r\n",
        "y_test = label\r\n",
        "x_test = [p_test, h_test]\r\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes=2)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n54VFQ3RFU7K"
      },
      "source": [
        "# 建立模型"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_BRLWEo6FY7b"
      },
      "source": [
        "模型建立参考了：https://github.com/wangle1218/deep_text_matching  与match Zoo：https://github.com/NTMC-Community/MatchZoo  \r\n",
        "模型的结构在build函数里"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2hLHnybXCOph"
      },
      "source": [
        "#为什么要把这个作为一个类层定义呢，因为我之前用tf.nn.top_k函数，会报错，因为使用了自定义的层，必须转化为类才行，可以看：https://blog.csdn.net/qq_44930937/article/details/104509254\r\n",
        "from keras.engine import Layer, InputSpec\r\n",
        "class KMaxPooling(Layer):\r\n",
        "    \"\"\"\r\n",
        "    K-max pooling layer that extracts the k-highest activations from a sequence (2nd dimension).\r\n",
        "    TensorFlow backend.\r\n",
        "    \"\"\"\r\n",
        "    def __init__(self, k=1, **kwargs):\r\n",
        "        super().__init__(**kwargs)\r\n",
        "        self.input_spec = InputSpec(ndim=2)\r\n",
        "        self.k = k\r\n",
        "\r\n",
        "    def compute_output_shape(self, input_shape):\r\n",
        "        return (input_shape[0], self.k)\r\n",
        "\r\n",
        "    def call(self, inputs):\r\n",
        "        \r\n",
        "        # swap last two dimensions since top_k will be applied along the last dimension\r\n",
        "        # shifted_input = tf.transpose(inputs, [0, 2, 1])\r\n",
        "        \r\n",
        "        # extract top_k, returns two tensors [values, indices]\r\n",
        "        top_k = tf.nn.top_k(inputs, k=self.k, sorted=True, name=None)[0]\r\n",
        "        \r\n",
        "        # return flattened output\r\n",
        "        # return Flatten()(top_k)\r\n",
        "        return top_k\r\n",
        "\r\n",
        "    def get_config(self):\r\n",
        "        config = {'k':self.k}\r\n",
        "        base_config = super(KMaxPooling,self).get_config()\r\n",
        "        return dict(list(base_config.items()) + list(config.items()))\r\n"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O1C2C1R_Fz1o"
      },
      "source": [
        "from keras.models import Model\r\n",
        "import tensorflow as tf \r\n",
        "\r\n",
        "np.random.seed(1)\r\n",
        "tf.random.set_seed(1)\r\n",
        "\r\n",
        "class MVLSTM(object):\r\n",
        "\r\n",
        "    def __init__( self, params):\r\n",
        "        \"\"\"Init.\"\"\"\r\n",
        "        self._params = params\r\n",
        "\r\n",
        "    \r\n",
        "    def make_embedding_layer(self,name='embedding',embed_type='char',**kwargs):   #加载词向量的方法\r\n",
        "\r\n",
        "        def init_embedding(weights=None):\r\n",
        "            if embed_type == \"char\":\r\n",
        "                input_dim = self._params['max_features']\r\n",
        "                output_dim = self._params['embed_size']\r\n",
        "            else:\r\n",
        "                input_dim = self._params['word_max_features']\r\n",
        "                output_dim = self._params['word_embed_size']\r\n",
        "\r\n",
        "            return keras.layers.Embedding(\r\n",
        "                input_dim = input_dim,\r\n",
        "                output_dim = output_dim,\r\n",
        "                trainable = False,\r\n",
        "                name = name,\r\n",
        "                weights = weights,\r\n",
        "                **kwargs)\r\n",
        "\r\n",
        "        if embed_type == \"char\":\r\n",
        "            embed_weights = self._params['embedding_matrix']\r\n",
        "        else:\r\n",
        "            embed_weights = self._params['word_embedding_matrix']\r\n",
        "\r\n",
        "        if embed_weights == []:\r\n",
        "            embedding = init_embedding()\r\n",
        "        else:\r\n",
        "            embedding = init_embedding(weights = [embed_weights])\r\n",
        "\r\n",
        "        return embedding\r\n",
        "\r\n",
        "    def _make_multi_layer_perceptron_layer(self) -> keras.layers.Layer:   #多层感知机方法（全连接）\r\n",
        "        # TODO: do not create new layers for a second call\r\n",
        "        def _wrapper(x):\r\n",
        "            activation = self._params['mlp_activation_func']   #全连接激活函数\r\n",
        "            for _ in range(self._params['mlp_num_layers']):  #全连接层数\r\n",
        "                x = keras.layers.Dense(self._params['mlp_num_units'],   #全连接神经元数\r\n",
        "                                       activation=activation)(x)\r\n",
        "            return keras.layers.Dense(self._params['mlp_num_fan_out'],  #全连接最后一层神经元数\r\n",
        "                                      activation=activation)(x)\r\n",
        "\r\n",
        "        return _wrapper\r\n",
        "\r\n",
        "    def _make_inputs(self) -> list:        #定义输入层，这里只有两个输入，也就是要判断是否匹配的两段文本\r\n",
        "        input_left = keras.layers.Input(\r\n",
        "            name='text_left',\r\n",
        "            shape=self._params['input_shapes'][0]\r\n",
        "        )\r\n",
        "        input_right = keras.layers.Input(\r\n",
        "            name='text_right',\r\n",
        "            shape=self._params['input_shapes'][1]\r\n",
        "        )\r\n",
        "        return [input_left, input_right]\r\n",
        "\r\n",
        "    def _make_output_layer(self) -> keras.layers.Layer:   #定义输出层\r\n",
        "        \"\"\":return: a correctly shaped keras dense layer for model output.\"\"\"\r\n",
        "        task = self._params['task']\r\n",
        "        if task == \"Classification\":\r\n",
        "            return keras.layers.Dense(self._params['num_classes'], activation='softmax')  #分类使用softmax，两个文档输入就分类\r\n",
        "        elif task == \"Ranking\":\r\n",
        "            return keras.layers.Dense(1, activation='linear')          #排序使用linear，多个文档输入就排序\r\n",
        "        else:\r\n",
        "            raise ValueError(f\"{task} is not a valid task type.\"\r\n",
        "                             f\"Must be in `Ranking` and `Classification`.\")\r\n",
        "\r\n",
        "\r\n",
        "    def build(self):\r\n",
        "        \"\"\"Build model structure.\"\"\"\r\n",
        "        query, doc = self._make_inputs()\r\n",
        "\r\n",
        "        # Embedding layer\r\n",
        "        embedding = self.make_embedding_layer(mask_zero=True)\r\n",
        "        embed_query = embedding(query)\r\n",
        "        embed_doc = embedding(doc)\r\n",
        "\r\n",
        "        # Bi-directional LSTM layer\r\n",
        "        rep_query = keras.layers.Bidirectional(keras.layers.LSTM(\r\n",
        "            self._params['lstm_units'],\r\n",
        "            return_sequences=True,\r\n",
        "            dropout=self._params['dropout_rate']\r\n",
        "        ))(embed_query)\r\n",
        "        rep_doc = keras.layers.Bidirectional(keras.layers.LSTM(\r\n",
        "            self._params['lstm_units'],\r\n",
        "            return_sequences=True,\r\n",
        "            dropout=self._params['dropout_rate']\r\n",
        "        ))(embed_doc)\r\n",
        "\r\n",
        "        # Top-k matching layer\r\n",
        "        matching_matrix = keras.layers.Dot(\r\n",
        "            axes=[2, 2], normalize=False)([rep_query, rep_doc])\r\n",
        "        matching_signals = keras.layers.Reshape((-1,))(matching_matrix)\r\n",
        "        # matching_topk = keras.layers.Lambda(\r\n",
        "        #     lambda x: tf.nn.top_k(x, k=self._params['top_k'], sorted=True)[0]\r\n",
        "        # )(matching_signals)\r\n",
        "        matching_topk = KMaxPooling(k=self._params['top_k'])(matching_signals)\r\n",
        "\r\n",
        "        # Multilayer perceptron layer.\r\n",
        "        mlp = self._make_multi_layer_perceptron_layer()(matching_topk)\r\n",
        "        mlp = keras.layers.Dropout(\r\n",
        "            rate=self._params['dropout_rate'])(mlp)\r\n",
        "\r\n",
        "        x_out = self._make_output_layer()(mlp)\r\n",
        "        model = keras.Model(inputs=[query, doc], outputs=x_out)\r\n",
        "        return model"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D1e3S0tjNibw"
      },
      "source": [
        "# 加载模型训练"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S-YbfumJH8Uz"
      },
      "source": [
        "params = base_params\r\n",
        "backend = MVLSTM(params)"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "io6Qv_rvJ21Z"
      },
      "source": [
        "model = backend.build()"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q3iwKspxL4Cp"
      },
      "source": [
        "model.compile(\r\n",
        "        loss='categorical_crossentropy', \r\n",
        "        optimizer='adam', \r\n",
        "        metrics=['accuracy']\r\n",
        "        )"
      ],
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d6dVcvRdL9Wf",
        "outputId": "9c313d5b-00b7-4193-af79-cff77c209cb6"
      },
      "source": [
        "print(model.summary())"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "text_left (InputLayer)          [(None, 64)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "text_right (InputLayer)         [(None, 64)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding (Embedding)           (None, 64, 100)      170000      text_left[0][0]                  \n",
            "                                                                 text_right[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_6 (Bidirectional) (None, 64, 128)      84480       embedding[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_7 (Bidirectional) (None, 64, 128)      84480       embedding[1][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "dot_4 (Dot)                     (None, 64, 64)       0           bidirectional_6[0][0]            \n",
            "                                                                 bidirectional_7[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "reshape_4 (Reshape)             (None, 4096)         0           dot_4[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "k_max_pooling_1 (KMaxPooling)   (None, 50)           0           reshape_4[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "dense_4 (Dense)                 (None, 128)          6528        k_max_pooling_1[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "dense_5 (Dense)                 (None, 128)          16512       dense_4[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dropout_1 (Dropout)             (None, 128)          0           dense_5[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dense_6 (Dense)                 (None, 2)            258         dropout_1[0][0]                  \n",
            "==================================================================================================\n",
            "Total params: 362,258\n",
            "Trainable params: 192,258\n",
            "Non-trainable params: 170,000\n",
            "__________________________________________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XWz0ftrfMVMg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6b212955-7e5b-4889-b30c-bc0d7d7330fd"
      },
      "source": [
        "!mkdir output"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mkdir: cannot create directory ‘output’: File exists\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LC3Lp-9-MGtf",
        "outputId": "2e88ebb6-f4fe-4495-d41f-e60ad2aab1be"
      },
      "source": [
        "earlystop = keras.callbacks.EarlyStopping(\r\n",
        "        monitor='val_accuracy', \r\n",
        "        patience=4, \r\n",
        "        verbose=2, \r\n",
        "        mode='max'\r\n",
        "        )\r\n",
        "model_name = 'mvlstm'\r\n",
        "bast_model_filepath = './output/best_%s_model.h5' % model_name\r\n",
        "checkpoint = keras.callbacks.ModelCheckpoint(\r\n",
        "    bast_model_filepath, \r\n",
        "    monitor='val_accuracy', \r\n",
        "    verbose=1, \r\n",
        "    save_best_only=True,\r\n",
        "    mode='max'\r\n",
        "    )\r\n",
        "model.fit(\r\n",
        "    x=x, \r\n",
        "    y=y, \r\n",
        "    batch_size=64, \r\n",
        "    epochs=15, \r\n",
        "    validation_data=(x_eval, y_eval), \r\n",
        "    shuffle=True, \r\n",
        "    callbacks=[earlystop,checkpoint]\r\n",
        "    )"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/15\n",
            "1563/1563 [==============================] - 59s 30ms/step - loss: 0.6662 - accuracy: 0.5840 - val_loss: 0.5772 - val_accuracy: 0.7075\n",
            "\n",
            "Epoch 00001: val_accuracy improved from -inf to 0.70750, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 2/15\n",
            "1563/1563 [==============================] - 45s 29ms/step - loss: 0.5889 - accuracy: 0.6898 - val_loss: 0.5516 - val_accuracy: 0.7206\n",
            "\n",
            "Epoch 00002: val_accuracy improved from 0.70750 to 0.72060, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 3/15\n",
            "1563/1563 [==============================] - 45s 29ms/step - loss: 0.5581 - accuracy: 0.7178 - val_loss: 0.5222 - val_accuracy: 0.7431\n",
            "\n",
            "Epoch 00003: val_accuracy improved from 0.72060 to 0.74310, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 4/15\n",
            "1563/1563 [==============================] - 44s 28ms/step - loss: 0.5335 - accuracy: 0.7341 - val_loss: 0.5208 - val_accuracy: 0.7488\n",
            "\n",
            "Epoch 00004: val_accuracy improved from 0.74310 to 0.74880, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 5/15\n",
            "1563/1563 [==============================] - 44s 28ms/step - loss: 0.5142 - accuracy: 0.7462 - val_loss: 0.5056 - val_accuracy: 0.7538\n",
            "\n",
            "Epoch 00005: val_accuracy improved from 0.74880 to 0.75380, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 6/15\n",
            "1563/1563 [==============================] - 44s 28ms/step - loss: 0.5023 - accuracy: 0.7556 - val_loss: 0.5036 - val_accuracy: 0.7580\n",
            "\n",
            "Epoch 00006: val_accuracy improved from 0.75380 to 0.75800, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 7/15\n",
            "1563/1563 [==============================] - 43s 27ms/step - loss: 0.4859 - accuracy: 0.7666 - val_loss: 0.4981 - val_accuracy: 0.7562\n",
            "\n",
            "Epoch 00007: val_accuracy did not improve from 0.75800\n",
            "Epoch 8/15\n",
            "1563/1563 [==============================] - 42s 27ms/step - loss: 0.4728 - accuracy: 0.7739 - val_loss: 0.4805 - val_accuracy: 0.7699\n",
            "\n",
            "Epoch 00008: val_accuracy improved from 0.75800 to 0.76990, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 9/15\n",
            "1563/1563 [==============================] - 42s 27ms/step - loss: 0.4636 - accuracy: 0.7809 - val_loss: 0.4976 - val_accuracy: 0.7577\n",
            "\n",
            "Epoch 00009: val_accuracy did not improve from 0.76990\n",
            "Epoch 10/15\n",
            "1563/1563 [==============================] - 41s 26ms/step - loss: 0.4452 - accuracy: 0.7930 - val_loss: 0.4826 - val_accuracy: 0.7785\n",
            "\n",
            "Epoch 00010: val_accuracy improved from 0.76990 to 0.77850, saving model to ./output/best_mvlstm_model.h5\n",
            "Epoch 11/15\n",
            "1563/1563 [==============================] - 41s 26ms/step - loss: 0.4377 - accuracy: 0.7986 - val_loss: 0.4747 - val_accuracy: 0.7750\n",
            "\n",
            "Epoch 00011: val_accuracy did not improve from 0.77850\n",
            "Epoch 12/15\n",
            "1563/1563 [==============================] - 42s 27ms/step - loss: 0.4263 - accuracy: 0.8065 - val_loss: 0.4781 - val_accuracy: 0.7774\n",
            "\n",
            "Epoch 00012: val_accuracy did not improve from 0.77850\n",
            "Epoch 13/15\n",
            "1563/1563 [==============================] - 42s 27ms/step - loss: 0.4166 - accuracy: 0.8090 - val_loss: 0.4859 - val_accuracy: 0.7725\n",
            "\n",
            "Epoch 00013: val_accuracy did not improve from 0.77850\n",
            "Epoch 14/15\n",
            "1563/1563 [==============================] - 42s 27ms/step - loss: 0.4066 - accuracy: 0.8167 - val_loss: 0.4800 - val_accuracy: 0.7736\n",
            "\n",
            "Epoch 00014: val_accuracy did not improve from 0.77850\n",
            "Epoch 00014: early stopping\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f3924d300f0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RinZM_KhNoj_"
      },
      "source": [
        "# 保存模型\r\n",
        "model_frame_path = \"./output/%s_model.json\" % model_name\r\n",
        "model_json = model.to_json()\r\n",
        "with open(model_frame_path, \"w\") as json_file:\r\n",
        "    json_file.write(model_json)"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjL7-L8RN-8n"
      },
      "source": [
        "# 测试模型"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aS99ULO0N112",
        "outputId": "f2148377-9270-4507-c236-aa9a08c5fbfd"
      },
      "source": [
        "model.load_weights(bast_model_filepath)\r\n",
        "model.compile(\r\n",
        "    loss='categorical_crossentropy', \r\n",
        "    optimizer='adam', \r\n",
        "    metrics=['accuracy']\r\n",
        "    )\r\n",
        "\r\n",
        "loss, acc = model.evaluate(\r\n",
        "    x=x_test, \r\n",
        "    y=y_test, \r\n",
        "    batch_size=128, \r\n",
        "    verbose=1\r\n",
        "    )\r\n",
        "print(\"Test loss:\",loss, \"Test accuracy:\",acc)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "79/79 [==============================] - 8s 13ms/step - loss: 0.5056 - accuracy: 0.7599\n",
            "Test loss: 0.5093920230865479 Test accuracy: 0.7590000033378601\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}