Skip to content

Latest commit

 

History

History
140 lines (96 loc) · 6.05 KB

README.md

File metadata and controls

140 lines (96 loc) · 6.05 KB
Raw

Conversational Robot

Robotics Club Summer Project 2020

Mentors

Team Members

Aim

The aim of this project was to make a Talking bot, one which can pay attention to the user's voice and generate meaningful and contextual responses according to their intent, much like human conversations.

Ideation

This project was divided into overall three parts :

Overall Pipeline of the Project

overall pipeline

Speech Recognition

We used google-speech-to-text (gstt) API for the conversion of speech to text transcripts with a WER(Word Error Rate) of 4.7%.

Response Generation

We used a subset of the OpenSubtitles [4] dataset to train our response generation model, which was a combination of Context-based and Topic-based Attention Model.
This model has an encoder network which produces context vector for an input sentence followed by an attention mechanism which decides how much attention is to be paid to a particular word in a sentence and finally a decoder network which uses attention weights and context vectors to generate words of the output sentence i.e. response. We also added an AIML pipeline to our model for responding to some specific pattern of inputs which include greetings, emotions, jokes etc and also added Weather Forecasting and Googling capabilities.
Some of the output examples that we've produced with our model are:

Text to speech conversion

We used the google-text-to-speech (gtts) API for the conversion of text transcripts of responses back to speech.
The API uses playsound to play a temporary mp3 file created from the model's textual response.

Usage

Install the required dependencies :

$pip install -r requirements.txt
$sudo apt-get install gstreamer-1.0

Training checkpoints, LDA model weights and tokens can be found here

Required File Structure:

Response Generation
├── bin
│   ├── LDA
│   ├── Tokens.txt
│   ├── topic_dict.dict
│   ├── training_checkpoints
│   └── glove.42B.300d.txt
└── ...

Running the bot

usage: bot.py [-h] [-m {msg,trigger}]

The bot.

optional arguments:
  -h, --help            show this help message and exit
  -m {msg,trigger}, --mode {msg,trigger}
                        Mode of execution : Message box/ Trigger word
                        detection

Modes

  • Message Box - Provides a GUI for the user to start the conversation at the click of a button.
  • Trigger Word Detection - The program listens in the background and starts the conversation upon hearing the trigger word.
    • Commencement Trigger - Hello
    • Concluding Trigger - Bye

Functionality

  • Casual Conversations
  • Google search along with an explicit search feature for images
  • Weather Information

Demonstration

The video demonstration of this project can be found here.

References

  1. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

  2. Topic Aware Neural Response Generation

    • Link : [https://arxiv.org/abs/1606.08340]
    • Authors : Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, Wei-Ying Ma
    • Tags : Neural response generation; Sequence to sequence model; Topic aware conversation model; Joint attention; Biased response generation
    • Published : 21 Jun 2016 (v1), 19 Sep 2016 (v2)

  3. Topic Modelling and Event Identification from Twitter Textual Data

    • Link : [https://arxiv.org/abs/1608.02519]
    • Authors : Marina Sokolova, Kanyi Huang, Stan Matwin, Joshua Ramisch, Vera Sazonova, Renee Black, Chris Orwa, Sidney Ochieng, Nanjira Sambuli
    • Tags : Latent Dirichlet Allocation; Topic Models; Statistical machine translation
    • Published : 8 Aug 2016

  4. OpenSubtitles (Dataset)