Pytorch code for deformation-invariant line-level Handwritten Text Recognition, as proposed in paper (accepted to ICIP'21).
Motivation:
Image deformations under simple geometric restrictions are crucial for Handwriting Text Recognition (HTR), since different writing styles can be viewed as simple geometrical deformations of the same
textual elements.
Contibutions: 1) Exploration of different existing strategies for ensuring deformation invariance, including spatial transformers and deformable convolutions, under the
context of text recognition.
2) Introduction of a new deformation-based algorithm, inspired by adversarial learning, which aims to reduce character output uncertainty during evaluation time.
DNN Architecture: A Convolutional-only HTR system is presented (see paper),
where the output of a convolutional backbone, which transforms the images into a sequence of feature vectors, is fed into a cascade of 1-D convolutional layers.
Model architecture can be modified by changing the the cnn_cfg and cnn_top variables in config.py.
Specifically, CNN backbone is consisted of multiple stacks of ResBlocks and the default setting cnn_cfg = [(2, 32), 'M', (4, 64), 'M', (6, 128), 'M', (2, 256)] is interpeted as follows:
the first stack consists of 2 resblocks with output channels of 32 dimensions, the second of 4 resblocks with 64 output channels etc.
The head, consisted of three 1-D convolutional layers, can be modified through the cnn_top variable, which controls the number of output channels in these layers.
Selected Features:
- Dataset is saved in a '.pt' file after the initial preprocessing for faster loading operations
- All images are resized to 128x1024 (using padding if possible in order to retain aspect ratio).
- Transformations used during training: global affine, local deformations (elastic net), local morphological operations.
File valid_deforms.py contains:
- local_deform(img): creates a grid of smooth offsets in order to perform an elastic net distortion.
- morphological(img): creates a grid which control the morhpological operation to apply in each neighborhod.
- uncertainty_reduction(net, img): the proposed uncertainty reduction method to be used at evaluation. It discovers small local deformations (akin to local_deform method) which maximize the per character softmax output, i.e. minimize the uncertainty to select these characters.
Installation:
python3 -m venv venv_defhtr # Create a virtual environment.. (optional)
source venv_defhtr/bin/activate # ..and activate the virtual environment (optional)
pip3 install --upgrade pip # Upgrade pip
pip3 install -r requirements.txt # Install required libraries
cat utils/iam_config.py # Update the folder where IAM resides in your hard drive
python3 train_htr.py # Have fun!!
Note: Local paths of IAM dataset (https://fki.tic.heia-fr.ch/databases/iam-handwriting-database) are hardcoded in utils/iam_config.py