Skip to content

Developer Guide

Jump to bottom
peacock edited this page Jun 26, 2022 · 4 revisions

Introduction

This guide intended to provide insights into internal working of data store.

Concept

In JAVA there are many options to store the data via Properties file, Serialization API and Databases. But there are very little third party API to provide implementation for persistent queues.

Datastore is developed to provide FIFO and LIFO persistent queues with the following goals.

  • Provide simple API to read, write and delete the data
  • Data hash and data length is checked before providing the data to application
  • Make best efforts to retain the last written data even under abrupt power off conditions.
  • Limit the file space usage per queue
  • Light weight

Design

Interface DataStore provides API definition for persistent queue.

Class FileDataStoreQueue implements DataStore interface to provide implementation for FIFO persistent queue.

Class FileDataStoreStack implements DataStore interface to provide implementation for LIFO persistent queue.

FileDataStoreQueue

Class FileDataStoreQueue uses single RandomAccessFile to write, read and delete data. The file has two data blocks

  1. Meta block
  2. Data block

Meta block contains FIFO queue head and tail information, this block is located in the begining the file and it is updated on every call of sync method.

Data block contains header and queue data, this block is located after meta block and it is updated on every call of write method. The header part contains data size and data hash information and queue data is added next to header. The data block is written in circular manner as data is added to the queue.

The format of the file structure is shown below

Meta block
Data block 1
Data block 2
Data block 3
Data block .
Data block .
Data block n

write operation

  • Check for sufficient space to write header and data to the file
  • Calculate data hash and prepare header information with data length and data hash
  • Write header into file location starting at queue head position
  • Write data into the file location after header.
  • Increment head position and data count.

Two file IO is performed to store the data.

read operation

  • Check if data is available
  • Read header information starting at queue tail position
  • Validate length information
  • Read data after followed by header.
  • Verify data hash.

Two file IO is performed to read the data. Data is not removed from queue in read operation.

remove operation

  • Check if data is available
  • Read header information starting at tail position
  • Validate length information
  • Increment the tail position to point next data.

One file IO is performed to remove the data.

sync operation

Head and tail position is loaded in RAM. Application must call sync method to save head and tail position information from RAM to Disk.

  • Prepare meta information with head and tail information
  • Compute meta block hash
  • Write meta block and hash starting at zero offset.

This meta block is read during queue creation to initialize head and tail information.

FileDataStoreStack

Class FileDataStoreStack uses RandomAccessFile to write, read and delete data and a BackUp file to store last read data. The RandomAccessFile file has two data blocks

  1. Meta block
  2. Data block

Meta block contains LIFO queue head information, this block is located in the begining the file and it is updated on every call of sync method.

Data block contains header and queue data, this block is located after meta block and it is updated on every call of write method. The header part contains data size and data hash information and queue data is added next to header. The data block is written in stack manner as data is added to the queue.

The format of the file structure is shown below

Meta block
Data block 1
Data block 2
Data block 3
Data block .
Data block .
Data block n

The BackUp file is the simple to save last read data, since in LIFO after read the data must not be discarded and the same data must be given to application until it is explicitly removed.

write operation

  • Check for sufficient space to write header and data to the file
  • Calculate data hash and prepare header information with data length and data hash
  • Write header into file location starting at stack head position
  • Write data into the file location after header.
  • Increment head position and data count.

Two file IO is performed to store the data.

read operation

  • Check if data in BackUp file, if data is present return the data otherwise perform the below steps.
  • Read header information starting at queue tail position
  • Validate length information
  • Read data after followed by header.
  • Verify data hash and store it to the BackUp file.
  • Decrement the head position to next data

Three file IO is performed to read the data. Data is backed up and removed from queue in read operation.

remove operation

  • Check if data in BackUp file, if data is present remove the data otherwise perform the below steps.
  • Read header information starting at queue tail position
  • Validate length information
  • Decrement the head position to next data

One file IO is performed in remove the data.

sync operation

Head and count information is loaded in RAM. Application must call sync method to save head and count information from RAM to Disk.

  • Prepare meta information with head and count information
  • Compute meta block hash
  • Write meta block and hash starting at zero offset.

This meta block is read during stack creation to initialize head and count information.

© peacock05 2022

Clone this wiki locally