updating preprocesor-shards.py to support additional input features

[mdasadul]

To update preprocessor-shards.py along with preprocessor.py

[guillaumekln]

Thank you for updating this script as well.

@yoonkim Do you plan to factorize the preprocessing code?

[yoonkim]

Hmm yeah preprocess-shards should really be part of preprocess with a --shardsize option... I'll see if I have a chance to factorize it