IO Module

IO commands are found in behemoth-io.job.

usage: com.digitalpebble.behemoth.io.nutch.NutchSegmentConverter <segment> <output>
<segment>             The Nutch segment on HDFS.
<output>              The output path on HDFS.

Converts a Nutch segment into a Behemoth corpus.

usage: com.digitalpebble.behemoth.io.warc.WARCConverterJob <archive> <output>
<archive>             The WARC archive on HDFS.
<output>              The output path on HDFS.

Converts a WARC archive into a Behemoth corpus.

Behemoth Modules | Home