Home

Welcome to the storm-crawler wiki!

Getting Started

• Introduction
• Configuration: how to configure the storm-crawler
• Registering Metadata for Serialization: If your topology doesn't extend ConfigurableTopology, you will need to manually register storm-crawler's Metadata class for serialization in Storm.

Components

• FetcherBolt(s)
• Protocols: Network protocols that are usable in storm-crawler
• JSoupParserBolt: parse HTML documents
• SiteMapParserBolt: how to handle sitemaps
• URLFilters: how to filter or normalise outlinks
• ParseFilters: extract metadata from documents
• IndexingBolts

Resources

• Presentations
• Powered By
• External plugins