more docs and samples

LICENSE.md

@@ -2,7 +2,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
 
-    http://www.apache.org/licenses/LICENSE-2.0
+http://www.apache.org/licenses/LICENSE-2.0
 
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,

README.md

@@ -1,4 +1,4 @@
-FsShelter [![Windows Build](https://ci.appveyor.com/api/projects/status/c0oom3oyr8qnrsc8?svg=true)](https://ci.appveyor.com/project/et1975/fsshelter) [![Mono/OSX build](https://travis-ci.org/Prolucid/FsShelter.svg?branch=master)](https://travis-ci.org/Prolucid/FsShelter)
+FsShelter [![Windows Build](https://ci.appveyor.com/api/projects/status/c0oom3oyr8qnrsc8?svg=true)](https://ci.appveyor.com/project/et1975/fsshelter) [![Mono/OSX build](https://travis-ci.org/Prolucid/FsShelter.svg?branch=master)](https://travis-ci.org/Prolucid/FsShelter) [![NuGet version](https://badge.fury.io/nu/fsshelter.svg)](https://badge.fury.io/nu/fsshelter)
 =======
 
 A library for defining and running Apache Storm topologies in F# using statically typed streams.
@@ -10,7 +10,7 @@ See [docs][docs] for for an intro and an overview.
 
 Join the conversation: [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](TBD)
 
-# Limitations
+## Limitations
 * At the moment FsShelter doesn't support direct emits.
 * [STORM-1644](https://issues.apache.org/jira/browse/STORM-1644): Currently, when running on Windows, the process will run under cmd.exe incurring slight overhead.
 
@@ -24,12 +24,12 @@ or on Linux/OSX:
 ./build.sh
 ```
 
-# Running the tests
+## Running the tests
 Building from command line runs the unit tests.
 
 IDE: Install NUnit plugin for VS or MonoDevelop to see the unit-tests in Test Explorer and step through the code under debugger.
 
-# Submitting the topology
+## Submitting the topology
 Have a local [Storm](https://storm.apache.org/downloads.html) installed and running.
 ```
 samples\WordCount\bin\Release\WordCount submit-local
@@ -39,13 +39,13 @@ or, if running on Mono:
 mono samples/WordCount/bin/Release/WordCount.exe submit-local
 ```
 
-# Seeing the topology in action
+## Seeing the topology in action
 Open [Storm UI](http://localhost:8080/) and see the Storm worker logs for runtime details.
 
-# License
+## License
 FsShelter is Apache 2.0 licensed and free to use and modify.
 
-# Commercial support
+## Commercial support
 Contact [Prolucid](http://prolucid.ca) for commercial support.
 
 [docs]:https://prolucid.github.io/FsShelter/

docs/content/LICENSE.md

@@ -2,7 +2,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
 
-    http://www.apache.org/licenses/LICENSE-2.0
+http://www.apache.org/licenses/LICENSE-2.0
 
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,

docs/content/index.fsx

@@ -5,22 +5,34 @@
 #r "FsShelter.dll"
 
 open System
+open FsShelter
 (**
-FsShelter
-======================
-
 Overview
 -------
-FsShelter is a library for implementation of [Apache Storm](https://storm.apache.org/) components and definition of topologies in F# DSL.
-The Management module also provides wrappers for Nimbus Thrift API, allowing to bundle and submit a topology for execution.
-FsShelter is based on and a major rewrite of [FsStorm](https://github.com/FsStorm). It departs from FsStrom in significant ways and therefore has been split off into itsown project.
+FsShelter is a library for implementation of [Apache Storm](https://storm.apache.org/) components and topologies in F#.
+FsShelter is based on and a major rewrite of [FsStorm](https://github.com/FsStorm). It departs from FsStrom in significant ways and therefore has been split into itsown project.
+
+Overall, the librabry provides "batteries included" experience with wrappers for Nimbus API as well as support for packaging and exporting:
+
+- bundle and submit a topology for execution w/o needing JDK or Storm CLI
+- include Storm-side serializer along
+- kill a running topology
+- generate a topology graph as part of your build
+
 The topology and the components could be implemented in a single EXE project and are executed by Storm via its [multilang](https://storm.apache.org/documentation/Multilang-protocol.html) protocol as separate processes - one for each task/instance.
-Corresponding [ProtoShell](https://github.com/prolucid/protoshell) and [ThriftShell](https://github.com/prolucid/thriftshell) libraries facilitate Protobuf and Thrift serialization, which improve throughput of FsShelter components as compared to standard JSON.
+Corresponding [ProtoShell](https://github.com/prolucid/protoshell) and [ThriftShell](https://github.com/prolucid/thriftshell) libraries facilitate Protobuf and Thrift serialization, which improve throughput of FsShelter topologies as compared to standard JSON.
+See samples to learn how to bundle the assemblies and a serializer for upload to Storm.
+
+Bring your own, if you need it:
+
+- command line parser
+- logging
+- custom serializer
 
 FsShelter topology schema
 -----------------------
-While Storm tuples are dynamically typed and to large extend the types are transparent to Storm itself, they are not types-less. 
-Mistakes and inconsistencies between declared outputs and tuple consumers could easily lead to errors detectable at run-time only and may be frustrating to test for, detect and fix.
+While Storm tuples are dynamically typed and to a large extend the types are transparent to Storm itself, they are not types-less. 
+Mistakes and inconsistencies between declared outputs and tuple consumers could easily lead to errors detectable at run-time only and may be frustrating to test, detect and fix.
 FsShelter introduces concept of topology schema, defined as F# discriminated union:
 *)
 
@@ -29,7 +41,8 @@ type BasicSchema =
     | Incremented of int
 
 (**
-where every DU case becomes a distinct stream in the topology. 
+where every DU case becomes a distinct stream in the topology. The fields of each DU case will become tuple fields in Storm.
+
 It is often handy to define a type that's shared across streams and FsShelter supports defining cases with records:
 *)
 
@@ -54,17 +67,17 @@ FsShelter "flattens" the first immediate "layer" of the DU case so that all the
 
 FsShelter components
 -----------------------
-
-FsShelter components are defined as simple functions:
+Some of the flexibility of Storm has been hidden to provide simple developer experience for authoring event-driven solutions.
+For exmple, FsShelter components are implemeted as simple functions:
 *)
 
 // numbers spout - produces messages
 let numbers source = async { return Some(Original(source())) }
 
 (**
-the async body is expected to return an option if there's a tuple to emit.
+The async body of a spout is expected to return an option if there's a tuple to emit or None if there's nothing to emit at this time.
 
-Bolts can get a touple on any number of streams, and so we pattern match:
+Bolts can get a tuple on any number of streams, and so we pattern match:
 *)
 
 // add 1 bolt - consumes and emits messages to Incremented stream
@@ -77,8 +90,8 @@ let addOne (input, emit) =
     }
 
 (**
-The bolt can also emit at any time, and can hold on to the passed emit function.
-The can be as many arguments for the component functions as needed:
+The bolt can also emit at any time, and we can hold on to the passed emit function (with caveates).
+Also, there can be as many arguments for the component functions as needed, the specifics will be determined when the components are put together in a topology.
 *)
 
 // terminating bolt - consumes messages
@@ -90,7 +103,11 @@ let logResult (info, input) =
     }
 
 (**
-the specifics will be determined when the components are put together in a topology:
+
+Using F# DSL to define the topology
+--------------------
+
+Storm topology is a graph of spouts and bolts connected via streams. FsShelter provides an embedded DSL for defining the topologies, which allows for mix and match of native Java, external shell and FsShell components:
 *)
 
 // define our source dependency
@@ -121,21 +138,20 @@ let sampleTopology =
     }
 
 (**
-Storm will start (a copy of) the same EXE for every component instance in the topology and will instruct each instance with the task it supposed to execute.
+Storm will start (a copy of) the same EXE for every component instance in the topology and will assign each instance a task it supposed to execute.
 
-The compiled topology can be submitted using embedded Thrift client, see the examples for details.
+The topology can be packaged with all its dependecies and submitted using embedded Nimbus client, see the examples for details.
 
 Exporting the topology graph in DOT format (GraphViz) using F# scripts
 -----------------------
+Once the number of components grows beyond handful it is often handy to be able to visualize them and FsStrom includes a simple way to export the topology into a graph:
 *)
 
-#r "../../build/WordCount.exe"
-
-open FsShelter
-
 sampleTopology |> DotGraph.writeToConsole
 
 (**
+See the samples included for further details.
+
 Samples & documentation
 -----------------------
 

docs/content/wordcount.fsx

@@ -3,6 +3,7 @@
 #r "FsShelter.dll"
 
 open System
+open FsShelter
 
 (**
 Defining the schema
@@ -112,7 +113,7 @@ let increment =
 Using F# DSL to define the topology
 --------------------
 
-Storm topology is a graph of spouts and bolts connected via streams. FsShelter provides an embedded DSL for defining the topologies, which allows mix and match of native java, external shell and FsShell components:
+Storm topology is a graph of spouts and bolts connected via streams that can be defined via `topology` computation expression:
 *)
 
 open FsShelter.DSL
@@ -155,7 +156,7 @@ The lambda arguments for the "run" methods privde the opportunity to carry out c
 * emit is the function to emit another tuple
 
 "log" and "cfg" are fixed once (curried) and as demonstrated in logBolt mkArgs lambda, one time-initialization can be carried out by inserting arbitrary code before "tuple" and "emit" arguments.
-This initialization will not be triggered unless the task execution is actually requsted by Storm for this specific instance of the process.
+This initialization will not be triggered unless the task execution is actually requested by Storm for this specific instance of the process.
 
 
 Exporting the topology graph 
@@ -166,13 +167,13 @@ FsShelter includes a completely customizable GraphViz (dot) export functionality
 ![SVG](svg/WordCount.svg "WordCount (SVG)")
 
 The dotted lines represent "unanchored" streams and the number inside the `[]` shows the parallelism hint.
-Which was achived by a simple export to console:
+This was achived by a simple export to console:
 *)
 
 sampleTopology |> DotGraph.writeToConsole
 
 (**
-Followed by a convertion into into SVG:
+Followed by further conversion into a desired format by piping the markup into GrapViz:
 
 ```bash
 mono samples/WordCount/bin/Release/WordCount graph | dot -Tsvg -o build/WordCount.svg

docs/tools/generate.fsx

@@ -9,16 +9,16 @@
 // for binaries output to root bin folder please add the filename only to the 
 // referenceBinaries list below in order to generate documentation for the binaries.
 // (This is the original behaviour of ProjectScaffold prior to multi project support)
-let referenceBinaries = ["FsShelter.dll"; "FsJson.dll"; "FsLogging.dll"]
+let referenceBinaries = ["FsShelter.dll"]
 // Web site location for the generated documentation
 let website = "/FsShelter"
 
-let githubLink = "http://github.com/FsShelter/FsShelter"
+let githubLink = "http://github.com/Prolucid/FsShelter"
 
 // Specify more information about your project
 let info =
   [ "project-name", "FsShelter"
-    "project-author", "Faisal Waris, Eugene Tolmachev"
+    "project-author", "Eugene Tolmachev"
     "project-summary", "F# DSL and runtime for Storm topologies"
     "project-github", githubLink
     "project-nuget", "http://nuget.org/packages/FsShelter" ]

src/FsShelter/Task.fs

@@ -5,8 +5,10 @@ open System.IO
 open FsShelter.Multilang
 open FsShelter.Topology
 open System
-
+
+/// Logger signature    
 type Log = (unit -> string) -> unit
+/// Task signature
 type Task<'t> = ComponentId -> Runnable<'t>
 
 // diagnostics pid shortcut

src/FsShelter/Topology.fs

@@ -4,38 +4,49 @@
 module Topology =
     open Multilang
 
+    /// Tuple id
     type TupleId = string
+    /// Stream id
     type StreamId = string
+    /// Component id
     type ComponentId = string
+    /// Signature for anchoring implementation
     type ToAnchors = TupleId->TupleId list
+    /// Signature for pluggable IO implementation
     type IO<'t> = (unit->Async<InCommand<'t>>)*(OutCommand<'t>->unit)
+    /// Signature for a final runnable component
     type Runnable<'t> = IO<'t>->Conf->Async<unit>
 
+    /// Storm Componend abstraction
     type Component<'t> = 
         | FuncRef of Runnable<'t>
         | Shell of command : string * args : string
         | Java of className : string * args : string list
 
+    /// Storm Spout abstraction
     type Spout<'t> = {
         MkComp:unit->Component<'t>
         Parallelism:uint32
         Conf:Conf option
     } with static member WithConf (s,conf) = {s with Conf = Some conf}
            static member WithParallelism (s,p) = {s with Parallelism = p}
 
+    /// Storm Bolt abstraction
     type Bolt<'t> = {
         MkComp:(StreamId->ToAnchors)->Component<'t>
         Parallelism:uint32
         Conf:Conf option
     } with static member WithConf (s,conf) = {s with Bolt.Conf = Some conf}
            static member WithParallelism (s,p) = {s with Bolt.Parallelism = p}
 
+    /// Storm stream grouping abstraction
     type Grouping<'t> = 
         | Shuffle
         | Fields of names:string list
         | All
         | Direct
 
+    /// Storm Stream abstraction
     type Stream<'t> = {
         Src:ComponentId
         Dst:ComponentId
@@ -44,6 +55,7 @@ module Topology =
         Schema:string list
     }
 
+    /// Storm Topology abstraction
     type Topology<'t> = { 
         Name:string 
         Spouts:Map<ComponentId,Spout<'t>>