Skip to content
Alex Tan Hong Pin edited this page Apr 1, 2018 · 2 revisions

Patterns that can be achieved with go's channel:

  • take - take a subset of items from the channel
  • generator - generate a pseudo-random or constant value and return the channel
  • shard/split - split the result from the in channel to multiple out channels, with each channel receiving the same item
  • worker - spawn multiple workers that streams data from the in channel that process the items concurrently before returning to a single out channel
  • backpressure - when the queue is filled, the channel will be blocked
  • rate-limit - throttle the delivery of items to channel by limiting the items per duration, e.g. 100 req/s
  • throttle/debounce - there are a few patterns to this, such as limiting the maximum number of in-flight messages (through buffered channels), or by filling a buffered channels and gradually release them at a fixed interval
  • to-type - perform type conversion from interface to a determined typed
  • repeat - takes an interface and repeatedly send the same item to the channel
  • error - pattern for handling error in channel
  • or-done - slightly advance pattern of handling items in a channel, rather than the conventional for i := range ch
  • future - asynchronous processing of item and pending for the item to be returned
  • fan-in/multiplex - similar to worker, split the processing across different channels/streams
  • fan-out/demultiplex - merge multiple channels into a single channel
  • cancellation - pattern to stop all goroutines when the context is cancelled
  • deadline - pattern to stop all goroutines when a deadline is achieved
  • timeout - pattern to stop all goroutines when a timeout is achieved
  • pause - pattern to stop all goroutine, can be achieved by sync.Cond
  • self-healing - pattern to stop unhealthy goroutines and restarting them
  • ring buffer - dropping the messages when the message is full
  • retry - retries an operation for a max number of tries, either through constant, linear (the time doubles on every retry), or exponential backoff
  • pool - pattern to reuse a set of available objects rather than recreating them

Patterns of Event-Driven Processing

Based on the book, Designing Distributed System.

  • Copier takes a single stream of work items and duplicate it out into two or more identical streams.
  • Filter reduces the stream of work items to smaller stream of work items by filtering out work items that does not meet specific criteria.
  • Splitter evaluates some criteria - just like filter - but instead of eliminating input, the splitter sends different inputs to different queues based on that criteria.
  • Sharder divides up a single queue into an evenly divided collection of work items based upon some sharding function.
  • Merger is the opposite of copier - it takes two different work queues and turn them into a single work queue.

Things to note

  • the fastest you can process items in an event-driven processing is determined by your slowest channel
  • handle errors by retrying them for a certain threshold - if fail, drop them

TODO

Look at elixir's OTP implementation on supervisor, and see how to implement them into the go's concurrency pattern, especially the stop-all, stop one etc.

Clone this wiki locally