GA release

Highlights in this release

• The 2.0.3~rc3 version of SCPF has been stable in production for some weeks now and is hereby promoted to GA (v2.0.3).
• This version completes adding HTTP/2 and JWT authentication to APNS.
• The resource leak issues we were having with chatterbox have been ironed out. At some point we will see if upstream is interested in the chatterbox modifications we made in our fork of chatterbox.
• The JWT auth seems to be working flawlessly, finally relieving us of the burden of APNS push certificate renewals.

Summary of changes (to apns_erlv3, mostly)

• Use SilentCircle chatterbox ETS table mod that addresses performance and unbounded memory growth.
• Decrease default burst size to 50.
• Decrease default burst interval to 50 ms.
• Update documentation to reflect changes.
• Rewrite parts of apns_erlv3_session.erl to handle load and errors better.
• Modify CT tests to do a better job of flood testing (add sync_flood_and_disconnect test).
• Modify CT tests to work with JWT auths, together with apns_erl_sim JWT support and apns_tools JWT token generation.
• Fix dialyzer warnings.
• Various other small improvements.

Informal Roadmap

• Add metrics; we're planning on using exometer.
• Move most database operations from Mnesia to Postgres.
• Add "soft" documentation:
  • Describe the suite of repositories that comprise SCPF. There are some very useful goodies in there that are not well exposed because they only have API documentation.
  • Explain how to set up SCPF and how the bits and pieces play together.
  • Present the rationale behind the design of SCPF, as well as a bit of history to put some decisions into context.
  • Describe how to add new push mechanisms to SCPF.
• Fix up the extensibility. Although SCPF was designed from the start to be extensible, the purpose being to add more push mechanisms besides Google Cloud Messaging (gcm) and Apple Push (APNS), the vagaries of startup life intruded, so it's not as clean as it could be. We'll probably turn the "plugins" into OTP behaviors to make them tidier. There's also the perennial question of "support everything" vs "least common denominator" in terms of the various features each push service supports. The way it's done right now could be improved a lot.
• Refactor apns_ervl3 into a more formal producer-consumer architecture. Right now it's in an over-complicated gen_fsm with a kicker mechanism to send bursts of notifications every so many ms. It's untidy and performance-limiting, and needs to be split up into multiple processes with better abstractions.
• In the same vein, we think a pool of HTTP/2 clients might be better, as opposed to a single client connection buffered by queues. Although APNS seems to set MCS to at least 1000, so theoretically there's plenty of concurrency, it may help to have two or three connections so that if one goes down (because, say, that APNS node is having maintenance done), any buffered notifications could be handed off seamlessly to the other client(s) while the current one finds a new APNS server to use. The jury's out on that one.
• There hasn't been much performance tuning or benchmarking. It would be good to know the limits of (in particular) the Web API.
• We may consider replacing webmachine, which has done sterling work for the Web API, with cowboy, which is used in our APNS simulator.