2014-04-leamington-black-pepper-open-dev-open-data.html

---
layout: reveal
title: Open^3
description: Open Development, Open Data, and the Open Data Institute
author: Stuart Harrison
twitter: pezholio
---

{% include odi_logo.html %}

{% include standard_title.html %}


<style type="text/css">
@import url(https://fonts.googleapis.com/css?family=Quicksand:400,700,400italic,700italic);
@import url(https://fonts.googleapis.com/css?family=Open+Sans:400italic,700italic,400,700);

reveal {
  font-family: "Open Sans", sans-serif;
  font-size: 36px;
  font-weight: 200;
  letter-spacing: 0.6em;
  color: #333333; }

.reveal h1,
.reveal h2,
.reveal h3,
.reveal h4,
.reveal h5,
.reveal h6 {
  margin: 0 0 20px 0;
  color: #333333;
  font-family: "Quicksand", sans-serif;
  line-height: 1.2em;
  letter-spacing: -0.08em;
  text-transform: uppercase;
  text-shadow: none;
 }

</style>


<section>
	<h2>WTF is the<br/>Open Data Institute?</h2>
	<ul class="centred-list">
		<li>non-profit, non-partisan</li>
		<li>founded 2012 by Tim Berners-Lee and Nigel Shadbolt</li>
		<li>&quot;helping others be successful with open data&quot;</li>
		<li>economic, social and environmental value</li>
	</ul>
  <aside class="notes"></aside>
</section>

<section>
	<h2>WTF is<br/>open data?</h2>
  <aside class="notes"></aside>
</section>

<section data-markdown>
  <aside class="notes">
    <p>The simplest way to describe it is...</p>
  </aside>
	<p>
		<blockquote>Open data is information that is available for anyone to use, for any purpose,<br/>at no cost.</blockquote>
		&mdash;&nbsp;<a href="http://opendefinition.org/">http://theodi.org/guide/what-open-data</a>
	</p>
</section>

<section>
  <aside class="notes">
    <p>Licensing is important too...</p>
    <ul>
      <li>Examples of licenses include Creative Commons Zero, Open Government Licence etc<li>
    </ul>
  </aside>
	<ul class="centred-list">
		<li><strong>open data</strong><br/>must have have a <em>licence</em> to say it is open</li>
		<li><strong>the license</strong><br/>may impose some constraints:<br/><em>attribution</em> and/or <em>share-alike</em></li>
	</ul>
	&mdash;&nbsp;<a href="http://opendefinition.org/licenses/">http://opendefinition.org/licenses/</a>
</section>

<section>
  <aside class="notes">
    <p>This is how the Open Knowledge foundation describe open data in the Open Definition</p>
  </aside>
	<p>
		<blockquote>A piece of data or content is open if anyone is free to use, reuse, and redistribute it &mdash; subject only, at most, to the requirement to attribute and/or share-alike.</blockquote>
		&mdash;&nbsp;<a href="http://opendefinition.org/">http://opendefinition.org/</a>
	</p>
</section>

<section>
	<h2>So What?</h2>
  <aside class="notes">
    <p>Why should you care?</p>
      <ul>
        <li>As a consumer: Because open data frees you up to build cool things without having to pay for the data or collect it yourself</li>
        <li>As a data owner: Because your clients/customers will probably start to care.</li>
      </ul>
  </aside>
</section>


<section>
  <p><img src='2013-09-bristol-brug-open-dev-open-data/OSMinecraft.png'></p>
  <small><a href='http://www.ordnancesurvey.co.uk/innovate/developers/minecraft-map-britain.html'>http://www.ordnancesurvey.co.uk/innovate/developers/minecraft-map-britain.html</a></small>
  <aside class="notes">
    <p>So thinking about some examples of interesting data sets and how these have been used...</p>
    <p>This is a Minecraft representation of the whole of Great Britain, built using Ordnance Survey open data by one of OS's summer interns</p>
  </aside>
</section>

<section>
  <aside class="notes">
    <p>This was a piece of work carried out with one of our startups, Mastodon C, together with Dr Ben Goldacre - by analysing open prescription data to see how much is spent on expensive “branded” medicines, for one class of drugs, namely statins, in England.</p>
    <p>Identified a £200 million saving</p>
  </aside>
  <p><img src='2013-09-bristol-brug-open-dev-open-data/prescribinganalytics.png'></p>
  <small><a href='http://prescribinganalytics.com/'>http://prescribinganalytics.com/</a></small>
</section>

<section>
  <aside class="notes">
    <p>Show me the money was our first project involving open data from the private sector</p>
    <p>Showing were peer-to-peer lending is coming from and going to - London to everywhere else</p>
  </aside>
  <p><img src='2013-09-bristol-brug-open-dev-open-data/smtm.png'></p>
  <small><a href='http://smtm.labs.theodi.org/'>http://smtm.labs.theodi.org/</a></small>
</section>

<section>
  <aside class="notes">
    <p>OpenCorporates - another startup we incubate</p>
    <p>They carry open data on over 60 million companies from company registers all around the world</p>
    <p>Mapped out data on corporate networks around the world. With countries size altered by number of companies.</p>
    <p>This is Goldman Sachs. That big country is the Cayman Islands. Nuff said</p>
  </aside>
  <p><img src='2014-02-birmingham-wmrug-open-dev-open-data/open-corporates.png'></p>
  <small><a href='http://opencorporates.com/viz/financial/index.html'>http://opencorporates.com/viz/financial/</a></small>
</section>

<!--
<section>
  <aside class="notes"></aside>
  <p><img src='2013-09-bristol-brug-open-dev-open-data/telefonicafootfall.png'></p>
  <small><a href='http://dynamicinsights.telefonica.com/488/smart-steps'>http://dynamicinsights.telefonica.com/488/smart-steps</a></small>
</section> -->

<section>
	<h2>Good Open Data</h2>
	<ul class="centred-list">
    <li><strong>can be linked to</strong><br/>so that it can be easily shared and talked about</li>
    <li><strong>is available in a standard, structured format</strong><br/>so that it can be easily processed</li>
    <li><strong>has guaranteed availability and consistency over time</strong><br/>so that others can rely on it</li>
    <li><strong>is traceable, through any processing</strong><br/>so others can work out whether to trust it</li>
  </ul>
  <aside class="notes">
    So what does good open data look like?
  </aside>
</section>

<section>
	<h2>Open Data Certificates</h2>
  <aside class="notes">
    But how can you check?
  </aside>
</section>

<section>
  <p><img src="2013-09-bristol-brug-open-dev-open-data/certificates.png"></p>
  <small><a href="https://certificates.theodi.org/">https://certificates.theodi.org/</a></small>
   <aside class="notes">
    <p>Open Data Certificates is a simple questionnaire that allows data publishers to identify the Legal, technical, social and practical things that make it easier for people to reuse data.</p>
    <p>Not just an award you can put on your website, a tool that shows you what good open data looks like, and how to improve</p>
   </aside>
</section>

<section>
	<h3>legal, technical,<br/>social, practical...</h3>
  <aside class="notes">

  </aside>
</section>

<section>
  <p><img src="2014-04-leamington-black-pepper-open-dev-open-data/csvlint.png"></p>
  <small><a href="">https://csvlint.io</a></small>
   <aside class="notes">
    <p>CSVlint is a tool we've developed and is currently in alpha</p>
    <p>Allows data publishers to check if they're publishing CSVs in a way that maximises reuse</p>
   </aside>
</section>

<section>
	<h2>The ODI: Redux</h2>
  <aside class="notes">
  </aside>
</section>

<section>
	<ul class="centred-list">
		<li>non-profit, non-partisan</li>
		<li>founded 2012 by Tim Berners-Lee and Nigel Shadbolt</li>
		<li>multiple funding sources</li>
		<li><strong>&quot;helping others be successful with open data&quot;</strong></li>
	</ul>
  <aside class="notes"></aside>
</section>

<section>
	<ul class="centred-list">
		<li>incubating startups</li>
		<li>training</li>
		<li>community building</li>
		<li>research and policy</li>
		<li><strong>software and service development</strong></li>
	</ul>
  <aside class="notes">
    <p>So, how do we help?</p>
  </aside>
</section>

<section>
	<h2>Open Development<br/>at the ODI</h2>
  <aside class="notes"></aside>
</section>

<section id="tech-team">
		<aside class="notes"></aside>
		<h2>ODI Tech Team</h2>
		<ul class="centred-list">
				<li><strong>started January 2013</strong></li>
				<li><strong>mixture of skills</strong><br/>
					data science<br/>
					statistics<br/>
					public data<br/>
					devops<br/>
					engineering<br/></li>
				<li><strong>building tools for open data</strong></li>
				<li><strong>showing best practice</strong></li>
				<li><strong>open culture in code</strong></li>
		</ul>
</section>

<section id="who-are-we">
  <img src='open-development/who-are-we.jpg'/>
  <aside class="notes">
      Started by defining principles
      Who are we - what do we believe
  </aside>
</section>

<section id='what-do-we-do'>
    <img src='open-development/what-do-we-do.jpg'/>
    <aside class="notes">
        how does that translate into practice?
    </aside>
</section>

<section id='open-by-default'>
		<h2>Open By Default</h2>
		<ul>
				<li>encourage public engagement</li>
				<li>plan in the open</li>
				<li>comments, forks, contributions</li>
		</ul>
		<blockquote>
				There are more good ideas in the world than there are in the ODI
		</blockquote>
</section>


<section>
    <aside class="notes"></aside>
    <blockquote>... being Open By Default is exhilarating. But I suspect
        that streaking at a cricket match is exhilarating, too.
        And sometimes it does feel like we're running around naked...</blockquote>
    <small>Sam Pikesley - <a href='http://www.theodi.org/blog/better-living-through-openness'>http://www.theodi.org/blog/better-living-through-openness</a>
    </small>
</section>

<section id='borrow-mercilessly'>
    <h2>Borrow Mercilessly</h2>
    <ul>
        <li>We will build on other's work</li>
        <li>Avoid reinvention</li>
        <li>Focus efforts into new areas</li>
        <li>Contribute back</li>
        <li>Show appreciation for what others have done</li>
    </ul>
</section>

<section id='experiment-and-learn'>
    <h2>Experiment and Learn</h2>
    <ul>
        <li>Try new tools and ways of working</li>
        <li>Reflect continuously on what works</li>
        <li>Adapt to suit, or move on</li>
        <li>Continuous improvement</li>
    </ul>
</section>

<section id='share-everything'>
		<h2>Share Everything</h2>
		<ul>
				<li>Document and discuss experiences</li>
				<li>Create reusable components
						<ul>
								<li>Eat your own dogfood</li>
						</ul>
				</li>
				<li>Share learning to help others</li>
		</ul>
</section>

<section id='be-ourselves'>
    <h2>Be Ourselves</h2>
    <ul>
        <li>We are all individuals</li>
        <li>We work as a team and in a community because we have common purpose</li>
        <li>Express individual opinions, considerately.</li>
    </ul>
</section>

<section>
		<h2>Practicalities</h2>
		<ul>
				<li>Agile Approach</li>
				<li>Open Tools</li>
				<li>Open Communication</li>
				<li>Open Collaboration</li>
		</ul>
</section>



<section id='open-tools'>
    <h2>Open Tools</h2>
</section>

<section>
    <section>
        <aside class="notes">
            This is a tweet from team member Sam from last year
        </aside>
        <h2>Git and GitHub</h2>
        <blockquote>
            How did anybody ever do anything before GitHub?
        </blockquote>
        <small><a href="https://twitter.com/pikesley/status/303100904037556225">https://twitter.com/pikesley/status/303100904037556225</a>
        </small>
    </section>

    <section id="github-theodi">
        <aside class="notes">
            Here's our Github page, with some of our projects. We don't have any private repos. That number is actually out-of-date now, we have over 100 repos, which is remarkable for 12 months work. The particular thing that Github do that we really love is...
        </aside>
        <img src='open-development/github.png'/>
        <small><a href='http://github.com/theodi'>http://github.com/theodi</a></small>
    </section>

    <section id="pull-requests">
        <aside class="notes">
            The Pull Request model. This is how Github work themselves, if you're interested I seriously recommend
            watching this presentation by Zach Holman. The mechanism is...
        </aside>
        <h2>Pull Requests</h2>

        <p>
            Fork (or branch) and Pull
        </p>

        <p>
            <a href='http://zachholman.com/talk/how-github-uses-github-to-build-github/'>How GitHub Uses GitHub to Build
                GitHub</a>
        </p>
        <blockquote>
            I've added something to your code, and I want you to merge it upstream.
        </blockquote>
    </section>

    <section id="workflow">
        <h2>Workflow</h2>
        <ul>
            <li>Clone the repository</li>
            <li>Create a branch</li>
            <li>Write some code (with tests!)</li>
            <li>Publish branch</li>
            <li>Open a pull request back to master</li>
        </ul>
        <aside class="notes">
            GitHub are calling this "GitHub Flow".
            Important aspects: short-lived branches, minimal merges. We shall return to this theme later
        </aside>
    </section>

    <section id="pull-request-description">
        <aside class="notes">
            Here's an example, here James has opened a PR on some of Jeni's code
        </aside>
        <img src='open-development/pull_request_1.png'/>
        <small><a href='https://github.com/theodi/member-directory/pull/72'>https://github.com/theodi/member-directory/pull/72</a>
        </small>
    </section>

    <section id="pull-request-commits">
        <aside class="notes">
            Here we can see the files that have been changed
        </aside>
        <img src='open-development/pull_request_2.png'/>
        <small><a href='https://github.com/theodi/member-directory/pull/72'>https://github.com/theodi/member-directory/pull/72</a>
        </small>
    </section>

    <section id="pull-request-files">
        <aside class="notes">
            And here are the actual changes within the code, all highlighted
        </aside>
        <img src='open-development/pull_request_3.png'/>
        <small><a href='https://github.com/theodi/member-directory/pull/72'>https://github.com/theodi/member-directory/pull/72</a>
        </small>
    </section>

    <section id="pull-request-discussion">
        <aside class="notes">
            A PR is not just about "here's my code", it's also way to facilitate a discussion. We usually open a PR on
            the first commit, and as you can see here, a further commit has been made halfway through, with a
            description of what's changed. It's like a comment thread, but without the shouting.
        </aside>
        <img src='open-development/pull_request_4.png'/>
        <small><a href='https://github.com/theodi/member-directory/pull/72'>https://github.com/theodi/member-directory/pull/72</a>
        </small>
    </section>

    <section id="pull-request-merge">
        <aside class="notes">
            And then the PR is merged. And this is the last place any humans touch the code, but more of that later...
        </aside>
        <img src='open-development/pull_request_5.png'/>
        <small><a href='https://github.com/theodi/member-directory/pull/72'>https://github.com/theodi/member-directory/pull/72</a>
        </small>
    </section>

</section>

<section id="robots">
    <h2>Robots</h2>
    <blockquote>All watched over by machines of loving grace</blockquote>
    <small>Richard Brautigan</small>
</section>

<section>
    <section id="continuous-deployment">
        <aside class="notes">
            Continuous deployment has been a Holy Grail for a long time, but it's always been too scary to
            attempt. As we were starting from scratch, we could bake this in from
            the beginning.
        </aside>
        <h2>Continuous<br/>Deployment</h2>
    </section>

    <section id="jenkins">
        <aside class="notes">
            Jenkins is our Continuous Integration robot, it checks out the code and runs the tests just the same as we
            would do on our laptops
        </aside>
        <img src='open-development/jenkins.png'/>
        <small><a href='http://jenkins.theodi.org'>http://jenkins.theodi.org</a></small>
    </section>

    <section id="ok-to-merge">
        <aside class="notes">
            And we can now hook it into Github so we can see in the PR that this branch has passing tests
        </aside>
        <img src="open-development/good-to-merge.png"/>
    </section>


    <section id="cattle-not-pets">
        <aside class="notes">
            So Jenkins has told us that the code is good to go, so we need to get it out into the wild. In the bad
            old days, we would own a bunch of actual servers, which we'd have to look after and care for over a long
            period of time, and deploy to them probably by hand, which is error-prone and unpleasant. This is the
            classic Snowflake anti-pattern. Not any more. Now we have ephemeral cloud servers, and deployment is in the
            hands of the Chef robots.
        </aside>
        <h2>Treat your servers as cattle, not as pets</h2>
        <a href="http://www.flickr.com/photos/pikesley/2879361159/in/photolist-5ortKV-5udMe4-6hXXdM/">
            <img src="open-development/cows.jpg"/>
        </a>

        <p>No more snowflakes!</p>
    </section>

    <section id="chef">
        <aside class="notes">
            We can now describe our infrastructure as code. Executable Infrastructure is a beautiful thing. This ties
            in nicely with the short-lived, frequently-merged branches from the Git Flow model, and it means we can
            deploy to production many, many times per day without ever thinking about it. As soon as the PR is merged,
            the robots take over.

            Deployments should not be like going to the dentist - we've done this on a Friday afternoon without any real
            terror.
        </aside>
        <h2>Chef</h2>

        <p style="text-align: center;">
            <a href='http://www.opscode.com/chef/'>http://www.opscode.com/chef/</a>
        </p>
        <ul>
            <li>Infrastructure == code</li>
            <li>Continuous deployment</li>
            <li>Deploys latest release</li>
            <li>Holds the keys!</li>
            <li>Code live in 10 minutes</li>
            <li>No human intervention</li>
            <li><a href='http://www.cucumber-chef.org/'>cucumber-chef</a></li>
        </ul>
    </section>

    <section id="cucumber-chef">
        <aside class="notes">
            And it's all testable. Test-driven Executable Infrastructure is an even more beautiful thing. This describes
            what I want this server to do, which provides a lovely safety-harness when I'm making changes.

            If your machines are Identical, Repeatable and Disposable, you'll have a good time.
        </aside>
        <img src="open-development/cuke-chef.png"/>
    </section>
</section>

<section>
    <section>
        <aside class="notes">
            We also use a lot of tools to keep our code up to scratch
        </aside>
        <h2>Code Maintenance </h2>
    </section>

    <section id="codeclimate">
        <aside class="notes">
            We use CodeClimate to get us thinking about how elegant our code is
        </aside>
        <img src='open-development/codeclimate.png' style='max-height: 500px'/>
        <small><a href='https://codeclimate.com/github/theodi/open-orgn-services'>https://codeclimate.com/github/theodi/open-orgn-services</a>
        </small>
    </section>

    <section id="codeclimate-security">
        <aside class="notes">
          (and also let us know about any security issues)
        </aside>
        <img src="open-development/codeclimate-security-alert.png" style='max-height: 500px'/>
    </section>

    <section id="gemnasium">
      <aside class="notes">
        And Gemnasium keeps us notified about our dependencies
      </aside>
        <img src='open-development/gemnasium.png' style='max-height: 500px'/>
        <small><a href='http://gemnasium.com/theodi'>http://gemnasium.com/theodi</a></small>
    </section>
    <section id="bimble">
      <aside class="notes">
        We also have another robot, called Bimble, which runs on our Jenkins box, clones down all our repos, runs `bundle update`, pushes a branch to github and opens a pull request. Our code coverage is good, so if it's green it gets merged.
      </aside>
        <h2>Bimble</h2>

        <img src='open-development/bimble_pr.png' style='max-height: 500px'/>
        <small><a href='https://github.com/theodi/bimble'>https://github.com/theodi/bimble</a></small>
    </section>

    <section id="forkbomb">
      <aside class="notes">
        Forkbomb is a 20% project we worked on just before Christmas, this keeps ours (and your) forks up to date in a similar way to Bimble. It opens a pull request at a given frequency from the base repo to our fork. Very useful for our website (which uses the same codebase as gov.uk - another presentation in itself!)
      </aside>
      <h2>Forkbomb</h2>
      <img src="open-development/forkbomb.png" style='max-height: 500px'/>
      <small><a href='http://alpha.forkbomb.io/'>http://alpha.forkbomb.io/</a></small>
    </section>
</section>

<section id='open-communication'>
    <h2>Open Communication</h2>
</section>
<!--
<section id="blogging">
    <aside class="notes">
        Sam

        There _should_ be a tech team blog post every week, and we also publish weeknotes, wherein we describe what
        we've been up to and what new things we've discovered.
    </aside>

    <h2>Blogging</h2>

    <p style="text-align: center">
        <a href='http://theodi.org/blog'>http://theodi.org/blog</a>
    </p>
    <ul>
        <li>Regular blog schedule</li>
        <li>Tech team weeknotes</li>
    </ul>
</section>-->

<section>
    <section id="irc">
        <aside class="notes">
            We also have a completely open IRC channel. This is where we work (because we're often a distibuted team)
            but this is open to anybody. Here you can see two people who don't actually work at the ODI having a
            discussion that's only tangentially-related to ODI stuff.
        </aside>
        <img src='open-development/irc.png' style='max-height: 500px'/>
        <p style="text-align: center"><a href='irc://irc.freenode.net/theodi'>irc.freenode.net #theodi</a></p>
    </section>

    <section id="odibot">
        <aside class="notes">
            We also have a robot in the IRC channel, who can remind us when stand-ups are happening, describe things
            that have happened in Github, and show us pictures of Admiral Ackbar. He will do more in the future.
        </aside>
        <img src='open-development/odibot.png' style='max-height: 500px'/>
    </section>
</section>

<section>
    <section id="dashboards">
        <h2>Dashboards</h2>
    </section>
    <section id="dashing">
      <aside class="notes">
          We also keep track of all our branches, uptime, and Github stats in a dashing dashboard, which we have in the office.
      </aside>
        <img src="open-development/dashing.png"/>
        <small><a href='http://dashboards.theodi.org/tech'>http://dashboards.theodi.org/tech</a></small>
    </section>

    <section id="company-dashboard">
      <aside class="notes">
          Our CEO loved them so much, he had us do dashboards for the whole organisation (more to come!)
      </aside>
        <img src="2014-02-birmingham-wmrug-open-dev-open-data/company-dashboard.png"/>
        <small><a href='http://dashboards.theodi.org/company/2013'>http://dashboards.theodi.org/company/2013</a></small>
    </section>

</section>

<section>
    <section id='open-collaboration'>
        <h2>Open Collaboration</h2>
    </section>

    <section id="certificate-issues">
        <aside class="notes">
            So all of this working in the open allows things like this to happen. This came about when the certificate
            was not much more than an idea on Jeni's laptop
        </aside>
        <img src='open-development/certificate-issues.png'/>
        <small><a href='https://github.com/theodi/open-data-certificate/issues'>https://github.com/theodi/open-data-certificate/issues</a>
        </small>
    </section>

    <section id="russian-translation">
        <img src='open-development/russian-translation.png'/>
        <small><a href='https://github.com/theodi/open-data-certificate/issues/24'>https://github.com/theodi/open-data-certificate/issues/24</a>
        </small>
    </section>

</section>

<section id="summary">
    <aside class="notes">
        James
    </aside>

    <h2>Working In The Open</h2>
    <ul>
        <li>stops us cutting corners</li>
        <li>catches more mistakes</li>
        <li>gives better results</li>
        <li>helps us find collaborators</li>
        <li>gives something back</li>
        <li>is fun!</li>
    </ul>
</section>



<!-- mention that the slides are data too! -->



{% include odi_tech_team.html %}