Reg_PAPslides.tex

% !TEX TS-program = pdflatex
% !TEX encoding = UTF-8 Unicode

% This file is a template using the "beamer" package to create slides for a talk or presentation
% - Talk at a conference/colloquium.
% - Talk length is about 20min.
% - Style is ornate.

% MODIFIED by Jonathan Kew, 2008-07-06
% The header comments and encoding in this file were modified for inclusion with TeXworks.
% The content is otherwise unchanged from the original distributed with the beamer package.

\documentclass{beamer}
%\usepackage{natbib}

% Copyright 2004 by Till Tantau <tantau@users.sourceforge.net>.
%
% In principle, this file can be redistributed and/or modified under
% the terms of the GNU Public License, version 2.
%
% However, this file is supposed to be a template to be modified
% for your own needs. For this reason, if you use this file as a
% template and not specifically distribute it as part of a another
% package/program, I grant the extra permission to freely copy and
% modify this file as you see fit and even to delete this copyright
% notice. 


\mode<presentation>
{
  \usetheme{Warsaw}
  % or ...

  \setbeamercovered{transparent}
  % or whatever (possibly just delete it)
}

\usepackage{tikz}
\usepackage{graphicx}
\usepackage[english]{babel}
% or whatever

\usepackage[utf8]{inputenc}
% or whatever

\usepackage{times}
\usepackage[T1]{fontenc}
% Or whatever. Note that the encoding and the font should match. If T1
% does not look nice, try deleting the line with the fontenc.


\title[Registrations and Pre-Analysis Plans] % (optional, use only with long paper titles)
{Registrations and Pre-Analysis Plans}

\subtitle
{Making research more transparent and reproducible}

\author[Author, Another] % (optional, use only with lots of authors)
{Garret Christensen\inst{1}}
% - Give the names in the same order as the appear in the paper.
% - Use the \inst{?} command only if the authors have different
%   affiliation.

\institute[Universities of Somewhere and Elsewhere] % (optional, but mostly needed)
{
  \inst{1}%
  Berkeley Initiative for Transparency in the Social Sciences\\
  University of California Berkeley\\
  and\\
  Center for Open Science
}
% - Use the \inst command only if there are several affiliations.
% - Keep it simple, no one is interested in your street address.

\date % (optional, should be abbreviation of conference name)
{BITSS Annual Meeting, 2014}
% - Either use conference name or its abbreviation.
% - Not really informative to the audience, more for people (including
%   yourself) who are reading the slides online

\subject{Research Transparency}
% This is only inserted into the PDF information catalog. Can be left
% out. 



% If you have a file called "university-logo-filename.xxx", where xxx
% is a graphic format that can be processed by latex or pdflatex,
% resp., then you can add a logo as follows:

% \pgfdeclareimage[height=0.5cm]{university-logo}{university-logo-filename}
% \logo{\pgfuseimage{university-logo}}



% Delete this, if you do not want the table of contents to pop up at
% the beginning of each subsection:
%\AtBeginSubsection[]
%{
%  \begin{frame}<beamer>{Outline}
%    \tableofcontents[currentsection,currentsubsection]
%  \end{frame}
%}


% If you wish to uncover everything in a step-wise fashion, uncomment
% the following command: 

\beamerdefaultoverlayspecification{<+->}


\begin{document}


{ % all template changes are local to this group.
    \setbeamertemplate{navigation symbols}{}
    \begin{frame}[plain]
   \textbf{ Welcome to \#BITSS2014!}

Agenda:

Thursday 9A-12P Training

Thursday 1P-5P Conference

Friday 8:30A-2:30P Research Seminar

  \end{frame}
}     



\begin{frame}
  \titlepage
\end{frame}


\begin{frame}{Outline}
  \tableofcontents
  % You might wish to add the option [pausesections]
\end{frame}



% Structuring a talk is a difficult task and the following structure
% may not be suitable. Here are some rules that apply for this
% solution: 

% - Exactly two or three sections (other than the summary).
% - At *most* three subsections per section.
% - Talk about 30s to 2min per frame. So there should be between about
%   15 and 30 frames, all told.

% - A conference audience is likely to know very little of what you
%   are going to talk about. So *simplify*!
% - In a 20min talk, getting the main ideas across is hard
%   enough. Leave out details, even if it means being less precise than
%   you think necessary.
% - If you omit details that are vital to the proof/implementation,
%   just say so once. Everybody will be happy with that.

\section{Problems}

\subsection{Publication Bias}
\begin{frame}{Publication Bias}%{Subtitles are optional.}
  % - A title should summarize the slide in an understandable fashion
  %   for anyone how does not follow everything on the slide itself.
  \pause
  \begin{itemize}
  %\item
   %The distibution of published p-values jumps around .05 (Brodeur et al 2013).
  \item
  There is a higher fraction of rejected hypothesis tests in the social sciences than in physical sciences (Fanelli 2010).
  \item
  	Published null results are disappearing over time, in all disciplines (Fanelli 2011). 
  \item
    This is very unlikely to represent the true state of the universe.
  \item
  	Data on the full set of experiments run with a large survey shows strong results are 40pp more likely to be published, and 60pp more likely to be written up. The file drawer problem is massive. (Franco, Malhotra, Simonovits 2014---see this afternoon)
  \end{itemize}
\end{frame}

\begin {frame}{Publication Bias}
If we only write up/publish significant results, and we have no record of all the insignificant results, we have no way to tell if our `significant' results are real, or if they're the 5\% we should expect due to randomness.
\end{frame}

\subsection{P-Hacking}
\begin{frame}{P-Hacking}
\begin{itemize}
\pause
\item
Also called fishing, researcher degrees of freedom, data mining, data massaging, or specification searching.
\item
Definition: flexibility in data analysis allows portrayal of \textit{anything} as below an arbitrary p-value threshhold; significance loses its meaning.
\item
Not something only evil people do. It can be subconcious---humans are really good at motivated reasoning.
\end{itemize}
\end{frame}

{ % all template changes are local to this group.
    \setbeamertemplate{navigation symbols}{}
    \begin{frame}[plain]
    Does this 
    
    actually happen?
    
    (JLP 2011)
        \begin{tikzpicture}[remember picture,overlay]
            \node[at=(current page.center)] {
                \includegraphics[height=\paperheight]{Prelec1.PNG}
            };
        \end{tikzpicture}
     \end{frame}
     
        
    \begin{frame}[plain]
         \begin{tikzpicture}[remember picture,overlay]
            \node[at=(current page.center)] {
                \includegraphics[height=\paperheight]{Prelec2.PNG}
            };
        \end{tikzpicture}
     \end{frame}
     
     \begin{frame}[plain]
         \begin{tikzpicture}[remember picture,overlay]
            \node[at=(current page.center)] {
                \includegraphics[height=\paperheight]{Prelec3.PNG}
            };
        \end{tikzpicture}
     \end{frame}
}
\begin{frame}{P-Hacking}
\begin{itemize}
\item
Do people actually do this? (Previous---John, Loewenstein, Prelec 2011)
\item
Listening to the Beatles' ``When I'm Sixty-Four'' makes you younger. (Simmons, Nelson, Simonsohn 2011)
\item
Inordinately many .049 p-values, and indordinately few .051's. (Brodeur et al 2013)
\item 
Political ideologues literally see in black and white (Nosek, Spies, Motyl 2012)
\end{itemize}
\end{frame}



\section{Solutions}
\subsection{Registration}
\begin{frame}{Solutions}
\begin{itemize}
\item
Publication Bias---Registration
\item
P-Hacking---Pre-Analysis Plans
\end{itemize}
\end{frame}

\subsection*{Registrations}
\begin{frame}{Registrations}
\pause
 \begin{itemize}
  \item
   Publicly stating all research you will do, and what hypotheses you will test, prospectively.
   %\item If we know every hypothesis test that is run on a given subject, we have a better idea of how seriously to take the significant results.
  \item Store this statement in a public registry.
  \item
   Near universal adoption in medical RCTs. Top journals won't publish if it's not registered. \url{http://clinicaltrials.gov}
  \item
   Even better if registry requires registering outcomes after study. Currently limited, but NIH is moving on this.
\end{itemize}
\end{frame}
\begin{frame}{Registrations}
\begin{itemize}
   \item Newer to social sciences, but:
   \begin{itemize}
   \item
   	AEA registry, currently only for RCTs. \url{http://socialscienceregistry.org}
   \item
    EGAP registry \url{http://egap.org/design-registration}
   \item 
    3ie registry, for developing country evaluations. \url{http://ridie.3ieimpact.org}
   \item
   	Open Science Framework \url{http://osf.io}
   	\begin{itemize}
   	\item
   	Open format
   	\item
   	Will soon sync with above
   	\end{itemize}
   \end{itemize}
  \end{itemize}  
\end{frame}



\subsection{Pre-Analysis Plan}
\begin{frame}{Pre-Analysis Plan}
\pause
\begin{itemize}
\item
Often part of a registration
\item
From 3ie: ``A pre-analysis plan is a detailed description of the analysis to be conducted that is written in advance of seeing the data on impacts of the program being evaluated. It may specify hypotheses to be tested, variable construction, equations to be estimated, controls to be used, and other aspects of the analysis. A key function of the pre-analysis plan is to increase transparency in the research. By setting out the details in advance of what will be done and before knowing the results, the plan guards against data mining and specification searching. Researchers are encouraged to develop and upload such a plan with their study registration, but it is not required for registration.''
\end{itemize}
\end{frame}

\begin{frame}{Glennerster, Takavarasha Suggestions}
\textit{Running Randomized Evaluations}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  the main outcome measures,
\item
  which outcome measures are primary and which are secondary,
\item
  the precise composition of any families that will be used for mean
  effects analysis,
\item
  the subgroups that will be analyzed,
\item
  the direction of expected impact if we want to use a one-sided test,
  and
\item
  the primary specification to be used for the analysis.
\end{enumerate}
\end{frame}

\begin{frame}{McKenzie Suggestions}
\pause
\href{http://blogs.worldbank.org/impactevaluations/a-pre-analysis-plan-checklist}{World Bank Development Impact Blog}

\begin{enumerate}
\item
  Description of the sample to be used in the study
\item
  Key data sources
\item
  Hypotheses to be tested throughout the causal chain
\item
  Specify how variables will be constructed
\item
  Specify the treatment effect equation to be estimated
\item
  What is the plan for how to deal with multiple outcomes and multiple
  hypothesis testing?
\item
  Procedures to be used for addressing survey attrition
\item
  How will the study deal with outcomes with limited variation?
\item
  If you are going to be testing a model, include the model
\item
  Remember to archive it
\end{enumerate}
\end{frame}

\begin{frame}{Simmons, Nelson, Simonsohn (2011)}
\pause
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  Authors must decide the rule for terminating data collection before
  data collection begins and report this rule in the article.
\item
  Authors must collect at least 20 observations per cell or else provide
  a compelling cost-of-data-collection justification.
\item
  Authors must list all variables collected in a study.
\item
  Authors must report all experimental conditions, including failed
  manipulations.
\item
  If observations are eliminated, authors must also report what the
  statistical results are if those observations are included.
\item
  If an analysis includes a covariate, authors must report the
  statistical results of the analysis without the covariate.
\end{enumerate}

\end{frame}

\begin{frame}{Examples}
Wide range of when to write and how detailed to make the plan. At the extreme level of detail you would have your entire code already written before you got any data.
\begin{itemize}
\item
J-PAL Hypothesis Registry (11), see \url{http://www.povertyactionlab.org/Hypothesis-Registry}
\item
AEA Registry has relatively few, plentiful in EGAP.
\item
Casey, Glennerster, Miguel, ``\href{http://www.nber.org/data-appendix/w17012/GBF_Supplementary_Appendix_2011-10-07.pdf}{Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan}'' \textit{QJE} 2012. (\href{http://http://eml.berkeley.edu/~emiguel/pdfs/miguel_gbf.pdf}{Paper},\href{http://www.nber.org/data-appendix/w17012/GBF_Supplementary_Appendix_2011-10-07.pdf}{Plan})
\begin{itemize}
 \item Government-sponsored program.
 \item Broad program (Community Driven Development)
 \item Broad outcomes (trust, public goods, public services, community groups, information, participation, crime, welfare, attitudes)
 \end{itemize} 
\end{itemize} 
\end{frame}

{ % all template changes are local to this group.
    \setbeamertemplate{navigation symbols}{}
    \begin{frame}[plain]
         \begin{tikzpicture}[remember picture,overlay]
            \node[at=(current page.center)] {
                \includegraphics[width=\paperwidth]{GoBifo1.PNG}
            };
        \end{tikzpicture}
     \end{frame}
}

\section{Conclusion}
\begin{frame}{Conclusion}
\pause
 \begin{itemize}
 \item
  Neither registration nor pre-analysis plans are limited to randomized trials. 
 \item
  Spectrum from confirmatory to exploratory research, all has value.
 \item
  I'd just like know which research is which.
\end{itemize}

\end{frame}

\end{document}