Custom curator url for webapp-only installation

.gitignore

@@ -20,5 +20,8 @@ GITHUB_CLIENT_SECRET
 GITHUB_APP_PRIVATE_KEY_PEM
 smasher/lib/json-simple-1.1.1.jar
 deploy/setup/CONFIG
+env
+web2py
+restart.sh
 webapp/static/statistics/
 lint/

LICENSE.txt

@@ -1,7 +1,7 @@
-Copyright (c) 2013, Jonathan Rees
-Copyright (c) 2013, Mark Holder
-Copyright (c) 2013, Jim Allman
-Copyright (c) 2013, Stephen Smith
+Copyright (c) 2013-2016, Jonathan Rees
+Copyright (c) 2013-2016, Mark Holder
+Copyright (c) 2013-2016, Jim Allman
+Copyright (c) 2013-2016, Stephen Smith
 
 All rights reserved.
 

README.md

@@ -1,101 +1,89 @@
-opentree
-========
+# opentree
 
-This is the repository for the Open Tree of Life web applications, one of many subsystems making up the Open Tree of Life project code.
+This is the repository for the Open Tree of Life web applications, one of many subsystems making up
+the Open Tree of Life project code.
+For Open Tree of Life documentation, see
+[the germinator repository's wiki](https://github.com/OpenTreeOfLife/germinator/wiki).
+The 'deployment system' and web API documentation sources that formerly resided in this
+repository now live in the [germinator repository](https://github.com/OpenTreeOfLife/germinator).
+The following instructions have not been reviewed in a long time.
+For local installation a better place to start might be
+[this wiki page](https://github.com/OpenTreeOfLife/opentree/wiki/Installing-a-local-curator-and-tree-browser-test-server).
 
-For Open Tree of Life documentation, see [the germinator repository's wiki](https://github.com/OpenTreeOfLife/germinator/wiki). The 'deployment system' and web API documentation sources that formerly resided in this repository now live in the [germinator repository](https://github.com/OpenTreeOfLife/germinator).
-
-The following instructions have not been reviewed in a long time. For local installation a better place to start might be [this wiki page](https://github.com/OpenTreeOfLife/opentree/wiki/Installing-a-local-curator-and-tree-browser-test-server).
-
-Installation
-============
-See the phylografter instructions for
-more details about using web2py.
+## Installation
 
 We strongly recommend using a virtual environment to manage the version of
-Python and installed modules. We're currently running opentree with Python
-v2.7.3. Newer versions of python2.7 should work, but **NOTE that web2py is not
+Python and installed modules.
+We're currently running opentree with Python
+v2.7.3.
+Newer versions of python2.7 should work, but **NOTE that web2py is not
 compatible with Python 3**.
+The final invocation to create your virtualenv should look something like:
 
-If necessary, compile Python2.7 and use it when making your virtualenv.  You
-should be able to safely install multiple versions of python using your
-preferred package manager, or by configuring Python2.7 with the --prefix
-option and 'make altinstall'.
-
-So the final invocation to create your virtualenv should look something like:
-```
-$ virtualenv --python=/usr/bin/python2.7 --distribute <path/to/new/virtualenv/>
-```
+    $ virtualenv --python=(which python2.7) --distribute env
+    $ source env/bin/activate
 
-Or, if you're using virtualenvwrapper (http://virtualenvwrapper.readthedocs.org/en/latest/index.html):
-```
-$ mkvirtualenv --python=python2.7 --no-site-packages --distribute opentree
-```
 
 The included **requirements.txt** file lists known-good versions of all the required
 python modules for opentree, plus a few convenience modules. To [install these modules 
 using pip](http://www.pip-installer.org/en/latest/cookbook.html#requirements-files), 
 
-<pre>
-pip install -r requirements.txt
-</pre>
+    pip install -r requirements.txt
+
+### install web2py and link to applications
 
 The contents of the webapp subdirectory are a web2py application.  Make a symbolic 
 link called "opentree" in a web2py/applications directory to the webapp directory.
 You should be able to launch web2py and see the app running at http://127.0.0.1:8000/opentree/
 
-There is now a second web2py app for the curation tool, which will also need a
-symlink. This will be available at http://127.0.0.1:8000/curator/
+    wget --no-verbose -O web2py_2.8.2_src.zip \
+        https://github.com/web2py/web2py/archive/R-2.8.2.zip
+    unzip web2py_2.8.2_src.zip
+    mv web2py_2.8.2_src web2py
+    cd web2py/applications
+    ln -s ../../webapp opentree
+    cd -
+    cp -p oauth20_account.py web2py/gluon/contrib/login_methods/
+    cp -p rewrite.py web2py/gluon/
+    cp -p custom_import.py web2py/gluon/
+    cp -p SITE.routes.py web2py/routes.py
 
-Briefly:
 
-1. Download and unpack the source code version of web2py from 
-http://www.web2py.com/examples/default/download MTH used version 2.4.2 of web2py
+Optionally, you can install a second web2py app for the curation tool, which will also need a
+symlink. This will be available at http://127.0.0.1:8000/curator/
 
-   NOTE: This version of web2py includes basic support for OAuth 2.0, but it needs
-   a minor patch to support for login via the GitHub API v3.  (The curation app
-   uses GitHub for its datastore and attribution. The tree browser also uses it
-   for its issue tracker, with optional authentication for convenience.) Replace
-   this web2py file with a modified version in the same folder as this README:
-   <pre>
-   {web2py-2.4.4}/gluon/contrib/login_methods/oauth20_account.py
-   </pre>
+    cd web2py/applications
+    ln -s ../../curator curator
+    cd -
 
-2. Create the sym links for the main web app and the study curation tool.
+## Configuring the application
+Copy and tweak the template `config` files to provide the variables that will
+be available during execution of the web applications:
 
-   <pre>
-   cd web2py/applications
-   ln -s /full/path/to/opentree/webapp opentree
-   ln -s /full/path/to/opentree/curator curator
-   </pre>
+    for app in webapp curator ; do 
+        cp ${app}/private/config.example ${app}/private/config
+    done
 
-3. Customize web2py's site-wide routing behavior using "SITE.routes.py"
+Edit each `${app}/private/config` file to use the API servers and properties that you
+    want to use for debugging purposes.
 
-   <pre>
-   # return to main web2py directory
-   cd ..  
-   cp /full/path/to/opentree/SITE.routes.py routes.py
-   </pre>
-
-   This routing file works in tandem with the opentree app router and lets us have
-   proper URLs with hyphens instead of underscores.
 
-4. Launch web2py
+## Launch web2py for debugging
 
-   <pre>
-   cd /full/path/to/web2py
-   python web2py.py --nogui -a '&lt;recycle&gt;'
-   </pre>
+    cd web2py
+    python web2py.py --nogui -a '<recycle>'
 
    Where the -a flag is allowing you to reuse the previous admin password that you used
    with this instance of web2py.
 
+## For an instance that allows logging in
+
    **To test with login and proper domain name**, modify your test system's
    `/etc/hosts` file (or equivalent) to resolve the domain `devtree.opentreeoflife.org`
    to localhost (127.0.0.1). Then run web2py on (privileged) port 80 like so:
 
    <pre>
-   cd /full/path/to/web2py
+   cd web2py
    sudo python web2py.py --nogui -p 80 -a '&lt;recycle&gt;'
    </pre>
 

curator/controllers/collection.py

@@ -14,7 +14,7 @@
 #########################################################################
 
 from applications.opentree.modules.opentreewebapputil import(
-    get_opentree_services_method_urls, 
+    get_opentree_api_endpoints, 
     fetch_current_TNRS_context_names,
     fetch_trees_queued_for_synthesis,
     get_maintenance_info)
@@ -27,7 +27,7 @@ def index():
     Show list searchable/filtered list of all collections
     (default filter = My Collections, if logged in?)
     """
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['maintenance_info'] = get_maintenance_info(request)
     if auth.is_logged_in():
         # user is logged in, filter to their own collections by default?
@@ -46,7 +46,7 @@ def view():
     ? OR can this include work-in-progress from a personal branch?
     """
     response.view = 'collection/edit.html'
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['maintenance_info'] = get_maintenance_info(request)
     #view_dict['taxonSearchContextNames'] = fetch_current_TNRS_context_names(request)
     view_dict['collectionID'] = request.args[0] +'/'+ request.args[1]
@@ -65,7 +65,7 @@ def create():
     if maintenance_info.get('maintenance_in_progress', False):
         redirect(URL('curator', 'default', 'index', vars={"maintenance_notice":"true"}))
         pass
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['message'] = "collection/create"
     return view_dict
 """
@@ -80,7 +80,7 @@ def edit():
             args=request.args))
     # Fetch a fresh list of search contexts for TNRS? see working example in
     # the header search of the main opentree webapp
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['taxonSearchContextNames'] = fetch_current_TNRS_context_names(request)
     view_dict['treesQueuedForSynthesis'] = fetch_trees_queued_for_synthesis(request)
     view_dict['collectionID'] = request.args[0] +'/'+ request.args[1]
@@ -108,6 +108,17 @@ def load():
 def store():
     return dict(message="collection/store")
 
+def synthesis_dashboard():
+    """
+    Allow any visitor to view (read-only!) a queue of recent custom-synthesis runs
+    """
+    response.view = 'collection/synthesis_dashboard.html'
+    view_dict = get_opentree_services_method_urls(request)
+    view_dict['maintenance_info'] = get_maintenance_info(request)
+    view_dict['taxonSearchContextNames'] = fetch_current_TNRS_context_names(request)
+    view_dict['userCanEdit'] = auth.is_logged_in() and True or False
+    return view_dict
+
 """ TODO: Adapt this for current collection status, based on new APIs """
 def _get_latest_synthesis_details_for_collection_id( collection_id ):
     # Fetch the last SHA for this collection that was used in the latest
@@ -118,7 +129,7 @@ def _get_latest_synthesis_details_for_collection_id( collection_id ):
         import json
         import requests
 
-        method_dict = get_opentree_services_method_urls(request)
+        method_dict = get_opentree_api_endpoints(request)
 
         # fetch a list of all studies and collections that contribute to synthesis
         # TODO: Request that these fields be added
@@ -137,15 +148,6 @@ def _get_latest_synthesis_details_for_collection_id( collection_id ):
         # Draft code is based on schema proposed in
         # https://github.com/OpenTreeOfLife/phylesystem-api/issues/228
 
-        # fetch the full source list, then look for this study and its trees
-        commit_SHA_in_synthesis = None
-        # if key (collection ID, e.g. "opentreeoflife/default") matches, read its details
-        for c_id, collection_details in source_dict.items():
-            if c_id == collection_id:
-                # this is the collection we're interested in!
-                commit_SHA_in_synthesis = collection_details['git_sha']
-        return commit_SHA_in_synthesis  # TODO: return more information?
-
         # fetch the full source list, then look for this collection and its SHA
         # if key (collection ID, e.g. "opentreeoflife/default") matches, read its details
         for c_id, collection_details in source_dict.items():
@@ -155,4 +157,5 @@ def _get_latest_synthesis_details_for_collection_id( collection_id ):
         return None
     except Exception, e:
         # throw 403 or 500 or just leave it
-        raise HTTP(500, T('Unable to retrieve latest synthesis details for collection {u}'.format(u=collection)))
+        ##raise HTTP(500, T('Unable to retrieve latest synthesis details for collection {u}'.format(u=collection_id)))
+        raise HTTP(500, T('Unable to retrieve latest synthesis details for collection {u}:\n\n{e}'.format(u=collection_id, e=e)))

curator/controllers/default.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 from applications.opentree.modules.opentreewebapputil import(
-    get_opentree_services_method_urls,
+    get_opentree_api_endpoints,
     extract_nexson_from_http_call,
     fetch_github_app_auth_token,
     get_maintenance_info)
@@ -29,7 +29,7 @@ def index():
     a logged-in user.
     """
     #response.flash = T("Welcome to web2py!")
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['maintenance_info'] = get_maintenance_info(request)
 
     if False:  ## auth.is_logged_in():
@@ -45,12 +45,12 @@ def collections():
 
     TODO: move to collection/index?
     """
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['maintenance_info'] = get_maintenance_info(request)
     return view_dict
 
 def error():
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     return view_dict
 
 @auth.requires_login()
@@ -78,7 +78,7 @@ def profile():
     shows a personalized profile for any user (default = the current logged-in user) 
     http://..../{app}/default/profile/[username]
     """
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['maintenance_info'] = get_maintenance_info(request)
 
     # if the URL has a [username], try to load their information
@@ -192,7 +192,7 @@ def _get_opentree_activity( userid=None, username=None ):
         'added_collections':[],
         'curated_collections':[]
     }
-    method_dict = get_opentree_services_method_urls(request)
+    method_dict = get_opentree_api_endpoints(request)
 
     # Use GitHub API to gather comments from this user, as shown in
     #   https://github.com/OpenTreeOfLife/feedback/issues/created_by/jimallman
@@ -470,6 +470,7 @@ def to_nexson():
     _LOG = get_logger(request, 'to_nexson')
     if request.env.request_method == 'OPTIONS':
         raise HTTP(200, T('Preflight approved!'))
+
     orig_args = {}
     is_upload = False
     # several of our NexSON use "uploadid" instead of "uploadId" so we should accept either
@@ -622,7 +623,7 @@ def to_nexson():
                 try:
                     assert(os.path.exists(exe_path))
                 except:
-                    response.view = 'generic.json'; return {'hb':exe_path}
+                    #response.view = 'generic.json'; return {'hb':exe_path}
                     _LOG.warn("Could not find the 2nexml executable")
                     raise HTTP(501, T("Server is misconfigured for 2nexml conversion"))
                 invoc = [exe_path, '-f{f}'.format(f=inp_format), ]

curator/controllers/study.py

@@ -12,7 +12,7 @@
 #########################################################################
 
 from applications.opentree.modules.opentreewebapputil import(
-    get_opentree_services_method_urls, 
+    get_opentree_api_endpoints, 
     fetch_current_TNRS_context_names,
     fetch_trees_queued_for_synthesis,
     get_maintenance_info)
@@ -25,7 +25,7 @@ def index():
     Show list searchable/filtered list of all studies
     (default filter = My Studies, if logged in?)
     """
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
 
     if auth.is_logged_in():
         # user is logged in, filter to their own studies by default?
@@ -44,7 +44,7 @@ def view():
     ? OR can this include work-in-progress from a personal branch?
     """
     response.view = 'study/edit.html'
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['maintenance_info'] = get_maintenance_info(request)
     #view_dict['taxonSearchContextNames'] = fetch_current_TNRS_context_names(request)
     view_dict['studyID'] = request.args[0]
@@ -61,7 +61,7 @@ def create():
     if maintenance_info.get('maintenance_in_progress', False):
         redirect(URL('curator', 'default', 'index', vars={"maintenance_notice":"true"}))
         pass
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['message'] = "study/create"
     return view_dict
 
@@ -76,7 +76,7 @@ def edit():
             args=request.args))
     # Fetch a fresh list of search contexts for TNRS? see working example in
     # the header search of the main opentree webapp
-    view_dict = get_opentree_services_method_urls(request)
+    view_dict = get_opentree_api_endpoints(request)
     view_dict['taxonSearchContextNames'] = fetch_current_TNRS_context_names(request)
     view_dict['treesQueuedForSynthesis'] = fetch_trees_queued_for_synthesis(request)
     view_dict['studyID'] = request.args[0]
@@ -93,7 +93,7 @@ def _get_latest_synthesis_details_for_study_id( study_id ):
         import json
         import requests
 
-        method_dict = get_opentree_services_method_urls(request)
+        method_dict = get_opentree_api_endpoints(request)
 
         # fetch a list of all studies that contribute to synthesis
         fetch_url = method_dict['getSynthesisSourceList_url']