Merge branch 'release/1.19.0'

WikiWatershed · Jul 20, 2017 · ab914d4 · ab914d4
2 parents 22541c6 + 3ce4ef3
commit ab914d4
Show file tree

Hide file tree

Showing 219 changed files with 9,262 additions and 3,998 deletions.
diff --git a/README.md b/README.md
@@ -121,6 +121,15 @@ Spark Job Server       | 8090 | [http://localhost:8090](http://localhost:8090)
 
 In order to speed up things up, you may want to consider leveraging the `vagrant-cachier` plugin. If installed, it is automatically used by Vagrant.
 
+To speed up geoprocessing, those requests are cached in Redis. To disable this caching for development purposes, set the value of `MMW_GEOPROCESSING_CACHE` to `0`:
+
+```bash
+$ vagrant ssh worker -c 'echo "0" | sudo tee /etc/mmw.d/env/MMW_GEOPROCESSING_CACHE'
+$ vagrant ssh worker -c 'sudo service celeryd restart'
+```
+
+To enable the geoprocessing cache simply set it to `1` and restart the `celeryd` service.
+
 ### Test Mode
 
 In order to run the app in test mode, which simulates the production static asset bundle, reprovision with `VAGRANT_ENV=TEST vagrant provision`.
@@ -135,6 +144,12 @@ $ ./scripts/test.sh
 
 ##### Python
 
+To check for Python lint:
+
+```bash
+$ ./scripts/check.sh
+```
+
 To run all the tests on the Django app:
 
 ```bash
@@ -151,6 +166,12 @@ More info [here](https://docs.djangoproject.com/en/1.8/topics/testing/).
 
 ##### JavaScript
 
+To check for JavaScript lint:
+
+```bash
+$ ./scripts/npm.sh run lint
+```
+
 When creating new tests or debugging old tests, it may be easier to open the testem page, which polls for changes to the test bundle and updates the test state dynamically.
 
 First, start the testem process.
@@ -203,7 +224,7 @@ This flag is for troubleshooting purposes only.
 
 #### Adding JS dependencies
 
-To add a new JS depenency, update the `JS_DEPS` array in `bundle.sh`, and `package.json` accordingly.
+To add a new JS dependency, update the `JS_DEPS` array in `bundle.sh`, and `package.json` accordingly.
 Because our dependencies are shrinkwrapped, follow the [instructions](https://docs.npmjs.com/cli/shrinkwrap#building-shrinkwrapped-packages) for adding a dependency to a shrinkwrapped package.
 Rebuild the vendor bundle using `./scripts/bundle.sh --vendor`.
 `npm` commands can be run using `./scripts/npm.sh`.
diff --git a/Vagrantfile b/Vagrantfile
@@ -93,6 +93,9 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
     # Path to RWD data (ex. /media/passport/rwd-nhd)
     worker.vm.synced_folder ENV.fetch("RWD_DATA", "/tmp"), "/opt/rwd-data"
 
+    # AWS
+    worker.vm.synced_folder "~/.aws", "/aws"
+
     # Docker
     worker.vm.network "forwarded_port", {
       guest: 2375,

diff --git a/deployment/RELEASES.md b/deployment/RELEASES.md
@@ -35,7 +35,7 @@ If you cannot see columns for `Branch`, `Environment`, and `Service`, use the ge
 
 ## Release Testing
 
-After the `staging-deployment` job completes, `mmw-dev.azavea.com` should reflect the current release. Be sure to run any outstanding database migrations or data imports.
+After the `staging-deployment` job completes, `staging.app.wikiwatershed.org` and `staging.portal.bigcz.org` should reflect the current release. Be sure to run any outstanding database migrations or data imports.
 
 ## AMI Promotion
 
@@ -78,7 +78,7 @@ $ ./mmw_stack.py launch-stacks --aws-profile mmw-prd \
                                --activate-dns
 ```
 
-Within 60 seconds, `app.wikiwatershed.org` and `mmw.azavea.com` should reflect the current release.
+Within 60 seconds, `app.wikiwatershed.org` and `portal.bigcz.org` should reflect the current release.
 
 ## Repository & Jenkins Cleanup
 

diff --git a/deployment/ansible/group_vars/all b/deployment/ansible/group_vars/all
@@ -32,7 +32,7 @@ nodejs_npm_version: 2.1.17
 
 apache_version: "2.4.7-*"
 
-java_version: "7u121-*"
+java_version: "7u131-*"
 
 graphite_carbon_version: "0.9.13-pre1"
 graphite_whisper_version: "0.9.13-pre1"
@@ -46,7 +46,8 @@ sjs_host: "localhost"
 sjs_port: 8090
 sjs_container_image: "quay.io/azavea/spark-jobserver:0.6.1"
 
-geop_version: "1.2.0"
+geop_version: "2.1.0"
+geop_cache_enabled: 1
 
 nginx_cache_dir: "/var/cache/nginx"
 observation_api_url: "http://www.wikiwatershed-vs.org/"
diff --git a/deployment/ansible/roles/model-my-watershed.app/templates/testem-harness.html.j2 b/deployment/ansible/roles/model-my-watershed.app/templates/testem-harness.html.j2
@@ -13,6 +13,7 @@
 <body>
 <div id="mocha"></div>
 <div id="map" style="display: none;"></div>
+<div id="layer-picker-region" style="display: none;"></div>
 <script src="js/test.vendor.js"></script>
 <script src="js/test.bundle.js"></script>
 <script>

diff --git a/deployment/ansible/roles/model-my-watershed.base/defaults/main.yml b/deployment/ansible/roles/model-my-watershed.base/defaults/main.yml
@@ -10,6 +10,7 @@ envdir_config:
   MMW_CACHE_HOST: "{{ redis_host }}"
   MMW_CACHE_PORT: "{{ redis_port }}"
   MMW_TILER_HOST: "{{ tiler_host }}"
+  MMW_GEOPROCESSING_CACHE: "{{ geop_cache_enabled }}"
   MMW_GEOPROCESSING_HOST: "{{ sjs_host }}"
   MMW_GEOPROCESSING_PORT: "{{ sjs_port }}"
   MMW_GEOPROCESSING_VERSION: "{{ geop_version }}"
@@ -19,4 +20,5 @@ envdir_config:
   MMW_STACK_COLOR: "{{ stack_color }}"
   MMW_TILECACHE_BUCKET: "{{ tilecache_bucket_name }}"
   MMW_STACK_TYPE: "{{ stack_type }}"
+
   ROLLBAR_SERVER_SIDE_ACCESS_TOKEN: "Unknown"
diff --git a/deployment/ansible/roles/model-my-watershed.spark-jobserver/tasks/main.yml b/deployment/ansible/roles/model-my-watershed.spark-jobserver/tasks/main.yml
@@ -7,12 +7,6 @@
     - "{{ sjs_home }}"
     - "{{ sjs_data_dir }}"
 
-- name: Copy AWS Crendentials
-  copy: src=~/.aws/ dest=/aws
-  when: "['development', 'test'] | some_are_in(group_names)"
-  notify:
-    - Restart Spark Job Server
-
 - name: Configure Spark Job Server
   template: src="{{ item }}.j2"
             dest="/opt/spark-jobserver/{{ item }}"

diff --git a/...oyment/ansible/roles/model-my-watershed.spark-jobserver/templates/spark-jobserver.conf.j2 b/...oyment/ansible/roles/model-my-watershed.spark-jobserver/templates/spark-jobserver.conf.j2
@@ -38,4 +38,9 @@ spark {
 # spray #
 #########
 
-spray.can.server.parsing.max-content-length = 250m
+spray.can.server {
+  parsing.max-content-length = 250m
+  # Match the backend's timeout set here src/mmw/apps/modeling/tasks.py
+  idle-timeout = 45s
+  request-timeout = 42s
+}
diff --git a/deployment/cfn/application.py b/deployment/cfn/application.py
@@ -20,6 +20,7 @@
     EC2_INSTANCE_TYPES,
     GRAPHITE,
     HTTP,
+    HTTP_ALT,
     HTTPS,
     POSTGRESQL,
     REDIS,
@@ -159,7 +160,7 @@ def set_up_stack(self):
         self.app_server_auto_scaling_schedule_start_recurrence = self.add_parameter(  # NOQA
             Parameter(
                 'AppServerAutoScalingScheduleStartRecurrence', Type='String',
-                Default='0 13 * * 1-5',
+                Default='0 12 * * 1-5',
                 Description='Application server ASG schedule start recurrence'
             ), 'AppServerAutoScalingScheduleStartRecurrence')
 
@@ -173,7 +174,7 @@ def set_up_stack(self):
         self.app_server_auto_scaling_schedule_end_recurrence = self.add_parameter(  # NOQA
             Parameter(
                 'AppServerAutoScalingScheduleEndRecurrence', Type='String',
-                Default='0 1 * * *',
+                Default='0 0 * * *',
                 Description='Application server ASG schedule end recurrence'
             ), 'AppServerAutoScalingScheduleEndRecurrence')
 
@@ -331,7 +332,7 @@ def create_security_groups(self):
                     IpProtocol='tcp', CidrIp=ALLOW_ALL_CIDR, FromPort=p,
                     ToPort=p
                 )
-                for p in [HTTP, HTTPS]
+                for p in [HTTP, HTTP_ALT, HTTPS]
             ],
             Tags=self.get_tags(Name=app_server_security_group_name)
         ))

diff --git a/deployment/cfn/data_plane.py b/deployment/cfn/data_plane.py
@@ -433,11 +433,19 @@ def create_elasticache_replication_group(self):
             SubnetIds=Ref(self.private_subnets)
         ))
 
+        elasticache_parameter_group = self.add_resource(ec.ParameterGroup(
+            'ecpgCacheCluster',
+            CacheParameterGroupFamily='redis2.8',
+            Description='Parameter group for the ElastiCache instances',
+            Properties={'maxmemory-policy': 'allkeys-lru'}
+        ))
+
         return self.add_resource(ec.ReplicationGroup(
             'CacheReplicationGroup',
             AutomaticFailoverEnabled=True,
             AutoMinorVersionUpgrade=True,
             CacheNodeType=Ref(self.elasticache_instance_type),
+            CacheParameterGroupName=Ref(elasticache_parameter_group),
             CacheSubnetGroupName=Ref(elasticache_subnet_group),
             Engine='redis',
             EngineVersion='2.8.19',

diff --git a/deployment/cfn/tiler.py b/deployment/cfn/tiler.py
@@ -150,7 +150,7 @@ def set_up_stack(self):
         self.tile_server_auto_scaling_schedule_start_recurrence = self.add_parameter(  # NOQA
             Parameter(
                 'TileServerAutoScalingScheduleStartRecurrence', Type='String',
-                Default='0 13 * * 1-5',
+                Default='0 12 * * 1-5',
                 Description='Tile server ASG schedule start recurrence'
             ), 'TileServerAutoScalingScheduleStartRecurrence')
 
@@ -164,7 +164,7 @@ def set_up_stack(self):
         self.tile_server_auto_scaling_schedule_end_recurrence = self.add_parameter(  # NOQA
             Parameter(
                 'TileServerAutoScalingScheduleEndRecurrence', Type='String',
-                Default='0 1 * * *',
+                Default='0 0 * * *',
                 Description='Tile server ASG schedule end recurrence'
             ), 'TileServerAutoScalingScheduleEndRecurrence')
 

diff --git a/deployment/cfn/utils/constants.py b/deployment/cfn/utils/constants.py
@@ -23,6 +23,7 @@
 GRAPHITE = 2003
 GRAPHITE_WEB = 8080
 HTTP = 80
+HTTP_ALT = 8080
 HTTPS = 443
 KIBANA = 5601
 POSTGRESQL = 5432

diff --git a/deployment/cfn/vpc.py b/deployment/cfn/vpc.py
@@ -17,6 +17,7 @@
     ALLOW_ALL_CIDR,
     EC2_INSTANCE_TYPES,
     HTTP,
+    HTTP_ALT,
     HTTPS,
     VPC_CIDR
 )
@@ -253,14 +254,14 @@ def nat_security_group(self):
                                           IpProtocol='tcp', CidrIp=VPC_CIDR,
                                           FromPort=p, ToPort=p
                                       )
-                                      for p in [HTTP, HTTPS]
+                                      for p in [HTTP, HTTP_ALT, HTTPS]
                                   ],
                                   SecurityGroupEgress=[
                                       ec2.SecurityGroupRule(
                                           IpProtocol='tcp',
                                           CidrIp=ALLOW_ALL_CIDR,
                                           FromPort=port, ToPort=port
-                                      ) for port in [HTTP, HTTPS]
+                                      ) for port in [HTTP, HTTP_ALT, HTTPS]
                                   ],
                                   Tags=self.get_tags(Name=nat_security_group_name)),  # NOQA
                                   'NATSecurityGroup'

diff --git a/deployment/cfn/worker.py b/deployment/cfn/worker.py
@@ -156,7 +156,7 @@ def set_up_stack(self):
         self.worker_auto_scaling_schedule_start_recurrence = self.add_parameter(  # NOQA
             Parameter(
                 'WorkerAutoScalingScheduleStartRecurrence', Type='String',
-                Default='0 13 * * 1-5',
+                Default='0 12 * * 1-5',
                 Description='Worker ASG schedule start recurrence'
             ), 'WorkerAutoScalingScheduleStartRecurrence')
 
@@ -170,7 +170,7 @@ def set_up_stack(self):
         self.worker_auto_scaling_schedule_end_recurrence = self.add_parameter(  # NOQA
             Parameter(
                 'WorkerAutoScalingScheduleEndRecurrence', Type='String',
-                Default='0 1 * * *',
+                Default='0 0 * * *',
                 Description='Worker ASG schedule end recurrence'
             ), 'WorkerAutoScalingScheduleEndRecurrence')
 

diff --git a/deployment/default.yml.example b/deployment/default.yml.example
@@ -10,7 +10,7 @@ BastionHostInstanceType: 't2.micro'
 BastionHostAMI: ''
 PrivateHostedZoneName: 'mmw.internal'
 PublicHostedZoneName: 'staging.app.wikiwatershed.org'
-BackwardCompatPublicHostedZoneName: 'mmw-dev.azavea.com'
+BackwardCompatPublicHostedZoneName: 'staging.portal.bigcz.org'
 RDSInstanceType: 'db.t2.micro'
 RDSDbName: 'modelmywatershed'
 RDSUsername: 'modelmywatershed'

diff --git a/doc/arch/adr-001-feature-flags.md b/doc/arch/adr-001-feature-flags.md
@@ -0,0 +1,57 @@
+# 001 - Feature Flags
+
+## Context
+
+We need to add a mechanism to toggle features in MMW so that we can deploy
+multiple implementations from the same codebase. For example, in BiG CZ,
+we'll need to disable Analyze mode and other MMW-specific features such as
+ITSI integration.
+
+Primary deciding factors:
+
+* How will this impact development?
+* How easy will it be to switch from MMW to BiGCZ?
+* How will this impact deployments?
+
+Here are the options that I considered:
+
+### [gutter-django](https://github.com/disqus/gutter-django)
+* Too complicated
+* Not actively maintained
+
+### [gargoyle-yplan](https://gargoyle-yplan.readthedocs.io/en/latest/)
+* Actively maintained fork
+* Documentation not great
+* Can add default values in `settings.py`
+* Flags are stored in database
+
+### [django-waffle](https://waffle.readthedocs.io/en/v0.11.1/)
+* Popular
+* Good documentation
+* Not very active
+* Has JavaScript interface
+* Flags are stored in database
+
+### None of the above (custom code)
+* No database required
+* We already have some conventions in place for custom settings
+
+## Decision
+
+We have decided to follow existing conventions to add custom settings
+to MMW to support BiG CZ instead of adding a feature flipper library.
+
+Because flags will rarely change, and we don't need any advanced features
+beyond swiching boolean flags, the overhead of adding and maintaining another
+dependency may not be worthwhile.
+By storing custom settings in Python instead of the database, we'll avoid
+the need to enhance our deployment process to support loading data
+fixtures. Switching between MMW and BiG CZ will be done by using
+an environment variable.
+
+## Consequences
+
+There may be long-running consequences to disabling entire features using
+flags, no matter which implementation we choose. We'll need to be
+careful to make sure that the work we do on BiG CZ doesn't negatively
+impact the functionality of MMW and vice versa.