This migration script is to port over the U.S.F.S. Open Forest system to a new organization within cloud.gov, or presumably another cloud foundry instance. It consists of three applications with code in two separate repositories.
The repositories are the [https://github.com/18f/fs-open-forest-platform] and the [https://github.com/18f/fs-open-forest-middlelayer-api].
Each application is deployed in development, staging, and production environments housed in separate corresponding spaces within the cloud.gov org. For more on cloud.gov orgs and spaces, please see the cloud foundry docs
This migration script does not provide any data held in the s3 buckets or databases.
This repo may also help you manage the connected services of the application, bound within VCAP_SERVICE in the environment variables.
- Overview
- Table of contents
- Migration Requirements
- Environment Variables
- Migration
- After Migration
- Updating Environment Variables
- Utilities
- Notes
- Contributing
- Public Domain
All scripts in this repository are written using BASH. If you are using MacOS, Linux, Unix, this will be available in your default shell. If you are using Windows, please contact your IT support to determine the best way to run BASH scripts.
Prior to running the migration, you will need a local copy of the environment variables for each application.
The user provided services a cloud foundry feature to help manage external 3rd party services. This ecosystem makes heavy use of them in order to manage the credentials of integrated services.
These user provided services are parsed as environment variables in a vcap-constant file in both the platform definitions and middlelayer services. While each repository will give a better indication of the services use, here are some of the expected values and how to generate them.
For more on why we use then instead of environment variables.
These should be organized in a group of json files, one per application:
json-envs/intake/intake-services-dev.jsonjson-envs/intake/intake-services-staging.jsonjson-envs/intake/intake-services-production.jsonjson-envs/middlelayer/middlelayer-services-dev.jsonjson-envs/middlelayer/middlelayer-services-staging.jsonjson-envs/middlelayer/middlelayer-services-production.json
Each file should contain an array of the required services and credentials in the format below. See VCAP_SERVICES for more information.
[
{
"name": "servicename1",
"credentials": {
"key1": "anytype",
"key2": "anytype"
}
},
{
"name":"servicename2",
"credentials" : {
"key3": "anytype",
"key4": "anytype"
},
...
]
For details on the required values see the documentation in the respective repository for each application:
To obtain all environment variables programmatically from existing Cloud Foundry infrastructure:
- Make sure you are in the root of this repository and have installed of the requirements
- Log into the correct
orgwith the Cloud Foundry CLI Ex.cf login -a api.fr.cloud.gov -o usda-forest-service --sso ./bin/get-credentials.sh-- Note this will overwrite existing local files so save a copy if necessary.
./migration.sh
If you would like to not run the migration tasks and just create the new apps and corresponding services within an org run:
./migration.sh false
For each environment, a set of "deployer credentials" is generated for use in other systems that need to interact with Cloud[.]gov programmatically. Circle CI is configured with these credentials in order to trigger deploys to staging and production environments on the successful completion of automatic tests. In addition, certain other credentials, such as those for additional S3 buckets, are generated in Cloud[.]gov to leverage their brokered services. If any of these change, they will need to be updated in the Circle CI user interface as well:
See Middlelayer for instructions on how to generate a user. After a user is generated, the configuration for the 'platform' in the corresponding environment will have to be updated with the generated authentication credentials. See Updating Environment Variables for details.
When credentials change, the environment variables in Cloud[.]gov will need to be updated and the corresponding application restaged. For the example below, we will assume a credential changed for the staging intake application. WARNING these changes are IRREVERSABLE so make sure to use the appropriate Cloud[.]gov space, application, and env variable file.
- Update your local copy by following the instructions here
- Update the value in
json-envs/<env variable file>.json cf t -s <space>cf multi-cups-plugin -p json-envs/<env variable file>.jsoncf restage <application>- where
- env variable file:
intake/intake-services-staging - space:
open-forest-platform-staging - application:
open-forest-platform-staging
- env variable file:
For our vendor integrations we are required to generate and provide certificates in order to sign and/or encrypt the transactions. We provide some useful scripts and instructions to help with this process.
Login[.]gov and EAuth require X509 certificates to integrate. Generate a unique public certificate and private key for each integration and environment:
-
Create a configuration file named
openssl-<integration-environment>.confin thecertificates/conffolder where "integration-environment" is one of:login-tree(-> Open Forest staging)login-prodeauth-dev(-> Open Forest staging)eauth-prod
-
Populate the configuration file with the following:
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
prompt = no
[ req_distinguished_name ]
commonName = <API_APP_URL>
organizationName = USDA
organizationalUnitName = FORESTSERVICE
localityName = Washington
stateOrProvinceName = DC
countryName = US
emailAddress = [email protected]
where "API_APP_URL" is the URL of the application.
-
run
./certificate-generation/make.sh <integration-environment>where "integration-environment" corresponds to the configuration fileThis will generate the following files:
./certificates/keys/saml.key.enc.usda-forest-service-epermit-<integration-environment>./certificates/certs/saml.crt.usda-forest-service-epermit-<integration-environment>
-
If necessary, create a Json Web Key (JWK) from the private key by following the steps in JWK Generation.
-
The appropriate public certificates in
certificates/certsshould be shared with the Eauth/Login[.]gov while the private keys (or corresponding JWK) will be used in our application configuration.
The Login[.]gov integration requires that the private key be provided in the form of a Json Web Key or JWK.
npm installnode certificate-generation/jwkmaker.js ./certificates/keys/saml.key.enc.usda-forest-service-epermit-<integration-environment>"
Pay[.]gov requires a Public Key Certificate to integrate. This section is not verified...
-
obtain
paygov.pfxFROM Pay[.]gov? -
run
openssl pkcs12 -in paygov.pfx -out outfile.pem -nodesThe resulting file will have the
cert, the leveragedcertificate certsand the private key. Thecertificatewill need to be an array of the certificates as strings in the following format:"certificate": [ "-----BEGIN CERTIFICATE-----\ncert1-----END CERTIFICATE-----\n", "-----BEGIN CERTIFICATE-----\ncert2-----END CERTIFICATE-----\n", "-----BEGIN CERTIFICATE-----\ncert3-----END CERTIFICATE-----\n", "-----BEGIN CERTIFICATE-----\ncert4-----END CERTIFICATE-----\n", "-----BEGIN CERTIFICATE-----\ncert5-----END CERTIFICATE-----\n" ],The private key should be added to the
private_keyin the user-provided service. Thepasswordused to generate will need to be added as well.
cf ssh is currently disabled for the production Cloud[.]gov middlelayer-production and public-production spaces.
See CONTRIBUTING for additional information.
This project is in the worldwide public domain. As stated in CONTRIBUTING:
This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.