README.md

Integrating KoboToolbox and GeoNode

This integration aims to connect your projects in KoboToolbox to the open source, web-based application GeoNode, a platform for developing geospatial information systems (GIS) and for deploying spatial data infrastructures (SDI). In order to connect the two platforms together, an external “integration layer” is necessary to pass information from a project in KoboToolbox, restructure it, and send it off to a GeoNode layer for plotting.

The connection relies on the REST Service feature of KoboToolbox which allows for webhook connections with external applications via a URL. Once the REST Service has been configured in your project, each new submission will be pushed off to the external service, resulting in a one-way data connection. In this case, the external service, from your project’s perspective, is the integration layer which will ultimately complete the connection to your layer on the GeoNode server.

If your project is collecting geographic data using the geopoint question type in your KoboToolbox form, and you are familiar with or have access to a GeoNode server, then this integration may aid your monitoring and analysis processes.

For advanced use-cases, forms can pull existing data from external CSV or XML files using the pulldata() method, and additionally use the KoboToolbox feature of Dynamic Data Attachments for pre-populating fields based on previously collected data.

This tutorial will go through the following steps:

1. Creating a form in KoboToolbox
2. Creating a layer in GeoNode
3. Hosting the integration later
4. Configuring the project’s REST Service in KoboToolbox
5. Viewing submissions on GeoNode layer

Whether you start with creating your form in KoboToolbox or your layer in GeoNode is not important, however there are cases where you need to move between the different platforms as they depend upon each other.

Note that this integration and tutorial has been developed specifically for DHI and their GeoNode server, using DigitalOcean as the hosting service. If changes are needed to enable your GeoNode server to use the integration or you would like to use a service other than DigitalOcean, please create a GitHub issue here or a Pull Request with your changes.

Creating a form in KoboToolbox

When creating a project in KoboToolbox and layer in GeoNode, it is essential that the question names in your form and layer attribute names match, otherwise the connection will not succeed — once the layer has been created, attributes cannot be added or modified. For more details, refer to the GeoNode documentation here.

If you are just getting started with KoboToolbox, you can refer to the support article here on the basics of form creation in the Form Builder. In this example, we will use a simple form with the following questions (download the XLSForm here):

type	name	label
geopoint	location	Record your location
begin_group	person	Person
text	name	What is your name?
integer	age	How old are you?
end_group

The form will have the following appearance in the Enketo Web Form:

Once your form has been created, you can then click DEPLOY in the FORM tab of your project in KoboToolbox.

Handling grouped questions

If your form has grouped questions and you intend to have those fields included in the layer, it is important to note that the JSON representation of the question name includes the full path hierarchy, separated by forward slashes. For example, if we use the form shown above, and a response to the question of “What is your name” is “Bob”, and “How old are you?” is “42”, the JSON structure of the submission will be the following:

{
  "person/name": "Bob",
  "person/age": 42
}

However, GeoNode attribute names cannot contain forward slashes, and therefore need to be accounted for in an alternative manner. When setting up the layer attributes, group hierarchies must be separated with double underscores instead. In the above example, we will set the attribute names to person__name and person__age.

Note that repeat groups are not handled by this integration and cannot be included in the GeoNode layer.

Attribute names can also only contain lowercase letters. The integration layer converts all form question names to lowercase to account for this, while additionally replacing forward slashes with double underscores: a name of Person/Name will be sent to your GeoNode layer as person__name.

Creating a layer in GeoNode

Log into your GeoNode profile. In the navigation menu at the top of the GeoNode interface, click on the Data dropdown menu, and then “Create Layer.”

You will be directed to a page where layer details, attributes, permissions, etc. can be configured. Below is an example of a layer named “kobo_geonode” with “String” attributes of person__name and person__age, matching the form’s question names in KoboToolbox as shown in the previous section. The double underscores account for the group hierarchy of the question.

Note that only the “Points” Geometry type is currently supported in this integration.

After creating the layer, you can add it to any map or explore it as shown below:

Obtain your GeoNode Access Token and Server URL

The GeoNode Access Token is required for authenticating data coming from your KoboToolbox project to the GeoNode server, and the Server URL is required for configuring the integration layer.

If you have an admin account on the GeoNode server, you can find your Access Token by entering the Admin console:

1. Click on the dropdown menu in the top right corner of the interface where your name is displayed and then click on Admin.
2. Within the Admin console, navigate to the section titled “Django OAuth Toolkit” and then to “Access tokens”
3. Within that page, you can find the token associated with your account, as well as all other user account tokens.

If you have a non-admin user account, you can find your Access Token through the following steps:

1. Click on the dropdown menu in the top right corner of the interface where your name is displayed and click on Profile.
2. Click on the link titled “User layers WMS GetCapabilities document” and copy the access_token included in the URL of that page:

http://<geonode-url>/capabilities/user/<username>/?access_token=<your-access-token>

Within the XML document at the above location, search for the term “geoserver/ows” which will be within an element named LegendURL. Copy the URL address:

http://<geo-server-url>:<port>/geoserver/ows

The values <geo-server-url> and <port> will be used to configure the integration’s environment variables.

Hosting the integration layer

This section describes the necessary steps to publish the integration layer between KoboToolbox and a GeoNode server using the cloud service DigitalOcean. The process should be transferable to other cloud service providers, although may require minor code changes. The integration has been built to be deployed as either a serverless function or a Dockerized application, each having their own advantages depending on the use-case.

Option 1: Severless Function App

1. Fork the kobotoolbox/kobo-geonode repository to your own GitHub account.
2. After creating a project in DigitalOcean, click on Create App, or the Create dropdown menu and then Apps.

3. Select GitHub as the Service Provider, authenticate your account (if not already done) then choose kobo-geonode as the Repository, leaving the Branch as main. Set the Source Directory to /serverless/digitalocean. Once complete, click Next.

4. There is no need to configure resources in Edit Plan, however you can change the function’s name. In this example, we will use kobo-geonode. If you changed the name, click Back and then Next.

5. In the Environment Variables section, add the following variables in the kobo-geonode environment (encryption optional): GEO_SERVER_URL, GEO_SERVER_PORT. The values of these variables are found on the GeoNode server, as described in the section titled Obtain GeoNode Access Token and Server URL above. Once complete, click Save and then Next.

Note that if you update any of the environment variables, you will need to deploy the app again by clicking on the Actions dropdown, and then Deploy.

6. Click Next in the Info section.
7. In the Review section, click Create Resources and wait for the application to build — this should take about a minute.
8. Once the application has finished building, there will be a URL in the HTTP ROUTES section of the Overview tab, which will allow you to access the application from the internet. This URL is the endpoint for invoking the function and will be used directly in the REST Services configuration. For example: https://goldfish-app-qwerty12345.ondigitalocean.app/geonode/geonode

Option 2: Dockerized App

1. Fork the kobotoolbox/kobo-geonode repository to your own GitHub account.
2. After creating a project in DigitalOcean, click on Create App, or the Create dropdown menu and then Apps.

3. Select GitHub as the Service Provider, authenticate your account (if not already done) then choose kobo-geonode as the Repository, leaving the Branch as main and source directory as /. Once complete, click Next.

4. This page allows you to edit the resources the application will use. For testing purposes or small loads, you can lower the monthly cost of the application by clicking Edit Plan, select Basic as the plan type and then “$5.00/mo - Basic” as the resource size. Click Back and then Next to the following section.

5. In the Environment Variables section, we will again set the following variables: GEO_SERVER_URL, GEO_SERVER_PORT. Click Save and then Next.
6. Click Next in the Info section.
7. In the Review section, click Create Resources and wait for the application to build, which may take a few minutes. You can watch the build logs in the Activity tab:

8. Once the application has finished building, there will be a URL in the HTTP ROUTES section of the Overview tab, which will allow you to access the application from the internet. We will use this URL when configuring the REST Service in KoboToolbox, pointing to the API endpoint of: <your-app-url>/api/geonode. For example, the full endpoint URL will be something like: https://shark-app-qwerty12345.ondigitalocean.app/api/geonode.

Configuring the project’s REST Service in KoboToolbox

Once the form has been created and deployed, the layer has been created in GeoNode and the integration later is live, we can finally configure the REST Service for the project in KoboToolbox.

1. Within your project in KoboToolbox, navigate to the SETTINGS tab and then to REST Services.

2. Click on REGISTER A NEW SERVICE which will display a modal for configuring the service.

Within the modal, there are several fields which require values obtained in previous steps, such as the integration’s endpoint URL from DigitalOcean (Endpoint URL for the REST Service), and the layer name and Access Token from GeoNode which will be included in the Custom HTTP Headers section as follows:

• geo-layer: <layer name>
• geo-token: <Access Token>

In the Select fields subset section, it is important to only include the fields that were created as attributes in the GeoNode layer. This is with the exception of the field _geolocation, which must be included in the subset, and will be used to plot the point on the layer.

Based on the example layer, kobo_geonode, created earlier, the following fields must be included in the Select fields subset section:

• _geolocation
• person/name
• person/age

If we use the serverless function approach for the hosted integration layer on DigitalOcean, then here is an example of what the REST Service settings will be:

Once these fields have been configured, click CREATE.

Viewing submissions on GeoNode layer

Within your project on KoboToolbox, navigate to the FORM tab, and then click OPEN within the Collect data section. This will open the Enketo Web Form where you can create your first submission.

Enter some values in the form and then click Submit.

Navigate back to your layer in GeoNode, refresh the page, and you should see a point displayed on the map. Click on the point, and our submission values will be displayed alongside: