Static site conversion

[goldpbear]

Updates/additions/thoughts to this issue welcome!

Converting Mapseed from a dynamically rendered site dependent on Django to a static site would have performance and deployment benefits, and is an important step on the road to being able to offer Mapseed as a service.

To migrate to a static site, we'd need to accomplish the following:

• Extend base.html with content in flavor index.html files (Do flavor-specific extensions within build process #703)
  • One possible approach here would be to rebuild all HTML templates as Handlebars templates, then use Handlebars partials to insert flavor-specific content within our build script
• Find a replacement for Django's localization system (Localize on client side #688)
  • Possibilities here include creating separate static bundles for different languages, or loading localized content dynamically from a microservice
• Replicating the proxy functionality found in views.py client-side (Proxy-less #687)
  • A natural place for this functionality would be to route the endpoints in views.py in the routes.js file instead, and implement handlers for those routes as separate modules (ie shareabouts-api-client.js)
• Migrate email configuration and email logic to client-side js or microservice
• Replace Django's config parsing functionality (Replace Django's config parsing functionality #726)
• Migrate user_token logic from Django's views.py to the client

...are there other to-dos on the way to a static site that we're missing here?

[zmbc]

One possible approach here would be to rebuild all HTML templates as Handlebars templates, then use Handlebars partials to insert flavor-specific content within our build script

To expand on this, we should probably take a look at changing our current method of flavor-specific overrides. Our current system of being able to override things on a file-by-file basis creates very little separation between our public API and our private internals. In order to customize something, a copy of a file is made before being tweaked. That means that an updated version of Mapseed isn't guaranteed (or even particularly likely) to be compatible with old overrides.

In my opinion, we should improve the override system so that:

• Many customizations can be made using a purely public API that is clearly defined (think callbacks, etc)
• If low-level internals need to be changed, they can be changed at the smallest possible scale (i.e. per function) to minimize problems when upgrading

[goldpbear]

That is a fair point. An API that we could use would be great, although I wonder how difficult it would be to abstract something like that effectively? To take a specific example, think about the feature in the pboakland flavor that swapped the show list button and add a place button when a page was open, then swapped them back when a page was closed. Making that work involved tweaking and overriding several methods inside the app view. To abstract something like that, I feel like we'd need generic "insert button" and "hide button" api calls, combined with events we can listen to that signal state changes (user opened a page, user closed a page, etc.). All of which would be great, but the use case is pretty specific.

Still, maybe we can come up with a list of functionality that would make sense to abstract based on the kind of flavor-specific overrides that we've encountered so far. Maybe a nice starting place would be a robust system of events that signal state changes?

And I totally agree with your second bullet above-- no more completely overwritten views! I take the blame for that--I didn't understand how to use the flavor mechanism properly in the past when I extended pboakland's views.

[futuresoup]

Can we resolve this by just having each mapseed deployment be it's own repo instead a flavor within a megarepo like we're currently doing? Soon enough we'll have 20 flavors within the platform repo, which is getting to be a burden, maybe we can solve that problem and not do so many flavor specific overrides?

[goldpbear]

Splitting up the flavors into their own repositories seems like a good thing too--I know you guys have been discussing that. If I'm understanding the scope of this correctly, I think we can make improvements like what @zmbc notes above in addition to splitting flavors up into their own repositories. One change would be on the level of code abstraction and the other on the level of code organization, but both seem like good steps forward.

[modulitos]

in re to your comment @zmbc:

In my opinion, we should improve the override system so that:

Many customizations can be made using a purely public API that is clearly defined (think callbacks, etc)
If low-level internals need to be changed, they can be changed at the smallest possible scale (i.e. per function) to minimize problems when upgrading

And to clarify on what was mentioned by goldpbear, flavors are no longer overriding entire base views (ie the entire file). Here is the PR where this was fixed: https://github.com/mapseed/platform/pull/631/files and here is the line of code where we are using Backbone to extend a view at the function-level: https://github.com/mapseed/platform/blob/master/src/flavors/pboakland/static/js/views/place-detail-view.js#L7

I think this concept of "overriding" should be it's own issue, and implemented after our static site migration is complete. I made an issue for it here: #656, which addresses the need to create a spec for the API.

So I think we should continue with our current implementation where flavors extend the base through the filesystem, which will get us through the static site migration. But I'm open to other ideas! Happy to discuss this at Wayard as well.

[zmbc]

If I'm understanding the scope of this correctly, I think we can make improvements like what @zmbc notes above in addition to splitting flavors up into their own repositories. One change would be on the level of code abstraction and the other on the level of code organization, but both seem like good steps forward.

Agreed.

@Lukeswart wow, not sure how I missed that. That's awesome! We should also do something similar for templates, if we don't already.

On the more practical side of this issue, I've had a branch proxy-less for a little while now. I think I've successfully removed dependency on the proxy from the client-side app, but I've been struggling to remove it from the server, as I'm pretty unfamiliar with that code. Another step that I've not even started yet is to remove the DATASET_KEY check from the API side of things (it's only required on POSTs which is why my branch works at all).

[goldpbear]

@zmbc @Lukeswart -- I've been working on the static site conversion for the last few days. One issue has come up that I'm unsure about: how should we manage static assets that are split between the base project and a given flavor directory? For example, images and marker icons.

Currently Django does all the heavy lifting here, such that we can use a relative url like /static/css/images/markers/marker-housing.png and assume that it will resolve correctly, whether marker-housing.png is in the base project or in the current flavor directory.

But how should the static site manage this? Should we copy all static assets from the base project to flavors during our build process? If we do, should we .gitignore stuff copied in this way? Or, should we assume we'll have one S3 bucket that will host static assets from the base project and separate S3 buckets for each flavor, and make the build process smart enough to covert relative urls like the one above to absolute urls pointing to the correct bucket, depending on the location of a given static asset?

Is there another approach?

[modulitos]

@goldpbear Answers inline, or feel free to ping me anytime:

how should we manage static assets that are split between the base project and a given flavor directory?

I think the general solution here is that we'll create our own CDN somewhere like assets.mapseed.org to serve static assets used by our client-side bundles. For example, we can add assets like assets.mapseed.org/images/markers/marker-housing.png or assets.mapseed.org/maps/bioregions.geojson.

Should we copy all static assets from the base project to flavors during our build process?

We should definitely avoid complicating our build process in this manner.

should we assume we'll have one S3 bucket that will host static assets from the base project and separate S3 buckets for each flavor

This is on the right track - we'll have a CDN/bucket for our static assets (assets.mapseed.org) but each flavor will have its own bucket, where we'll put the bundles of client code. These bundles will include our flavor-specific assets, and webpack will take care of that. For example, they'll be accessible from our flavor bucket in a location like example-flavor.com/images/my-pic.png.

make the build process smart enough to covert relative urls like the one above to absolute urls pointing to the correct bucket, depending on the location of a given static asset?

Maybe I'm missing something - or this might be case-specific - but I think we should be explicit about the locations of our assets. The build process should not interfere with these locations. If the path is relative, then it will be located in our flavor's bucket, ie: example-flavor.com/images/my-pic.png.

This will require some refactoring, because I believe our flavors Django app is inheriting its file structure from our base app, allowing our flavors to override files in the base app. And this Django app logic will need to go, so instead we'll use javascript to reference the correct assets.

And of course, keep me posted! I'm happy to help, when I get some time :-P

[goldpbear]

@Lukeswart -- cool, thanks for the feedback. Some followup questions:

This will require some refactoring, because I believe our flavors Django app is inheriting its file structure from our base app, allowing our flavors to override files in the base app. And this Django app logic will need to go, so instead we'll use javascript to reference the correct assets.

As far as I can tell there isn't so much a flavor app as there is a careful use of the staticfiles app. The staticfiles app is configured to look first in flavor subdirectories, then in the base project. By default it will use the first matching static file it finds. I believe this is how the flavor "override" behavior actually works.

So when we lose this functionality, I'm imagining we'll need to replace it in our build process. Assuming we want to keep the same flavor directory structure and still be able to reference all static files in the same way as we do now (using relative urls, with flavor-specific assets overriding base project assets), how about this approach:

1. Have our build script check all static asset paths (in the flavor config, js templates, and in the flavor's index.html), and determine if a given asset lives in the base project or in the flavor
2. If an asset lives in the flavor, leave the path alone (so we'll have relative urls like /static/css/images/markers/marker-housing.png, which will resolve to the flavor's S3 bucket in production)
3. If an asset lives in the base project, prepend the absolute path (so the above would become https://assets.mapseed.org/static/css/images/markers/marker-housing.png in production)

Is that too convoluted? If I'm understanding correctly the alternative will be to refactor all configs and all templates to convert relative urls to one of two absolute urls: the main mapseed assets store, or the given flavor assets store?

Also, how will any of this work in local development?

[goldpbear]

I've been thinking about this some more. I'm wondering if maintaining a separate assets.mapseed.org bucket is adding unneeded complexity.

Instead, why not just replicate what Django's static files app is already doing in production: create a single staticfiles/ directory for each flavor, and use a build script perform the equivalent of the collect static Django command. In other words, we'd walk a given flavor's static subdirectory and move all static assets there into the staticfiles folder, then walk the base project's static files subdirectory and copy remaining assets. The staticfiles folder would be .gitignored. All references to static files in the config, templates, etc. would then resolve to the flavor's staticfiles directory.

I think this approach is similar to what @Lukeswart cautioned against above, but I think it has some advantages:

• it's effectively the same process that Django already performs to manage static assets
• it makes each flavor bundle completely self-reliant, reducing complexity
• it allows us to continue referencing static assets in the way we always have been, preventing the need for flavor creators to distinguish between flavor assets and base assets
• it maintains the "override" functionality for any static asset

Some potential drawbacks:

• duplication of base static assets across flavor buckets (although storage space is not really at a premium for us)
• the need to propagate changes/updates to base assets out to all flavor buckets (although we could have a batch-update script for this purpose)

What do you think?

[zmbc]

@Lukeswart @goldpbear

We also need to think about this from the perspective of someone using Mapseed, not just us hosting the current flavors. If Mapseed builds sites that reference our S3 buckets,

• We're forcing all Mapseed users to rely on our buckets (if the buckets go down, their site will break)
• If we ever change the S3 buckets, we'll need to maintain the old ones so that older versions of Mapseed don't break

It's possible that we really don't like having static file duplication (although, as Trevor said, I don't think storage space is really that big of a deal for us), but if we decide to do anything about it, that should be some hack/modification to Mapseed, not part of the Mapseed core.

[modulitos]

@goldpbear
I've been thinking about how to address the flavor overrides, and I think my earlier comment got out of scope.
It's not clear to me what the best solution is, but here are my thoughts:

I'm wondering if maintaining a separate assets.mapseed.org bucket is adding unneeded complexity.

In the context of this issue, I think you're definitely right. We can consider the benefits of having an assets bucket at a later point, but it's out of scope. I probably shouldn't have brought that up. @zmbc thanks for your thoughts on this too - we'll keep those in mind.

Instead, why not just replicate what Django's static files app is already doing in production: create a single staticfiles/ directory for each flavor, and use a build script perform the equivalent of the collect static Django command. In other words, we'd walk a given flavor's static subdirectory and move all static assets there into the staticfiles folder, then walk the base project's static files subdirectory and copy remaining assets. The staticfiles folder would be .gitignored. All references to static files in the config, templates, etc. would then resolve to the flavor's staticfiles directory.

This sounds like a good idea :-) But at what point in our build step should we copy over the base static assets into the flavor's staticfiles dir? What happens when we make modifications to our base or flavor's static assets - will we have to re-run our build script to perform the copy again? Since we don't often change our static assets, I think this approach might be worth the tradeoff.

I think I was too optimistic when I thought we could handle the flavor's assets outside of our build step. I do believe we'll come up with a more elegant solution when we refactor our flavor modules, outlined in #656, but that's out of scope for now.

[zmbc]

@Lukeswart I don't think that requiring a build step to copy files is all that bad. Depends, obviously, on how long the build step takes 😃. But I know Jekyll requires a rebuild to update static assets, and I think the same is probably true of most static site generators.

[goldpbear]

I'm thinking we could set up a watch task on the static files subdirectory (in both the base project and the current flavor), then only copy/modify static assets that change during development. We'd have to cover the case where a brand new asset is added, an existing asset is modified, and an asset is deleted.

[zmbc]

Yes, for development we will want to have an overall watch task for everything (static files, webpack, etc).

[zmbc]

🎉 🎉 🎉