Scope of the OWA SWMM Project

[bemcdonnell]

@bemcdonnell I thought that the focus of this project was to supply SWMM with a comprehensive API, only making changes to the core engine code when it was absolutely necessary, thus making it easier to keep in synch with the official EPA version of the code. Although I'm sure there are lots of places where refactoring might be called for, my response would be "if it ain't broke don't fix it".

That being said, I would endorse someone or group forking the code to another project (not just a branch) and re-architecting it from the ground up (not just applying piecemeal refactoring to the current code). It's been almost 15 years since SWMM 5 was first released so maybe the software development cycle has run its course.

One source of ideas for what the re-design might look like can be found in a white paper I wrote for the re-design of EPANET. EPANET and SWMM have a lot in common from a structural point of view so the ideas presented there may be relevant.

Originally posted by @LRossman in #252 (comment)

[bemcdonnell]

@LRossman, It's clear to me that the community perceives this project to be only extending the API to support pyswmm. We do not want to be limited to that at all. The story here is more one the hues to the tone of modernizing the engine. We have worked on porting code, unit/regression testing, API, coverage, autodocumentation so far… and we want to take on bigger architectural challenges that support plugins (features and solvers) and such. We are building a platform to launch from.

But what I feel is happening in our goal of modernization is that we are unlocking the most value by expanding the API. It's a quick win and it's a void that hasn't been filled much with SWMM (unlike EPANET They've had programmatic access for many years so now the discussion is on architecture more than API). Much of the roadmapping inspiration for this project has come form EPANET (like reentrancy/thread safety).

The biggest question I have comes back to human resource allocation. Since funding for major code refactoring isn't typically available, I think it's important to be realistic and strategic on how we get to the end goal. It's not glamorous work per se, but will have to be done eventually. OWA is a great place to host the rearchitecting of the engine. At the same time, progress toward the goal will best take place with the incorporation of key "code developer" stakeholders. Watching the last couple years of thought-centers dreaming up the future of SWMM has been interesting. The ideas that percolate up and out are nice and they are important things to consider when we think of new features/new solvers/ and new architecture.

I think the approach to refactoring this huge codebase could be done in smaller consumable pieces. Dropping a codebase like this and starting from scratch is a HUGE effort and can take years. What I am imagining instead is over time doing something more staged and coordinated. Think of it like points and vectors. We can take minor steps toward our end-goal. We draw the final resultant from now to where we want to be and we start the journey to get there. It might take longer but the fact is, we will always have solid benchmarks to compare back to. It’s broken into consumable steps so that our contributors are all working toward the same end goal.

One high-level roadmap might be like this:

1. encapsulating the data model and exposing functional access to the getters and setters.
2. separate concerns in the project I/O. Disaggregate functions that parse, validate, and set data model into three separate functions
3. separate concerns - allowing functions to have argument inputs (thinking about unit testing here)
4. Simultaneously growing the API

Each step along this proposed roadmap has solid benchmarks and unit tests. Lots of code will be added at first only to be removed later. But that’s part of the process. Re-writing from scratch is an enormous mountain to climb. It can certainly be done, but I would worry that along the way there will be lots of incredible challenges with resources.

The other important detail is that this is a community-driven roadmap. If, for example, I came up with the complete plan and went after it, people would probably not care at all! But if we approach this plan and have open discussions about it (like next-net), we might have great luck with success. The issue the community is facing is that no-one knows where to have this discussion. Fact is, talking about code is one thing, doing it is another. Who else has made so much visible code progress? I’m excited for the community to put ideas into action. Moreover, I’m excited to see a community provide solutions instead of criticisms so we can all feel good about making SWMM’s future bright.

One day we will all wake up after all these steps we took and think to ourselves. “You know what??? I think we just released SWMM6” :-)

[dickinsonre]

Thanks for the very detailed white paper @LRossman on EPANET 3 and I can see how many of your ideas can be applied to SWMM as well. Thank you as well @bemcdonnell for your time and lengthy comments on OWA. I think your Agile approach is much better and more current than a Waterfall approach to refactoring.

[bemcdonnell]

@dickinsonre and @LRossman, Thank you back! It's both humbling and meaningful to me that you are both willing to take time out of your days to participate in the discussions and share code here. @LRossman, I read the EPANET3 proposed plan before and I think there is a lot of great content in there!