Load test megathreads

[zorbathut]

No description provided.

[Raserei408]

I did some very basic testing of large threads. Caveats: running on a 2018 MBP. Tested a mostly flat comment structure with a few deeply nested threads because it was easy to do.

I found, even at ~5000 comments and with nesting up to ~50 threads didn't truncate and it would faithfully render the full thing. (Nested comments eventually stop visibly nesting to maintain a minimum width, but the site preserves the actual structure.) However, substantially before that point page load time suffered considerably. With ~1000 comments page loads took 2.5-3.5 seconds and it took ~10-15 seconds with 5000 (a number I think CW threads have hit on popular weeks in the past).

I don't feel totally confident in this for a few reasons, but it seems like the majority of the time came not from loading the data from the database but from filling out the HTML template. I hoped this would come from some badly written SQL that we could easily fix; this seems much harder to make better without digging deep into the template engine or drastically changing the way the front-end works. We may want to try to implement some form of partial loading / paging to make loading big threads reasonable.

Last note: we should move the "Post a comment" box above the list of comments; it's annoying to have to scroll to the bottom of a very long thread to post a new top-level comment, and weird when it will appear at the top of the thread.

[zorbathut]

From raserei408:

I'm having trouble with my github account (don't want to use my primary, work-associated one; old one I revived got flagged) so I'll say here that I looked at "load-testing megathreads". To summarize, it works in that it seems to generate the page you'd expect with nothing filtered out. However pages take a long time to load at >1000 comments in a thread. (2.5-3.5 seconds for ~1000; 10-15 seconds for 5000). It seems like the super-majority of the time comes from the HTML-templating system (jinja), which might make it hard to improve it directly. We may want to look into some form of load-more / paging functionality to put an upper bound on that. We should also move the "Post a comment" box to sit above all the comments.

[zorbathut]

Further notes:

Basically:

1. Disable rate limiting by commenting out the body of request_methods.apply_rate_limit.
2. Post ~5000 comments in a loop using cURL.
3. Reload page in browser; time taken from dev console window.

Technically I had some nested comments, but I saw problems without it.