analytics.dwyl.com 502 -> Please Investigate 🔍 🕵️ 🙏

[nelsonic]

Just tried to view the analytics to grab a screenshot for the issue/task I'm preparing #11
and it's 502-ing ... 😕
https://analytics.dwyl.com/

[nelsonic]

@iteles is aware of and is going to help on this. 👌

[nelsonic]

This is a blocker for our work-work.

[LuchoTurtle]

I restarted all the containers and it's back up again - https://analytics.dwyl.com/.

But I wanted to check for the files and there should be a plausible folder in the droplet (as it was cloned from git).

I wonder what happened to it, can't find it in the VM :/

[iteles]

I forgot my necessary hardware to sort this from the forest this morning but will be home in the afternoon and it's my top priority. Grateful restarting the containers worked 👌

[LuchoTurtle]

I believe this this may have been a resource management issue (it's running on the cheapest machine possible in Droplet).

It's running normally (with a low number of visitors) but the memory is at over 70%. If there's a spike of user activity, memory consumption may increase and can cause the VM to crash, which is the likeliest scenario (the logs indicated connection errors with the app server, which may have been lost because one container may have "blew up").

Ever since restarting, it's working normally.

[nelsonic]

Still getting 502:

[nelsonic]

A week later still 502:

Guessing it's a resource issue.

[LuchoTurtle]

I'll take a look later. Tho I feel like it's a memory issue. Plausible recommends 2GB of RAM, and I chose the cheapest droplet. Should I upgrade?

[nelsonic]

It's nuts to me that Plausible needs 2GB of RAM to run a service that gets a few tens/hundreds of requests per day! 🤯
There must be a memory leak that is causing it to crash. 🤦‍♂️
In terms of the value of our Time ⏳ == 💸 ... we should just upgrade (double the cost) to $12/month:

But I'm not super happy about how the default Plausible instance is deployed and will need to revisit it in future. 💭
Specifically: want(ed) Plausible to use the Managed Postgres instance on DigitalOcean not have all the stack on the same Droplet. 🙃
I get that from a performance and "simplicity" perspective it makes sense to deploy on one instance.
But then when it crashes and burns there is no backup for the data! 🤦‍♂️

[LuchoTurtle]

If that's the case, I'll try to spin up a PostgreSQL instance separate. I've never done a managed instance in DigitalOcean, so it may be a good time to learn. 👌

The problem is how, because the way docker compose for the Community Version is laid out, it assumes everything goes to the same VM. So I'll probably have to do some tweaking

I'm sorting through some stuff, but I'll take a look at it once done

[nelsonic]

We already have the Managed Postgres: https://cloud.digitalocean.com/databases/38253546-fe28-4a33-9a44-cb175b6f698f?i=933568

But it's not currently used for anything cause I hit a brick wall with connecting to it from Fly.io: dwyl/learn-devops#93 (comment)

[LuchoTurtle]

I don't know if something's changed since last time I accessed the server but I can't seem to do it now.

I always get this error, even after regenerating my ssh key. Even on my terminal:

[email protected]: Permission denied (publickey).

I get this error.

Can I destroy and build a droplet from scratch?

[LuchoTurtle]

In order to not do that (so you wouldn't have to change the domain CNAME records again), I restarted the droplet, set a new root password and enabled ssh properly again.

https://www.digitalocean.com/community/questions/i-have-port-22-operation-timed-out-on-my-droplets

Sent the new pw thru Signal

[LuchoTurtle]

It should now be working.

I had to reset the Droplet because I couldn't connect to it through ssh. So I booted the recovery console and reset the password. I've sent all the credentials for both the server and https://analytics.dwyl.com/sites to @nelsonic .

Regarding deployment, I was considering using the PostgreSQL instance Nelson mentioned. However, I went against it because:

• unless PostgreSQL is being used for other applications, it is more expensive. See the images below:

PostgreSQL instance

Droplet

PostgreSQL alone is 15 dollars. The cheapest droplet (if we were to run the Plausible Elixir App there) is 7 dollars. This would mean 22 dollars to run everything.
But even then, the bottleneck is with Clickhouse. According to Plausible:

• CPU must support SSE 4.2 or NEON instruction set or higher (required by ClickHouse).
• At least 2 GB of RAM is recommended for running ClickHouse and Plausible without fear of OOMs.

Therefore, it is much simpler to run everything in a single machine (to maintain/upgrade in the future, since we just need to run a command) instead of separating into two different machines (which is harder to implement). Additionally, it is cheaper to run everything in a single Droplet that is more powerful. I chose a Basic with Premium AMD, which runs for 14 dollars per month. It should work like this (and it has more storage than the PostgreSQL instance too - 25gb over 15gb.

@nelsonic you can delete the PostgreSQL instance if you want to, to save some bucks. I think running everything in the same Droplet is easier to manage in the long run 👌

If everything's oki doki, feel free to close the issue 🙏

[nelsonic]

Using Clickhouse for a site that gets <100 views per day is like using a Lambo to go grocery shopping. 🙃 #overkill
Thanks for investigating this and doing the thorough write-up. ✍️ 👌
DigitalOcean is silly expensive for these smaller projects. 💸 🔥
Currently investigating hetzner dwyl/learn-devops#97 ref: dwyl/mvp#488 (comment)

If I'm able to deploy Postgres Cluster on hetzner and the MVP via Kamal: dwyl/mvp#488 💭 (thoughts?)