fix download counter

[mreid-tt]

Description

This PR implements the download counter function and includes:

• rewrite catalog download links using md5 hashes for id
• allow downloads of noarch builds to pass arch checks
• rewrite nas tests for new url structure

fixes #22

Notes

While this implementation is functional, it lacks efficiency as it generates a database entry for every download request from a client. As mentioned in #112 (comment), the platform currently manages approximately 3.5 million requests per day with a 97% hit ratio on Fastly as CDN. Implementing this would lead to a significant surge in server traffic, potentially causing platform outages for users due to approximately a 33x increase in load.

Given the above, help is requested to find a more efficient implementation of this function.

[hgy59]

@mreid-tt If you find a solution for the download counters, it would be nice to display the counters somewhere in the admin UI.

[mreid-tt]

hey @Diaoul @publicarray, would either of you be able to work with me to get the logs from the Fastly CDN to begin developing a parsing process for them?

[publicarray]

@mreid-tt Fastly can send logs to another platform but doesn't store logs itself. At least not with our current plan
https://docs.fastly.com/en/guides/integrations#_streaming-logs

FYI we average about 38 requests per second.

I also remember that I did for a brief period had logs go to sumologic so that I could debug an issue and after the issue was fixed the consensus at the time was not to send logs to it anymore

@Diaoul

[publicarray]

The last 24h:

[mreid-tt]

@publicarray Thanks for your response! Regarding the logging options, we don’t need anything too complex. A daily log dump via FTP or SFTP to our server would suffice. From there, the log could be parsed, ingested into our database, and then deleted.

Based on our traffic of 38 requests/second (approximately 3.3 million requests/day), the uncompressed log file might be around 3.1 GB. To manage this, we could allocate a dedicated 5 GB partition for uploads, ensuring it only holds one day of logs. This way, if a cron job fails, uploads will be blocked rather than risking the server running out of space.

Would love to hear your thoughts on this approach.

[publicarray]

Good Idea, Fastly can push logs via SFTP and auto rotate logs: https://docs.fastly.com/en/guides/log-streaming-sftp. But aren't we usually low on storage on that server?

[mreid-tt]

Good Idea, Fastly can push logs via SFTP and auto rotate logs: https://docs.fastly.com/en/guides/log-streaming-sftp. But aren't we usually low on storage on that server?

You are correct, the last daily storage report looks like this:

Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1        79G   62G   14G  82% /
53G     /etc/synocommunity/data

We may not need as much space as a whole day since the documentation for setting this up has this line:

By default, logs are placed in your root directory every hour using the file naming format YYYY-mm-ddThh:mm:ss-.

Thus if we process the logs hourly as they are received we should have much less of a storage risk.

[Diaoul]

It some point I would love to move hosting because currently I don't think I'm paying a fair price for this VPS at 15 EUR per months for 2 vcores, 4GB of RAM and 80 GB storage.

If you have some hosting recommendation I'm all ears!

[mreid-tt]

It some point I would love to move hosting because currently I don't think I'm paying a fair price for this VPS at 15 EUR per months for 2 vcores, 4GB of RAM and 80 GB storage.

If you have some hosting recommendation I'm all ears!

Interesting. Are we also hosting content with Fastly? If so, this would involve not only moving the VPS service but also the CDN. Alternatively, will Fastly continue to provide CDN services if we choose to migrate our VPS to one of their designated cloud partners?

EDIT: After a quick search, I found the existing pricing to be surprisingly reasonable in comparison. Here are a few options worth exploring:

1. Hetzner Cloud
2. Contabo VPS
3. IONOS VPS
4. A2 Hosting (Unmanaged VPS)
5. Hostinger VPS Hosting

Additionally, this video highlights some other factors to consider.

[Diaoul]

Hetzner wanted to verify my ID and all plus it comes with restriction on egress traffic. Given we upload a lot of data, I don't want to be limited by that so I gave up.

I've taken a VPS Comfort at OVH. It's twice everything we have today but at a cheaper price. Now the challenge will be to move everything there 😅

[mreid-tt]

I've taken a VPS Comfort at OVH. It's twice everything we have today but at a cheaper price. Now the challenge will be to move everything there 😅

From the website, I noticed that the Comfort plan is currently offered as part of a Black Friday promotion. Are you certain it will renew at this discounted price after the initial period? Additionally, I saw this note on the webpage:

For any VPS solutions hosted in Sydney and Singapore: 3 TB/month for the Comfort range. Bandwidth is reduced to 10 Mbps once the monthly quota has been exceeded.

Regarding migration, I assume the process would involve cloning our current configuration, conducting some tests, and piloting with a few users using a new URL for the Package Sources configuration. Once everything is confirmed to be working as expected, we could proceed to update the DNS records to point to the new server. During this process, it would be a good opportunity to explore logging options and potentially move this PR forward.