Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Storage #81

Open
palitu opened this issue Jul 6, 2023 · 6 comments
Open

S3 Storage #81

palitu opened this issue Jul 6, 2023 · 6 comments

Comments

@palitu
Copy link

palitu commented Jul 6, 2023

Hi, i am trying to work out how the service exposes the gifs.

  • Is it done through the local webserver (ie, the server grabs the data from S3, then forwards it),
  • is it done by the browser, where the server sends a reference to the S3 object, and the client browser gets the asset directly from the object store.

I am going to use B2 (backblaze) and there is a question whether the new bucket should be public or private. I will start with private, as i do not think that i will allow others to access it, but was curious all the same.

thanks.

@pietvanzoen
Copy link
Owner

Media hosting is managed by your S3 provider. In AWS that would be Cloudfront with a bucket behind it.

The Gifable app is only responsible for managing the library, uploading images to the bucket, then knowing enough about the S3 setup to generate a URL that gets saved to the DB.

The key to that is this config:

S3_STORAGE_BASE_URL= # e.g. "https://my.bucket.me" - the url that maps to your bucket

I've not used B2 backblaze before. I'd love to hear what the setup is like when you get it working. Thinking I should setup some docs for config on various providers. I'm using Cloudflare R2 with app.gifable.club

@jason-weiser
Copy link

@palitu if you are running this using Backblaze, would you be open to sharing how you configured it using the .env? I can’t seem to get it working at all and wonder if there are little idiosyncrasies I’m missing when it comes to all the different things I need to input.

@pietvanzoen
Copy link
Owner

pietvanzoen commented Dec 17, 2023

Hey @jason-weiser. I managed to get backblaze working for me with this config:

S3_ENDPOINT=s3.us-east-005.backblazeb2.com
S3_BUCKET=gifable-club
S3_USE_SSL=true # required I think
S3_ACCESS_KEY=<key_id>
S3_SECRET_KEY=<secret>
S3_STORAGE_BASE_URL=https://f005.backblazeb2.com/file/gifable-club/

I found the endpoint on the buckets page:
image

The base url I found by just browsing files and looking at the "Friendly URL" in the file details modal.

Let me know if this helps. If not can you share what errors or behaviour you're seeing?

@Phibonacci
Copy link

Phibonacci commented Feb 20, 2024

I am running Gifable with docker and I installed MinIO as an s3 on the same server. I can connect to MinIO successfully using the mc tool in command line but Gifable is giving me the "Invalid" error anytime I try to upload a gif.

I do not know how to investigate this further.

S3 variables of my .env look this way:

S3_ENDPOINT=s3.mydomain.net # as shown in `mc admin --json info mys3` in info>servers>endpoint
S3_BUCKET=gif # As created in MinIO web UI
S3_STORAGE_BASE_URL=https://s3.mydomain.net/gif/ # Typing that in my browser shows an XML to the right bucket 
S3_ACCESS_KEY=access_key
S3_SECRET_KEY=secret_key # generated through the MinIO web UI
S3_REGION=
S3_BASE_PATH=
S3_PORT=443
S3_USE_SSL=true # I configured my nginx proxy properly and I can connect to https://s3.mydomain.net with the mc tool

In my browser https://s3.mydomain.net/gif/ shows the following (since I'm not sending the keys):

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied.</Message>
<BucketName>gif</BucketName>
<Resource>/gif/</Resource>
<RequestId>17A6DBF9817C650F</RequestId>
<HostId>
da9024bbb4ad464b049177c95ea2ebf374d3b3fd1af9251148b658df7ca2e3e8
</HostId>
</Error>

The only logs I get when refreshing the page are:

prisma:query SELECT 1
prisma:query SELECT `main`.`User`.`id`, `main`.`User`.`username`, `main`.`User`.`isAdmin`, `main`.`User`.`theme` FROM `main`.`User` WHERE (`main`.`User`.`id` = ? AND 1=1) LIMIT ? OFFSET ?
GET /media/new?uploadType=file 200 - - 5.080 ms

Clicking "upload" does not generate new logs.

Edit: I got it! It does not like spaces in file names! Now I can figure out the next error as the upload has failed.

I realise it's not even s3 related, yet.

Error: User not found
    at action5 (/app/build/index.js:2403:11)
    at Object.callRouteActionRR (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/data.js:35:16)
    at callLoaderOrAction (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:3550:16)
    at submit (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2817:16)
    at queryImpl (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2752:22)
    at Object.queryRoute (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2702:18)
    at handleDataRequestRR (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/server.js:63:20)
    at requestHandler (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/server.js:37:18)
    at /app/node_modules/@remix-run/express/dist/server.js:39:22
POST /media/new?uploadType=file&_data=routes%2Fmedia.new 500 - - 97.844 ms

Alright, so you cannot upload as an admin. I created an user. Also you cannot set DISABLE_SIGNUP to 0 you need to comment it. Now I have an s3 error.

InvalidEndpointError: Invalid endPoint : s3.mydomain.net # e.g. "s3.amazonaws.com"
    at new TypedClient (/app/node_modules/minio/dist/main/internal/client.ts:112:13)
    at new Client (/app/node_modules/minio/dist/main/minio.js:76:8)
    at new S3Storage (/app/build/index.js:779:24)
    at storage2 (/app/build/index.js:775:10)
    at storeBuffer (/app/build/index.js:999:13)
    at action5 (/app/build/index.js:2406:23)
    at Object.callRouteActionRR (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/data.js:35:16)
    at callLoaderOrAction (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:3550:16)
    at submit (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2817:16)
    at queryImpl (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2752:22)

I tested with play.min.io to see if I would get the same error and I did, "Invalid endPoint".

Wait... is that the file comment I see in the log? Removing all trailing comments... ok, now it's working, I can upload a gif without an error message.

Now I need to figure out why the actual file fails to download, only showing a rectangle with the name of the file.

The file is not visible because the bucket is not public. But then wouldn't it expose the bucket to upload from anyone directly interacting with it?

Yes it would, so I kept it private and added a custom rule to make downloads public: mc anonymous set download mys3/gif

It's working, problem solved.

@pietvanzoen
Copy link
Owner

Hey @Phibonacci . Sorry it took me a while to get to this. Sounds like you battled through! Glad it's working now.

If you have any suggestions for docs or improved error messaging please let me know!

@Phibonacci
Copy link

It's perfectly fine. I kept editing my message so you could use my struggles as a user feedback.

It would help to have more explicit messages on the web interface when a filename is invalid. Maybe allowing space in filenames is doable?

I would suggest not using trailing comments in the .env file example as they are not supported. Moving them on the previous line should be fine.

I found the MinIO documentation hard to follow. Maybe this ticket will help others. I also had to fight with my nginx reverse proxy, although MinIO has a good documentation on the topic.

On a completely different topic I was surprised Gifable does not support MP4 as it is the preferred format of most "gif" creators nowadays. I converted mines using this great article. Maybe a future feature allowing to upload multiple formats of the same gif to then be able to pick one depending on the targeted platform?

Great work nonetheless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants