Skip to content
Permalink

Comparing changes

This is a direct comparison between two commits made in this repository or its related repositories. View the default comparison for this range or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: gocardless/bucket-store
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: c943758009645ecc7b43a1e922afb63aa8b85789
Choose a base ref
..
head repository: gocardless/bucket-store
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 8237dc22853b44c87a55e0d5d3fda5c046d76149
Choose a head ref
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -13,13 +13,13 @@ references:
cache_bundle: &cache_bundle
save_cache:
key: bundle-<< parameters.ruby_version >>-{{ checksum "file_storage.gemspec" }}-{{ checksum "Gemfile" }}
key: bundle-<< parameters.ruby_version >>-{{ checksum "bucket_store.gemspec" }}-{{ checksum "Gemfile" }}
paths:
- vendor/bundle

restore_bundle: &restore_bundle
restore_cache:
key: bundle-<< parameters.ruby_version >>-{{ checksum "file_storage.gemspec" }}-{{ checksum "Gemfile" }}
key: bundle-<< parameters.ruby_version >>-{{ checksum "bucket_store.gemspec" }}-{{ checksum "Gemfile" }}

jobs:
code_quality:
11 changes: 10 additions & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
@@ -30,7 +30,16 @@ RSpec/LetSetup:
RSpec/MultipleExpectations:
Max: 10

RSpec/Rails/AvoidSetupHook:
Enabled: true

# Disabled as it's breaking dependabot
Gemspec/RequiredRubyVersion:
Exclude:
- 'file_storage.gemspec'
- 'bucket_store.gemspec'

# Disabled as it's breaking rubocop, see:
# https://github.com/rubocop/rubocop/issues/10208
# https://github.com/rubocop/rubocop/issues/10210
Layout/DotPosition:
Enabled: false
2 changes: 1 addition & 1 deletion .ruby-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.0.1
3.0.2
12 changes: 10 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
v0.3.0
v0.5.0
------

- Add `presigned_url` method.

v0.4.0
------
- Add an `.exists?` method that returns `true`/`false` depending on whether a given
key exists or not.

v0.3.0
------
- Add support for S3

v0.2.0
------

2 changes: 1 addition & 1 deletion Gemfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# frozen_string_literal: true

source 'https://rubygems.org'
source "https://rubygems.org"

gemspec
63 changes: 33 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
# FileStorage
# BucketStore

An abstraction layer on the top of file cloud storage systems such as Google Cloud
Storage or S3. This module exposes a generic interface that allows interoperability
between different storage options. Callers don't need to worry about the specifics
of where and how a file is stored and retrieved as long as the given key is valid.

Keys within the `FileStorage` are URI strings that can universally locate an object
Keys within the `BucketStorage` are URI strings that can universally locate an object
in the given provider. A valid key example would be
`gs://a-gcs-bucket/file/path.json`.

## Usage
This library is distributed as a Ruby gem, and we recommend adding it to your Gemfile:

```ruby
gem "file-storage"
gem "bucket_store"
```

Some attributes can be configured via `FileStorage.configure`. If using Rails, you want to
add a new initializer for `FileStorage`. Example:
Some attributes can be configured via `BucketStore.configure`. If using Rails, you want to
add a new initializer for `BucketStore`. Example:

```ruby
FileStorage.configure do |config|
BucketStore.configure do |config|
config.logger = Logger.new($stderr)
end
```
@@ -29,13 +29,13 @@ If using RSpec, you'll probably want to add this line to RSpec's config block (s
the *Adapters* section for more details):

```ruby
config.before { FileStorage::InMemory.reset! }
config.before { BucketStore::InMemory.reset! }
```

For our policy on compatibility with Ruby versions, see [COMPATIBILITY.md](docs/COMPATIBILITY.md).

## Design and Architecture
The main principle behind `FileStorage` is that each resource or group of resources must
The main principle behind `BucketStore` is that each resource or group of resources must
be unequivocally identifiable by a URI. The URI is always composed of three parts:

- the "adapter" used to fetch the resource (see "adapters" below)
@@ -48,47 +48,50 @@ As an example, all the following are valid URIs:
- `inmemory://bucket/separator/file.xml`
- `disk://hello/path/to/file.json`

Even though `FileStorage`'s main goal is to be an abstraction layer on top of systems such
Even though `BucketStore`'s main goal is to be an abstraction layer on top of systems such
as S3 or Google Cloud Storage where the "path" to a resource is in practice a unique
identifier as a whole (i.e. the `/` is not a directory separator but rather part of the
key's name), we assume that clients will actually want some sort of hierarchical
separation of resources and assume that such separation is achieved by defining each
part of the hierarchy via `/`.

This means that the following are also valid URIs in `FileStorage` but they refer to
This means that the following are also valid URIs in `BucketStore` but they refer to
all the resources under that specific hierarchy:

- `gs://gcs-bucket/path/subpath/`
- `inmemory://bucket/separator/`
- `disk://hello/path`

## Configuration
`FileStorage` exposes some configurable attributes via `FileStorage.configure`. If
`BucketStore` exposes some configurable attributes via `BucketStore.configure`. If
necessary this should be called at startup time before any other method is invoked.

- `logger`: custom logger class. By default, logs will be sent to stdout.

## Adapters

`FileStorage` comes with 3 built-in adapters:
`BucketStore` comes with 4 built-in adapters:

- `gs`: the Google Cloud Storage adapter
- `s3`: the S3 adapter
- `disk`: a disk-based adapter
- `inmemory`: an in-memory store

### GS adapter
This is the Google Cloud Storage adapter and what you'll most likely want to use in
production. `FileStorage` assumes that the authorisation for accessing the resources
has been set up outside of the gem.
This is the adapter for Google Cloud Storage. `BucketStore` assumes that the authorisation
for accessing the resources has been set up outside of the gem.

### S3 adapter
This is the adapter for S3. `BucketStore` assumes that the authorisation for accessing
the resources has been set up outside of the gem (see also
https://docs.aws.amazon.com/sdk-for-ruby/v3/api/index.html#Configuration).

### Disk adapter
A disk-backed key-value store. This adapter will create a temporary directory where
all the files will be written into/read from. The base directory can be explicitly
defined by setting the `DISK_ADAPTER_BASE_DIR` environment variable, otherwise a temporary
directory will be created.


### In-memory adapter
An in-memory key-value storage. This works just like the disk adapter, except that
the content of all the files is stored in memory, which is particularly useful for
@@ -100,65 +103,65 @@ content though via a `.reset!` method. In RSpec this would translate to adding t
in the `spec_helper`:

```ruby
config.before { FileStorage::InMemory.reset! }
config.before { BucketStore::InMemory.reset! }
```

## FileStorage vs ActiveStorage
## BucketStore vs ActiveStorage

ActiveStorage is a common framework to access cloud storage systems that is included in
the ActiveSupport library. In general, ActiveStorage provides a lot more than FileStorage
the ActiveSupport library. In general, ActiveStorage provides a lot more than BucketStore
does (including many more adapters) however the two libraries have different use cases
in mind:

- ActiveStorage requires you to define every possible bucket you're planning to use
ahead of time in a YAML file. This works well for most cases, however if you plan to
use a lot of buckets this soon becomes impractical. We think that FileStorage approach
use a lot of buckets this soon becomes impractical. We think that BucketStore approach
works much better in this case.
- FileStorage does not provide ways to manipulate the content whereas ActiveStorage does.
- BucketStore does not provide ways to manipulate the content whereas ActiveStorage does.
If you plan to apply transformations to the content before uploading or after
downloading them, then probably ActiveStorage is the library for you. With that said,
it's still possible to do these transformations outside of FileStorage and in fact we've
it's still possible to do these transformations outside of BucketStore and in fact we've
found the explicitness of this approach a desirable property.
- FileStorage approach makes any resource on a cloud storage system uniquely identifiable
- BucketStore approach makes any resource on a cloud storage system uniquely identifiable
via a single URI, which means normally it's enough to pass that string around different
systems to be able to access the resource without ambiguity. As the URI also includes
the adapter, it's possible for example to download a `disk://dir/input_file` and
upload it to a `gs://bucket/output_file` all going through a single interface.
ActiveStorage is instead focused on persisting an equivalent reference on a Rails model.
If your application does not use Rails, or does not need to persist the reference or
just requires more flexibility in general, then FileStorage is probably the library for
just requires more flexibility in general, then BucketStore is probably the library for
you.


## Examples

### Uploading a file to a bucket
```ruby
FileStorage.for("inmemory://bucket/path/file.xml").upload!("hello world")
BucketStore.for("inmemory://bucket/path/file.xml").upload!("hello world")
=> "inmemory://bucket/path/file.xml"
```

### Accessing a file in a bucket
```ruby
FileStorage.for("inmemory://bucket/path/file.xml").download
BucketStore.for("inmemory://bucket/path/file.xml").download
=> {:bucket=>"bucket", :key=>"path/file.xml", :content=>"hello world"}
```

### Listing all keys under a prefix
```ruby
FileStorage.for("inmemory://bucket/path/").list
BucketStore.for("inmemory://bucket/path/").list
=> ["inmemory://bucket/path/file.xml"]
```

### Delete a file
```ruby
FileStorage.for("inmemory://bucket/path/file.xml").delete!
BucketStore.for("inmemory://bucket/path/file.xml").delete!
=> true
```

## License & Contributing

* FileStorage is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
* Bug reports and pull requests are welcome on GitHub at https://github.com/gocardless/file-storage.
* BucketStore is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
* Bug reports and pull requests are welcome on GitHub at https://github.com/gocardless/bucket-store.

GoCardless ♥ open source. If you do too, come [join us](https://gocardless.com/about/careers/).
21 changes: 11 additions & 10 deletions file_storage.gemspec → bucket_store.gemspec
Original file line number Diff line number Diff line change
@@ -1,30 +1,31 @@
# frozen_string_literal: true

require File.expand_path("lib/file_storage/version", __dir__)
require File.expand_path("lib/bucket_store/version", __dir__)

Gem::Specification.new do |s|
s.name = "file_storage"
s.version = FileStorage::VERSION.dup
s.name = "bucket_store"
s.version = BucketStore::VERSION.dup
s.authors = ["GoCardless Engineering"]
s.email = ["engineering@gocardless.com"]
s.summary = "A helper library to access cloud storage services"
s.description = <<-DESCRIPTION
A helper library to access cloud storage services such as Google Cloud Storage.
A helper library to access cloud storage services such as Google Cloud Storage or S3.
DESCRIPTION
s.homepage = "https://github.com/gocardless/file-storage"
s.homepage = "https://github.com/gocardless/bucket-store"
s.license = "MIT"

s.files = Dir["lib/**/*", "README.md"]

s.required_ruby_version = ">= 2.6"

s.add_dependency "google-cloud-storage", "~> 1.31"
s.add_dependency "aws-sdk-s3", ">= 1.104"
s.add_dependency "google-cloud-storage", ">= 1.34"

s.add_development_dependency "gc_ruboconfig", "~> 2.25"
s.add_development_dependency "gc_ruboconfig", "~> 2.29"
s.add_development_dependency "pry-byebug", "~> 3.9"
s.add_development_dependency "rspec", "~> 3.10"
s.add_development_dependency "rspec_junit_formatter", "~> 0.4.1"
s.add_development_dependency "rubocop", "~> 1.12"
s.add_development_dependency "rubocop-performance", "~> 1.10"
s.add_development_dependency "rubocop-rspec", "~> 2.2"
s.add_development_dependency "rubocop", "~> 1.22"
s.add_development_dependency "rubocop-performance", "~> 1.11"
s.add_development_dependency "rubocop-rspec", "~> 2.5"
end
10 changes: 5 additions & 5 deletions docs/COMPATIBILITY.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Compatibility

Our goal as FileStorage maintainers is for the library to be compatible with all supported
Our goal as BucketStore maintainers is for the library to be compatible with all supported
versions of Ruby.

Specifically, any CRuby/MRI version that has not received an End of Life notice
@@ -9,18 +9,18 @@ is supported.

To that end, [our build matrix](../.circleci/config.yml) includes all these versions.

Any time FileStorage doesn't work on a supported version of Ruby, it's a bug, and can be
Any time BucketStore doesn't work on a supported version of Ruby, it's a bug, and can be
reported [here](https://github.com/gocardless/file-storage/issues).

# Deprecation

Whenever a version of Ruby falls out of support, we will mirror that change in FileStorage
Whenever a version of Ruby falls out of support, we will mirror that change in BucketStore
by updating the build matrix and releasing a new major version.

At that point, we will close any issues that only affect the unsupported version, and may
choose to remove any workarounds from the code that are only necessary for the unsupported
version.

We will then bump the major version of FileStorage, to indicate the break in compatibility.
Even if the new version of FileStorage happens to work on the unsupported version of Ruby, we
We will then bump the major version of BucketStore, to indicate the break in compatibility.
Even if the new version of BucketStore happens to work on the unsupported version of Ruby, we
consider compatibility to be broken at this point.
28 changes: 14 additions & 14 deletions lib/file_storage.rb → lib/bucket_store.rb
Original file line number Diff line number Diff line change
@@ -1,33 +1,33 @@
# frozen_string_literal: true

require "file_storage/version"
require "file_storage/configuration"
require "file_storage/key_context"
require "file_storage/key_storage"
require "bucket_store/version"
require "bucket_store/configuration"
require "bucket_store/key_context"
require "bucket_store/key_storage"

# An abstraction layer on the top of file cloud storage systems such as Google Cloud
# Storage or S3. This module exposes a generic interface that allows interoperability
# between different storage options. Callers don't need to worry about the specifics
# of where and how a file is stored and retrieved as long as the given key is valid.
#
# Keys within the {FileStorage} are URI strings that can universally locate an object
# Keys within the {BucketStore} are URI strings that can universally locate an object
# in the given provider. A valid key example would be:
# `gs://gcs-bucket/file/path.json`.
module FileStorage
module BucketStore
class << self
attr_writer :configuration

def configuration
@configuration ||= FileStorage::Configuration.new
@configuration ||= BucketStore::Configuration.new
end

# Yields a {FileStorage::Configuration} object that allows callers to configure
# FileStorage's behaviour.
# Yields a {BucketStore::Configuration} object that allows callers to configure
# BucketStore's behaviour.
#
# @yield [FileStorage::Configuration]
# @yield [BucketStore::Configuration]
#
# @example Configure FileStorage to use a different logger than the default
# FileStorage.configure do |config|
# @example Configure BucketStore to use a different logger than the default
# BucketStore.configure do |config|
# config.logger = Logger.new($stderr)
# end
def configure
@@ -46,8 +46,8 @@ def logger
#
# @param [String] key The reference key
# @return [KeyStorage] An interface to the adapter that can handle requests on the given key
# @example Configure {FileStorage} for Google Cloud Storage
# FileStorage.for("gs://the_bucket/a/valid/key")
# @example Configure {BucketStore} for Google Cloud Storage
# BucketStore.for("gs://the_bucket/a/valid/key")
def for(key)
ctx = KeyContext.parse(key)

Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# frozen_string_literal: true

module FileStorage
module BucketStore
class Configuration
def logger
@logger ||= Logger.new($stdout)
end

# Specifies a custom logger.
#
# Note that {FileStorage} uses structured logging, any custom logger passed must also
# Note that {BucketStore} uses structured logging, any custom logger passed must also
# support it.
#
# @example Use stderr as main output device
Loading