Skip to content

Feature: conditional upload API #178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

alsenz
Copy link

@alsenz alsenz commented Apr 21, 2025

Resolves #129

Adds support for conditional object uploads.

Extends ObjectAttributes with optional ObjectVersion (if supported), and Upload(...) with three new ObjectUploadOption, depending on provider support.

  1. IfMatch: the object is written only if the existing object matches the provided version
  2. IfNotMatch: the object is written only if the existing object does not match the provided version
  3. IfNotExists: the object is written only if no object already exists for the provided key

Supports two forms of ObjectVersion:

  1. Generational versions (i.e. increasing integers)
  2. Etags

Not all providers support conditional write. The following providers are supported:

  1. Filesystem: uses extended filesystem attributes. Only supported if the host system supports extended attributes.
  2. InMem
  3. GCS
  4. Azure
  5. S3: IfNotMatch is not yet supported (as it is not supported by AWS).

Clients can check conditional API by calling SupportedObjectUploadOptions on the Bucket interface.

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

API changes

  1. Added 3 new ObjectUploadOptions: IfMatch, IfNotMatch and IfNotExists
    1.1 Made ObjectUploadOption typed, repeating the existing IterOption pattern.
    1.2 Added three new parameter modifiers- WithIfMatch, WithIfNotExists and WithIfNotMatch, and WithContentType to support the previous un-typed ObjectUploadOption
  2. Added optional ObjectVersion field to ObjectAttributes
  3. Added acceptance tests for IfMatch, IfNotMatch, IfNotExists and for Attributes API
  4. Added Bucket provider implementations for:
    1.1 filesystem: this now uses Extended Attributes, if supported by the host system
    1.2 inmem
    1.3 GCS
    1.4 Azure
    1.5 S3, excepting IfNoneMatch which is not fully supported (by AWS) yet
    1.6 Wrappers

Backwards compatibility

API changes are additive and should be backwards compatible.

  • The Upload API changes use a trailing variadic parameter so should be backwards compatible

  • The ObjectAttributes add new field so should be backwards compatible

  • The SupportedObjectUploadOptions interface method potentially breaks existing Bucket implementations. Library users who implement their own providers will need to implement this method. Full or stub implementations are provided for all provider implementations in the libary.

Dependencies

  • Adds dependency on github.com/pkg/xattr, which is BSD-2-clause licensed.

Verification

Added acceptance tests for

  • IfMatch
  • IfNotMatch
  • IfExists
  • Versions on Attributes

Did not run integration tests on other object store providers, as I do not have access to test environments, but these should be unchanged.

@alsenz alsenz changed the title Feature: conditions API Feature: conditional upload API Apr 21, 2025
@alsenz alsenz force-pushed the feat_conditions_API_129 branch from c21b930 to 1bb876b Compare April 21, 2025 13:57
@alsenz
Copy link
Author

alsenz commented May 1, 2025

@bwplotka - would you be the appropriate maintainer to submit this for a first review?

Greatly appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: conditions API
1 participant