Skip to content

ChecksumCRC32 get_object field is not present after uploading via a presigned URL with x-amz-checksum-crc32 #20343

@MrCreosote

Description

@MrCreosote

When uploading an object via a presigned URL with the x-amz-checksum-crc32 field set, based on the documentation I assumed the ChecksumCRC32 field would be available in a get_object call, but it doesn't appear to be there.

Reference https://minio.slack.com/archives/C3NDUB8UA/p1724866846844969
cc @marktheunissen

Expected Behavior

When uploading an object via a presigned URL with the x-amz-checksum-crc32 field set, the ChecksumCRC32 should be present in the get_object response for that object.

Current Behavior

The field does not appear to be there.

Possible Solution

Possibly I'm doing something wrong, although the logs show that Minio is receiving the CRC as it's sending it back in response to the POST.

If I'm not doing something wrong and I'm interpreting the get_object documentation correctly, then the ChecksumCRC32 presumably needs to be added to the get_object response.

Steps to Reproduce (for bugs)

  • Start Minio
MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password ~/minio/RELEASE/2024-08-17T01-24-54Z/minio server ~/minio/data
  • Use the method of your choice to create a bucket called test
  • Execute the following Python code
$ ipython
Python 3.11.3 (main, Apr  5 2023, 14:14:40) [GCC 7.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.26.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from aiobotocore.session import get_session

In [2]: import pprint

In [3]: import requests

In [4]: sess = get_session()

In [5]: async with sess.create_client(
   ...:     "s3",
   ...:     endpoint_url="http://localhost:9000",
   ...:     aws_access_key_id="admin",
   ...:     aws_secret_access_key="password"
   ...: ) as client:
   ...:     url = await client.generate_presigned_post(
   ...:         Bucket="test", Key="somefile")
   ...: 

In [6]: # modifying the crc causes the upload to fail

In [7]: url["fields"]["x-amz-checksum-crc32"] = "T/xSCA=="

In [8]: with open("test/testdata/random_bytes_10kB", "rb") as f:
   ...:     files = {"file": ("somefile", f)}
   ...:     res = requests.post(url["url"], data=url["fields"], files=files)
   ...: 

In [9]: res.status_code
Out[9]: 204

In [18]: async with sess.create_client(
    ...:     "s3",
    ...:     endpoint_url="http://localhost:9000",
    ...:     aws_access_key_id="admin",
    ...:     aws_secret_access_key="password"
    ...: ) as client:
    ...:     res = await client.get_object(
    ...:         Bucket="test",
    ...:         Key="somefile",
    ...:         ChecksumMode="ENABLED")
    ...:     pprint.pp(res)
    ...: 
{'ResponseMetadata': {'RequestId': '17F00E8EA6676D5C',
                      'HostId': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8',
                      'HTTPStatusCode': 200,
                      'HTTPHeaders': {'accept-ranges': 'bytes',
                                      'content-length': '10240',
                                      'content-type': 'application/octet-stream',
                                      'etag': '"3291fbb392f6fad06dbf331dfb74da81"',
                                      'last-modified': 'Thu, 29 Aug 2024 '
                                                       '00:57:24 GMT',
                                      'server': 'MinIO',
                                      'strict-transport-security': 'max-age=31536000; '
                                                                   'includeSubDomains',
                                      'vary': 'Accept-Encoding',
                                      'x-amz-id-2': 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8',
                                      'x-amz-request-id': '17F00E8EA6676D5C',
                                      'x-content-type-options': 'nosniff',
                                      'x-ratelimit-limit': '3865',
                                      'x-ratelimit-remaining': '3865',
                                      'x-xss-protection': '1; mode=block',
                                      'date': 'Thu, 29 Aug 2024 01:24:23 GMT'},
                      'RetryAttempts': 0},
 'AcceptRanges': 'bytes',
 'LastModified': datetime.datetime(2024, 8, 29, 0, 57, 24, tzinfo=tzutc()),
 'ContentLength': 10240,
 'ETag': '"3291fbb392f6fad06dbf331dfb74da81"',
 'ContentType': 'application/octet-stream',
 'Metadata': {},
 'Body': <StreamingBody at 0x7f26576f1fc0 for ClientResponse at 0x7f265c6b6510>}

trace:

$~/minio/mc/RELEASE/2024-08-17T11-33-50Z/mc admin trace -v myminio/
localhost:9000 [REQUEST s3.PostPolicyBucket] [2024-08-28T17:57:24.823] [Client IP: 127.0.0.1]
localhost:9000 POST /test
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 Content-Type: multipart/form-data; boundary=9667e218262b3f77e245aa5079b79d47
localhost:9000 User-Agent: python-requests/2.32.3
localhost:9000 Accept: */*
localhost:9000 Accept-Encoding: gzip, deflate
localhost:9000 Connection: keep-alive
localhost:9000 Content-Length: 11018
localhost:9000 <BLOB>
localhost:9000 [RESPONSE] [2024-08-28T17:57:24.862] [ Duration 39.01ms TTFB 0s ↑ 11 KiB  ↓ 0 B ]
localhost:9000 204 No Content
localhost:9000 Content-Type: text/plain; charset=utf-8
localhost:9000 ETag: "3291fbb392f6fad06dbf331dfb74da81"
localhost:9000 Location: http://localhost:9000/test/somefile
localhost:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
localhost:9000 X-Amz-Checksum-Crc32: T/xSCA==
localhost:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
localhost:9000 X-Ratelimit-Limit: 3865
localhost:9000 X-Amz-Request-Id: 17F00D15DC40BADF
localhost:9000 X-Content-Type-Options: nosniff
localhost:9000 X-Xss-Protection: 1; mode=block
localhost:9000 Accept-Ranges: bytes
localhost:9000 Vary: Origin,Accept-Encoding
localhost:9000 Content-Length: 0
localhost:9000 Server: MinIO
localhost:9000 X-Ratelimit-Remaining: 3865
localhost:9000 <BLOB>
localhost:9000 
localhost:9000 [REQUEST s3.GetObject] [2024-08-28T18:24:23.122] [Client IP: 127.0.0.1]
localhost:9000 GET /test/somefile
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 Authorization: AWS4-HMAC-SHA256 Credential=admin/20240829/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date, Signature=b10d8899750fbc65d02d8a0dd7b1d9f1854dca05d73be6aa26fb2b720ae19300
localhost:9000 User-Agent: aiobotocore/2.13.2 md/Botocore#1.34.131 ua/2.0 os/linux#4.15.0-213-generic md/arch#x86_64 lang/python#3.11.3 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.131
localhost:9000 X-Amz-Checksum-Mode: ENABLED
localhost:9000 X-Amz-Date: 20240829T012423Z
localhost:9000 Accept: */*
localhost:9000 Accept-Encoding: identity
localhost:9000 Amz-Sdk-Invocation-Id: e4ee95fc-291d-43a7-9a97-e01e4986fc47
localhost:9000 Amz-Sdk-Request: attempt=1
localhost:9000 Content-Length: 0
localhost:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
localhost:9000 <BLOB>
localhost:9000 [RESPONSE] [2024-08-28T18:24:23.123] [ Duration 548µs TTFB 491.735µs ↑ 158 B  ↓ 10 KiB ]
localhost:9000 200 OK
localhost:9000 X-Amz-Request-Id: 17F00E8EA6676D5C
localhost:9000 X-Content-Type-Options: nosniff
localhost:9000 X-Ratelimit-Limit: 3865
localhost:9000 Last-Modified: Thu, 29 Aug 2024 00:57:24 GMT
localhost:9000 ETag: "3291fbb392f6fad06dbf331dfb74da81"
localhost:9000 Server: MinIO
localhost:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
localhost:9000 X-Xss-Protection: 1; mode=block
localhost:9000 Content-Type: application/octet-stream
localhost:9000 Content-Length: 10240
localhost:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
localhost:9000 Accept-Ranges: bytes
localhost:9000 X-Ratelimit-Remaining: 3865
localhost:9000 Vary: Origin,Accept-Encoding
localhost:9000 <BLOB>
localhost:9000 

It seems to be returned with the original POST but not with the get_object request.

Context

I'm trying to download objects from Minio and ensure that data integrity is maintained via a CRC check.

Regression

I don't know; I have no reason to think so.

Your Environment

  • Version used (minio --version):
$ RELEASE/2024-08-17T01-24-54Z/minio --version
minio version RELEASE.2024-08-17T01-24-54Z (commit-id=72cff79c8a7cc59bccb591995e3c3ed6aa2f4cd5)
Runtime: go1.22.6 linux/amd64
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Copyright: 2015-2024 MinIO, Inc.
  • Server setup and configuration:
    Just my laptop
  • Operating System and version (uname -a):
    Ubuntu 18.04 (yeah, I need to upgrade)
$ uname -a
Linux andbusinessisgood 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions