mirror of
https://github.com/multica-ai/multica.git
synced 2026-08-05 17:40:11 +02:00
eb14e73a20
* fix(storage): disable request checksum in buffered S3 Upload
* test(storage): make the checksum-trailer regression test actually fail without the fix
The test built its client with s3.New(s3.Options{...}), which leaves
RequestChecksumCalculation unset. An unset value emits no checksum at all,
so the assertions passed with or without the fix in Upload(). Production
builds the client via config.LoadDefaultConfig, which resolves the field to
WhenSupported.
Pin that production default in the test client and serve over TLS, since the
SDK only switches to a trailing checksum when the request is HTTPS. Removing
the option from Upload() now fails buffered_upload on the exact
STREAMING-UNSIGNED-PAYLOAD-TRAILER value Aliyun OSS rejects, while
streaming_upload still passes.
Also gofmt s3.go: the added option callback used spaces instead of tabs.
Co-authored-by: multica-agent <github@multica.ai>
* fix(storage): scope the checksum workaround to non-AWS endpoints
Downgrading RequestChecksumCalculation to WhenRequired is what makes Aliyun
OSS and Tencent COS accept the buffered upload, but it drops the client-side
checksum entirely rather than moving it out of the trailer: on TLS the request
goes out as UNSIGNED-PAYLOAD with no x-amz-checksum-* at all. Applying that to
every backend would change requests that work today, and AWS buckets with a
default Object Lock retention require a checksum to be present.
Gate the option on the endpoint instead. Real AWS S3 — no AWS_ENDPOINT_URL, or
an explicit amazonaws.com host — keeps whatever the SDK resolved, including any
operator-set AWS_REQUEST_CHECKSUM_CALCULATION. Only S3-compatible endpoints get
WhenRequired.
Adds TestS3StorageAWSUploadKeepsChecksumTrailer so the AWS path is pinned from
the other side: making the workaround unconditional now fails that test, while
removing it entirely still fails buffered_upload.
Also drops the incorrect claim that the seekable body stays covered by a real
SigV4 payload hash.
Co-authored-by: multica-agent <github@multica.ai>
* fix(storage): match the AWS endpoint on the host label boundary
usesAWSEndpoint substring-matched "amazonaws.com" anywhere in the configured
endpoint, which got the decision wrong in both directions: an uppercase
"https://S3.US-EAST-1.AMAZONAWS.COM" was treated as third-party and lost its
checksum, while "https://notamazonaws.com", "https://s3.amazonaws.com.evil.net"
and any URL merely carrying the string in its path or query were treated as AWS
and never got the fix they needed.
Parse the endpoint and compare the hostname on the label boundary instead,
covering the China partition. Scheme-less values are retried as https, since
url.Parse otherwise reads the whole value as a path.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: renjianjun <renjianjun@angelalign.com>
Co-authored-by: Bohan-J <bohan.optimism@gmail.com>
Co-authored-by: multica-agent <github@multica.ai>