[S3] Object Storage MinIO compatibility

#1

Hello,

Since Scaleway doesn’t offer a proper SDK for its ObjectStorage solution (OSS), I started to use the MinIO Javascript SDK (with NodeJS) for all S3 operations, and I encountered some problems along the way:

  • copy OBJECT: MinIO SDK URL-escapes all fields in headers, including ‘/’ -> OSS won’t recognize this, and the request will fail. Since for this particular operation, a prefix (of URL type /my/folder/) is usually used, it should be able to handle URL encoded fields in headers.
  • post OBJECT: MinIO SDK offers a presignedPostPolicy() that you can use to perform Browser direct uploads. OSS is supposed to support this, but I can’t seem to make it work. I event replaced the MinIO solution with the very old ‘aws-s3-form’ package (like provided in Scaleway sample code for post OBJECT operation), and I just can’t make it work (with every params/signatures/urls looking good) ! This is crucial, as for me, it is THE feature os S3 solutions.
  • MinIO SDK is ‘Bucket Policy’ oriented, and does not manage ACL at all. This is a choice from the MinIO team, and even though the OSS chose to use ACL, it would be great if the API would provide, in a near future, an API to be Bucket Policy compatible.

Anyway, working with a MinIO server locally while developing had me wasting so much time to be OSS compatible, that I might just simply drop ObjectStorage with Scaleway to go with AWS or Google Cloud. Which is a shame, since I really enjoy working with Scaleway solutions far.

#2

Thanks for the detailed feedback! We will investigate our compatibility with the Minio JS SDK.

#3

Thank you for the quick response.

Yes, I indeed use this SDK: https://github.com/minio/minio-js

I haven’t tried personally the Javascript AWS-SDK directly, but I read some posts talking about how the AWS-SDK is not really friendly with other S3 services (endPoint configuration limited, default region based in ‘us-east-1’, bucket/region URL defined AWS style, etc…).
I would definitely integrate AWS-SDK (instead of MinIO) if you can assure me that you base services works with it (put/post/delete/copy OBJECT) !

Thank you anyway.

#4

I created an internel task so that we can investigate the compatibility with the Minio JS SDK. If you could provide us with code snippets in which you encountered these issues, we would appreciate it.

To be honest, I can’t guarantee that the AWS JS SDK will for for your uses cases, but for what it’s worth we never received any report of incompatibility, and it’s also what the Scaleway Console uses.

#5

Hello,

Just so you know, I successfully integrated AWS SDK S3 client in my project. It works perfectly with Scaleway Object Storage, at the very least for these basic operations:

  • GET object
  • PUT object
  • HEAD object
  • COPY object
  • DELETE object
  • GET object presigned URL
  • POST object presigned URL

Turns out you can easily initialize the client using specific endpoint / regions:

const AWS = require("aws-sdk");
const AWSCli = new AWS.S3({
  accessKeyId: [ACCESS KEY],
  secretAccessKey:[SECRET KEY],
  endpoint: 's3.fr-par.scw.cloud',
  region: 'fr-par',
  s3ForcePathStyle: true,
  signatureVersion: 'v4'
});

AWSCli.putObject(...)

Thank you anyway.

1 Like
#6

Thanks for the feedback :slight_smile: