S3
Getting started
S3 API is an Amazon S3 API based API for working with object storage resources. With S3 API you can:
- View information about the number and volume of containers and objects within an account;
- Create and delete containers;
- upload, view, copy, move, download, and delete objects in containers;
- manage segmented loading of objects;
- manage CORS, etc.
API access
Only authorized access to the S3 API is possible.
To access the container via the S3 API, the user must have a role with access to the object store, see the instructions for details Manage access in object storage.
The address (URL) when using Path-Style addressing (default) can be viewed in the URL list.
If Virtual Hosted addressing is enabled, you can address the container at <container_name>.<s3_domain>.
Authentication
Authentication in the S3 API is accomplished by signing requests. We support AWS Signature Version 4 and AWS Signature Version 2.
To form a signature, access keys will be required, which can be obtained by issuing an S3 key to the user:
- Access Key ID — field value Access key from the S3 key;
- Secret Access Key — field value Secret key from the S3 key.
You can sign the request using one of the methods:
- via HTTP header Authorization, more details in the instruction Authenticating Requests: Using the Authorization Header (AWS Signature Version 4) Amazon documentation;
- using query parameters or a Presigned URL. With this method you can specify all query parameters in the URL, see the instructions for more details Authenticating Requests: Using Query Parameters (AWS Signature Version 4) Amazon documentation.
The lifetime of a request signature is 15 minutes.
Compatibility
The S3 API is compatible with the following Amazon S3 API features:
AWS SDK
Python
Boto
boto3 — are development kits (SDKs) for Python 3.x programming languages. SDKs are designed to work with AWS services.
- Install boto, more details in the documentation boto3.
- In your home directory, create a configuration file
~/.aws/credentials:
[default]
aws_access_key_id = <access_key>
aws_secret_access_key = <secret_key>
Specify:
<access_key>— field value Access key from S3 key;<secret_key— field value Secret key from the S3 key.
- In your home directory, create a configuration file
~/.aws/config:
[default]
region=<pool>
Examples of operations
#!/usr/bin/env python
#-*- coding: utf-8 -*-
import boto3
# Authorization
s3 = boto3.client(
service_name='s3',
endpoint_url='<url>'
)
# Upload object from string
s3.put_object(Bucket="BucketName", Key="ObjectName1", Body="Test")
# Upload object from file
s3.upload_file("data.docx", "BucketName", "ObjectName2")
# Get list of objects in the container
for key in s3.list_objects(Bucket="BucketName")["Contents"]:
print(key["Key"])
# Download object
get_object_response = s3.get_object(Bucket="BucketName", Key="ObjectName2")
print(get_object_response["Body"].read())
# Delete multiple objects
objects_to_delete = [{"Key": "ObjectName1"}, {"Key": "ObjectName2"}]
s3.delete_objects(Bucket="BucketName", Delete={"Objects": objects_to_delete})
Example for boto3
Create an s3 client by providing an endpoint_url, a pool, and an S3 key (EC2 key), user-issued:
s3 = boto3.client("s3", endpoint_url="<url>", region_name="<pool>", aws_access_key_id="access_key", aws_secret_access_key="secret_key")
Call the method generate_presigned_post(), passing the name of the bucket to be loaded into, and the name of the key, which may contain a pattern ${filename} for the filename provided by the user at the time of upload. This call will return a dictionary with the URL to which the form should be sent and the dictionary fields with all necessary fields filled in for this form (X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Signature, Policy).
post = s3.generate_presigned_post(bucket, key)
If additional fields need to be added to the object (and form), such as a Content-Type header, pass a dictionary with these fields with the Fields argument. Also, according to policy specifications We need to describe each additional field in the array. conditions (argument Conditions in boto).
post = s3.generate_presigned_post(bucket, key,
Fields={"Content-Type": "image/webp"},
Conditions=[["eq", "$content-type", "image/webp"]])
From the received data, you can compose an HTML form or reproduce the request via requests:
requests.post(post["url"], data=post["fields"], files=[("file", ("filename", b"body_data"))])
When generating an HTML form, it is worth considering that the file field with the data of the uploaded file should be at the end of the form.
PHP
AWS SDK for PHP — is a development kit for working with AWS services.
The SDK is a modern open source PHP library that simplifies the integration of a PHP application with the S3 object store.
Follow these steps to connect the library using the Composer dependency management tool:
- Install Composer:
curl -sS 'https://getcomposer.org/installer' | php
- Run the Composer command to install the latest stable version of the SDK:
php composer.phar require aws/aws-sdk-php
- Include aws-sdk-php in your script. The values required for authorization are Access Key и Secret Key from S3 key.
Examples of operations
<?php
require "vendor/autoload.php";
use Aws\S3\S3Client;
// Create client
$s3Client = new S3Client([
"version" => "latest",
"region" => "<pool>",
"use_path_style_endpoint" => true,
"credentials" => [
"key" => "<access_key>",
"secret" => "<secret_key>",
],
"endpoint" => "<url>"
]);
// Upload object
$s3Client->putObject([
"Bucket" => "BucketName",
"Key" => "ObjectName",
"Body" => "Test"
]);
// Download object
$result = $s3Client->getObject([
"Bucket" => "BucketName",
"Key" => "ObjectName"
]);
echo $result["Body"];
Specify:
<access_key>— field value Access key from S3 key;<secret_key— field value Secret key from the S3 key.
JavaScript
AWS SDK for Node.js — is a development kit for running JavaScript with AWS services in a Node.js environment.
Example of work for Node.js
var S3 = require("aws-sdk/clients/s3");
var s3 = new S3({
credentials: {
accessKeyId: "<access_key>",
secretAccessKey: "<secret_key>"
},
endpoint: "<url>",
s3ForcePathStyle: true,
region: "<pool>",
apiVersion: "latest"
});
// Upload object
var params = {
Bucket: "BucketName",
Key: "ObjectName",
Body: "Test"
};
s3.upload(params, (err, data) => {
if (err) {
console.log(err, err.stack);
} else {
console.log(data);
}
/*
data = {
ETag: "0cbc6611f5540bd0809a388dc95a615b",
Location: "<url>/BucketName/ObjectName",
key: "ObjectName",
Key: "ObjectName",
Bucket: "BucketName"
}
*/
});
// Get object metadata
var params = {
Bucket: "BucketName",
Key: "ObjectName"
};
s3.headObject(params, (err, data) => {
if (err) {
console.log(err, err.stack);
} else {
console.log(data);
}
/*
data = {
AcceptRanges: "bytes",
LastModified: 2019-12-03T17:29:15.000Z,
ContentLength: 4,
ETag: "0cbc6611f5540bd0809a388dc95a615b",
ContentType: "application/octet-stream",
Metadata: {}
}
*/
});
// Get object
var params = {
Bucket: "BucketName",
Key: "ObjectName"
};
s3.getObject(params, (err, data) => {
if (err) {
console.log(err, err.stack);
} else {
console.log(data);
}
/*
data = {
AcceptRanges: "bytes",
LastModified: 2019-12-03T17:29:15.000Z,
ContentLength: 4,
ETag: "0cbc6611f5540bd0809a388dc95a615b",
ContentType: "application/octet-stream",
Metadata: {},
Body: <Buffer 54 65 73 74>
}
*/
});
// Delete object
var params = {
Bucket: "BucketName",
Key: "ObjectName"
};
s3.deleteObject(params, (err, data) => {
if (err) {
console.log(err, err.stack);
} else {
console.log(data);
}
/*
data = {
}
*/
});
Specify:
<access_key>— field value Access key from S3 key;<secret_key— field value Secret key from the S3 key.
Java
// Configure S3 client connection
AWSCredentials credentials = new BasicAWSCredentials(
"<access_key>",
"<secret_key>"
);
EndpointConfiguration endpoint =
new EndpointConfiguration("<url>", "<pool>");
AmazonS3 s3client = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withPathStyleAccessEnabled(true)
.withEndpointConfiguration(endpoint)
.build();
// Create bucket
String bucketName = "s3bucket";
String objectName = "s3object";
if(!s3client.doesBucketExistV2(bucketName)) {
s3client.createBucket(bucketName);
}
// Upload object
s3client.putObject(
bucketName,
objectName,
"sample-data"
);
// Download object
S3Object s3object = s3client.getObject(bucketName, objectName);
S3ObjectInputStream inputStream = s3object.getObjectContent();
inputStream.transferTo(new FileOutputStream("downloaded-object"));
// Delete object
s3client.deleteObject(bucketName, objectName);
// Delete bucket
s3client.deleteBucket(bucketName);
Specify:
<access_key>— field value Access key from S3 key;<secret_key— field value Secret key from the S3 key.