Working with S3: Difference between revisions

From Public wiki of Kevin P. Inscoe
Jump to navigation Jump to search
mNo edit summary
 
(2 intermediate revisions by the same user not shown)
Line 6: Line 6:


http://aws.amazon.com/s3/developer-resources/
http://aws.amazon.com/s3/developer-resources/
==Amazon S3 Path Deprecation Plan==
See https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/


== Publish ==
== Publish ==
Line 55: Line 59:
Please read: http://stackoverflow.com/questions/20264928/s3-static-website-index-document
Please read: http://stackoverflow.com/questions/20264928/s3-static-website-index-document


IMPORTANT: This assumes the entirety of web site example.hmhco.com will be hosted by S3.
IMPORTANT: This assumes the entirety of web site example.com will be hosted by S3.


http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html
http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html
Line 61: Line 65:
Also look at  http://stackoverflow.com/questions/5043616/amazon-s3-static-web-sites-custom-domain-or-subdomain
Also look at  http://stackoverflow.com/questions/5043616/amazon-s3-static-web-sites-custom-domain-or-subdomain


1. Now create a S3 bucket with a name of example.hmhco.com. It must be exactly that. It must match your intended fully qualified domain named (FQDN).
1. Now create a S3 bucket with a name of example.com. It must be exactly that. It must match your intended fully qualified domain named (FQDN).


2. Register DNS it does not need to be managed by Route 53. In this case we will assume example.hmhco.com.
2. Register DNS it does not need to be managed by Route 53. In this case we will assume example.com.


3. IMPORTANT! Point DNS name example.hmhco.com to S3 endpoint as speficied in the bucket (look in Properties in the S3 Console) you created above in Step #1 as a DNS CNAME. <b>Do not use s3.amazonaws.com this will not fully work</b>.
3. IMPORTANT! Point DNS name example.com to S3 endpoint as specified in the bucket (look in Properties in the S3 Console) you created above in Step #1 as a DNS CNAME. <b>Do not use s3.amazonaws.com this will not fully work</b>.


4. Under the S3 Console for your newly created bucket under the section called "Static Website Hosting" check "Enable website hosting" and set an Index Document. Usually it's called index.html but it can be anything.
4. Under the S3 Console for your newly created bucket under the section called "Static Website Hosting" check "Enable website hosting" and set an Index Document. Usually it's called index.html but it can be anything.
Line 87: Line 91:
],
],
"Resource": [
"Resource": [
"arn:aws:s3:::example.hmhco.com/*"
"arn:aws:s3:::example.com/*"
]
]
}
}
Line 94: Line 98:
</pre>
</pre>


6. Create a Group for publishers (say example.hmhco.com). Attach a managed policy in that group so that members of this group can publish to the bucket. After creating the group go to Permissions->Managed Policies -> Inline Policies with the name like "example.hmhco.com-group-policy".
6. Create a Group for publishers (say example.com). Attach a managed policy in that group so that members of this group can publish to the bucket. After creating the group go to Permissions->Managed Policies -> Inline Policies with the name like "example.com-group-policy".


A sample policy for the IAM Group example.hmhco.com:
A sample policy for the IAM Group example.com:


<pre>
<pre>
Line 122: Line 126:
                 "s3:DeleteObject"
                 "s3:DeleteObject"
             ],
             ],
             "Resource": "arn:aws:s3:::example.hmhco.com/*"
             "Resource": "arn:aws:s3:::example.com/*"
         }
         }
     ]
     ]
Line 129: Line 133:


7. Now using your favorite tool upload content to the S3 bucket and dont forget to update your index document.
7. Now using your favorite tool upload content to the S3 bucket and dont forget to update your index document.
==S3 as a file-system==
https://github.com/danilop/yas3fs

Latest revision as of 21:21, 16 September 2020

S3 Documentation

http://aws.amazon.com/documentation/s3/

http://aws.amazon.com/s3/getting-started/

http://aws.amazon.com/s3/developer-resources/

Amazon S3 Path Deprecation Plan

See https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

Publish

To publish to S3 there are several ways:

You can do this from the AWS console. Login as your userid to the S3 console at https://console.aws.amazon.com/s3.

Follow this guide at http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html

Hostng images on S3 - http://www.labnol.org/internet/host-images-files-on-amazon-s3-storage/4923/

Third party tools for accessing your S3 bucket

Besides using the AWS console, AWS Command line tools (AWS CLI) or API's for accessing your S3 bucket there are other more GUI like tools for this.

Import/Export

There are also some import/export features available: http://aws.amazon.com/importexport/tools/

Command line tools

$ aws s3 ls s3://mybucket

Syncing

Notes about syncing data to S3 storage

Hosting your domain on S3

Please read: http://stackoverflow.com/questions/20264928/s3-static-website-index-document

IMPORTANT: This assumes the entirety of web site example.com will be hosted by S3.

http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html

Also look at http://stackoverflow.com/questions/5043616/amazon-s3-static-web-sites-custom-domain-or-subdomain

1. Now create a S3 bucket with a name of example.com. It must be exactly that. It must match your intended fully qualified domain named (FQDN).

2. Register DNS it does not need to be managed by Route 53. In this case we will assume example.com.

3. IMPORTANT! Point DNS name example.com to S3 endpoint as specified in the bucket (look in Properties in the S3 Console) you created above in Step #1 as a DNS CNAME. Do not use s3.amazonaws.com this will not fully work.

4. Under the S3 Console for your newly created bucket under the section called "Static Website Hosting" check "Enable website hosting" and set an Index Document. Usually it's called index.html but it can be anything.

5. Create a policy to allow anonymous users (the internet) to access the site. By default only the GetObject action in the policy will be used which means each object must be referenced by the index document or some other page referenced originally by the index document. In other words S3 does not support directory lstings or indexes like Apache web server does. For more on S3 Access Control have a look here: http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html

A sample anonymous policy:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AddPerm",
			"Effect": "Allow",
			"Principal": "*",
			"Action": [
				"s3:GetObject"
			],
			"Resource": [
				"arn:aws:s3:::example.com/*"
			]
		}
	]
}

6. Create a Group for publishers (say example.com). Attach a managed policy in that group so that members of this group can publish to the bucket. After creating the group go to Permissions->Managed Policies -> Inline Policies with the name like "example.com-group-policy".

A sample policy for the IAM Group example.com:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::example.com/*"
        }
    ]
}

7. Now using your favorite tool upload content to the S3 bucket and dont forget to update your index document.

S3 as a file-system

https://github.com/danilop/yas3fs