Kevin's AWS CloudFormation Notes: Difference between revisions

From Public wiki of Kevin P. Inscoe
Jump to navigation Jump to search
(Created page with "== CloudFormation Mantra == *Start thinking of deployment as code. Think first like a developer but also as an admin for cost containment. *Do E.V.E.R.Y.T.H.I.N.G. as CloudFo...")
 
 
Line 155: Line 155:


<pre>
<pre>
aws cloudformation describe-stack-resource --stack-name wordpress \
$ aws cloudformation describe-stack-resource --stack-name wordpress \
  --logical-resource-id DBInstance \
  --logical-resource-id DBInstance \
  | jq '.StackResourceDetail .PhysicalResourceId' \
  | jq '.StackResourceDetail .PhysicalResourceId' \

Latest revision as of 19:29, 29 March 2017

CloudFormation Mantra

  • Start thinking of deployment as code. Think first like a developer but also as an admin for cost containment.
  • Do E.V.E.R.Y.T.H.I.N.G. as CloudFormation (AWS resource related). it s a new way of thinking. Again think code more than manual steps. To paraphrase Spider Man: "With great Stacks come great responsibility". When Stacks dies everything (attached to it) dies but that's ok if we are being thoughtful.
  • Stacks can be updated to reflect changed resources. http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-stack.html. It seems more code easy to use CLI (rake update_stacks command) to do this then Console.
  • Save and track templates in Git. Check your revisions and changes.
  • Versioning of Stacks? I don't think it's needed if you think stack updates.
  • Use Stack Policies to protect against mistakes and errant deployments particularly in production.
  • AWS Best Practise: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html

CloudFormation Practise

  • Use parameterization heavily - See Parameterization
  • Use rake to update templates into AWS. Why? See Rake

JSON

Need to understand JSON first: http://json.org/

JSON validation tools:

http://jsonlint.com -

Command line (or offline) version of jsonlint - https://www.npmjs.com/package/jsonlint -OR- just call the web site from the command line with: https://www.npmjs.com/package/jheri-curl (see also http://www.commandlinefu.com/commands/view/5074/validate-json) See also Installing jheri-curl on a Mac.

Some other useful JSON tools such as formatters, pretty-print, etc..:

From command line (if you have Python 2.6+ installed): $ python -m json.tool < AWS_CFN_KPI_POC_01.template

https://github.com/exhuma/braindump/tree/master/jsonformat

http://www.restlessprogrammer.com/2013/03/how-to-pretty-print-json-from-command.html

http://stackoverflow.com/questions/352098/how-can-i-pretty-print-json

https://github.com/atomicbird/jsonlint (for Mac only?)

JSON Editors:

Vim has plug-ins: I have not tried these yet.

On Mac I currently use Cocoa JSON Editor: http://www.cocoajsoneditor.com/ (Mac only).

Online editors a plenty!

http://jsonviewer.stack.hu/

There are cross-platform editors out there.

Sublime was heavily liked ($70) - http://www.sublimetext.com/

More cross platform tools at https://pods.iplantcollaborative.org/wiki/display/sciplant/JSON+Tools+and+Editors

CloudFormation tutorials I found useful

http://blog.engelke.com/2012/12/19/learning-about-aws-cloudformation/

http://blogs.sequoiainc.com/blogs/automate-application-deployment-with-aws-cloudformation-part-1-parameters-3da69e21-a33b-42ec-ae9c-6ea737daa58a

http://blogs.sequoiainc.com/blogs/automate-application-deployment-with-aws-cloudformation-part-2-resources-125f46b3-491e-4d03-a275-948444358004

CloudFormation creation and generation of templates

CloudFormation is described as a JSON (JavaScript Object Notation) template. It's a model-driven template in that the AWS infrastructure is instantiated according to its own specification of proper order of execution. It is not a procedural language. If you are writing a CloudFormation template, you only need to follow the rules of the CloudFormation external Domain Specific Language (DSL) in JSON notation. Editing templates is not difficult, but it is a process that could require some time (not to mention validation). Once you create a template updating it is not difficult but the initial creation particularly for new applications and environments can be a lot of upfront effort.

One feature of CFN that may make validation amd selection easier is [[1]].

Some tools to help us:

CFN Validation

"Two biggest things I've found that help with cfn:

commit often, diffs will save your life run them often with the CLI.

I run them every time I add a resource or change anything. If you're apprehensive about running your template because it creates a lot of costly resources, at least validate it with the CLI:

http://docs.aws.amazon.com/cli/latest/reference/cloudformation/validate-template.html 

$ aws cloudformation validate-template --template-body file:////home//local//test//sampletemplate.json

http://blog.controlgroup.com/2013/11/04/review-cloudformation-cfn-summarize-cfn-show/

CFN Editors

Loose notes from my class notes. Need to clean up.

AWS prefers Java Eclipse IDE using AWS Toolkit - (install instructions at http://aws.amazon.com/eclipse/) http://docs.aws.amazon.com/AWSToolkitEclipse/latest/GettingStartedGuide/Welcome.html see also AWS Explorer - http://docs.aws.amazon.com/AWSToolkitEclipse/latest/GettingStartedGuide/aws_explorer.html

Would love a really RAD environment validation editor but that does not exist and is somewhat an anti-pattern in a cloud world.

http://www.reddit.com/r/aws/comments/27ek6s/cloudformation_editor/

https://github.com/cloudtools/troposphere

CFN Visualization

Visualization of diagram from template https://www.nephorider.com

CFN Management

YAML? - http://yaml.org/

Lots of talk of using Yaml to create JSON.

YAML vs JSON:

http://stackoverflow.com/questions/1726802/what-is-the-difference-between-yaml-and-json-when-to-prefer-one-over-the-other

"json is not a subset (although it's close), and the incompatibilities are infuriating when you encouter them. json libraries are generally faster... (stackoverflow.com/questions/2451732/…). yaml proponents will insist that it is a subset. if readability is a concern, use yaml. if interoperability and speed are a concern, use JSON."

Observations:

Python programmers are generally big fans of YAML, because of the use of indentation, rather than bracketed syntax, to indicate levels. Many programmers consider the attachment of "meaning" to indentation a poor choice.

YAML is very human readable however JSON is natively supported.

JSON is not self documenting but can have comments.

Cumulus takes YAML input and creates Stacks from it.

http://awsadvent.tumblr.com/post/106058327586/aws-advent-2014-repeatable-infrastructure-with

https://github.com/cotdsa/cumulus 

$ AMID=ami-1q23123123 cumulus -y example_stack.yaml -a create

would be seen by cloudformation with the ami id of ami-1q23123123

"The problem:

Amazon CloudFormation (CF) allows you to instantiate multiple AWS resources in a repeatable, ordered and structured method. As our infrastructure grew, so did our CF templates and soon they were monolithic and complex to maintain. We looked at splitting these templates into smaller chunks, which worked as a short term solution but created a new problem. With multiple templates dependant on other declared resources, we were forced to manually pass parameters for inter-stack operability. This greatly affected the repeatability of our stacks as we did not have an easy method to keep track of what parameters were used, especially those relating to physical resource IDs.

The solution:

Cumulus attempts to solve the problem by introducing a layer above CF templates, a stack configuration YAML file. This allows multiple CF stacks to be created in order and maintained respecting their dependencies. The YAML file stores values for parameters to be passed into each of the stacks. Parameters can be assigned with static values or will source the value of a parameter, output or resource of another stack described in the YAML file. Cumulus actively translates reference values to physical resource values on creation of the stack."

CFN Descriptions

Look at JSON query tools like command line tool: http://stedolan.github.io/jq/

Can be used to look at specific parameters: 

$ aws cloudformation describe-stack-resource --stack-name wordpress \
 --logical-resource-id DBInstance \
 | jq '.StackResourceDetail .PhysicalResourceId' \
 | xargs aws rds describe-db-instances --db-instance-identifier \
 | jq '.DBInstances[0] .Endpoint .Address'

Instead of the complete JSON response, we get exactly the piece of data we care about:

WDXXXXXXX22.chXXXXXXXXXimf.us-east-1.rds.amazonaws.com

Describe Stack resources:

http://blogs.aws.amazon.com/application-management/post/TxIGF3TLJQDKE5/CloudFormation-and-the-New-AWS-CLI

http://blog.controlgroup.com/2013/11/04/review-cloudformation-cfn-summarize-cfn-show/

Ruby Rake

Rake job for installing updated Stacks. I have this code from lab. (Upload later?)

You need the AWS keys in your environment variable before running this:

(these are not real keys) 

AWS_ACCESS_KEY_ID="AKIAJPGHQH5CWGXOSNNQ"; export AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY="BNV7ZbTtdLHjgaMh7SlhkpQMUoJoE99Qbwv3pTpg"; export AWS_SECRET_ACCESS_KEY
$ rake update_stack["$AMM_STACK_NAME","AppEC2InstanceType=m3.medium"]

If you see this error: 

$ rake update_stack["$AMM_STACK_NAME","AppEC2InstanceType=m3.medium"]
rake aborted!
cannot load such file -- aws-sdk
/Users/kevin/Dropbox/AWS/Class/AWS_Advanced_Ops/amm/Rakefile:1:in `<top (required)>'
(See full trace by running task with --trace)

To fix:

Now install aws-sdk-v1 and modify the Rakefile to use aws-sdk-v1:

Installs v1 of the SDK (which uses AWS:: rather than v2 which uses Aws:: calls).

The V2 that's installed results in "uninitialized constant AWS" errors. 

$ sudo gem install aws-sdk-v1

Edit Rakefile and change line: 

require 'aws-sdk-v1'

CFN Parameterization

Template: 

 "Parameters":{
 ...
 "AppEC2InstanceType":{
            "Description":"EC2 Instance Type to use. 32-bit or 64-bit.",
            "Type":"String",
            "Default":"m3.medium"
        },
 ...
 "ASLaunchConfiguration":{
 ...
 "InstanceType":{"Ref":"AppEC2InstanceType"},

Misc Tools

Retrieved from "https://kevininscoe.com/wiki/index.php?title=Kevin%27s_AWS_CloudFormation_Notes&oldid=344"