I will keep this document updated as I move along.
Summary
Security Considerations
Under the instructions the Amazon Security Blog you need to do a few things to get started.
First, you need to create a new file called credentials in ~/.aws and set the rights to 600.
The credentials file needs to look like this:
[default]
aws_access_key_idx = "value here" <-- "This is the Access Key ID from IAM for the core user"
aws_secret_access_key = "value here" <-- "This is the secret ID from the CSV file that matches the access key"
Some things also need to be variables it seems. This is the default .bash_profile:
export AWS_ACCESS_KEY_ID="value here"
export AWS_SECRET_ACCESS_KEY="vale here"
export AWS_SSH_KEY_ID="PEM key name without the .pem"
export AWS_SSH_KEY="$HOME/.ssh/pem key with the .pem"
This is a bit of belt and suspenders, but it works and doesn’t throw irrational errors that keep you chasing your tail. Ideally you should not need the AWS_ACCESS_KEY and ID in your .bash_profile file, but some functions seem to need it.
You may want to set up a config file in ~/.ssh similar to:
# contents of $HOME/.ssh/config
Host chef
User ubuntu
HostName 52.91.89.20 <-- public IP address of instance
IdentityFile ~/.ssh/awskey.pem <-- aws key
Drivers
You will need the EC2 Drivers from GitHub You will also need to install the AWS SDK for Ruby v2 gem.
To install the gems:
$ gem install aws-sdk
$ gem install ec2
Instantiate the kitchen:
$ kitchen init --driver=kitchen-ec2 --create-gemfile
create .kitchen.yml
create test/integration/default
create Gemfile
append Gemfile
append Gemfile
You must run `bundle install' to fetch any new gems.
The .kitchen.yml file
Modify/tweak your .kitchen.yml file to look like either of these or use the baseline sample:
Ubuntu Sample
---
driver:
name: ec2 <-- Driver name
security_group_ids: ["security group"]
require_chef_omnibus: true
region: us-east-1 <-- Verify
availability_zone: d <-- Verify
subnet_id: "subnet-x"
associate_public_ip: true <-- If you want to connect from outside.
interface: private <-- To connect from in AWS
transport:
ssh_key: "/home/ubuntu/.ssh/AWSKEY.pem" <-- set to your key name
username: ["ubuntu"] <-- Connect user name (needs quotes and brackets)
provisioner:
name: chef_solo
platforms:
- name: ubuntu-14.04 <-- Descriptive name
driver:
image_id: ami-d05e75b8 <-- Verify
instance_type: t2.micro <-- Verify
block_device_mappings: <-- Optional
- ebs_device_name: /dev/sdb
ebs_volume_type: gp2
ebs_virtual_name: test
ebs_volume_size: 8
ebs_delete_on_termination: true
suites:
- name: default
run_list:
attributes:
CentOS/RHEL Sample
---
driver:
name: ec2
security_group_ids: ["security group"]
require_chef_omnibus: true
region: us-east-1 <-- zone may need verification
availability_zone: e <-- may need verification
subnet_id: "subnet-yoursubnet"
associate_public_ip: true
interface: private <-- when building from inside AWS
transport:
ssh_key: ~/.ssh/AWS.pem <-- set to your key name
username: ["ec2-user"] <-- may need to be root for CentOS, ubuntu for ubuntu
provisioner:
name: chef_solo
platforms:
- name: centos-6.4
driver:
image_id: ami-26cc934e <-- Verify
instance_type: t1.micro <-- Verify
block_device_mappings:
- ebs_device_name: /dev/sdb
ebs_volume_type: gp2
ebs_virtual_name: test
ebs_volume_size: 8
ebs_delete_on_termination: true
suites:
- name: default
run_list:
attributes:
Baseline file sample for both Ubuntu and CentOS/RHEL
---
driver:
name: ec2
require_chef_omnibus: true
aws_ssh_key_id: AWSKEY <-- AWS Key name (no .pem)
security_group_ids: ["sg-...f"] <-- security group
region: us-east-1 <-- verify your region
associate_public_ip: true <-- if you need to access the node outside AWS
interface: private <-- set to _private_ if you are inside AWS
provisioner:
name: chef_solo
transport:
ssh_key: "/location/.ssh/key.pem" <-- don't know why, but this has to be here and not in the individual sections.
platforms:
- name: rhel-7.1 <-- RHEL is not officially supported but will work
driver:
image_id: ami-12663b7a <-- verify the image
instance_type: t2.micro <-- verify the instance type and size
availability_zone: e <-- verify the zone it can run in
transport.username: ["ec2-user"] <-- user will vary _ec2-user_ is the default for RHEL, but may need _root_
subnet_id: "subnet-...2" <-- verify the subnet with the zone
block_device_mappings:
- ebs_device_name: /dev/sdb
ebs_volume_type: gp2
ebs_virtual_name: test
ebs_volume_size: 8
ebs_delete_on_termination: true
- name: ubuntu-14.04
driver:
image_id: ami-d05e75b8 <-- verify the image
instance_type: t2.micro <-- verify the instance type and size
availability_zone: d <-- verify the zone it can run in
subnet_id: subnet-...c <-- verify the subnet with the zone
transport.username: ["ubuntu"] <-- default name for Ubuntu
block_device_mappings:
- ebs_device_name: /dev/sdb
ebs_volume_type: gp2
ebs_virtual_name: test
ebs_volume_size: 8
ebs_delete_on_termination: true
suites:
- name: default
run_list:
attributes:
If you want to assign a static address to the host, you have to do it at kitchen create stage. In the platforms section add:
network:
- ["private_network", {ip: "172.31.47.69"}]
Using Kitchen
Kitchen List: Check your Instances and Actions
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
default-rhel-71 Ec2 ChefSolo Busser Ssh <Not Created>
default-ubuntu-1404 Ec2 ChefSolo Busser Ssh <Not Created>
Kitchen Create: Create an instance
$ kitchen create default-ubuntu-1404
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-ubuntu-1404>...
If you are not using an account that qualifies under the AWS free-tier, you may be charged to run these suites.
The charge should be minimal, but neither Test Kitchen nor its maintainers are responsible for your incurred costs.
Instance <i-d4f71865> requested.
EC2 instance <i-d4f71865> created.
Waited 0/300s for instance <i-d4f71865> to become ready.
Waited 5/300s for instance <i-d4f71865> to become ready.
Waited 10/300s for instance <i-d4f71865> to become ready.
Waited 15/300s for instance <i-d4f71865> to become ready.
Waited 20/300s for instance <i-d4f71865> to become ready.
Waited 25/300s for instance <i-d4f71865> to become ready.
Waited 30/300s for instance <i-d4f71865> to become ready.
Waited 35/300s for instance <i-d4f71865> to become ready.
EC2 instance <i-d4f71865> ready.
Waiting for SSH service on 172.31.63.224:22, retrying in 3 seconds
Waiting for SSH service on 172.31.63.224:22, retrying in 3 seconds
Waiting for SSH service on 172.31.63.224:22, retrying in 3 seconds
[SSH] Established
Finished creating <default-ubuntu-1404> (1m9.39s).
-----> Kitchen is finished. (1m9.46s)
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
default-rhel-71 Ec2 ChefSolo Busser Ssh <Not Created>
default-ubuntu-1404 Ec2 ChefSolo Busser Ssh Created
Kitchen Destroy: Destroy an Instance
$ kitchen destroy default-ubuntu-1404
-----> Starting Kitchen (v1.4.2)
-----> Destroying <default-ubuntu-1404>...
EC2 instance <i-d4f71865> destroyed.
Finished destroying <default-ubuntu-1404> (0m0.82s).
-----> Kitchen is finished. (0m0.87s)
Kitchen Setup: Install Chef on a node
$ kitchen setup default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-rhel-71>...
If you are not using an account that qualifies under the AWS free-tier, you may be charged to run these suites.
The charge should be minimal, but neither Test Kitchen nor its maintainers are responsible for your incurred costs.
Instance <i-387a1fc1> requested.
EC2 instance <i-387a1fc1> created.
Waited 0/300s for instance <i-387a1fc1> to become ready.
Waited 5/300s for instance <i-387a1fc1> to become ready.
Waited 10/300s for instance <i-387a1fc1> to become ready.
Waited 15/300s for instance <i-387a1fc1> to become ready.
Waited 20/300s for instance <i-387a1fc1> to become ready.
Waited 25/300s for instance <i-387a1fc1> to become ready.
Waited 30/300s for instance <i-387a1fc1> to become ready.
Waited 35/300s for instance <i-387a1fc1> to become ready.
EC2 instance <i-387a1fc1> ready.
Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
Please login as the user "ec2-user" rather than the user "root".
Please login as the user "ec2-user" rather than the user "root".
Finished creating <default-rhel-71> (1m47.75s).
-----> Converging <default-rhel-71>...
Preparing files for transfer
Preparing dna.json
Preparing current project directory as a cookbook
Removing non-cookbook files before transfer
Preparing solo.rb
Please login as the user "ec2-user" rather than the user "root".
Please login as the user "ec2-user" rather than the user "root".
-----> Starting Kitchen (v1.4.2)
-----> Converging <default-rhel-71>...
Preparing files for transfer
Preparing dna.json
Preparing current project directory as a cookbook
Removing non-cookbook files before transfer
Preparing solo.rb
-----> Installing Chef Omnibus (install only if missing)
Downloading https://www.chef.io/chef/install.sh to file /tmp/install.sh
Trying curl...
Download complete.
Downloading Chef for el...
downloading https://www.chef.io/chef/metadata?v=&prerelease=false&nightlies=false&p=el&pv=7&m=x86_64
to file /tmp/install.sh.10715/metadata.txt
trying curl...
url https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
md5 9333136ba8a11bd6cad6d28fcd26a2c7
sha256 7a937d8c0ab68a1f342aba4ad33417fc4ba8cb1a71f46e4a18b5e76c363e4075
downloaded metadata file looks valid...
downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
to file /tmp/install.sh.10715/chef-12.5.1-1.el7.x86_64.rpm
trying curl...
Comparing checksum with sha256sum...
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
You are installing an omnibus package without a version pin. If you are installing
on production servers via an automated process this is DANGEROUS and you will
be upgraded without warning on new releases, even to new major releases.
Letting the version float is only appropriate in desktop, test, development or
CI/CD environments.
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
Installing Chef
installing with rpm...
warning: /tmp/install.sh.10715/chef-12.5.1-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
Preparing... ################################# [100%]
Updating / installing... ################################# [100%]
Thank you for installing Chef!
Transferring files to <default-rhel-71>
Starting Chef Client, version 12.5.1
Compiling Cookbooks...
Converging 0 resources
Running handlers:
Running handlers complete
Chef Client finished, 0/0 resources updated in 00 seconds
Finished converging <default-rhel-71> (0m39.27s).
-----> Setting up <default-rhel-71>...
Finished setting up <default-rhel-71> (0m0.00s).
-----> Kitchen is finished. (0m39.32s)
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
default-rhel-71 Ec2 ChefSolo Busser Ssh Set Up
default-ubuntu-1404 Ec2 ChefSolo Busser Ssh <Not Created>
Kitchen Converge: Deploying a file to a node
Modify your .kitchen.yml file, and update the suites section with the recipe:
suites:
- name: default
run_list:
- recipe[motd::default]
attributes:
Then run the kitchen converge command:
$ kitchen converge default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-rhel-71>...
If you are not using an account that qualifies under the AWS free-tier, you may be charged to run these suites.
The charge should be minimal, but neither Test Kitchen nor its maintainers are responsible for your incurred costs.
Instance <i-af402556> requested.
EC2 instance <i-af402556> created.
Waited 0/300s for instance <i-af402556> to become ready.
Waited 5/300s for instance <i-af402556> to become ready.
Waited 10/300s for instance <i-af402556> to become ready.
Waited 15/300s for instance <i-af402556> to become ready.
Waited 20/300s for instance <i-af402556> to become ready.
Waited 25/300s for instance <i-af402556> to become ready.
EC2 instance <i-af402556> ready.
Waiting for SSH service on 172.31.45.65:22, retrying in 3 seconds
Waiting for SSH service on 172.31.45.65:22, retrying in 3 seconds
[SSH] Established
Finished creating <default-rhel-71> (1m4.66s).
-----> Converging <default-rhel-71>...
Preparing files for transfer
Preparing dna.json
Preparing current project directory as a cookbook
Removing non-cookbook files before transfer
Preparing solo.rb
-----> Installing Chef Omnibus (install only if missing)
Downloading https://www.chef.io/chef/install.sh to file /tmp/install.sh
Trying curl...
Download complete.
Downloading Chef for el...
downloading https://www.chef.io/chef/metadata?v=&prerelease=false&nightlies=false&p=el&pv=7&m=x86_64
to file /tmp/install.sh.5483/metadata.txt
trying curl...
url https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
md5 9333136ba8a11bd6cad6d28fcd26a2c7
sha256 7a937d8c0ab68a1f342aba4ad33417fc4ba8cb1a71f46e4a18b5e76c363e4075
downloaded metadata file looks valid...
downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
to file /tmp/install.sh.5483/chef-12.5.1-1.el7.x86_64.rpm
trying curl...
Comparing checksum with sha256sum...
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
You are installing an omnibus package without a version pin. If you are installing
on production servers via an automated process this is DANGEROUS and you will be upgraded without warning on new releases, even to new major releases.
Letting the version float is only appropriate in desktop, test, development or CI/CD environments.
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
Installing Chef
installing with rpm...
warning: /tmp/install.sh.5483/chef-12.5.1-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
Preparing... ################################# [100%]
Updating / installing... ################################# [100%]
Thank you for installing Chef!
Transferring files to <default-rhel-71>
Starting Chef Client, version 12.5.1
Compiling Cookbooks...
Converging 1 resources
Recipe: motd::default
* cookbook_file[/etc/motd] action create
- update content in file /etc/motd from e3b0c4 to 295b84
--- /etc/motd 2013-06-07 10:31:32.000000000 -0400
+++ /etc/.motd20151210-10819-18peqj2 2015-12-10 14:02:01.757471882 -0500
@@ -1 +1,10 @@
+ __________________________________
+/ You are on a simulated Chef node \
+\ environment /
+ ----------------------------------
+ \ ^__^
+ \ (oo)\_______
+ (__)\ )\/\
+ ||----w |
- restore selinux security context
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 00 seconds
Finished converging <default-rhel-71> (0m32.21s).
-----> Kitchen is finished. (1m36.95s)
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
default-rhel-71 Ec2 ChefSolo Busser Ssh Converged
default-ubuntu-1404 Ec2 ChefSolo Busser Ssh <Not Created>
$ ssh -i ~/.ssh/awskey.pem ec2-user@52.91.126.45
Last login: Thu Dec 10 14:02:00 2015 from ip-172-31-60-114.ec2.internal
__________________________________
/ You are on a simulated Chef node \
\ environment /
----------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[ec2-user@ip-172-31-45-65 ~]$ exit
logout
Connection to 52.91.126.45 closed.
Metadata.rb modifications
When you are creating a new recipe, you need to edit the metadata.rb file. For example, in the apache cookbook example, the file will look like:
name 'apache'
maintainer 'David A. Lane'
maintainer_email 'david.lane@gmx.com'
license 'All rights reserved'
description 'Installs/Configures apache'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '0.1.0'
Writing a recipe: Modifying recipe/default.rb
When you want to install a package, you will need to modify the default.rb file in the recipe subdirectory. An example, for installing apache is as follows:
#
# Cookbook Name:: apache
# Recipe:: default
#
# Copyright 2015, YOUR_COMPANY_NAME
#
# All rights reserved - Do Not Redistribute
#
package "httpd" do
action :install
end
Once you make that modificaiton run a kitchen converge [node] and it will install apache.
[ec2-user@ip-172-31-47-69 ~]$ rpm -qa httpd
httpd-2.4.6-40.el7.x86_64
Service Resource
You can take it a step further to install, and activate the package once it is installed by modifying the default.rb like this:
package "httpd"
service "httpd" do
action [ :enable, :start ]
end
Which should result in ouput like this:
$ kitchen converge default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Converging <default-rhel-71>...
Preparing files for transfer
Preparing dna.json
Preparing current project directory as a cookbook
Removing non-cookbook files before transfer
Preparing solo.rb
-----> Chef Omnibus installation detected (install only if missing)
Transferring files to <default-rhel-71>
Starting Chef Client, version 12.5.1
Compiling Cookbooks...
Converging 2 resources
Recipe: apache::default
(up to date)
- enable service service[httpd]
- start service service[httpd]
Running handlers:
Running handlers complete
Chef Client finished, 2/3 resources updated in 03 seconds
Finished converging <default-rhel-71> (0m5.05s).
-----> Kitchen is finished. (0m5.11s)
And on the server, you get:
[ec2-user@ip-172-31-47-69 ~]$ systemctl list-unit-files | grep httpd
httpd.service enabled
Template Resource
Modify the default.rb to add the template line as shown:
package "httpd"
service "httpd" do
action [ :enable, :start ]
end
template "/var/www/html/index.html" do
source 'index.html.erb'
mode '0644'
end
And then you need to create the index.html.erb file. Start by running chef generate template <file>:
$ chef generate template index.html
and then change into templates/default and edit the index.html.erb file with what you want to include, such as:
This site was set up by <%= node['hostname'] %>
and run another kitchen converge.
Check the output:
$ kitchen converge default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Converging <default-rhel-71>...
Preparing files for transfer
Preparing dna.json
Preparing current project directory as a cookbook
Removing non-cookbook files before transfer
Preparing solo.rb
-----> Chef Omnibus installation detected (install only if missing)
Transferring files to <default-rhel-71>
Starting Chef Client, version 12.5.1
Compiling Cookbooks...
Converging 3 resources
Recipe: apache::default
(up to date)
(up to date)
(up to date)
- create new file /var/www/html/index.html
- update content in file /var/www/html/index.html from none to b2f6ae
--- /var/www/html/index.html 2015-12-11 12:49:17.376524243 -0500
+++ /var/www/html/.index.html20151211-19185-1lfz25z 2015-12-11 12:49:17.376524243 -0500
@@ -1 +1,2 @@
+This site was set up by
- restore selinux security context
Running handlers:
Running handlers complete
Chef Client finished, 1/4 resources updated in 03 seconds
Finished converging <default-rhel-71> (0m5.03s).
-----> Kitchen is finished. (0m5.09s)
And then on the host, you can verify the installation:
[ec2-user@ip-172-31-47-69 ~]$ curl localhost
This site was set up by ip-172-31-47-69
Using Knife
Creating a Knife file
$ knife cookbook create motd --cookbook-path .
WARNING: No knife configuration file found
** Creating cookbook motd in /home/ubuntu/git/motd
** Creating README for cookbook: motd
** Creating CHANGELOG for cookbook: motd
** Creating metadata for cookbook: motd
$ kitchen init --create-gemfile
conflict .kitchen.yml
Overwrite /home/ubuntu/git/motd/.kitchen.yml? (enter "h" for help) [Ynaqdh] n
skip .kitchen.yml
conflict chefignore
Overwrite /home/ubuntu/git/motd/chefignore? (enter "h" for help) [Ynaqdh] y
force chefignore
create Gemfile
append Gemfile
append Gemfile
You must run `bundle install' to fetch any new gems.
$ bundle install
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using mixlib-shellout 2.2.5
Using net-ssh 2.9.2
Using net-scp 1.2.1
Using safe_yaml 1.0.4
Using thor 0.19.1
Using test-kitchen 1.4.2
Using kitchen-vagrant 0.19.0
Using bundler 1.10.6
Bundle complete! 2 Gemfile dependencies, 8 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.