Test Kitchen to support Amazon Web Service (AWS) AMIs

I will keep this document updated as I move along.

Summary


Security Considerations

Under the instructions the Amazon Security Blog you need to do a few things to get started.

First, you need to create a new file called credentials in ~/.aws and set the rights to 600.

The credentials file needs to look like this:

[default]
aws_access_key_idx = "value here" <-- "This is the Access Key ID from IAM for the core user"
aws_secret_access_key = "value here" <-- "This is the secret ID from the CSV file that matches the access key"

Some things also need to be variables it seems. This is the default .bash_profile:

export AWS_ACCESS_KEY_ID="value here"
export AWS_SECRET_ACCESS_KEY="vale here"
export AWS_SSH_KEY_ID="PEM key name without the .pem"
export AWS_SSH_KEY="$HOME/.ssh/pem key with the .pem"

This is a bit of belt and suspenders, but it works and doesn’t throw irrational errors that keep you chasing your tail. Ideally you should not need the AWS_ACCESS_KEY and ID in your .bash_profile file, but some functions seem to need it.

You may want to set up a config file in ~/.ssh similar to:

# contents of $HOME/.ssh/config
Host chef
    User ubuntu
    HostName 52.91.89.20  <-- public IP address of instance
    IdentityFile ~/.ssh/awskey.pem <-- aws key

Drivers

You will need the EC2 Drivers from GitHub You will also need to install the AWS SDK for Ruby v2 gem.

To install the gems:

 $ gem install aws-sdk
 $ gem install ec2

Instantiate the kitchen:

$ kitchen init --driver=kitchen-ec2 --create-gemfile
  create  .kitchen.yml
  create  test/integration/default
  create  Gemfile
  append  Gemfile
  append  Gemfile
You must run `bundle install' to fetch any new gems.

The .kitchen.yml file

Modify/tweak your .kitchen.yml file to look like either of these or use the baseline sample:

Ubuntu Sample

---
driver:
  name: ec2 <-- Driver name
  security_group_ids: ["security group"]
  require_chef_omnibus: true
  region: us-east-1 <-- Verify
  availability_zone: d <-- Verify
  subnet_id: "subnet-x"
  associate_public_ip: true <-- If you want to connect from outside.
  interface: private <-- To connect from in AWS

transport:
  ssh_key: "/home/ubuntu/.ssh/AWSKEY.pem" <-- set to your key name
  username: ["ubuntu"] <-- Connect user name (needs quotes and brackets)

provisioner:
  name: chef_solo

platforms:
  - name: ubuntu-14.04 <-- Descriptive name
  driver:
    image_id: ami-d05e75b8 <-- Verify
    instance_type: t2.micro <-- Verify
    block_device_mappings: <-- Optional
      - ebs_device_name: /dev/sdb
        ebs_volume_type: gp2
        ebs_virtual_name: test
        ebs_volume_size: 8
        ebs_delete_on_termination: true

  suites:
    - name: default
    run_list:
    attributes:

CentOS/RHEL Sample

---
driver:
  name: ec2
  security_group_ids: ["security group"]
  require_chef_omnibus: true
  region: us-east-1 <-- zone may need verification
  availability_zone: e <-- may need verification
  subnet_id: "subnet-yoursubnet"
  associate_public_ip: true
  interface: private <-- when building from inside AWS

transport:
  ssh_key: ~/.ssh/AWS.pem <-- set to your key name
  username: ["ec2-user"] <-- may need to be root for CentOS, ubuntu for ubuntu

provisioner:
  name: chef_solo

platforms:
  - name: centos-6.4
driver:
  image_id: ami-26cc934e <-- Verify
  instance_type: t1.micro <-- Verify
  block_device_mappings:
    - ebs_device_name: /dev/sdb
      ebs_volume_type: gp2
      ebs_virtual_name: test
      ebs_volume_size: 8
      ebs_delete_on_termination: true  

suites:
  - name: default
    run_list:
    attributes:         

Baseline file sample for both Ubuntu and CentOS/RHEL

---
driver: 
  name: ec2
  require_chef_omnibus: true
  aws_ssh_key_id: AWSKEY <-- AWS Key name (no .pem)
  security_group_ids: ["sg-...f"] <-- security group
  region: us-east-1 <-- verify your region
  associate_public_ip: true <-- if you need to access the node outside AWS
  interface: private <-- set to _private_ if you are inside AWS

provisioner:
   name: chef_solo
transport:
   ssh_key: "/location/.ssh/key.pem" <-- don't know why, but this has to be here and not in the individual sections. 

platforms:
   - name: rhel-7.1 <-- RHEL is not officially supported but will work
     driver:
       image_id: ami-12663b7a <-- verify the image 
       instance_type: t2.micro <-- verify the instance type and size
       availability_zone: e <-- verify the zone it can run in
       transport.username: ["ec2-user"] <-- user will vary _ec2-user_ is the default for RHEL, but may need _root_
       subnet_id: "subnet-...2" <-- verify the subnet with the zone
       block_device_mappings:
         - ebs_device_name: /dev/sdb
           ebs_volume_type: gp2
           ebs_virtual_name: test
           ebs_volume_size: 8
           ebs_delete_on_termination: true

- name: ubuntu-14.04
     driver:
     image_id: ami-d05e75b8 <-- verify the image
     instance_type: t2.micro <-- verify the instance type and size
     availability_zone: d <-- verify the zone it can run in
     subnet_id: subnet-...c <-- verify the subnet with the zone
     transport.username: ["ubuntu"] <-- default name for Ubuntu
     block_device_mappings:
       - ebs_device_name: /dev/sdb
         ebs_volume_type: gp2
         ebs_virtual_name: test
         ebs_volume_size: 8
         ebs_delete_on_termination: true

suites:
  - name: default
    run_list:
    attributes:

If you want to assign a static address to the host, you have to do it at kitchen create stage. In the platforms section add:

network:
   - ["private_network", {ip: "172.31.47.69"}]

Using Kitchen

Kitchen List: Check your Instances and Actions

$ kitchen list
Instance             Driver  Provisioner  Verifier  Transport   Last Action
default-rhel-71      Ec2     ChefSolo     Busser    Ssh         <Not Created>
default-ubuntu-1404  Ec2     ChefSolo     Busser    Ssh         <Not Created>

Kitchen Create: Create an instance

$ kitchen create default-ubuntu-1404
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-ubuntu-1404>...
    If you are not using an account that qualifies under the AWS free-tier, you may be charged to run these suites. 
    The charge should be minimal, but neither Test Kitchen nor its maintainers are responsible for your incurred costs.

   Instance <i-d4f71865> requested.
   EC2 instance <i-d4f71865> created.
   Waited 0/300s for instance <i-d4f71865> to become ready.
   Waited 5/300s for instance <i-d4f71865> to become ready.
   Waited 10/300s for instance <i-d4f71865> to become ready.
   Waited 15/300s for instance <i-d4f71865> to become ready.
   Waited 20/300s for instance <i-d4f71865> to become ready.
   Waited 25/300s for instance <i-d4f71865> to become ready.
   Waited 30/300s for instance <i-d4f71865> to become ready.
   Waited 35/300s for instance <i-d4f71865> to become ready.
   EC2 instance <i-d4f71865> ready.
   Waiting for SSH service on 172.31.63.224:22, retrying in 3 seconds
   Waiting for SSH service on 172.31.63.224:22, retrying in 3 seconds
   Waiting for SSH service on 172.31.63.224:22, retrying in 3 seconds
       [SSH] Established
       Finished creating <default-ubuntu-1404> (1m9.39s).
-----> Kitchen is finished. (1m9.46s)

$ kitchen list
Instance             Driver  Provisioner  Verifier  Transport   Last Action
default-rhel-71      Ec2     ChefSolo     Busser    Ssh         <Not Created>
default-ubuntu-1404  Ec2     ChefSolo     Busser    Ssh         Created

Kitchen Destroy: Destroy an Instance

$ kitchen destroy default-ubuntu-1404
-----> Starting Kitchen (v1.4.2)
-----> Destroying <default-ubuntu-1404>...
       EC2 instance <i-d4f71865> destroyed.
       Finished destroying <default-ubuntu-1404> (0m0.82s).
-----> Kitchen is finished. (0m0.87s)

Kitchen Setup: Install Chef on a node

$ kitchen setup default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-rhel-71>...
If you are not using an account that qualifies under the AWS free-tier, you may be charged to run these suites. 
The charge should be minimal, but neither Test Kitchen nor its maintainers are responsible for your incurred costs.

   Instance <i-387a1fc1> requested.
   EC2 instance <i-387a1fc1> created.
   Waited 0/300s for instance <i-387a1fc1> to become ready.
   Waited 5/300s for instance <i-387a1fc1> to become ready.
   Waited 10/300s for instance <i-387a1fc1> to become ready.
   Waited 15/300s for instance <i-387a1fc1> to become ready.
   Waited 20/300s for instance <i-387a1fc1> to become ready.
   Waited 25/300s for instance <i-387a1fc1> to become ready.
   Waited 30/300s for instance <i-387a1fc1> to become ready.
   Waited 35/300s for instance <i-387a1fc1> to become ready.
   EC2 instance <i-387a1fc1> ready.
   Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
   Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
   Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
   Waiting for SSH service on 172.31.41.13:22, retrying in 3 seconds
   Please login as the user "ec2-user" rather than the user "root".

   Please login as the user "ec2-user" rather than the user "root".

   Finished creating <default-rhel-71> (1m47.75s).
-----> Converging <default-rhel-71>...
   Preparing files for transfer
   Preparing dna.json
   Preparing current project directory as a cookbook
   Removing non-cookbook files before transfer
   Preparing solo.rb
   Please login as the user "ec2-user" rather than the user "root".

   Please login as the user "ec2-user" rather than the user "root".

-----> Starting Kitchen (v1.4.2)
-----> Converging <default-rhel-71>...
   Preparing files for transfer
   Preparing dna.json
   Preparing current project directory as a cookbook
   Removing non-cookbook files before transfer
   Preparing solo.rb
-----> Installing Chef Omnibus (install only if missing)
   Downloading https://www.chef.io/chef/install.sh to file /tmp/install.sh
   Trying curl...
   Download complete.
   Downloading Chef  for el...
   downloading https://www.chef.io/chef/metadata?v=&prerelease=false&nightlies=false&p=el&pv=7&m=x86_64
     to file /tmp/install.sh.10715/metadata.txt
   trying curl...
   url  https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
   md5  9333136ba8a11bd6cad6d28fcd26a2c7
   sha256   7a937d8c0ab68a1f342aba4ad33417fc4ba8cb1a71f46e4a18b5e76c363e4075
   downloaded metadata file looks valid...
   downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
     to file /tmp/install.sh.10715/chef-12.5.1-1.el7.x86_64.rpm
   trying curl...
   Comparing checksum with sha256sum...

   WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

   You are installing an omnibus package without a version pin.  If you are installing
   on production servers via an automated process this is DANGEROUS and you will
   be upgraded without warning on new releases, even to new major releases.
   Letting the version float is only appropriate in desktop, test, development or
   CI/CD environments.

   WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

   Installing Chef 
   installing with rpm...
   warning: /tmp/install.sh.10715/chef-12.5.1-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
   Preparing...             ################################# [100%]
   Updating / installing... ################################# [100%]
   Thank you for installing Chef!
   Transferring files to <default-rhel-71>
   Starting Chef Client, version 12.5.1
   Compiling Cookbooks...
   Converging 0 resources

   Running handlers:
   Running handlers complete
   Chef Client finished, 0/0 resources updated in 00 seconds
   Finished converging <default-rhel-71> (0m39.27s).
-----> Setting up <default-rhel-71>...
   Finished setting up <default-rhel-71> (0m0.00s).
-----> Kitchen is finished. (0m39.32s)

$ kitchen list
Instance             Driver  Provisioner  Verifier  Transport   Last Action
default-rhel-71      Ec2     ChefSolo     Busser    Ssh         Set Up
default-ubuntu-1404  Ec2     ChefSolo     Busser    Ssh         <Not Created>

Kitchen Converge: Deploying a file to a node

Modify your .kitchen.yml file, and update the suites section with the recipe:

suites:
  - name: default
    run_list:
      - recipe[motd::default]
    attributes:

Then run the kitchen converge command:

$ kitchen converge default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-rhel-71>...
   If you are not using an account that qualifies under the AWS free-tier, you may be charged to run these suites. 
   The charge should be minimal, but neither Test Kitchen nor its maintainers are responsible for your incurred costs.

   Instance <i-af402556> requested.
   EC2 instance <i-af402556> created.
   Waited 0/300s for instance <i-af402556> to become ready.
   Waited 5/300s for instance <i-af402556> to become ready.
   Waited 10/300s for instance <i-af402556> to become ready.
   Waited 15/300s for instance <i-af402556> to become ready.
   Waited 20/300s for instance <i-af402556> to become ready.
   Waited 25/300s for instance <i-af402556> to become ready.
   EC2 instance <i-af402556> ready.
   Waiting for SSH service on 172.31.45.65:22, retrying in 3 seconds
   Waiting for SSH service on 172.31.45.65:22, retrying in 3 seconds
   [SSH] Established
   Finished creating <default-rhel-71> (1m4.66s).
-----> Converging <default-rhel-71>...
   Preparing files for transfer
   Preparing dna.json
   Preparing current project directory as a cookbook
   Removing non-cookbook files before transfer
   Preparing solo.rb
-----> Installing Chef Omnibus (install only if missing)
   Downloading https://www.chef.io/chef/install.sh to file /tmp/install.sh
   Trying curl...
   Download complete.
   Downloading Chef  for el...
   downloading https://www.chef.io/chef/metadata?v=&prerelease=false&nightlies=false&p=el&pv=7&m=x86_64
     to file /tmp/install.sh.5483/metadata.txt
   trying curl...
   url  https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
   md5  9333136ba8a11bd6cad6d28fcd26a2c7
   sha256   7a937d8c0ab68a1f342aba4ad33417fc4ba8cb1a71f46e4a18b5e76c363e4075
   downloaded metadata file looks valid...
   downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/7/x86_64/chef-12.5.1-1.el7.x86_64.rpm
     to file /tmp/install.sh.5483/chef-12.5.1-1.el7.x86_64.rpm
   trying curl...
   Comparing checksum with sha256sum...

   WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

   You are installing an omnibus package without a version pin.  If you are installing
    on production servers via an automated process this is DANGEROUS and you will be upgraded without warning on new releases, even to new major releases.
   Letting the version float is only appropriate in desktop, test, development or CI/CD environments.

   WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

   Installing Chef 
   installing with rpm...
   warning: /tmp/install.sh.5483/chef-12.5.1-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
   Preparing...             ################################# [100%]
   Updating / installing... ################################# [100%]
   Thank you for installing Chef!
   Transferring files to <default-rhel-71>
   Starting Chef Client, version 12.5.1
   Compiling Cookbooks...
   Converging 1 resources
   Recipe: motd::default
     * cookbook_file[/etc/motd] action create
       - update content in file /etc/motd from e3b0c4 to 295b84
       --- /etc/motd    2013-06-07 10:31:32.000000000 -0400
       +++ /etc/.motd20151210-10819-18peqj2 2015-12-10 14:02:01.757471882 -0500
       @@ -1 +1,10 @@
       + __________________________________
       +/ You are on a simulated Chef node \
       +\ environment                      /
       + ----------------------------------
       +        \   ^__^
       +         \  (oo)\_______
       +            (__)\       )\/\
       +                ||----w |

       - restore selinux security context

   Running handlers:
   Running handlers complete
   Chef Client finished, 1/1 resources updated in 00 seconds
   Finished converging <default-rhel-71> (0m32.21s).
-----> Kitchen is finished. (1m36.95s)

$ kitchen list
Instance             Driver  Provisioner  Verifier  Transport   Last Action
default-rhel-71      Ec2     ChefSolo     Busser    Ssh         Converged
default-ubuntu-1404  Ec2     ChefSolo     Busser    Ssh         <Not Created>

$ ssh -i ~/.ssh/awskey.pem ec2-user@52.91.126.45
Last login: Thu Dec 10 14:02:00 2015 from ip-172-31-60-114.ec2.internal
 __________________________________
/ You are on a simulated Chef node \
\ environment                      /
----------------------------------
        \       ^__^
         \      (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||
[ec2-user@ip-172-31-45-65 ~]$ exit
logout
Connection to 52.91.126.45 closed.

Metadata.rb modifications

When you are creating a new recipe, you need to edit the metadata.rb file. For example, in the apache cookbook example, the file will look like:

name             'apache'
maintainer       'David A. Lane'
maintainer_email 'david.lane@gmx.com'
license          'All rights reserved'
description      'Installs/Configures apache'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version          '0.1.0'

Writing a recipe: Modifying recipe/default.rb

When you want to install a package, you will need to modify the default.rb file in the recipe subdirectory. An example, for installing apache is as follows:

#
# Cookbook Name:: apache
# Recipe:: default
#
# Copyright 2015, YOUR_COMPANY_NAME
#
# All rights reserved - Do Not Redistribute
#

package "httpd" do
  action :install
end

Once you make that modificaiton run a kitchen converge [node] and it will install apache.

[ec2-user@ip-172-31-47-69 ~]$ rpm -qa httpd
httpd-2.4.6-40.el7.x86_64

Service Resource

You can take it a step further to install, and activate the package once it is installed by modifying the default.rb like this:

package "httpd" 

service "httpd" do
  action [ :enable, :start ]
end

Which should result in ouput like this:

$ kitchen converge default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Converging <default-rhel-71>...
   Preparing files for transfer
   Preparing dna.json
   Preparing current project directory as a cookbook
   Removing non-cookbook files before transfer
   Preparing solo.rb
-----> Chef Omnibus installation detected (install only if missing)
   Transferring files to <default-rhel-71>
   Starting Chef Client, version 12.5.1
   Compiling Cookbooks...
   Converging 2 resources
   Recipe: apache::default
    (up to date)

       - enable service service[httpd]

       - start service service[httpd]

   Running handlers:
   Running handlers complete
   Chef Client finished, 2/3 resources updated in 03 seconds
   Finished converging <default-rhel-71> (0m5.05s).
-----> Kitchen is finished. (0m5.11s)

And on the server, you get:

[ec2-user@ip-172-31-47-69 ~]$ systemctl list-unit-files | grep httpd
httpd.service                               enabled 

Template Resource

Modify the default.rb to add the template line as shown:

package "httpd"

service "httpd" do
  action [ :enable, :start ]
end

template "/var/www/html/index.html" do
  source 'index.html.erb'
  mode '0644'
end

And then you need to create the index.html.erb file. Start by running chef generate template <file>:

$ chef generate template index.html

and then change into templates/default and edit the index.html.erb file with what you want to include, such as:

This site was set up by <%= node['hostname'] %>

and run another kitchen converge.

Check the output:

$ kitchen converge default-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Converging <default-rhel-71>...
   Preparing files for transfer
   Preparing dna.json
   Preparing current project directory as a cookbook
   Removing non-cookbook files before transfer
   Preparing solo.rb
-----> Chef Omnibus installation detected (install only if missing)
   Transferring files to <default-rhel-71>
   Starting Chef Client, version 12.5.1
   Compiling Cookbooks...
   Converging 3 resources
   Recipe: apache::default
    (up to date)
    (up to date)
    (up to date)

       - create new file /var/www/html/index.html
       - update content in file /var/www/html/index.html from none to b2f6ae
       --- /var/www/html/index.html 2015-12-11 12:49:17.376524243 -0500
       +++ /var/www/html/.index.html20151211-19185-1lfz25z  2015-12-11 12:49:17.376524243 -0500
       @@ -1 +1,2 @@
       +This site was set up by 

       - restore selinux security context

   Running handlers:
   Running handlers complete
   Chef Client finished, 1/4 resources updated in 03 seconds
   Finished converging <default-rhel-71> (0m5.03s).
-----> Kitchen is finished. (0m5.09s)

And then on the host, you can verify the installation:

[ec2-user@ip-172-31-47-69 ~]$ curl localhost
This site was set up by ip-172-31-47-69 

Using Knife

Creating a Knife file

$ knife cookbook create motd --cookbook-path .
WARNING: No knife configuration file found
** Creating cookbook motd in /home/ubuntu/git/motd
** Creating README for cookbook: motd
** Creating CHANGELOG for cookbook: motd
** Creating metadata for cookbook: motd

$ kitchen init --create-gemfile
conflict  .kitchen.yml
Overwrite /home/ubuntu/git/motd/.kitchen.yml? (enter "h" for help) [Ynaqdh] n
    skip  .kitchen.yml
conflict  chefignore
Overwrite /home/ubuntu/git/motd/chefignore? (enter "h" for help) [Ynaqdh] y
   force  chefignore
  create  Gemfile
  append  Gemfile
  append  Gemfile
You must run `bundle install' to fetch any new gems.

$ bundle install
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using mixlib-shellout 2.2.5
Using net-ssh 2.9.2
Using net-scp 1.2.1
Using safe_yaml 1.0.4
Using thor 0.19.1
Using test-kitchen 1.4.2
Using kitchen-vagrant 0.19.0
Using bundler 1.10.6
Bundle complete! 2 Gemfile dependencies, 8 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

To Dos