I’ve been neglecting this blog for a while now however I haven’t been away from Chef and Chef on Windows in the intervening period. In that time I’ve expanded the number of cookbooks and problems I’ve solved using Chef on Windows and thought I’d start posting some code samples and snippets that might be useful. Windows is a fundamentally different beast than Linux and there are unique challenges that after a while I’ve found patterns that solve these.

As such I’ve thrown up two Gists on Github on  my profile. The first is a series of Powershell based guards that can be used in resources to control the execution of actions: chef-windows-guards

The second is a series of powershell_script resource scripts that fill in some gaps in the available resources: chef-windows-scripts

I hope to add others soon as I refine some of my thinking and practices. Also I’m keen for others to try them out and submit any changes or improvements.

Advertisement

1. Discovering that the last 2u space in the rack is actually blocked by a rogue power cable threaded through the rails.

2. Not being able to figure out how to remove the crusty old rails that the last engineer left behind. Resorting to brute force leaves a smear of blood behind on the rack where your hand got caught on the sharp edge of the rail.

3. Losing a chunk of a finger trying to remove an old metal screw mount left behind by the last engineer.

4. Trying to lift a 30kg server up and onto flimsy metal rails at around face height and not being able to get all the notches lined up and locked in.

5. Discovering you’ve put the server in upside down.

6. The cable management arm won’t fit because the rack has ultra wide PDU’s. Create a new gash on your hand while trying to pull the arm back off of the rail.

7. While running the network cables you discover you have only 1m or 10m Ethernet cables when you really need 3m.

8. Powering on the server only to discover that one phase from the UPS is at capacity and that it’s the one that feeds your rack.

9. Dropping one of the bolts as you are removing the server to relocate it and it drops down the side of the rack into that weird sill at the bottom of the rack you can only get to when there are no servers in the rack. Smear a bit more blood on the rack trying to fish it out.

10. Discovering that the physical memory capacity is no longer enough due to more and larger VM’s. Then having having to order and replace all the RAM in all your hosts. Again.

This is part of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise.

At some point we will hit the need to use DSC resources in Chef recipes. The reality is the Chef cookbooks for Windows only go so far and most Microsoft products outside of Windows itself have no coverage at all. All the DSC resources available are now open source just like Chef cookbooks and are expanding all the time:

I suspect we will see more and more DSC resources come out of Microsoft’s product teams as well.

Chef DSC Resources

There are a couple of key requirements for computers to leverage DSC in Chef. Windows Managmement Framework 5 (Feb Preview or better) is needed for DSC to be available and Chef Client 12.5 or better is required to provide the two Chef resources that allow DSC resources to be consumed in cookbooks.

UPDATE: Chef Client 12.6 has just been released which removes the limitations below (see the changelog for full details, strangely the inclusion of the timeout attribute isn’t in the changelog but was included in the master branch). There is a requirement for a recent version of WMF 5 to be installed as Microsft relaxed the LCM requirements.

There are two Chef resources for using DSC with chef; dsc_script and dsc_resource. These are effectively mutually exclusive as they require the DSC Local Configuration Manager’s RefreshMode to be configured to Push (dsc_script) or Disabled (dsc_resource).

Currently the main limitation of dsc_resource is that this resource has a hard timeout of 10 minutes, thus its not useful for DSC resources like xSQLServerSetup that can run for some time. I’ve logged this issue with Chef in GitHub. It is also a closer analogue of other Chef resources than dsc_script is which makes writing declarations easy if you have prior Chef experience.

dsc_script is more flexible in terms of being able to use more of the DSC functionality, like being able to pass configuration data or use powershell code within the DSC definition. Downside is that it converts the declaration into a MOF file so securing credentials requires that you use the DSC method for encrypting strings. This involves having a certificate that supports encryption loaded on the computer and defining that certificate thumbprint in your Chef recipe. This leads to a bit of double handling where you should use Chef Vault to secure the credentials in the recipe but then have to effectively unencrypt and re-encrypt the secured string for DSC to embed into a MOF file.

Personally I prefer the dsc_resource approach as it’s clean and simple to use in a Chef recipe and I don’t have to deal with the re-encryption of secure strings (and all the troubleshooting pain it brings).

Getting Modules Installed

There are a few ways you can achieve this, you could bundle up the modules as zip files and distribute them within the a Chef cookbook or you can use the new package manager in WMF5..

The easy way – WMF5 Package Manager

By using the Chef resource “powershell_script” you can invoke the install-package cmdlet and use the get-package cmdlet as a guard. Note the use of the -force switch which will force the install even if the package repository is trusted. This isn’t good practice and a better way to do it is to  pre-define the trusted repositories and not use the -force switch.

powershell_script "install dsc module" do
  code 'install-package -name "xResourceModule" -force
  not_if 'if(get-package -name "xResourceModule"){$true}else{$false}

end

Some good references for Powershell’s package manager are below:

This is part of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise.

DCOM. It strikes fear and bewilderment into IT Pros everywhere. Usually because some third party application is installed that loads some DCOM components and at some point someone broke the security permissions or activation settings; and no one knows what the correct settings were.

If there was ever a case for automation and infrastructure as code then this is it. Personally, I have one fairly important third party framework that, occasionally over the years, has needed to be deployed/redeployed. And every time there is a period of head scratching as to how to get it configured correctly. It has a couple of DCOM objects that get installed but these don’t ever have the correct permissions or identity set after installation. Enter Chef and dcomperm.exe, a compiled tool from Microsoft’s SDK sample code, that allows for programmatic control of DCOM permissions.

It seems setting DCOM permissions is actually pretty hard to automate, the permission ACL’s are binary strings in the registry and a slight mistake trying to set a binary string renders the ACL unusable. There aren’t any Powershell commands out of the box so you either assemble a script that hits WMI or you use the compiled tool.

Fortunately this application actually ships with dcomperm.exe, as I suspect it tries to use it itself to set the permissions at some point. As such I’ll focus on how I bundled this into a Chef recipe and the limitations of doing so. Hopefully I’ll have an improved solution at some point in the future to share.

Set out below are the snippets of a cookbook, as one way of using dcomperm.exe to set permissions on a DCOM object. You need to retrieve the objects GUID in advance for this example. If you are setting the identity for the object to run under then please note that you should securely encrypt the password in a Chef vault, rather than just leaving it as plain text in the attributes!

In the attributes I set an array of options that I want the dcomperm.exe tool to run to set my object’s permissions and identity. Then in the recipe I loop through each of those strings of options and use an “execute” resource to then run dcomperm.exe with those options. You will need to distribute the dcomperm.exe in the cookbook as well, but I have not included that step in the sample for the sake of brevity.

The important thing to note is that there is no idempotence. Every time the recipe is run this section will run regardless of current settings. On the positive side it is a very fast set of commands to run and it does not affect the operation of the DCOM object when it is applied.

Attributes

default[‘poal_openroad’][‘dcom_options’] = [
  ‘-runas {9804E901-495A-11D4-A083-00C04F740D56}   domain\username password’,
  ‘-al {9804E901-495A-11d4-A083-00C04F740D56} default’,
  ‘-aa {9804E901-495A-11d4-A083-00C04F740D56} default’,
  ‘-al {9804E901-495A-11d4-A083-00C04F740D56} set domain\username permit’,
  ‘-aa {9804E901-495A-11d4-A083-00C04F740D56} set domain\username permit’
]

Recipe

#Set DCOM Permissions
node[‘cookbook’][‘dcom_options’].each do |dcom_options|
  execute “#{dcom_options}” do
    command “dcomperm.exe #{dcom_options}”
    cwd “#{node[‘cookbook’][‘bin’]}”
  end
end

This is the first of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise company.

To understand the nuances of running Chef on Windows you need to keep in mind that it evolved in the Linux\Open Source world. It’s a bit like humans trying to live on Mars, it can be done but you need to provide a special habitat and it takes longer to get things done sometimes.

Core to Chef is Ruby as the base programming layer that it defines its DSL in. Ruby’s performance and efficiency isn’t great in the Windows world. It’s a port from Linux and it isn’t optimised for Windows. What you’ll see is consistently high CPU usage during a Chef run, generally on a single cpu core. (I’ve found single core virtual machines suffer significantly during a Chef run in terms of other operations happening in parallel)

It also manifests in longer Chef runs especially with deploying a large number of Windows features for example.

The approach I’m taking more and more now is to utilise DSC resources within the Chef recipe which effectively hands off processing of the Windows specific configurations from Chef to the native DSC engine. A further post will go into the detail of how I’m implementing DSC in Chef.

Updating the Chef client is also a bit tricky, it ships as an MSI (A well built MSI that actually adheres to the MSI best practices) but you can’t update it during a Chef run for obvious reasons… Ultimately you’ll need to update it out of band to the Chef run, either with a different tool or by using a Chef recipe which sets up a scheduled task to run to do the update.

You will also need to make a decision on how the Chef client will run, will you trigger it on demand or run it as a service and have it automatically trigger chef runs. The good news is that the Chef client operates properly within a Powershell environment both locally and via remoting so there are plenty of options depending on your deployment processes.

The other key point to note is that Chef has ensured that Powershell is an option within recipes, both as a resource and as a guard interpreter (More on this later as well).

This is part of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise company.

The first question is why I chose Chef, especially considering it’s roots are definitely not in the Windows space!

Ultimately a choice had to be made and at the time it was made because of the following points.

  • A large and vibrant community (Helped by the fact that Steve Murawski, an ardent fan and early DSC adopter, joined the Chef team).
  • Chef identified early that it needed to embrace Windows and demonstrated a clear path to incorporating DSC into it’s DSL.
  • Templates! Oh how templates makes deploying configuration files, where one or two settings need to be dynamic, so simple. I can’t highlight this enough, being able to drop text files and then dynamically modify a single field based on environment or node specific variables is a godsend.
  • All the core functionality is open source and provides a complete system for managing and deploying configurations.
  • The Chef server and in built search tools give operations the ability to do some simple but dynamic service discovery and configuration. Not as good as doing service discovery native within the applications but better than static configuration files.

Of course the big question is why not straight Powershell DSC? I spent a fair bit of time with DSC before I adopted Chef and I found there were some fairly large hurdles to overcome. Some of them are still relevant today and some have been remediated or will be in the not to distant future.

The really painful problem relates back to managing text based configuration files which have some dynamic content within them. There’s no native functionality within DSC that will allow you to handle config files in the same manner as say Chef, so you have to implement your own logic. I went right down a rabbit hole writing a custom DSC resource to manage nodes in a XML file for one POC before I realised I was going to have to do something similar for every text based file format.

There’s a basic framework around DSC configuration deployment that requires you to write a lot of Powershell to assemble a deployment system from base DSC to deployed configuration. It can be done but it takes a lot of work to maintain that sort of system and I’d rather spend my time writing configurations than the plumbing to keep deployments working. DSC being primarily a DSL means you can use it different deployment and configuration scenarios but ultimately it’s not a singular tool for the end user. Microsoft and others (like Chef) are going to wrap their own deployment methodologies around the DSC DSL.

So Chef got chosen as the platform for attempting to automate the Windows beast. Stay tuned for the follow up articles where I discover it isn’t all roses in the automation world.

To get started with Chef hit the root of their official documentation at https://docs.chef.io/ or for a guided set of tutorials https://learn.chef.io/tutorials/

Shortly after my first blush of sucess with Chef, it become clear I needed to understand how to securely distribute credentials and file content such as SSL certificates. Off I went in search of the solution with old friend Google….

A bit later, having come to the conclusion that the chef-vault gem (open-sourced by Nordstrum and bundled into the Chefdk omnibus) was the answer I then went in seach of the real question “How do I actually use this?”. The answers were few and far between and only one blog really ran through the process when it came to file contents. @jttimberman’s post “Managing Secrets with Chef Vault” was the closest thing I came across that walked through the process. Unfortunately it was nearly two years out of date and in the world of OSS that’s an eternity.

Distributing passwords was simple enough but tackling a PEM SSL cert meant I ran into a bunch of baffling errors, especially with encoding the file content into a JSON format. After a lot of trial and error I found a way forward and considering the lack of info on the web and the very likely chance I’ll forget how I did it, I’ve posted the process below. I haven’t gone into detail on explaining what each step does and why it’s necessary; @jtimberman’s post still explains that well along with the official GitHub repo for reference.

This is the step by step updated procedure for distributing sensitive file contents via chef and chef-vault, as I figured it out. It’s valid as of the following versions

  • chef-dk 0.4.0
  • chef-vault 2.5
  • Chef client 12.0.2

1) Update the Chef-Vault gem (From what I can tell the omnibus doesn’t have this latest version which seems to have a number of fixes in it).

2) Update knife.rb with chef-vault settings.

knife[:vault_mode] = ‘client’ (If you are using Chef Server)

3) Upload the file contents using knife.

knife vault create <databag name> <item name> –file <full path to file to encrypt> –search ‘role:<role- names(s)>

4) Update your recipe to load the chef-vault gem and decrypt the file contents.

chef_gem “chef-vault”

require “chef-vault”

vault = ChefVault::Item.load(‘<vault name>’,'<item name>’)

 file “<path and name of file>” do

    content vault[‘file-content’]

    action :create

    sensitive true

end

References:
Github Chef-Vault repo

 

A quick update today that version 1.0.1 of my Powershell script and module for uninstalling Java has been released.

One fix and one update are in this version. The fix is for the detection of running Java processes which weren’t triggering the while loop to wait and/or kill before uninstalling Java. The update was to update the query string to pick up Java 8 installations.

https://poshuninstalljava.codeplex.com/

I still need to put together a more robust testing process, right now my thoughts are to use Teamcity and Pester to run up VM’s and do automated testing against all the permutations that are now possible.

Managing Configuration Drift

Posted: October 7, 2014 in Uncategorized

Much has been written on the rise of the devops culture and the use of configuration as code tools such as Chef. Puppet and DSC.

However once you have your infrastructure configuration built as code and deployed, how do you validate that it is actually being applied as expected?

Once you have more than a handful of nodes and multiple roles you can’t easily just manually log on to the servers and check. (Not to mention this is totally counter intuitive in the pets vs cattle approach to infrastructure management)

I’ve run into a tool from a start up called Scriptrock called Guardrail. It’s whole reason for existing is to solve the problem of validating deployed configuration. It takes the philosophy that you can’t rely on the police to police themselves. If Chef or DSC says it applied the config can you be sure it actually did?

Guardrail also can take a config policy and scaffold out a Chef, puppet or DSC configuration to speed up automating deployments.

I wholeheartedly recommend checking not out. They have a basic hosted trial option that allows for a rapid POC.

Today I’ve released what I’m considering a 1.0 release of the Uninstall-Java Powershell script and Module code.

I’ve performed some testing of both the script and module on Windows 7, 8 and 8.1 and the code seems to be stable and clean. As always I highly recommend testing in your environment.

This project started after the furor over major security bugs in Java and now has developed into a full project aimed at simplifying the management of Java versions in large environments by ensuring wither java is not installed or only the approved version is deployed. Previous articles (here and here) I wrote go into some of the detail and the development progress is tracked in some of the comments.

In the process I started my first Codeplex project as a way to distribute the code. There was a bit of a learning curve in using Codeplex and Git and along with some character encoding issues led to code being available which didn’t run when downloaded from Codeplex. These have been fixed (sorry to anyone who tried to use the damaged code) and is now available from the Codeplex project download page at https://poshuninstalljava.codeplex.com.

I’d appreciate any feedback or results of any testing that any of you do, either on the Codeplex project site or as a comment against this post. Documentation should be sufficient for anyone with basic Powershell knowledge and as time allows I’ll try and improve both the documentation in Codeplex and within the code itself.