Archive for the ‘Automation’ Category

This is part of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise.

DCOM. It strikes fear and bewilderment into IT Pros everywhere. Usually because some third party application is installed that loads some DCOM components and at some point someone broke the security permissions or activation settings; and no one knows what the correct settings were.

If there was ever a case for automation and infrastructure as code then this is it. Personally, I have one fairly important third party framework that, occasionally over the years, has needed to be deployed/redeployed. And every time there is a period of head scratching as to how to get it configured correctly. It has a couple of DCOM objects that get installed but these don’t ever have the correct permissions or identity set after installation. Enter Chef and dcomperm.exe, a compiled tool from Microsoft’s SDK sample code, that allows for programmatic control of DCOM permissions.

It seems setting DCOM permissions is actually pretty hard to automate, the permission ACL’s are binary strings in the registry and a slight mistake trying to set a binary string renders the ACL unusable. There aren’t any Powershell¬†commands out of the box so you either assemble a script that hits WMI or you use the compiled tool.

Fortunately this application actually ships with dcomperm.exe, as I suspect it tries to use it itself to set the permissions at some point. As such I’ll focus on how I bundled this into a Chef recipe and the limitations of doing so. Hopefully I’ll have an improved solution at some point in the future to share.

Set out below are the snippets of a cookbook, as one way of using dcomperm.exe to set permissions on a DCOM object. You need to retrieve the objects GUID in advance for this example. If you are setting the identity for the object to run under then please note that you should securely encrypt the password in a Chef vault, rather than just leaving it as plain text in the attributes!

In the attributes I set an array of options that I want the dcomperm.exe tool to run to set my object’s permissions and identity. Then in the recipe I loop through each of those strings of options and use an “execute” resource to then run dcomperm.exe with those options. You will need to distribute the dcomperm.exe in the cookbook as well, but I have not included that step in the sample for the sake of brevity.

The important thing to note is that there is no idempotence. Every time the recipe is run this section will run regardless of current settings. On the positive side it is a very fast set of commands to run and it does not affect the operation of the DCOM object when it is applied.

Attributes

default[‘poal_openroad’][‘dcom_options’] = [
  ‘-runas {9804E901-495A-11D4-A083-00C04F740D56}   domain\username¬†password’,
  ‘-al {9804E901-495A-11d4-A083-00C04F740D56} default’,
  ‘-aa {9804E901-495A-11d4-A083-00C04F740D56} default’,
  ‘-al {9804E901-495A-11d4-A083-00C04F740D56} set¬†domain\username permit’,
  ‘-aa {9804E901-495A-11d4-A083-00C04F740D56} set¬†domain\username permit’
]

Recipe

#Set DCOM Permissions
node[‘cookbook’][‘dcom_options’].each do |dcom_options|
  execute “#{dcom_options}” do
    command “dcomperm.exe #{dcom_options}”
    cwd “#{node[‘cookbook’][‘bin’]}”
  end
end

This is the first of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise company.

To understand the nuances of running Chef on Windows you need to keep in mind that it evolved in the Linux\Open Source world. It’s a bit like humans trying to live on Mars, it can be done but you need to provide a special habitat and it takes longer to get things done sometimes.

Core to Chef is Ruby as¬†the base programming layer that it defines its DSL in. Ruby’s performance and efficiency isn’t great in the Windows world. It’s a port from Linux and it isn’t optimised for Windows. What you’ll see is¬†consistently high CPU usage during a Chef run, generally on a single cpu core. (I’ve found single core virtual machines suffer significantly during a Chef run in terms of other operations happening in parallel)

It also manifests in longer Chef runs especially with deploying a large number of Windows features for example.

The approach I’m taking more and more now is to utilise DSC resources within the Chef recipe which effectively hands off processing of the Windows specific configurations from Chef to the native DSC engine. A further post will go into the detail of how I’m implementing DSC in Chef.

Updating the Chef client is also a bit tricky, it ships as an MSI (A well built MSI that actually adheres to the MSI best practices) but you can’t update it during a Chef run for obvious reasons… Ultimately you’ll need to update it out of band to the Chef run, either with a different tool or by using a Chef recipe which sets up a scheduled task to run to do the update.

You will also need to make a decision on how the Chef client will run, will you trigger it on demand or run it as a service and have it automatically trigger chef runs. The good news is that the Chef client operates properly within a Powershell environment both locally and via remoting so there are plenty of options depending on your deployment processes.

The other key point to note is that Chef has ensured that Powershell is an option within recipes, both as a resource and as a guard interpreter (More on this later as well).

This is¬†part of a series of posts on automating Windows and deploying systems in the real world of enterprise operations. I’ve focused the series on the those nuances and problems that I’ve hit trying to deploy the various software stacks found in a typical Windows centric enterprise company.

The first question is why I chose Chef, especially considering it’s roots are definitely not in the Windows space!

Ultimately a choice had to be made and at the time it was made because of the following points.

  • A large and vibrant community (Helped by the fact that Steve Murawski,¬†an ardent fan and early DSC adopter, joined the Chef team).
  • Chef identified early that it needed to embrace Windows and demonstrated a clear path to incorporating DSC into it’s DSL.
  • Templates! Oh how templates makes deploying configuration files, where one or two settings need to be dynamic, so simple. I can’t highlight this enough, being able to drop text files and then dynamically modify a single field based on environment or node specific variables is a godsend.
  • All the core functionality is open source and provides a complete system for managing and deploying configurations.
  • The Chef server and in built search tools give operations the ability to do some simple but¬†dynamic service discovery and configuration. Not as good as doing¬†service discovery native within the applications but better than static configuration files.

Of course the big question is why not straight Powershell DSC? I spent a fair bit of time with DSC before I adopted Chef and I found there were some fairly large hurdles to overcome. Some of them are still relevant today and some have been remediated or will be in the not to distant future.

The really painful problem relates back to managing text based configuration files which have some dynamic content within them. There’s no native functionality within DSC that will allow you to handle config files in the same manner as say Chef,¬†so you have to implement your own logic. I went right down a rabbit hole writing a custom DSC resource to manage nodes in a XML file for one POC before I realised I was going to have to do something similar for every text based file format.

There’s¬†a basic¬†framework around DSC configuration deployment that requires you to write a lot of Powershell to assemble a deployment system from base DSC to deployed configuration. It can be done but it takes a lot of work to maintain that sort of system and I’d rather spend my time writing configurations than the plumbing to keep deployments working. DSC being primarily a DSL means you can use it different deployment and configuration scenarios but ultimately it’s not a singular tool for the end user. Microsoft and others (like Chef) are going to wrap their own deployment methodologies around the DSC DSL.

So Chef got chosen as the platform for attempting to automate the Windows beast. Stay tuned for the follow up articles where I discover it isn’t all roses in the automation world.

To get started with Chef hit the root of their official documentation at https://docs.chef.io/ or for a guided set of tutorials https://learn.chef.io/tutorials/

Shortly after my first blush of sucess with Chef, it become clear I needed to understand how to securely distribute credentials and file content such as SSL certificates. Off I went in search of the solution with old friend Google….

A bit later, having come to the conclusion that the chef-vault gem (open-sourced by Nordstrum and bundled into the Chefdk omnibus) was the answer I then went in seach of the real question “How do I actually¬†use this?”. The answers were few and far between and only one blog really ran through the process when it came to file contents. @jttimberman’s post “Managing Secrets with Chef Vault” was the closest thing I came across that walked through the process. Unfortunately it was nearly two years out of date and in the world of OSS that’s an eternity.

Distributing passwords was simple enough but tackling a PEM SSL cert meant¬†I ran into a bunch of baffling errors, especially with encoding the file content into a JSON format. After a lot of trial and error I found a way forward and considering the lack of info on the web and the very likely chance I’ll forget how I did it, I’ve posted the process below. I haven’t gone into detail on explaining what each step does and why it’s necessary; @jtimberman’s post still explains that well along with the official GitHub repo for reference.

This is the step by step updated procedure for distributing sensitive file contents via chef and chef-vault, as I figured it out. It’s valid as of the following versions

  • chef-dk 0.4.0
  • chef-vault 2.5
  • Chef client 12.0.2

1) Update the Chef-Vault gem (From what I can tell the omnibus doesn’t have this latest version which seems to have a number of fixes in it).

2) Update knife.rb with chef-vault settings.

knife[:vault_mode] = ‘client’ (If you are using Chef Server)

3) Upload the file contents using knife.

knife vault create <databag name> <item name> –file <full path to file to encrypt> –search ‘role:<role- names(s)>

4) Update your recipe to load the chef-vault gem and decrypt the file contents.

chef_gem “chef-vault”

require “chef-vault”

vault = ChefVault::Item.load(‘<vault name>’,'<item name>’)

¬†file “<path and name of file>” do

¬†¬†¬† content vault[‘file-content’]

    action :create

    sensitive true

end

References:
Github Chef-Vault repo

 

A quick update today that version 1.0.1 of my Powershell script and module for uninstalling Java has been released.

One fix and one update are in this version. The fix is for the detection of running Java processes which weren’t triggering the while loop to wait and/or kill before uninstalling Java. The update was to update the query string to pick up Java 8 installations.

https://poshuninstalljava.codeplex.com/

I still need to put together a more robust testing process, right now my thoughts are to use Teamcity and Pester to run up VM’s and do automated testing against all the permutations that are now possible.

Today I’ve released what I’m considering a 1.0 release of the Uninstall-Java Powershell script and Module code.

I’ve performed some testing of both the script and module on Windows 7, 8 and 8.1 and the code seems to be stable and clean. As always I highly recommend testing in your environment.

This project started after the furor over major security bugs in Java and now has developed into a full project aimed at simplifying the management of Java versions in large environments by ensuring wither java is not installed or only the approved version is deployed. Previous articles (here and here) I wrote go into some of the detail and the development progress is tracked in some of the comments.

In the process I started my first Codeplex project as a way to distribute the code. There was a bit of a learning curve in using Codeplex and Git and along with some character encoding issues led to code being available which didn’t run when downloaded from Codeplex. These have been fixed (sorry to anyone who tried to use the damaged code) and is now available from the Codeplex project download page at¬†https://poshuninstalljava.codeplex.com.

I’d appreciate any feedback or results of any testing that any of you do, either on the Codeplex project site or as a comment against this post. Documentation should be sufficient for anyone with basic Powershell knowledge and as time allows I’ll try and improve both the documentation in Codeplex and within the code itself.

I ran into this post whilst browsing Powershell.Org, for useful tidbits for improving my Powershell skills. If you’re starting to write modules or functions, then you have probably started to think about validating inputs. For a few modules I wrote big chunks of code to try and catch invalid inputs. Now, having read the following article, I have slapped myself silly for not finding out about Powershell’s validation capabilities earlier..

http://learn-powershell.net/2014/02/04/using-powershell-parameter-validation-to-make-your-day-easier/

I’ve come to realise with Powershell that if ¬†something seems complicated then you’re probably doing it wrong.