Integrate Chef with Commander

You can integrate Chef's automation software with Commander to automate repetitive configuration tasks for provisioning VMs.

Requested VMs can be automatically deployed and configured as Chef nodes. You can create workflows that specify the configuration for a node in the Chef server. You can also set up automatic VM decommissioning driven by expiry date or by a change request process, including removal of the node configuration from the Chef server.

If you have integrated with the Chef IT automation system, a Chef tab appears in the Infrastructure and Applications views for any VMs identified as Chef nodes. This tab displays recipe and role information.

If the tab contains no information, or if you want to retrieve new information, click Refresh Chef Information.

You can integrate multiple Chef servers or multiple Chef organizations with Commander, depending on how you're using Chef. This multi-tenant support extends throughout Commander's automated provisioning, including the service catalog, request forms, workflows, and workflow variables. For example, if you have one Chef organization for Linux and one for Windows, you can add both of these as separate integrations, so that Commander is aware of the two distinct sets of users, cookbooks, recipes, and roles. You can set the Chef organization in the service catalog blueprint, which means that you can easily configure the service definition for use by a specific tenant.

You can specify the Chef environment (such as development, testing, staging and production) for a component in the service catalog blueprint. Workflow variables allow you to return the specified environment. Both Commander and Service Portal users can view the intended Chef environment for a Chef node, along with its roles and recipes.

Integrating Commander with Chef allows you to:

• Set default values for environments and run-lists in the service catalog.
• Optionally, present Chef configuration choices to end users when requesting new services.
• Configure Chef-specific workflow steps for common functions such as creating nodes, bootstrapping nodes, supplying roles and recipes for node configuration, and deleting nodes during decommissioning.
• Access VM configuration, metadata, and request form data when configuring a node in the Chef server.
• Identify Chef nodes and view their configuration in Commander and the Service Portal.
• Synchronize with the Chef server on a configurable schedule.

Overview of tasks

The tasks involved in integrating with Chef are:

1. Add System credentials for the Chef server
2. Add a Chef server or Chef organization to Commander
3. Configure Chef information in the service catalog
4. Optional: Allow users to set Chef roles and recipes on the request form
5. Optional: Allow users to set the Chef environment on the request form
6. Edit the example completion workflows for new requests
7. Set up a decommissioning workflow for Chef nodes
8. Optional: Adjust the schedule for Chef synchronization

To set this up, you must use Chef 12.x (either hosted or installed).

Add system credentials for the Chef server or Chef organization

Chef integration requires an RSA key in PEM format. When you create credentials for integrating Chef with Commander, you must have a .PEM file to authenticate with.

RSA public-key pairs are used to authenticate the Chef-client with the Chef server every time a Chef-client needs access to data that's stored on the Chef server. This prevents any node from accessing data that it shouldn't and ensures that only nodes that are properly registered with the Chef server can be managed. Each client has its own RSA key.

As shown in Edit the example completion workflows for new requests, when you use the Configure Chef workflow step to create a Chef node, Commander creates a private RSA key for the client. You can then copy this key to the target VM in .PEM format with a Copy File workflow step.

For each Chef server or Chef organization you want to integrate with Commander, follow this procedure:

1. On the Credentials page, click Add.
2. In the Add Credentials dialog, from Credential Type, choose RSA Key.
3. Enter the credentials for an account that Commander can use to connect to the Chef server. You must use a .PEM file to authenticate, so you must paste the contents of the .PEM file into the RSA Key field.
4. Provide a description, like "Chef", to serve as a memory aid to administrators when configuring tasks requiring credentials.
5. From Category, choose System Credentials.
6. Click OK.

Add Chef server or Chef organizations to Commander

For each Chef server or Chef organization you want to integrate with Commander, follow this procedure:

1. Click the Integration tab.
2. On the Integration page, click Add > Chef Server.
3. In the Host/IP field, type the host name or IP address and port number for the Chef server. The default port is 443.

   You must use Chef 12.x (either installed or hosted).

4. From Credentials, choose the Chef credentials you created in the previous procedure.
5. The Node Name field allows Commander to match any VMs currently managed by Commander to nodes registered with the Chef server. The field is prepopulated with the variable #{target.dnsName}, which returns the DNS name of the VM. If you have a different naming convention for Chef nodes, enter it.

   Commander uses variable substitution to identify Chef nodes in your infrastructure. Commander obtains a list of registered node names from the Chef server then compares them with VM names in your infrastructure. When there are matches, it marks the VM as a Chef node.

6. In the Organization field, type the name of the Chef organization you want to manage with Commander.
7. The Display Name field is prepopulated with the value from the Organization field. If you want to use a different display name for this integration, enter it.
8. Click Test to test the connection.

   Commander tries to establish a connection to the specified server. This test doesn't verify the Node Name.

9. Once you see a Success message, click OK to save the configuration.

After you click OK, Commander creates and runs the Chef Synchronization task. This task retrieves the currently configured environments, roles and recipes from the Chef server. It also matches any VMs currently managed by Commander to nodes registered with the Chef server.

You can double-click the Chef Synchronization task in the Tasks tab at the bottom of the Commander console to view the associated events. The following image shows that Commander found Chef nodes, recipes, roles, and environments:

This task is scheduled to run nightly. See Synchronizing with Chef and Adjusting the Synchronization Schedule below for more information.

Configure Chef information in the service catalog

You can configure Chef information in the service catalog. You can then use variables to pass the default values to a completion workflow that instructs the Chef server to apply the configuration to requested VMs (as explained Edit the example completion workflows for new requests later in this topic).

If you want to allow users to specify Chef information on the request form (as discussed in Optional: Allow users to set a Chef run-list on the request form), the defaults you select in this procedure appear as defaults on the form.

The following are ideas of how you might set up your service catalog blueprint to match your requirements:

• If you don't want to allow users to make changes on the form, configure the run-list on the Chef tab in the catalog blueprint, and don't add any Chef elements to the Form tab.
• If your requesters are developers familiar with Chef, add the Chef Run-List control to the Form tab in the catalog, so that they can configure their own run-list. You can configure defaults on the Chef tab of the service catalog blueprint if you like.
• If your requesters just need to specify one or more additional applications for a requested service, add the simpler Chef Roles or Chef Recipes elements to the form.

To configure Chef information in the service catalog:

1. Click the Catalog tab.
2. On the Catalog tab, locate the service in the list and click Edit.
3. Go to the component's page in the wizard. If you have integrated with Chef, you should see a Chef tab.
4. Click the Chef tab.

   This page displays the Chef organizations, environments, roles and recipes that are retrieved from the Chef server.

5. If you have added multiple Chef servers or organizations, select a server or organization from the Chef Organization menu. Otherwise, the Chef organization is displayed as a read-only value.
6. Choose an environment from the Chef Environment menu.

   The roles and recipes for the selected environment are displayed.

7. Select one or more default roles and recipes from the Available Roles and Recipes pick-list.

   Ctrl-click to select multiple roles and recipes. Use the arrow buttons between the lists to move your selections into the Current Run-List. Then, use the arrow buttons to the right of Current Run-List to order the roles and recipes properly.

   If you don't select default roles and recipes, no defaults are applied.

8. Click Finish. Or, if you want to allow users to configure roles and recipes when requesting this service, continue to the next section.

Optional: Allow users to set a Chef run-list on the request form

If you want to allow users to configure a Chef run-list or choose Chef roles and recipes when requesting a new service, edit the service catalog entry. In this example, you want users to specify the Chef run-list for new VMs. You can then use the run-list configured on the form as input to a VM completion workflow or command workflow (which we'll create next).

• If you want to customize the list of recipes or roles on the form (for example, if you want to display "friendlier" role and recipe names), you can create a list-type custom attribute for use on the form instead of using the Chef Recipes and Chef Roles form elements.
• The Chef Run-List form element can't co-exist with either the Chef Recipes or the Chef Roles form element.

To allow users to set a Chef run-list on the request form:

1. Click the Catalog tab.
2. Continuing from the previous section, in the Service Catalog blueprint, click the Form tab.
3. In the Toolbox at the right side of the page, under Chef, click the Chef Run-List form element.

   The run-list is added as a service request form component.

   

4. In the newly added component, optionally enter a custom display label for the form element.
5. From the Selectable Values list, Ctrl-click all values you want to allow users to select.
6. Do one of the following:
• If you want users requesting a service to be able to override the defaults you specify, enable Override Default Values.
• If you don't want users requesting a service to be able to override the defaults you specify, clear Override Default.
7. To force users to select one or more values, enable Required.
8. Click OK and Save.
9. If you want to predefine the Chef environment and/or allow requesters to specify the environment, continue to Optional: Allow users to set the Chef environment on the request form.
10. To test your form, go to Configuration > Self-Service > Forms.
11. Select the Default Service form in the list and click Preview. In the list of services, select the service you configured for Chef.

Optional: Allow users to set the Chef environment on the request form

The Configure Chef workflow step allows you to specify the Chef environment. If you want to allow requesters to select the Chef environment on the form, you need to create a list-type custom attribute and add it to the service catalog entry.

To allow users to set the Chef environment on the request form:

1. Continuing from the previous section, in the Service Catalog, click the Attributes tab.
2. Click Add Attributes.
3. At the bottom of the Add Attributes dialog, click Create New Attribute.
4. In the Configure Custom Attribute dialog, enter a name for the new custom attribute, for example, "Chef Environment".
5. From Type, choose List.
6. From Applies To, choose Form to create a custom attribute to capture information required only during the service request process (form attributes are not attached to deployed services).
7. Clear the Edit in Service Portal option, then click Next.
8. On the Configure Attribute page, enter a comma-separated list of values that users can select from a menu list for the attribute. For example: "development,production,staging,testing", click Add, then Finish.
9. In the Add Attributes dialog, the new custom attribute is selected. Click OK.
10. On the Attributes tab, the new attribute is displayed. Optionally choose a default environment.
11. If you want to allow requesters to specify the Chef environment, click to the Form tab.
12. The new custom attribute is listed in the Toolbox at the right side of the tab. Click the new attribute to add it to the form.
13. Locate the attribute that was added to the Chef Environment form, and click Edit on the Chef Environment form element and enable the Required checkbox, if a value for this form element is required.

    If you did not specify a default environment on the Attributes tab, you should enable Required.

14. Click OK, and click Finish to save the catalog entry.
15. To test your form, do either of the following:
    • Go to Configuration > Self-Service > Form Designer. Select the Default Service form in the list and click Preview. In the list of services, select the service you configured for Chef.
    • Sign in to the Service Portal and request the Chef service you just configured.

Edit the example completion workflows for new requests

Commander includes two example Chef workflows—one for Linux and one for Windows. These examples create the Chef node, install and configure the Chef client on the VM, and configure the Chef server. You must customize these example workflows. And once you've customized a completion workflow, you can copy it and make modifications for other operating systems.

Prerequisite: Ensure that the VM's networking and host name are properly configured, either through a customization spec or through workflow steps. For more information, including guidance on credentials, see Configure OS Networking Through Workflow Steps.

To edit the example completion workflows for new requests:

1. Click the Completion tab.
2. Select either the Chef for Linux Example or the Chef for Windows Example and click Edit.
3. On the Name page, customize the name if you want.
4. On the Steps page, the following steps are preconfigured:

   

5. The Wait for VM to Be Ready step waits for the service to obtain an IP address and DNS name.

   

   For Windows VMs, it may take longer than the default 300 seconds (five minutes) to obtain an IP address and DNS name. You may want to use two steps: one that waits for guest OS customization to complete, and one that waits for IP address and DNS name.

6. The Create Node step creates a Chef node, using a run-list by default. This step also creates the private RSA key for the chef-client. We'll access this key in a later step.
   • Organization: By default, the organization is determined automatically.

   The Automatic setting only works with a single Chef integration. If you have integrated multiple Chef servers, or if you have separately integrated multiple Chef organizations, you must select an organization from the drop-down list.

   • Action: By default, this step uses a run-list to configure the node. If you want to configure the node using a list of roles and recipes instead, select Create Node (Roles and Recipes) instead.
   • Node: By default, this field is prepopulated with the variable #{target.dnsName}. Edit this value if required.

     Commander uses variable substitution to identify Chef nodes in your infrastructure. Commander obtains a list of registered node names from the Chef server, then compares them with VM names in your infrastructure. When there is a match, it marks the VM as a Chef node.

   • Environment : If you added the Chef environment to the request form as a custom attribute (as described above), enter the following variable in the Environment field:

     #{target.settings.customAttribute['attribute name']}

     For example, if you created a custom attribute called Chef Environment, enter the following variable:

     #{target.settings.customAttribute['Chef Environment']}

     Otherwise, leave the default value, #{target.settings.chef.environment}.

     In any text field that supports variables, click  to open the script editor and select variables for the current context.

   • Recipes and Roles : If the Action is Create Node (Roles and Recipes), these fields are prepopulated with variables, to return the values selected on the form, or in the service catalog, if not selected on the form. Edit this value if required.
   • Run-List : If the Action is Create Node (Run-List), this field is prepopulated with a variable that returns the run-list configured on the form, or in the service catalog, if not selected on the form. Edit this value if required. This field accepts a comma-separated list of roles and recipes, in Chef run-list format. For example:

     role[linux],recipe[logrotate],recipe[logrotate::global]

     The RSA key is stored at the following location, so that you can access it later if required for troubleshooting:

     #{system.directory}\temp\#{target.chefNodeName}.key

     where #{system.directory} is the location of the Tomcat directory in your Commander installation.

     

7. The Install Chef Client step installs the chef-client on the node.
   • Credentials: Select credentials from the drop-down menu or click Add Credentials. On Windows, Administrator credentials are required. See Workflow Steps Reference for credentials guidance.
   • Command Line: Enter the install command that you want to run.

   Linux example:

   curl -L https://www.chef.io/chef/install.sh | bash -s -- && mkdir /etc/chef

   Windows example:

   msiexec /qn /i https://opscode-omnibus-packages.s3.amazonaws.com/windows/2008r2/x86_64/chef-client-12.3.0-1.msi ADDLOCAL="ChefClientFeature"

   

8. The Copy client.rb step loads the client.rb file, which is used to specify configuration details for the chef-client.

   When you integrate a Chef server or organization, an example configuration file is saved to:

   #{system.directory}\public\$org-client.rb

   where #{system.directory} is the location of the Tomcat directory in your Commander installation, and $org is the name of the Chef organization.

   For example, if you integrated a Chef organization named "dev", the file would be located here:

   C:\Program Files\Embotics\Commander\tomcat\public\dev-client.rb

   You can customize this configuration file. The default file looks similar to the following:

   log_level        :auto
   log_location     STDOUT
   chef_server_url  "https://chef.example.com:443/organizations/dev"
   ssl_verify_mode  :verify_none;

   Do the following in the details section:

   • Credentials: Choose the appropriate credentials from the list or click Add Credentials. See Workflow Steps Reference for credentials guidance.
   • Source: Specify the located of the client.rb file. Variables are used to specify the location of the client.rb file, as explained above:

     #{system.directory}\public\#{target.settings.chef.organization}-client.rb

   • Destination: Customize the path to the chef-client installation on the target VM if necessary.

     

9. The Copy client.pem step uses the output from the Create Node step, the RSA private key, to create a .PEM file and copy it to the target VM:
• Credentials: Choose the proper credentials from the menu. See Workflow Steps Reference for credentials guidance.
• Source: The following path to the RSA key is prepopulated:
  #{system.directory}\temp\#{target.chefNodeName}.key
  where #{system.directory} is a variable that returns the location of the Tomcat directory in your Commander installation.
• Destination: Customize the path to the chef-client installation on the target VM if necessary.



10. The final Run Chef Client step runs the chef-client:
    • Credentials: Select credentials from the  menu. See Workflow Steps Reference for credentials guidance.
    • Command Line for Linux: Customize the following if necessary:
      chef-client -l error
    • Command Line for Windows: Customize the following if necessary:
      c:\opscode\chef\bin\chef-client.bat -l error

    

11. Click Next.
12. On the Assigned Components page, select the service components that you want to assign to the workflow.
    
13. Click Next.
14. On the Summary page, review the information and click Finish.

Workflow errors are written to the workflow step comment log. See Troubleshooting and Tracking Workflows for more information.

For more details on completion workflows, see Completion Workflows.

Once you've set up one completion workflow, you can copy it and make modifications for other operating systems. Select the workflow on the Workflows tab and click Copy.

Set up decommissioning workflows for Chef nodes

This example creates a completion workflow that instructs the Chef server to delete the Chef node, power off the VM and delete the VM from disk. You can also create a command workflow for decommissioning, using the same steps and actions.

The example completion workflow uses the following steps:

• Delete Node
• Power Off
• Perform Remove Action

To set up a decommissioning workflow for Chef nodes:

1. Click the Completion tab, then click Add.
2. On the Name page, provide a name such as "Decommission Chef Node", and from Apply this Workflow, choose after a Change Request is fulfilled, then click Next.
3. On the Steps page, add a Configure Chef step with a Delete Node action.

The Configure Chef step uses the credentials you created for integration with the Chef server.

Choose Add > Guest OS > Configure Chef, then in the details section, do the following:

• Change Step Name to "Delete Node".
• Organization is set to Automatic by default.

Unless the node was created by Commander, the Automaticsetting only works with a single Chef integration. If you have integrated multiple Chef servers, or if you have separately integrated multiple Chef organizations, you can must select a listed organization.



4. Next add a Perform Power Action step with a Stop action that will execute only on running VMs.

   Choose Add > Perform Power Action, then in the details section, do the following:

   • From Step Execution, choose Execute when conditions are met, click Edit, enter the following, then click OK:

     #{target.state} -eq "Running"

     

     For more information on conditional steps, see Making Workflow Steps Conditional.

5. Now add a Perform Remove Action step with a Delete from Disk action.

   Choose Add > Perform Remove Action, then in the details section, in the Step Name field, type "Delete from Disk", then use the default settings.

6. Click Next.
7. On the Assigned Forms page, select your Decommissioning form.
8. Click Next and Finish.

   Now when a decommissioning change request is fulfilled, this workflow will run.

What happens next

When a user requests the service you configured with Chef information, the VM will be automatically deployed and the completion workflow will run on the deployed VM. Once the Chef client on the deployed VM connects to the Chef server, the client will download its configuration and apply it to the VM.

View Chef node configuration for a VM

When Commander identifies a VM as a Chef node, a Chef tab is added to the Guest OS Details section for the VM.

The name displayed on the Chef tab is the name of the node found on the Chef server.

Initially, the Chef tab contains no information. Click Refresh Chef Information to retrieve the intended role and recipe configuration for this node from the Chef server.

Once you have retrieved the configuration for a Chef node in Commander, the information appears for Service Portal users. Service Portal users see the Refresh Chef Information button for VMs identified as Chef nodes as well.

Synchronize with Chef and adjust the synchronization schedule

When you integrate with a Chef server, Commander creates and runs a Chef Synchronization task, to retrieve the currently configured roles and recipes from the Chef server as well as match any VMs currently managed by Commander to nodes registered with the Chef server.

The task is scheduled to run nightly at 12:30 a.m.

To synchronize manually:

• On the Scheduled Tasks page, select the Chef Synchronization task and click Run Now.

  Depending on how you use Chef and how much churn there is in your Chef environment, you may want to synchronize less frequently, synchronize once and then disable the task, or not synchronize at all.

To disable the task or change its schedule:

1. On the Scheduled Tasks page, select the Chef Synchronization task and click Edit.
2. To disable the scheduled task, uncheck the Enabled checkbox.
3. To change the schedule, edit the Frequency and/or the Recurrence.

   It's not possible to delete this scheduled task, but the task is automatically deleted if the Chef server integration is removed.

Disable or remove Chef servers

Disabling a Chef server makes the server unavailable for connections but saves all of your settings, meaning that you can return to the configuration dialog later and re-enable it. Disabling a server also disables the scheduled Chef synchronization task for that server. The Chef tab on the VM's Guest OS Details section remains visible, but users won't be able to refresh Chef information. The settings on the Chef tab and the Form tab in the service catalog are also preserved.

Removing a Chef server clears the settings, meaning that you must reconfigure all of the settings if you want to reintegrate later. Removing a Chef server also removes the scheduled Chef synchronization task and the Chef tab on the VM's Guest OS Details section.

To disable a Chef server:

1. Click the Integration tab.
2. On the Integration page, locate the Chef server and click Edit.
3. Clear the Enabled checkbox and click OK.

To remove a Chef server:

1. Click the Integration tab.
2. On the Integration page, locate the Chef server and click Remove.
3. Click Yes to confirm the change.