Terraform, Nutanix AHV and Windows

Deploy a Windows Template using Terraform to Nutanix AHV

If you just want the template, check the github repo.

This post assumes you already have a Windows image for your desired Server OS. The Autounattend.xml included here should work fine on Server 2016 through 2022 and will need some tweaks for a client OS.

I build my AHV images using Packer and have one for each OS type. The Terraform template assumes your image is using UEFI boot, but could easily be modified for BIOS boot. You will need to modify main.tf to add your image name to the map, or copy your template and name it the same as mine - efi-rf2-2022-packer - for instance.

Your image must be sysprepped. 

# Select the correct image and product key
variable "images" {
  type = map(any)
  default = {
    "2016"      = "efi-rf2-2016-packer"
    "2016-core" = "efi-rf2-2016-core-packer"
    "2019"      = "efi-rf2-2019-packer"
    "2019-core" = "efi-rf2-2019-core-packer"
    "2022"      = "efi-rf2-2022-packer"
    "2022-core" = "efi-rf2-2022-core-packer"
  }
}

The image name in Nutanix for my 2022 server template is efi-rf2-2022-packer so if you just want to quickly test, replace the image you already have in the map, e.g… "2022" = "myServerTemplate"

The os variable in terraform.tfvars maps into the table above (and also the product key map.)

Breaking down the template

Taking a look at the main.tf file, the first section tells terraform that we need the Nutanix module

terraform {
  required_providers {
    nutanix = {
      source = "nutanix/nutanix"
    }
  }
}

The next part sets up the connection to the Nutanix endpoint - Prism Central or Elements - pulling variables from our terraform.tfvars. Once connected, the data section gets the UUID of the first cluster. Since I’m connecting directly to Prism Elements, that’s all that is needed for me. If you’re connecting to Prism Central you may need to include additional variables for the cluster.

# Connection Variables
provider "nutanix" {
  endpoint     = var.nutanix_endpoint
  port         = var.nutanix_port
  insecure     = var.nutanix_insecure
  wait_timeout = var.nutanix_wait_timeout
}

# Get Cluster uuid
data "nutanix_clusters" "clusters" {
}
locals {
  cluster1 = data.nutanix_clusters.clusters.entities[0].metadata.uuid
}

The next section is a couple of maps which serve as a lookup table to convert the os variable into other useful variables for later use. if os is 2022, then image_name = var.images[var.os] sets the image_name to efi-rf2-2022-packer.

# Select the correct image and product key
variable "images" {
  type = map(any)
  default = {
    "2016"      = "efi-rf2-2016-packer"
    "2016-core" = "efi-rf2-2016-core-packer"
    "2019"      = "efi-rf2-2019-packer"
    "2019-core" = "efi-rf2-2019-core-packer"
    "2022"      = "efi-rf2-2022-packer"
    "2022-core" = "efi-rf2-2022-core-packer"
  }
}

# These are KMS keys available from Microsoft at:
# https://learn.microsoft.com/en-us/windows-server/get-started/kms-client-activation-keys
variable "product_keys" {
  type = map(any)
  default = {
    "2016"      = "CB7KF-BWN84-R7R2Y-793K2-8XDDG"
    "2016-core" = "CB7KF-BWN84-R7R2Y-793K2-8XDDG"
    "2019"      = "WMDGN-G9PQG-XVVXX-R3X43-63DFG"
    "2019-core" = "WMDGN-G9PQG-XVVXX-R3X43-63DFG"
    "2022"      = "WX4NM-KYWYW-QJJR4-XV3QB-6VM33"
    "2022-core" = "WX4NM-KYWYW-QJJR4-XV3QB-6VM33"
  }
}

data "nutanix_image" "disk_image" {
  image_name = var.images[var.os]
}

This just gets the nutanix subnet using the name supplied in the variable

#pull desired subnet data
data "nutanix_subnet" "subnet" {
  subnet_name = var.subnet_name
}

This is the interesting bit, the template_file section is what injects variables into the Autounnatend.xml for autologon, domain join and executes a ps1 script from your webserver for onward configuration of the system.

Right now, it’s obfuscates the local admin password, but the domain join password is added in plain text. There is an option to obfuscate this using the AccountData xml section instead, but it’s still easily reversible.

On security, this xml file is added to a CD-ROM image attached to the new VM, and will be left mounted. You should have your first run script eject the CD-ROM so that the passwords contained in it are removed.

If you’ve not used Terraform before, then the state files that it creates are also considered to be sensitive since they will contain usernames, passwords etc. If you’re planning to do this in production then investigate using remote state.

# Unattend.xml template
data "template_file" "autounattend" {
  template = file("${path.module}/Autounattend.xml")
  vars = {
    ADMIN_PASSWORD       = textencodebase64(join("", [var.admin_password, "AdministratorPassword"]), "UTF-16LE")
    AUTOLOGON_PASSWORD   = textencodebase64(join("", [var.admin_password, "Password"]), "UTF-16LE")
    ORG_NAME             = "Terraform Org"
    OWNER_NAME           = "Terraform Owner"
    TIMEZONE             = var.timezone
    PRODUCT_KEY          = var.product_keys[var.os]
    VM_NAME              = var.vm_name
    AD_DOMAIN_SHORT      = var.domain_shortname
    AD_DOMAIN            = var.domain
    AD_DOMAIN_USER       = var.domain_user
    AD_DOMAIN_PASSWORD   = var.domain_pw
    AD_DOMAIN_OU_PATH    = var.ou_path
    FIRST_RUN_SCRIPT_URI = var.first_run_script_uri
  }
}

Next is the main virtual machine resource section, VM settings are hard coded here but can be parameterised easily.

# Virtual machine resource
resource "nutanix_virtual_machine" "virtual_machine_1" {
  # General Information
  name                 = var.vm_name
  description          = "Terraform Test VM"
  num_vcpus_per_socket = 4
  num_sockets          = 1
  memory_size_mib      = 8192
  boot_type            = "UEFI"

  guest_customization_sysprep = {
    install_type = "PREPARED"
    unattend_xml = base64encode(data.template_file.autounattend.rendered)
  }

  # VM Cluster
  cluster_uuid = local.cluster1

  # What networks will this be attached to?
  nic_list {
    subnet_uuid = data.nutanix_subnet.subnet.id
  }

  # What disk/cdrom configuration will this have?
  disk_list {
    data_source_reference = {
      kind = "image"
      uuid = data.nutanix_image.disk_image.id
    }
  }

  disk_list {
    # defining an additional entry in the disk_list array will create another.
    disk_size_mib   = 10240
  }
}

The last bit just outputs the IP of the new machine

# Show IP address
output "ip_address" {
  value = nutanix_virtual_machine.virtual_machine_1.nic_list_status[0].ip_endpoint_list[0].ip
}

And for completeness, here’s the Autounattend.xml template

Building a VM

To get started, install terraform and make sure it’s available in your path. I’m using Windows / PowerShell for this but it shouldn’t matter if you’re on another OS for your build host.

Clone the repo

git clone https://github.com/bobalob/tf-nutanix

Modify the example terraform.tfvars file with your settings.

Set environment variables for usernames/passwords so they aren’t stored in plain text! Make sure to escape any PowerShell specific characters like $ and `. If automating this step, consider using a password vault solution like Azure Key Vault.

Make sure you’re in the right folder and run terraform init. This will download the required plugins and get you ready to run.

If this looks okay, run terraform plan

Again, if this looks OK you can apply with terraform apply

And here’s the script from the webserver executing

Have fun!

Written with StackEdit.

Packer for Nutanix AHV Windows Templates

Packer for Nutanix AHV

Packer automates the creation of virtual machine images. It’s quite a bit simpler than SCCM to set up if you just want basic, up to date images that you can deploy virtual machines from. Packer uses ‘builders’ to create virtual machine images for various hypervisors and cloud services. Unfortunately, there isn’t yet a builder for Nutanix AHV. AHV is based on KVM for virtualisation though, so it’s possible to generate images using a basic KVM hypervisor and then upload them to the image service ready to deploy.

Since it’s not possible to create the templates natively in the platform, a helper virtual machine is needed to run KVM and build the images. In this post, I’ll go through the set up for an Ubuntu virtual machine and give a Windows Server 2019 example ready to deploy.

I used Nutanix CE 5.18 in most of my testing, but it’s also possible to run the KVM builder on a physical machine or any VM that supports CPU passthrough such as VMware workstation, Hyper-V or ESXi. If you’re running an older version of Nutanix AOS/AHV then it may still be possible with caveats. Check the troubleshooting section for more information.

Create the builder virtual machine

Create the VM in AHV

• Create VM, 2 vCPU, 4 GB, I’m using the name packer
• Add disk, ~100 GB
• Enable CPU passtrhough in the Nutanix CLI

SSH to a CVM as nutanix and run the following command

acli vm.update <VMNAME> cpu_passthrough=true

• Install Ubuntu 20.04, I chose a minimal installation
• Install XRDP if required. This script appears to configure Ubuntu for XRDP quite well. If you run this script, you should check it’s contents. I have only done this on a non-production test VM, but it appears to work well.

Install packer with the following commands. First party guide here. Make sure you update with the latest version, the URL here is just an example, but it is the version I used.

sudo apt update
sudo apt -y upgrade
wget https://releases.hashicorp.com/packer/1.6.5/packer_1.6.5_linux_amd64.zip
unzip packer_1.6.5_linux_amd64.zip
sudo mv packer /usr/local/bin/

Run the packer binary to make sure it’s in the path and executable

packer

Download the packer windows update provisioner and install per the instructions.

wget https://github.com/rgl/packer-provisioner-windows-update/releases/download/v0.10.1/packer-provisioner-windows-update_0.10.1_linux_amd64.tar.gz
tar -xvf packer-provisioner-windows-update_0.10.1_linux_amd64.tar.gz
chmod +x packer-provisioner-windows-update
sudo mv packer-provisioner-windows-update /usr/local/bin/

Install qemu, vnc viewer & git

sudo apt -y install qemu qemu-kvm tigervnc-viewer git

Check you have virtualisation enabled

kvm-ok

Add your local user to the kvm group and reboot

sudo usermod -aG kvm dave
sudo reboot

I have built an example windows build on github here

Clone the example files to your local system with the following command

git clone https://github.com/bobalob/packer-examples

you will need to download the Windows 2019 Server ISO from here, the Nutanix VirtIO Drivers from here and LAPS from here.

Place the Windows installation media in the iso folder and the 2 MSI files in the files folder. They must be named exactly as in the win2019.json file for this to work. Update the json file if you have differing MSI or ISO file names. Use this as a base to build from. You can upload additonal MSIs or add scripts. Experiment with the packer provisioners.

If you run with a different ISO you will either need to obtain the sha256 hash for the ISO or just run the packer build command and it will tell you what hash it wants. Be careful here, I trusted my ISO file so I just copied the hash that packer wanted into my json and ran the build again.

If you wish to change the password for the build, change the variable in the win2019.json file and the plain text password in the Autounattend.xml file.

My folder structure looks like this:

Run packer build

cd packer-examples/win2019/
packer build win2019.json

Once the machine is built, upload the artifact from vm/win2019-qemu/win2019 to the image service in Prism

Once uploaded you can create a VM from the image. Hopefully it will have all the correct VirtIO drivers installed.

Troubleshooting

In all cases where a build fails it’s useful to set the PACKER_LOG environment variable as follows

PACKER_LOG=1 packer build win2019.json

==> qemu: Failed to send shutdown command: unknown error Post “http://127.0.0.1:3931/wsman”: dial tcp 127.0.0.1:3931: connect: connection refused

In my case this was because I had configured my sysprep command in a regular script. Since the sysprep runs and shuts the machine down, there is no longer a winrm endpoint for packer to connect to.

The issue with this is that packer attempts to cleanup once it has run the script and then run the shutdown_command. I removed my sysprep from the script and included it as my shutdown_command.

Build hangs when installing VirtIO MSI

I realised this is because the network driver installs and disconnects the network for a second causing packer to hang and not receive output from the script. Changing the build VM nic to e1000 in the json file means the NIC doesn’t get disconnected when installing VirtIO.

openjdk / java issue with ncli

System default is
but java 8 is required to run ncli

edit the ncli file in a text editor, replace

JAVA_VERSION=`java -version 2>&1 | grep -i "java version"

with

JAVA_VERSION=`java -version 2>&1 | grep -i "openjdk version"

MSR 0xe1 to 0x0 error on Ryzen system

Fix here

Essentially run the following and try again, if this fixes it, try the linked blog for the permenant fix.

echo 1 | sudo tee /sys/module/kvm/parameters/ignore_msrs

Windows build hangs the host VM or the guest

I think this is a problem in AHV 5.10, Tested working on AHV 5.18 CE. A workaround is changing the machine type to pc or pc-i440fx-4.2. Unfortunately this appears to be REALLY slow! Might be worth experimenting with different machine types. q35 is just as slow.

Update the Qemu args to include the machine type:

"qemuargs": [
    [
      "-m",
      "2048"
    ],
    [
      "-smp",
      "2"
    ],
    [
      "-machine",
      "pc"
    ]
  ]

Written with StackEdit.

Windows Process Command Line Logging

Introduction

There are many reasons to gather process launch information in a Windows environment. Troubleshooting, Security forensics, Performance Analysis to name a few. By default, Windows does not log process launch information so it will need to be configured using local or group policy.

There are some security consequences of enabling process command line logging. Some commands, including some legacy AD commands require using a password or secret key as a parameter. Ideally we wouldn’t want these secrets appearing in logs, unfortunately the the only way to exclude them is to disable command line logging which substantially reduces the usefulness of enabling process logging in the first place.

Reading the logs requires local administrator rights on the system, however this could lead to privilege escalation; for instance if a logged command line included a Domain Admin password. The usefulness in forensics though, may be more benefit than the risk of having secrets in the Windows Event log. It may be prudent to configure log forwarding to a hardened server and regular log rotation on client and server systems.

Required Policy Settings

Warning: Enabling the Audit: Force audit policy subcategory settings policy will override any basic policy settings that are configured on the system. These should be translated to the equivalent advanced audit settings if they are required.

In order for Windows to log event ID 4688 so that you can see Windows process creation events and their command lines, there are several policy settings to configure:

Enable Audit: Force audit policy subcategory settings policy in Computer Configuration\Policies\Windows Settings\Local Policies\Security Options

Enable the Audit Process Creation policy in Computer Configuration\Policies\Windows Settings\Security Settings\Advanced Audit Configuration\Audit Policies\Detailed Tracking

Enable the Include command line in process creation events setting in Administrative Templates\System\Audit Process Creation

Audit: Force audit policy subcategory settings

Audit Process Creation

Include command line in process creation events

If using Group Policy, once the policy is saved, run gpupdate /force on the targeted system. Once applied, running auditpol /get /category:* should show the appropriate audit policies being applied.

Troubleshooting

While deploying this group policy, I noticed that Process Creation was not being set when running auditpol /get /category:*. After enabling an unrelated advanced audit policy such as Account Lockout in the Logon/Logoff category, the Process Creation policy was applied. I was then able to remove Account Lockout while still keeping Process Creation. I’m not exactly sure why this issue ocurred, but I think it happened due to the order the policy elements were applied.

Reviewing the Windows Event Log

Once the above policy and audit settings are in place then the system will begin logging event ID 4866 in the System event log

Gathering Process History for a machine in PowerShell

Looking at windows event logs can be time consuming and tedious, so here is a short PowerShell script to parse the event log on a target machine for recent commands:

Script Output:

Extending with SysInternals Sysmon

While logging process creation events with the full command line is a powerful forensic tool, there is much more detailed logging available in SysInternals Sysmon. Next time I will be installing and configuring Sysmon to generate more verbose forensic logs including process creation and termination, generation of suspicious file modification and deletion logs, Registry modification and even WMI event logging.

Gathering Domain Controller Audit Policy

I’ve also written a quick script to gather the audit policy for all Domain Controllers in the domain. You can find it here.

Written with StackEdit.

Creating Windows images on Azure Resource Manager with Packer

Building from the last post about creating infrastructure on Azure Resource Manager with Terraform, I wanted to try out packer on the platform for creating Windows Server images.

Getting this running requires the following:

1. choco install -y packer - or manually download and install on windows machine
2. Set up application and permissions in Azure
3. Add the required environment variables
4. Create json file for packer
5. packer build windows-example.json

In much the same way as Terraform, you will need to create an application in Azure and assign permissions so that Packer can create resources to build your image.

Unfortunately the official packer script for auth setup doesn't work for me and the commands on the site are specific to Linux. I created a PowerShell script that will create the relevant objects in Azure for authorisation. If you want to manually create these objects you will need to follow the packer documentation here.

Once the API keys have been loaded into environment variables, I created the an example Windows build json file. An important thing to note is the winrm communicator section in the file, without it the build will fail. There is also an official packer example file here.

Packer requires a resource group and storage account to store your completed images. These can be created any way you choose but will need to be there for the build to complete successfully.


Once the file has been created, run the build with

    packer build .\windows-example.json

Packer should then build a keystore and VM in a temporary resource group, run any provisioners configured and then output build artifacts ready to use as a vm template. Once the template is created, you will be given a path to a vhd and an Azure json template for building VMs.

All scripts and files can be found on my GitHub.

Next step for this is to use Terraform to build a VM from the custom image.