Install Nutanix CE on an AMD Ryzen CPU

What’s the issue?

Nutanix CE requires an Intel CPU according to Nutanix. (portal.nutanix.com) Although it’s not supported you can modify an installation to run on a modern AMD CPU. I’m using an AMD Ryzen 3700X system running VMware Workstation 15.

You can probably modify this config to run on bare metal just by changing the minimum_reqs.py to allow AMD.

System Used

• AMD Ryzen 3700X
• 32 GB 3200Mhz RAM
• Windows 10 1909
• VMware Workstation 15
• Nutanix CE 2019.11.22 image

Thanks

• The VMware part of this guide is made possible by the work of Tim Smith and his post here (tsmith.co)

Get Started

• Download the “Disk Image-based Full Install” from here (next.nutanix.com)
• Extract ce-2019.11.22-stable.img from ce-2019.11.22-stable.img.gz. I used 7-Zip.

Create the Nutanix CE virtual machine

• Create a new folder for your vm, I called mine nutanix
• Move ce-2019.11.22-stable.img into the folder
• Rename ce-2019.11.22-stable.img to ce-flat.vmdk
• Create a new file called ce.vmdk and insert the following:

The ce.vmdk disk descriptor file, more information here (kb.vmware.com)

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=4a23b86a
parentCID=ffffffff
createType="vmfs"
  
# Extent description
RW 14540800 VMFS "ce-flat.vmdk" 0
  
# The Disk Data Base
#DDB
  
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "905"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "39ab32063800e361c1c248034a23b86a"
ddb.uuid = "60 00 C2 91 19 55 99 b4-0c 1e 38 af 74 3f 10 2d"
ddb.virtualHWVersion = "14"

• Open VMware Workstation and create a new virtual machine with the following specs:
  • 1 vCPU, 4 Cores
  • 16 GiB RAM
  • Virtualize Intel VT-x/EPT or AMD-V/RVI enabled (see fig 2.1)
  • Attach the ce.vmdk as the first hard disk, select SATA as the bus
  • Add a new 250 GiB disk on an SSD backed volume, select SATA as the bus
  • Add a new 500 GiB disk, select SATA as the bus

fig 2.1:

• Start her up

AMD Specifics

• Once the system is booted, login with root and nutanix/4u
• Edit the minimum_reqs.py

code:

nano -c /home/install/phx_iso/phoenix/minimum_reqs.py

• Find line 52, replace vmx with svm
• Find line 70, replace Intel with AMD :)

fig 2.2:

VMware Specifics

• Modify the capabilities xml file:

code:

nano /var/cache/libvirt/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml

• Delete the line pc-i440fx-rhel7.2.0 near the very bottom (CTRL+K)
• Edit the line containing pc-i440fx-rhel7.3.0 modify to pc-i440fx-rhel7.2.0

fig 2.3:

• Modify the CVM default.xml

code:

nano /home/install/phx_iso/phoenix/svm_template/kvm/default.xml

• Add <pmu state='off'/> to the <features> section

I believe this is disabling the “Performance Monitoring Unit” (linux-kvm.org) in libvirt

fig 2.4:

Install Nutanix

• type exit to go back to the login screen, login with install no password, then follow the instructions

Troubleshooting

• If the VM doesn’t boot and errors with dracut-initqueue timeout complaining it can’t find disk by UUID. Make sure your disks are all set to SATA on the bus
• If the CVM won’t start after running install make sure you made the relevant VMware specific modifications.
• If the installer won’t run complaining Intel VT-x is not running, make sure you have nested virt enabled on the vCPU - see fig 2.1. Also make sure you made the right changes to the minimum_reqs.py file.

Written with StackEdit.

Replace the MS Advanced Threat Analytics (ATA) Center Certificate

Foreword

This guide is based on the Microsoft Document but goes into a little more detail and should be clearer, you should review the Microsoft guide as well as this one.
https://docs.microsoft.com/en-us/advanced-threat-analytics/modifying-ata-center-configuration
THIS MUST BE DONE BEFORE CERTIFICATE EXPIRES!!
I am purposely not using auto enrolment. This may generate a new thumbprint if the certificate auto renews, causing all the gateways to stop talking to the ATA Center server. This process must be done manually before the certificate expires each time!
This guide assumes you have a PKI infrastructure in your domain. The certificate you generate must be trusted by the gateway for this to work otherwise the gateways will disconnect from the ATA Center.
In this guide, I will:

• Add a second server certificate
• Update all the gateways so they recognise the new certificate
• Switch to the new certificate
• Update all the gateways again so they only use the new certificate.

Replacing the certificate

• Check the existing certificate in the management panel. Currently all the gateways only have this certificate pinned in their configuration and will only talk to the Center if it’s using this certificate.

Generating the New Certificate

• Log into the ATA Center Server and open “Manage Computer Certificates”
• Open Personal > Certificates
• Right click in the right pane
• Select All Tasks > Advanced Options > Create Custom Request.

• Follow the enrolment wizard and select web server certificate template

• On the Certificate Information screen, expand details and click properties

• Fill out the form, include all the relevant details including alternative names

• Deselect Microsoft DH Provider in the Private Key tab and change the Key size to 2048 bits

• Save the CSR somewhere handy

• Copy the CSR to your issuing CA
  
• Run the following certreq command to generate the certificate
  certreq -submit -config “SERVER1\ADCS Issuing CA-1” server1.csr server1.cer
  
• Copy the resulting files back to the ATA Center server
  

Importing the Certificate

• Open the certificate, note the thumbprint, then install the certificate into the local computer, Personal store

Replacing the Certificate in ATA Center

• Log into ATA Center web console
• Configuration > Center
• Select the new certificate, check the thumbprint matches the newly installed cert

IMPORTANT STEP, DON’T CLICK ACTIVATE YET

• Click Save and wait for all gateways to sync the config – do not click activate

• Once you see the Green message that all gateways have synced the config click Activate and wait for all gateways to sync the config again.

• You can now restart the ATA Center service in Windows

When you reload the page in a fresh browser, the certificate should now be the new trusted cert.

CPU hardware assisted virtualisation exposed with PCI Passthrough on the same VM in ESXi 6.7 u2

If you have a GPU or PCI device passed through to your virtual machine, you've likely seen the error message in VMware that you cannot expose VT-d to the VM at the same time:

Failed to reconfigure virtual machine commando. PCI passthrough devices cannot be added when Nested Hardware-Assisted Virtualization is enabled.

If you need both PCI passthrough and VT-d (nested virtualisation) on the same VM in ESXi 6.7 u2, here's how:

Back up, then edit your VMX file and include the following 2 lines:

vhv.enable = "TRUE"
vhv.allowPassthru = "TRUE"

Doing so will enable experimental support for both to be passed through to the VM. Unfortunately this causes some side effects. The vmmem process appears to be running wild and taking 25% CPU on this 4-core VM (so likely a full thread.)

Since this is unsupported, I guess this is just an issue you need to live with until there is a fix (if ever.) If you happen to know one weird trick (sysadmins HATE him) to fix this please leave a comment. For now, I'll just enable in the VM when I really need to have VT-d exposed.

Error while attempting to install Windows Server 2019 App Compatibility FOD

Windows Server 2019 now comes with the ability to install various GUI binaries into a core install of windows. This feature is really useful, but if you are in an enterprise environment with WSUS you may have trouble installing.

The error I was receiving when trying to install the pack:

Add-WindowsCapability -Online -Name ServerCore.AppCompatibility~~~~0.0.1.0

Add-WindowsCapability failed. Error code = 0x8024500c

The resolution for me was to temporarily disable WSUS on the machine.

vCenter Server 6.7 - Error trying to join AD, error code [41887]

Problem:

When trying to join vCenter to the Active Directory domain for Single Sign on, the following error is displayed:

idm client exception: Error trying to join AD, error code [41887], user [username@domain], domain [domain.local], orgUnit []

Cause:

Joining Active Directory fails in vCenter due to the user attempting to authenticate using a logon name that is not their user principal name (UPN.) Check the user's account object in Active Directory Users & Computers - user logon name - in Account tab.

Solution:

Use the user's UPN to authenticate to AD and the configuration should succeed.

Creating Custom Azure RBAC Roles with PowerShell

Background

Azure has a bunch of built in roles but sometimes you need someone or something to be able to do a single task and don’t want to over permission their account.

Azure RBAC allows you to define a custom role with really granular permissions. To do this you can use PowerShell to pull one of Azure’s pre-defined templates, modify it in a text editor using JSON, then push it back as a custom defined role to assign to your user.

My example will be to create a user role that’s able to read BGP status information from the subscription. Initially I created a user and gave it the ‘Reader’ role but I hit the following error.

Take a note of the permission (Action) required, as this will be used to create the new role definition.

'Microsoft.Network/virtualNetworkGateway/getBgpPeerStatus/action'

Find a suitable role to copy

Check the list of RBAC roles by attempting to add role to a user on a subscription, resource group or resource in the portal. You can also run the following PowerShell command to get a list of all the resources in your subscription.

Get-AzureRmRoleDefinition

Once you’ve selected a template that’s similar to what you want, then get the definition and view the current permissions. I’m just using the ‘Reader’ role as it’s really simple and I only need a couple of additional permissions.

Get-AzureRmRoleDefinition "Reader"

You can now export the definition to a JSON file for editing

Get-AzureRmRoleDefinition "Reader" | ConvertTo-Json | Out-File C:\Temp\CustomReader.json

Edit the file in a text editor. You need to remove the id tag and change IsCustom to true. Change the Name, Description and add in the Actions required.

{
    "Name":  "Reader",
    "Id":  "f3323452-47a2-4221-bc0c-d66f17e14e98",
    "IsCustom":  false,
    "Description":  "Can read all monitoring data.",
    "Actions":  [
                "*/read"
    ],
    "NotActions":  [
    ],
    "AssignableScopes":  [
                          "/"
    ]
}

And here is my custom file, note I have set this to be limited to a subscription. Also, I have modified the Action to include all actions for virtualNetworkGateways.

{
"Name":  "BGP Status Reader",
"IsCustom":  true,
"Description":  "Can read BGP Status data.",
"Actions":  [
                "*/read",
                "Microsoft.Network/virtualNetworkGateways/*/action"
            ],
"NotActions":  [

               ],
"AssignableScopes":  [
                         "/subscriptions/ae015742-7715-42e3-bfbd-5beb36e89d18"
                     ]
}

Once you’re happy with the modifications, you can use it to create a custom role definition.

New-AzureRmRoleDefinition -InputFile C:\Temp\CustomReader.json

You can now assign this role definition to your user account.

And re-run the problematic command.

If you have difficulty and need to remove your custom role, you can run the following command.

Get-AzureRmRoleDefinition | 
	Where-Object { $_.isCustom } | 
	Where-Object { $_.Name -eq 'BGP Status Reader' } | 
	Remove-AzureRmRoleDefinition

Once the role is removed you can recreate it with the above commands. There is also a Set-AzureAzureRmRoleDefinition but this may require modifying your JSON.

Written with StackEdit.

One-Liner's for AD Time Synchronisation Information

After finding that some of my domain controller VMs were set to sync with the host, I had a time synchronisation issue across my domiain. Here are a couple of commands that assisted in resolving the problem.

Show all Domain Controller times with 1 sample

(Get-ADForest).GlobalCatalogs | sort | % { Write-Host "$($_): " -foregroundcolor Yellow -nonewline ; w32tm /stripchart /computer:$_ /dataonly /samples:1 | Select -Last 1}

The first column is the DC name, 2nd is local time (machine you're running the command from) and the 3rd is the DC's offset from local time

Force Sync on all DCs

(Get-ADForest).GlobalCatalogs | % { w32tm /resync /computer:$_ /nowait}

Slightly more full featured script: