Hardware Planning for Skype for Business 2015 Enterprise

I've been looking through the MS hardware and virtualisation guides for Skype for Business 2015 (and Lync 2013) and put together some notes to try to translate some of the best practices.

There aren't many corners that can be cut when setting this up and the hardware requirements are pretty large. The smallest enterprise set up that Microsoft details is for 10,000 seats. Microsoft won't support a configuration with hardware less than recommended which makes the hardware requirements for an organisation with under 10,000 seats seem even higher.

Most of the notes are bullet points taken from the Lync 2013 virtualisation guide, but I have put a section at the bottom to try and translate the official 2015 hardware requirements. The disk IO and capacity requirements are also listed here.

References

Lync Server 2013 Virtualization 
Capacity Planning Doc 
Lync Server 2013 Stress Testing Guide 
2015 Hardware Specifications 
RAID IOPS Calculator 


MS Assumptions in 2013 virtualisation guide

• 10k users on 3 Front Ends 
• Shared resource techniques, including processor over subscription, memory over commitment, and I/O virtualisation, cannot be used because of their negative impact on Lync scale and call quality.

General Information

• Need to do own testing with LSS – A must - Lync Server 2013 Stress Testing Guide 
• There is a section (virtualisation guide) on KHIs (Key Hardware Indicators) which should be checked during testing.
• No vMotion / Live Migrate – VMs can only be moved while powered off. 
• Mixing physical and virtual in the same pool (lync role) is not supported 
• Each server in the pool must have identical resources 
• Physical servers must be fully redundant (ie PSU, RAID) 
• Using a lower specification of server should be done with caution and it is highly recommended to use Stress and Performance tools to verify the final solution. Support will not do anything unless HW specs are met.

CPU

• Disable Hyperthreading 
• 1:1 vCPU to pCPU 
• Host must support nested page tables and extended page tables (NPT and EPT) 
• Disable NUMA spanning 
• MS Config uses 8 x HPDL560 G8, 4x E5-4650 (8c/16t) 2.70 Ghz 
• 6-10 percent overhead for VMs in guide.
• Microsoft guide does not account for NUMA home nodes at all, therefore spanning VMs across NUMA nodes or CPUs should not be an issue. (They have 12-core VMs on hosts with 8-core Intel CPUs, therefore the VM must span NUMA)

Memory

• No over commitment 
• No Dynamic Memory (or VMware ballooning - must reserve all)
• MS Config uses 8 x HPDL560 G8, 128 GiB (8 * 16 GiB)

Networking

• Must Use VMQ (Virtual Machine Queue) 
• Physical NIC segregation between Host and Guest communication.
• SR-IOV is recommended.
• MS Config uses 8x HPDL560 G8 with 4x 1 Gb NIC 
• Each host must have at least 1 dedicated NIC for lync workload.
• Lync server media workload can reach over 500 Mbps. 
• If more than 1 VM on a host size NIC accordingly, consider 10 GbE or multiple 1 GbE ie. 3x1Gb NICs Teamed. 
• Synthetic NICs in guest are preferred, also use physical NIC offloads if available.
• Legacy NIC not supported in Lync media workloads.
• Use only IPv4 -OR- IPv6 on a NIC

Storage

• Fixed / Pass Through disks (NOT Dynamic) - VHDX format 
• VM storage must be dedicated to VMs (ie. Don't use hypervisor system drive for VMs) 
• VM Checkpoints not supported 
• 'be aware' of contention between VMs 
• MS Config uses 4 * 300 GiB RAID 1 local system drives 
• MS Config uses 8 * Drive Enclosures with 12 600 GiB SAS 15k drives each (96 drives) 
• Each physical host has 1 x 600 GB array and 3 x 1.2 TB arrays each with 700 read and 350 write IOPS 
• VM: IDE for boot disk, SCSI for all other attached drives 
• ISCSI for data drives supported 
• Normal best practice for OS and binaries on OS drive, DBs and data on data drives 
• Implement MPIO for back end storage

Software (OS)

• Hypervisor – 2012R2, 2012, 2008R2, or SVVP tested platform 
• All Lync Server workloads supported in VM 
• Use VM Templates – Sysprep cannot be applied after Lync installation 
• Guest OS – 2012R2, 2012, 2008 R2 required

DR

• Front end pools in both sites, both active, both pools must be phys or virt (not mixed) 
• Admin can fail over from one site to the other 
• Both pools should handle all users

SQL

• SQL HA using SQL Mirroring + witness is recommended. Checking the 2015 hardware requirements, SQL AlwaysOn is also supported and is likely the best choice for a 2015 deployment.
• See below for supported SQL Server versions.

Supported Versions

• MS SQL 2014 Ent or Std with CU6.
• MSSQL 2012 Ent or Std with latest SP.
• MSSQL 2008 R2 Ent or Std with latest SP.
• Mirroring, Clustering and HA all supported, mirroring only in the Topology Builder.
• Active/Passive only, do not use passive for anything else.

Hypervisor Considerations

• Place VMs in the same application tier on different hosts for HA.
• Lync Server 2013 can be deployed on Windows Server 2012 Hyper-V and later technology, or any third-party hypervisor that has been validated under the SVVP. (implies 2016 is supported -- need to check with MS to be sure.) 
• Resource allocations not explicitly required unless oversubscribed - Seems an absurd statement since you cannot over commit?
• If you deploy AV on host, ensure exclusions are in place (doesn't detail exclusions.) 
• Disable virtual CD/DVD ROM 
• Lync unable to use HA or DR capabilities of Hypervisor (SRM, Hyper-V Replica)

Microsoft Virtualisation Guide VM to Host Placement

Skype for Business 2015 Hardware Requirements (Per VM)

Front End, Back End, Standard Edition and Persistent Chat

Microsoft Specification	Translation	Comment
64-Bit dual, hex-core 2.26 Ghz	12 Core
32 gigabytes (GB)	32 GB
8 * 10k rpm "with 72 GB free" --or— SSD with similar performance 2 in RAID1 6 in RAID10	232 IOPS, 72 GB 697 IOPS, 216 GB	Lync 2013 doc suggests 66/33 read/write IO profile (700/350 iops per LUN)
1 dual-port 1 Gbps NIC –or— 2 single NIC Teamed with single MAC	1 Gbps redundant	Doesn't say how teaming to be done, so NFT only seems appropriate
OS	2012R2 or 2012	Specific KBs are required, see MS hardware spec site

Edge, Standalone Mediation, Video Interop and Directors

Microsoft Specification	Translation	Comment
64-Bit dual, quad-core 2.26 Ghz	12 Core
16 gigabytes (GB)	32 GB
4 * 10k rpm "with 72 GB free" --or— SSD with similar performance 2 in RAID1 + 2 in RAID1	232 IOPS, 72 GB 232 IOPS, 72 GB	Lync 2013 doc suggests 66/33 read/write IO profile (700/350 iops per LUN)
1 dual-port 1 Gbps NIC –or— 2 single NIC Teamed with single MAC	1 Gbps redundant	Doesn't say how teaming to be done, so NFT only seems appropriate
OS	2012R2 or 2012	Specific KBs are required, see MS hardware spec site

Disk calculations are based on using 72 GB 10,000 RPM drives as detailed in the Microsoft Spec.

2 x 72 GB drives in RAID 1 gives: 72 GB capacity and 232 mixed total IOPS.

6 x 72 GB drives in RAID 1 gives: 216 GB capacity and 697 IOPS.

See the IO calculator linked for details.

You can see the guidelines are pretty detailed and once you translate them they become quite clear. Once the hypervisor servers and VMs are built, I plan to post a script to configure as per best practices.

Azure Resource Manager (ARM) Templates Getting Started Guide

Preface
I've been doing some research and development work on ARM templates recently and thought I would put together a getting started guide while I was at it.

There are some great online references for ARM template creation:

• Official Microsoft Documentation
• Azure QuickStart Template Gallery
• QuickStart Templates on GitHub
• Microsoft Resource Manager Template Walkthrough

Getting Started

The basics

ARM templates are JSON formatted files that describe Azure Resources. You can send the file off to Azure in a number of ways and the resources described will be implemented by the Service. ARM resources are deployed in parallel, so even complicated templates can be deployed quickly.

The ARM template is broken down into four sections: Parameters, Variables, Resources and Outputs. Only one section is actually required to have content - resources - the others help to make the templates more dynamic and flexible.

I use Visual Studio Code for creating and editing ARM templates as it has some excellent plugins which provide intellisense when creating resources. Using this means that you can just hit Ctrl+Space to get a hint at the possible resource properties on the fly. To set up VS Code for intellisense, the official guide is here.

You can also download templates from the Azure Portal to edit later. You can download templates from created resources, from the last page of the resource creation process, or even from entire resource groups.

Here is a really basic, valid ARM template to create a Storage Account. It's pretty rigid in that the storage account name is hard-coded, to fix that, parameters can be used to take user input.



Execution

ARM templates can be deployed in a number of ways: PowerShell, X-plat CLI and the Azure Portal are some common ways.

The following PowerShell code will create a Resource Group and deploy the template into it.

Taking input

The parameters section allows the template to take input. This input can be from the user at deployment time, from a parameters answer file or from another template. A basic parameter looks like this:

Storing Variables and Transforming input

The variables section is useful for holding regularly used values or for transforming input. For example in some of my templates, I use a resources prefix which is concatenated with resource names so that related resources can be seen at a glance.

The example includes a variable to apply a tag to a resource. In this example there is only a single resource, but in more complicated templates, the variable can be reused in all of the resources.

Producing Output

It can be useful to produce outputs from a template to get information on the resources that were created. In this example the DNS name assigned to the public IP is returned after deployment. Outputs can also be used in chained templates.

Creating multiple copies of a resource

A really powerful feature of ARM templates is the copy loop. This means you can create multiple, identical copies of specific resources in your template. For example you can create 3 copies of a VM using one code block. When you use a copy loop, you can reference the copyIndex() function in order to generate unique names for resources. In the following example, there is a simple copy loop to generate 3 public IPs. 

Using arrays in loops

Arrays can be combined with the copy loop in order to name resources with a list of predefined or runtime input names. In the example, the array is generated as a parameter and then indexed using the copyIndex() function. The count of the copy property can be set to the length of the array so that all array elements are indexed dynamically.

Running scripts in VMs using the extension resource

A nice part of Azure Resource Manager is the ability to add virtual machine extensions. One such extension is the script extension which allows the Resource Manager service to run scripts inside a virtual machine. No external access is required to the virtual machine since Azure executes the script for you. Scripts need to be hosted on an accessible web server or storage account. If there is any sensitive information in the script you should use storage keys and a key vault to access the script privately from the template. The following example just runs a publicly accessible script in the virtual machine 'myVirtualMachine'.

Linking to other templates for complex solutions

Chaining templates is super useful for creating complex solutions. For instance you can create a template to build an availability group of VMs, then call that template from another template with the copy loop and parameters to build multi-tier applications. 

I have a full implementation of this on my GitHub, the implementation will loop through several templates and create multiple availability groups and VMs. Be careful if you run this as it will spawn a lot of resources.

Here is the 'master' template which calls the other templates in the solution:

And here are all the resources it creates!

iDRAC Password Change PowerShell Script

I knocked up a quick script for changing the password on iDRAC cards.

This function can be looped through to change a local user password on a bunch of iDRAC cards for when the auditors come! :)

It requires you have the racadm command line utility installed and in $ENV:Path

Configure Hyper-V Client Tools on Windows 10 to manage non-domain Server 2012 R2

Out of the box, it's not possible to manage a non-domain joined Hyper-V 2012 R2 server from Windows 10. The following steps can be taken to enable WinRM and allow management.

Check the scripts before running them as they will install Hyper-V and the Management tools. If you've already done this you can remove those lines.

Bear in mind this uses NTLM encryption (Negotiate authentication) over HTTP. You will need to check to see if this is secure enough for you. Here is some further reading.

Windows Editions Used

Server: Windows Server 2012 R2 (I used the RTM disk, fresh installed no updates)
Client: Windows 10 Pro (I used Insider Preview 14965, fresh installed no updates)

Server Configuration Script



Client Configuration Script


Manual Client Configuration
This is an additional step to the client configuration script and must be completed for this to work

• Click Start > Run > type 'dcomcnfg' > OK
• Browse 'Component Services' > 'Computers'
• Right Click 'My Computer' and Click 'Properties'

• Click the 'COM Security' Tab, then click the 'Edit Limits' button in the 'Access Permissions' section.

• Check 'Allow' on 'Remote Access' for the 'ANONYMOUS LOGON' user group. Click OK.

• Management should now be possible.

If you want to manage the host with a non-admin account you can additionally follow this guide.

PowerShell DSC Getting Started Guide - Part 4, Partial Configurations

Part 1 - Pushing a configuration and Credentials
Part 2 - Pull Server Setup
Part 3 - Custom Scripts
Part 4 - Partial Configurations.

Partial Configurations

Sometimes, having multiple configurations for a single server could be beneficial. For example, all servers built may follow a single base configuration, with another configuration to install and setup an application.

Another example could be the 2 or more teams are responsible for the ultimate setup of a server with each team managing some portion of the configuration. Both teams can run a pull server with their own configurations.

To handle this, DSC now supports partial configurations. Partial configurations can be set up using push, pull or some combination of both.

In this example, the LCM on the client is configured to pull 2 configurations from the pull server using the ConfigurationID method. ConfigurationID should not be an easily guessable number therefore New-Guid is a good method to generate this.


The following is a basic test configuration for the machine using the 2 partial configuration names set up in the LCM configuration. It is using the same $AllNodes configuration data as the above LCM configuration.

 


In the above configuration, using the same $AllNodes configuration data, the mof files are published to the correct location on the pull server.

It is possible to configure this using the Configuration Names method also. Further reading on this topic can be found on MSDN here.

PowerShell DSC Getting Started Guide - Part 3, Custom Scripts

Part 1 - Pushing a configuration and Credentials
Part 2 - Pull Server Setup
Part 3 - Custom Scripts
Part 4 - Partial Configurations.

The Script Resource

There are lots of pre-defined modules and resources to use in a PowerShell DSC configuration, but what if none of them do exactly what you need?

Enter the script resource. The script resource is actually very similar to how full DSC resources are created, so if the script is fairly complex or will be used for more than just a few tests then I would recommend creating a full custom resource.

The syntax for this resource looks quite different from a regular resource:

There are 3 sections which need to be completed for this to work.

GetScript
This is required to return a hashtable with a single 'result' key. The value doesn't need to be anything specific as nothing is done with the output. I normally return something to do with the resource or the following:

@{ Result = "Not Implemented" }

TestScript
This is used to test to see if the work that the SetScript section implements is already done. For instance if you were creating a certificate, you could use Get-Certificate in here and return $true or $false if the certificate exists or not. This section is required and should output $false if the SetScript section needs to be run and $true if everything is in order.

SetScript
This section does the actual work and is not required to return anything. 

Below is an example script that I made that will ensure a specific ODBC connection exists. This should really be expanded so that the TestScript section checks each section of the ODBC connection to make sure the server, database and name all match the correct information. For the purpose of this example though, it should demonstrate a working script resource.


Variables are not processed at execution time, they are expanded when the .mof file is created. The special $Using:varname syntax is required to expand them at creation time.

Throughout the script, I use Write-Verbose so that the text is hidden during normal runs of the configuration.

Further information on the script resource can be found here.

Part 4 - Partial Configurations.

PowerShell DSC Getting Started Guide - Part 2, Pull Servers

Part 1 - Pushing a configuration and Credentials
Part 2 - Pull Server Setup
Part 3 - Custom Scripts
Part 4 - Partial Configurations.

Pull Server Configs - SMB / HTTP / HTTPS

A DSC Pull Server can be configured using SMB by creating a fileshare that allows the machine account of the clients to read the share. Simply dropping configurations in the correct format and configuring the LCM will allow clients to pull a configuration from the server. The share should be secured to only allow trusted admins to read and write data.

Alternate methods are HTTP and HTTPS which are identical except for TLS. HTTP configuration is not recommended since all communications are done in the clear and configuration data is sensitive. HTTP also opens the possibility for man in the middle attacks.


Configure HTTPS pull server

An HTTPS pull server can be bootstrapped using a DSC configuration with the xWebService DSC module.

The full MSDN guide can be found here.

The following .ps1 file will create and configure a web server as a DSC pull server on the machine it's run on. This will be using a self signed certificate, so should not be used in production.

Once this has completed, it should write on screen the thumbprint for the new self-signed certificate. This is required to configure the client.




Configure HTTPS Pull Client

To configure the pull client, the LCM on the machine needs to be set up to collect it's configuration from the new pull server. To do this run the following DSC configuration, then apply it with the Set-DSCLocalConfigurationManager command. Make sure to note the certificate thumbprint from the pull server creation script and add it to the client configuration below.



Then run:

  Set-DSCLocalConfigurationManager -ComputerName dscclient.example.com -Path C:\DSC\PullClient -Verbose


Create and deploy a configuration on the Pull Server

Once the client is configured, a configuration for it can be created and added to the pull server. There are 2 methods for adding configurations to a pull server. The configuration .mof files can be named the same as the GUID configured in the LCM, or the configurations can be named friendly names and referenced in the LCM. The friendly names require registration of the pull client with the server. Further information can be found here.

Configurations held on a pull server also require a checksum to be created for them. This is shown a bit further down.

The above LCM configuration uses the GUID method. When it is run it should write to the screen the GUID for the client as configured. Be careful as the script will create a new GUID if run again.

To set up a basic configuration to be pulled to the client, create and run the following configuration:



Once the mof file is created, run the following script on the pull server to move the mof file to the correct location, name it appropriately and create the required checksum file.


It's now set up for the client to pull the config. You can force the client to pull the config and watch the results with the following command:

  Update-DscConfiguration -ComputerName dscclient.example.com -Wait -Verbose 

To run the config after download run:

  Start-DscConfiguration -ComputerName dscclient.example.com -Wait -Verbose -Force -UseExisting

Part 3 - The Custom Script Resource

Additional useful commands:

Remove pending configuration after seeing the error:

Warning "LCM state is changed by non-DSC operations"

Remove-DscConfigurationDocument -CimSession 'ComputerName' -Stage Pending -Force

PowerShell DSC Getting Started Guide - Part 1, Pushing a configuration and Credentials

Part 1 - Pushing a configuration and Credentials
Part 2 - Pull Server Setup
Part 3 - Custom Scripts
Part 4 - Partial Configurations.

What is PowerShell DSC

PowerShell DSC - Desired State Configuration - is a declarative platform for provisioning Windows (and non Windows) machines with configuration information. Under the covers, there is PowerShell code which enforces the configuration on the machine.

PowerShell DSC can be used to install and configure software and enforce configuration of the Operating System.

PowerShell DSC is idempotent, which means a configuration can be run against a machine over and over and if the machine is already in the described state, then nothing will be changed.

Included in-box are 'modules' which contain 'resources' that allow installation of packages and Windows features, copying of files and folders, ensuring registry entries are set to specific values and more.

A very basic PowerShell DSC configuration, using the file resource looks a lot like a normal PowerShell function. It can also take parameter values like a function:


Running the above configuration creates a .mof file for the 'localhost' computer in the folder C:\DSC\BasicConfig. To push this configuration, execute the following command:

  Start-DscConfiguration -ComputerName 'localhost' -Path C:\DSC\BasicConfig -wait -verbose -force

During execution, DSC will ensure C:\Temp exists. Running the same configuration again will result in no change, since the folder already exists.

Omitting the -wait switch will create a PSJob and run the configuration in the background, omitting -verbose reduces the output from the configuration job and -force just tells DSC to ignore any other jobs in progress and force this one to run.



Modules, Resources and Extending PowerShell DSC

View installed resources and modules:

  Get-DscResource

Finding resources on the PowerShell Gallery can be done directly from PowerShell with:

  Find-Module

Resources can then be installed with:

  Install-Module

Quick syntax information for a resource:

  Get-DSCResource -Name User -Syntax

The PowerShell Gallery hosts a ton of available DSC resources which can be downloaded directly.
PowerShell Gallery

The PowerShell GitHub page has source code for many of the first party DSC modules. Viewing the source for a module can aid debugging a configuration considerably.
PowerShell GitHub



The Local Configuration Manager (LCM)

The LCM runs on all versions of WMF 4 and above, it is responsible for getting and applying the configuration to the machine. The configuration of the LCM can be queried with:

  Get-DscLocalConfigurationManager



Modes: Push / Pull

PowerShell DSC supports two modes of getting a configuration to a client. 

In push mode, the LCM will apply a configuration sent to it using the Start-DscConfiguration command.

In pull mode, the LCM is set up to check a server for it's configuration, download and apply it. 



Push a basic configuration to another machine

DSC Client Config

✔ WinRM needs to be set up and working as a prerequisite to this, to make this easy both machines should be a member of the same domain. 

✔ WMF 5.0 installed.

DSC Push Workstation Config

Create and execute the following .ps1 file



This will create a .mof file for dscclient.example.com and save it in C:\DSC\BasicPush.

Pushing the configuration can be done with the following command:

  Start-DscConfiguration -ComputerName 'dscclient.example.com' -Path C:\DSC\BasicPush -wait -force -verbose



Set up certificates for credential usage

For most non-trivial configurations, it's likely that credentials will be required for access to installation packages or AD for example.

Passing credentials with DSC requires a special encryption certificate to be created, with the private key installed on the remote client and the public key accessible on the workstation where the configurations are created.

Executing a configuration will encrypt the credential with the public key so that the remote machine can decrypt and use it with it's private key. 

It's worth remembering that anyone with access to the private key can decrypt the credential from the stored mof file. Mof files and private keys should be secured appropriately. Anyone with admin level access on the client will have access to the private key in the certificate store and therefore be able to decrypt the password. Any credentials used should have the minimum amount of rights to get the job done.

Credential management appears to be broken between versions 4 and 5 of the WMF. I would therefore advise installing the production release of WMF 5 on Server 2012 R2 and Windows 8.1 to ensure credentials can be encrypted and recovered properly.

To create the certificate, you can use Windows PKI and create a special template. I have detailed this process in another blog post here. 

An interesting alternative, that I haven't tried myself may be to use the xDSCUtils module to bootstrap with self-signed certs. 



Push a configuration with credentials

Prerequisites for the client

✔ Generate the certificate and add it to the local computer personal store on the client ( cert:\LocalComputer\My )

From your push workstation

✔ Run the following PowerShell script to export the certificate public key to the local machine. This will store the key in a local .cer b64 encoded file. The script will also generate 'configdata' for the push command.

  New-DscClient.PS1 -ComputerName 'dscclient.example.com'

✔ Generate the configuration .mof file with the following script

✔ Configure the LCM on the remote machine to use the correct certificate. The LCM configuration information was generated in the mof file using the LocalConfigurationManager section in the DSC-InstallApp.PS1 above. The command below will push the LCM configuration to the remote machine:

  Set-DscLocalConfigurationManager -Path C:\DSC\CredentialPush -verbose

✔ Execute the configuration on the remote machine.

  Start-DscConfiguration -Path C:\DSC\CredentialPush -verbose -Force -Wait

✔ It's worth opening the .mof file in a text editor so that you can see the encrypted credential in the file.

Continued in part 2 - Pull Server Setup

How to Create a Certificate Template for Powershell DSC Credential Encryption

PowerShell DSC credential signing requires a specific certificate type. To create this in a Windows PKI environment, I did the following:

Log into your PKI Certificate Authority server and open the Certification Authority mmc console.

Right click on the Certificate Templates folder and click manage.

Right click on the Computer template and click Duplicate.

The settings I changed are as follows:

Compatibility Tab 
  Certification Authority: Windows Server 2012
  Certificate Recipient: Windows 7 / Server 2008 R2
General Tab
  Template Display Name: DSC Signing Certificate
Request Handling Tab
  Purpose: Encryption
Cryptography Tab
  Minimum key size: 2048 bits
Subject Name Tab
  Subject name format: DNS name
Ensure Microsoft RSA SChannel Cryptographc Provider is the only selected provider
Extensions Tab:
  Application Policies: Remove all entries and add a new policy.
    Name the policy Document Encryption
    Enter the Object identifier: 1.3.1.4.1.311.80.1
  Key Usage: Click Edit and tick Allow encryption of user data

A useful thing to do at this point is to create an AD security group and add any DSC configured computers to it. Then in the security tab of the template, give the read, enroll and autoenroll permissions. This will automatically create a certificate for each machine in the group. There may be some additional configuration required for this to work which is detailed here.

Below are some screenshots of the certificate template creation