Note: in addition to the Windows-specific points here, much of the operations troubleshooting guide is applicable for Windows blueprints.
If you can't get WinRM to work at all, see the notes on the Winrm4j client which includes detailed troubleshooting for basic connectivity.
User metadata service requirement
WinRM requires activation and configuration before it will work in a standard Windows Server deployment. To automate
this, Brooklyn will place a setup script in the user metadata blob. Services such as Amazon EC2's
will automatically load and execute this script. If your chosen cloud provider does not support
a similar package, or if your cloud provider does not support user metadata, then you must pre-configure a Windows image
with the required WinRM setup and make Brooklyn use this image.
If the configuration options
userMetadataString are used on the location, then this will override
the default setup script. This allows one to supply a custom setup script. However, if userMetadata contains something
else then the setup will not be done and the VM may not not be accessible remotely over WinRM.
Credentials and privileges requiring special configuration
When a script is run over WinRM over HTTP, the credentials under which the script are run are marked as 'remote' credentials, which are prohibited from running certain security-related operations. This may prevent certain operations. The installer from Microsoft SQL Server is known to fail in this case, for example. For a workaround, please refer to How and Why to re-authenticate withing a PowerShell script above.
In some cases where security-related operation are to be executed, it may require the use of
CredSSP to obtain
the correct Administrator privileges: you may otherwise get an access denied error. See the sub-section
How and Why to re-authenticate within a powershell script for more details.
WebServiceException: Could not send Message
We detected a
WebServiceException and different
during deployment of long-lasting Application Blueprint against VcloudDirector.
Launching the blueprint below was giving constantly this type of error on launch step.
services: type: org.apache.brooklyn.entity.software.base.VanillaWindowsProcess brooklyn.config: pre.install.command: echo preInstallCommand install.command: echo installCommand > C:\\install.txt post.install.command: echo postInstallCommand customize.command: echo customizeCommand pre.launch.command: echo preLaunchCommand launch.powershell.command: | Start-Sleep -s 400 Write-Host Test Completed post.launch.command: echo postLaunchCommand checkRunning.command: echo checkRunningCommand stop.command: echo stopCommand
With series of tests we concluded that on the Vcloud Director environment we were using a restart was happening about 2 minutes after the VM is provisioned. Logging in the host and search for System event of type 1074 in Windows Event Viewer, we found two 1074 events where the second one was
The process C:\Windows\system32\winlogon.exe (W2K12-STD) has initiated the restart of computer WIN-XXXX on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Upgrade (Planned) Reason Code: 0x80020003 Shutdown Type: restart Comment:
Normally on other clouds only one restart event is registered and the first time WinRM connection is made the Windows VM is ready for use.
For this particular case when you want this second restart to finish we made
waitWindowsToStart location parameter
which basically adds additional check assuring the Windows VM provisioning is done.
For example when using
waitWindowsToStart: 5m location parameter, Apache Brooklyn will wait 5 minutes to see if a disconnect occurs.
If it does, then it will again wait 5m for the machine to come back up.
The default behaviour in Apache Brooklyn is to consider provisioning done on the first successful WinRM connection, without waiting for restart.
To determine whether you should use this parameter you should carefully inspect how the image you choose to provision is behaving. If the description above matches your case and you are getting connection failure message in the middle of the installation process for your blueprints, a restart probably occurred and you should try this parameter.
Before using this parameter we advice to check whether this is really your case.
AMIs not found
If using the imageId of a Windows community AMI, you may find that the AMI is deleted after a few weeks. See Windows AMIs on AWS above.
VM Provisioning Times Out
In some environments, provisioning of Windows VMs can take a very long time to return a usable VM. If the image is old, it may install many security updates (and reboot several times) before it is usable.
On a VMware vCloud Director environment, the guest customizations can cause the VM to reboot (sometimes several times) before the VM is usable.
This could cause the WinRM connection attempts to timeout. The location configuration option
waitForWinRmAvailable defaults to
30m (i.e. 30 minutes). This can be increased if required.
Incorrectly prepared Windows templates can cause the deployment to time-out expecting an interaction by the user. You can verify if this is the case by RDPing to the in-progress deployment. It is recommended that any new Windows template be tested with a manually deployment to verify that it can be used for unattended installations and it doesn't wait and/or require an input by the user. See Windows template settings for an Unattended Installation under Known Limitations below.
Windows log files
Details of the commands executed, and their results, can be found in the Brooklyn log and in the Brooklyn web-console's activity view.
There will also be log files on the Windows Server. System errors in Windows are usually reported in the Windows Event Log -
see https://technet.microsoft.com/en-us/library/cc766042.aspx for more information.
Additional logs may be created by some Windows programs. For example, MSSQL creates a log in
%programfiles%\Microsoft SQL Server\130\Setup Bootstrap\Log\ - for more information see
WinRM Commands Fail on Java Version 8u161
This is caused by the Java bug JDK-8196491.
The error within Brooklyn will look like:
org.apache.brooklyn.util.core.internal.winrm.WinRmException: (Administrator@22.214.171.124:5985) : failed to execute command: SOAPFaultException: Marshalling Error: Entity References are not allowed in SOAP documents at org.apache.brooklyn.util.core.internal.winrm.winrm4j.Winrm4jTool.propagate(Winrm4jTool.java:257) at org.apache.brooklyn.util.core.internal.winrm.winrm4j.Winrm4jTool.exec(Winrm4jTool.java:214) at org.apache.brooklyn.util.core.internal.winrm.winrm4j.Winrm4jTool.executeCommand(Winrm4jTool.java:117) ... Caused by: java.lang.UnsupportedOperationException: Entity References are not allowed in SOAP documents at com.sun.xml.internal.messaging.saaj.soap.SOAPDocumentImpl.createEntityReference(SOAPDocumentImpl.java:148) ...
The workaround is to downgrade Java to 8u151 or similar, or upgrade to 8u192 or later.