Category Archives: Operations Manager

SCOM 2016 – Agent Maintenance Mode

I was pleased to present a session about “Why we want you to migrate to SCOM 2016” with Bob Cornelissen at MMSMOA in Minneapolis. During this session I presented about SCOM 2016 Agent maintenance mode.

With System Center Operations Manager 2016 we finally have our scheduled maintenance mode and also the option to directly set the maintenance mode from the agent. We do not need custom management packs/solutions anymore to get this functionality. Everything is build in :-).

When I tried the Agent maintenance mode first, I followed this blog post. I want to share some details here:

  • Open PowerShell as Administrator, otherwise the module does not work. I tried to set permissions on the registry key, which is used, but that did not help. The module itself seems to check the permissions.
    2.AgentMaintenance-AdminError.PNG
  • You need to load the module with the full path, otherwise it is not working:
    import-module “C:\Program Files\Microsoft Monitoring Agent\Agent\MaintenanceMode.dll”
  • The Syntax to start maintenance mode is this:
    Start-SCOMAgentMaintenanceMode -Duration 10 -Reason ApplicationInstallation -Comment “Test”
    4.AgentMaintenance-SetMaint.PNG
    I recommend to use the PowerShell ISE as you then also get the entries for the reason.
  • The cmdlet writes the information into the registry.
    hklm:\Software\Microsoft\Microsoft Operations Manager\3.0\MaintenanceMode\Record
    8.AgentMaintenance-RegKey.PNG
  • When you try again to set the maintenance mode and the system is not in maintenance mode yet, then you get this feedback:
    5.AgentMaintenance-SetMaint-Pending
    Last Maintenance Mode Request pending. Try afterwards.
  • There is a rule running on the agent, which checks the registry key every 5 min.
    Agent Initiated Maintenance Mode Rule
    The rule runs a PowerShell script (ReadMaintenanceModeRegEntry,ps1), which writes a OperationsManager Event Log entry with the EventID 2222. After that appears, the maintenance mode should been set successfully.
    7.AgentMaintenance-Event2222
    After the maintenance mode is set, you get this message when you try to set it again during the maintenance mode:
    6.AgentMaintenance-ProbablyMaint
  • If you want to unset the maintenance mode and set a new one, then you can delete the registry key, unset the maintenance mode in the console and then set the maintenance mode again.
  • You can also easily read the key through PowerShell and give users who normally do not have registry access the possibility to check the current maintenance mode window.
    function Get-SCOMAgentMaintenanceMode {
    $mm=(Get-ItemProperty “hklm:\software\microsoft\microsoft operations manager\3.0\maintenancemode”).Record
    $split=$mm.split(“|”)
    write-host “Duration:”$split[0]”
    Reason:”$split[1]”
    User:”$split[2].split(“:”)[0]”
    Comment:”$split[2].split(“:”)[1]”
    StartTime:”$split[3]
    }
    Get-SCOMAgentMaintenanceMode

Have fun with it!

SCOM:Veeam VMware Collector: VMware connection data processing errors

This is a short blog post to give you some alert troubleshooting details.

The alert VMware connection data processing errors from the Veeam VMware management pack (8.x) needs a collector server and a Run As Account with permissions in VCenter. Permissions are described in this article.

Alert details:
“The Veeam VMware Collector service was not able to process data it received from the VMware target. Some data could be missing in the VMware API response or it could be a parsing error. Some performance data may not be published. The exact error received from Collector: VP050 Error encountered while retrieving performance metrics for a cluster on system name. The VMware API error returned was: ‘One or more errors occurred.’.”

The alert details talk about performance metrics, which confused me.

The Veeam logs (you get them through the Veeam Enterprise Console) show these details:

[VP120] retrieveDatastores failed, A: No ‘datastore’ objects returned for VC,
Collector cannot access a datastore at the vCenter.
Then, [VP038] buildInventory failed [system name]
Cannot build topology always for system name

And after that you seeing [VP050] performanceDaemon One or more errors occurred.

Possible root causes:

1. system does not have a datastore at all
2. no permissions => re check permissions (see link above)

The permissions need to be set on the top level of the cluster not below!

To check, if the system has a datastore, you can browse the following webpage: http://system name/mob or https://system name/mob. Login with the Run As Account.

veeamcontent

Then click the content link.

veeamrootfolder

Click the link beside the rootFolder entry.

veeamchildentry

Click the link beside the childEntitiy entry.

veeamdatastore

If you see one or more datastores in the list, then the permissions are correct.

You can also click on one of the datastore links to verify.

veeamdatastoredetails

When both topics are resolved (permissions and datastore access) then the alert should resolve itself.

SCOM: Sample Maintenance Mode MP works on SCOM 2016

With all the great changes related to Maintenance Mode in SCOM 2016 you probably only miss the possibility to easily set Maintenance Mode on the agent without the need of knowing the PowerShell script details.

My old Sample Maintenance Mode management pack can help you with this also on SCOM 2016. I have imported it into my SCOM 2016 test environment and set a server into maintenance mode through it without any problem.

It was required in the past, that you deploy the files separately to the agents to have the Splash Screen available. Now I have added the files to the Visual Studio solution and deploy them to c:\it\mom\mm. The solution also adds a shortcut to the default user desktop and to the public startup folder on Windows Server 2012 and above (also applies to the corresponding client versions). I have used the examples from David Allen’s blog post.

I have posted the sample sealed mpb file on github including also the whole solution.

To adjust the solution to your needs I recommend to change the text in the file OpsMgrMM.ps1 which runs the Splash Screen.
Also you can change the target directory in the DeployableFile.ps1:

$TargetDirectory = “C:\IT\MOM\MM”

You will find both files in the Resources folder.

When you have done your adjustments, then build the solution (seal the mps) and import the mpb file into your SCOM environment. If you had a previous version installed, then you will need to remove that first.

Have fun with it!

 

OMS Management Packs

If you ever have to author a SCOM management pack, which references Microsoft OMS resources, then it would be good, if you have the management packs to reference.

I was recently searching for the Microsoft.IntelligencePacks.Performance.mp and Microsoft.IntelligencePacks.Types.mp, but could only find the xml here. Sealing this management pack did not help as I needed the original MP (with the signature from Microsoft) for my management pack.
But SystemCenterCore.com provided the information that the MP is part of the Microsoft.IntelligencePack.Core.mpb. And the last hint came from a MVP colleague (thanks Stefan!).

You can find all OMS (formerly Advisor) management pack bundles on the SCOM Management Server which is already connected to OMS. They all get downloaded to C:\Windows\Temp.

omsmbps

Don’t get confused by the names, you can use them as they are. So, for my reference, I selected the Microsoft.IntelligencePack.Core_635779185101086268.mpb in Visual Studio and all included Management Packs appeared – also the Microsoft.IntelligencePacks.Types.mp :-).

 

 

AzureRM: Move Virtual Machines to a new subscription

I have a subscription which ends soon, which forces me to migrate my resources to a new subscription. I have used the DevTestLab service to create my System Center test environment, which is a nice feature, but also has some limitations as some resources are locked. I tried to use the Azure PowerShell commands to migrate the resources or tried it directly in the console, but there were always parts which could not be migrated so the Validation failed. The other option I found is directly copy the blob storage. Here are two blog posts about it:
http://gauravmantri.com/2012/07/04/how-to-move-windows-azure-virtual-machines-from-one-subscription-to-another/
https://blogs.msdn.microsoft.com/laurelle/2015/10/01/how-to-move-azure-vm-between-subscriptions/

Both reference the old portal. That cannot be used for virtual machines which are part of a DevTestLab as they are AzureRM resources, which can only be managed in the new Azure Portal.

Both solutions need a tool to browse and copy the blob storage. I have also used the free CloudBerry Explorer for Azure Blob Storage. It reminded me a bit of the old File Commander ;-).

So to move my virtual machines, I had to find out, which storage account was used in my old DevTestLab. Open up your DevTestLab and click on the resource group name in the overview.

ResGroup.jpg

You will see all resources which belong to this resource group:

ResGroupDetails.jpg

I have two storage accounts, by selecting one-by-one, I find out which one holds the virtual machines.

storageaccount

Click on Blobs and then on Vhds.vhds

The entries with the Page blob type are the vhds, the Block blob type are the VM Status files.

Now I know the Storage account. Then I created a new DevTestLab in my new subscription and checked which resources are there. There were two storage accounts, both with blob storage assigned, but without entries. Okay. I do not have the details which one you should select, but I selected the first one in the list. Now I have the names of both storage accounts.

To reference these accounts for the copy job, I need one more thing: the Access key. To find that open up the Storage accounts service in your Azure Portal.
StorageAccounts.JPG
Select your storage account, click on Access keys and copy the first key.

You will need to do that for both storage accounts when you create your connection in CloudBerry Explorer.

So open up CloudBerry Explorer and create the connection to both of your storage accounts (old subscription, new subscription).

CloudBerry-NewAzureblobStorage.JPG
Enter a Display name, the name of the Storage account and the shared key (which you copied before).

CloudBerry.JPG

When you have entered both Storage accounts, then you can browse them. On the old storage account browse to Root\Vhds and on the new one to Root\Uploads.

Then select the vhd files one by one and click copy. A queued copy job will start.

CloudBerryUploads.JPG
The job will close as soon as it is finished.

Now you can go back to the Azure Portal and to your new DevTestLab. The next step is to create Custom Images and the last step is to create the VMs with it.

  1. Create Custom Images
    In you DevTestLab Service, select your lab. Then select Configuration, Custom Images:
    CreateCustomImage.JPG
    Click Add.
    CustomImage.JPG
    Enter a Name for the Image, select the correct VHD and select the OS configuration.
    Click OK and the custom image gets created. Wait until this is finished successful.
  2. Create new VMs
    Now go back to My Virtual Machines in your new DevTestLab.
    newvm
    Click Add. And you will see your Custom Images at the top of the base selection.vmbase

You can go on with your VM creation as you are used to within your first DevTestLab. If you create a Domain Controller, as I still have it (yes, I should move to Azure AD, will do that later ;-)), then remember to give it a static IP and enter that Ip in the DNS Server configuration of your Azure Network – not in the VM!

When you have verified that the VMs are running in your new DevTestLab, then you can delete your old one and you are done.

Surely you can also use this process to move any other VM which is not part of a DevTestLab.

That is it! Have fun with it :-).

Azure: Move OMS Resource Group to new Subscription

Sometimes you have to move Azure resources to a new subscription. One reason could be that your current subscription will end and you have a new one, which you want to use. Another reason could be that you need to migrate it to a subscription of someone else (demerger, etc.).

Anyhow if you do not want to loose the connection with your SCOM environment or your directly connected agents, how can I get this done? One way is always PowerShell, but I want to show you here, how you can do that within the Azure portal.

So open up http://portal.azure.com and Login.

Then go to Log Analytics and check which resource group you need to move.

oms

Now go to Resource Groups and select your resource group (here OMS).

In the Details view of the resource group you will have at the top the Move button.

OMSMove.JPG

You can now select all resources which belong to this resource group.

move

Select the new subscription and create a new resource group or select an existing one in the new subscription.

The problem which we get now is that the validation fails. Why? When you have Solutions selected in your OMS environment, which you normally have, then these cannot be moved. You need to remove them first and then the move works. The error gives the details:

Error: Resource move is not supported for resources that have plan. Resources are ‘Microsoft.OperationsManagement/solutions/ADAssessment(Lab),Microsoft.OperationsManagement/solutions/ADReplication(Lab),Microsoft.OperationsManagement/solutions/AgentHealthAssessment and tracking id is ‘xxx’. (Code: ResourceMoveFailed)

So open up OMS (http://oms.microsoft.com/oms) and remove your Solutions.

omsagenthealth

Click Remove.

When you check in the azure portal, then you will see that the resource is deleted after the solution is removed. Do that for all Solutions in your OMS resource.

At the end it should look like this, when you click Move again:

omsmove2

Do not miss to select the check box at the bottom.

Then click OK.

The old resource group will be empty when the move task is finished. So you can delete it afterwards.

You can then enable the OMS solutions again, now that all your OMS resources are in the new resource group.

The necessarity to remove the solutions from OMS can be a problem in productive environments, not for test environments. So be cautious there.

SCOM 2012: Monitor MSAs with the HP Storageworks MP 4.2.1

If you ever implemented the HP Storageworks Management Pack for System Center Operations Manager 2012 then you will find that it needs some improvements. Beside the missing monitor alerts and the view properties there is one other topic: it would require a better documentation about the prerequisites on the monitored systems.

I have recently implemented the HP Storageworks MSA Management Pack which is one part of this Storageworks solution and followed Chiyo’s blog to do the initial configuration. But I had problems to connect some MSAs to the Management Server (Error: “Unable to connect to the remote system” in the HP Storage Management Pack User Configuration Tool) and there was nothing in the Management Pack documentation about how to fix it.

Here is what I found out, what was missing:

  • The Firmware had to  be updated (Firmware TS240* or later is required for SMI-S)
  • SMI-S unencrypted had to be enabled (new setting in the newer Firmware)
  • The monitoring user (which you enter in the HP Storage Management Pack User Configuration Tool) needs SMI-S access permissions.

With that the error went away and SCOM started to monitor.

I hope this helps you too.

 

SCOM 2012: Updated SCUtils APC PDU MP V1.1

I already wrote about the SCUtils APC PDU Management Pack for SCOM 2012 in this post. That was about the version 1.0 which had problems with monitoring the PDUs from a Gateway Server.

SCUtils has now released the new Version 1.1, which enables the monitoring from Gateway Servers. I have tested it and it works. So download the latest version and implement it.

Here is what you need to do to get this working (directly taken out of the SCUtils APC PDU MP documentation):

1. Install the Operations Manager console on each gateway server that is a member of a resource pool for monitoring APC PDUs.

2. If the gateway server and management server are separated by a firewall, you have to adjust the firewall to open the following ports in both directions between the gateway server and its management server:

1. TCP 5723
2. TCP 5724

3. Prepare an account that is a member of Operations Manager Read-Only Operators group.

4. Copy SetAccountToRegistry.exe from the installation folder (the default path is ‘C:\Program Files\SCUtils\ SCUtils Management Pack for APC PDU’) to each gateway server that is a member of a resource pool for monitoring APC PDUs.

5. Run SetAccountToRegistry.exe using a local administrator account. Fill in all the required fields. And click on Test button.

Gateway.jpg

6. If the connectivity test has succeeded, click on Save button (the password will be encrypted before saving). Otherwise, reenter the information and try to test again.

7. Repeat the procedure on each gateway server that is a member of a resource pool for monitoring APC PDUs.

 

 

SCUtils APC PDU monitoring

I already wrote a blog post about the free SCUtils APC UPS monitoring pack a while ago. Now SCUtils provided the promised APC PDU Monitoring pack for System Center Operations Manager 2012 SP1/R2. The only disadvantage is that this one is with costs. You pay 716,14€ without VAT (852,21€ incl. VAT) per license which includes monitoring for 10 devices and you get one year support.

The management pack monitors two types of PDUs: PDU and PDU2, which are two different generations of APC PDUs. PDU objects are first generation devices of the AP7000 series. PDU2 objects are from the second generation AP8000 series, they have sensors included which are monitored also.

Here are some more details about the monitoring pack.

Included MPs:
SCUtils.AdminSettings.mp
SCUtils.APC.PDU.MP.mpb
SCUtils.APC.PDU.FirstDiscovery.Overrides.xml (only required to speed up the discoveries => changes frequency to 700 sec. You can remove that after all PDUs are discovered.)

The regular discoveries run ever 12 hours, that is ok. Most of the monitors run every 5 min, rules run every 10 min. All are enabled by default.

Monitors:
Monitors.JPG

Rules:Rules

The following folder/view structure gets created:
views

Diagram View:  (PDU Generation 1 => PDU2 Generation 2 would have additional objects)
diagram

You need to install the management pack on the management server and activate the license through a task. To do that go to the administration pane in the SCOM console and find the SCUtils Settings.
SCUtilsSettings

Select SCUtils Products Activation, then the Activation view is shown.
ActivationView

On the right side you have three Tasks:
ActivateTask
Check a license Task
Get unique ID for offline activation
If your console machine has internet access, then you can run the Activate task. Otherwise you can also use the Get Unique ID for an offline activation task to request the activation through email.

In the Activate task you override the LicenseKey and the CompanyName fields:ActivateTaskDetails

The management pack will work after activation.

I only tested PDU devices not PDU2.

This management pack closes the APC monitoring gap. So with both offered management packs (UPS/PDU) you can monitor your APC environment.

Attention! Version 1.0 of this MP only works if you monitor the devices through a management server not on a gateway server! A new version of this MP has been released, which also works on a Gateway Server.

SCOM 2012: Get Pool Member monitoring details

I recently had a problem that a custom rule was not running correctly, so I wanted to find out which of my SCOM 2012 Management Servers was running the All Management Servers Resource Pool instance, where the rule was targetted to.

I could not find a something which matched in the web so I contacted some of my great SCOM colleagues and got feedback from Kevin Holman with the correct solution. Thanks Kevin!

There are two tasks in SCOM already, which can give you more deatils about which Management Server takes care of which instance (class).

MSTasksI

I will show now where you find them and what you need to enter.

Both tasks you require the ID of the resource pool which handles the instances. In my case it is the ‘All Management Servers Resource Pool’.

To find that run the Operations Manager Shell and enter the command:

Get-SCOMResourcePool | FT Displayname, IDgetscomresourcepool

The output shows the resource pool names and the IDs. So copy the ID of the pool you need.

Then go to your Operations Manager Console.

Open the Management Servers State Dashboard view:
MSServerStateView

Select one of the Management Servers in the Management Server State section.

MSServerStateView2

Then run the task ‘Get the Pool Member Monitoring a Top Level Instance’.

Task1Details

Here you need to override the PoolId and the ManagedEntityID. In my case the IDs are both the same as I want to know which pool member of the All Management Servers Resource Pool manages the pool. In your case you perhaps want to know that for another class. You can find the ManagedEntityID of the class through the PowerShell command: (Get-SCOMClass -Displayname ‘xxx’).Id.

The output of the Task gives you the Management Server Name:

Task1Output

The second task has another approach. It gives you all top level instances which a Pool Member monitors.

So run the ‘Get Top Level Instances Monitored by a Pool Member’ task.

Task2Details

Here you only need to override the PoolId.

Task2Output

The output lists all classes monitored by the pool member you selected in the Management Server State view.

With that information you can now go on and troubleshoot the logs why things are not working correctly on that Management Server.