Category Archives: Orchestrator

SCSM 2012: Asset Management Part 6 – Runbook/Automation details

This is the sixth and last part of the blog series about my Asset Management solution for SCSM 2012 R2.
Part 1: General overview
Part 2: Authoring – Classes and Relationships
Part 3: Folders and Views
Part 4: Authoring – Forms
Part 5: Reports

This part of the series covers the runbooks which are created with System Center Orchestrator 2012 R2.

All Orchestrator Runbook servers need the following software and Integration Packs installed:

Nearly all runbooks use the SCSMServer variable, which you need to set:
scsmservervariable

There are three main runbooks. All check the status of Windows Computers and create or update assets. All three inform site owners at the end through email. These activities also need adjustment as you need to enter the SMTP server information and email addresses.

You can read this blog to understand the dependency between the Windows Computer object and the Deployed Computer object from SCCM. There you also see that some information, which we need to create the asset, is in the Deployed Computer object (as the serial number).

1.Create assets:

createassets

This is the main runbook to create the computer assets. It checks if the asset for a deployed Windows Computer exists, if it exists, then it only checks the relationship and updates the asset. If it does not exist, then it gets the required information through the sub runbooks and then creates the asset and the necessary relationships.

2. Update assets:
updateassets
This runbook only updates the location information, if the asset was re-imaged at a new site.

3. Check deleted:
checkdeleted

This runbook only checks for deleted objects, but does not delete the asset, as it should stay in the database.

The following sub runbooks get additional information for the assets.

Get Location information:
getlocationinfo
This runbook checks if the list entries exist for the country and site and if they are new, then it creates the enum list entry.

This runbook in the “Get Location Info” activity checks a local text file, which has the site information in this format Shortname – Site – Country – Region. Example: FRA – Frankfurt – Germany – Europe. You can also query that information from a database if you have it there.

In the “Check Country/Site” activity it checks if the enum value exists and if not, then it creates it. You will need to check, if you have another GUID for your enum lists:

$CountryEnum=’c94c8568-8cc8-2f32-ec3a-8b8b04cc9848′

$SiteEnum=’9d07bd6a-9e08-439e-486c-4ba4e7f88b30′

Check Model/Manufacturer:
checkmodelmanufacturer

This runbook checks if the list entries exist for the model and manufacturer and if they are new, then it creates the enum list entry.

You will need to check, if you have another GUID for your enum lists:

$ManufacturerEnum=’6e131742-a95b-6143-05c8-ee0f1aabae06′
$ModelEnum=’457cb61d-0c0c-7229-62ef-7b343f7b7941′

Get Warranty information: Find detailed information in this post.
getwarranty

Create AssetToComputerAsset Relationship:

createassetrelationship

Create AssetCustodian Relationship:

setcustodian

Check Relationships:

checkrelationships
Additionally we have this runbook, which checks the relationships for all classes to the AssetManagementBase class and creates them, if they do not exist.

These runbooks all only check the Windows Computer objects. If you have another source which you can use to create/update other asset object types, then you can create your own runbooks for it.

Have fun!

 

 

SCOR 2012: Monitor expired CA certificates

A lot people might think, why should I monitor expiring certificates with System Center Orchestrator 2012. There are already solutions to monitor certificates in the local store on Windows computers – for example the wonderful solution from Raphael Burri (PKI Certificate Verification MP).

But we had an issue with an expired certificate on non-windows systems. With that problem the idea came up to directly query expired certificates from the certificate authority (CA) – our internal cert store. We also wanted to notify the requesters directly, when a certificate expires within the next 30 days, send him/her reminders and last but not least inform the management a few days before the certificate expires, if no one has taken an action. With that idea we through about an automation solution – System Center Orchestrator.

Here is the description of the solution.

Prerequisites:

  • System Center Orchestrator 2012 R2
  • Exchange Mailbox to send and receive emails and a user which has permissions to it
  • Exchange Users Integration Pack
  • PS PKI PowerShell module installed on all Runbook servers
  • Microsoft Certification Authority
  • SQL Temp database (Orchestrator Temp)

Temp Database config:
certificates-table

Mailbox folders:
outlookfolders

Exchange Users Integration Pack Config:
GetEmail:
getemailconfig

Move Email:
moveemailconfig

For both configurations you need to know the path of your Exchange Server EWS service: https://autodiscover.fqdn/EWS/Exchange.asmx and the user with domain and password, which owns the mailbox.

After importing the runbooks, you will also need to configure the SQL activites (enter DB instance and Database) and the Send email activities (SMTP server and addresses). There will be some more things to change, but I will mention them in the runbook descriptions.

I suggest to run the runbooks through a scheduled task on a daily basis.

The first runbook reads the certificates and writes them to the database:

Certificates.WriteToDB

certificates-writetodb
Update the template list, you want to check, in the GetCerts Activity:
$Templist=@(
“WebServer”, #Web Server
“1.3.6.1.4.1.311.21.8.15817755.12287325.10963384.6580300.14252506.110.2574398.13148790” #Web Servers
)

Enter the names of the CAs which should be checked:
$canames=@(‘CA1FQDN‘,’CA2FQDN‘)

Certificates.GetRequesterEmail
certificates-getrequesteremail
Update the domain names and FQDNs which should be checked in the Get Requester Email activity.

function get-domain($dn)
{
if($dn -like “*,dc=domainshortname1*”){$domain = “domain1fqdn“}
if($dn -like “*,dc=domainshortname2*”){$domain = “domain2fqdn“}
return $domain
}

Now that we have all expiring certificates in the database, we need a runbook, which sends out the notifications.

Certificates.Notifications

certificates-noifications

The first email will be send out when 30 days are left and it should be send to the service desk to create a ticket. The requester will be on CC. The second email is send to the requester from day 29 to day 4. And then the last emails will be send to the management and the requester as CC.

We also created two additional runbooks to update the requester, if this information is not current anymore or to stop the notifications, if the certificate is already extended.
The requester has the option in his emails to click on a link which creates a new pre-filled email for both situations. If the certificate PPR changed, then he/she should enter the email address of the new PPR into the CC field and send the email. If the certificate was extended, then he/she only needs to send the email.

CertificateEmail.JPG

The emails will be sent to the mailbox for which we configured the Exchange User IP before.

Information:Certificate extended email
certificateextendedemail

Information:Change certificate PPR emailcertificate-newrequesteremail

Certificates.UpdateRequesterEmailcertificates-updaterequesteremail

This runbook takes the email address of the CC field and writes it to the database, then it moves the email to the Certificates.NewRequester folder in the mailbox.

Certificates.SetDeleted

certificates-setdeleted

This runbook only sets a field in the database to deleted for the mentioned certificate in the email. Then it moves the email to the Certificates.Finished folder in the mailbox.

This is the whole solution.

You can download it from github.

Most of the activities are PowerShell activities, so it should not be a big effort to also migrate the runbooks to Azure Automation or SMA.

 

 

 

 

 

 

SCSM 2012: Asset Management

This blog will be the first part of a series about my Asset Management solution for SCSM 2012 R2. It will give a general overview about the solution. The following parts will go into the details.

Part 2: Authoring – Classes and Relationships
Part 3: Authoring – Folders and Views
Part 4: Authoring – Forms
Part 5: Reports
Part 6: Runbook/Automation details

I was asked by a colleague end of last year, if we can manage hardware assets with System Center Service Manager 2012 R2. I found a solution from Steve Beaumont for SCSM 2010 besides the solutions with costs from Cireson or Provance for example. I contacted Steve and after discussing the options I started to create a new solution for SCSM 2012 R2 based on Steves template. The new solution is designed to meet the requirements of the requesting company but can also be adopted to other companies needs.

The solution covers the following requirements:

  • Sync with Active Directory (User), SCCM (PCs)
  • Assets should stay in the DB
  • Custodian should be linked to Asset
  • Read PC warranty information from Dell
  • Fill location information automatically
  • Additional categories for Computer Assets: Kiosk, Lab, Test
  • Create Assets automatically based on existing Windows Computer objects in SCSM (sync with SCCM)
  • Email notifications for new/deleted/updated objects

Prerequisites:

  • The Windows Computer names need to follow this structure: ❤ letter site Code><Serialnumber>.
  • Enable AD and SCCM connectors in SCSM.
  • Install Service Management console and SMlets on the Orchestrator Runbook Server.

How is this implemented?

To achieve that the asset stays in the DB, the solution creates an Asset class based on the ComputerHardware class. When a new Asset gets created then it needs to be assigned to the Windows Computer object. The Orchestrator automation runbooks are handling that. So it can happen that multiple Windows Computer objects which are created in SCSM get linked to one Asset object. With that you can see the history of the Asset deployment.

The Custodian also needs to be linked to the Asset, because the Asset is the hardware that should have a defined owner, which does not change when the system gets reimaged. The automation will take the primary user of the deployed Windows Computer as first Custodian, but this needs to be manually reviewed, because this could not be the real owner of the hardware (shared machines, etc.).

Classes and Relationships:

Classes

An Asset Management workspace is created and views for the new classes and some additional useful views.

views

This form shows which information can be entered for the ComputerAsset. The other asset forms look nearly the same (only the Category field is missing).

ComputerAsset

The Custodian can be selected in the AssetCustodian field. All other drop down fields reference predefined liss. The Windows Computer can be assigned in the Related Items tab. Also the Order can be referenced there.

The order form is very short and simple.Order

Some runbooks are created in Orchestrator to automate the creation of the ComputerAsset objects in SCSM and keep them updated, . Here is a short overview:

processes.JPG

All three main runbooks (Create Asset, Update Asset, Check deleted Windows Computer) check the Windows Computer objects first, then the Computer (Deployed) objects (synced from SCCM) and at last checks if an Asset object exists.

Additionally to that sub workflows are called to check the location, get the warranty, etc.

General considerations

  • Not everything can be automated, so every created/updated/deleted asset needs to be reviewed. Emails are send out to inform responsible teams to review.
  • The solution is an example how hardware asset management can be done with SCSM. It can be extended with additional class types, other properties, different automation, etc.

The solution can be downloaded from Github and you can watch a video on YouTube, which shows the solution in the SCSM console and the Orchestrator runbooks. I already presented about this solution at a German conference. So if you want to watch a German video, then you find it here.

In the other parts of my series I will go into more details how the solution is designed. So stay tuned ;-).

 

 

 

PowerShell: Temperature monitoring

If you want to monitor the temperature of your server rooms, then you have a lot of options. One is a temperature module, which is directly connected to your network and where you can access the temperature value through a XML file like: http://moduleIP/state.xml.

state.xml

We have used a solution from ControlByWeb, a PoE module with one sensor.

The idea is to have a System Center Orchestrator runbook, which checks the temperature of all sensors and creates a SCOM alert when the temperature is higher than the threshold of 30°C.

CheckTemp.xps.1

Then we also wanted to have a view directly in SCOM with the current values for all sensors. I used the PowerShell Web Widget for this.

TempSensorSCOM

The main part for all of this is a PowerShell script.

You can even use parts of the script and collect the data in SCOM.

Graph

But herefore you will need one rule for each sensor.

Functionality description:

The script reads a text file from a share with all IP addresses and names of the temperature modules.
Example:
192.168.10.110, Frankfurt
192.168.10.111, Paris

Then it connects to each module, loads the state.xml and reads the value of the first sensor.
With that data it creates an HTML table and writes that to a HTML file in a share on a web server.
The last step is that it can load the web page in the PowerShell Web Widget.

You can download the script on TechNet Gallery.

 

 

 

Orchestrator 2012: Start server patching from Service Manager

In my MMS 2015 session “Real world Automation with Service Manager and Azure Automation” with Steve Buchanan I showed how you can patch Servers initialized from a Service Manager Change Request.

The idea behind that is that there are systems which cannot be patched (and rebooted) during normal patch windows because the application owners need to control the outage times by themselves. They only know when production can handle a server outage. With Service Manager they can follow the ITIL Standards and create a Change Request, select a SCCM Collection with its Servers and the Software Updates to be applied. The Change Request will then call an Orchestrator Runbook and implement the Patches on all Servers in the given Collection.

Prerequisites:

  • The Software Updates need to be pre-deployed to all effected Servers through SCCM (Deployment Type: Available).
  • System Center Orchestrator 2012 R2, System Center Service Manager 2012 R2, System Center Configuration Manager 2012 R2
  • Log Database on SQL to store process Information
  • Sync SCCM Collections with SCSM

Temp DB Setup:

tempdb

MMSPatch

serverstatus

SoftwareUpdate

SoftwareUpdateInstall

Service Manager:

Select Template: (Patch Server)
template

Enter Title:
CR

Select Config Items to Change – SCCM Collection (Collection Info):
ConfigItems

Select Related Items – Configuration Items: Computers, Services and People (Software Update):
RelatedItems

Runbook Automation Activity:
Activity

Runbooks:

The following screenshots show the runbooks which are used for this solution.

The main runbook:

Install Software Updates (called from SCSM)MMS - Install Software Updates

Sub runbooks:

Get CR Details (writes all necessary CR information to the DB)

MMS - Get CR Details

Get Software Updates (write Software Update Information to the DB)MMS - Get Software Updates

Get Collection IDs (writes SCCM Collection Information to the DB)MMS - Get Collection IDs

Split Patching by Server (gets all Servers within the Collection)
MMS - Split By Server

Split by Patch (reads all updates from the DB)

SCCM - Split By Patch

Check Updates (checks if the Patch is available on the machine)
MMS - Check Updates

Install Update (installs the update on the machine)
SCCM - Install Updates

Update CR (updates the Change Request)
MMS - Update CR

Improvement ideas:

  • Use Service Request instead of CR
  • Import SCCM Software Update Groups into SCSM and select them

This YouTube-Video shows you the process in action.

The complete solution can be downloaded here.

Microsoft System Center Reporting Cookbook available soon

A new System Center book is on the horizon which covers the very important reporting topic. It will be published Friday 27th. You can find the link to the book and more information about it on the blog of Steve Buchanan, MVP and technical reviewer of the book.

Why is this book special?

Reporting is essential in the System Center world. What is for example Sccm without patch compliance reports? But where can you find good information about how to design System Center reports besides searching the web? This book gives you guidance with easy to follow recipes and a lot of useful information about setup, report design and other options besides SSRS like PowerPivot.

A big thank from me goes to Sam Erskine, one of the authors, who had the idea for the book. He managed the publication from the beginning to the end and it is really his baby. He made it possible that I was a technical reviewer of this book, that I saw how it grew and I am proud as a nurse which helped to bring a baby to live, that I had a small part in it.

So buy it, read it and share it ;-).

SCOM 2012: SC Orchestrator Additions Management Pack

There are already some management packs available, to monitor System Center Orchestrator 2012 with System Center Operations Manager 2012:

I am missing in those for example the monitoring of the Orchestrator database. After I wrote my last post about the Policy_Publish_Queue filling up in Orchestrator, I decided to create a mangement pack to monitor that and also added some tasks I thought that they could be useful.

You can find the management pack here

I would be glad about any comment or improvement idea.

 

Orchestrator 2012: Too much queued policy instances caused Orchestrator to slow down dramatically

Recently I had a situation with my System Center Orchestrator 2012 SP1 environment, where the Runbook Designer behaved strangely. I saw that when I started a runbook, it was not updating the log only the log history, when the runbook was finished. It also seemed to take longer than normal until the runbook was finished.
I started to check some things in my environment:

  • I checked the size of my database: with 2GB it was not too big
  • I checked the performance of my Management and Runbook servers. All looked normal.
  • I restarted the services. That did not help
  • I cleaned up some things in the DB => cleaned orphaned log entries from runbooks, deleted some old runbooks, which were not required anymore, purged the logs.
  • Then I checked the logging settings for all runbooks. With that I found one runbook, where the logging was enabled and it was currently running. But I could not stop it! It gave me an error like “Unable to un-deploy the runbook“. (sorry, I missed to create a screenshot of it 😉 ) I saw that the job history showed current entries and created always new ones. This runbook was invoked by another one, this invokation filled up the queue.

I searched around and found some SQL queries I could use to investigate more. So I logged on to the SQL server with the Orchestrator instance on it and ran the following query:

SELECT * FROM POLICY_PUBLISH_QUEUE

This gave me all instances of policies which were queued right now. And I had 350000 in there! That was the problem. I looked through the results and saw that most entries came from one policy/runbook. So I used this query to find more details about it:

SELECT POLICYINSTANCES.PolicyID ,POLICYINSTANCES.TimeStarted, POLICYINSTANCES.TimeEnded, POLICYINSTANCES.ProcessID, POLICYINSTANCES.SeqNumber, POLICIES.Name FROM POLICYINSTANCES INNER JOIN POLICIES ON POLICYINSTANCES.PolicyID = POLICIES.UniqueID WHERE POLICYINSTANCES.PolicyID = ‘PolicyID’

With that I could verify that it was the runbook, which was not stopping. So I used the next query to delete the entries from this policy out of the queue:

DELETE FROM [POLICY_PUBLISH_QUEUE] WHERE [PolicyID] =’PolicyID’

Now the queue only had 10 entries left in it :-).

I shrinked the database and checked the Orchestrator performance again and it was back to normal.

Wonderful!

Orchestrator 2012: SCOM activities are failing with error “Input string was not in a correct format”

I had recently a problem with my System Center Operations Manager 2012 (RTM) activities on my Orchestrator 2012 SP1 runbook servers.

All runbooks with SCOM activities failed. So I created a test runbook with only on SCOM activity (Get Monitor), enabled logging and checked what the error is.

The error text was: Failed to load the object properties. The exception was “Input string was not in a correct format.”.

SCOMActivityfailure

The web search did not help and I tried a lot: restarted server, redeployed SCOM integration pack, started SCOM console, which was working, tried with a new SCOM connection. Nothing helped.

So I opened a ticket with Microsoft support and they really helped very fast. Thanks!

The solution was this:

The problem was the Operations Manager Console cache, which was corrupted.

  1. To clean up this, recreate the SCOM connections with the same name.
  2. Start the SCOM console with the clear cache option: "C:\Program Files\System Center Operations Manager 2012\Console\Microsoft.EnterpriseManagement.Monitoring.Console.exe" /clearcache

 

Orchestrator 2012: Parallel reboot of server groups

As stated in my post “Orchestrator 2012: Patch a server with SCCM 2012” we had a request to reboot and patch groups of servers in parallel. The requirements were: Restart servers from different groups parallel, manual or scheduled start, do not go on with the rest of the servers in a group if one fails in this group.

How can we do that? First of all: use System Center Orchestrator 2012 – the automation tool from Microsoft.

Then I use SQL to provide the server names and store the status of the process.

I have a OrchestratorTemp database with two tables in there (plus the table described for the patching – see my blog):

ServerStatus:
ServerStatus

Service:
Service

The ServerStatus table has some entries already filled, when the runbook starts:
Servername, Grouping.

The Grouping had the values “OWA” and “General”. So servers of these two groups can be rebooted in parallel.

The start workflow looks like this:

startsite

It has to be started with the following parameter: Patch = Yes/No. This defines if patches should be applied or not.

If you need to schedule the reboots then you can add a schedule runbook in front of it which checks the date and initializes this runbook with the required start parameter.

It initiates the “Start groups” runbook and waits for completion. After the reboots it checks the patch status, checks the overall status and empties the tables (in the server status table it only deletes the fields which show the status).

Start Groups

This runbook enables the parallelity and can be extended with more groups.

startgroups
The “Get Server Groups” activity runs the following query: Select Distinct Grouping from dbo.ServerStatus.

The output will be used to start the “Control” runbook and fill the parameter “Grouping”. The parameter “Patch” is also provided to the sub runbook.

Control
control

This runbook helps to ensure that the rest of the servers in a group are skipped if one server fails.
It has a “Job concurrency” of the number of groups.

Details:

  • “Get failed Server in Group” : Select Servername from dbo.ServerStatus where Grouping =’%Grouping%’ AND Status = ‘Failed’
  • “Get next Server in Group” : Select Top 1 Servername from dbo.ServerStatus where Grouping =’%Grouping%’ AND Status is NULL.
  • When it found a server then it initiates the “Maintenance” runbook with the parameters: “Servername”,”Patch” and “Group”. It waits for completion.

Maintenance

This is the main reboot runbook. It could be split up to multiple sub runbooks, but I only took the patching part out of it. It can also be extended with pre or post activities to stop services or do other tasks around the reboot.
This runbook has a “Job concurrency” of the number of parallel groups.

maintenance

The easiest way to follow this workflow is to go straight from top to down. The enties in on the left and right are only for logging.

The main things this workflow is doing are: ping the computer, start SCOM maintenance mode, install patches, reboot, check netlogon service to see that the system is up, check patch status, check services and restart if necessary, check general service status, stop maintenance. Additionally it logs the status of the steps and sends out emails.

Here are the details of the non standard activities:

  • Get NetLogon Service Status: This is a “Run Command” activity running on the local on the runbook server. sc \\%Servername% query netlogon
  • Get Citrix Services: This is a “Run .Net” activity to get application specific services – here for Citrix. It runs a PowerShell script  and publishes the variable “output” :
    $output=@()
    $services = get-wmiobject win32_service -computername %Servername% | where {($_.displayname -like ‘*Citrix*’) -and ($_.Startmode -eq ‘Auto’)}
    foreach ($service in $services)
    {
    $output+=$service.displayname
    }
  • The Get-FQDN activity is described here.

Neil Peterson has written also a complex runbook about patching a Hyper-V cluster. He used some other methods to intialize the patching and presented the whole process on the MMS2013. You can get the details and watch the session here: http://blogs.technet.com/b/neilp/archive/2013/04/15/mms2013-session-now-on-channel-9-patching-a-hyper-v-cluster-with-orchestrator-configuration-manager-including-downloadable-runbook-exports.aspx

The Runbooks can be downloaded here.