Orchestrator 2012: Parallel reboot of server groups

As stated in my post “Orchestrator 2012: Patch a server with SCCM 2012” we had a request to reboot and patch groups of servers in parallel. The requirements were: Restart servers from different groups parallel, manual or scheduled start, do not go on with the rest of the servers in a group if one fails in this group.

How can we do that? First of all: use System Center Orchestrator 2012 – the automation tool from Microsoft.

Then I use SQL to provide the server names and store the status of the process.

I have a OrchestratorTemp database with two tables in there (plus the table described for the patching – see my blog):

ServerStatus:
ServerStatus

Service:
Service

The ServerStatus table has some entries already filled, when the runbook starts:
Servername, Grouping.

The Grouping had the values “OWA” and “General”. So servers of these two groups can be rebooted in parallel.

The start workflow looks like this:

startsite

It has to be started with the following parameter: Patch = Yes/No. This defines if patches should be applied or not.

If you need to schedule the reboots then you can add a schedule runbook in front of it which checks the date and initializes this runbook with the required start parameter.

It initiates the “Start groups” runbook and waits for completion. After the reboots it checks the patch status, checks the overall status and empties the tables (in the server status table it only deletes the fields which show the status).

Start Groups

This runbook enables the parallelity and can be extended with more groups.

startgroups
The “Get Server Groups” activity runs the following query: Select Distinct Grouping from dbo.ServerStatus.

The output will be used to start the “Control” runbook and fill the parameter “Grouping”. The parameter “Patch” is also provided to the sub runbook.

Control
control

This runbook helps to ensure that the rest of the servers in a group are skipped if one server fails.
It has a “Job concurrency” of the number of groups.

Details:

  • “Get failed Server in Group” : Select Servername from dbo.ServerStatus where Grouping =’%Grouping%’ AND Status = ‘Failed’
  • “Get next Server in Group” : Select Top 1 Servername from dbo.ServerStatus where Grouping =’%Grouping%’ AND Status is NULL.
  • When it found a server then it initiates the “Maintenance” runbook with the parameters: “Servername”,”Patch” and “Group”. It waits for completion.

Maintenance

This is the main reboot runbook. It could be split up to multiple sub runbooks, but I only took the patching part out of it. It can also be extended with pre or post activities to stop services or do other tasks around the reboot.
This runbook has a “Job concurrency” of the number of parallel groups.

maintenance

The easiest way to follow this workflow is to go straight from top to down. The enties in on the left and right are only for logging.

The main things this workflow is doing are: ping the computer, start SCOM maintenance mode, install patches, reboot, check netlogon service to see that the system is up, check patch status, check services and restart if necessary, check general service status, stop maintenance. Additionally it logs the status of the steps and sends out emails.

Here are the details of the non standard activities:

  • Get NetLogon Service Status: This is a “Run Command” activity running on the local on the runbook server. sc \\%Servername% query netlogon
  • Get Citrix Services: This is a “Run .Net” activity to get application specific services – here for Citrix. It runs a PowerShell script  and publishes the variable “output” :
    $output=@()
    $services = get-wmiobject win32_service -computername %Servername% | where {($_.displayname -like ‘*Citrix*’) -and ($_.Startmode -eq ‘Auto’)}
    foreach ($service in $services)
    {
    $output+=$service.displayname
    }
  • The Get-FQDN activity is described here.

Neil Peterson has written also a complex runbook about patching a Hyper-V cluster. He used some other methods to intialize the patching and presented the whole process on the MMS2013. You can get the details and watch the session here: http://blogs.technet.com/b/neilp/archive/2013/04/15/mms2013-session-now-on-channel-9-patching-a-hyper-v-cluster-with-orchestrator-configuration-manager-including-downloadable-runbook-exports.aspx

The Runbooks can be downloaded here.

Advertisements
Post a comment or leave a trackback: Trackback URL.

Comments

  • Mark  On February 28, 2014 at 4:37 pm

    Hello, hope you are doing well, you wouldn’t by chance have these runbooks listed in the article available for download?

    • opsmgrtipps  On March 18, 2014 at 12:01 pm

      Hi Mark,

      I have updated my blog with the link to the runbooks.

      Thanks,
      Natascia

      • Mark  On March 20, 2014 at 2:31 pm

        Thank you!

  • jason  On December 15, 2014 at 8:43 pm

    Great post, I was looking at leveraging task sequences but cam across the issue of stopping a group if one failed and came across your post on the patch runbooks. That link led me here.

  • wdjenkins  On January 14, 2015 at 9:11 pm

    Trying to figure out in the “Start” runbook, in the step “Query Patch Status” there is a part of the query that isn’t clear to me “WHERE ServerStatus.Version='{from “Initialize Data”}”. It is referencing a field in the table “ServerStatus” called “Version”, but I don’t see where this is defined in your example of the table, and I don’t see where it referenced anywhere else so I’m not sure what its looking for or how it will be populated.

    I’m trying to diagnose why when I test the runbook starting with “Start” I get no errors but starts the maintenance, but if I test the runbook “Maintenance” (providing a target server name manually) it runs exactly as expected.

    Hopefully I’m not missing something obvious.

    • wdjenkins  On January 14, 2015 at 9:27 pm

      I realized I had a typo in my initial question. Below is how the sentence should have read:

      “I’m trying to diagnose why when I test the runbook starting with “Start” I get no errors but DOESN’T START the maintenance”

    • opsmgrtipps  On January 19, 2015 at 11:34 am

      Hi, that was a mistake from my side – that was an entry from a previous Version. So please delete everything with Version out. You can also try to download the runbook again. I have deleted that part out.

      Thanks,
      Natascia

      • wdjenkins  On January 27, 2015 at 10:24 pm

        Last question, I hope, If a server in the list fails for any particular reason shouldn’t the maintenance continue on down the list? What I’m seeing is that if, for example, one of the servers is offline (can’t even ping it) it gets marked “Failed” for status but then the job stops. If the failed server is in “Group1” then no other servers in “Group1” get processed after that, but “Group2” continues processing until it hits a “Failed”. Is this what you would expect, or have I made a mistake with my logic?

        Feel free to respond to me via email with my screen name @ yahoo com

      • wdjenkins  On January 28, 2015 at 4:34 pm

        The most recent download of your exported runbook doesn’t appear to work. After running the import nothing actually is imported. I even tried on a fresh server with nothing else installed. I had no issues importing the previous version you had up.

  • Hugh McIntyre (@HughMc)  On February 26, 2015 at 12:05 am

    I can import the latest version after coming back to this from a while back to see if i can get it to work. The workflow makes sense however a few things are missing. Version? for eg. Mentioned a lot in the SQL queries but not in the SQL tables at all.

    • opsmgrtipps  On February 26, 2015 at 7:36 am

      Have you downloaded the file recently? There should not be any Version reference in there anymore.

      Natascia

  • ML  On May 20, 2015 at 8:16 pm

    Ho can I download this runbook

  • Ashen  On October 17, 2016 at 4:13 pm

    Hi,

    Thank you very much for the valuable article. however the Download link is broken.could you fix that if possible..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: