Creating a Runbook to update Aged Alerts


Server administrators often use scom notifications to receive an e-mail for critical alerts. A common problem with this is that the e-mail is sent once and if no action is taken, the alert will stay in the scom console untill there is a system failure. After the failure everyone is going to search for a scapegoat.Why didn’t the scom operators take action? To whom was the e-mail sent? Why didn’t this person take action? Blahblahblah… too late…

With this runbook we can change the ResolutionState of an alert that reaches a certain age and for which no action was taken. This can be a trigger to send out mail to a somebody that’s higher up the ladder. This person can then take action to see who’s not doing his/her job correctly.

 Ingrediënts:

System Center Orchestrator with SC 2012 Operations manager Integration Pack

  • a valid Microsoft System Center Operations Manager Connection (options > SC 2012 Operations Manager)
  • The following runbook activities
    – Monitoring Date/Time Activity
    – Run .NET Script
    – Get Alert
    – Update Alert
  • powershell cmd
    – $AlertDateBefore = (Get-Date).AddDays(-2) | get-date -Format ‘yyyy-MM-ddTHH:mm:ss’
System Center Operations Manager
  • A new Alert resolution State (i.e. “AlertAge Passed 1 day”)
  • An active alert to test the runbook
  • The MonitoringRuleId of the active alert if you want to update only certain alerts (not really necessary)

 Lets start

In your Orchestrator Runbook designer add the four runbook activities and connect them.


 
 

1) In the Monitor “Date/Time” activity you can set when and how frequent you want to run the runbook.

2) In the “Run .Net Script” activity paste the powershell command 


 

3) In the “Get Alert” activity specify the filters to granularly select which alerts you want to update.
– MonitoringRuleId: you can get this through using the “get-scomalert” cmdlet + some parameters
– TimeRaised: For this one you have to subscribe to the published data from our previous step. Just right click in the blank field 😉
– ResolutionsState: We are only going to update New alerts.
– Severity: We are only going to update Critical alerts.


 4) In the “Update Alert” activity, right click the Alert ID field. Choose subscribe > Published Data and select “Id” from the Get Alert activity. Use the “Select field…” to add the Resolution State item. You can select the new resolution state you’ve create in scom (see ingrediënts)

That’s it!! Now test your runbook.
Samuel.

Advertisements

3 Responses to Creating a Runbook to update Aged Alerts

  1. Mark D says:

    Nice! You can do the same thing with alerting settings in SCOM. Of course this is an across the board change so your solution is more elegant.

    • David Biot says:

      It is indeed possible to do this with alert ageing. Orchestrator is a more dynamic way to do this of course.

      Even better would be to generate tickets in service manager based on critical alerts in operations manager. In service manager you can use SLA’s on incidents, which enhances reporting on aged incidents.

  2. hmmmm, pretty interesting

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s