OEM 24ai Blackouts and Maintenance Windows: Stop the Alert Storm Before It Starts
Oracle observability post #4 — the last post covered incident rules and alert routing. This one is about what happens when you need to take systems down for maintenance and why skipping this step turns every planned window into an unplanned incident.
Here's a scenario I've walked into more than once.
A DBA team schedules a Saturday night patching window for a cluster of production databases. Work goes cleanly — patches applied, systems restarted, services confirmed up. Team wraps at midnight. Then the alerts start. Tablespace thresholds that were crossed during startup. Archive log space triggered during the redo application phase. Listener availability events from the restart sequence. By the time the on-call engineer processes the volume, it's 1am, and the team has spent 45 minutes triaging alerts that have already resolved themselves.
This is what a patching window without a configured blackout looks like. And it's the most common thing I clean up on day one of an OEM engagement.
Blackouts and maintenance windows aren't optional configuration. They're the difference between a monitoring system that the ops team trusts and one that they've learned to ignore.
The Two Types of Blackout: Make the Right Choice Before You Click
The most important thing to understand before configuring anything is that OEM 24ai has two fundamentally different blackout types, and picking the wrong one causes exactly the kind of problem you're trying to avoid.
Full Blackout — suspends everything. The OEM Agent stops collecting metrics entirely. No data flows to the OMS during the window. No events are generated. No incidents open. No notifications fire. When the blackout ends, metric collection resumes, but any threshold crossings that happened during the window are permanently gone — not retroactively evaluated, not queued. The slate is clean.
Use Full Blackout when: the target is genuinely offline (hardware replacement, OS-level maintenance, network isolation), or you're performing work that would generate meaningless metric noise regardless — firmware updates, storage migrations, full system rebuilds. The key signal is that you don't need to know what happened to the system during the window.
Notification Blackout — keeps the OEM Agent running and collecting metrics. Threshold crossings generate events. Events promote to incidents in Incident Manager. But notifications don't fire. The data is all there; it just doesn't ring anyone's phone.
Use Notification Blackout when: you're performing in-place maintenance (patching, configuration changes, rolling restarts) where you still want visibility into what the system is doing, but you don't want to page the on-call for transient states that are expected during the work. You want the audit trail; you don't want the noise.
Notification Blackout Subtypes: The SLA Detail That Trips People Up
Within Notification Blackout, there's a second decision that has compliance implications.
Maintenance type (-notification_only without -is_sla_required) — the blackout window is explicitly excluded from OEM's availability calculations. Your system's uptime percentage doesn't take a hit for a planned maintenance window. This is correct for scheduled patching, planned upgrades, and any work on the maintenance calendar.
Notification-only type (-notification_only -is_sla_required) — the window is not excluded from availability calculations. OEM still computes downtime against SLA during this period. This is the right choice for unplanned situations where the system is degraded but you need to suppress alert noise while the team works — because that downtime should count against your availability metrics.
The default behavior most teams get wrong: they use Maintenance type for everything, including unplanned outages. Their availability reports look cleaner than reality. That's a reporting integrity issue.
EM CLI: The Right Way to Configure Blackouts
The OEM Console blackout wizard works, but EM CLI is what you want for anything you'll run more than once. It's repeatable, scriptable, and auditable.
Basic Notification Blackout (the most common case)
emcli create_blackout \
-name="Sat_Patch_Window" \
-targets="PRODDB01:oracle_database,PRODDB02:oracle_database" \
-notification_only \
-schedule="startTime:2026-06-21 22:00;duration:04:00;tzinfo:US/Central"
This creates a 4-hour Notification Blackout (Maintenance type) for two specific database targets starting Saturday night at 22:00 Central. Metric collection stays live; no pages fire during the window.
Full Blackout for Hardware Maintenance
emcli create_blackout \
-name="Storage_Migration_BK" \
-targets="DBHOST01:host" \
-schedule="startTime:2026-06-28 06:00;duration:08:00;tzinfo:US/Central"
Host-level target with Full Blackout. All targets managed by the agent on that host are automatically suspended.
Group Blackout with Target Type Exclusions
When you're patching a large group but need to exclude certain target types:
emcli create_blackout \
-name="Prod_Group_Patch" \
-targets="Production_DBs:group" \
-propagate_targets \
-exclude_types="oracle_dbsys,weblogic_domain" \
-notification_only \
-schedule="startTime:2026-07-05 21:00;duration:06:00;tzinfo:US/Central"
-propagate_targets expands the group to all member targets. -exclude_types lets you carve out specific target types — in this case, Oracle DB systems and WebLogic domains stay unblacked while individual database instances go quiet. Useful when patching the DB tier without touching middleware targets in the same group.
Recurring Scheduled Blackout
Weekly recurring maintenance window for a regular batch job that generates predictable alert noise:
emcli create_blackout \
-name="Weekly_Batch_Window" \
-targets="BATCHDB01:oracle_database" \
-notification_only \
-schedule="frequency:weekly;startTime:2026-06-22 23:00;duration:2:00;days:7;tzinfo:US/Central"
days:7 = Sunday. Once configured, this runs without intervention. This is the pattern I recommend for any batch-heavy database that generates load and metric spikes on a predictable schedule.
Stopping a Blackout Early
Maintenance finished ahead of schedule — stop it immediately rather than letting the window run out:
emcli stop_blackout -name="Sat_Patch_Window"
Resume monitoring immediately. Don't let a 4-hour blackout continue 90 minutes after the work is done.
Auditing Blackouts
emcli list_blackouts -format="name:pretty"
This shows all active and scheduled blackouts. Run this before any incident response to quickly rule out "the target is in a blackout" as the reason alerts aren't firing.
emctl: Agent-Side Blackout for Simple Cases
For situations where you need a quick blackout without OMS access — or when you're scripting something at the agent level — emctl provides the agent-side interface.
# Blackout specific targets for 2 hours 30 minutes
emctl start blackout BK_PATCH PRODDB01 PRODDB02 -d 02:30
# Blackout all targets on the host (node-level)
emctl start blackout BK_HOST_MAINT -nodeLevel -d 04:00
# Stop a named blackout
emctl stop blackout BK_PATCH
# Check current blackout status on the agent
emctl status blackout
One important constraint: emctl always allows EM jobs to run during a blackout. If you need to block EM jobs from executing on a target during the maintenance window, you must use the Console or EM CLI — not emctl. Job blocking is configured at the OMS level, not the agent level.
Also note: emctl blackouts can only target the targets managed by that specific agent. For cross-host or group-level blackouts, use EM CLI from the OMS.
Privilege Requirements: Who Can Configure Blackouts
This trips up teams that have multiple DBAs with different OEM privilege levels.
Creating a blackout on a target requires the "Create Blackout" privilege on that target (or the group containing it). This is a named target privilege in OEM, not just a generic role.
Operators with the default "Operator" role can view blackout status but cannot create or stop blackouts on targets they don't explicitly have the privilege for. If your DBAs are reporting that they can't create blackouts on new targets, this is almost always the cause.
# Check who has Blackout Target privilege on a specific target
emcli list_target_privileges -target_name="PRODDB01" -target_type="oracle_database"
Assign via: Setup → Security → Administrators → select user → Target Privileges → Create Blackout.
ZDT Monitoring: Keeping the OMS Itself Running During Maintenance
Blackouts address target-side maintenance. There's a different problem on the OMS side: what happens to monitoring when OEM itself needs to be patched?
OEM 24ai introduces Zero Downtime Monitoring (ZDT). When you apply a Release Update to the OMS using the Zero Downtime Patching framework, OEM continues monitoring, alerting, and sending notifications throughout the patching process. The OMS stays operational during the update — no monitoring gap, no blackout needed for the management plane.
ZDT Monitoring replaces the older Always-On Monitoring (AOM) feature from previous releases. If you're still running EM 13.5 or earlier and using AOM during OMR upgrades, this is the EM 24ai equivalent — and it's significantly more capable. AOM required separate setup and had limitations on notification coverage. ZDT is built into the patching workflow.
This matters practically: in EM 24ai, you should no longer need to choose between "patch OEM" and "monitor your production estate." The two can happen simultaneously with proper planning.
The Anti-Patterns (What Causes the 2am Alert Storm)
No blackout configured for maintenance windows. The most common failure mode. Work happens, systems restart, transient threshold crossings fire as incidents, on-call gets paged for events that are already resolving. Always create the blackout before starting work, not during it.
Using Full Blackout when Notification Blackout is appropriate. If you take a Full Blackout for a rolling database patch, you lose metric history during the window. If something actually went wrong — a parameter change caused an issue, a tablespace got miscalculated — you have no data. Notification Blackout gives you the data; you just suppress the pages.
Forgetting to stop the blackout when work finishes early. A 6-hour blackout window with work done at hour 2 means 4 hours of unmonitored production. Build a checklist step: "stop blackout" before signing off on the maintenance window.
Group blackouts without exclusions. Blacking out an entire group when you're only patching one target type. If your Production_DBs group contains Oracle databases and WebLogic targets, a full group blackout silences everything. Use -exclude_types to be precise.
Using Maintenance type for unplanned downtime. If a system goes down unexpectedly and you create a Notification Blackout to suppress the noise while you troubleshoot — that's valid. But use -is_sla_required so the downtime counts against availability. Don't hide an unplanned outage inside what looks like a maintenance window in your reports.
No recurring blackout for predictable noise. If there's a batch job that runs every Sunday at midnight and spikes CPU and I/O, that generates weekly alert noise. Configure a recurring Notification Blackout for that window. The team should know about it the first time. After that, it shouldn't interrupt anyone.
What Good Looks Like
When blackout configuration is right, here's the experience:
The maintenance window gets created via emcli create_blackout as part of the change ticket workflow — not as an afterthought five minutes before the work starts. The notification blackout means the DBA team can see in Incident Manager what the system was doing during the window, but no pages fire for expected behavior.
When the window closes or the blackout is stopped early, monitoring resumes cleanly. Any genuine issues that arose — not just transient restart states — generate incidents normally. The difference between "tablespace at 91% because we ran a purge job and it's collecting itself" and "archive log destination actually full and the DB is going read-only" is visible, because the metrics were collected the whole time.
The list_blackouts command is part of every on-call runbook. First thing you check when a target isn't alerting as expected.
And the OEM 24ai ZDT Monitoring means that when the OMS itself gets patched on a quarterly basis, your production monitoring doesn't have a gap. No scramble to set up AOM. No temporary monitoring blind spot while the management plane is updated.
The Bottom Line
Full Blackout and Notification Blackout serve different purposes. Getting that choice wrong is the difference between a clean maintenance window and either missing real issues (wrong direction) or drowning in false alerts (other direction).
The decision tree is simple: if the system is genuinely offline and you don't need the metric data, use Full Blackout. If you're doing in-place maintenance and want visibility but not noise, use Notification Blackout with Maintenance type. If the downtime is unplanned and you're suppressing while you respond, use Notification Blackout with -is_sla_required.
Configure recurring blackouts for predictable noise sources. Use EM CLI for anything you'll run more than once. Stop the blackout when the work is done.
Alert fatigue and trust in the monitoring system are directly correlated. Every unnecessary 2am page trains the ops team to start ignoring the ones that matter.
Next in this series: OEM 24ai Metrics, Templates and Monitoring Framework — how to standardize what you monitor across a large Oracle estate, push threshold changes to thousands of targets in minutes, and avoid the drift that turns a well-tuned monitoring setup into an inconsistent mess over time.
