Every Oracle environment has a list of tasks that run on a schedule. Statistics gathering. Log archiving. RMAN backup verification. Tablespace usage reports. Health checks before maintenance windows. On a small estate, cron jobs and DBMS_SCHEDULER handle this well enough. On a larger estate — 50 databases, 200 hosts, multiple data centers — that approach creates a coordination problem. The jobs run, but the output is scattered across servers, nobody is watching for failures, and a missed execution goes unnoticed until something breaks downstream.

OEM 24ai's job system solves this by centralizing job scheduling, execution, and monitoring across your entire estate. One place to define what runs, where, and when. One place to see what succeeded and what failed. One place to get paged when something doesn't run. No logging into individual servers, no reconciling output from disparate cron logs.

Job Types in OEM 24ai

OEM supports several job types, each designed for a specific class of work:

OS Command — runs a shell script or command on the agent host. Good for host-level maintenance: log rotation, file cleanup, pre/post-patching steps, space management. The script runs in the OS context of the Management Agent user.

SQL Script — executes SQL against an Oracle database target. Script runs in the database context, output is captured in OEM. Good for statistics operations, data purges, custom health checks, report generation.

RMAN Script — executes an RMAN script against a database. The job handles connection and output capture. Good for backup validation, image copies, crosschecks, restore testing.

SQL*Plus Script — runs SQLPlus with a script you supply. More flexible than SQL Script for operations that use SQLPlus-specific commands or complex formatting. Use when you need control over output formatting or SQL*Plus settings.

Multi-Task Job — a sequence of steps where each step is its own job type. Steps can run serially or in parallel, with pass/fail conditions controlling flow. This is where OEM's job system becomes genuinely powerful for operational automation.

Creating and Scheduling Jobs (EM CLI)

The console job wizard works for one-off setup. For repeatable, scriptable job management, EM CLI is the right tool.> Note: EM CLI syntax below is representative. Exact parameter names vary by OEM version and verb type. Always validate against your environment: emcli help create_job and emcli help get_jobs.

Create a SQL script job targeting multiple databases

emcli create_job \
  -name="PROD_STATS_GATHER_WEEKLY" \
  -type="SQL Script" \
  -target_list="PRODDB01:oracle_database,PRODDB02:oracle_database,PRODDB03:oracle_database" \
  -input_file="sql_script:/opt/oracle/scripts/gather_stats.sql" \
  -schedule_type="weekly" \
  -start_time="2026-10-05 02:00:00" \
  -repeat_hours=168 \
  -tz="America/Chicago"

Create an OS command job targeting a host group

emcli create_job \
  -name="PROD_HOST_LOG_CLEANUP" \
  -type="OSCommand" \
  -target_list="Production_Hosts:group" \
  -os_command="/opt/oracle/scripts/cleanup_alert_logs.sh" \
  -schedule_type="daily" \
  -start_time="2026-10-01 01:00:00" \
  -tz="America/Chicago"

Submit a job immediately (outside its schedule)

emcli submit_job \
  -name="PROD_STATS_GATHER_WEEKLY"

Get job execution history

emcli get_jobs \
  -name="PROD_STATS_GATHER_WEEKLY" \
  -status="Succeeded,Failed,Running"

Output includes execution time, duration, target, and exit status. For failed jobs, use the console to drill into full output and error messages.

Job Libraries: Reusable Templates

If the same job runs against multiple target sets with minor variations, the Job Library saves you from redefining it each time.

A Job Library entry is a saved job definition without a schedule or specific target list attached. When you need to run it, you create a job from the library entry, add the target list and schedule, and submit.

Console path: Enterprise → Job → Library

This is particularly useful for:

• Standard maintenance scripts that run against different groups on different cadences (production on Sunday, development on Tuesday)

• Jobs that different DBAs submit against their own target subsets

• Patching and upgrade pre-checks that get rerun at each patching cycle

Build the library as you build the jobs. By the time you have 10–15 standardized operations in the library, onboarding a new DBA to your team's operational practices becomes significantly easier.

Multi-Task Jobs: Chained Workflows

A Multi-Task Job is a sequence of steps where each step is its own job type. Steps can be:

• Serial — step 2 starts only after step 1 completes

• Parallel — steps run concurrently; a final step waits for all to complete

• Conditional — step 3 runs only if step 2 succeeded; step 4 runs only if step 2 failed

Example: pre-maintenance health check workflow

1. Run tablespace usage query (SQL Script) — verify no tablespace is above 85%

2. If step 1 succeeds → run invalid objects check (SQL Script)

3. If step 1 or 2 fails → generate alert incident (via notification rule)

4. Run RMAN crosscheck (RMAN Script) — confirm backup catalog is clean

5. Run listener status check (OS Command) — confirm listeners are up

This kind of workflow would previously live in a shell script that called multiple utilities, parsed exit codes, and had its own error handling. OEM's Multi-Task Job does it natively: per-step status, per-step output capture, conditional branching, and full visibility in the console — no custom orchestration code.

Console path: Enterprise → Job → Activity → Create Job → Multi-Task

Job Output and Failure Visibility

Every job execution captures:

• Start and end time

• Target name and host

• Exit status (Succeeded, Failed, Stopped)

• Full output log — stdout/stderr for OS commands, SQL output for database scripts

Console path: Enterprise → Job → Activity — filterable by status, target, time range, and job name.

For long-running jobs, you can monitor live output in the console rather than waiting for completion or tailing a log on the host.

Connecting to incident management: Create incident rules that fire on job failure. A failed RMAN crosscheck at 3am becomes an OEM incident, which triggers your on-call notification via the same infrastructure that handles your performance alerts. One notification framework for both real-time monitoring and scheduled job failures.

Console path for notification rules: Enterprise → Monitoring → Incident Rules → Create Rule → select "Job Execution" as the event type → filter on status = Failed and severity = your threshold.

What to Automate First

If you're starting from scratch with OEM job automation, three categories give the most return for the setup investment:

Statistics gathering. Most environments still run this ad-hoc or through DBMS_SCHEDULER jobs with no centralized visibility. Moving it to OEM gives you a single place to confirm it ran on every production database, see how long it took, and get paged if it fails. The group targeting approach means one job definition covers your entire production database fleet.

Log and trace file cleanup. Alert logs, trace files, and audit files accumulate on every Oracle host. An OS Command job targeting your host group, running weekly, keeps disk pressure under control without requiring manual intervention. Schedule it to run before your disk utilization monitoring alerts fire — not after.

RMAN backup verification. Your backups run. But is someone checking that they succeeded? An RMAN crosscheck job with a failure notification rule turns "I assume the backups are good" into "I know the backups are good, and here's when I last verified." This one change has caught real backup failures in multiple customer environments before the database team realized there was a problem.

The Anti-Patterns

Using OEM jobs to replace DBMS_SCHEDULER. For database-internal scheduling — maintenance window management, stats gathering that runs inside a maintenance window, application-layer batch jobs — DBMS_SCHEDULER is the right tool. OEM jobs are for cross-estate coordination and operations that need centralized visibility and failure alerting. Don't duplicate the same job in both systems.

No output retention policy. OEM stores job output in the Management Repository. Long-running jobs with verbose SQL output accumulate fast. Set a retention period in the job settings (Enterprise → Job → Settings) rather than letting the repository grow unchecked.

Submitting jobs manually when automation is available. If a DBA is running the same operation by hand every week, it should be scheduled. The manual step adds risk — forgotten runs, inconsistent timing, no output retention — without adding value. If you're doing it more than twice, build the job.

No failure notification on scheduled jobs. A job that fails silently is worse than no job at all. It gives you false confidence that the task ran. Every scheduled job that matters should have a notification rule on failure. This is a one-time setup cost per job type.

One monolithic OS command job. Breaking complex operations into Multi-Task Job steps gives you per-step status, per-step output, and conditional branching. A shell script that does everything in sequence tells you "it failed" — a Multi-Task Job tells you which step failed, what its output was, and which subsequent steps were skipped or redirected.

Ignoring duration trends. Statistics gathering that used to take 45 minutes and now takes 3 hours is a signal. OEM captures job duration for every execution. Review it periodically. An unexplained duration increase is often your first indicator of a growing data volume problem, degraded I/O, or a locking issue.

What Good Looks Like

In a well-configured estate, the OEM job system handles the operational repetition: statistics, cleanup, verification, health checks, pre-maintenance checks. All of it runs on schedule, against the right target groups, with output captured and failures surfaced as incidents. DBAs aren't logging into servers to run manual checks; they're responding to alerts when automation finds a problem.

The Job Library has reusable templates for the 10–15 standard operations your team runs regularly. Multi-Task Jobs handle workflows that used to live in shell scripts nobody maintained. The job activity view gives you a single answer to "did everything run last night?"

When something fails — and eventually something will — you know immediately, you have the full output, and you can correlate the failure with other events on the same target.

The Bottom Line

OEM 24ai's job system turns routine DBA operations from manual tasks into managed, monitored automation. OS commands, SQL scripts, RMAN scripts, and chained Multi-Task workflows cover the operational workloads that most teams are still running manually or through undocumented cron jobs.

The investment is upfront: defining the jobs, building the library, wiring up the notification rules. The return is every subsequent execution running automatically, with output retained and failures paged — not discovered days later when someone notices a side effect.

Coming soon in this series: OEM 24ai Incident Management and Notification Rules — how to build a notification framework that gets the right alert to the right person at the right time, and how to avoid the alert fatigue that makes monitoring systems useless.

Every conversation about OEM eventually gets to the same uncomfortable question: what's the DR plan for OEM itself?

Most teams have a rock-solid answer for their production databases — Data Guard, RAC, tested failover, quarterly drills. Ask the same team about their OMS and OMR, and the answer is usually "we back it up nightly" or, worse, silence. That's a gap. OEM isn't a nice-to-have dashboard. It's the system that pages your on-call team when a production database is in trouble, runs your patching workflows, and enforces your compliance standards. When it goes down, you don't just lose a UI — you lose visibility into everything it was watching, at exactly the moment you're least equipped to notice.

This post covers the reference topologies for making OEM 24ai itself highly available, the mechanics of adding a standby repository, and the failure modes that catch people who treat OEM HA as an afterthought.

Why OEM HA Is a Different Problem Than Database HA

An OMS is stateless-ish but not quite stateless — it holds a shared software library, in-flight job state, and an active console session layer. The OMR is a full Oracle database, so RAC and Data Guard apply directly. The Management Agents deployed across your estate are the resilient part almost by accident: they retry, buffer, and re-upload, so a short OMS outage doesn't lose monitoring data, it just delays it.

The design question isn't "can we make this resilient" — every component here is buildable with standard Oracle HA/DR patterns. The question is which combination of RTO and RPO your estate actually needs, because the complexity curve is steep once you move past a single OMS.

Four Reference Topologies

For the vast majority of enterprise customers I work with, Topology 3 is the right landing spot. It's the point where the RTO/RPO numbers actually match what the business expects from "our monitoring platform," without dragging in the operational overhead of a full active/active design that almost nobody needs for a management tier.

If you're still on Topology 1, that's fine for a lab. If it's your production OMS, you're one hardware failure away from flying blind on your entire Oracle estate until someone restores from backup.

Topology 2: Multi-OMS HA, in Practice

This is the minimum bar for a production-grade OEM deployment. The pieces:

• Two or more OMS nodes behind a Server Load Balancer

• Shared software library on NFS or DBFS — writable by every OMS node

• Single OMR, typically a 2-node RAC database

The part that trips people up isn't the OMS nodes themselves — it's the load balancer configuration. The SLB has to terminate TLS, support sticky sessions for console traffic, and — this is the one that actually causes incidents — provide persistence on the Agent Upload channel.

SLB ports that need to be load balanced correctly:

If Agent Upload isn't load balanced with proper persistence, agents don't fail over gracefully — they pile onto whichever OMS answered first, and you get a silent, uneven load skew that looks like a performance problem until someone traces it back to the SLB config. This is the single most common misconfiguration I see in multi-OMS deployments.

Topology 3: Adding a Standby OMR with Data Guard

Once the OMS tier is resilient, the repository is the remaining single point of failure. A backup-and-restore RTO measured in hours is not acceptable for most production OEM deployments — that's hours of no monitoring, no incident rules firing, no patching visibility, during whatever event took the OMR down in the first place.

Adding Data Guard closes that gap:

• Primary OMR at the production site, standby at the DR site

• Sync transport where the network supports it — aim for round-trip latency under 10ms

• Async transport for cross-continent DR pairs

• The standby stays read-only. Do not point OMS at it during normal operations — that's a Data Guard misconfiguration waiting to cause split-brain-style confusion, not a valid load-balancing shortcut.

Switchover, at a High Level

Note: EM CLI syntax below is representative. Exact parameter names vary by OEM version and job type. Always validate against your environment: emcli help <verb> and Oracle's official EM CLI reference.

# 1. Stop OMS on all nodes
emctl stop oms -all

# 2. Switch over the OMR (run from the standby side once it's ready)
# SQL*Plus / DGMGRL:
# ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

# 3. Update the OMS connect descriptor to point at the new primary
emctl config oms -store_repos_details \
  -repos_conndesc "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dr-omr-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=EMREP_DR)))" \
  -repos_user sysman

# 4. Restart OMS
emctl start oms

# 5. Verify agent uploads are resuming
emctl status agent
emctl status oms -details

A well-rehearsed switchover is a 20-to-30-minute operation. The keyword is rehearsed. I've sat in more than one post-incident review where the DR runbook for OEM hadn't been touched in over a year, and step 4 referenced a hostname that had since changed. Rehearse the switchover at minimum twice a year — treat it exactly like you'd treat a database DR drill, because functionally, that's what it is.

Agent Resilience Isn't Something You Build — It's Something You Configure Correctly

Here's the good news: Management Agents don't need their own HA design. They're effectively stateless from a monitoring-continuity standpoint — if they can't reach an OMS, they buffer locally and retry, then catch up once connectivity returns. No monitoring data is lost during a short OMS interruption, only delayed.

What does need attention is agent failover across OMS nodes:

• Agents can be configured with a comma-separated list of OMS URLs, so a single agent knows about more than one management server

• If you're using an SLB VIP as the single agent-facing URL, failover is automatic — the SLB handles routing, and the agent never needs to know a specific OMS node went down

Note: EM CLI syntax below is representative. Exact parameter names vary by OEM version and verb type. Always validate against your environment: emcli help <verb> and Oracle's official EM CLI reference.

# Check which OMS URLs an agent currently knows about
emctl getemhome
emctl status agent | grep -i "OMS URL"

# Re-secure an agent against a new OMS/VIP after a DR cutover
emctl secure agent -emdWalletSrcUrl https://dr-oms-vip.example.com:4903/em

The pitfall shows up specifically at DR cutover, when the DR site uses a different VIP than production. If your DR runbook doesn't include a mass agent re-secure step, every agent in the estate silently stops reporting the moment you fail over — and nobody notices until someone asks why a target has been "green" for six hours with no data behind it. Where possible, use the same VIP DNS name across sites so cutover doesn't require touching every agent.

The Anti-Patterns

Software library on local disk. Works fine for a single OMS. The moment you add a second OMS node, a local software library means only one node can actually run software-library-dependent jobs correctly. Validate shared storage before you validate anything else in a multi-OMS build.

No SLB persistence on Agent Upload. Covered above, but worth repeating because it's the most common production issue: without it, you get a "thundering herd" onto one OMS during heartbeat cycles, and it presents as a mysterious performance problem instead of the load balancer misconfiguration it actually is.

OMR backups without Data Guard. A backup strategy is not a DR strategy for a system whose entire job is real-time visibility. A 4-hour restore window is a long time to have zero eyes on a production Oracle estate.

DR site with different OMS hostnames. This forces certificate and wallet rebuilds on cutover, on top of everything else you're already dealing with during a DR event. Standardize the VIP DNS name across sites during initial design, not during the incident.

Skipping the bi-annual DR drill. Every runbook I've reviewed that hadn't been drilled in the last six months had at least one stale step. Not most — every one. The drill is what finds the stale step before a real event does.

What Good Looks Like

A well-architected OEM HA/DR setup looks boring, which is exactly the point. Two or more OMS nodes sit behind a properly configured SLB that load-balances console, agent upload, agent registration, and BI Publisher traffic correctly. The software library lives on shared storage every node can write to. The OMR runs as a 2-node RAC primary with a Data Guard standby at the DR site, sync or async depending on network distance. Agents are configured to fail over automatically through the SLB VIP, and the DR runbook — switchover steps, connect descriptor updates, mass agent re-secure if needed — has been rehearsed within the last six months, not just written once and filed away.

When someone asks "what happens if we lose the OMS," the answer is "nothing, the second node picks it up," not a long pause.

The Bottom Line

OEM watches your Oracle estate, which means OEM's own availability is a Tier 1 concern, not an operational nice-to-have. Multi-OMS HA behind a correctly configured load balancer is the production floor. Adding a Data Guard standby OMR gets you from a backup-restore RTO measured in hours to a switchover measured in minutes. None of the individual pieces are exotic — it's standard Oracle MAA thinking applied to the management tier instead of the application tier. The part that actually determines whether it works during a real event is whether you've rehearsed it.

Next in this series: OEM 24ai Patch Management and Fleet Patching — how Gold Images and Database Lifecycle Management turn a 50-database patch cycle from a multi-week fire drill into a scheduled, repeatable operation.

Most Oracle environments have a “we ran a security assessment once” problem. A hardening checklist is applied at provisioning time. An auditor samples controls annually. In between, parameters drift, accounts accumulate, and privileges expand — and nobody knows until the next review.

Oracle Enterprise Manager 24ai Compliance Management gives you a repeatable way to evaluate managed targets against defined configuration, security, storage, and operational standards. It does not make a database secure by itself, and it is not a substitute for a security assessment. What it does provide is continuous evidence: defined rules, target associations, evaluation results, and a documented exception process.

Start with the Compliance Library

OEM ships with Oracle-provided compliance frameworks and standards for Oracle hardware and software. The exact content installed in your environment depends on the Enterprise Manager release updates and Self Update content you have applied, so treat the Compliance Library as the source of truth rather than assuming a particular benchmark or database release is present.

For database security work, common starting points include:

• CIS Compliance Standards — self-update-enabled standards, with availability and supported database versions defined by the current Compliance Standards Reference.
• Security Technical Implementation Guide (STIG) standards — also delivered through the compliance content lifecycle; use the version that matches your database platform and release.
• Security Configuration for Oracle Database and other Oracle-provided standards — baseline configuration checks supplied with Enterprise Manager.
• Your internal standards — organization-specific requirements for parameters, auditing, privileged accounts, encryption, backup, and operational controls.

Before associating a security standard, enable the required database security configuration metric collections. Oracle documents an Oracle Certified monitoring template for this purpose: Oracle Certified-Enable Database Security Configuration Metrics.

Two practical cautions:

1. Do not claim CIS, STIG, GDPR, or any other framework coverage until you have verified the actual standard, version, and rules in your own Compliance Library.
2. A passed rule means the target met that rule as evaluated. It is not a blanket statement that the database is secure or compliant with a regulation.

Associate a Standard and Evaluate It

A compliance standard is associated with managed targets from Enterprise → Compliance → Library. Select the standard, click Associate Target, select the eligible targets, and save the association. The initial evaluation runs as a background job; results normally appear after the evaluation completes.

For a small number of targets, use the documented EM CLI verb. First identify the standard’s internal name, author, and version with emcli list_standards; Oracle-provided standards require those exact values.

emcli associate_cs_targets   -name="<standard_internal_name>"   -version="<standard_version>"   -author="<standard_author>"   -target_list="PRODDB01"

You can include a group in the target list by appending :Group:

emcli associate_cs_targets   -name="<standard_internal_name>"   -version="<standard_version>"   -author="<standard_author>"   -target_list="Production_DBs:Group"

For a large estate, group association is the useful operating model. When eligible new targets are added to an associated group and meet the target property filter, OEM can associate them automatically. This is a group-association capability — it is not the same thing as assuming every Administration Group will automatically apply a compliance standard.

Read the Results Correctly

Console path: Enterprise → Compliance → Results

OEM lets you view results by compliance framework, standard, and target. The Target Compliance view is especially useful for identifying the least-compliant targets across the standards currently associated with them.

The score is a computed result. It considers factors such as violations, rule severity, and the importance assigned to rules and folders in the standard. That means scores are most useful for trending the same standard against a comparable population — not for declaring that an 88% score on one standard is intrinsically better than an 82% score on another.

When reviewing results, prioritize:

1. Violations with the highest security or operational impact.
2. New violations and unexpected changes in the score trend.
3. Evaluation errors — an error is not a pass and can leave you with an incomplete picture.
4. Rules marked manual, where the evidence and decision must be recorded outside an automated check.

Remediation and Exceptions

Each violation should lead to one of two outcomes:

Remediate it. The rule details can include description, impact, recommendation, and corrective-action guidance. Corrective actions in OEM are scripts that fix a violation; they can be manual or automatic when configured through incident rules. Test every corrective action in a representative non-production environment before broad use.

Document a time-bound exception. OEM’s documented mechanism is violation suppression, not a generic “waiver.” The suppression workflow records a reason and can include a suppress_until date. Use it only for an approved exception, and track the business owner, compensating control, review date, and expiration in your security governance process.

For automation, the documented EM CLI verb is:

emcli suppress_compliance_rule_violations   -cs_iname="<standard_internal_name>"   -author="<standard_author>"   -version="<standard_version>"   -rule_iname="<rule_internal_name>"   -target_type="oracle_database"   -target_name="PRODDB01"   -suppress_until="12-31-2026"   -reason="Approved exception; compensating control documented"

A suppressed violation is still a signal that the rule failed. Suppression makes the risk decision explicit; it does not make the underlying control compliant.

Create Standards That Match Your Environment

Oracle-provided standards are a useful baseline, but they will not represent every internal requirement. Create a user-defined standard when you need to continuously check requirements such as mandated initialization parameters, audit configuration, approved account patterns, or environment-specific configuration rules.

Console path: Enterprise → Compliance → Library → Compliance Standards → Create

A standard applies to a single target type and can contain rules, rule folders, and included standards. OEM supports several rule types, including repository, agent-side, monitoring, configuration drift, configuration consistency, manual, and missing-patch rules. Choose the lightest rule type that can reliably collect the evidence you need.

For a quick starting point, use Create Like to copy an existing standard and then tailor it. In 24.1.0.3 and later, Oracle also documents an export/import path for enhancing eligible out-of-box CIS content. Do not directly edit an Oracle-provided standard in the library; preserve the vendor baseline and keep your variation clearly identified.

An Operating Model That Holds Up

A practical production model is:

• Associate the relevant, verified standards with the right target population.
• Review results on a defined cadence — monthly is a reasonable starting point; use a shorter cadence for high-change environments.
• Triage new and material violations first.
• Remediate, or record a time-bound approved exception with compensating controls.
• Investigate score drops and evaluation errors promptly.
• Reassess associations after Enterprise Manager release updates that change standards or rules; Oracle notes that re-association can be required for updated compliance content.

The anti-pattern is running compliance only before an audit. The value is not a point-in-time score. It is the operating history: what was checked, what changed, what failed, who accepted an exception, and when it expires.

The Bottom Line

Compliance Management in OEM 24ai turns configuration and security assurance into an operational process. It gives you a framework for Oracle-provided and internal standards, target-level evaluations, score and result analysis, and corrective-action or exception workflows.

Use it as evidence, not as a certification. Validate the standards installed in your environment, align them with your security team’s control objectives, and make the review-and-remediation cadence part of normal database operations.

References

• Oracle Enterprise Manager 24ai — Overview of Compliance Management
• Oracle Enterprise Manager 24ai — Configure Compliance Management
• EM CLI: associate_cs_targets
• EM CLI: suppress_compliance_rule_violations

Next in this series: OEM 24ai Jobs and Automation Framework — how to use OEM’s job system to automate routine DBA tasks, schedule maintenance operations across your estate, and build multi-step workflows that run without manual intervention.

---

Here's what monitoring drift looks like in practice.

A DBA tunes the tablespace warning threshold on a production database from 85% to 90% because the application has a regular purge cycle and 85% generates weekly noise. Makes sense. Six months later, that DBA has tuned 12 databases the same way — all done manually through the OEM console, all slightly different because someone had a different idea of the right value on a different day. Meanwhile, 40 other databases still have the OEM default of 85%. And the three databases added last quarter? They have whatever was in the default profile, which nobody reviewed.

This is monitoring drift. And it shows up on every large Oracle estate I've worked with that doesn't have Monitoring Templates configured.

The problem isn't that people are making bad decisions about these thresholds. The problem is that there's no mechanism to make those decisions once and apply them everywhere.

---

## What Monitoring Templates Control

A Monitoring Template in OEM 24ai is a named collection of monitoring settings for a specific target type. It can define:

• Warning and critical thresholds — numeric values or pattern matches, depending on the metric type

• Collection schedule — how frequently the metric is sampled

• Number of occurrences before an alert triggers — prevents single-sample spikes from generating incidents

• Corrective actions — automated responses when a threshold is crossed

• Template description — documentation for the standard and its intended scope

When you apply a template, its metric settings are copied to the target. Be deliberate about the apply options: the default preserves thresholds for metrics that are not in the template; using -replace_metrics=1 clears those thresholds and can stop alerting for them.

Creating and Applying Templates (EM CLI)

OEM 24ai supports template management through the console and EM CLI. For anything at scale — more than a handful of targets, or anything you'll repeat — EM CLI is the right path.

### Create from a tuned target, then export the template

To create a baseline from an already tuned target, use the Monitoring Template wizard and choose the option to copy that target's settings. Once the template exists, export it with EM CLI:

emcli export_template \
-name="Prod_DB_Standard" \
-target_type="oracle_database" \
-output_file="/tmp/prod_db_standard.xml"

Use -archive instead when the template includes a Metric Extension.

Apply a template to specific targets

emcli apply_template \
-name="Prod_DB_Standard" \
-targets="PRODDB01:oracle_database;PRODDB02:oracle_database;PRODDB03:oracle_database"

Apply to all members of a group

emcli apply_template \
-name="Prod_DB_Standard" \
-targets="Production_DBs:composite"

This is where the leverage kicks in. A single command applies your standard configuration to every database in the group. When you update the template and reapply, every target gets the new values in one operation.

Validate a target against its standard

EM CLI provides list_templates to inventory templates, but it does not provide a get_template_applied verb. For a target-level check, use the console's Compare Monitoring Template workflow or inspect the affected metric settings and template-override flags. That shows whether the target is aligned with the intended standard and whether a documented exception is in effect.

Administration Groups: Automatic Template Assignment

Manual template application works. Administration Groups (AdGroups) make it automatic.

An AdGroup is a dynamic group that uses target-property criteria to enroll targets. When a target joins the group, OEM applies the monitoring templates in the group's associated template collection; synchronization can run on a schedule or be started on demand.

Membership criteria use supported target properties such as Lifecycle Status, Line of Business, Location, Target Type, Target Version, Contact, Cost Center, Department, and Customer Support Identifier. Use the hierarchy to model combinations and exceptions; Administration Group membership is mutually exclusive.

Example AdGroup structure for a typical enterprise:

When a new production database is discovered and its Lifecycle Status is set to "Production," it joins the appropriate Administration Group and receives the template collection's monitoring settings at synchronization. The template collection can include a database template plus templates for other target types.

Setting Up Administration Groups

Administration Groups are configured from Setup -> Add Target -> Administration Groups.

Setup sequence:

1. Create the Administration Group hierarchy and membership criteria

2. Create the monitoring templates

3. Create a template collection and add the required templates (one template per target type)

4. Associate the template collection with the applicable Administration Group

5. Synchronize the group or configure the synchronization schedule

A target belongs to at most one Administration Group. Within the hierarchy, settings inherited from a higher level apply to lower levels; for duplicate metric settings, the lower-level template collection takes precedence.

Exceptions and Precedence

OEM does not use a universal four-tier priority stack for monitoring templates. A direct apply_template operation writes the selected template settings to its targets. In an Administration Group hierarchy, lower-level template-collection settings override inherited higher-level settings only where the same metric setting is defined.

For a documented target-specific exception, set the metric's Template Override. A metric marked as a template override is protected from subsequent template applies. Review those overrides regularly, because they are intentional exceptions to the standard.

Separating Templates by Environment Tier

One template for all environments is a common mistake. Dev and production should have meaningfully different configurations:

**Production template characteristics:**

- Tighter thresholds — alert sooner because degradation has user impact

- Shorter collection intervals on critical metrics (CPU, active sessions, availability)

- Corrective actions enabled (automated recovery is appropriate in production)

- Number of occurrences = 2-3 for volatile metrics to suppress transient spikes

**Development template characteristics:**

- Looser thresholds — developers expect heavy workloads during testing

- Longer collection intervals — less frequent sampling reduces agent overhead on shared dev infrastructure

- No automated corrective actions — a script that restarts a dev database will interrupt someone's active session

- Higher occurrence counts before alerting — less urgency for transient states

**Stage template:** closer to production thresholds (you want to catch real issues in staging), but no corrective actions that would interfere with planned test runs.

---

## Metric Extensions: When Built-in Metrics Aren't Enough

OEM 24ai ships with hundreds of built-in metrics for Oracle targets. For cases the built-in set doesn't cover, Metric Extensions let you define custom metrics.

Custom metric sources:

- **OS command** — run a shell script on the agent host, parse the output as metric values

- **SQL query** — query the Oracle database directly; the result set becomes the metric data

- **JMX** — pull from Java MBeans (for WebLogic and Java application targets)

Once created, a Metric Extension behaves exactly like any built-in metric — it can be included in a Monitoring Template, thresholded, and pushed across your estate.

Common use cases: checking for specific initialization parameters your team requires (e.g., `ARCHIVELOG` mode, `ENABLE_DDL_LOGGING`), monitoring application schema objects (queue depths, custom job status tables, application-specific row counts), verifying backup catalog state that OEM doesn't track natively.

Create in console: Enterprise -> Monitoring -> Metric Extensions -> Create.

---

## The Anti-Patterns

Relying on OEM default thresholds in production. Oracle-supplied defaults are recommended baselines, not a substitute for thresholds calibrated to your workload and service objectives. If normal operation regularly crosses a default threshold, tune it to the environment instead of masking the alerts with suppression rules.

Applying templates manually without AdGroups. Every new target added to OEM becomes an exception to your standard. The longer you run without AdGroups, the more targets fall outside your standard configuration. Manual application is better than nothing; AdGroups are better than manual.

One template for all environments. If dev, stage, and production share the same thresholds and collection intervals, you'll either over-alert in dev or under-alert in production. Separate templates per lifecycle tier are non-negotiable on a large estate.

Setting "occurred 1 time" for volatile metrics. CPU, active sessions, redo generation, and similar metrics can spike transiently during normal operations. If your policy fires on a single sample, you'll generate incidents for events that resolve in the next collection cycle. Set occurrences to 2-3 for performance metrics.

Not documenting threshold rationale. Six months later, nobody remembers why Physical Reads is set to 500K instead of the default. Record the reason in the template description and your operational runbook or change record. "Set to 90% per 2026-03 review — PURGE_JOB runs weekly and fills to 88% before clearing" is useful. "Updated" is not.

Forgetting to synchronize after template updates. Editing a template changes the repository definition, not the target's active settings. Reapply a directly assigned template with apply_template. For Administration Groups, run synchronization on demand or confirm the configured synchronization schedule.

What Good Looks Like

In a well-configured estate, every target type has a template tuned by the team that understands what those metrics mean in that environment. Production, staging, and dev have separate templates. Administration Groups handle enrollment automatically.

When a DBA provisions a new database, registers it with OEM, and sets the appropriate target properties, it is enrolled automatically. When application behavior changes, update the template and synchronize the Administration Group—or reapply the template directly—to propagate the new setting.

Monitoring drift stops being a problem because there is no easy mechanism for it to accumulate. Individual overrides are documented and intentional. The default state for any target in OEM is "correctly configured," not "whatever the agent shipped with."

And when you're asked in an audit or incident review "what is the warning threshold for tablespace usage on production databases?" — the answer is in the template, documented, and consistent across every production database in the relevant Administration Group.

The Bottom Line

Monitoring Templates are the difference between a configuration that was set up once and slowly drifted, and one that's actively managed. The template is the source of truth. AdGroups make application automatic. Metric Extensions handle what the built-in set doesn't cover.

The investment is a few hours upfront for a typical estate. The return is years of consistent, trustworthy alerting — and the end of "why is this threshold different on this database" conversations during an incident.

Get the templates right, and the monitoring system becomes something the ops team can actually rely on.

---

*Next in this series: OEM 24ai Compliance Standards and Security Frameworks — how to use OEM's built-in compliance library to continuously assess your Oracle estate against CIS benchmarks, Oracle security baselines, and custom standards, and how to turn compliance scores into actionable remediation workflows.*

---

Here's a scenario I've walked into more than once.

A DBA team schedules a Saturday night patching window for a cluster of production databases. Work goes cleanly — patches applied, systems restarted, services confirmed up. Team wraps at midnight. Then the alerts start. Tablespace thresholds that were crossed during startup. Archive log space triggered during the redo application phase. Listener availability events from the restart sequence. By the time the on-call engineer processes the volume, it's 1am, and the team has spent 45 minutes triaging alerts that have already resolved themselves.

This is what a patching window without a configured blackout looks like. And it's the most common thing I clean up on day one of an OEM engagement.

Blackouts and maintenance windows aren't optional configuration. They're the difference between a monitoring system that the ops team trusts and one that they've learned to ignore.

The Two Types of Blackout: Make the Right Choice Before You Click

The most important thing to understand before configuring anything is that OEM 24ai has two fundamentally different blackout types, and picking the wrong one causes exactly the kind of problem you're trying to avoid.

Full Blackout — suspends everything. The OEM Agent stops collecting metrics entirely. No data flows to the OMS during the window. No events are generated. No incidents open. No notifications fire. When the blackout ends, metric collection resumes, but any threshold crossings that happened during the window are permanently gone — not retroactively evaluated, not queued. The slate is clean.

Use Full Blackout when: the target is genuinely offline (hardware replacement, OS-level maintenance, network isolation), or you're performing work that would generate meaningless metric noise regardless — firmware updates, storage migrations, full system rebuilds. The key signal is that you don't need to know what happened to the system during the window.

Notification Blackout — keeps the OEM Agent running and collecting metrics. Threshold crossings generate events. Events promote to incidents in Incident Manager. But notifications don't fire. The data is all there; it just doesn't ring anyone's phone.

Use Notification Blackout when: you're performing in-place maintenance (patching, configuration changes, rolling restarts) where you still want visibility into what the system is doing, but you don't want to page the on-call for transient states that are expected during the work. You want the audit trail; you don't want the noise.

Notification Blackout Subtypes: The SLA Detail That Trips People Up

Within Notification Blackout, there's a second decision that has compliance implications.

Maintenance type (-notification_only without -is_sla_required) — the blackout window is explicitly excluded from OEM's availability calculations. Your system's uptime percentage doesn't take a hit for a planned maintenance window. This is correct for scheduled patching, planned upgrades, and any work on the maintenance calendar.

Notification-only type (-notification_only -is_sla_required) — the window is not excluded from availability calculations. OEM still computes downtime against SLA during this period. This is the right choice for unplanned situations where the system is degraded but you need to suppress alert noise while the team works — because that downtime should count against your availability metrics.

The default behavior most teams get wrong: they use Maintenance type for everything, including unplanned outages. Their availability reports look cleaner than reality. That's a reporting integrity issue.

EM CLI: The Right Way to Configure Blackouts

The OEM Console blackout wizard works, but EM CLI is what you want for anything you'll run more than once. It's repeatable, scriptable, and auditable.

Basic Notification Blackout (the most common case)

emcli create_blackout \
  -name="Sat_Patch_Window" \
  -targets="PRODDB01:oracle_database,PRODDB02:oracle_database" \
  -notification_only \
  -schedule="startTime:2026-06-21 22:00;duration:04:00;tzinfo:US/Central"

This creates a 4-hour Notification Blackout (Maintenance type) for two specific database targets starting Saturday night at 22:00 Central. Metric collection stays live; no pages fire during the window.

Full Blackout for Hardware Maintenance

emcli create_blackout \
  -name="Storage_Migration_BK" \
  -targets="DBHOST01:host" \
  -schedule="startTime:2026-06-28 06:00;duration:08:00;tzinfo:US/Central"

Host-level target with Full Blackout. All targets managed by the agent on that host are automatically suspended.

Group Blackout with Target Type Exclusions

When you're patching a large group but need to exclude certain target types:

emcli create_blackout \
  -name="Prod_Group_Patch" \
  -targets="Production_DBs:group" \
  -propagate_targets \
  -exclude_types="oracle_dbsys,weblogic_domain" \
  -notification_only \
  -schedule="startTime:2026-07-05 21:00;duration:06:00;tzinfo:US/Central"

-propagate_targets expands the group to all member targets. -exclude_types lets you carve out specific target types — in this case, Oracle DB systems and WebLogic domains stay unblacked while individual database instances go quiet. Useful when patching the DB tier without touching middleware targets in the same group.

Recurring Scheduled Blackout

Weekly recurring maintenance window for a regular batch job that generates predictable alert noise:

emcli create_blackout \
  -name="Weekly_Batch_Window" \
  -targets="BATCHDB01:oracle_database" \
  -notification_only \
  -schedule="frequency:weekly;startTime:2026-06-22 23:00;duration:2:00;days:7;tzinfo:US/Central"

days:7 = Sunday. Once configured, this runs without intervention. This is the pattern I recommend for any batch-heavy database that generates load and metric spikes on a predictable schedule.

Stopping a Blackout Early

Maintenance finished ahead of schedule — stop it immediately rather than letting the window run out:

emcli stop_blackout -name="Sat_Patch_Window"

Resume monitoring immediately. Don't let a 4-hour blackout continue 90 minutes after the work is done.

Auditing Blackouts

emcli list_blackouts -format="name:pretty"

This shows all active and scheduled blackouts. Run this before any incident response to quickly rule out "the target is in a blackout" as the reason alerts aren't firing.

emctl: Agent-Side Blackout for Simple Cases

For situations where you need a quick blackout without OMS access — or when you're scripting something at the agent level — emctl provides the agent-side interface.

# Blackout specific targets for 2 hours 30 minutes
emctl start blackout BK_PATCH PRODDB01 PRODDB02 -d 02:30

# Blackout all targets on the host (node-level)
emctl start blackout BK_HOST_MAINT -nodeLevel -d 04:00

# Stop a named blackout
emctl stop blackout BK_PATCH

# Check current blackout status on the agent
emctl status blackout

One important constraint: emctl always allows EM jobs to run during a blackout. If you need to block EM jobs from executing on a target during the maintenance window, you must use the Console or EM CLI — not emctl. Job blocking is configured at the OMS level, not the agent level.

Also note: emctl blackouts can only target the targets managed by that specific agent. For cross-host or group-level blackouts, use EM CLI from the OMS.

Privilege Requirements: Who Can Configure Blackouts

This trips up teams that have multiple DBAs with different OEM privilege levels.

Creating a blackout on a target requires the "Create Blackout" privilege on that target (or the group containing it). This is a named target privilege in OEM, not just a generic role.

Operators with the default "Operator" role can view blackout status but cannot create or stop blackouts on targets they don't explicitly have the privilege for. If your DBAs are reporting that they can't create blackouts on new targets, this is almost always the cause.

# Check who has Blackout Target privilege on a specific target
emcli list_target_privileges -target_name="PRODDB01" -target_type="oracle_database"

Assign via: Setup → Security → Administrators → select user → Target Privileges → Create Blackout.

ZDT Monitoring: Keeping the OMS Itself Running During Maintenance

Blackouts address target-side maintenance. There's a different problem on the OMS side: what happens to monitoring when OEM itself needs to be patched?

OEM 24ai introduces Zero Downtime Monitoring (ZDT). When you apply a Release Update to the OMS using the Zero Downtime Patching framework, OEM continues monitoring, alerting, and sending notifications throughout the patching process. The OMS stays operational during the update — no monitoring gap, no blackout needed for the management plane.

ZDT Monitoring replaces the older Always-On Monitoring (AOM) feature from previous releases. If you're still running EM 13.5 or earlier and using AOM during OMR upgrades, this is the EM 24ai equivalent — and it's significantly more capable. AOM required separate setup and had limitations on notification coverage. ZDT is built into the patching workflow.

This matters practically: in EM 24ai, you should no longer need to choose between "patch OEM" and "monitor your production estate." The two can happen simultaneously with proper planning.

The Anti-Patterns (What Causes the 2am Alert Storm)

No blackout configured for maintenance windows. The most common failure mode. Work happens, systems restart, transient threshold crossings fire as incidents, on-call gets paged for events that are already resolving. Always create the blackout before starting work, not during it.

Using Full Blackout when Notification Blackout is appropriate. If you take a Full Blackout for a rolling database patch, you lose metric history during the window. If something actually went wrong — a parameter change caused an issue, a tablespace got miscalculated — you have no data. Notification Blackout gives you the data; you just suppress the pages.

Forgetting to stop the blackout when work finishes early. A 6-hour blackout window with work done at hour 2 means 4 hours of unmonitored production. Build a checklist step: "stop blackout" before signing off on the maintenance window.

Group blackouts without exclusions. Blacking out an entire group when you're only patching one target type. If your Production_DBs group contains Oracle databases and WebLogic targets, a full group blackout silences everything. Use -exclude_types to be precise.

Using Maintenance type for unplanned downtime. If a system goes down unexpectedly and you create a Notification Blackout to suppress the noise while you troubleshoot — that's valid. But use -is_sla_required so the downtime counts against availability. Don't hide an unplanned outage inside what looks like a maintenance window in your reports.

No recurring blackout for predictable noise. If there's a batch job that runs every Sunday at midnight and spikes CPU and I/O, that generates weekly alert noise. Configure a recurring Notification Blackout for that window. The team should know about it the first time. After that, it shouldn't interrupt anyone.

What Good Looks Like

When blackout configuration is right, here's the experience:

The maintenance window gets created via emcli create_blackout as part of the change ticket workflow — not as an afterthought five minutes before the work starts. The notification blackout means the DBA team can see in Incident Manager what the system was doing during the window, but no pages fire for expected behavior.

When the window closes or the blackout is stopped early, monitoring resumes cleanly. Any genuine issues that arose — not just transient restart states — generate incidents normally. The difference between "tablespace at 91% because we ran a purge job and it's collecting itself" and "archive log destination actually full and the DB is going read-only" is visible, because the metrics were collected the whole time.

The list_blackouts command is part of every on-call runbook. First thing you check when a target isn't alerting as expected.

And the OEM 24ai ZDT Monitoring means that when the OMS itself gets patched on a quarterly basis, your production monitoring doesn't have a gap. No scramble to set up AOM. No temporary monitoring blind spot while the management plane is updated.

The Bottom Line

Full Blackout and Notification Blackout serve different purposes. Getting that choice wrong is the difference between a clean maintenance window and either missing real issues (wrong direction) or drowning in false alerts (other direction).

The decision tree is simple: if the system is genuinely offline and you don't need the metric data, use Full Blackout. If you're doing in-place maintenance and want visibility but not noise, use Notification Blackout with Maintenance type. If the downtime is unplanned and you're suppressing while you respond, use Notification Blackout with -is_sla_required.

Configure recurring blackouts for predictable noise sources. Use EM CLI for anything you'll run more than once. Stop the blackout when the work is done.

Alert fatigue and trust in the monitoring system are directly correlated. Every unnecessary 2am page trains the ops team to start ignoring the ones that matter.

Next in this series: OEM 24ai Metrics, Templates and Monitoring Framework — how to standardize what you monitor across a large Oracle estate, push threshold changes to thousands of targets in minutes, and avoid the drift that turns a well-tuned monitoring setup into an inconsistent mess over time.

There's a pattern I see at almost every customer site after a fresh OEM 24ai deployment.

The monitoring is working. Metrics are being collected. Thresholds are set. But six months in, the DBAs have started ignoring the alert emails — because there are too many, half of them don't apply to the person receiving them, and the format tells you something fired but not what to actually do about it.

That's not a monitoring problem. That's an incident rules problem.

Getting an alert is useless if the wrong person gets it, at the wrong time, in the wrong format, with no context about severity. This post is about getting that right in OEM 24ai — the rule set architecture, the common failure patterns I see in the field, and how to connect OEM's notification pipeline into the broader ops toolchain, including OCI Notifications for hybrid environments.

The OEM Incident Pipeline: What Actually Happens

Before you touch a rule set, you need to understand how OEM moves from a raw metric value to a page on someone's phone. There are four stages:

Metric → raw data collected by the OEM Agent on the target. CPU utilization, tablespace usage, active sessions, etc. Collected on a polling interval, stored in the OMS.

Event → a metric threshold crossing. When CPU utilization exceeds your configured warning or critical threshold, OEM generates an event. Events are also generated by target availability changes, compliance violations, and EM jobs. Incident → a grouping of related events, managed through Incident Manager. By default OEM creates one incident per event, but you can configure rules to correlate multiple events into a single incident. This is how you avoid 47 separate incidents when an Exadata cell node goes offline and cascades.

Notification → an action triggered by an incident rule. Email, SNMP trap, PagerDuty webhook, OCI Notifications endpoint, or an EM CLI script. This is the step where routing decisions are made.

The rule set layer sits between incident creation and notification. Rules evaluate: For this incident, at this severity, on this target type — who gets told, how, and what do they see?

Rule Sets vs. Rules: The Container Model

In OEM 24ai, notification routing is controlled by Incident Rule Sets. The hierarchy is:

• Rule Set — a container with an ordered list of rules, evaluated top-to-bottom. Rule sets are assigned to a scope: all targets, a specific group, or a named target list.

• Rule — an individual condition + action pair. "For Severity 1 and 2 incidents on targets in the Production-DBs group, send email to dba-oncall@company.com and create a PagerDuty incident."

• Action — what happens when the rule matches. Email, SNMP, custom script, OCI endpoint.

The key thing to understand: rule evaluation stops at the first matching rule within a rule set. Order matters. If your generic "send all alerts to dba-team" rule is first, your targeted routing rules below it will never fire.

What Ships Out of the Box (and Why It's Not Enough)

OEM 24ai ships with two out-of-box (System Generated) enterprise rule sets, visible under Setup → Incidents → Incident Rules. Note that these Oracle-supplied rule sets cannot be exported or imported via EM CLI. They are:

Incident management rule set for all targets — the main system-generated set, scoped to all targets. It bundles roughly twenty rules that create, compress, and clear incidents for events like target-down and agent-unreachable, high-availability events, critical metric alerts, compliance violations, and SLA alerts. Its job is to create and manage incidents in Incident Manager — none of these default rules sends email or any other notification on its own.

A metric-alert rule set — fires email notifications for all Warning and Critical metric alerts. Default recipient: whatever email address was configured during OMS installation. The second one is the source of most of the alert fatigue I see. It sends everything, to one address, at all hours. Within a few months in a real environment — dozens of targets, hundreds of monitored metrics — that mailbox is noise. DBAs stop reading it.

The fix is to replace the default routing with rule sets that are built around how your team actually works.

Building Rule Sets That Work

Here's the model I use with enterprise customers. Three rule sets, ordered by specificity:

Rule Set 1: Production Critical (Severity 1–2)

Scope: A named group containing your production Oracle databases, RAC clusters, and Exadata targets.

Rules inside:

1. Severity 1 incidents → immediate email + PagerDuty → DBA on-call rotation + database manager

2. Severity 2 incidents → email → DBA team distribution list, business hours only (06:00–22:00 on weekdays)

3. All target availability Down events → immediate email + SMS → DBA on-call, any hour

Key configuration detail: Set Notification Repeat Interval for Severity 1 to every 30 minutes until acknowledged. OEM supports this natively — set it in the Advanced section of the notification action. Without it, you get one email when the incident opens and silence afterward.

Rule Set 2: Non-Production and Dev/Test (Severity 1–3)

Scope: Non-prod target group.

Rules inside:

1. Severity 1 incidents only → email → DBA team DL, business hours only

2. Severity 2–3 incidents → email daily digest to team lead (not to individuals)

Non-prod environments don't need paging. A daily digest is enough visibility without generating noise that erodes trust in the alerting system.

Rule Set 3: Catch-All (Severity 4–5 / Everything Else)

Scope: All targets.

Rules inside:

1. All Severity 4–5 incidents → log to Incident Manager only, no notification

2. All unmatched incidents → email to monitoring team alias, low priority flag

Severity 4–5 in OEM are advisory. They should be reviewed periodically, not interrupt anyone's workflow.

Path: Setup → Incidents → Incident Rules → Create Rule Set

Notification Templates: Stop Sending Raw Alerts

Default OEM email notifications look like this:

Target: PRODDB01
Metric: CPU Utilization
Value: 94.3
Threshold: 90

That tells you what fired. It doesn't tell you whether this is normal weekend batch behavior, whether this same alert fired last Tuesday, or what the on-call engineer should look at first.

OEM 24ai supports custom notification message templates. Use them.

Path: Setup → Notifications → Notification Methods → Manage Notification Methods → OS Command or Email → Edit Message Template

A better template includes:

• Target name and type

• Metric name, current value, threshold breached

• Severity level

• Time of event

• Direct link to the incident in Incident Manager (via the URL substitution variables available in OEM notification templates)

• Last 3 occurrences of this metric alert on the same target (query from EM CLI)

The last point requires a small wrapper script — but a message that says "this is the 4th time this week" changes how urgently an on-call DBA responds versus "CPU was high one time."

Escalation Rules: What Happens When No One Responds

OEM supports escalation rules within a rule set. An escalation rule fires when an incident has been open (or acknowledged but unresolved) for a defined time window.

Common pattern:

• Severity 1 incident not acknowledged within 15 minutes → escalate to DBA manager

• Severity 1 incident acknowledged but not resolved within 2 hours → escalate to database team lead + incident bridge

Configure this under the rule's Actions → Add Escalation section. The escalation target is a separate notification method, so you can route initial alerts to individuals and escalations to a group.

This is the piece most teams skip — and it's the piece that matters at 2am when the primary on-call is unavailable.

EM CLI: Audit and Manage Rules Programmatically

If you're managing more than one OMS or handing rule sets across environments, EM CLI is the right approach. You don't want to manually replicate a 15-rule set through the console.

Note: EM CLI has no verb that lists incident rules. You review rule sets in the console under Setup → Incidents → Incident Rules. To capture a rule set's full definition for review or backup, export it to XML:

emcli export_incident_rule_set -rule_set_name="Production Critical" -rule_set_owner=sysman -export_file="/tmp/"

Import the rule set into another OMS (for multi-OMS HA environments):

emcli import_incident_rule_set -import_file="/tmp/Production_Critical.xml" -alt_rule_set_name="Production Critical"

This is especially useful in OEM 24ai HA environments where you run multiple OMS nodes behind a load balancer. Rule sets replicate through the OMR, but having version-controlled exports in a git repo means you can always roll back a bad rule change without hunting through Incident Manager history.

Connecting OEM to the Broader Notification Stack

For teams running hybrid environments — which, as I covered in the last post, is most Oracle shops today — OEM's notification pipeline needs to connect to the same tools the rest of the ops team uses. The DBA team might live in OEM and email. The cloud ops team is in Slack and PagerDuty. Leadership wants a dashboard, not an inbox.

Three patterns I've seen work well:

Pattern 1: OEM → SNMP → PagerDuty OEM has native SNMP trap support. PagerDuty's generic webhook integration can receive SNMP traps via a bridge like OpsGenie's SNMP integration or a lightweight snmptrapd → HTTP relay. Lower latency than email, works without custom code. Pattern 2: OEM → EM CLI Custom Notification → OCI Notifications Set up a custom OS command notification that calls a Python script. That script POSTs the incident payload to an OCI Notifications topic via the OCI SDK. From there, the topic fans out to Slack, PagerDuty, email — whatever your cloud ops team uses. I covered the OEM REST API endpoint for this in the last post:

https://<OMS_HOST>:<OMS_PORT>/em/api/v1/incidents   # verify the exact host, port, and path for your OMS install

The same endpoint works in reverse — your notification script can pull full incident details and include them in the payload, not just the sparse EM notification defaults.

Pattern 3: OCI Notifications as the Single Alerting Plane If you've already set up OCI Monitoring alarms (as described in Step 4 of the last post), consider routing OEM Severity 1–2 notifications and OCI Monitoring alarms to the same OCI Notifications topic. One topic, one subscriber list, one PagerDuty service. Your on-call doesn't need to know whether the alert originated from OEM or OCI — they need to know something's broken and where to look.

This pattern requires some discipline: you need to deduplicate. If both OEM and OCI Monitoring are watching CPU on the same host and both alert, you get double pages. Split the responsibility by layer — OEM monitors Oracle process-level metrics, OCI Monitoring handles host-level infrastructure metrics. Don't overlap.

The Anti-Patterns (What I Fix First at New Customer Sites)

Alert on everything = alert on nothing. When the DBA team has 200 unread alert emails, the one that matters gets missed. Start by auditing which alerts have fired more than 50 times in the last 30 days. If it's that frequent, it's either a false positive or a chronic issue that needs a fix, not a repeated notification.

No severity segmentation. Warning and Critical going to the same mailbox, at the same priority, with the same notification format. If an on-call can't distinguish a "hey watch this" from a "wake up now," the severity system is broken. No escalation path. A single point of contact for Severity 1 alerts with no escalation rule is a production risk. People travel, phones die, PagerDuty apps get removed. If your monitoring system has no fallback when the primary path fails, you've created a single point of failure in your incident response.

Notification bloat on test environments. Non-prod alerts drowning prod alerts in the same inbox is one of the most common problems I fix on day one of an engagement. Separate rule sets, separate distribution lists, or at minimum separate email subjects that let filters do the work.

No blackouts configured. Maintenance windows with no blackout = alert storms at 2am. Post #4 will cover OEM blackout configuration in detail — because this is its own topic worth getting right.

What Good Looks Like

When OEM 24ai's incident routing is working correctly, here's the experience:

• Severity 1 fires → the right on-call engineer gets paged within 60 seconds, with enough context (target, metric, value, link to Incident Manager) to act immediately

• Severity 2 fires → the DBA team distribution list gets an email during business hours, formatted with enough detail to triage without opening OEM first

• Non-prod fires → a daily digest review, not individual notifications

• A 15-minute silence from the paged engineer → the manager gets a follow-up automatically

• Post-incident → Incident Manager has a full audit trail, linked to the original metric event, with timestamps for acknowledgment and resolution

That's not a complex setup. It's around three rule sets, a handful of rules each, and a couple of notification methods — scale that up or down to fit your environment. The complexity isn't in the implementation — it's in taking the time to think through how your specific team works before clicking through the console.

The Bottom Line

OEM 24ai's monitoring is only as effective as its notification routing. An alert that goes to the wrong person, arrives with no context, or gets buried in noise from lower-severity events might as well not exist.

Start with a rule set audit: review your rule sets in the console (Setup → Incidents → Incident Rules), exporting them with export_incident_rule_set if you want a full offline copy, and map what you have against what your team actually receives and responds to. In most environments I've seen, there's a significant gap between what OEM is configured to send and what the ops team treats as actionable.

Fix the routing first. The rest of the observability work in this series depends on people trusting that when OEM fires, it means something real.

Next in this series: OEM 24ai Blackouts and Maintenance Windows — how to suppress alerts during planned maintenance without creating monitoring blind spots or drowning your team when the window ends.

There's a debate that comes up in almost every Oracle architecture conversation I have these days.

"Should we use OEM or OCI for monitoring?"

The short answer: both. The right answer: they're not doing the same job.

A quick note on OCI Stack Monitoring: If you've been using OCI Stack Monitoring, you need to know it's officially deprecated with End of Life on January 23, 2027. Oracle's guidance is to migrate to OCI Monitoring (infrastructure metrics), OCI Logging Analytics (log correlation), and OCI Database Management (DB fleet performance) before that date. This post is written against the current OCI Observability stack — the tools you should be building on right now.

Treating OEM 24ai and the OCI Observability suite as alternatives is like choosing between a hospital and an ambulance. One is purpose-built for a specific, deep function. The other gets you connected to a broader system. You need both, and you need them talking to each other.

After 17 years of Oracle deployments — from pure on-prem Oracle estates to hybrid cloud migrations and full OCI lifts — this is one of the architectural decisions I've seen teams get wrong most consistently. And the cost shows up later: blind spots in production, duplicate alerting, and two disconnected dashboards that no one trusts.

Let's fix that.

What OEM 24ai Actually Does Well

Oracle Enterprise Manager 24ai is a deep-stack tool. Its core strength is native Oracle target management at a level nothing else matches.

Where OEM is irreplaceable:

• Oracle Database internals — AWR, ASH, SQL monitoring, blocking sessions, undo usage, buffer cache hit ratios. OEM reads internal Oracle metrics that no third-party tool can access without significant custom work.

• Lifecycle management — Patching, provisioning, compliance standards, gold image management. This is not monitoring, it's management. OCI doesn't do this.

• EM Jobs — Scheduled scripts, backup jobs, RMAN management, DB cloning. These are operational workflows, not just observability.

• Exadata and RAC — Cell server metrics, smart scan offload efficiency, InfiniBand/RoCE interconnect health, ASM disk group monitoring. OEM 24ai has native Exadata awareness no other product can replicate.

• Target hierarchy — OEM's concept of a monitoring hierarchy (RAC → Instance → PDB → Service) with inheritance and override is sophisticated. You can apply a single metric template to 5,000 targets and push threshold changes in minutes.

• EM CLI + REST API — Automation-first. You can script everything: target discovery, job execution, template application, incident management.

If you're running Oracle Database on-prem or on Oracle Cloud Infrastructure Dedicated Infrastructure, OEM 24ai is the right primary tool for that layer. Full stop.

What OCI Observability Services Do That OEM Can't

The OCI Observability and Management suite was designed to answer a different question: How does your full application stack look from the cloud?

The suite covers four distinct functions, each a separate service:

OCI Monitoring handles infrastructure-level telemetry — host CPU, memory, disk, network — through the Management Agent. It's where you create alarms, set thresholds, and wire notifications to PagerDuty, Slack, or OCI Functions. For OCI Compute instances, coverage is near-instant after agent deployment.

OCI Logging Analytics handles log ingestion, parsing, and correlation at scale. If your Oracle Database alert logs, listener logs, OS syslogs, and application logs need to be correlated into a single query surface — this is where that happens. The ML-based anomaly detection finds patterns in log data that threshold-based monitoring misses entirely.

OCI Database Management covers Oracle Database fleet performance from the cloud plane. SQL performance analytics, wait event analysis, AWR data aggregation across multiple databases, and space management — all accessible without needing OEM Console access. For your cloud ops team, this eliminates the need for OEM credentials to diagnose a DB performance issue.

OCI Ops Insights handles capacity analytics and long-range SQL trend analysis. It answers questions like "which databases will run out of tablespace in 90 days" or "which SQL statement has degraded most in the last 30 days across our fleet." This is planning intelligence that OEM generates per-target but doesn't aggregate across the estate the way Ops Insights does.

The key point: none of these services are trying to replace OEM. They're giving you observability above the database layer and around the cloud boundary that OEM was never designed to cover.

The Architecture That Actually Works

Here's how I recommend structuring this for a hybrid Oracle environment — which is 90% of what customers are running today:

On-Premises                              OCI
─────────────────────────────────────    ────────────────────────────────────
Oracle DB (19c/21c/23ai)                 Oracle DB@Azure / ExaCS / BaseDB
  └── OEM 24ai Agent                       └── OCI Management Agent
        │                                        │
        ▼                                        ▼
    OEM OMS ───────────────────────────► OCI Monitoring
    (OMR on Data Guard)                  (Infrastructure alarms + notifications)
        │                                        │
        ▼                                        ▼
    EM CLI / REST API                    OCI Database Management
                                         (Fleet SQL analytics + wait events)
                                                 │
                                                 ▼
                                         OCI Logging Analytics
                                         (Log correlation + ML anomaly detection)
                                                 │
                                                 ▼
                                         OCI Ops Insights
                                         (Capacity planning + SQL trends)

The bridge is the OCI Management Agent running on your on-prem Oracle hosts. Once deployed, it streams host metrics into OCI Monitoring and enables Database Management's external database feature for on-prem Oracle DBs — giving you cloud-plane visibility into databases that live on-premises.

The Decision Table: Which Tool Monitors What

The pattern is clear: OEM owns the Oracle stack deeply. The OCI Observability suite owns breadth, log intelligence, fleet analytics, and cloud-native alerting.

Step-by-Step: Connecting Your On-Prem Oracle Targets to OCI Observability

Here's the practical path to get both tools working together. These steps assume you already have OEM 24ai running on-prem.

Step 1: Deploy the OCI Management Agent on Your Oracle Hosts

The Management Agent is distinct from the OEM Agent. It runs alongside it without conflict.

1. Download the Management Agent from OCI Console → Observability & Management → Management Agents

2. Deploy to your Oracle DB hosts (requires JDK 8u261+ or JDK 11)

3. Agent key: each agent registers against an Agent Install Key you generate in OCI — scope this to a compartment that mirrors your on-prem environment

# Example install on Oracle Linux
chmod +x oracle.mgmt_agent-<version>.rpm
sudo rpm -ivh oracle.mgmt_agent-<version>.rpm
sudo /opt/oracle/mgmt_agent/bin/setup.sh -f /tmp/input.rsp

The input.rsp file contains your agent install key and compartment OCID. Scope to the correct compartment — don't use root compartment for production.

Step 2: Enable OCI Database Management for On-Prem Databases

With the Management Agent running, you can connect your on-prem Oracle databases to OCI Database Management as "external databases."

1. OCI Console → Observability & Management → Database Management

2. Create External Database → Provide connection details and credentials

3. Use a read-only monitoring user — dbsnmp or a custom low-privilege role

4. Assign the compartment that aligns with your on-prem environment segmentation

Once connected, Database Management gives your cloud ops team AWR summaries, top SQL, wait event analysis, and space trends — without OEM Console access.

Step 3: Set Up OCI Logging Analytics for Oracle Log Ingestion

This is the step most teams skip. Don't.

1. OCI Console → Observability & Management → Logging Analytics → Log Sources

2. Enable the pre-built Oracle log parsers: Oracle Database Alert Log, Oracle Listener Log, Oracle Audit Log

3. Configure log collection via the Management Agent's log collection plugin — edit /etc/oracle/mgmt_agent/conf/emd.properties to include log paths

4. In Logging Analytics, create a Log Group per environment (prod, non-prod) and assign retention policies

Within 24 hours you'll have correlated log views across your DB tier that would take days to build manually.

Step 4: Align OCI Monitoring Alarms with Your OEM Baselines

This is where most teams skip and regret it later. If OEM already has tuned metric thresholds, align your OCI Monitoring alarms to the same values. Mismatched thresholds = duplicate noise.

In OCI Monitoring: Alarms → Create Alarm → reference the same threshold values you've set in OEM metric templates.

For the same metric (e.g., CPU utilization on the DB host), pick one alerting path:

• OEM → incidents for the Oracle DBA team

• OCI Monitoring → OCI Notifications → ops/cloud team via PagerDuty, Slack, or email

Split by audience, not by metric.

Step 5: Push OEM Incident Data to OCI (Optional but High Value)

If you want a single pane across both environments, use the OEM REST API to forward incident data to OCI Events or a custom endpoint.

OEM 24ai exposes the Incident Manager API at:

https://<OMS_HOST>:7803/em/api/incidents

A simple poller (Python, cron, or OCI Functions) can sync open OEM incidents into an OCI Notifications stream. This isn't a native integration today — it's a custom bridge — but I've implemented it at several enterprise customers and it's worth the 2-day effort. The payoff is one unified incident view for leadership that doesn't require toggling between tools.

What You Get When It's Connected

When OEM 24ai and the OCI Observability suite are both running and your data is bridged, here's what actually changes:

For your DBA team: Nothing changes. OEM is still their primary tool. Same workflows, same console, same EM CLI scripts.

For your cloud ops team: They get Oracle DB health in OCI Console via Database Management — without needing OEM access. Less friction, fewer permissions headaches.

For security and compliance: Logging Analytics ingesting Oracle audit logs gives you a queryable, correlated audit trail across the full estate. No more ad-hoc log file archaeology.

For your leadership: One dashboard showing full-stack health — from Exadata cells to OCI Compute. No more "which screen shows the real status?"

For incident response: When your application degrades, Logging Analytics' correlation view shows you whether the issue is a DB wait event, a listener failure, or an OS-level resource constraint — in one query, not three separate tools.

The Bottom Line

OEM 24ai is the best Oracle database monitoring tool that exists. Nothing comes close for deep Oracle internals, lifecycle management, and Exadata coverage.

The OCI Observability suite — Monitoring, Logging Analytics, Database Management, Ops Insights — is the best stack for infrastructure telemetry, log intelligence, fleet analytics, and cloud-native alerting integration.

If you've been running OCI Stack Monitoring, start planning your migration now. EOL is January 23, 2027, which sounds far away until you're mid-fiscal-year trying to retrofit monitoring architecture in production.

The teams that try to pick one tool end up with gaps. The teams that connect OEM with the full OCI Observability stack end up with the kind of visibility that actually lets you sleep at night.

Start with the Management Agent deployment. It's the lightest-weight step and it unlocks everything else.

Next in this series: Setting up proper incident rules and notification routing in OEM 24ai — because getting an alert is useless if the wrong person gets it at the wrong time in the wrong format.