Bulk relocation of OEM targets related to former agents

I currently work on a 4-node cluster with Grid Infrastructure installed, hosting several 11g and 12c databases. As it is a cold failover cluster, databases are not RAC databases and are active on only one node at a time.

Thoses databases share the same action script. The “start” section of this action script is quite complex and also manages the relocation of the database target on Enterprise Manager But this specific task is time-consuming (approximately 30 seconds for 1 target) and is not always successful. Which means I often have to relocate some targets in Enterprise Manager by hand.

Recently, 2 nodes of our cluster did not survive a major storage outage. After their reboot, the databases restarted (quite slowly) according to the action script. But target relocation did not work as expected for all targets and I had to manually fix the mess.

After this experience, we decided with my team to completely remove the target relocation from the action script and take care of this specific task separately, applying the following method :

    • Find all the databases running on the local server, except ASM proxy instance, ASM instance and MGMTDB instance :
ps -ef | grep [p]mon | egrep -v '\+|\-'

(And format the result as you want … I used an awful chain of sed …)

    • From Enterprise Manager repository, generate the list of targets (and their wrongly associated agent) that should be related to the local agent but are not :
SELECT target_name, agent_name 
FROM mgmt$agents_monitoring_targets
WHERE target_name IN ('<list_of_databases_running_locally>')
AND agent_name != '<local_server:agent_port>';
    • For each target, relocate it with emcli :
emcli relocate_targets \
-src_agent=<agent_name_from_previous_query> \
-dest_agent=<local_server:agent_port> \
-target_name=<target_name> \
-target_type=oracle_database \
-copy_from_src \

Here are the descriptions of copy_from_src and force emcli parameters according to official documentation :

copy_from_src : Use the same type of properties from the source Management Agent to identify the target. 
This is a MANDATORY parameter. 
Not supplying this parameter may result in the corruption of the target definition.
force : Force dependencies (if needed) to failover as well.

Keep in mind that relocating several targets at once may cause the agent to need a lot of memory, so you might need to tune agent memory settings.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s