Cloning an Oracle eBusiness Suite environment from one instance to another is one of the most common tasks required of an Oracle Apps DBA.
The basic process is documented in Oracle Note Cloning Oracle Applications Release 12 with Rapid Clone (Doc ID 406982.1), but the actual steps required can be surprisingly different from system to system. It can depend on factors such as the size of the database; the backup method being used; whether there is direct connectivity between the source and target environments; whether there are any snapshot technologies available at storage level; what filesystem is in use and so on.
Nevertheless, there are some common tips and tasks that apply to all systems, which are not often documented by Oracle, but an experienced Apps DBA will always tend to include.
The list below is hardly exhaustive, but provides a flavour of the sort of thing a good Apps DBA should be doing as a standard for your business!
The first step of any clone is to run “perl adpreclone.pl” on the database tier and each application tier to prepare the source system for cloning.
If you are therefore planning on performing a clone based on last night’s backup, you will have had to manually run pre-clone at some point before the backup started. But, what if the clone request has come through as an emergency and you didn’t know this had to be done?
One option is to run pre-clone on your source environment and then take a new backup of the appsutil directories on database tier and each application tier, however, this adds an unnecessary manual step to the clone.
Instead, schedule a job to automatically run pre-clone on each tier on the source environment as a nightly job, either as a standalone task or as part of the backup itself.
You can then be confident that any backup you use as the source of your clone will already have had pre-clone run against it.
The script to do this can be very simple – on UNIX something like this would do the job nicely (with some appropriate logging around it)
. <APPL_TOP>/APPS<SID>_<server>.env
cd $INST_TOP/admin/scripts
perl adpreclone.pl appsTier
This can then be scheduled to run in the crontab.
If you have a Dataguard-managed DR database, chances are that this will either be on the same server – or at the very least in the same datacentre –where your non-production environments are hosted.
You can therefore save considerable time and network bandwidth by cloning from DR to Non-Prod as opposed, to Prod to Non-prod. You can temporarily shut down your DR environment and perform a simple cold copy of the datafiles to their new location, or use a backup of your DR environment as the clone source.
Once you backup, copy and extract the database and application tier filesystems to the target environment, there will be a number of soft links which will potentially point to a production file path which is not a valid location on the target.
On UNIX, the following command can be run to list all soft links which are present in the current directory and downwards:
find . -type l -ls -exec ls -lrt {} \;
Typically, you will find some in your CUST_TOP/bin directory if your business has developed some bespoke shell script code called via a concurrent request. These will vary from site to site, although there are also always some more to be found in the standard eBusiness Suite filesystem.
Usually, the application will work perfectly fine with these pointing to invalid paths, however, it can become an issue when you try to patch or relink the technology stack components on the target environment. So, I usually always try to correct these as a standard post-clone step.
On eBusiness Suite 12.1 application tier:
cd <10.1.2_ORACLE_HOME>
cd bin
ln -sf <10.1.2_ORACLE_HOME>/webcache/bin/webcachectl/webcachectl webcachectl
ln -sf <10.1.2_ORACLE_HOME>/nls/lbuilder/lbuilder lbuilder
cd ../webcache/examples
ln -sf <10.1.2_ORACLE_HOME>/webcache/docs/readme.examples.html readme.examples.html
cd ../../sysman/webapps/emd/WEB-INF/lib
ln -sf <10.1.2_ORACLE_HOME>/sysman/jlib/log4j-core.jar log4j-core.jar
cd <10.1.3_ORACLE_HOME>
cd bin
ln -sf <10.1.3_ORACLE_HOME>/nls/lbuilder/lbuilder lbuilder
Database tier:
cd <ORACLE_HOME>
cd lib
ln -sf <ORACLE_HOME>/lib/libclntsh.so libclntsh.so.10.1
cd ../bin
ln -sf <ORACLE_HOME>/nls/lbuilder/lbuilder lbuilder
Chances are the $ORACLE_HOME/oraInst.loc file in each Oracle Home on the extracted filesystem will point to a location which is not valid for the target environment.
The clone process will still work fine and the application will come up normally, but you may find that opatch complains about inventory problems, if you try to later apply a database or application server patch onto a cloned environment.
My trick to avoid this is to always have a step to correct the paths to the oraInventories before you run adcfgclone.pl
On the database tier:
vi <ORACLE_HOME>/oraInst.loc
inventory_loc=<ORACLE_HOME>/oraInventory
inst_group=dba
mkdir -p <ORACLE_HOME>/oraInventoryOn the application tier:
vi <10.1.3_ORACLE_HOME>/oraInst.loc
inventory_loc=<10.1.3_ORACLE_HOME>/oraInventory
inst_group=dba
vi <10.1.2_ORACLE_HOME>/oraInst.loc
inventory_loc=<10.1.2_ORACLE_HOME>/oraInventory
inst_group=dba
mkdir -p <10.1.3_ORACLE_HOME>/oraInventory
mkdir -p <10.1.2_ORACLE_HOME>/oraInventory
There will almost certainly be a number of standard and custom directories defined on your source environment and these may point to paths that are not valid for the target instance.
As a post-clone step, always check to see if there are any that should be updated:
SELECT directory_name, directory_path
FROM dba_directories;
CREATE OR REPLACE DIRECTORY <name> as <new_path>;
Your application may have additional, custom paths defined in UTL_FILE_DIR and these paths may not be valid for the target instance.
As a post-clone step, always check to see if there are any that should be updated.
In addition, by default the first entry in UTL_FILE_DIR will be /usr/tmp on a UNIX system and the application will use this to create temporary files for concurrent requests (the $APPLPTMP directory). This can lead to problems with permissions on files in that, if there is more than one environment hosted on the same server and all environments are using the same location. As a best practice in that situation, I would always change the first entry in UTL_FILE_DIR to be /usr/tmp/<TARGET_ENV> and set $APPLPTMP on the application tier to this directory (making sure to create this directory first of course!)
Most standard Oracle profile options will be corrected on the target environment when AutoConfig is run. However, there are usually some that don’t get updated, plus whatever custom profile options you may have defined which have to be manually updated.
The following script can be run to quickly list all profile options which may point to production paths, names or ports and which may need to be corrected
select decode(pov.level_id, '10001', 'SITE',
'10002', 'APPLICATION',
'10003', 'RESPONSIBILITY',
'10004', 'USER',
'10005', 'SERVER',
'10007', 'SERVRESP',
'Undefined') SET_AT_LEVEL,
decode(pov.level_id, '10002', fa.application_short_name,
'10003', frt.responsibility_name,
'10004', fu.user_name,
'10005', fn.node_name,
'10006', org.name,
'10007', 'Serv/resp',
pov.level_value) ASSIGNED_TO,
po.profile_option_name,
pot.user_profile_option_name,
pov.profile_option_value,
po.profile_option_id
from applsys.fnd_profile_options po,
applsys.fnd_profile_options_tl pot,
applsys.fnd_profile_option_values pov,
applsys.fnd_responsibility_tl frt,
applsys.fnd_user fu,
applsys.fnd_application fa,
applsys.fnd_nodes fn,
hr_operating_units org
where po.profile_option_id = pov.profile_option_id
and po.profile_option_name = pot.profile_option_name
and pov.level_value = frt.responsibility_id (+)
and pov.level_value = fu.user_id (+)
and pov.level_value = fa.application_id (+)
and pov.profile_option_value like '%prod%'
and fn.node_id (+) = pov.level_value
and org.organization_id (+) = pov.level_value
order by 3,1,2
Change the highlighted text to what is appropriate for your environment – run the script a few times to check for the source database name, server names and ports.
Once you have identified that some profile options need to be changed, this can either be done in the front-end or more quickly can be done via a script – this also saves having to start the application, change the profile options and then bouncing the application again.
This script will look something like this:
DECLARE
value1 Boolean;
value2 Boolean;
BEGIN
value1 := fnd_profile.save('<PROFILE_OPTION1>','<VALUE>','SITE');
value2 := fnd_profile.save('<PROFILE_OPTION2>','<VALUE>','SITE');
END;
/
commit;
I would also generally recommend always changing these profile options:
At database level (11gR1 and later), alert logs, trace files and dump files are stored in the Automatic Diagnostic Repository (ADR) and are then purged from the system at defined intervals.
It is not usually required to retain these files on non-production environment for as long as it would be on the live system, so disk space can be saved by reducing the ADR retention policies on the non-production environments.
Diagnostic data Purging in ADR is controlled by two attributes:
I usually reduce these to 6 days and 36 days respectively with the ADRCI command line utility as follows:
adrci
adrci> show homes
ADR Homes:
diag/rdbms/<env>/<ENV>
diag/tnslsnr/<server>/<env>
adrci> set homepath diag/rdbms/<env>/<ENV>
adrci> set control (SHORTP_POLICY = 360)
adrci> set control (LONGP_POLICY = 2160)
adrci> set homepath diag/tnslsnr/<server>/<env>
adrci> set control (SHORTP_POLICY = 360)
adrci> set control (LONGP_POLICY = 2160)
By default you will get a 1GB SGA and 1GB PGA in a cloned environment.
This might be sufficient, but for a pre-prod environment you should try and match production values; I also generally see better stability if I provide an SGA of at least 2GB.
Always define a test/override email address on cloned environment to prevent spurious emails being sent out to users, employees and suppliers – this will ensure that any emails that are sent out will all go to a ‘bucket’ email address instead.
This can be set in a 12.1 environment by running script $FND_TOP/sql/afsvcpup.sql
Also check and correct the Workflow-related section of the AutoConfig $CONTEXT_FILE (the section tagged <!– Workflow –>) so that the values for SMTP server, IMAP server, usernames and Reply To address are valid for the target environment.
If you are using an IMAP server for Workflow inbound processing, define a separate mailbox for each environment in your estate. This prevents locking issues with too many Workflow Mailers trying to poll the same mailbox at the same time, or emails valid for one environment being moved to the DISCARD folder by the Mailer for another environment.
As good practice always change the passwords in your cloned environment so that they are something different from the source environment, especially when cloning from production.
Typically, change the following passwords:
The number of workshifts defined for each of your concurrent Managers will be optimised for the production environment. But, this may not be appropriate for a cloned, non-production environment which may be running on a server with fewer hardware resources and which will never expect the same workload as the live environment.
In consultation with the expected users of the environment (developers, testers etc) see whether any concurrent Managers can have their workshifts reduced to lessen the impact on the server.
There may be a number of overnight batch jobs scheduled to run the live environment. Are all of these also required on the non-production systems? If not – again in consultation with the expected users of the target environment – see whether any pending requests can be cancelled or on the cloned environment to reduce overnight server utilisation on the non-production servers.
There’s a couple of questions to ask yourself before doing this:
If you have no requirements to backup/restore your non-prod environments then it makes no sense for them to be in ARCHIVELOGMODE – therefore reduce server disk space utilisation by putting the target environment into NOARCHIVELOGMODE as a post-clone step.
Your production server will have been optimised to run with X amount of parallel JVM threads in the oacore, forms and oafm groups as appropriate for the expected user workload and hardware capacity on the production servers.
These values will be inherited by the cloned environments but will probably be over-powered on a development environment, so should be reduced in the $CONTEXT_FILE and AutoConfig re-run e.g. s_oacore_nprocs to eliminate unnecessary CPU and memory consumption on the non-production server.
If you have a CUSTOM_TOP defined in a custom environment file in your APPL_TOP, then make sure that the file exists on the target environment with the correct name and pointing to the correct paths and directories.
Always look for ways to improve or speed up the clone process on each iteration:
And finally the most important tip of all – clearly document the clone process in detail and always keep the document up to date!
Always write the document such that another DBA would be able to follow the process for the first time; create the sort of document that you would like if you were having to perform a clone on a new system.