Execute ASM Rebalance 1 time while adding & droppping disks

Sometimes it is required to add disks and drop disks sequentially in ASM environments. For example diskgroup migration between SAN , disk arrays etc .. , the disks are added and dropped . In this situation as default rebalance operation executes after every add & drop statements . 

alter diskgroup DATA add disk 

'ASMNEWDDISK_1',

'ASMNEWDDISK_2',

'ASMNEWDDISK_3' rebalance power 4;

alter diskgroup DATA drop disk 

'ASMOLDDDISK_1,

'ASMOLDDDISK_2,

'ASMOLDDDISK_3' rebalance power 4;

With this above statements rebalance operation executes 2 times , and not necessary. In order to execute rebalance in one operation , following statement can be used.

alter diskgroup DATA add disk 

'ASMNEWDDISK_1',

'ASMNEWDDISK_2',

'ASMNEWDDISK_3' 

drop disk 

'ASMOLDDDISK_1,

'ASMOLDDDISK_2,

'ASMOLDDDISK_3' rebalance power 4;

Open Standby Database due to Archive Gap

Renewing of TEST or Development environments , clone of standby database , can be used. At my last renew process for TEST , i have used last backup of production system which was used to open standby database . Production system has been backed up and standby controlfile created . After successfull standby database creation , by using same backup sets  new TEST environment has been restored to another location.

Whenever with command "alter database activate standby database" to open standby database as active database , below error encountered.

alter database activate standby database

ALTER DATABASE ACTIVATE [PHYSICAL] STANDBY DATABASE (XXX)

Begin: Standby Redo Logfile archival

End: Standby Redo Logfile archival

Signalling error 1152 for datafile 1!

Beginning standby crash recovery.

Serial Media Recovery started

Managed Standby Recovery starting Real Time Apply

Media Recovery Waiting for thread 1 sequence 318427

Fetching gap sequence in thread 1, gap sequence 318427-318427

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:19 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:29 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:39 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:49 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:59 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:55:09 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Standby crash recovery need archive log for thread 1 sequence 318427 to continue.

Please verify that primary database is transporting redo logs to the standby database.

Wait timeout: thread 1 sequence 318427

Standby crash recovery aborted due to error 16016.

ORA-16016: archived log for thread 1 sequence# 318427 unavailable

Recovery interrupted!

Completed standby crash recovery.

Signalling error 1152 for datafile 1!

ORA-1152 signalled during: alter database activate standby database...

Database has requested archivelog (318427) and could not get from primary system and finally timed out. 

Actually database has been restored and recovered with last backup sets and archivelogs and at this point database can be used as TEST environment.

Because of type of controlfile which was standby , could cause this situation and thought recreate controlfile as primary controlfile.

Got controlfile script by using below command

"alter database backup controlfile to trace"

Created controlfile with above output of command with resetlogs and opened .

txkWfClone.sh Timed out

While executing first autoconfig for clone system , txkWfClone.sh gives error that autoconfig did not complete successfully,

AutoConfig could not successfully execute the following scripts:

  Directory: /testapps/R12/inst/apps/TEST_testapp/admin/install

    txkWfClone.sh           INSTALL_SETUP      -1

AutoConfig is exiting with status 1

From log file,

AutoConfig Setup Phase

Running Setup Process 7 of 8 for AD_TOP

Executing script in InstantiateFile:

/testapps/R12/inst/apps/TEST_testapp/admin/install/txkWfClone.sh

script returned:

****************************************************

Timed out( 3750000 ): Interrupted Exception

If Workflow tables have high row counts related error can be observed . Below tables can be reviewed for row counts

select count(*) from wf_resources where name = 'WF_WEB_AGENT' and language = 'US';

select count(*) from WF_ITEM_ATTRIBUTE_VALUES;

select count(*) from WF_ITEM_ATTRIBUTES;

select count(*) from wf_activity_attr_values;

select count(*) from wf_process_activities;

select count(*) from wf_activity_attributes;

select count(*) from wf_notification_attributes;

select count(*) from wf_message_attributes;

Firstly "Purge Obsolete Workflow RunTime Data" concurrent program can be executed on source system to get less data for clone system.

In my situation , i have already restored source system for clone which is nearly 15 TB and no chance to run related concurrent program before system configuration.

To solve my issue , after getting error from adcfgclone script , I have executed txkWfClone.sh as single process from command line and completed adcfgclone successfully.

Change database JDBC URL from deployed ear file

When i would like to change database JDBC URL of deployed application in weblogic from console , it gave below error 

Console encountered the following error java.lang.IllegalArgumentException: [J2EE Deployment SPI:260140]Unable to remove descriptor bean 

weblogic.j2ee.descriptor.wl.JDBCPropertyBeanImpl@179efc1([MYAPP]/JDBCDriverParams/Properties/Properties[user]) due to 'Unable to remove bean since not defined in plan'. The remover should first check to see if the bean is removable in the deployment plan prior to removing it.

Of course there is a way to solve this issue at weblogic side , but i prefered changing database url from ear file directly changing related jdbc.xml file

Action Plan,

• Stop WebLogic
• Locate ear file of application 
• Backup ear file 
• Unzip ear file
• unzip XXXX_myapp.ear
• Change url entitiy in META-INF/MYAPP-jdbc.xml to desired url
• Save MYAPP-jdbc.xml file
• Using jar , update relate ear file
• jar uvf XXXX_myapp.ear META-INF/MYAPP-jdbc.xml 
• Start Weblogic

You can observe the change from weblogic console 

Capture DDL of Dumpfile

Sometimes dump files are wanted to import your system . But if you do not know source system structure import process can be trouble.Without knowing source tablespace , schema knowledge  import will produce errors .

In order to solve this issue and do successful import , impdp are called with sqlfile parameter to get all DDL statements . Output contains all DDL statements and all necessary statements can be captured .

impdp $DBUSER directory=$DIRECTORY dumpfile=$DUMPFILE.dmp sqlfile=impfile.sql

After executing above statement impfile.sql generated.For example tablespaces can be seen with ,

grep TABLESPACE impfile.sql

command which lists all "TABLESPACE" statements . You can create tablespaces according to output or "remap_tablespace" parameter for impdp is used to reconfigure tablespaces according to your system.

ORA-01378 The logical block size error

For some reason client asked me to open production database in another environment until specific time.After successfull restore and recover process , clear logfile gave the below error ,

ORA-01378: The logical block size (4096) of file +RECO is not compatible with the disk sector size (media sector size is 512 and host sector size is 512)

Production system 's redo log files have 4K sector size and clearing log file on 512 byte is not possible.

Recreating controlfiles without redo logs statements which are 4K size has solved our issue.I have added new redo logfiles to system before recreating controlfile which area group 11,12,13

create controlfile reuse database "testdb" noresetlogs force logging archivelog

  maxlogfiles 

  max...

  ...

logfile2

   group 2('+RECO/TESTDB/log2a','+RECO/TESTDB/log2b') size 100M blocksize 4096,

   group 11('+RECO/TESTDB/log11a','+RECO/TESTDB/log11b') size 100M blocksize 512,

   group 12('+RECO/TESTDB/log12a','+RECO/TESTDB/log12b') size 100M blocksize 512

   group 13('+RECO/TESTDB/log13a','+RECO/TESTDB/log13b') size 100M blocksize 512

datafile

...

...

New Controlfile script : 

create controlfile reuse database "testdb" noresetlogs force logging archivelog

  maxlogfiles 

  max...

  ...

logfile2

   group 11('+RECO/TESTDB/log11a','+RECO/TESTDB/log11b') size 100M blocksize 512,

   group 12('+RECO/TESTDB/log12a','+RECO/TESTDB/log12b') size 100M blocksize 512

   group 13('+RECO/TESTDB/log13a','+RECO/TESTDB/log13b') size 100M blocksize 512

datafile

...

...

/usr/bin/ld: cannot find -ljava

When applying patch to Oracle EBS , below error occured while relinking. Because of clone system , some enviroments were still pointing old home enviroments

/usr/bin/ld: cannot find -ljava

For this error , entry "JRE_LIB_DIR" in $ORACLE_HOME/sysman/lib/env_sysman.mk has been checked if it points true directory.After adjusting to proper directory , relink has been completed successfully.