OBIEE Error ADF_FACES-60097 ADF_FACES-60096

When selecting "Coreapplication" in enterprise manager of OBIEE following error occured,

[2018-02-22T21:17:13.868+02:00] [AdminServer] [NOTIFICATION] [] [oracle.bi.management.adminservices] [tid: DmsThread-2] [userId: <WLS Kernel>] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:-8000-000000000000007d,1:16628] 
[APP: biadminservices#11.1.1] BIInstanceStatusMetricMBeanImpl; getServiceStatus: currentPrincipal is null
[2018-02-22T21:17:14.512+02:00] [AdminServer] [NOTIFICATION] [OBI-SYSMAN-1230] [oracle.bi.management.sysmancommon] [tid: DmsThread-2] [userId: <WLS Kernel>] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:-8000-00000000000000
7d,1:16628] [APP: biadminservices#11.1.1] Summary of current system component statuses; Up: 6; Down: 0; Up(HA): 0; Down(HA): 0; Error: 0
[2018-02-22T21:17:14.514+02:00] [AdminServer] [NOTIFICATION] [] [oracle.bi.management.adminservices] [tid: DmsThread-3] [userId: <WLS Kernel>] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:-8000-000000000000007d,1:16626] [A
PP: biadminservices#11.1.1] BIInstanceStatusMetricMBeanImpl; getServiceStatus: currentPrincipal is null
[2018-02-22T21:17:15.158+02:00] [AdminServer] [NOTIFICATION] [OBI-SYSMAN-1230] [oracle.bi.management.sysmancommon] [tid: DmsThread-3] [userId: <WLS Kernel>] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:-8000-00000000000000
7d,1:16626] [APP: biadminservices#11.1.1] Summary of current system component statuses; Up: 6; Down: 0; Up(HA): 0; Down(HA): 0; Error: 0
[2018-02-22T21:17:15.236+02:00] [AdminServer] [WARNING] [] [oracle.mds] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:
-8000-0000000000000137,0] [APP: em] A call to flushChanges on the current MDSSession does not specify the correct transaction key
[2018-02-22T21:17:15.241+02:00] [AdminServer] [ERROR] [ADFC-50018] [oracle.adfinternal.controller.application.AdfcExceptionHandler] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] 
[userId: weblogic] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:-8000-0000000000000137,0] [APP: em] ADFc: No exception handler was found for an application exception.
[[javax.faces.FacesException: java.io.IOException: Stream closed

[2018-02-22T21:17:15.243+02:00] [AdminServer] 
[WARNING] [] [oracle.adfinternal.view.faces.lifecycle.LifecycleImpl] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 7326609a41c1cc90:101d0126:161bee9fb6b:-8000-0000000000000137,0] [APP: em] ADF_FACES-60098:Faces lifecycle receives unhandled exceptions in phase RENDER_RESPONSE 6[[javax.faces.FacesException: java.io.IOException: Stream closed

Error can be observed from AdminServer-diagnostic logs.

According to metalink if your system ran out of disk space in any time this corrupted some files and caused this issue.In order to solve this issue,

1.Stop all OBIEE services and weblogic server

2.Find 0 byte files under $DOMAIN_HOME/sysman/mds directory

   cd $DOMAIN_HOME/sysman/mds

   find . -name "*" -size 0 

3.delete these files

4.Open weblogic and OBIEE services

Now "Coreapplication" should be accessible again and deleted files will be recreated.

Execute ASM Rebalance 1 time while adding & droppping disks

Sometimes it is required to add disks and drop disks sequentially in ASM environments. For example diskgroup migration between SAN , disk arrays etc .. , the disks are added and dropped . In this situation as default rebalance operation executes after every add & drop statements . 

alter diskgroup DATA add disk 

'ASMNEWDDISK_1',

'ASMNEWDDISK_2',

'ASMNEWDDISK_3' rebalance power 4;

alter diskgroup DATA drop disk 

'ASMOLDDDISK_1,

'ASMOLDDDISK_2,

'ASMOLDDDISK_3' rebalance power 4;

With this above statements rebalance operation executes 2 times , and not necessary. In order to execute rebalance in one operation , following statement can be used.

alter diskgroup DATA add disk 

'ASMNEWDDISK_1',

'ASMNEWDDISK_2',

'ASMNEWDDISK_3' 

drop disk 

'ASMOLDDDISK_1,

'ASMOLDDDISK_2,

'ASMOLDDDISK_3' rebalance power 4;

Open Standby Database due to Archive Gap

Renewing of TEST or Development environments , clone of standby database , can be used. At my last renew process for TEST , i have used last backup of production system which was used to open standby database . Production system has been backed up and standby controlfile created . After successfull standby database creation , by using same backup sets  new TEST environment has been restored to another location.

Whenever with command "alter database activate standby database" to open standby database as active database , below error encountered.

alter database activate standby database

ALTER DATABASE ACTIVATE [PHYSICAL] STANDBY DATABASE (XXX)

Begin: Standby Redo Logfile archival

End: Standby Redo Logfile archival

Signalling error 1152 for datafile 1!

Beginning standby crash recovery.

Serial Media Recovery started

Managed Standby Recovery starting Real Time Apply

Media Recovery Waiting for thread 1 sequence 318427

Fetching gap sequence in thread 1, gap sequence 318427-318427

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:19 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:29 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:39 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:49 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:54:59 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Wed Jan 24 23:55:09 2018

FAL[client]: Error fetching gap sequence, no FAL server specified

Standby crash recovery need archive log for thread 1 sequence 318427 to continue.

Please verify that primary database is transporting redo logs to the standby database.

Wait timeout: thread 1 sequence 318427

Standby crash recovery aborted due to error 16016.

ORA-16016: archived log for thread 1 sequence# 318427 unavailable

Recovery interrupted!

Completed standby crash recovery.

Signalling error 1152 for datafile 1!

ORA-1152 signalled during: alter database activate standby database...

Database has requested archivelog (318427) and could not get from primary system and finally timed out. 

Actually database has been restored and recovered with last backup sets and archivelogs and at this point database can be used as TEST environment.

Because of type of controlfile which was standby , could cause this situation and thought recreate controlfile as primary controlfile.

Got controlfile script by using below command

"alter database backup controlfile to trace"

Created controlfile with above output of command with resetlogs and opened .

txkWfClone.sh Timed out

While executing first autoconfig for clone system , txkWfClone.sh gives error that autoconfig did not complete successfully,

AutoConfig could not successfully execute the following scripts:

  Directory: /testapps/R12/inst/apps/TEST_testapp/admin/install

    txkWfClone.sh           INSTALL_SETUP      -1

AutoConfig is exiting with status 1

From log file,

AutoConfig Setup Phase

Running Setup Process 7 of 8 for AD_TOP

Executing script in InstantiateFile:

/testapps/R12/inst/apps/TEST_testapp/admin/install/txkWfClone.sh

script returned:

****************************************************

Timed out( 3750000 ): Interrupted Exception

If Workflow tables have high row counts related error can be observed . Below tables can be reviewed for row counts

select count(*) from wf_resources where name = 'WF_WEB_AGENT' and language = 'US';

select count(*) from WF_ITEM_ATTRIBUTE_VALUES;

select count(*) from WF_ITEM_ATTRIBUTES;

select count(*) from wf_activity_attr_values;

select count(*) from wf_process_activities;

select count(*) from wf_activity_attributes;

select count(*) from wf_notification_attributes;

select count(*) from wf_message_attributes;

Firstly "Purge Obsolete Workflow RunTime Data" concurrent program can be executed on source system to get less data for clone system.

In my situation , i have already restored source system for clone which is nearly 15 TB and no chance to run related concurrent program before system configuration.

To solve my issue , after getting error from adcfgclone script , I have executed txkWfClone.sh as single process from command line and completed adcfgclone successfully.