How to troubleshoot Concurrent Manager in 11i/R12 – Apps Listener


How to check if Concurrent Manager is Up and running ?

Login as user with System Administrator responsibility. Under Concurrent : Manager click on Administer (Actual should be equal to Target)

How to troubleshoot Concurrent Manager issue ?
Check Concurrent Manager log $SID_[MMDD] where MMDD is Concurrent Manager start Month & Date
 1. $APPLCSF/$APPLLOG/$SID_[MMDD].mgr

Could NOT contact Service Manager FNDSM_INNOWAVE03_visr12. 
 
The TNS alias could NOT be located, the listener process onINNOWAVE03 could NOT be contacted, 
 
OR the listener failed TO spawn the Service Manager process. 
 
     Process monitor session started : 24-SEP-2009 20:31:07Could NOT contact Service Manager FNDSM_INNOWAVE03_visr12. 
 
The TNS alias could NOT be located, the listener process onINNOWAVE03 could NOT be contacted, 
 
OR the listener failed TO spawn the Service Manager process.

2. Error message is pointing towards “FNDSM_INNOWAVE03_visr12″ so run tnsping to check status of TNS alias like

[avisr12@innowave03 ~]$ tnsping FNDSM_INNOWAVE03_visr12

3. Check Apps Listener status like
[avisr12@innowave03 ~]$ lsnrctl status APPS_visr12
LSNRCTL for Linux: Version 10.1.0.5.0 - 
 
Production on 24-SEP-2009 20:37:36Copyright (c) 1991, 2004, Oracle.  
 
All rights reserved.Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=innowave03.com)(Port=1638)) 
 
TNS-12541: TNS:no listener  
 
TNS-12560: TNS:protocol adapter error  
 
TNS-00511: No listener   
 
Linux Error: 111: Connection refused.

4. Start Apps Listener
[avisr12@innowave03 scripts]$ ./adalnctl.sh start

output like

adalnctl.sh version 120.3 
 
Checking for FNDFS executable. 
 
Starting listener process APPS_visr12. 
 
adalnctl.sh: exiting with status 0adalnctl.sh: check the logfile /oracle/ apps/ r12/ visr12/ inst/ apps/ visr12_innowave03/ logs/ appl/ admin/ log/ adalnctl.txt formore information ... 

5. Try again tnsping again
[avisr12@innowave03 scripts]$ tnsping FNDSM_INNOWAVE03_visr12
output like
TNS Ping Utility for Linux: Version 10.1.0.5.0 - 
 
Production on 24-SEP-2009 20:39:09Copyright (c) 1997, 2003, Oracle.  
 
All rights reserved.Used parameter files:Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=innowave03.com)(PORT=1638)) (CONNECT_DATA=(SID=FNDSM))) 
 
OK (0 msec)

6. Restart CM again.
If you like this post please comment ......

Concurrent manager issues - 11i / R12

Concurrent manager issues - 11i / R12

Inactive/No manager

A concurrent request has a life cycle consisting of the following phases: Pending, Running, and Completed. During each phase, a request has a specific status. Listed below are the possible statuses for each phase:

•Pending Phase - Normal, Standby, Scheduled, Waiting

•Running Phase - Normal, Paused, Resuming, Terminating

•Completed Phase - Normal, Error, Warning, Cancelled, Terminated

•Inactive Phase - Disabled, On Hold, No Manager

If a concurrent request is on hold or unable to run when there are no active manager processes that can run the request, the request is placed in an Inactive phase.

Review the following points when the concurrent request is in Inactive phase with No Manager status.

1. Verify that Internal Concurrent Manager(ICM) is up and running. Use any one navigation mentioned below to check the status details of Internal Manager.

i. Oracle Applications Manager(OAM) > Site Map > Monitoring > Availability > Internal Concurrent Manager > View Status.

OR

ii. System Administrator Responsibility > Concurrent > Manager > Administer

2. Verify that there is at least one active concurrent manager with/without specialization rules that allow the concurrent program to run.

i. Run the following query to check whether any specialization rule defined for any concurrent manager that includes/excludes the concurrent program in question. Query returns 'no rows selected' when there are no Include/Exclude specialization rules of Program type for the given concurrent program.

select 'Concurrent program '

fcp.concurrent_program_name

' is '

decode(fcqc.include_flag,'I','included in ','E','excluded from ')

fcqv.user_concurrent_queue_name specialization_rule_details from fnd_concurrent_queues_vl fcqv,fnd_concurrent_queue_content fcqc,fnd_concurrent_programs fcp where fcqv.concurrent_queue_id=fcqc.concurrent_queue_id and fcqc.type_id=fcp.concurrent_program_id and fcp.concurrent_program_name='';

Note: Program Short Name is visible when the program is queried in concurrent program definition form.

Example:

SQL> select 'Concurrent program '

fcp.concurrent_program_name

' is '

decode(fcqc.include_flag,'I','included in ','E','excluded from ')

fcqv.user_concurrent_queue_name specialization_rule_details from fnd_concurrent_queues_vl fcqv,fnd_concurrent_queue_content fcqc,fnd_concurrent_programs fcp where fcqv.concurrent_queue_id=fcqc.concurrent_queue_id and fcqc.type_id=fcp.concurrent_program_id and fcp.concurrent_program_name='XXRFG3041A';

SPECIALIZATION_RULE_DETAILS

-----------------------------------------------------------------------------

Concurrent program OKCRAQE is included in Contracts Core Concurrent Manager

Concurrent program OKCRAQE is excluded from Standard Manager

From the sample output above, it shows that the OKCRAQE(Listener for Events Queue) concurrent program has been excluded from the Standard Manager and included in Contracts Core Concurrent Manager. That means the concurrent request OKCRAQE can be run only by the Contracts Core Concurrent Manager which should be up and running to run and complete the OKCRAQE concurrent request.

Make sure that Concurrent Manager whose specialization rule includes the concurrent program is up and running.

ii. Ensure that standard concurrent manager is up and running.

Follow the below step only when you have confirmed the previous points and the issue is still remaining as there may be an issue with concurrent request queue view.

3. Manually re-create the concurrent request queue view for concurrent managers by entering the following command as an applmgr user at operating system prompt.

FNDLIBR FND FNDCPBWV apps/pass SYSADMIN 'System Administrator' SYSADMIN

-----------------------------------------------------------------------------------------------------------

Pending Standby

Check CRM queue,

Click on Application developer Responsibility --> Concurrent --> Program

Check for incompatibilities by clicking on Incompatibilities button.

If scheduled program has a conflict with other program then CRM will make sure to run the pending requests once the conflicting requests are completed .

Found dead process in ICM log file 

There is an issue occurred on my E-Business Suite.Many concurrent mangers are not started after restarted the concurrent manger server.The Internal Manager log file shows that the following information.

Found dead process: spid=(12099), cpid=(1125806),
ORA pid=(123), manager=(0/15293)

Found dead process: spid=(12053), cpid=(1125722),
ORA pid=(104), manager=(0/14293)

This was happened due to the controlling manager was not brought down gracefully.Gracefully meaning that it was brought down when and only when, all concurrent processes are completed.I have applied the following solution to resolve this issue.

Solution:

Step1. Stop all middle tier services including the concurrent managers and verify that all APPS-Tier processes are down :

ps -fu<APPS User> | grep FND
                OR
ps -fu<APPS User> | grep FND | wc -l

The command will show all processes on OS level, which are running for the Concurrent Managers. It can take some minutes until all CCM processes are stopped.

Step2. Stop the Database and verify, that all Database processes are down.

Step3. Start the database.

Step4. Execute the CMCLEAN.SQL script from the referenced note below (don’t forget to commit).

For details, please see NOTE:134007.1 – ‘CMCLEAN.SQL – Non Destructive Script to Clean Concurrent Manager Tables’.

Step5. Start the Middle Tier Services including your Concurrent Manager.

Step6. Verify the issue .

Here  Second step and Third Steps are optional.Normally it s not necessary to stop the database for run the cmclean.sql.In my case the cmclean script is hanged when i do run it and its needed to restart the database.

How to cancel a concurrent request from db level? 

• The following query returns sid” and “serial#” once you have provided “request_id” of a Concurrent Program which is currently running.
• Take the values “sid” and “serial#” and kill the sessions at the database level.

Step1:Connect to SQLPLUS as the APPS user:

SQL> SELECT ses.sid,

 ses.serial#

 FROM v$session ses,

 v$process pro

 WHERE ses.paddr = pro.addr

 AND pro.spid IN (SELECT oracle_process_id

 FROM fnd_concurrent_requests

 WHERE request_id = &request_id);

Enter value for request_id:

Step2:Connect to SQLPLUS as the SYSTEM user:

SQL> ALTER SYSTEM KILL SESSION '&sid,&serial#'IMMEDIATE;

Enter value for sid:

Enter value for serial:

Concurrent Manager FAQs

Which Concurrent Managers (CCMs) are supported by the Sysadmin team?


    Internal Concurrent Manager (ICM)
    Standard Manager
    Conflict Resolution Manager  (CRM)

The setup of product-specific managers related issues should

be directed to the associated product group, i.e. the Inventory manager

Q: Where do the concurrent request output and log files reside?

A: $APPLCSF/out or $Product_Top/out for output files

$APPLCSF/log or $Product_Top/log for log files

Q: Where are temporary files located?

A: Temporary files reside in:

$APPLPTMP$APPLTMP

$REPORTS25_TMP (Release 11.0)

$REPORTS60_TMP (Release 11.5)

Q: How to change PMON  method to LOCK.

To check the process monitor (PMON) method status:

Connect as database user “APPS”.

SELECT profile_option_value

FROM fnd_profile_option_values

WHERE level_id = 10001

AND level_value = 0

AND application_id = 0

AND profile_option_id =

(SELECT profile_option_id

FROM fnd_profile_options

WHERE profile_option_name = ‘CONC_PMON_METHOD’);

This should return one row with a value of ‘LOCK’

If the value is ‘RDBMS’ or ‘OS’ run the following script:

$FND_TOP/sql/AFIMPMON.SQL – this will set the PMON method to LOCK instead of RDBMS.

Bounce the database. If running on NT, restart the NT Server completely.

Within 11.5 Applications, the only PMON method is LOCK.

Q: How to start and shutdown the CCM on Unix as APPS

A:  To Start:

STARTMGR sysmgr=APPS/passwd

To Shutdown:

concsub apps/<APPS password> sysadmin ‘System Administrator’ SYSADMIN CONCURRENT FND ABORT

11.5 => 11.5.4

adcmctl.sh apps/passwd start/stop prod

11.5.5 =>

adcmctl.sh start/stop apps/passwd prod

Q: How to terminate a concurrent request that cannot be cancelled?

A: Identify the request number to terminate. Shut down the concurrent managers and issue the following sql command as applsys:

update fnd_concurrent_requests

set status_code = ‘E’, phase_code = ‘C’

where Request_id = <reqnum>; (reqnum = request number)

Q: A Manager is not activating.

A: Check the count in sys.dual, system.dual, & apps.fnd_dual. There should only be one (1) row for each. If the count is greater, change to one only. This can be done from sqlplus as  follows:

As user apps:

select *

from SYS.DUAL;

Any extra rows should be deleted.

delete rownum

from SYS.DUAL;

(rownum = the row number to delete)

Any extra rows for apps.fnd_dual must be removed by performing the following SQL command:

delete from fnd_dual

where rownum < (select max(rownum) from fnd_dual);

Bounce the Concurrent Managers.

Q: Internal Concurrent Manager.

A: If the ICM should go down, requests will continue to run normally, except for ‘queue control’ requests.

If the ICM should go down, you can restart it with ‘startmgr’. There is no need to shut down the other managers first.

Q: How to purge requests that are in Pending status.

A: The Purge Concurrent Requests program will only purge requests that are in Completed status. Set the status of the Pending requests  to Completed

before purging them. The Internal Manager Must be DOWN!

Using sqlplus as APPLSYS perform the following:

UPDATE fnd_concurrent_requests

SET phase_code = ‘C’

WHERE phase_code = ‘P’.

Lets classify how to start troubleshooting CM

Issues related to startup/shutdown of CM
If you try to start CM using adcmctl.sh or adstrtall.sh script & you encounter problem in starting CM then check for log files. All information related to why Concurrent Manager not coming up is recorder in log file at $APPLCSF/$APPLLOG with default name as $SID_$DD ( where DD is date on which you started Concurrent Manager ). This Concurrent Manager naming convention will be listed in file adcmctl.sh file (Concurrent Manager startup file). If $APPLCSF is not set then you can look for this file in $FND_TOP$APPLLOG

ICM is Up but some managers are still down
As you know adcmctl.sh script starts ICM (Internal Concurrent Manager) and its duty of ICM to start other Managers. If you see ICM is up with some other managers & Issue is with only Particular Managers then look for log files of these managers .

Concurrent -> Manager -> Administer -> Select manager which is not starting -> Click on "Processes" -> Click on "Manager Log" to find out error message.
These individual manager log file, you can access under $APPLCSF/$APPLLOG directory as well.

If you error message is not enough to find out cause of problem then you can enable debug on CM log files . You can enable diagnostics by changing DIAG_PARAM="" to DIAG_PARAM="Y" in adcmctl.sh (CM Startup script file)

Other things you can try for CM startup Issues :
Try to bounce CM & see all FNDLIBR processes are cleared before restrarting CM. You can run cmclean.sql command to clear any Invalid State against any request or manager. Check for any messages in log file. Check status of Manager from administer Concurrent Manager Window.
Concurrent -> Manager -> Administer (Login as System Administrator Responsibility)

Concurrent Manager(CM) Trouble shooting :


For Concurrent manager(CM) trouble shooting

1. First check the CM is up or not by using below any one procedure.
      i. ps –ef|grep FNDLIBR
     ii. adcmctl.sh status apps/<appsPWD>
    iii. Login as system administrator responsibilities go to the below navigation Concurrent -> manager -> Administer -> see the  Actual and target for all the managers

2. If CM in down check the internal manager Logfile in $APPLCSF/APPLLOG location for errors.
3. If any errors related to FNDFS then check the Application listener STATUS , if it is not running start it.
4. If CM is up/running then log file and output file are not able see the user from the front end then check the Application listener is status if it is not running then start it.

5. If any user is complained some request is running long time then

   i. First check the CM status and ICM status.
  ii. Check any Locks on DB particularly related to that program.
 iii. Check any INVALID objects related to that program.
 iv. Check any incompatible requests/programs are running for that program.
Login as system administrator responsibility and query the long running request /program. Navigate to concurrent program define query that long running request in the same form select incompatible requests .
If any incompatible request is running then find the user name who is running that program inform them regarding the incompatibilities.

Concurrent Manager not starting after Cloning ?
CM not starting after Cloning ?


Once after doing Rapid cloning some of concurrent managers were not starting.
The issue was that the node was wrongly defined as source node in Administer Manager screen.
Then i checked in the FND_NODES table which wrongly had the source nodes as well apart from the target nodes.

I performed the following action plan which resolved the issue->

* Shutdown the apps services

* EXEC FND_CONC_CLONE.SETUP_CLEAN;

   COMMIT;

   EXIT;

* Ran AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers.

* Start the apps services.

Actually FND_CONC_CLONE.SETUP_CLEAN clears up the tables with nodes information and when we run autoconfig, it repopulates these tables with correct node information.