| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Workflow crahes, FDC cut |
« View previous topic :: View next topic » |
| Author |
Message
|
| vennela |
 Posted: Wed Jun 11, 2003 8:58 am Post subject: Workflow crahes, FDC cut |
 |
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
Has anyone seen this before
| Code: |
+-----------------------------------------------------------------------------+
| |
| MQSeries First Failure Symptom Report |
| ===================================== |
| |
| Date/Time :- Wednesday June 11 08:38:34 CDT 2003 |
| Host Name :- localhostxxx (AIX 5.1) |
| PIDS :- 5765B73 |
| LVLS :- 520 |
| Product Long Name :- MQSeries for AIX |
| Vendor :- IBM |
| Probe Id :- XC130003 |
| Application Name :- MQM |
| Component :- xehExceptionHandler |
| Build Date :- Dec 11 2002 |
| CMVC level :- p520-CSD06G |
| Build Type :- IKAP - (Production) |
| UserID :- 00000202 (fmc) |
| Program Name :- fmcemain |
| Process :- 00019410 |
| Thread :- 00000001 |
| QueueManager :- FMCQM |
| Major Errorcode :- STOP |
| Minor Errorcode :- OK |
| Probe Type :- HALT6109 |
| Probe Severity :- 1 |
| Probe Description :- AMQ6109: An internal MQSeries error has occurred. |
| Arith1 :- 11 b |
| |
+-----------------------------------------------------------------------------+ |
Workflow dies and there is an FDC being generated
It is happening very often
MQWF 332 SP4
MQ 52 CSD06
Thanks
-------
Venny |
|
| Back to top |
|
 |
| Ratan |
| Posted: Wed Jun 11, 2003 9:03 am Post subject: |
|
|
Grand Master
Joined: 18 Jul 2002
Posts: 1245
|
Venny, A Little confusion. Why would there be an FDC if workflow dies? Is MQ Still running? Try to access all queues fmcemain accesses and see if you can put to those queues.
_________________
-Ratan |
|
| Back to top |
|
|
| Ratan |
| Posted: Wed Jun 11, 2003 9:12 am Post subject: |
|
|
Grand Master
Joined: 18 Jul 2002
Posts: 1245
|
Venny, Read this link
http://www.mqseries.net/phpBB2/viewtopic.php?t=4104&highlight=amq6109
This is an extract from the above thread
| XC130003 |
|----------->
>-----------------------------------------------------------------------|
| This is the most common probe our exception handler generates. It |
| indicates that a thread connected to MQSeries ran into unexpected |
| trouble. However, it is possible that the thread which created the |
| FDC might have been running customer code at the time. If the |
| function stack shows only xcsFFST, and if the last two entries in the |
| trace show a return from an MQI call followed by an entry into xcsFFST|
| , then the problem is again likely to be in the customer's code. If |
| the process is one of the core queue manager processes, or if the |
| function stack shows other MQSeries functions, then this FDC suggests |
| a problem within MQSeries. |
_________________
-Ratan |
|
| Back to top |
|
|
| clindsey |
| Posted: Wed Jun 11, 2003 5:33 pm Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
Venny,
The arith field is 11 or 0xb which is a SIGSEGV. The execution server is abending, possibly due to incorrectly handling null data or something along that line. It should catch it with an assertion check and report the error more cleanly.
I think you will probably want to report this in a pmr. It would help if you
can determine what the scenario is that causing the abend. If not, you can export AMQ_ABORT_ON_EXCEPTION=TRUE before starting fmcamain. This will generate a core file. You can use dbx to display the call stack and that will give some idea about what the execution server was doing at the time. Change to the directory where the core file is & enter
'dbx'. When it prompts for the executable name, plug in /usr/lpp/fmc/bin/fmcemain and the core should load into memory.
Enter 'where > /tmp/stack.out' and then 'quit'. You can post the stack.out
file here if you want. I can take a look at it.
If you do open a pmr, the level 2 guys will want to see it.
Charlie |
|
| Back to top |
|
|
| srao6 |
| Posted: Tue Jul 22, 2003 2:26 pm Post subject: FDC errors |
|
|
Newbie
Joined: 09 Aug 2002
Posts: 5
|
We are facing a similar situation. We have MQ, MQWF and WAS running on the same AIX system. We currently have a PMR opened with IBM.
Your workflow dies because your qmanager is not available. the FDC files are created when you have internal MQ Series errors
This is what I have found so far, it points to the segmentation errors caused by the application which causes the shared memory segment conflicts. please review the following. I will update you as soon as I have any further information on the PMR.
Here's what I found from the IBM support site:
Problem
A customer reported that his Java bindings application failed to connect to MQSeries with a 2059 (MQRC_Q_MGR_NOT_AVAILABLE) error. Each time his application runs, MQSeries generates an FDC showing Probe Id XY341019 from RetryConnectToSharedSubpool.
Solution
This is a well-known problem between MQSeries 5.2 and AIX which relates to the way AIX segments the application address space and maps shared memory to these segments. This customer was using both MQSeries and DB2 in his application. Because the JVM was using most lower-numbered segments and DB2 was claiming segment 12 for itself, the customer was able to solve the problem by adding the parameter 'IPCCBaseAddress=11' to his queue manager's stanza in the /var/mqm/mqs.ini file.
32-bit applications on AIX have an address space of 4 GB, with virtual addresses ranging from 0x00000000 through 0xffffffff. AIX divides this address space into 16 segments, each 256 MB in size. The high-order nibble of any virtual address identifies its segment:
Virtual address 0x0003900c --> Segment 0
Virtual address 0x40000000 --> Segment 4
Virtual address 0x7f98c04f --> Segment 7
Virtual address 0xffffffff --> Segment 15
Not all of this memory is available to the application. Some segments are reserved by the operating system for the AIX kernel, shared libraries, and executable code. Other segments are reserved for heap memory, which is accessed by calling malloc(), calloc(), realloc(), free() and similar functions. The remaining segments are available for use by shmat() and mmap() for shared memory and file mappings.
The number of segments used for heap storage is set when an AIX application is compiled; This information is held in the maxdata field of the XCOFF header on RS/6000 executables. By default this maxdata field is set to zero, limiting the application to 256 MB of heap storage allocated from segment 2:
Segment 15: Reserved [AIX]
Segment 14: Reserved [AIX]
Segment 13: Reserved [AIX]
Segment 12: Free
Segment 11: Free
Segment 10: Free
Segment 9: Free
Segment 8: Free
Segment 7: Free
Segment 6: Free
Segment 5: Free
Segment 4: Free
Segment 3: Free
Segment 2: Native Heap [Application]
Segment 1: Reserved [AIX]
Segment 0: Reserved [AIX]
Applications that require more than 256 MB of heap memory must be compiled with the -bmaxdata flag to specify their heap size. So- called maxdata applications follow the AIX large address space model; Their heap grows upward from segment 3 as opposed to segment 2. The following table shows the location of the heap as the maxdata setting varies up to the maximum of 0x80000000, or 2 GB:
Compile Statement Heap Storage Segments
===================================== ============ ========
xlC foo.c -o foo 256 MB 2
xlC foo.c -bmaxdata:0x00000000 -o foo 256 MB 2
xlC foo.c -bmaxdata:0x10000000 -o foo 256 MB 3
xlC foo.c -bmaxdata:0x20000000 -o foo 512 MB 3-4
xlC foo.c -bmaxdata:0x30000000 -o foo 768 MB 3-5
xlC foo.c -bmaxdata:0x40000000 -o foo 1024 MB 3-6
xlC foo.c -bmaxdata:0x50000000 -o foo 1280 MB 3-7
xlC foo.c -bmaxdata:0x60000000 -o foo 1536 MB 3-8
xlC foo.c -bmaxdata:0x70000000 -o foo 1892 MB 3-9
xlC foo.c -bmaxdata:0x80000000 -o foo 2048 MB 3-10
When an application establishes a bindings (non-client) connection to a queue manager, MQSeries 5.2 uses segment 8 to attach shared memory and complete the connection. If segment 8 is unavailable, the connection will fail with reason code MQRC_Q_MGR_NOT_AVAILABLE (2059), and the application will generate an FDC file in the /var/mqm/errors directory showing a Probe Id of XY341019 and a Component of RetryConnectToSharedSubpool.
Segment 8 will be unavailable to MQSeries if it belongs to the heap or if a shared memory or memory map region has already been established there. Since shmat() and mmap() generally allocate areas growing upward from the top of the heap, reducing the maxdata setting of an application can free segment 8 and allow MQSeries connections to succeed.
It is possible to change the maxdata setting of an application without recompiling it. The following environment variable can be exported to override the XCOFF maxdata setting. Be aware that this variable should be set only for those processes which need it; Other programs (including strmqm) will not run if it is set:
export LDR_CNTRL=MAXDATA=0x30000000
The maxdata setting can also be overridden by modifying the XCOFF header in an RS/6000 executable. The AIX manual "General Programming Concepts: Writing and Debugging Programs" (publication number SC23-4128) describes how to do so using the echo and dd programs in Chapter 8, "Large Program Support."
If an application calls shmat or mmap heavily before connecting to MQSeries, or if its maxdata value cannot be reduced, segment 8 may not be available for MQSeries. To accommodate these situations, MQSeries 5.2 introduced a new attribute to the QueueManager: stanza of the /var/mqm/mqs.ini file. IPCCBaseAddress names an alternate segment for MQSeries shared memory (either 4, 5, 8, 9, 10, 11 or 12). Different queue managers can use different values:
QueueManager:
Name=HOBBES
Prefix=/var/mqm
Directory=HOBBES
IPCCBaseAddress=11
For example, IBM JVMs allocate a separate Java heap using the mmap function. The -ms and -mx parameters determine the startup and the maximum size of this Java heap. On the Java2 platform these flags must be preceded by an 'X'; A JVM with maxdata of 0x30000000 started with command-line flags "-Xms512M -Xmx1024M" would require MQSeries to set 'IPCCBaseAddress=11' or 'IPCCBaseAddress=12' in order to make room for both native and Java heaps while leaving a little room for other shmat or mmap callers:
Segment 15: Reserved [AIX]
Segment 14: Reserved [AIX]
Segment 13: Reserved [AIX]
Segment 12: Free
Segment 11: Shmat [MQSeries]
Segment 10: Free
Segment 9: Java Heap [IBM JVM]
Segment 8: Java Heap [IBM JVM]
Segment 7: Java Heap [IBM JVM]
Segment 6: Java Heap [IBM JVM]
Segment 5: Native Heap [IBM JVM]
Segment 4: Native Heap [IBM JVM]
Segment 3: Native Heap [IBM JVM]
Segment 2: Reserved [C Runtime]
Segment 1: Reserved [AIX]
Segment 0: Reserved [AIX]
Since memory allocation in AIX generally grows up, higher values for IPCCBaseAddress are more likely to succeed. However, it may be necessary to experiment with IPCCBaseAddress and maxdata in tandem to find a setting which works for a given queue manager and its applications. Be aware that "fastbound" applications are less likely to succeed since MQSeries requires segments 3 and 7 as well, and these segment numbers cannot be changed. Export the environment variable MQ_CONNECT_TYPE=STANDARD to use standard bindings instead.
AIX 5.0 introduced a new feature called dynamic segment allocation (DSA) which can also affect memory layout. Applications compiled with the '/dsa' modifier on the maxdata setting (for example, 'xlC foo.c -bmaxdata:0x30000000/dsa') or applications running in an environment with DSA (eg. 'export LDR_CNTRL=MAXDATA=0x30000000/dsa') must be aware that shmat and mmap regions will be allocated top down. Therefore applications running with DSA may find that lower settings of IPCCBaseAddress are more likely to succeed.
If MQSeries connections fail in an application despite attempts to use alternative IPCCBaseAddress/maxdata combinations, it may be necessary to use client connections. MQSeries client connections rely on a network link to the queue manager instead of shared memory, and as a result there are immune to any of the problems described in this documentation. Instructions for making client connections to MQSeries are available in the "MQSeries Clients" manual, the "MQSeries Using C++" and "MQSeries Using Java" manuals, the "MQSeries Application Programming Guide" and "MQSeries Application Programming Reference."
WebSphere MQ 5.3 handles shared memory differently on AIX and is not generally subject to this problem. The IPCCBaseAddress parameter is ignored by WebSphere MQ 5.3. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
