| Author |
Message
|
| vincent |
 Posted: Mon Jun 05, 2006 9:37 am Post subject: |
 |
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
mvic,
Your calculation is right. Since i have enabled the trace, it has taken aproximatly 30 seconds to put 1500 messages. If trace is turned off, then it would take 12 minutes.
Both the clients and server are in the same machine. There is nothing on the server side to process. Basically checking the time taken to put the mesage between two servers ( old and new ).
The output is
grep " poll" AMQ3695.FMT
11:19:04.460352 3695.1 -----{ poll
11:19:04.554834 3695.1 -----}! poll rc=Unknown(1)
11:19:04.555193 3695.1 -----{ poll
11:19:04.555216 3695.1 -----}! poll rc=Unknown(1)
11:19:04.555990 3695.1 -----{ poll
11:19:04.572580 3695.1 -----}! poll rc=Unknown(1)
11:19:04.573123 3695.1 ------{ poll
11:19:04.579236 3695.1 ------}! poll rc=Unknown(1)
11:19:04.579821 3695.1 ------{ poll
11:19:04.609093 3695.1 ------}! poll rc=Unknown(1)
11:19:04.609614 3695.1 ------{ poll
11:19:04.632878 3695.1 ------}! poll rc=Unknown(1)
This is going on till another 25 seconds.
What do i find on the server and how do i find ? |
|
| Back to top |
|
 |
| vincent |
| Posted: Mon Jun 05, 2006 11:30 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
Jeff,
It is taking 50% of CPU time to write/read from I/O. Do you think that this could be the cause of the problem? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jun 05, 2006 11:33 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
YES.
And that is probably a DISK problem.
EDIT.
But do what mvic says. He's smarter than I am in this area.
_________________
I am *not* the model of the modern major general.
Last edited by jefflowrey on Mon Jun 05, 2006 11:53 am; edited 1 time in total |
|
| Back to top |
|
|
| mvic |
| Posted: Mon Jun 05, 2006 11:50 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vincent wrote: |
| What do i find on the server and how do i find ? |
Since it's on the same machine as the client, the server trace files should already be in /var/mqm/errors. Try
| Code: |
cd /var/mqm/trace
grep amqrmppa *.FMT
# From the output of the "grep", find the trace file(s)
# for the amqrmppa process(es). Here we use the
# number 123456 - but substitute the pid to get the
# correct trace file
grep " send" AMQ123456.FMT | grep 11:19:04
grep " poll" AMQ123456.FMT | grep 11:19:04 |
and post the output, or enough of the output to show a few puts. |
|
| Back to top |
|
|
| mvic |
| Posted: Mon Jun 05, 2006 12:06 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vincent wrote: |
grep " poll" AMQ3695.FMT
11:19:04.460352 3695.1 -----{ poll
11:19:04.554834 3695.1 -----}! poll rc=Unknown(1)
11:19:04.555193 3695.1 -----{ poll
11:19:04.555216 3695.1 -----}! poll rc=Unknown(1)
11:19:04.555990 3695.1 -----{ poll
11:19:04.572580 3695.1 -----}! poll rc=Unknown(1)
11:19:04.573123 3695.1 ------{ poll
11:19:04.579236 3695.1 ------}! poll rc=Unknown(1)
11:19:04.579821 3695.1 ------{ poll
11:19:04.609093 3695.1 ------}! poll rc=Unknown(1)
11:19:04.609614 3695.1 ------{ poll
11:19:04.632878 3695.1 ------}! poll rc=Unknown(1) |
Another bit of analysis: the above outputs show server+network times (yes I know this is the same machine so the network time should be negligible) of approximately 94.4, 0.023, 16.6, 6.1, 29.2 and 23.2 milliseconds.
If (if!!) these are all from equivalent puts - eg. same message size, persistence, syncpoint settings, commits - then these numbers vary by a large amount. The range is from 94.4 to 0.023 ms. Even ignoring these two exceptional cases, the spread in numbers is wide.
Jeff may be right that this is a disk problem. But to be certain, more analysis is required of what happens during the put once the request has reached the server.
Are there any entries in the syslog showing disk or other system problems?
Are any FDC files being written to /var/mqm/errors ? For example, post the output from
| Code: |
cd /var/mqm/errors
/opt/mqm/bin/ffstsummary |
|
|
| Back to top |
|
|
| vincent |
| Posted: Mon Jun 05, 2006 12:59 pm Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
mvic,
how do i take a output of ffstsumary since it is a binary file?
There is no error recorded in errors /var/mqm/errors/ directory |
|
| Back to top |
|
|
| mvic |
| Posted: Mon Jun 05, 2006 5:12 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vincent wrote: |
| how do i take a output of ffstsumary since it is a binary file? |
In my "code" snippet above I intended that you would first type "cd /var/mqm/errors" then enter, then type "/opt/mqm/bin/ffstsummary" then enter. Sorry I was not clear enough about that.
| Quote: |
| There is no error recorded in errors /var/mqm/errors/ directory |
Does this mean there are no files named *.FDC in /var/mqm/errors ? If there are no *.FDC files, then /opt/mqm/bin/ffstsummary will give no output, so no need to run that program. |
|
| Back to top |
|
|
| vincent |
| Posted: Tue Jun 06, 2006 5:52 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
mvic,
Here is the output:
11:19:04.554764 2061.39 -----{ send
11:19:04.554821 2061.39 -----}! send rc=Unknown(84)
11:19:04.555148 2061.39 -----{ send
11:19:04.555171 2061.39 -----}! send rc=Unknown(84)
11:19:04.572552 2061.39 -----{ send
11:19:04.572576 2061.39 -----}! send rc=Unknown(2C)
11:19:04.579209 2061.39 -----{ send
11:19:04.579229 2061.39 -----}! send rc=Unknown(D
11:19:04.609062 2061.39 -----{ send
11:19:04.609086 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.632853 2061.39 -----{ send
11:19:04.632873 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.650745 2061.39 -----{ send
11:19:04.650765 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.660892 2061.39 -----{ send
11:19:04.660913 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.686953 2061.39 -----{ send
11:19:04.686975 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.711194 2061.39 -----{ send
11:19:04.711215 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.726831 2061.39 -----{ send
11:19:04.726852 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.744753 2061.39 -----{ send
11:19:04.554940 2061.39 ------{ poll
11:19:04.555041 2061.39 ------}! poll rc=Unknown(1)
11:19:04.572696 2061.39 ------{ poll
11:19:04.572988 2061.39 ------}! poll rc=Unknown(1)
11:19:04.579359 2061.39 ------{ poll
11:19:04.579689 2061.39 ------}! poll rc=Unknown(1)
11:19:04.609223 2061.39 ------{ poll
11:19:04.609447 2061.39 ------}! poll rc=Unknown(1)
11:19:04.632975 2061.39 ------{ poll
11:19:04.633230 2061.39 ------}! poll rc=Unknown(1)
11:19:04.650896 2061.39 ------{ poll
11:19:04.651153 2061.39 ------}! poll rc=Unknown(1)
11:19:04.661051 2061.39 ------{ poll
11:19:04.668836 2061.39 ------}! poll rc=Unknown(1)
11:19:04.687100 2061.39 ------{ poll
11:19:04.687338 2061.39 ------}! poll rc=Unknown(1)
11:19:04.711351 2061.39 ------{ poll
11:19:04.711604 2061.39 ------}! poll rc=Unknown(1)
11:19:04.726989 2061.39 ------{ poll
11:19:04.727214 2061.39 ------}! poll rc=Unknown(1)
11:19:04.744920 2061.39 ------{ poll
11:19:04.745138 2061.39 ------}! poll rc=Unknown(1)
11:19:04.762829 2061.39 ------{ poll
11:19:04.763090 2061.39 ------}! poll rc=Unknown(1) |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jun 06, 2006 6:06 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
OK splicing these together and sorting gives a list below. The next step is to investigate what is happening between pairs of lines that show a long elapsed time. For example, the pairs of lines I have highlighted in bold text.
I think at this point the best advice is what others have said: call IBM Support for assistance. This type of analysis requires deep knowledge of MQ internals, and should be handled by IBM support engineers.
| Quote: |
11:19:04.554764 2061.39 -----{ send
11:19:04.554821 2061.39 -----}! send rc=Unknown(84)
11:19:04.554940 2061.39 ------{ poll
11:19:04.555041 2061.39 ------}! poll rc=Unknown(1)
11:19:04.555148 2061.39 -----{ send
11:19:04.555171 2061.39 -----}! send rc=Unknown(84)
11:19:04.572552 2061.39 -----{ send
11:19:04.572576 2061.39 -----}! send rc=Unknown(2C)
11:19:04.572696 2061.39 ------{ poll
11:19:04.572988 2061.39 ------}! poll rc=Unknown(1)
11:19:04.579209 2061.39 -----{ send
11:19:04.579229 2061.39 -----}! send rc=Unknown(D
11:19:04.579359 2061.39 ------{ poll
11:19:04.579689 2061.39 ------}! poll rc=Unknown(1)
11:19:04.609062 2061.39 -----{ send
11:19:04.609086 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.609223 2061.39 ------{ poll
11:19:04.609447 2061.39 ------}! poll rc=Unknown(1)
11:19:04.632853 2061.39 -----{ send
11:19:04.632873 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.632975 2061.39 ------{ poll
11:19:04.633230 2061.39 ------}! poll rc=Unknown(1)
11:19:04.650745 2061.39 -----{ send
11:19:04.650765 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.650896 2061.39 ------{ poll
11:19:04.651153 2061.39 ------}! poll rc=Unknown(1)
11:19:04.660892 2061.39 -----{ send
11:19:04.660913 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.661051 2061.39 ------{ poll
11:19:04.668836 2061.39 ------}! poll rc=Unknown(1)
11:19:04.686953 2061.39 -----{ send
11:19:04.686975 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.687100 2061.39 ------{ poll
11:19:04.687338 2061.39 ------}! poll rc=Unknown(1)
11:19:04.711194 2061.39 -----{ send
11:19:04.711215 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.711351 2061.39 ------{ poll
11:19:04.711604 2061.39 ------}! poll rc=Unknown(1)
11:19:04.726831 2061.39 -----{ send
11:19:04.726852 2061.39 -----}! send rc=Unknown(1F4)
11:19:04.726989 2061.39 ------{ poll
11:19:04.727214 2061.39 ------}! poll rc=Unknown(1)
11:19:04.744753 2061.39 -----{ send
11:19:04.744920 2061.39 ------{ poll
11:19:04.745138 2061.39 ------}! poll rc=Unknown(1)
11:19:04.762829 2061.39 ------{ poll
11:19:04.763090 2061.39 ------}! poll rc=Unknown(1) |
|
|
| Back to top |
|
|
| vincent |
| Posted: Tue Jun 06, 2006 6:26 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
Thansk mvic,
is it possible to give some clue for the following scenario
The messages are datagrams and no reply/ response is required.
Messages are persistant. One app puts and other app gets it. Both the are running in the same machine where server installed but runs as a client.
I am basically expecting more throughput so i have added more number of PUT process, the througput scaled up and the same way GET process too when i ran independently.
But both the 10 PUT processes and 10 GET processes runs simultaneoulsy. The throughput does not improve but stand still as it is in 1 and 1 pair of put and get. Any thoughts?
Another intersting thing..
1 put process takes 5 seconds to put 1500 messages
1 get process takes 3 seconds to get the same messages
when both runs it took 6 seconds to complete the both put and get messages
Any clue why? I am seing some kind of lock. The put process takes more time when get also runs and put and get finishes the job simultaneously. |
|
| Back to top |
|
|
| vincent |
| Posted: Tue Jun 06, 2006 6:39 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
I also got the access to run ffstsummary:
The report is : - The system is behaving badly after 19th of may. it is highligted.
AMQ04077.0.FDC 2006/05/03 21:47:22 dltmqm_nd 4077 1 AD034000 adiDeleteDir
xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ04077.0.FDC 2006/05/03 21:47:22 dltmqm_nd 4077 1 ZT072000 zutDeleteDirec
tory xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ04081.0.FDC 2006/05/03 21:48:09 dltmqm_nd 4081 1 ZT028001 zutCheckIPCCDi
rs xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ04558.0.FDC 2006/05/04 16:43:45 dltmqm_nd 4558 1 AD034000 adiDeleteDir
xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ04558.0.FDC 2006/05/04 16:43:45 dltmqm_nd 4558 1 ZT072000 zutDeleteDirec
tory xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ04561.0.FDC 2006/05/04 16:44:01 dltmqm_nd 4561 1 ZT028001 zutCheckIPCCDi
rs xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ16185.0.FDC 2006/05/04 21:40:51 runmqsc 16185 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ16185.0.FDC 2006/05/04 21:40:51 runmqsc 16185 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ22526.0.FDC 2006/05/04 21:42:41 runmqsc 22526 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ22526.0.FDC 2006/05/04 21:42:41 runmqsc 22526 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ00323.0.FDC 2006/05/04 21:44:54 runmqsc 323 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ00323.0.FDC 2006/05/04 21:44:54 runmqsc 323 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ02106.0.FDC 2006/05/04 22:00:38 runmqsc 2106 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ03412.0.FDC 2006/05/05 17:49:12 runmqsc 3412 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ03412.0.FDC 2006/05/05 17:49:12 runmqsc 3412 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ04280.0.FDC 2006/05/05 18:31:36 runmqsc 4280 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ04280.0.FDC 2006/05/05 18:31:36 runmqsc 4280 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ06226.0.FDC 2006/05/05 18:39:48 runmqsc 6226 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ06226.0.FDC 2006/05/05 18:39:48 runmqsc 6226 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ06676.0.FDC 2006/05/05 18:58:13 runmqsc 6676 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ06676.0.FDC 2006/05/05 18:58:13 runmqsc 6676 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ22768.0.FDC 2006/05/05 21:33:42 runmqsc 22768 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ22768.0.FDC 2006/05/05 21:33:42 runmqsc 22768 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ03228.0.FDC 2006/05/05 22:24:41 runmqsc 3228 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ03228.0.FDC 2006/05/05 22:24:41 runmqsc 3228 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ19241.0.FDC 2006/05/05 22:29:24 runmqlsr_nd 19241 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ12575.0.FDC 2006/05/08 16:09:24 runmqsc 12575 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ12575.0.FDC 2006/05/08 16:09:24 runmqsc 12575 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ16029.0.FDC 2006/05/08 22:04:51 runmqsc 16029 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ16029.0.FDC 2006/05/08 22:04:51 runmqsc 16029 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ03194.0.FDC 2006/05/09 19:16:28 runmqsc 3194 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ03194.0.FDC 2006/05/09 19:16:28 runmqsc 3194 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ21791.0.FDC 2006/05/09 20:07:51 runmqsc 21791 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ21791.0.FDC 2006/05/09 20:07:51 runmqsc 21791 2 XC267011 xehAsySignalMo
nitor xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ19407.0.FDC 2006/05/16 18:43:13 runmqtrm 19407 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ19407.0.FDC 2006/05/16 18:43:15 runmqtrm 19407 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ19407.0.FDC 2006/05/18 17:35:22 runmqtrm 19407 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL077075 mqlpgmrf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL047028 mqlpgadf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL077071 mqlpgmrf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL047028 mqlpgadf
hrcE_MQLP_BADLOG OK AMQ25070.0.FDC 2006/05/19 16:43:19 amqrmppa 25070 78 HL010002 hlgReserveLogS
pace xecF_E_UNEXPECTED_RC hrcE_MQLP_BADLOG
AMQ25070.0.FDC 2006/05/19 16:43:19 amqrmppa 25070 78 AL018001 almReserveSpac
e xecF_E_UNEXPECTED_RC hrcE_MQLO_UNEXPECTED_OS_ERROR
AMQ25070.0.FDC 2006/05/19 16:43:19 amqrmppa 25070 77 HL010002 hlgReserveLogS
pace xecF_E_UNEXPECTED_RC hrcE_MQLP_BADLOG
AMQ25070.0.FDC 2006/05/19 16:43:20 amqrmppa 25070 77 AL018001 almReserveSpac
e xecF_E_UNEXPECTED_RC hrcE_MQLO_UNEXPECTED_OS_ERROR
AMQ18445.0.FDC 2006/05/19 16:44:59 amqhasmx 18445 1 HL077071 mqlpgmrf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:44:59 amqhasmx 18445 1 HL047028 mqlpgadf
hrcE_MQLP_BADLOG OK
AMQ25070.0.FDC 2006/05/19 16:44:59 amqrmppa 25070 79 HL010002 hlgReserveLogS
pace xecF_E_UNEXPECTED_RC hrcE_MQLP_BADLOG
AMQ25070.0.FDC 2006/05/19 16:44:59 amqrmppa 25070 79 AL018001 almReserveSpac
e xecF_E_UNEXPECTED_RC hrcE_MQLO_UNEXPECTED_OS_ERROR
AMQ23835.0.FDC 2006/05/19 16:50:34 runmqsc 23835 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ18450.0.FDC 2006/05/19 17:40:09 amqzlaa0_nd 18450 237 XC307070 xlsRequestMute
x xecL_W_LONG_LOCK_WAIT OK
AMQ27026.0.FDC 2006/05/19 18:52:49 runmqsc 27026 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27026.0.FDC 2006/05/19 18:53:26 runmqsc 27026 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27026.0.FDC 2006/05/19 18:53:28 runmqsc 27026 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27088.0.FDC 2006/05/19 19:40:06 runmqsc 27088 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:43 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:43 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:43 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:43 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:44 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:44 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27112.0.FDC 2006/05/19 19:43:45 runmqsc 27112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27176.0.FDC 2006/05/19 20:29:39 endmqlsr 27176 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ18442.0.FDC 2006/05/19 20:35:02 amqzxma0_nd 18442 1 XC307040 xlsRequestMute
x xecL_W_LONG_LOCK_WAIT OK
AMQ19241.0.FDC 2006/05/19 20:39:51 runmqlsr_nd 19241 5 XC307070 xlsRequestMute
x xecL_W_LONG_LOCK_WAIT OK
AMQ27194.0.FDC 2006/05/19 20:57:00 endmqm 27194 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ27195.0.FDC 2006/05/19 20:57:22 endmqlsr 27195 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ28266.0.FDC 2006/05/22 14:32:05 endmqm 28266 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ28266.0.FDC 2006/05/22 14:51:29 endmqm 28266 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ28266.0.FDC 2006/05/22 14:51:34 endmqm 28266 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ19241.0.FDC 2006/05/22 15:00:13 runmqlsr_nd 19241 4 XC308010 xlsReleaseMute
x xecL_W_LONG_LOCK_WAIT OK
AMQ18450.0.FDC 2006/05/22 15:00:15 amqzlaa0_nd 18450 1 ZL000010 zlaMain
xecF_E_UNEXPECTED_RC xecP_E_INVALID_PID
AMQ18447.0.FDC 2006/05/22 15:00:24 amqrrmfa 18447 1 RM220005 rrmRepository
rrcE_MQGET_FAILED OK
AMQ28382.0.FDC 2006/05/22 15:03:53 endmqm 28382 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ28382.0.FDC 2006/05/22 15:03:53 endmqm 28382 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ28382.0.FDC 2006/05/22 15:03:54 endmqm 28382 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ18446.0.FDC 2006/05/22 15:05:32 amqzllp0 18446 1 XC308010 xlsReleaseMute
x xecL_W_LONG_LOCK_WAIT OK
AMQ18446.0.FDC 2006/05/22 15:05:43 amqzllp0 18446 1 XC307030 xlsRequestMute
x STOP_ALL OK
AMQ18446.0.FDC 2006/05/22 15:05:43 amqzllp0 18446 1 AL047011 alsCheckPointL
oop STOP_ALL OK
AMQ29198.0.FDC 2006/05/22 21:29:45 runmqsc 29198 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ29198.0.FDC 2006/05/22 21:29:45 runmqsc 29198 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ00814.0.FDC 2006/05/23 21:43:31 runmqsc 814 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ01112.0.FDC 2006/05/23 23:06:02 runmqlsr_nd 1112 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ01085.0.FDC 2006/05/23 23:07:13 runmqsc 1085 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ00318.0.FDC 2006/06/01 22:05:18 dltmqm_nd 318 1 HL005003 hlgDeleteLog
hrcE_MQLO_UNEXPECTED_OS_ERROR OK
AMQ00318.0.FDC 2006/06/01 22:05:18 dltmqm_nd 318 1 ZT072000 zutDeleteDirec
tory xecF_E_UNEXPECTED_SYSTEM_RC OK
AMQ00322.0.FDC 2006/06/01 22:05:51 dltmqm_nd 322 1 HL081010 mqlpgolf
xecF_E_UNEXPECTED_RC hrcE_MQLO_FNEX
AMQ00322.0.FDC 2006/06/01 22:05:51 dltmqm_nd 322 1 HL081010 mqlpgolf
xecF_E_UNEXPECTED_RC hrcE_MQLO_FNEX
AMQ01233.0.FDC 2006/06/02 15:22:47 amqzxma0_nd 1233 1 ZX000001 ExecCtrlrMain
xecF_E_UNEXPECTED_RC lpiRC_LOG_LOCATION_INVALID
AMQ01978.0.FDC 2006/06/02 21:14:10 runmqsc 1978 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK
AMQ02058.0.FDC 2006/06/05 14:16:50 runmqlsr_nd 2058 2 XC338001 xehAsySignalHa
ndler xecE_W_UNEXPECTED_ASYNC_SIGNAL OK |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jun 06, 2006 7:02 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vincent wrote: |
| is it possible to give some clue for the following scenario |
May I suggest getting some IBM consultants to help with these questions? Tuning for performance is never straightforward. Personally I don't know the answers to your questions. It could be that you have a system or disk bottleneck - the only way to tell is to analyse traces and system performance tools carefully, ideally with the assistance of a performance tuning expert.
| Quote: |
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL077075 mqlpgmrf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL047028 mqlpgadf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL077071 mqlpgmrf
hrcE_MQLP_BADLOG OK
AMQ18445.0.FDC 2006/05/19 16:43:19 amqhasmx 18445 1 HL047028 mqlpgadf
hrcE_MQLP_BADLOG OK |
For help with these, I recommend you call IBM Support. |
|
| Back to top |
|
|
| vincent |
| Posted: Thu Jun 08, 2006 9:21 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
| Thanks Jeff, mvic. It could be a DISK problem. SysAdmin changing the disk. |
|
| Back to top |
|
|
| vincent |
| Posted: Fri Jun 16, 2006 6:39 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
By adding another SAN based disk increased throughput of around 200% but that is not still sufficient for my requirement.
I found a interesting thing when i ran GET process, the iostat says that writting speed instead of reading spead.
Following is the snapshot:-
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0s2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0s6
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0s2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t16d0
0.0 1148.0 0.0 4986.0 0.0 1.1 0.0 0.9 0 90 c2t17d0
0.0 1148.0 0.0 4986.0 0.0 1.1 0.0 0.9 0 90 c2t17d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t17d0s2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t16d0
Do we have any clue?
Another think is that this statistic says the harddisk is utilized 90% of its resource to write. What could be wrong here because the utilization is always very high?
So when we add both PUT and GET process for the same QUEUE the throughput is reduced.
Any clue? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jun 17, 2006 5:21 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Get faster better performing disks with better cache?
Remember that this counts for both queue space and log space!!!
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
