| Author |
Message
|
| vinodsasidharan |
 Posted: Mon May 12, 2003 1:05 pm Post subject: best way to set trace |
 |
|
Apprentice
Joined: 25 Apr 2003
Posts: 47
Location: Norwich
|
we are hesitating to set a trace in our production system.
ours is an aix based workflow system and it is 3.3.0 versioned .
what is the best trace any of u have used and which is not going to affect performanc3.
like flip flop trace / external trace.
hope any one can give me the base trace settings as my admin server is hanging and we cannot find anything in logs and err files .and trace being only solution.plss reply fast .
_________________
Vinod sasidharan
Ibm Certfied MQ Admin 5.3
Ibm Certfied MQ Admin 6.0
Ibm Certfied WAS Admin 6.0
Ibm Certfied WMB Admin 5.0
Ibm Certfied Db2 Specialist.
Sun certified Java Programmer.
"Ai carte, ai parte ....................." |
|
| Back to top |
|
 |
| vinodsasidharan |
| Posted: Tue May 13, 2003 7:30 am Post subject: ???? any ideas plss |
|
|
Apprentice
Joined: 25 Apr 2003
Posts: 47
Location: Norwich
|
_________________
Vinod sasidharan
Ibm Certfied MQ Admin 5.3
Ibm Certfied MQ Admin 6.0
Ibm Certfied WAS Admin 6.0
Ibm Certfied WMB Admin 5.0
Ibm Certfied Db2 Specialist.
Sun certified Java Programmer.
"Ai carte, ai parte ....................." |
|
| Back to top |
|
|
| vennela |
| Posted: Tue May 13, 2003 7:47 am Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
Are you looking for the command to set trace or are you asking for the type of trace that would be trap your admin server failure.
If you are looking for command then it is fmczchk. When your admin server fails are there any kind of db2 errors or mq errors. |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Tue May 13, 2003 8:52 am Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
Vinod,
I understand your concern in regard to turn the trace on, especially on a production system. We, in the past have also had similar issues, but turning what specific trace is usually a question we turn to IBM for, simply because they are the only ones able to understand whatever the hec is contained in it. Most of the time they are wanting you to recreate the problem and also happen to tell you what type of trace they would like for you to run and for how long, otherwise I would put that on my list of things to ask. (You probably have already done this, but if not I would open up a pmr with IBM and put in as much information as possible so it is considered atleast a sev2/sev1.)
So what do we know, Obviously we understand the the more detailed of a trace you turnon, on any system the more logging of information is done hence requiring more performance from the server.
What concerns me even more is your mention of the admin server and how it keeps crashing.
Have you been able to get any information from any of the following log files:
(NOTE: These are the default installation locations for the logs files)
/var/mqm/errors/AMQERR01.LOG (MQSeries complaints)
/var/mqm/qmgrs/FMCQM/errors/AMQERR01.LOG (MQSeries complaints)
/home/db2inst1/sqllib/db2dump/db2diag.log (DB2 complaints)
/var/fmc/fmcerr.log (MQSeries Workflow complaints)
/var/fmc/fmcsys.log (MQSeries Workflow system history
Also what is your status on auditing, do you have full audits to db/mq turned on. How often are you performing db2 maintenance, reorgs etc.
Have you been able to gain any additional information in this regard, ever since your first post?
Thanks,Praveen
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| vinodsasidharan |
| Posted: Tue May 13, 2003 9:13 am Post subject: admin server not responding |
|
|
Apprentice
Joined: 25 Apr 2003
Posts: 47
Location: Norwich
|
Hi ,
The admin server is in a not responding (thats the problem) state .
if i give ps -ef | grep main .
i can find fmcamain running .
i checked all the logs from top to bottom and no trace of any errors .except a number of dead locks . but these dead locks are here from 1 year . i tried database reorg and rebind and i do it weekly .
So my next step is to make a trace . i want to know which will be a best trace option from via ibm can detect this . The trace enablement must not affect performance of servers too???????
_________________
Vinod sasidharan
Ibm Certfied MQ Admin 5.3
Ibm Certfied MQ Admin 6.0
Ibm Certfied WAS Admin 6.0
Ibm Certfied WMB Admin 5.0
Ibm Certfied Db2 Specialist.
Sun certified Java Programmer.
"Ai carte, ai parte ....................." |
|
| Back to top |
|
|
| jmac |
| Posted: Tue May 13, 2003 9:19 am Post subject: |
|
|
Jedi Knight
Joined: 27 Jun 2001
Posts: 3081
Location: EmeriCon, LLC
|
This is just a wild guess... but have you looked to ensure that there is space on the filesystems containing all of your log files?
_________________
John McDonald
RETIRED |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Tue May 13, 2003 9:39 pm Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
How are you able to determine that the admin server is not responding? Are you performing an fmcautil on the configuration?
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| vinodsasidharan |
| Posted: Wed May 14, 2003 5:07 am Post subject: well yes |
|
|
Apprentice
Joined: 25 Apr 2003
Posts: 47
Location: Norwich
|
as soon as we found that users were complaning we checked the admin servers status and execution server status and the ps -ef | grep main
found all were fine .
But on trying fmcautil to check the reality . fmcautil was not working .then could make out that admin server is not responding . What we did usually was killed the admin server and restarted . But i dont want this to happen again . that why i am analysing . why it happened .
_________________
Vinod sasidharan
Ibm Certfied MQ Admin 5.3
Ibm Certfied MQ Admin 6.0
Ibm Certfied WAS Admin 6.0
Ibm Certfied WMB Admin 5.0
Ibm Certfied Db2 Specialist.
Sun certified Java Programmer.
"Ai carte, ai parte ....................." |
|
| Back to top |
|
|
| vinodsasidharan |
| Posted: Wed May 14, 2003 5:19 am Post subject: space available |
|
|
Apprentice
Joined: 25 Apr 2003
Posts: 47
Location: Norwich
|
yes john,
space is adequate .
_________________
Vinod sasidharan
Ibm Certfied MQ Admin 5.3
Ibm Certfied MQ Admin 6.0
Ibm Certfied WAS Admin 6.0
Ibm Certfied WMB Admin 5.0
Ibm Certfied Db2 Specialist.
Sun certified Java Programmer.
"Ai carte, ai parte ....................." |
|
| Back to top |
|
|
| praveenchhangani |
| Posted: Wed May 14, 2003 5:42 am Post subject: |
|
|
Disciple
Joined: 20 Sep 2002
Posts: 192
Location: Chicago, IL
|
That makes sense Vinod.
This is an interesting situation here, because I believe what happens is when the admin server goes down, workflow is not able to take anymore logon reuests BUT all the existing users that currently have a valid session are still able to work. Users that try to logon to workflow after the admin server died will get errors.
Bouncing the admin server is a good idea, but keep in mind that as soon as you bounce that configuration, you have just kicked all the people that were currently logged on with a session(and these are users that still would have been able to work, had the workflow admin not bounced the admin server).
Have you tried to simply bring up fmcamain only? If so, do you encounter the workflow coredump or does it come up just fine?
Often times, what the problem with the admin server coming down usually boils down to a few things:
1. file system is full
2. db is full / corrupted / db service went down etc.
Look at this article from IBM which talks about the workflow database tuning and what not. We found this to be very very helpful.
http://www-3.ibm.com/software/ts/mqseries/txppacs/wd02.html
When we ran into such issues in the past, we had opened a PMR with IBM and in the meantime we had monitoring setup so that as soon as fmcamain goes down we would get paged and inturn we could go in there and turn fmcmain back on without even having the users notice a problem. (This ofcourse is only a temp. solution until you hear something back from IBM on your particular case)
_________________
Praveen K. Chhangani,
IBM Certified Solutions Designer -
MQ Workflow 3.4. |
|
| Back to top |
|
|
| olan022 |
| Posted: Wed Mar 24, 2004 7:30 pm Post subject: |
|
|
Acolyte
Joined: 17 Apr 2002
Posts: 50
|
vinodsasidharan (or anyone that has had such issues)
I'm finding our WF 3.4 SP4, MQ 5.3, DB2 7.2 FP6 AIX 3-tier configuration having similar issues to what you are seeing where either the fmcamain server dies or more recently, continues to run but is not resposive when accessed via fmcautil until we restart everything.
There are no space issues, reorgs and statistics run on a weekly basis, etc.
I was instructed to run a trace from IBM but am also leary of doing so in production, did the trace solve your issue? |
|
| Back to top |
|
|
| premkumar |
| Posted: Wed Mar 24, 2004 8:08 pm Post subject: |
|
|
Apprentice
Joined: 24 Feb 2004
Posts: 40
|
| olan022 wrote: |
vinodsasidharan (or anyone that has had such issues)
I'm finding our WF 3.4 SP4, MQ 5.3, DB2 7.2 FP6 AIX 3-tier configuration having similar issues to what you are seeing where either the fmcamain server dies or more recently, continues to run but is not resposive when accessed via fmcautil until we restart everything.
There are no space issues, reorgs and statistics run on a weekly basis, etc.
I was instructed to run a trace from IBM but am also leary of doing so in production, did the trace solve your issue? |
Sorry, I never faced this type of issue...
Please try to solve my problem
Posted : Urgent ! Workflow Runtime Database is growing
_________________
Thanks and Regards,
Premkumar.S |
|
| Back to top |
|
|
| manoj |
| Posted: Thu Mar 25, 2004 2:45 pm Post subject: |
|
|
Master
Joined: 30 Jan 2002
Posts: 237
Location: Virgina
|
Vinod,
From my past experiance i can tell you one thing..When you are not doin a graceful shut down of the admin server (say kill fmcemain) there is a possible scenario which will keep the database entry for the admin server state as RUNNING (i guess....also it's an integer value and i don't remember what and in which table). Since there's a session existing for the Admin server ,MQ Worklfow will not start another admin server instance. To my knowledge the only way to get around this problem is issue a delete statement for that row.
regards
_________________
-manoj |
|
| Back to top |
|
|
|
|
