| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Syslog: How to add unique id to the group of messages caused |
« View previous topic :: View next topic » |
| Author |
Message
|
| t603 |
 Posted: Mon Sep 16, 2013 2:15 am Post subject: Syslog: How to add unique id to the group of messages caused |
 |
|
Voyager
Joined: 16 Oct 2012
Posts: 88
Location: Prague, the Czech Republic, Europe
|
Hello,
I would like to process our WebSphere Message Broker 7.x syslog by some regex (and|or) sql script to get set of unique types of errors logged into syslog - something like SQL's "select group by".
As I undertand, there are mostly several (lines of) messages caused by one error within one message flow. For example these 4 messages below are (according to me) caused by 1 problem in 1 message flow.
| Code: |
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP4623E: Exception condition detected by input node 'CONN_IGAS_ODBC_CZ.C_parkingMsgs.TimeoutNotification'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/TimeoutControl/ImbTimeoutNotificationNode.cpp: 1575: ImbTimeoutNotificationNode::propagateToCatch: ComIbmTimeoutNotificationNode: CONN_IGAS_ODBC_CZ#FCMComposite_1_9.Components/C_parkingMsgs#FCMComposite_1_1
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP2230E: Error detected whilst processing a message in node 'CONN_IGAS_ODBC_CZ.testConnection'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/ImbComputeNode.cpp: 489: ImbComputeNode::evaluate: ComIbmComputeNode: CONN_IGAS_ODBC_CZ#FCMComposite_1_5
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP2393E: Database error: ODBC return code '-1' from data source 'MQSIIGASCZ' using ODBC driver manager 'libbipodbc.a(odbc.so)'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/ImbOdbc.cpp: 351: ImbOdbcHandle::checkRcInner: :
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP2322E: Database error: SQL State '28000'; Native Error Code '1017'; Error Text '[IBM][ODBC Oracle Wire Protocol driver][Oracle]ORA-01017: neplatn uivatelsk jmno nebo heslo; vstup do systmu odmtnut'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/ImbOdbc.cpp: 520: ImbOdbcHandle::checkRcInner: : |
My question is: Is it possible to configure WSMB 7.x to add some unique id to the group of messages caused by one error? There already are some ids [32505936] and [52519], but this looks like id of some object within wsmb (I guess broker and eg) rather that id of one error.
It would be sufficient to have this id unique within broker, execution group and let say timestamp up to second format, e.g. 2013-09-16-10-51-58-0001 . So messages should be something like (added id is surrounded by "***"):
| Code: |
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP4623E: ***201309161051580001*** Exception condition detected by input node 'CONN_IGAS_ODBC_CZ.C_parkingMsgs.TimeoutNotification'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/TimeoutControl/ImbTimeoutNotificationNode.cpp: 1575: ImbTimeoutNotificationNode::propagateToCatch: ComIbmTimeoutNotificationNode: CONN_IGAS_ODBC_CZ#FCMComposite_1_9.Components/C_parkingMsgs#FCMComposite_1_1
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP2230E: ***201309161051580001*** Error detected whilst processing a message in node 'CONN_IGAS_ODBC_CZ.testConnection'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/ImbComputeNode.cpp: 489: ImbComputeNode::evaluate: ComIbmComputeNode: CONN_IGAS_ODBC_CZ#FCMComposite_1_5
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP2393E: ***201309161051580001*** Database error: ODBC return code '-1' from data source 'MQSIIGASCZ' using ODBC driver manager 'libbipodbc.a(odbc.so)'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/ImbOdbc.cpp: 351: ImbOdbcHandle::checkRcInner: :
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP2322E: ***201309161051580001*** Database error: SQL State '28000'; Native Error Code '1017'; Error Text '[IBM][ODBC Oracle Wire Protocol driver][Oracle]ORA-01017: neplatn uivatelsk jmno nebo heslo; vstup do systmu odmtnut'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/ImbOdbc.cpp: 520: ImbOdbcHandle::checkRcInner: : |
I searched for the format of WSMB log in "http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/topic/com.ibm.etools.mft.doc/an04120_.htm" and "MQSeries.net", but without any success.
Thank You for any suggestion in advance. Stepan
Last edited by t603 on Mon Sep 16, 2013 4:30 am; edited 1 time in total |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Mon Sep 16, 2013 3:10 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
What product is WSMB ? Do you mean WebSphere Message Broker ? If so, the product identifier is WMB for version 7.0.0.6. What version are you on ? 7.X is not a valid version identifier.
Now to get to your question:
Is it possible in WMB to add some unique id to the group of messages caused by one error?
What you really mean to do is have robust error handling within your message flow, and yes, it is possible to construct your error output through java.util.logging to output specific identifiers. I often use MQMD.MsgId when outputting application log messages. In other cases, I use LocalEnvironment.ReplyId. There are innumerable correlation identifiers at your disposal.
Trying to parse content directly from syslog is not what you really want to do.
You want to build into your message flow development process some common tooling that you use in all projects. One or more of these tools would get the end-result you desire, which is associating one or more application log messages with a single event.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Tibor |
| Posted: Mon Sep 16, 2013 3:27 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Stepan: [32505936] and [52519] are PID and TID (Process / Thread ID) of the Execution Group which sends these entries into syslog.
Besides, lancelotlinc is right, first question is: what is your goal / intent with this script? |
|
| Back to top |
|
|
| t603 |
| Posted: Mon Sep 16, 2013 4:04 am Post subject: |
|
|
Voyager
Joined: 16 Oct 2012
Posts: 88
Location: Prague, the Czech Republic, Europe
|
1) Result of my "mqsiservice -v":
BIP8996I: Version: 7004
BIP8997I: Product: WebSphere Message Broker
BIP8998I: CMVC Level: S700-FP04
BIP8999I: Build Type: Production, 64 bit, rios_aix_4
2) The goal of my script? Ehm...
To watch if there are (in syslog) still the same errors as usually are and not any new kind of errors 
And I do not have sources (sometimes even source code, mostly time, often people, always money) to do any changes in all of my 1.128 (sometimes very old) message flows. So I need only to monitor current errors and watch, if there are not any new kind of errors (these new ones I can ask to repair). Sad, but simple.
Really, only change in the output to the syslog can help me now. So is there any such change?
3) On the other hand - identyfying the start and the end or better grouping one error according some id should be welcome by anyone, who is reading syslog, am I right? Otherwise one do not know, where ends one error and starts another one.
Thank you, Stepan |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Sep 16, 2013 4:07 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
syslog.d is governed by /etc/syslog.conf on most *nix. You can configure it any way you want.
Are you saying you have no source code ? You do not use source code repository ?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Sep 16, 2013 4:17 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| lancelotlinc wrote: |
What product is WSMB ? Do you mean WebSphere Message Broker ? If so, the product identifier is WMB for version 7.0.0.6. What version are you on ? 7.X is not a valid version identifier.
|
| t603 wrote: |
Code:
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP4623E: Exception condition detected by input node 'CONN_IGAS_ODBC_CZ.C_parkingMsgs.TimeoutNotification'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/TimeoutControl/ImbTimeoutNotificationNode.cpp: 1575: ImbTimeoutNotificationNode::propagateToCatch: ComIbmTimeoutNotificationNode: CONN_IGAS_ODBC_CZ#FCMComposite_1_9.Components/C_parkingMsgs#FCMComposite_1_1
|
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Sep 16, 2013 4:22 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| PeterPotkay wrote: |
| lancelotlinc wrote: |
What product is WSMB ? Do you mean WebSphere Message Broker ? If so, the product identifier is WMB for version 7.0.0.6. What version are you on ? 7.X is not a valid version identifier.
|
| t603 wrote: |
Code:
Sep 16 10:51:58 batch15 user:err|error WebSphere Broker v7004[32505936]: (BK2.ConnODBC_2)[52519]BIP4623E: Exception condition detected by input node 'CONN_IGAS_ODBC_CZ.C_parkingMsgs.TimeoutNotification'. : BK2.7e7e443f-3a01-0000-0080-a1c4966041ac: /build/S700_P/src/DataFlowEngine/TimeoutControl/ImbTimeoutNotificationNode.cpp: 1575: ImbTimeoutNotificationNode::propagateToCatch: ComIbmTimeoutNotificationNode: CONN_IGAS_ODBC_CZ#FCMComposite_1_9.Components/C_parkingMsgs#FCMComposite_1_1
|
|
No doubt. I was merely pointing out to the OP that 7.X lacks sufficient precision when discussing problems and issues. For example, the SOAP nodes were not really functional until 7.0.0.2 (query wsdl did not work).
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Sep 16, 2013 4:42 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
The exception list contains the information that you need. Exceptions are grouped into chains of related errors. If your 1000+ message flows had been designed using best practices then they would be processing the exception list using a standard subflow attached to the Catch terminal of every input node. ( or maybe they are doing that, but the subflow is not doing any logging? )
The entries in the syslog are not designed to contain the information that you need. They are almost always a single-line entry, and contain no context. That's going to make it hard to achieve your goals.
I hope you find a good solution, but I don't think you will get there simply by scanning the syslog.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| t603 |
| Posted: Mon Sep 16, 2013 5:15 am Post subject: |
|
|
Voyager
Joined: 16 Oct 2012
Posts: 88
Location: Prague, the Czech Republic, Europe
|
Thank You, kimbert, thank You. That is the answer I wanted to know - I should stop further searching for any exact solution.
Unfortunately, some or most (?) of our message flows do not catch just only few kinds of exception like missing queue, odbc problems, non well-formed xml, encoding violations etc. Other exceptions are catched OK. So I do not rely on syslog as the source of our pool of bugs, but in syslog are something like hard errors, not soft exceptions.
Meanwhile I wrote small ksh script, which clean-up syslog to the three columns brokerName|executionGroupName|fullExceptionMessage and then uniq and then sort and then finaly lastResult right outer join currentResult, which do little something like I wanted. |
|
| Back to top |
|
|
| t603 |
| Posted: Mon Sep 16, 2013 11:59 am Post subject: |
|
|
Voyager
Joined: 16 Oct 2012
Posts: 88
Location: Prague, the Czech Republic, Europe
|
If anyone will anytime solve similar task how to parse and make statistic from syslog, here is my all in one line KSH command, which takes as an input syslog and outputs SQL inserts to be better visible. Of course, if one has (in-memory) fast database, skip sort and uniq commands an make statistics in SQL database using select group by command in database.
| Code: |
| sed -e "s/'/''/g" < "wmbsyslog.log" | sed " s/.*WebSphere Broker v[0-9]*\[[0-9]*\]: (\([a-zA-Z0-9]*\)\.\([a-zA-Z0-9_]*\))\[[0-9]*\]\(.*\)/=insert into MQSI.syslogAnalyze (brokerName, egName, exceptionValue) values ('\1', '\2', '\3') /g" | sed "s/^[^=].*//g" | sed "/^$/d" | sed "s/^=\(.*\)/\1/g" | sort | uniq | sed '/.*/{G;s/\(.*\)/\1go/;}' |
|
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
