| Author |
Message
|
| kolthorr |
 Posted: Tue May 31, 2005 8:42 am Post subject: Pub/sub broker non-durable queue cleanup |
 |
|
Apprentice
Joined: 09 Mar 2002
Posts: 31
|
Hi all,
We're testing JMS pub/sub with WSMQ5.3 CSD08 (ie, integrated broker rather than MA0C) on Solaris. Sequence of events goes something like this:
- apps run normally for a while
- over time we have a buildup of SYSTEM.JMS.ND.XXXXX permdyn queues created (they don't always get removed)
- the 'amqfcxba' (broker) process generally has these leftover queues open for output, but nothing has them open for input
So far, fairly normal for apps that haven't disconnected cleanly. So I go to the good old jms cleanup utility. Shut everything down, do a 'NONDUR' cleanup, fire it up again. Old queues disappear.
BUT after that point the broker starts throwing 2085 errors in the logs (see below) related to these now-deleted queues, and further updates to the topics end up on the DLQ. To actually get it to 'forget' about these subscriptions I end up deleting the broker, running the jms cleanup app again, clearing down the DLQ/qmgr event q, and starting everything up. This seems rather heavy handed & I'd rather not have to do it every week in production 
Any thoughts on what might be happening and why the broker isn't letting go? We use non-durable subscriptions exclusively, no durable ones. All this time new clients can connect & receive updates normally, nothing 'breaks' from the client end.
Thanks very much,
Andrew
| Quote: |
----- amqfqxaa.c : 679 --------------------------------------------------------
05/31/05 15:29:35
AMQ5882: WebSphere MQ Publish/Subscribe broker has written a message to the
dead-letter queue.
EXPLANATION:
The broker has written a message to the dead-letter queue
(SYSTEM.DEAD.LETTER.QUEUE ) for reason
'2085:MQRC_UNKNOWN_OBJECT_NAME'. Note. To save log space, after the first
occurrence of this message for stream (SYSTEM.BROKER.DEFAULT.STREAM
), it will only be written periodically.
|
|
|
| Back to top |
|
 |
| vennela |
| Posted: Tue May 31, 2005 9:31 am Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| Quote: |
| So I go to the good old jms cleanup utility. |
What is that? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 31, 2005 11:46 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Did you unsubscribe the applications having lost connectivity ??
 |
|
| Back to top |
|
|
| kolthorr |
| Posted: Wed Jun 01, 2005 2:26 am Post subject: |
|
|
Apprentice
Joined: 09 Mar 2002
Posts: 31
|
Thanks for the replies. The JMS Cleanup utility is a java program supplied with MQ in /opt/mqm/java/bin, either a .bat file on Win or shell script. It just calls com.ibm.mq.jms.Cleanup and passes through the option you want.
We don't call unsubscribe() anywhere as we're only using non-durable connections - all we can do is call close(). When the app exits cleanly everything works as you'd expect - the temporary queues are removed. And even if it doesn't that should be ok - as long as we can clean things up afterwards. It's just the cleanup afterward that seems to be more difficult than I'd expected.
I'll do a bit more digging & see if something turns up in our config.
Cheers,
Andrew |
|
| Back to top |
|
|
| bower5932 |
| Posted: Wed Jun 01, 2005 5:43 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
I'd suggest that you run the DumpBroker utility at:
http://www.developer.ibm.com/isv/tech/sampmq.html
to see what the broker thinks it has subscribed to it. This might shed some light on what is wrong. You can run it before and after your cleanup to see if the cleanup worked or if there might be some other errors hiding. |
|
| Back to top |
|
|
| kolthorr |
| Posted: Wed Jun 01, 2005 6:47 am Post subject: |
|
|
Apprentice
Joined: 09 Mar 2002
Posts: 31
|
More & more interesting. I can reliably recreate 'orphaned' subscriptions by deliberately crashing our app. Then running a combo of DumpBroker, UnSubscribe and the JMS cleanup utility to clean things up - they don't get cleaned up!
Running DumpBroker gives output like:
| Quote: |
Publishers
--------------
Stream Name: SYSTEM.BROKER.DEFAULT.STREAM
Subscribers
--------------
Stream Name: SYSTEM.BROKER.DEFAULT.STREAM
Topic: dealing/orders/history/updates
BrokerCount: 0
ApplCount: 3
AnonymousCount: 0
RegistrationQMgrName: LONDLG01
RegistrationQName: SYSTEM.JMS.ND.4288C6DB20DDC903
RegistrationUserIdentifier: mqim
RegistrationOptions: 0 : MQREGO_NONE
RegistrationTime: 2005060114293278
RegistrationQMgrName: LONDLG01
RegistrationQName: SYSTEM.JMS.ND.4288C6DB20DDC603
RegistrationUserIdentifier: mqim
RegistrationOptions: 0 : MQREGO_NONE
RegistrationTime: 2005060114282449
RegistrationQMgrName: LONDLG01
RegistrationQName: SYSTEM.JMS.ND.4288C6DB20DDC403
RegistrationUserIdentifier: mqim
RegistrationOptions: 0 : MQREGO_NONE
RegistrationTime: 2005060114263857
Topic: MQ/S/LONDLG01 /Subscribers/Identities/*
BrokerCount: 0
ApplCount: 1
AnonymousCount: 1
|
Those queues exist. At this point further updates on the topic build up on those queues but are not consumed.
Then I ran the JMS Cleanup util with NONDUR and the queues were removed as expected. But re-running DumpBroker shows the same output again - in other words the broker still thinks the subscriptions are active, and we start seeing 2085 errors from the broker.
Running the UnSubscribe util with the above as input from a file like this:
java -Df=dumpout UnSubscribe LONDLG01
shows:
| Quote: |
Reading deregister data from file: dumpout
Deregister topic: dealing/orders/history/updates
Deregister stream: SYSTEM.BROKER.DEFAULT.STREAM
Deregister queue: SYSTEM.JMS.ND.4288C6DB20DDC903
Deregister correlid:
Unsubscribe failed for reason ReasonCode 3078. Enter mqrc 3078 on the command line to get a description.
|
(repeated for each of the subscriptions)
There was a brief discussion here about the 3078 errors but didn't seem to be resolved.
The only way I've found to make the broker forget is stop/delete/start of the broker - ugh.
Would it be worthwhile patching to CSD10 and retrying? We're on 08 currently...wonder if there's something there. I should mention we're using straight java JMS - nothing fancy like constructing our own broker commands. We're creating a TopicConnectionFactory programmatically and setting the appropriate props to connect as a client, then connecting as an mqim user:
| Code: |
String mquser = "mqim";
String clientId = null;
// Create a topic connection
tConn = tcf.createTopicConnection(mquser, "");
if (clientId != null) {
tConn.setClientID(clientId);
}
|
We're passing in null for clientId as these are non-durables, so the 'setClientID' call is never made.
If it doesn't work with CSD10 I'll raise some support with IBM & post back with the fix.
Cheers,
Andrew |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 01, 2005 1:04 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Are you sure leaving the setClientID = null is the right thing to do?
I would have expected subscriptions to distinguish clients using ClientID....
 |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Mon Jul 25, 2005 8:48 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
All,
This is interesting. I'm at a client site and they appear to be having the same type of issue except instead of the ClientID being blank, all clients (subscribers) are using the same ID.
It is happening on a Solaris 8 box with WMQ V5.3 CSD08.
I wonder if the broker is having a problem when the ClientID blank or the same across all subscribers? Maybe I missed it, but I have not read anything about subscribers requiring unique ClientIDs.
Another odd thing, how come DumpBroker does not output the Publisher information. I get the same output as kolthorr above.
i.e.
| Code: |
Publishers
--------------
Stream Name: SYSTEM.BROKER.DEFAULT.STREAM
Subscribers
--------------
Stream Name: SYSTEM.BROKER.DEFAULT.STREAM
Topic: blah, blah, blah |
Unders Subscribers, DumpBroker outputs lots of information about topics, subscribers, etc... but I always get nothing for Publishers. But I know there are active Publishers.
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jul 25, 2005 9:00 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There's a sample program included now, amqspsd, that provides substantially better information than DumpBroker.
And you don't have to compile it. It should be in <samples>/bin.
It does require the creation of a model queue with a fixed name.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| clindsey |
| Posted: Mon Jul 25, 2005 9:05 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
Roger,
Try running /opt/mqm/samp/bin/amqspsd to see if it gives you any publisher data.
DumpBroker was based on amqspsda.c a few years back (pre-JMS) as a sample and it only implemented around 80% of the features. It was intended to be a sample to show how java could be used as an alternative to c. I am certain it needs a few updates to catch up.
Charlie |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Mon Jul 25, 2005 9:23 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Ok I tried amqspsd and I do get more information but nothing for the Publishers.
| Code: |
mqm@ab02:/opt/mqm/samp/bin> amqspsd -m QMGRNAME
WebSphere MQ Publish/Subscribe Dumper
Start time: Mon Jul 25 13:17:37 2005
Broker Relations
----------------
QMgrName:
QMGRNAME
Parent:
None
Children:
None
Streams supported
-----------------
SYSTEM.BROKER.DEFAULT.STREAM
SYSTEM.BROKER.ADMIN.STREAM
Publishers
----------
StreamName: SYSTEM.BROKER.ADMIN.STREAM
None
StreamName: SYSTEM.BROKER.DEFAULT.STREAM
None
Subscribers
-----------
StreamName: SYSTEM.BROKER.ADMIN.STREAM
Topic: blah, blah, blah |
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jul 25, 2005 9:29 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| RogerLacroix wrote: |
| Ok I tried amqspsd and I do get more information but nothing for the Publishers. |
| Using Pub/Sub wrote: |
Choosing not to register
Publishers do not have to register with a broker. This saves the programming overhead of performing the registration, and of deregistering when the publisher has finished. However, other applications cannot find out about unregistered publishers because they do not appear in metatopics (see Metatopics for information about these). Unregistered publishers can send Publish command messages to the broker, specifying that they do not want the broker to perform an implicit registration, provided that both of the following statements are true:
The publisher does not need to be listed in the metatopics
The publisher's stream is already known to the broker |
If the publisher doesn't register, there's not a lot you can do about it.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Tue Jul 26, 2005 6:22 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
| Quote: |
| If the publisher doesn't register, there's not a lot you can do about it. |
Well, that's really annoying from an MQ Admin point of view. Now you cannot prove or disprove what the developer / support person is telling you.
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jul 26, 2005 6:25 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| RogerLacroix wrote: |
| Well, that's really annoying from an MQ Admin point of view. |
What isn't? 
| RogerLacroix wrote: |
| Now you cannot prove or disprove what the developer / support person is telling you. |
Neither can they prove they've actually published anything...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
