ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » Inactive queue load / unload messages to disc ?

Post new topic  Reply to topic
 Inactive queue load / unload messages to disc ? « View previous topic :: View next topic » 
Author Message
Jhap
PostPosted: Thu May 28, 2015 10:16 am    Post subject: Inactive queue load / unload messages to disc ? Reply with quote

Novice

Joined: 18 Feb 2010
Posts: 18

Hello !

Just need to clarify next claim.

If MQ has example one million persistent messages stored in local queue without continuously reading those messages out. After few minutes MQ start to behaves like that, it loads / unloads those messages to disc and also writes note from this operation to log between every 10 000 messages ?

During this operation all other functions concerning this queue, like receiver channel go to wait status ?
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu May 28, 2015 10:20 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

If MQ has 1 million messages loaded to a queue, your program is completely broken and you will get horrible horrible performance.

Your app developers should also be fired.

Or at least taken out back and trouted for a while.

Back to top
View user's profile Send private message
Vitor
PostPosted: Thu May 28, 2015 10:30 am    Post subject: Re: Inactive queue load / unload messages to disc ? Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Jhap wrote:
If MQ has example one million persistent messages stored in local queue without continuously reading those messages out. After few minutes MQ start to behaves like that, it loads / unloads those messages to disc and also writes note from this operation to log between every 10 000 messages ?


I fully agree with my most worthy associate. Even if the software can handle it, that's a truly horrible design.

Jhap wrote:
During this operation all other functions concerning this queue, like receiver channel go to wait status ?


All other functions will go into a wait state; the log is not at a queue level but at a queue manager level. It also contains one million duplicate copies of these messages so they can be recovered if the queue file is corrupted (you say log so I say persistent) so the I/O will be through the roof. Imagine the start up time if corruption occurs and if it has to roll through the log to recover. Imagine if you just restart it and it verifies one million persistent messages in the queue against the log images.

Even if they're not persistent, woe betide you if you do a controlled stop and controlled start of the queue manager and thus make the queue manager do an integrity check of the queue file. That's going to take a while. Woe will slightly less betide you if it crashes and just decides to restart - reinitializing a queue file that used to hold that much will still delay the start up.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Jhap
PostPosted: Thu May 28, 2015 11:00 am    Post subject: Reply with quote

Novice

Joined: 18 Feb 2010
Posts: 18

Thanks !

Problem is not poor program, problem is that there is no program at all Program is goin to start after few weeks.

Actually I just heard this and this is not my problem, I just want to know does MQ behaves like I wrote.

If messages are persistent, mq start after few minutes make copies ? Is this true ?

I thought that MQ makes copy immediately when persistent message arrives to queue and delete copy when message is put another queue and commit ?

What about if there is one million messages in queue and some program read messages slowly . MQ still makes copy of those messages which are still in queue ?
Back to top
View user's profile Send private message
Jhap
PostPosted: Thu May 28, 2015 11:15 am    Post subject: Reply with quote

Novice

Joined: 18 Feb 2010
Posts: 18

Now I understand that point that mq writes log after few minutes....

MQ start make copies immediately but first mark to log appears after few minutes when messages are started put to queue
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu May 28, 2015 11:19 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Jhap wrote:
I just want to know does MQ behaves like I wrote.


Not exactly how you wrote.

Jhap wrote:
If messages are persistent, mq start after few minutes make copies ? Is this true ?


False. It logs the message at the same time as it arrives. It also compares the log copy with the queue copy at start up and during recovery operations (e.g. media recovery)

Jhap wrote:
I thought that MQ makes copy immediately when persistent message arrives to queue and delete copy when message is put another queue and commit ?


Yes, but it removes it when the message is read off the queue and the read is committed. It doesn't have to be put anywhere.

Jhap wrote:
What about if there is one million messages in queue and some program read messages slowly . MQ still makes copy of those messages which are still in queue ?


MQ will make a copy of any new message that arrives, and retains a copy of any message not yet read off. If messages arrive while they're being read off, this causes a huge I/O penalty as the file pointer is moved to where the copy is being removed, then moved back to the end to add a record, then moved forward and so forth.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu May 28, 2015 11:20 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Jhap wrote:
MQ start make copies immediately but first mark to log appears after few minutes when messages are started put to queue




The queue manager doesn't report it in "real time", even though that's how the copies are made.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Jhap
PostPosted: Thu May 28, 2015 11:28 am    Post subject: Reply with quote

Novice

Joined: 18 Feb 2010
Posts: 18

Thanks a lot for clarifying
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu May 28, 2015 3:27 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

The message you see about 10,000 messages read/written is MQ buffer management internals telling you that it doing work for you - offloading/reloading buffers to/from disk.

When a persistent message is MQPUT to a queue, it is first written to the log portion of the file-system, then made available to any consuming (MQGETting) application in the message buffer in virtual storage.

If there is no MQGETting application, when the message buffer fills, buffer management software moves the message to the queue-holding portion of the file-system. So, worst case, two writes of the same message - one to the log and one to the queue-holding file-system.

Millions of messages produced, but no consumer? When, if ever, will the consumer come to life?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Andyh
PostPosted: Fri May 29, 2015 12:17 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

The AMQERRnn.LOG messages about "messages loaded" are nothing to do with MQ buffer management.

When a queue is first referenced, a representation of that queue is loaded into memory. One part of this memory representation is a queue index, and in order to create the queue index the queue manager needs to look at the control data (effectively the MQMD) of each message. The "messages loaded" messages are indicating how many 'MQMD's' have been accessed during this process.

The queue is loaded at first reference, and if the queue then becomes unreferenced during some sustained period of other activity the queue will be chosen to be unloaded, where the queue index will be discarded, and where buffers etc that haven't yet been written to disk would be written to disk. The intent here is to release memory that's associated with a queue that's no longer being used.

If you have a very deep queue, with an unusual access pattern (such as write only!) then it might be an idea to write a program that simply opens the queue and waits for queue manager termination. The presence of this open handle will be sufficient to prevent the queue from being unloaded, and thus the need to reload the queue at the next reference.

Of course the well known mantra that "MQ is not a database" is worth repeating here. There are some occasions where a deep queue is within acceptable messaging design goals (for example when some downstream system is unavailable a deep transmission queue), but storage of large volumes of long lived data isn't the sweet spot for a traditional messaging engine.
Back to top
View user's profile Send private message
Andyh
PostPosted: Sun Mar 18, 2018 12:23 pm    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

I cannot repeat the "MQ is not a database" mantra too frequently, but there's a little change in MQ 9.0.5 that might benefit some users of deep queues.

MQ 9.0.5 includes a small change to try to reduce the frequency of queue unload/reload.
Prior to 9.0.5 the algorithm that decides when a queue is unloaded used the time and amount of activity since the queue was last referenced in order to decide when to unload the queue.
From 9.0.5 the current depth of the queue is also taken into account in an attempt to reduce the frequency of unload/reload for deep queues.
Back to top
View user's profile Send private message
tczielke
PostPosted: Mon Mar 19, 2018 8:26 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

My two cents, but the feature that Kafka has to act as a psuedo-database (e.g. store the messages on disk for a week and allow the application to replay the messages, if needed) is a handy feature that our Kafka developers leverage and IBM MQ does not offer.
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
Andyh
PostPosted: Mon Mar 19, 2018 10:15 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

Kafka is more of a message streaming service than the traditional once and once only messaging service that MQ provides.
There are clearly applications more suited to one or the other, and neither approach plays particularly well in the wrong scenario.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Mar 19, 2018 3:23 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

Andyh wrote:
The AMQERRnn.LOG messages about "messages loaded" are nothing to do with MQ buffer management.

When a queue is first referenced, a representation of that queue is loaded into memory. One part of this memory representation is a queue index, and in order to create the queue index the queue manager needs to look at the control data (effectively the MQMD) of each message. The "messages loaded" messages are indicating how many 'MQMD's' have been accessed during this process.

The queue is loaded at first reference, and if the queue then becomes unreferenced during some sustained period of other activity the queue will be chosen to be unloaded, where the queue index will be discarded, and where buffers etc that haven't yet been written to disk would be written to disk. The intent here is to release memory that's associated with a queue that's no longer being used.

If you have a very deep queue, with an unusual access pattern (such as write only!) then it might be an idea to write a program that simply opens the queue and waits for queue manager termination. The presence of this open handle will be sufficient to prevent the queue from being unloaded, and thus the need to reload the queue at the next reference.

Of course the well known mantra that "MQ is not a database" is worth repeating here. There are some occasions where a deep queue is within acceptable messaging design goals (for example when some downstream system is unavailable a deep transmission queue), but storage of large volumes of long lived data isn't the sweet spot for a traditional messaging engine.

This is correct. If the queue depth is large (>100,000), the queue index reload can take a long time (many minutes) and can impact performance of the qmgr. eg. DISPLAY QLOCAL of the queue will hang until the reload is complete, which effectively hangs the MQ command processor for other users.

The solution is to not let queues grow to a large depth. The messages should be consumed continuously or at a regular short interval, so the that the depth does not grow to (say) over 1000.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General Discussion » Inactive queue load / unload messages to disc ?
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.