ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » calculate space required to store 20000 messages of 2MB

Post new topic  Reply to topic
 calculate space required to store 20000 messages of 2MB « View previous topic :: View next topic » 
Author Message
narayanarvr
PostPosted: Mon Aug 30, 2021 1:16 am    Post subject: calculate space required to store 20000 messages of 2MB Reply with quote

Voyager

Joined: 09 Oct 2012
Posts: 84

Hi All,

I have a requirement to store 20000 messages in a local queue, each message contains 2MB of data, no application takes from this queue, but this queue needs to be cleared at the end of the day once work finished using clear qlocal, my question is how much disk space is needed to store 20000 messages of 2MB each to all messages come and sit in a queue, I have 50GB space now, I am using MQ 9.1.0.4 and Linux 8 system, do we really need 20000 x 2MB = 40GB + space or is it bit different?

Thanks,
IN advance.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Aug 30, 2021 4:22 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Yep, simple math. Size of your application data payload plus size of MQMD plus any other headers times number of messages.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
narayanarvr
PostPosted: Mon Aug 30, 2021 5:46 am    Post subject: Reply with quote

Voyager

Joined: 09 Oct 2012
Posts: 84

Thank for time and quick reply, I will make a note of it
Back to top
View user's profile Send private message
Andyh
PostPosted: Mon Aug 30, 2021 6:33 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

Simple adding the payload sizes and the header sizes will give you a good initial estimate of the space needed for the queue file, but as with so many things when you dig a little deeper things become a little more complicated.
From your usage pattern this doesn't sound like a traditional queue, a 'put only' queue where the messages are "cleared" at the end of the day (by "cleared" do you mean purged ??). With a pattern of a series of MQPUT's and then a series of MQGET's that empties the Q (or a CLEAR QL) there won't be any space fragmentation issues, but in general it would be sensible to apply a fudge factor to allow for this sort of consideration.
For smaller messages it's also worth noting that the smallest unit of queue space allocation possible is 512 bytes, for example a 100 byte message with an MQMD will use 1KB of space, rather than 100 + sizeof(MQMD).
Not also tht if you're using linear logging you'll need to think carefully about the potential impact of a media image while the queue hosts 40GB of user data!
Back to top
View user's profile Send private message
narayanarvr
PostPosted: Mon Aug 30, 2021 6:52 am    Post subject: Reply with quote

Voyager

Joined: 09 Oct 2012
Posts: 84

Hi Andyh,

Thanks for your reply, as you mentioned it is 'put only' queue, I want to purge all the messages at the end of the day, after closure of business, they give 2 hours time to clear this queue, when you say potential impact of a media image while the queue hosts 40GB of user data! do you mean any more disk space cause issues? any suggestions? what would be better solution if in case moving 20000 messages to a local queue clearing at the end of the day is not good solution? we clear every day, there is a job running every day, any thoughts ?

Thanks.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Aug 30, 2021 7:00 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Andy’s is correct.

Are the messages Persistent? If so, an image of each message is written to the log-holding file system, as well as to the queue-holding file system.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
narayanarvr
PostPosted: Mon Aug 30, 2021 7:08 am    Post subject: Reply with quote

Voyager

Joined: 09 Oct 2012
Posts: 84

Hi Bruce,

Thanks for your point, as of now not yet decided on persistence, mostly it would be non-persistent from your point it is better.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Aug 30, 2021 9:43 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

narayanarvr wrote:
Hi Bruce,

Thanks for your point, as of now not yet decided on persistence, mostly it would be non-persistent from your point it is better.


I didn’t make a recommendation as to message persistence. That decision is based on business requirements.

Disk space is cheap and plentiful. Most of my clients have terabytes of it.

What is your concern, really?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Aug 30, 2021 4:04 pm    Post subject: Re: calculate space required to store 20000 messages of 2MB Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

narayanarvr wrote:
Hi All,
I have a requirement to store 20000 messages in a local queue, each message contains 2MB of data, no application takes from this queue, but this queue needs to be cleared at the end of the day once work finished using clear qlocal, my question is how much disk space is needed to store 20000 messages of 2MB each to all messages come and sit in a queue, I have 50GB space now, I am using MQ 9.1.0.4 and Linux 8 system, do we really need 20000 x 2MB = 40GB + space or is it bit different?
Thanks,
IN advance.

What happens if one day the app puts 30,000 messages, and you only have a 50GB file system? You should allow a significant margin for growth or peak usage, at least double of current usage. Disk space is cheap.

To reduce disk usage, you could clear the queue every hour, or have a long running app that consumes messages and does nothing with them.
_________________
Glenn
Back to top
View user's profile Send private message
narayanarvr
PostPosted: Mon Aug 30, 2021 9:08 pm    Post subject: Reply with quote

Voyager

Joined: 09 Oct 2012
Posts: 84

Hi Gbaddeley,

Yes, I will take your points and I will request for additional disk space as a margin of safety.

Thanks for your point.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Aug 31, 2021 3:34 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

If this is production, /var/mqm, /var/mqm/errors and /var/mqm/qmgrs should be on separate file systems. There should also be monitoring in place that will alert your support team if used space exceeds (say) 80% on any file system.

Quote:
I have a requirement to store 20000 messages in a local queue, each message contains 2MB of data, no application takes from this queue...

Can you elaborate on the requirement? From a MQ design standpoint, this does not make sense.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » calculate space required to store 20000 messages of 2MB
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.