| Author |
Message
|
| yps_sengar |
 Posted: Tue Feb 09, 2010 10:49 pm Post subject: Appending message records to a pre-existing file |
 |
|
Newbie
Joined: 05 Feb 2010
Posts: 9
|
Hi,
we have a requirement where we need to read messages from an MQ Queue and write the same to a pre-existing file in IBM-AIX unix system (Local broker box). The flow will read the messages in a batch mode say after each 30 minutes and write to the same file. at the end of day say 6:00 PM, it should FTP the file to a different FTP server (IBM-AIX) for some downstream systems to process further.
We have an existing appraoch which reads from MQ queue and store message information in database and at the end of the day it creates a file reading all the messages from database to a file. But we don't want to store the messages in database. we want to process these message directly from queue in multiple batch mode as explained before.
Please suggest the best design approach to be followed.
We are trying to make use of FileOutput node to append messages to a pre-existing file. We are struggling somehow to achieve this. Pls. let me know your suggestions. |
|
| Back to top |
|
 |
| zpat |
| Posted: Wed Feb 10, 2010 5:00 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Simple - keep the messages accumulating in the queue until 6 PM, then start a flow on a timer to get all the messages and write to a file (fileouput node).
Don't get hung up on appending to existing files. That is not the requirement. The requirement is to process the messages at 6PM and send a single file out via FTP. |
|
| Back to top |
|
|
| yps_sengar |
| Posted: Wed Feb 10, 2010 8:46 am Post subject: |
|
|
Newbie
Joined: 05 Feb 2010
Posts: 9
|
| Hi zpat, thank for your reply. We thought the same way before but the customer MQ Admin team is saying they will not allow accumulating messages in queue. As per them its not a best practice. We have database before to download messages as disccused before. But customer don't want to use database also. As downstream system expect flat file only. We need to devise a solution using file handling approach only. Let me know how we can achieve this. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 10, 2010 9:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| yps_sengar wrote: |
| We thought the same way before but the customer MQ Admin team is saying they will not allow accumulating messages in queue. As per them its not a best practice. |
Tell them not to be so pedantic. It's not best practice to store messages in a queue and use it like a database, or like a configuration file (example - a message in a queue contains business data & parameters which each application browses on startup) but it's perfectly legitimate to hold messages in a queue while they're waiting to be processed.
As I understand this your plan is to allow the messages to accumulate for 30 mins, read the queue until it's empty, writing the message contents into a file. Every 30 mins new data is appended to the file.
This is functionally identical to allowing the messages to accumulate for the 6, 8, or 24 hours (depending on business day & other considerations) until 6pm comes round again and the queue is emptied.
This is a perfectly workable solution and to say it's not "best practice" is hair splitting. If these messages were accumulating for an unknown period of time, or the 6pm only ever read the 1st 1000 on the queue then I'd agree. But the plan is to read the queue on a regular schedule and until empty.
Trout them.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| yps_sengar |
| Posted: Wed Feb 10, 2010 9:11 am Post subject: |
|
|
Newbie
Joined: 05 Feb 2010
Posts: 9
|
| Thanks a lot Grand. Our Architect has also asked the same question to MQ admin team. They say it is suggested by IBM. They should not accumulate messages in Queue. I am not sure if you guys are from IBM. Is it true? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 10, 2010 9:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| yps_sengar wrote: |
| They say it is suggested by IBM. |
Get them to name names, or post a link if they're been specifically advised to avoid this scenario. Remeber that in general they're right and cases such as the ones I named above are to be avoided. It sounds a lot like they're taking this general advice (and it's only advice) too literally.
| yps_sengar wrote: |
| I am not sure if you guys are from IBM. Is it true? |
Some of us are, some of us are not, no one (AFAIK) is authorised to speak for IBM via this forum.
All of us are WMQ professionals of varying levels of experience. Speaking personally, I've been using WMQ professionally for rather more than 2 decades and feel I can speak with some level of experience on general design points! 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 10, 2010 9:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
As licensed users of IBM software, you do of course have the right to ask IBM, referencing this thread, for advice in your specific case.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Feb 10, 2010 10:16 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| yps_sengar wrote: |
| Thanks a lot Grand. Our Architect has also asked the same question to MQ admin team. They say it is suggested by IBM. They should not accumulate messages in Queue. I am not sure if you guys are from IBM. Is it true? |
There are a number of posts on this forum which state 'IBM told us to do it this way' that (I'm guessing here) make the IBM'ers from Hursley who post here tear their hair out with anger at some of the so called advice that some illinformed of their colleagues give customers.
As with the saying, "Ask 6 lawyers a question and you will get 7 answers", there is often more than one way to solve the problem with WMQ/Broker.
In you case, your MQ Admins might also be saying this because they have a blanket Queue /monitoring rule setup that will set off all sorts of alarms if even 10 messages are in a queue (I've seen this in Production). And they can't be bothered (or don't know how to) to setup an exception for your queue(s)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| yps_sengar |
| Posted: Wed Feb 10, 2010 10:35 am Post subject: |
|
|
Newbie
Joined: 05 Feb 2010
Posts: 9
|
Thanks smdavies99 & Victor for your valuable view points.
we gone through multiple meeting with MQ team and our architects from customer end. They have not responded back saying why they don't want to accumulate messages in the Queue for a day which are waiting to be processed at the End of the day. I will definately discuss again with our team and MQ team and update you guys on that.
But I have one query in Mind:
If suppose we want to run our flow multiple times in a day and want to write to the same file. How can we achive that? if this is not a good design, can some one give some bullet point to prove that we should not be using this approach because of ...? This will help me to put my points strongly in the meeting....... |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Feb 10, 2010 10:42 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Just remember that if you do use the queue as your temporary data store, make sure the messages are persistent.
I just wondered if your MQ admins had a policy that made all app queues non persistent. Then if apps write messages with the MQPMO options AS_Q_DEF, the messages will be non persistent.
And we all know that happens when you restart a QMGR with queues holding non persistent messages.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 10, 2010 10:49 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| I just wondered if your MQ admins had a policy that made all app queues non persistent. |
It's an odd policy unless the admins a) have the ability to enfore the use of that PMO b) know that all app messages have the ability to be recreated by the app c) know that this will never change.
It's a fortunate position to be in if they are.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| yps_sengar |
| Posted: Wed Feb 10, 2010 11:00 am Post subject: |
|
|
Newbie
Joined: 05 Feb 2010
Posts: 9
|
| Quote: |
smdavies99 wrote:
I just wondered if your MQ admins had a policy that made all app queues non persistent. |
No, they supports both persistence & Non-persistence queues. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 10, 2010 2:24 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| yps_sengar wrote: |
| Quote: |
smdavies99 wrote:
I just wondered if your MQ admins had a policy that made all app queues non persistent. |
No, they supports both persistence & Non-persistence queues. |
As for being pedantic...queues have no persistence... messages do. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Feb 10, 2010 11:08 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Persistent messages are of course held by MQ in files.
However these files are managed for you by MQ, which is a much better option than trying to manage your own files for interim storage of messages.
Especially since the standard fileoutput node does not support your original design.
We are not (all) IBM, however we have decades of experience. Using a queue for a day's worth of messages is perfectly acceptable, providing your queue manager has adequate disk capacity and queue's max depth is set to a suitable value.
Don't forget messages only use the disk space needed, the maxmessagesize is just the max, and the maxdepth is not preallocated in space terms. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Fri Feb 12, 2010 12:19 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
But consider this scenario:
If for some reason the queue started collecting large amounts of messages, it could flood. If messages were then still being collected, the DLQ could flood, followed by transmission queues. When that happens, all flows depending on that broker could experience serious performance issues.
In my job, I deal with flows that process millions of messages in the course of one day. Average queue depth is measured in thousands, not millions.
So it's best not to use queues as a storage device. |
|
| Back to top |
|
|
|
|
