| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| MQ Get Allowed from IIB |
« View previous topic :: View next topic » |
| Author |
Message
|
| iibEnthu |
 Posted: Wed Jul 03, 2019 1:47 am Post subject: MQ Get Allowed from IIB |
 |
|
Newbie
Joined: 03 Jul 2019
Posts: 2
|
Hello,
Its been some time we are trying to put in place a resilient design but getting into some sort of issues everytime. Any pointers would be of great help.
In our organisation, direct DB interaction is disliked due to the fact that if the db is down, all messages would result in failure filling the logs / monitoring system and as well as the mq file system with messages piling up in the queue, which causes impact for others flows of other projects or domains running on the same iib node.
We wanted to put in place a resilient solution to handle all kinds of issues that occurs with a db interaction. Hence we came up with this solution.
In case the connection to DB fails or some rights issue with schema or table or other similar issue which could result in failure for all the messages, we inhibit the queue, log and send emails to the support. Besides which, we also send the message to another flow which triggers a timer, after 1hr, this timer flow disables the inhibit and makes the queue to process further.
Functional errors will not trigger this chain
The flow could easily receive 100000 messages on a given day
The solution works fine the first time when messages are received one after other from the source. In this case, when the db is down, the queue is inhibited and message stay in queue.
The problem comes when the get is enabled by the timer flow, After 1hr, the DB is still down, but as the timer flow get enables the queue and further messages are processed. This time, the queue has around 70000 messages piled up and hence this time inhibit does not happen until all these 70000 has been processed.
I use the similar type of code available on the site for making get inhibit with MQPCF commands sent to the system queue. I did see the reply as well and it says "OK" for inhibit. But since the queue is open, it is not able to inhibit in the middle if my understanding is correct.
I thought of using force option, but 1st place it didnt work as expected, meaning it didnt inhibit immeadiatly. I used "MQIACF_FORCE" as parameter and "MQFC_YES" as value. Even if it did work, not sure if its good to do that as it could force the change and break the mq connection with the queue...
Still wondering how to put in place this automatic process of get enabled. anypointers would be of great help. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jul 03, 2019 5:08 am Post subject: Re: MQ Get Allowed from IIB |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| iibEnthu wrote: |
| In our organisation, direct DB interaction is disliked due to the fact that if the db is down, all messages would result in failure filling the logs / monitoring system and as well as the mq file system with messages piling up in the queue, which causes impact for others flows of other projects or domains running on the same iib node. |
I don't see why this is a problem. It's going to be a lot easier / safer to stop the application / IIB flow that's interacting with the database (and is therefore guaranteed to fail) and let the messages pile up in the queue. If MQ is sitting on a file system small enough for this to be a problem, your file system is too small.
| iibEnthu wrote: |
| The flow could easily receive 100000 messages on a given day |
Unless these messages are very, very large that shouldn't fill up a reasonably sized file system. That's certainly not a large number of messages.
| iibEnthu wrote: |
| anypointers would be of great help. |
Code your monitoring system to shut down any component directly affected by the database outage (and raise a ticket with the support people obviously). Pile up outstanding requests in MQ. Once the database is restored, restart the applications and let them chew through the message backlog.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Wed Jul 03, 2019 6:11 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
hi...i agree with Vitor's comments.
Other things you could explore are,
- configuring alternative Database servers at the ODBC stanzas if you have high available database instances so that only the first message will fail with 'connection dead' error and the flow will switch over to secondary DB instance.
- For interfaces where database is used just for static data lookup implement caching logic (shared variables are more than enough) so that database outage will not impact at all.
- Explore the usage of 'failedMessageWaitTime' setting which will be useful for DB failures if you only have one DB node. This means the flow will wait for certain time before processing the second message which means messages will not fail in bulk.
Some notes here - http://www.mqseries.net/phpBB2/viewtopic.php?p=423008&highlight=#423008
It is always better to keep things simple wherever possible otherwise you will end up with a much complex error handling/recovery logic when compared to happy path logic |
|
| Back to top |
|
|
| iibEnthu |
| Posted: Fri Jul 05, 2019 12:06 am Post subject: |
|
|
Newbie
Joined: 03 Jul 2019
Posts: 2
|
Many Thanks for your replies. Many interesting pointers.
I did find the solution of managing the flow through monitoring system and the pointers on failedMessageWaitTime could possibly fit us. Got to go deep.
A little more history of the application. Actually we are receiving referentials from a master application which are sent by a batch process during night. They reach IIB and published there for any application to pick, and there are close to 250 different flows that subscribe these datas for various usages. Well, all consume at the same time. So we multiple 100000 by 250. But the messages are always smaller in some kb, definitely less than a MB. We also have real time tracking for all these message one entry footprint and one exit footprint, I let you imagine number of footprints for all these messages.
We have multiple nodes based on various platform domains, but still there are other very critical flows running on the same nodes and what the infra feared was that due to these less critical referential problem for some application, they don't want the critical flows to be affected. Hence they asked us to avoid tight coupling with DB and use MQ to push these messages to target application and then process it there, since it could be managed at project level and any impacts are left with that application. We do this as of now, but seeing the number of tools used their cost, maintenance, support, etc, I was asking why not make a design that does not affect other applications running on the same node. And here we are..
I agree that this is not a usecase for IIB and probably a case for a ETL tool, but when all this was started 20 years ago with probably 20 application and pub-sub being a hot feature, it was done this way and the we couldn't control the evolution. At last, we see that kafka might do some good for us, but the iib kafka node still misses certain features for producer and also lacks flexibility with less options. Hoping we might have more kafka feature in the future..
Couple of questions on the above 2 points
Managing the flow through monitoring system : How would the monitoring system know the DB is down (with pile up of messages in queue or does it monitor DB as well ?) it seems a tricky point there.
On the failedMessageWaitTime, i have not used it before. is it advisable to keep the retry interval as 1hr or more? I see this same as sleep function which we use for certain retries and its not recommended for a higher interval, no idea on this failedMessageWaitTime, is it the same ?
Once again thanks for the pointers. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 05, 2019 4:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| iibEnthu wrote: |
| I agree that this is not a usecase for IIB and probably a case for a ETL tool, but when all this was started 20 years ago with probably 20 application and pub-sub being a hot feature, it was done this way and the we couldn't control the evolution. At last, we see that kafka might do some good for us, but the iib kafka node still misses certain features for producer and also lacks flexibility with less options. Hoping we might have more kafka feature in the future.. |
I don't think anyone said this wasn't a use case for IIB. I think we simply disagreed with how you were handling your use case.
| iibEnthu wrote: |
| Managing the flow through monitoring system : How would the monitoring system know the DB is down (with pile up of messages in queue or does it monitor DB as well ?) it seems a tricky point there. |
Speaking entirely personally, I was referring to whatever monitoring software your site uses to determine problems in the infrastructure. This is, we assume, already checking to see if the DB is down so it can appropriately ticket the DBA team.
| iibEnthu wrote: |
| On the failedMessageWaitTime, i have not used it before. is it advisable to keep the retry interval as 1hr or more? I see this same as sleep function which we use for certain retries and its not recommended for a higher interval, no idea on this failedMessageWaitTime, is it the same ? |
It's not the same a sleep function and you should not be using a sleep function to control retries. Use a Timer node for retries.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
