| Author |
Message
|
| Takidoso |
 Posted: Tue Apr 10, 2007 2:26 am Post subject: MQ-Series Transaction safety / commit |
 |
|
Novice
Joined: 08 Mar 2007
Posts: 19
Location: Germany
|
Hello to all,
in my application i am just reading datasets from files of an directory produce messages and send tehm via mq. And it is also reading messages from mq and collect them in files. Each messge represents a dataset.
My question is now: Imagine my program would be killed, is there a kind of mq-mechanism making sure that my series of produced messages would not double when I restart the program with the file that did not go totaly through? Or on the other direction if teh program is killed reading from MQ is there a good way to make sure that none of teh messages get lostwhen it was stopped in reading the message but did not have any oportunity writing it to file?
In databases this would be done by giving a commit. Does something like this also exist in MQ-Series? If not what is the best practice?
Thanks for hints and suggestions in advance
Takidoso |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Apr 10, 2007 2:36 am Post subject: Re: MQ-Series Transaction safety / commit |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Takidoso wrote: |
In databases this would be done by giving a commit. Does something like this also exist in MQ-Series?
|
That exact mechanism exists in MQ. Look up "Syncpoint" in the documentation.
It's also possible (but optional) to link MQ via an XA mechanism to coordinate messages with database work. You may or may not need this. It's certainly not required for the simple non-duplication mechanism you're requesting.
Happy Reading! 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sieijish |
| Posted: Tue Apr 10, 2007 7:11 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2004
Posts: 67
Location: London
|
If you use sync point you won't lose any message but you have a good chance of getting duplicates.
put message to MQ at syncpoint
commit to MQ
Client can delete record / flag record as sent
So if the client crashes while committing, you don't know whether it is sent or not. So you will have to send it again.
You will have to invest a 2-phase style logic, with two messages, one to put the message in the Q and a second to release it from the queue. Without this you cannot ensure once & once only delivery. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Apr 10, 2007 7:17 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Just use MQ as the XA coordinator for 2-phase commit over the queue and the database if you really want to ensure sending once only.
If the client crashes before the commit, the message won't get sent. The window for duplicates is really very small with single-phase commit and you could always check for duplicates in the receiving application by examining the last stored message id against the one just got from the queue. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 10, 2007 7:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sieijish wrote: |
So if the client crashes while committing, you don't know whether it is sent or not. So you will have to send it again.
|
The window with single phase commit is fairly small, and good enough for a fair whack of business requirements. Most scenarios are more worried about messages not getting sent than messages getting sent twice.
Though for total safety you need XA coordinated 2-phase commit. Which isn't too hard with MQ and a database.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Takidoso |
| Posted: Wed Apr 11, 2007 2:49 am Post subject: |
|
|
Novice
Joined: 08 Mar 2007
Posts: 19
Location: Germany
|
Thanks for the hints, but I often read in ur suggestions abot database and MQ-Series. In my case I have just file-processing!!!
So what can be done in this situation?
I guess I need something that keeps track on my dataset in the file. Is there something?
Are there any code examples to look at?
Takidoso |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Apr 11, 2007 1:16 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Takidoso wrote: |
Thanks for the hints, but I often read in ur suggestions abot database and MQ-Series. In my case I have just file-processing!!!
So what can be done in this situation?
I guess I need something that keeps track on my dataset in the file. Is there something?
Are there any code examples to look at?
Takidoso |
SAP has a very good example there about moving Idocs from a file to a DB (same concept).
In a nutshell you need to use syncpoint and before every commit point write the information of where to start processing the file to your storage under syncpoint in a way you can retrieve it before restarting the processing....
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Takidoso |
| Posted: Mon Apr 16, 2007 3:50 am Post subject: |
|
|
Novice
Joined: 08 Mar 2007
Posts: 19
Location: Germany
|
| Quote: |
SAP has a very good example there about moving Idocs from a file to a DB (same concept).
In a nutshell you need to use syncpoint and before every commit point write the information of where to start processing the file to your storage under syncpoint in a way you can retrieve it before restarting the processing....
|
Hi fjb_saper,
Do I need to concept and programm my own syncpoint-mechanism for this matter or is something in mq-serries framwork i allready could use?
Takidoso |
|
| Back to top |
|
|
| ashoon |
| Posted: Mon Apr 16, 2007 5:14 am Post subject: you'll have to build it |
|
|
Master
Joined: 26 Oct 2004
Posts: 235
|
| since filesystems are generally non-transactional you are going to have to build your own syncpoint mechanism into your code for reading/writing to the file... thank goodness MQ has some syncpoint control so that takes away 1/2 of your coding... |
|
| Back to top |
|
|
|
|
