| Author |
Message
|
| abhyyy |
 Posted: Mon Nov 14, 2011 1:44 am Post subject: Reading unprocessed records from file after system restart. |
 |
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
Hi Friends,
I am doing a new project and got stuck at a point (explained below). If anybody has ever done this or have any idea about this, your view would really help.
Scenario : Flow is reading from a file having 50,000 records originally and have already processed 25,000 records and sent 1 message per record to SMTP server and now the sysem goes down.
Now, when the system restarts, as per my understanding the flow will again start reading from the file having 50,000 records (as records dont get deleted on processing, only file as a whole is deleted or moved) and will read the file from first record again and again for first 25,000 records , duplicate messages will be sent to users/SMTP server.
This is what I dont want !!
Is there a way that after system restart I can read from the last record that I read?
or Is there a way we can delete processed record from the file being read on record by record basis??
Thanks and Regards,
Abhishek Gupta.
Nobody Knows Everything. |
|
| Back to top |
|
 |
| zpat |
| Posted: Mon Nov 14, 2011 2:35 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes, use MQ messages and stop using files for transactions. It's very old-fashioned and one of the main reasons that "modern" distributed systems are so fragile and prone to errors.
To reduce the problem, code a simple flow to move the file records to MQ messages output (and do nothing else in this flow). This will run quickly and there is little that can go wrong with this.
Then code your main flow as MQ messages input (the same queue) to SMTP (I assume you are using the new Email node for this). |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Nov 14, 2011 5:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Even better, change the upstream process that is creating the file to send MQ messages in the first place. |
|
| Back to top |
|
|
| abhyyy |
| Posted: Tue Nov 15, 2011 12:14 am Post subject: |
|
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
Thanks for your responses. 
But the only 2 ways I can reveive data is via a File or in a database.
If I use
1) i/p from a file and throw the records from a file into MQ one by one then when the system goes down, after system restart how to read the unprocessed records from file as file originally holds 50,000 to 100,000 records and MQ cant hold these many records.
2) If I want to read from the database , is there a way that my flow can be triggered from database records??
_________________
----------------------
NeVeR StOp LeaRnInG. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Nov 15, 2011 1:46 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| abhyyy wrote: |
If I use
1) i/p from a file and throw the records from a file into MQ one by one then when the system goes down, after system restart how to read the unprocessed records from file as file originally holds 50,000 to 100,000 records and MQ cant hold these many records.
|
Eh? Where did you read that WMQ can't hold 100K messages in a queue?
I have developed a broker flow in the past that took a file and wrote individual messages to a Queue for onward processing. There were many times when the queue help well in excess of 250K messages.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqmatt |
| Posted: Tue Nov 15, 2011 2:49 am Post subject: |
|
|
Grand Master
Joined: 04 Aug 2004
Posts: 1213
Location: Hursley, UK
|
You can use the V7 Resequence node downstream; this will discard duplicate records when you restart, as it can persist state.
You'll need to set the sequence identifier to be LocalEnvironment.File.Record so that the sequence number is derived from the record number in the file. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 15, 2011 5:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| abhyyy wrote: |
| file originally holds 50,000 to 100,000 records and MQ cant hold these many records. |
Rubbish. Unless you're short on disk.
Or left max depth at the default.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 15, 2011 6:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
For the record, the minimum maximum (if you see what I mean) given here is 640,000 records in a queue. On all common platforms it's 999,999,999.
Subject to disk space obviously, but you've got the original file stored on something.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 15, 2011 12:29 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
For the record, SAP has been dealing very well with this problem for years.
If you need some details talk with your friendly SAP ALE specialist.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| abhyyy |
| Posted: Sun Nov 20, 2011 12:32 am Post subject: |
|
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
Thanks everybody for your view and time.
I tried it with MQ and problem seems to be solved now.
_________________
----------------------
NeVeR StOp LeaRnInG. |
|
| Back to top |
|
|
|
|
