| Author |
Message
|
| zpat |
 Posted: Sat Jun 29, 2013 10:52 pm Post subject: |
 |
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
But then you consume a lot of memory. What happens if a thousand messages arrive at once - how many instances would you need to have?
Coming from a mainframe background (when times were hard and memory expensive) I prefer a design that is ultra-efficient, and it can also be scaled up to provide truly massive throughput.
However this may not be necessary for low volumes. But always consider what needs to happen if no response is received. I have seen developers who simply ignore that possibility - which is ludicrous. |
|
| Back to top |
|
 |
| Tibor |
| Posted: Sun Jun 30, 2013 12:21 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| zpat, basically I had agreed with you from the beginning, I just thought about a technical solution for the question. |
|
| Back to top |
|
|
| Sunrise |
| Posted: Tue Jul 02, 2013 2:16 am Post subject: |
|
|
Newbie
Joined: 27 Jun 2013
Posts: 3
|
Finally i Convinced my Manager used for MQ Get Node, It was working properly
Thank you folks. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 02, 2013 2:20 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Sunrise wrote: |
Finally i Convinced my Manager used for MQ Get Node, It was working properly
Thank you folks. |
Well done Sir! Get a gold star!
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| MasterGD |
| Posted: Tue Aug 19, 2014 10:42 pm Post subject: |
|
|
Novice
Joined: 21 Feb 2013
Posts: 10
|
Dearr Sunrise,
Could u please explain us how did u achieve this . im facing same issue im trying to delay the flow using MQ get node but its not working .. Kindly Explain us the flow ..
My flow is
mqinput----> compute (Some DB insert and update operations)
In esql part:
im using condition for different operations..
---some esql code
ex: If logpoint = res
PROPARATE to TERMINAL 'out1' finalize none delete none;
--Some more esql code--
this out1 is connected to MQ getnode with wait interval 10000 ms..
but as i observed once it it propagated to terminal out1 control is not coming back to compute node..
kindly let me know if there is any mistake |
|
| Back to top |
|
|
| MasterGD |
| Posted: Tue Aug 19, 2014 11:10 pm Post subject: |
|
|
Novice
Joined: 21 Feb 2013
Posts: 10
|
Great Delay is working properly with Mqget Node..
previously the mistake was , if condition is not satisfying so propagate statement was not executed...
THANKS:-)  |
|
| Back to top |
|
|
| mehta10987 |
| Posted: Tue Oct 15, 2019 2:32 pm Post subject: Causing Delay for one instance alone |
|
|
Novice
Joined: 12 Oct 2019
Posts: 13
Location: USA
|
We have a requirement to process very huge file(with multiple records and received once in a day) record by record and write failures of transactions into an Error File.
Approach we are following
First Flow
-> Using File Input Node to Read the File Record by Record.
-> Each Record is converted to XML Message
--> XML message is dropped to a Queue
Second Flow
--> MQ INput Node would read the XML messages from Queue
--> perform two DB calls, one service call and enrich the message and finally call to another backend service with overall data
In case of failures/exceptions with any backend, the records need to be written into Error file.
Since the Input file received is huge , we are creating multiple instances of the Second flow Queue Component to read messages from Queue and then write to File(we are using "stage to mqsitransit directory.. " method just to make sure that file is moved to actual folder when writing to it is completed) in case of Errors.
While doing Unit testing in local(with 1200+ records) it is seen that the records in Error file is not matching with actual failures. It seems that the indicator in MQ which tells the "End of File = TRUE" is read first and the file is closed even before other messages are still being processed in other instances of Second Flow.
Since the file will be received once in Day, I was thinking to put sleep of the instance which receives the End Of File Indicator for 60 seconds.
Since the file would contain 100K records and even with 30 instances running in parallel, it would take anywhere around 3+ hours to consume all messages.
SO putting the thread to sleep for 60 seconds .. Is it bad?
Also if I use in ESQL
DECLARE returnValue BOOLEAN;
SET returnValue = SLEEP(60000);
If someone can suggest a better option in this?
 |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Oct 15, 2019 3:21 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Usually if there an MQ Input node for processing, all errors an exceptions should be logged on a MQ Output queue, eg. .FAULT, .ERROR, with messages containing details of the error and the original input data. This is much easier to manage than writing an Error File. It also allows a design capability to move the messages back into the input queue to reprocess them, if there is a transient backend DB error or service error.
Sleeping is a very bad idea from a design standpoint.
Do you really need an End Of File indicator message? Do the multiple instances of the second flow actually need to do anything specific at "End of File" ?
_________________
Glenn |
|
| Back to top |
|
|
| mehta10987 |
| Posted: Tue Oct 15, 2019 3:33 pm Post subject: How to delay the Message Flow using esql code in MB6.1 |
|
|
Novice
Joined: 12 Oct 2019
Posts: 13
Location: USA
|
The input file to IIB is a huge pipe delimiter file, which is to be read record by record and then XML is to be created(for each record) for hitting the final destination service in second flow.
We are creating the Error Report File for business. They would not be actually verse with looking into MQ.
We are also propagating the pipe delimiter record in the second flow so that it could be written to the Error file. The reason for that is, In case there is data related issue for the record that would be corrected and send later with another batch(pipe delimiter file).
SO the ask is our Error File should have all the records (in pipe delimiter format) which got error,
The file which we are creating is a csv file which will have Error, ErrorCode, UniqueRecordNumber and ActualRecord(in pipedelimiter format) so that they take all the data error related records and update with correct data and push again with the second batch file. |
|
| Back to top |
|
|
| mehta10987 |
| Posted: Tue Oct 15, 2019 3:38 pm Post subject: |
|
|
Novice
Joined: 12 Oct 2019
Posts: 13
Location: USA
|
| gbaddeley wrote: |
Usually if there an MQ Input node for processing, all errors an exceptions should be logged on a MQ Output queue, eg. .FAULT, .ERROR, with messages containing details of the error and the original input data. This is much easier to manage than writing an Error File. It also allows a design capability to move the messages back into the input queue to reprocess them, if there is a transient backend DB error or service error.
Sleeping is a very bad idea from a design standpoint.
Do you really need an End Of File indicator message? Do the multiple instances of the second flow actually need to do anything specific at "End of File" ? |
-- when records are read from the file and put into the queue, there is also an indicator message in Queue which tells the End Of File = TRUE.
We are using this to close the file getting written |
|
| Back to top |
|
|
| timber |
| Posted: Fri Oct 18, 2019 7:59 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
How about this:
- When Flow 1 writes the end-of-file message, it includes the record count (i.e. the number of records in the input file).
- Flow 2 declares and uses a SHARED INTEGER counter. This is visible to all instances of Flow 2. Whenever Flow 2 completes the processing of a message (i.e. after the FileOutput node), it performs a thread-safe increment of the counter. This will require some careful use of ATOMIC blocks in your ESQL.
- When an instance of Flow 2 receives the end-of-file message, it goes into a loop and waits for the shared counter to reach its final value. This cannot happen until all the other instances have finished their processing.
I'm probably missing some important detail, but it will be entertaining to find out what it is. |
|
| Back to top |
|
|
| mehta10987 |
| Posted: Fri Oct 18, 2019 8:42 am Post subject: How to delay the Message Flow using esql code in MB6.1 |
|
|
Novice
Joined: 12 Oct 2019
Posts: 13
Location: USA
|
When you mention
".. it goes into a loop and waits for the shared counter to reach its final value.."
What I understand to put that instance to sleep and then return in the loop to check if counter gets match, If Yes then
This is exactly what we tried with Shared Variable but ended up with counter increment discrepancy.
Hence it went to infinite loop and had to connect with Infra Team to kill the resources on Server.
I proposed to either go ahead with MQGet Node and put a Delay there but
Since this would be one time run on a daily basis and even with 30 instances running in parallel, it would take 3+ hours to process 100K records, they were ok to put the thread on sleep for 120 seconds.
Since before that other threads running(with last batch of 30 records) would either have completed the processing or would have time out error.
So after two minutes the Error File would get closed and file will be moved from mqsitransit to main folder. |
|
| Back to top |
|
|
| timber |
| Posted: Sun Oct 20, 2019 10:50 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
What I understand to put that instance to sleep and then return in the loop to check if counter gets match, If Yes then
This is exactly what we tried with Shared Variable but ended up with counter increment discrepancy. |
Did you mention this? If you did, then I missed it.
If you did implement the solution that I suggested then I suspect that you got your ATOMIC blocks wrong, and got into a deadlock situation.
Go ahead with the delay - it sounds as if it will not be too risky. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Oct 20, 2019 2:59 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| timber wrote: |
| ...I'm probably missing some important detail, but it will be entertaining to find out what it is. |
An essential piece of detail is to consider what will happen in all possible failure modes. eg. One or more messages are missing / bad format / rolled back. The counter never reaches its "end of file" value.
It is a bad idea to mix transaction messages and indicator messages on one queue, if there are multiple consumer flows.
Given that there is one huge file per day, that takes several hours to process, you could use a processing threshold technique to determine end of file. eg. Per time interval (eg. 60 seconds YMMV), record how many messages the flow has processed. If the number has dropped from a significant number to zero over successive intervals, treat it as end of file. Set a flag so that other instances of the flow do not also treat it as end of file.
_________________
Glenn |
|
| Back to top |
|
|
|
|
