| Author |
Message
|
| yesh |
 Posted: Fri Aug 28, 2015 2:18 am Post subject: Adding one header in file for multiple request. |
 |
|
Newbie
Joined: 19 Jan 2015
Posts: 1
|
Hi,
I am using IIB 9.0.0
I have a requirement like, I will get many request via queue and create a one output file for every hour with one header followed by all the incoming messages.
I have created DFDL with header, and used Timeout Notification node which will trigger every hour and connected to 'Finish File' in FileOutput node. Here the problem is for every message i receive, one header is created.
For eg: if i am receiving 5 messages, the output file has 5 headers plus five messages.
Please help me to solve this issue or suggest any other way for my requirement. |
|
| Back to top |
|
 |
| maurito |
| Posted: Fri Aug 28, 2015 3:16 am Post subject: Re: Adding one header in file for multiple request. |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| yesh wrote: |
Hi,
I am using IIB 9.0.0
I have a requirement like, I will get many request via queue and create a one output file for every hour with one header followed by all the incoming messages.
I have created DFDL with header, and used Timeout Notification node which will trigger every hour and connected to 'Finish File' in FileOutput node. Here the problem is for every message i receive, one header is created.
For eg: if i am receiving 5 messages, the output file has 5 headers plus five messages.
Please help me to solve this issue or suggest any other way for my requirement. |
when the flow is triggered, create the header, then read all the messages from the queue, when no more messages propagate to file node. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 28, 2015 5:56 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Check if it's been an hour or not.
IF
Create a new file name.
Create the header record. Send it.
Send the current message.
ELSE
send the current message.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Aug 28, 2015 6:39 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| mqjeff wrote: |
Check if it's been an hour or not.
IF
Create a new file name.
Create the header record. Send it.
Send the current message.
ELSE
send the current message. |
WHAT ???
What is wrong with the OP approach of using a timer node to do the triggering ? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 28, 2015 6:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| maurito wrote: |
| mqjeff wrote: |
Check if it's been an hour or not.
IF
Create a new file name.
Create the header record. Send it.
Send the current message.
ELSE
send the current message. |
WHAT ???
What is wrong with the OP approach of using a timer node to do the triggering ? |
How will the OP know whether to add the header or not?
What will the OP do if the messages being read - and accumulated in the same flow - and overwhelm the EG memory?
I agree that checking the time is not strictly necessary - you can use a boolean in the Environment or global cache...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Aug 28, 2015 7:09 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| mqjeff wrote: |
| maurito wrote: |
| mqjeff wrote: |
Check if it's been an hour or not.
IF
Create a new file name.
Create the header record. Send it.
Send the current message.
ELSE
send the current message. |
WHAT ???
What is wrong with the OP approach of using a timer node to do the triggering ? |
How will the OP know whether to add the header or not?
|
have you ever heard of ESQL shared variables ?... |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 28, 2015 7:14 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| maurito wrote: |
WHAT ???
What is wrong with the OP approach of using a timer node to do the triggering ? |
The Timer node is exactly that - a timer. It's not a scheduler. Any number of actions can cause the Timer to reset.
Draining a queue with an MQGet node is the fastest and most reliable way to run an execution group out of memory. If the queue has any real depth when the flow is triggered, it's going to end badly, and in out of heap errors.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 28, 2015 7:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| maurito wrote: |
| have you ever heard of shared variables ?... |
I'm sure that's a rhetorical or sarcastic question.
It also doesn't address my questions about memory usage...
And processing one message at a time can also be scaled better - at least if the timer node is broken into a separate flow.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Aug 28, 2015 7:22 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| mqjeff wrote: |
| maurito wrote: |
| have you ever heard of shared variables ?... |
I'm sure that's a rhetorical or sarcastic question.
It also doesn't address my questions about memory usage...
And processing one message at a time can also be scaled better - at least if the timer node is broken into a separate flow. |
yes, it was a sarcastic question.
the timer can be triggered on a UDP set by the user according to their needs. I am sure you know how can that be determined |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 28, 2015 7:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| maurito wrote: |
| the timer can be triggered on a UDP set by the user according to their needs. |
Unless their need is for it to go off on a regular schedule.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Aug 28, 2015 7:25 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| Vitor wrote: |
| maurito wrote: |
WHAT ???
What is wrong with the OP approach of using a timer node to do the triggering ? |
The Timer node is exactly that - a timer. It's not a scheduler. Any number of actions can cause the Timer to reset.
Draining a queue with an MQGet node is the fastest and most reliable way to run an execution group out of memory. If the queue has any real depth when the flow is triggered, it's going to end badly, and in out of heap errors. |
There are ways of programmatically controlling how many messages are drained out of the queue if that is worrying you.
Last edited by maurito on Fri Aug 28, 2015 7:33 am; edited 1 time in total |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Aug 28, 2015 7:28 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| Vitor wrote: |
| maurito wrote: |
| the timer can be triggered on a UDP set by the user according to their needs. |
Unless their need is for it to go off on a regular schedule. |
so what is your proposal then? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 28, 2015 7:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| maurito wrote: |
| There are ways of programmatically controlling how many messages are drained out of the queue if that is worrying you. |
It doesn't worry me because I don't do that.
I'm also interested on how you would programmatically determine that's you're short on memory, or how the OP would tell whoever gave him the requirement that there are 2 hourly files because the flow ran out of memory.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 28, 2015 7:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| maurito wrote: |
| Vitor wrote: |
| maurito wrote: |
| the timer can be triggered on a UDP set by the user according to their needs. |
Unless their need is for it to go off on a regular schedule. |
so what is your proposal then? |
Any of the OS level schedulers. On Unix cron is a popular choice because it's ubiquitous, on Windows there's the Windows Scheduler and on z/OS there's CA7 (or something - I've never seen a z/OS installation that didn't have a complex batch schedule).
All of these tools will handle unexpected events, missed schedule dates and support more complex scheduling scenarios than the simple (but entirely valid) one described by the OP. An more complex but equally plausible example would be once an hour between 8am and 5pm on weekdays.
Obviously the Windows Scheduler does this as effectively and reliably as any piece of Windows software. But it's still less vulnerable to external events than the Timer node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Aug 28, 2015 8:21 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
Obviously the Windows Scheduler does this as effectively and reliably as any piece of Windows software. But it's still less vulnerable to external events than the Timer node. |
I've found issues with short repeat intervals with the Windows Scheduler.
As a resuly I tried a number of alternatives. My choice is Z-Cron. Yes the server version cost a bit but not that much really.
One of my sites uses a 5 minute repeat interval and has been running without issue for nearly 3 years.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
