| Author |
Message
|
| ibmpro |
 Posted: Thu Jun 23, 2016 2:00 pm Post subject: how to split large file into smaller files esql command |
 |
|
Novice
Joined: 20 Jun 2016
Posts: 11
Location: USA
|
any one can help .
how to split large file into smaller files esql command .
we are fetching big volume of data records from DB2 .. is there any mechanism to have these files fetched splited not in big volume .. and then sent to output file node to be saved at external folder and be saved in one batch file back .
Thank you
Last edited by ibmpro on Fri Jun 24, 2016 5:21 am; edited 1 time in total |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri Jun 24, 2016 4:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
ESQL doesn't know anything about files.
You can use it to send smaller portions of the logical message tree to the next parts of the message tree.
There are file handling samples that come with the Toolkit.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ibmpro |
| Posted: Fri Jun 24, 2016 5:19 am Post subject: |
|
|
Novice
Joined: 20 Jun 2016
Posts: 11
Location: USA
|
Thank you for your reply
bellow is the esql command am using , the complain is it consume the resources ,
CREATE COMPUTE MODULE MF_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
CALL CopyMessageHeaders();
SET OutputRoot.Properties.MessageFormat='DFDL';
SET OutputRoot.Properties.MessageType='SCHMSG';
SET OutputRoot.DFDL.SCHMSG.record[] = SELECT * FROM Database.VREQUEST AS A;
So is there any way to enhance the above statement to manage the file transfers without affecting the flow performance ? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 24, 2016 5:23 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Review the documentation on the answer to your question here
Or use a different node to start your flow - something that knows how to read databases, perhaps.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ibmpro |
| Posted: Fri Jun 24, 2016 5:30 am Post subject: |
|
|
Novice
Joined: 20 Jun 2016
Posts: 11
Location: USA
|
Thank you ,
this was my post .. that to use stored procedure .. unfortunately i didn't get buy-in from technical team to use stored-procedure for this purpose .
Regards |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 24, 2016 5:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Then ask them to clarify the requirements for the processing time of the flow.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ibmpro |
| Posted: Fri Jun 24, 2016 6:03 am Post subject: |
|
|
Novice
Joined: 20 Jun 2016
Posts: 11
Location: USA
|
it is ones Daily , almost 50000 records at ones saved in db2 table ,
is there a way to divide these to 4 or 5 separated files on ESQL level ? and then use a loop commands ?
Thanks |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 24, 2016 6:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What are you using to start the flow?
The ESQL select statement can't do paging.
The ESQL Passthru statement can do any function that the database native select allows.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ibmpro |
| Posted: Fri Jun 24, 2016 6:24 am Post subject: |
|
|
Novice
Joined: 20 Jun 2016
Posts: 11
Location: USA
|
| am using time out notification node , which set to run at 12:00AM Midnight .. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 24, 2016 6:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You could use database input node.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 24, 2016 6:40 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jun 24, 2016 7:49 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ibmpro wrote: |
| am using time out notification node , which set to run at 12:00AM Midnight .. |
No it isn't. It's set to trigger after a given period of time has elapsed. Any number of activities (broker restart, eg reload, etc) will restart that timer. So the fact that now it goes off every 24 hours at midnight doesn't mean it will always do that; if (for example) someone applies maintenance to the OS and restarts the server at 3am, your flow will start going off every day at 3am.
Also (as I pointed out in your previous thread) the problem is the SELECT *. Even if you split this up into 4 or 5 "files", your database people will still complain about the actual result set sucking all the resources.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 24, 2016 7:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
And, as has been pointed out, the ESQL select can't do what you need.
You need to use a different means of getting records from the database. Including the other ESQL database statement.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Jun 25, 2016 9:16 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
| ibmpro wrote: |
| am using time out notification node , which set to run at 12:00AM Midnight .. |
No it isn't. It's set to trigger after a given period of time has elapsed. Any number of activities (broker restart, eg reload, etc) will restart that timer. So the fact that now it goes off every 24 hours at midnight doesn't mean it will always do that; if (for example) someone applies maintenance to the OS and restarts the server at 3am, your flow will start going off every day at 3am.
|
This is the default behaviour BUT is can be overridden to make a time fire at the same time every minute/hour/day/etc
All it takes is a little bit of ESQL.
Our systems have a master timer flow.
When this starts it works out where in the minute it is and sets the next firing to happen at 00 seconds and repeats every minute forever.
This flow then fires off timed event messages to other flows according to the schedule DB.
Yes we could use Cron and I would if it was a Unix or Linux system but it is Windows so we made sure that all the times events were totally under our control.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ibmpro |
| Posted: Sun Jun 26, 2016 7:55 am Post subject: |
|
|
Novice
Joined: 20 Jun 2016
Posts: 11
Location: USA
|
I like the idea of a master timer flow. .. can you explain this thanks
where we can set them .. and how.
appreciate |
|
| Back to top |
|
|
|
|
