| Author |
Message
|
| Bichu |
 Posted: Fri Jul 22, 2016 5:41 am Post subject: Fetch lakhs of rows from IIB |
 |
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Hi Folks,
I have a requirement of fetching a database on a daily basis, but the resultset will contain lakhs of rows. I need to get this data and should expose as a web service in iib. I thought of options including limiting the result set, or invoking a SP which limits the results. But since I need to communicate the data as a single webservice, I doubt the feasibility of the above options.
Can anyone suggest a better design for this?
Thanks in advance. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Jul 22, 2016 5:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
So you're fetching a result set from a database and sending the data to a web service endpoint?
Or you need to hold the result set in IIB and allow access to it via an IIB hosted web service?
Exactly how many rows are we talking? How many Mb of data?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Fri Jul 22, 2016 5:51 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Yes. I need to access the database from IIB and need to act as a webservice provider so that data can be fetched from IIB.
It will be around 5lakhs+ rows but not sure about the size in MB. Probably, it might be greater than the 100 MB capacity of broker. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 22, 2016 5:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| It will be around 5lakhs+ rows but not sure about the size in MB. |
For anyone else wondering:
https://en.wikipedia.org/wiki/Lakh
(To remove any doubt, I didn't happen to know but Googled it)
| Bichu wrote: |
| Probably, it might be greater than the 100 MB capacity of broker. |
Where do you get 100 Mb from? The JVM especially will typically be larger than that.
A better question is what's the client going to do when you send them all this through as a web service response? I bet they're expecting something smaller.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Fri Jul 22, 2016 6:21 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks for your replies.
But as per my understanding, MQ have the maximum size limit as 100 Mb. Hence message flows designed to be run on MQ will have also 100 MB.
I know we can increase the file node capacity, but am I right in the above point. I wonder if its not. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 22, 2016 6:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| But as per my understanding, MQ have the maximum size limit as 100 Mb. Hence message flows designed to be run on MQ will have also 100 MB. |
Where's the MQ in your solution? You've mentioned database, you've mentioned web services, you've never said anything about using MQ before now.
| Bichu wrote: |
| I know we can increase the file node capacity, but am I right in the above point. I wonder if its not. |
If you need to increase the File node limit past the 100Mb default you need to question your design. That limit is not for the file it's for the record, and if you're trying to read an entire 100+Mb file as a single record something is very odd. If you're trying to read a 100+Mb file one 2Kb record at a time you don't need to touch that limit.
For MQ 100Mb is the limit for a single message. You can use other strategies to pass more data than that through MQ, and ideally would even if you had a lot of data but less than 100Mb.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
