| Author |
Message
|
| fjb_saper |
 Posted: Wed Dec 16, 2015 5:22 am Post subject: |
 |
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Why does IIB have to post from a queue into the cloud via HTTP???
This design makes no sense at all.
Use MQ in the cloud and directly read from the queue and post to your application. No HTTP involved. Assured delivery with persistent messages and if the application reading off the queue and posting fails, the message will await in the queue (Syncpoint). Of course the single point of failure is reading from the queue and posting...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Dec 16, 2015 5:34 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RangaKovela wrote: |
We are moving files from backend directory to IIB Directory (directory from which FileInput Node picks up the messages) using batch script hourly basis.
We have designed two flows here.
Flow1 - which picks Files using fileInput node and posts Files into MQ
Flow2. - Process MQ message received from flows and post them in Cloud base web application using HTTPS |
Your design becomes more awesome by the moment.
Why exactly are you putting files onto MQ and then posting them to the web? What value does this add?
We have implemented backout mechanism in case of any intermittent connectivity failures in Flow2 to ensure sequential processing.
| RangaKovela wrote: |
| We are planning to get disable the main queues using PCF messages from Flow2 if in case connectivity issues persists after 5 retries. An e-mail notification/text message will be sent out to Support team. Support team will have to retry Messages in backout queue followed by enabling of main queue once backend system is up and running. |
I do hope this isn't the "value" MQ is adding. This is just horrible.
| RangaKovela wrote: |
| This will ensure sequential processing. |
No it won't.
| RangaKovela wrote: |
| Any suggestions |
Redesign this. It's an accident waiting to happen on a number of fronts.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Wed Dec 16, 2015 5:35 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
We would consider using PCF messages as part of normal operations to be a design "smell".
But this system just seems to be setup to be a maintenance and operational nightmare.
Files - check.
Message/file affinity - check.
Multiple protocols (file, MQ, HTTP) - check.
Ouch. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Dec 16, 2015 4:34 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Sometimes the source data is in a file. What is the difference between the application reading the file and converting the records into MQ messages versus a WMB message flow that reads the file and converts the records into MQ messages?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Dec 16, 2015 4:35 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| joebuckeye wrote: |
We would consider using PCF messages as part of normal operations to be a design "smell".
But this system just seems to be setup to be a maintenance and operational nightmare.
Files - check.
Message/file affinity - check.
Multiple protocols (file, MQ, HTTP) - check.
Ouch. |
Windoze - check
Coding MQ administrative functions into a message flow - check
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| RangaKovela |
| Posted: Thu Dec 17, 2015 2:17 am Post subject: |
|
|
Apprentice
Joined: 10 May 2011
Posts: 38
|
| fjb_saper wrote: |
Why does IIB have to post from a queue into the cloud via HTTP???
This design makes no sense at all.
Use MQ in the cloud and directly read from the queue and post to your application. No HTTP involved. Assured delivery with persistent messages and if the application reading off the queue and posting fails, the message will await in the queue (Syncpoint). Of course the single point of failure is reading from the queue and posting...
Have fun |
Unfortunately Cloud based web application does support reading or writting data from MQ. They don't seem to have MQ adapters. Hence we have to use API calls to export data from CRM system to Cloud app or import data into CRM system from Cloud app. Cloud application supports only API based integration over HTTPS. |
|
| Back to top |
|
|
| RangaKovela |
| Posted: Thu Dec 17, 2015 2:27 am Post subject: |
|
|
Apprentice
Joined: 10 May 2011
Posts: 38
|
| PeterPotkay wrote: |
| Sometimes the source data is in a file. What is the difference between the application reading the file and converting the records into MQ messages versus a WMB message flow that reads the file and converts the records into MQ messages? |
*) Application capability to write messages into MQ over network
*) Though MQ Client is for free, some developers of application just don't like to change. I tried my best to convince them to use MQAdapters but no luck
--
*) From WMB perspective, inorder to take advantage of MQ backout mechanism built in MQ Input node though FileinputNode has a backout mechanism there is overhead of doing repeated SFTP logins on to the remote fileserver . |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Dec 17, 2015 3:07 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You have to use the right protocol for the job.
You cannot accept the limitations of an inexperienced developer otherwise all IT systems will be ruined over time.
Learning to use MQ for simple getting of messages takes about 60 minutes.
There are various samples around for different languages. Find out which one he wants to use and give him a ready to use sample.
Which language does he want (for .dot net is there a complete trigger monitor sample assembly available).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 17, 2015 5:29 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| zpat wrote: |
You have to use the right protocol for the job.
You cannot accept the limitations of an inexperienced developer otherwise all IT systems will be ruined over time.
Learning to use MQ for simple getting of messages takes about 60 minutes.
There are various samples around for different languages. Find out which one he wants to use and give him a ready to use sample.
Which language does he want (for .dot net is there a complete trigger monitor sample assembly available). |
And as the developer is working on a "Web" application, don't forget to give him a JMS sample. If the developer is using Node.js or one of it's derivates have him / her use the AMQP protocol (MQ Light) to talk to MQ V8.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 17, 2015 5:38 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RangaKovela wrote: |
| *) From WMB perspective, inorder to take advantage of MQ backout mechanism built in MQ Input node |
As has been said previously in this thread, this is a poor design and doesn't guarantee what you see to think it does.
| RangaKovela wrote: |
| FileinputNode has a backout mechanism there is overhead of doing repeated SFTP logins on to the remote fileserver |
This is the most serious problem you can think of? Seriously?
I echo the comments of my worthy associates. You seem to be tying yourself in knots trying to overcome the wrong problems, and a key element of your design seems to be "what the developer knows how to write".
This is a recipe for disaster. 6 months to a year from now, this system will be abandoned because the administrators spend so much time unraveling the back out problems and fixing the file transfers it's faster for them to do the whole thing manually.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Dec 17, 2015 6:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
MQTT can use a WebSockets interface to talk to a queue manager using the MQTT extensions, or better to MessageSight.
If your cloud based "web application" doesn't support WebSockets, then it should be beaten on until it does.
And any web developer that can't master WebSockets can't apparently use sample code, and should probably be relegated to help desk for a while until they remember how.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
