| Author |
Message
|
| alethea |
 Posted: Wed Oct 03, 2007 2:09 am Post subject: web services |
 |
|
Apprentice
Joined: 30 Jul 2007
Posts: 45
|
Hi,
I'm looking for some input on different ways we can automate processing for when web services is down.
Currently if web services is up, the messages processed while it is down will go to a failure queue and need to be manually handled from that point.
Here is the problem:
1. some records compared changed(in a node); web service is down and it goes to failure queue
2.Web services comes up
3.records compared and changed start again and processed.
4.First change were then moved from failure queue to input queue(that too manually), it would overlay the first change.
One of the solutions to retry messages a defined number of times after a specified time interval would help but then it wouldnt overcome overlay problem.
Thanks |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Oct 03, 2007 3:00 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Apparently you suffer from an Affinity problem.
Before processing the message you will need to check the timestamp and compare to the latest update...
If the timestamp of current message < latest update discard... 
Which means that you cannot send a delta of the object unless you do that check for all fields in the object...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| alethea |
| Posted: Wed Oct 03, 2007 3:35 am Post subject: |
|
|
Apprentice
Joined: 30 Jul 2007
Posts: 45
|
Thanks for the reply fjb_saper.
But here is how the flow works. Hope it gives the clear picture of the update process. Please ask if you need further information to help with this.
Interface will generate a SOAP message to request the existing data through a web service. If the particular data doesn’t exist , a create data will be generated and provided. When the receiving application receives the create transaction, it will create the data.
If the data already exists, the broker will dynamically identify what sub-data are candidates for being overwritten, compare the sub-data with the data values received from web service, and prepare an update message consisting of the key values and only those data fields that have actually been changed. If no data values that trigger an update have changed, an update message will not be provided to receiving application. |
|
| Back to top |
|
|
| dilse |
| Posted: Wed Oct 03, 2007 8:32 am Post subject: |
|
|
Master
Joined: 24 Jun 2004
Posts: 270
|
| Quote: |
Interface will generate a SOAP message to request the existing data through a web service. If the particular data doesn’t exist , a create data will be generated and provided. When the receiving application receives the create transaction, it will create the data.
If the data already exists, the broker will dynamically identify what sub-data are candidates for being overwritten, compare the sub-data with the data values received from web service, and prepare an update message consisting of the key values and only those data fields that have actually been changed. If no data values that trigger an update have changed, an update message will not be provided to receiving application. |
Alethea,
Description is still not clear to me. Her is my impression fron your description:
1. The source app is putting an MQ message with the key value in it.
2. You prepare the SOAP request containing the same key.
3. In Broker, you are calling the Web service with the SOAP request, which gives you either the 'create' or 'Update' result set.
4. I f you get 'Create' transaction you simply send it out to receiver with Create action, if not in Broker you compare the current value with the Web service response values and find out the difference and send it out with the Update action.
I did not understand your problem here. Try to clarify, otherwise we cannot help you. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 03, 2007 8:35 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I think the problem is that an update may come in, get errored out, go to a backout queue, and then another update comes in and succeeds. Then the update on the backout queue gets retried, and also succeeds... which then undoes the last update.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| ashoon |
| Posted: Wed Oct 03, 2007 8:45 am Post subject: maybe a queueing mechanism might work? |
|
|
Master
Joined: 26 Oct 2004
Posts: 235
|
Instead of using HTTP for the transport why not use SOAP/JMS and do a FIFO therefore not having to worry about which message was not processed when the com. link went down?!
_________________
IBM Certified - SOA Solution Designer & WebSphere Datapower SOA Appliances |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 03, 2007 8:51 am Post subject: Re: maybe a queueing mechanism might work? |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| ashoon wrote: |
| Instead of using HTTP for the transport why not use SOAP/JMS and do a FIFO therefore not having to worry about which message was not processed when the com. link went down?! |
Right, because putting a SOAP/JMS message on a backout queue changes the problem entirely, rather than putting an SOAP/HTTP message on a backout queue.
The problem is that updates that are stored for retry - regardless of what transport the updates came in on - may overwrite updates that were generated LATER, but processed EARLIER.
This is a requirements/ business logic problem and not a technology problem.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| dilse |
| Posted: Wed Oct 03, 2007 10:30 am Post subject: |
|
|
Master
Joined: 24 Jun 2004
Posts: 270
|
| Quote: |
| I think the problem is that an update may come in, get errored out, go to a backout queue, and then another update comes in and succeeds. Then the update on the backout queue gets retried, and also succeeds... which then undoes the last update. |
I think I have a solution. It may not really optimized but I think it works.
1. Create a table with some columns which mainly includes the message Id and key value.
2. When a failure occurs log the table with the key value, messageID and Have the message to go into a separate(failure) queue. Make the key value as primary key so that we will log only the first message of a specfic key value but not all the following messages.
3. Everytime you get an update, we will check to see if that key value exist in the table and if it exists, send that message directly to the failure queue.
4. Delete the log entry in the table only when the first logged message id & key value pair is successful.
This way you can maintain the order of actions in the order in which we recieved irrespective of any failure
Hope this helps.... |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 03, 2007 11:06 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
That may solve the technical problem, and may not match the business requirements.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mahek |
| Posted: Wed Oct 03, 2007 12:08 pm Post subject: |
|
|
Voyager
Joined: 10 Sep 2004
Posts: 87
|
Jeff,
I Guess we can use the try catch node before the http request node so when the webservice is down the message comes back to the catch terminal and connect the catch terminal back to the try catch node, thus the message will be resubmitted to the webservice till the we service is up and running.
Or
Use a time control node |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 03, 2007 12:26 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| mahek wrote: |
Jeff,
I Guess we can use the try catch node before the http request node so when the webservice is down the message comes back to the catch terminal and connect the catch terminal back to the try catch node, thus the message will be resubmitted to the webservice till the we service is up and running.
Or
Use a time control node |
That may solve the technical problem, and not match the business requirements.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mahek |
| Posted: Wed Oct 03, 2007 12:54 pm Post subject: |
|
|
Voyager
Joined: 10 Sep 2004
Posts: 87
|
Jeff,
Just wanted to know when message broker processes the messages does it processs multiple messages at the same time or single-thread the messages?
If we are using the try catch node
1)when the first message comes in (when the web service is down it goes to the catch terminal which is connected to the try catch node). At the same instance an update messages comes in and web services is up and running will the second message that came in as an update be processed before the first message ( the first messages is still not processed it is going back to the try catch node) |
|
| Back to top |
|
|
| ashoon |
| Posted: Wed Oct 03, 2007 12:55 pm Post subject: Re: maybe a queueing mechanism might work? |
|
|
Master
Joined: 26 Oct 2004
Posts: 235
|
the messages that can't be sent via the SOAP/HTTP connection are stored for later processing... instead of using HTTP using a queue would just buffer the messages in a FIFO (if you setup your flows that way)... and there would be no messages stored for later processing...
| jefflowrey wrote: |
| ashoon wrote: |
| Instead of using HTTP for the transport why not use SOAP/JMS and do a FIFO therefore not having to worry about which message was not processed when the com. link went down?! |
Right, because putting a SOAP/JMS message on a backout queue changes the problem entirely, rather than putting an SOAP/HTTP message on a backout queue.
The problem is that updates that are stored for retry - regardless of what transport the updates came in on - may overwrite updates that were generated LATER, but processed EARLIER.
This is a requirements/ business logic problem and not a technology problem. |
_________________
IBM Certified - SOA Solution Designer & WebSphere Datapower SOA Appliances |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 03, 2007 1:01 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Ashoon - the problem is this.
1) A message comes in. The WS is unavailable. The message gets put on the storage queue.
2) Another message comes in. The WS is avaliable. The update in the message gets applied.
3) The message on the storage queue gets reprocessed. The WS is available. The update in the OLDER message gets applied OVER the update in the NEWER message.
EDIT. Okay, I think I see what you mean.
It's not clear that the WS that does the update can be made asynchronous.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| ashoon |
| Posted: Thu Oct 04, 2007 12:35 pm Post subject: reliable transports for SOA |
|
|
Master
Joined: 26 Oct 2004
Posts: 235
|
|
| Back to top |
|
|
|
|
