| Author |
Message
|
| sguruprasanna |
 Posted: Tue May 10, 2005 11:50 am Post subject: design question |
 |
|
Apprentice
Joined: 29 Jul 2002
Posts: 37
|
Hi,
At input, I receive multiple messages containing comma seperated data.
I need to convert them into XML message. Each input message has multiple lines (delimited by CR,LF) that maps to an XML segments. For instance, input message will have 100 lines data corresponding to 100 purchase orders and output XML message will be like:
| Code: |
<ROOT>
<PurchaseOrder>
<count>1</count>
...
...
<PurchaseOrder>
<PurchaseOrder>
<count>2</count>
...
...
<PurchaseOrder>
and so on till
<PurchaseOrder>
<count>100</count>
...
...
<PurchaseOrder>
</ROOT>
|
The issue is, irrespective of the number of input files, at the output, I need to create XML message each containing not more than 3000 xml segments (in the example above it was 100). ie even if I receive 10000 input msgs, i need to create only as many XML messages each with ~3000 xml segments... Can this be done?
How do I accumulate the converted xml data and create output msgs containing exactly 3000 segments? Please suggest me a way to acheive this...
regards
Guru |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue May 10, 2005 11:59 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Have you looked into XML and the DOM model ? |
|
| Back to top |
|
|
| sguruprasanna |
| Posted: Tue May 10, 2005 12:21 pm Post subject: |
|
|
Apprentice
Joined: 29 Jul 2002
Posts: 37
|
I'm sorry, I havent looked into them..
I dont understand what you mean by "DOM model" though I have used DOM parser to parse xml using java...
Can you explain me a bit more pls? Any pointers is also appreciated...
Also, aggregation node will be of any help for the above problem? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue May 10, 2005 12:44 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| sguruprasanna wrote: |
| Also, aggregation node will be of any help for the above problem? |
Some. But it will take some doing. Aggregation nodes are designed more for sending out multiple requests and then building one reply.
You may be better off inserting each record into a table, and then selecting all "unpicked" rows. If the count is >=3000, then mark the first 3000 rows and propagate a message. If it's <3000, then don't mark any rows and wait for the next incoming message.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| maha20 |
| Posted: Tue May 10, 2005 4:05 pm Post subject: |
|
|
Voyager
Joined: 23 Sep 2003
Posts: 79
|
Wow! cool design Jeff ..
_________________
IBM Certified System Administrator -WMQ V5.3
SCJP 1.4 |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue May 10, 2005 4:29 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| maha20 wrote: |
| Wow! cool design Jeff .. |
Ehh.
It has some flaws.
What if the last message of the day comes in, and leaves the table with 2999 rows that haven't been picked?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sguruprasanna |
| Posted: Wed May 11, 2005 7:03 am Post subject: |
|
|
Apprentice
Joined: 29 Jul 2002
Posts: 37
|
Can I use MQGET plugin (IA09) for this scenario? I am using MQSI V2.1 with CSD 6.
With this, I can store the intermediate xml messages in a HOLD queue, and then use MQGET plugin to read from the HOLD queue and frame xml messages having ~ 3000 xml segments..
I have never used IA09 before, and will this work?
Can this plugin node read all the messages from the queue at once? I searched in this forum, and found that we need to loop this node against a queue to read all the messages. I am not understaning how to loop it. Can you help me here? |
|
| Back to top |
|
|
| maha20 |
| Posted: Wed May 11, 2005 7:53 am Post subject: |
|
|
Voyager
Joined: 23 Sep 2003
Posts: 79
|
I am not sure how MQGET node works but may be u can think of implementing Jeff's idea. The flaw in Jeff's design can be handled by marking the last message with some special MQMD property and then in the mssg flow you can check for that property value. If the value is found send the last message with 3000 segments and an additional message with unpicked values.
_________________
IBM Certified System Administrator -WMQ V5.3
SCJP 1.4 |
|
| Back to top |
|
|
| martinrydman |
| Posted: Wed May 11, 2005 8:13 am Post subject: |
|
|
Centurion
Joined: 30 Jan 2004
Posts: 139
Location: Gothenburg, Sweden
|
Hi,
I'm trying to understand the details of your requirements. Must you always produce *exactly* 3000 Purchase Orders / message? If not, the easiest way is to split incoming messages if they contain > 3000 records (using PROPAGATE).
If it *is* a requirement (and it seems mighty strange  ), what would you do in Jeffs scenario?
If you have 3 messages containing 2000, 800 and 300 recs, must you concatenate 2000+800+200 (=3000) and then leave the remaining 100 for the next batch? Must these 100 recs wait until another 2900 recs have arrived?
Normally, there is some property of the messages making up a batch of inbound messages that you can query in order to know when to produce the outbound batch.
Either way, the way to aggregate several incoming messages to one outgoing message (barring request/reply) is to use a database table as a holding area. But there must be some specific criteria to tell you when to empty the table and produce the outgoing message.
The MQGET way seems a bit strange too. As you probably know, an instance of a flow is triggered as soon as a message arrives on its in-queue. As far as I understand, you would use the same queue to read messages with MQGET as you would to trigger the flow, and I can't see how that could possibly work.
/Martin |
|
| Back to top |
|
|
| sguruprasanna |
| Posted: Wed May 11, 2005 9:53 am Post subject: |
|
|
Apprentice
Joined: 29 Jul 2002
Posts: 37
|
| Quote: |
As far as I understand, you would use the same queue to read messages with MQGET as you would to trigger the flow |
I thought of doing something like:
MQInput (From INQ) -> MQGet (reads from HOLDQ) -> Compute ->
Filter (check for a flag) ->(true) ->MQOutput (OUTQ)
->(false) -> MQOutput(HOLDQ)
Read a comma seperated message from INQ, also read whatever is there in HOLDQ (initially empty) using MQGET, convert commaseperated msg into xml, append it to whatever was read from HOLDQ, check for the size, if it is ~ 3000, set a flag=true else to false. This will tell the filter node to pass the output message to either OUTQ or HOLDQ.
I havent used MQGET even once, can you advise if the plugin will work the way mentioned above? Using database is anyway the only option, if nothing else work. I wanted to avoid using database because of the performance and maintanence issues it gets in. |
|
| Back to top |
|
|
| martinrydman |
| Posted: Wed May 11, 2005 10:18 am Post subject: |
|
|
Centurion
Joined: 30 Jan 2004
Posts: 139
Location: Gothenburg, Sweden
|
Hi,
I haven't used it either. Maybe someone who have will chip in. Logically, you do the same as a db table (a queue being a virtual db table of sorts, anyway). I'm not sure you will gain much performance-wise, especially if you make your HOLDQ persistent. If performance is really a crucial issue, some comparsion tests between the two approaches might be interesting. As for maintanence, you already have a db at hand for the broker. Another table won't add much.
I've used the db approach without any problems, and saved myself another hassle: dealing with support-packs (always an adventure when the base product gets upgraded - more than one support-pack has been withdrawn over the years)
/martin |
|
| Back to top |
|
|
|
|
