| Author |
Message
|
| dwitherspoon |
 Posted: Fri Aug 13, 2004 10:17 am Post subject: Receiving file segments through MQ |
 |
|
Acolyte
Joined: 09 Dec 2003
Posts: 59
|
Hi folks,
Care to comment on a design issue? I need to build something in Java (base classes) that's going to be able to handle a stream of non-segmented messages as well as segmented messages. The segmented messages will be file chunks, and I want to process them in order only after they have all arrived. Also, I don't want the segmented message work to hold up processing of non-segmented messages. That is, if I have a 250MB file (in segments) coming through, I don't want to block all the other message processing while that's going on.
My current thinking on the flow is:Browse the current message specifying MQGMO_ALL_SEGS_AVAIL and MQGMO_LOGICAL_ORDER
If it's segmentedIf it's the first segment start a SegmentedMessageReader thread passing the groupId
Else Skip to the next message and go around again Else start a NonSegmentedMessageReader thread passing the MessageId A SegmentedReaderThread would:Issue a Get matching on groupId specifying MQGMO_LOGICAL_ORDER
Save this file part to disk
If it's the last segmentCommit messages off the queue
Reassemble the file parts into a single file
Stop Else, go around again for the next message A NonSegmentedReaderThread would:Issue a Get matching on MessageId
Process this message
Stop My strategy is to have one thread that just browses to figure out if he sees a new segmented message group, and when he does, he moves that work to another thread. That browser can then spin past message segments until he gets to the next thing. If he finds a non-segmented message, then he moves that work to a separate thread (should finish quickly). And he browses on. He can go as fast as he has worker threads to complete the tasks, and I think this lets single messages speed through the system and not be blocked by large segment blocks.
A couple of questions I have:
1. Will matching on groupId be slow? Or MessageId?
2. Is there a common pattern for doing segmented receives?
I would welcome any comments!
_________________
Good...Fast...Cheap. Choose any two. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Aug 13, 2004 11:01 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
What you are looking at are 2 types of messages on the queue.
1) regular messages that need being serviced ASAP
2) Ftp via mq segmentation.
You could choose following approach
Set priority of regular messages to 5
Set priority of ftp messages to 0
set queue message delivery sequence to priority.
Have 2 listeners each with priority selector.
Now you can have a connection/listener for each priority.
Regular messages get processed as they arrive.
ftp messages get consumed when they arrive (after the regular ones).
This set up should allow you to receive regular messages in the middle of an ftp transmition as their greater priority would push them through.
However be very careful with ftp and channel trouble. Qdepth and physical space can become very quickly harsh limiting factors.
You might as well want to look into the ftp reference messages implementation.
Enjoy |
|
| Back to top |
|
|
| dwitherspoon |
| Posted: Fri Aug 13, 2004 11:34 am Post subject: |
|
|
Acolyte
Joined: 09 Dec 2003
Posts: 59
|
I like that approach...much simpler! I'll check into that.
If I cannot count on my partner application to set message priority, I'd be curious to know if my design has any holes...
_________________
Good...Fast...Cheap. Choose any two. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Aug 13, 2004 12:07 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have your partner app use alias queues to put the messages on.
The underlying queue may be the same but the priority will be different by alias.
That should do the trick.
Enjoy |
|
| Back to top |
|
|
| dwitherspoon |
| Posted: Fri Aug 13, 2004 12:36 pm Post subject: |
|
|
Acolyte
Joined: 09 Dec 2003
Posts: 59
|
So let me get this clear. I have two alias queues, say Low and High. They both actually point to TheQueue. However, my partner apps that send regular (non-segmented) messages should send them to the High queue, while FTP messages go to the Low queue. I can set a default priority on Low and High, such that messages going through the Low alias get a priority of 5 assigned to them, while messages through the High alias get a priority of 0.
Is that the idea?
If so, very snazzy. 
_________________
Good...Fast...Cheap. Choose any two. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 13, 2004 4:56 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Quote: |
Have 2 listeners each with priority selector.
|
but
| Quote: |
I need to build something in Java (base classes)
|
You can't select by priority in Base Jave.
If the partner app can put to 2 different queues, and you have 2 different threads, why don't you just make 2 separate queues on your side as well?
The app puts big messages to the Queue1 on their side, which resolves to Queue1 on your side, where you have the appropriate thread listening. And do the same for small messages and Queue2.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| dwitherspoon |
| Posted: Mon Aug 16, 2004 9:51 am Post subject: |
|
|
Acolyte
Joined: 09 Dec 2003
Posts: 59
|
Yep...2 queues makes more sense from a couple of different angles.
_________________
Good...Fast...Cheap. Choose any two. |
|
| Back to top |
|
|
|
|
