ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » Workload balancing not working as expected

Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next
 Workload balancing not working as expected « View previous topic :: View next topic » 
Author Message
Nigelg
PostPosted: Tue Sep 28, 2004 3:18 am    Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

The final check is NOT on date/time, it is on the channel sequence number.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Sep 28, 2004 12:40 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

Well Nigelg, sequence #s are used, but in an interesting way.

QM1, QM2, QM3 and QM4 are in CLUSTER1.

QM1 and QM2 host QueueA, and QueueA is clustered on both.
QM2 and QM3 host QueueB, and QueueB is clustered on both.
QM4 hosts no queues, it is a gateway.

AppA connects to QM4, and starts putting a message every second to QueueA. As expected, the messages round robin between QM1 and QM2.

A few minutes later, (while AppA is still running) AppB connects to QM4, and starts putting 300 messages a second to QueueB. As expected, those messages equally round robin between QM2 and QM3. BUT, all of a sudden, all of AppA's messages start going to QM1 only.

It seems that QM4 is weighing in the heavy use of the channel to QM2 by AppB against AppA's round robining, presumambly by seeing that the sequence number is rising much faster on channel TO.QM2 (because of AppB) than TO.QM1.

Once AppB ends, AppA's messages go back to round robining between QM1 and QM2 evenly. But the "damage" has been done. All the while the AppB was running, AppA's messages would not go to QM2 (but I expect if they were explicitly addresses to QM2, they would).

So it looks like when the algorithim compares the sequence numbers between 2 channels, it compares not which sequence number is bigger, but which one has changed more since the last time it checked. Note in the above test that after AppB was done, the sequence number on TO.QM2 was much bigger than TO.QM1, but was no longer *changing* rapidly, so AppA got its round robining behaviour back.

Kind of lame that one heavy duty app can disrupt the round robining behaviour for every other app in a cluster.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Nigelg
PostPosted: Wed Sep 29, 2004 12:30 am    Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

Peter, I do have access to the algorithm code (as you asked earlier), and I can assure you that the sequence number check is a straight comparison, which one is higher. The msg goes on the channel with the lowest sequence number.

You can see this in a trace, where each possible choice has some details dumped:

Choice:nn ChannelName XXX ChannelStatus n ThisPriority xNNNN NetPriority n ChannelSequence nnnn

ThisPriority is a combination of the flags mentioned in your earlier post before getting to the NetPriority and ChannelSequence.

You will see that if ThisPriority and NetPriority are the same, that the channel with the lowest ChannelSequence is chosen.

I agree with you that this is a rather crude way to decide, and that an app making heavy use of queues hosted on a subset of qmgrs in the cluster would bias the behaviour of other apps.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Sep 29, 2004 7:07 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

Nigelg wrote:
Peter, I do have access to the algorithm code (as you asked earlier), and I can assure you that the sequence number check is a straight comparison, which one is higher. The msg goes on the channel with the lowest sequence number.


If AppB send 1,000,000 messages down channels TO.QM2 and TO.QM3, while AppA slowly sends 1 message a second down TO.QM1 and TO.QM2, wouldn't you agree that the sequence number for TO.QM2 is going to be much higher for a loooong time after AppB ends compared to the sequence number for TO.QM1? And if your statement is true, then messages wouldn't round robin between QM1 and QM2 for a looooong time after AppB ended? Which is not happening. As soon as AppB ends, the sequence number stops *rising* rapidly on TO.QM2, and round robining starts working normally for AppA. So I gotta think its not a straight comparison of which sequence number is higher, but rather a comparison of what sequence numbers have changed the most since the last check.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
RatherBeGolfing
PostPosted: Wed Sep 29, 2004 7:28 am    Post subject: Reply with quote

Centurion

Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA

I have finally had some time to revisit my scenario - way back on page 1 of this thread. I looked at the sequence numbers on my CLUSRCVR channels for my QM2 and QM3 managers. Sure enough, the seqno for QM2 was 68,618, the seqno for QM3 was 223. I couldn't figure out how to reset the sequence numbers (probably have to visit each QMgr who has a CLUSSDR to QM2 or QM3 and reset at that end?), so I forced 68,345 messages to flow to QM3 only, thereby raising it's seqno to 68,618.

Then, I repeated my original test, trying to round robin between a clusterQ defined to both QM2 and QM3.

Once again, all 100 messages went to QM3, even though the sequence numbers on the CLUSRCVR channels are the same!!!


Here was my original post back on page 1:
I have a cluster Queue (QueueA) that is defined on both QM2 and QM3. Bind option set to Not Fixed.

I have a COBOL program that is putting say 10 messages to QueueA. I specify MQOO-BIND-NOT-FIXED as an open option. The program issues an MQConn once, then for each message to be put it issues an MQOpen, MQPut and MQClose. Following the last message to be put, I issue an MQDisc. I would have expected 5 messages to appear on QM1's instance of QueueA and 5 to appear on QM2's instance of QueueA.
_________________
Cheers,
Larry
MQ Certifiable
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Sep 29, 2004 8:26 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

RatherBeGolfing wrote:
I looked at the sequence numbers on my CLUSRCVR channels for my QM2 and QM3 managers. Sure enough, the seqno for QM2 was 68,618, the seqno for QM3 was 223. I couldn't figure out how to reset the sequence numbers (probably have to visit each QMgr who has a CLUSSDR to QM2 or QM3 and reset at that end?), so I forced 68,345 messages to flow to QM3 only, thereby raising it's seqno to 68,618.


There is no way, no how, that an app should have to go thru something like this. And of course, when things are working right, they don't have to do this. (My point about how it can't be a straight compare of the current value of the sequence #)

Anyway, if it is still doing this, it is one of three things:
1. The app is coding the QM name in the MQOPEN call, or is asking for BIND_ON_OPEN (please post the code).
2. The queue is defined BIND_ON_OPEN (please post the queue definition).
3. You found a bug (what version of MQ and CSD are you at?).
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
RatherBeGolfing
PostPosted: Wed Sep 29, 2004 11:38 am    Post subject: Reply with quote

Centurion

Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA

I believe I'm specifying blanks for the queue manager name and also bind not fixed on the open options. Sorry for the length of the post. I thought about chopping out comments and such from the program, but I may chop out something that is causing my problem!

Code:
     IDENTIFICATION DIVISION.

       PROGRAM-ID.
           IMP00028.
CLUSTR* THIS PROGRAM IS A CLONE OF IMP00008, EXCEPT FOR THE LOGIC
CLUSTR* OF OPEN OPTIONS - LINES MARKED CLUSTR IN COLS 1-6.

       AUTHOR.
           LARRY LACHANSE

       DATE-WRITTEN.
           JULY 27, 1998.

      *REMARKS
      *****************************************************************
      * 1. THIS PROGRAM IS A COMMON "PUT" ROUTINE FOR APPLICATIONS WITH
      *    A NEED TO PLACE MESSAGES ONE AT A TIME ON AN MQ/SERIES QUEUE.
      *    THE PROGRAM HAS BEEN LINKED IN 3 "MODES" USING 3 DISTINCT
      *    MQ/SERIES STUB MODULES:
      *      LOAD MODULE IMPUTB01 USES STUB CSQBSTUB FOR MVS BATCH
      *      LOAD MODULE IMPUTC01 USES STUB CSQCSTUB FOR CICS ONLINE
      *      LOAD MODULE IMPUTI01 USES STUB CSQQSTUB FOR IMS/DC ONLINE
      *    ALL 3 LOAD MODULES HAVE THIS PROGRAM AS THEIR BASE, EACH ONE
      *    USING A UNIQUE ENTRY POINT.  LINKAGE PARAMETERS CONTROL THE
      *    MQ/SERIES CALLS.
      *
      * 2. TO RUN THE PROGRAM, YOU NEED TO SUPPLY THE FOLLOWING VALUES
      *    VIA LINKAGE:
      *    --> ENVIRONMENT                 -  1 BYTE
      *        (VALUES: C FOR CICS, I FOR IMS, B FOR BATCH)
      *    --> QUEUE MANAGER NAME          - 48 BYTES
      *    --> QUEUE NAME                  - 48 BYTES
      *    --> LENGTH OF INPUT DATA        - 9 BYTES (MUST BE NUMERIC)
      *        (THIS LENGTH BECOMES MESSAGE LENGTH - THE PROGRAM PUTS
      *         MESSAGES UP TO A MAXIMUM LENGTH OF 64000 BYTES.)
      *    --> MESSAGE PERSISTENCE         - 1 BYTE (MUST BE Y OR N)
      *        (PERSISTENT MESSAGES ARE WRITTEN TO BOTH LOGS AND DATA
      *         FILES.  IF A QUEUE MANAGER IS RESTARTED AFTER A FAILURE,
      *         HE RECOVERS THESE PERSISTENT MESSAGES AS NECESSARY FROM
      *         THE LOGGED DATA.  MESSAGES THAT ARE NOT PERSISTENT ARE
      *         DISCARDED IF A QUEUE MANAGER STOPS.)
      *    --> BATCH PROCESS CODE          - 1 BYTE
      *        (NOTE: THIS CODE ONLY APPLIES TO APPLICATIONS
      *            WHICH CALL THIS PROGRAM USING THE BATCH ENTRY POINT)
      *        (VALUES: P - WILL PROCESS PUT, BUT NOT COMMIT MESSAGE
      *                 C - WILL BYPASS PUT AND PROCESS COMMIT
      *                 B - WILL BYPASS PUT AND PROCESS BACKOUT
      *                 X - WILL PROCESS PUT AND COMMIT MESSAGE)
      *    --> MESSAGE ID                  - 24 BYTES
      *        (THIS FIELD, COMBINED WITH CORRELATION ID, ALLOW A UNIQUE
      *         48 BYTE STRING TO BE STORED IN THE MESSAGE DESCRIPTOR
      *         PORTION OF THE MESSAGE ON THE QUEUE - NOT PART OF THE
      *         PHYSICAL MESSAGE, BUT SORT OF LIKE A PREFIX.  IN CONCERT
      *         WITH THE CORRELATION ID (NEXT FIELD), IT WILL PROVIDE A
      *         UNIQUE "KEY" SO THAT PROGRAMS RETRIEVING MESSAGES FROM A
      *         COMMON QUEUE WILL KNOW WHICH ONE TO "GET").
      *    --> CORRELATION ID              - 24 BYTES
      *        (THIS FIELD, COMBINED WITH MESSAGE ID, ALLOW A UNIQUE
      *         48 BYTE STRING TO BE STORED IN THE MESSAGE DESCRIPTOR
      *         PORTION OF THE MESSAGE ON THE QUEUE - NOT PART OF THE
      *         PHYSICAL MESSAGE, BUT SORT OF LIKE A PREFIX.  IN CONCERT
      *         WITH THE MESSAGE ID (PRIOR FIELD), IT WILL PROVIDE A
      *         UNIQUE "KEY" SO THAT PROGRAMS RETRIEVING MESSAGES FROM A
      *         COMMON QUEUE WILL KNOW WHICH ONE TO "GET").
      *    --> DATA PORTION OF THE MESSAGE - 64000 BYTES
      *    --> RETURN CODE                 - 2 BYTES (NUMERIC)
      *    --> RETURN ERROR MESSAGE        - 50 BYTES
      *
      * 3. THE PROGRAM'S HIGH-LEVEL FLOW IS AS FOLLOWS:
      *    --> LINKAGE AREA IS BRIEFLY EXAMINED FOR ACCURACY
      *    --> CONNECTS TO THE QUEUE MANAGER
      *    --> OPENS THE QUEUE CONNECTION
      *    --> PUTS THE MESSAGE FOUND IN LINKAGE TO THE QUEUE
      *    --> COMMITS THE MESSAGE TO THE QUEUE, IF REQUESTED
      *    --> CLOSES THE QUEUE CONNECTION
      *    --> DISCONNECTS FROM THE QUEUE MANAGER
      *    --> RETURNS A CODE AND A MESSAGE TO THE CALLING MODULE
      *
      *****************************************************************
      *****************************************************************

       ENVIRONMENT DIVISION.
       CONFIGURATION SECTION.
       SOURCE-COMPUTER.    IBM-370.
       OBJECT-COMPUTER.    IBM-370.

       INPUT-OUTPUT SECTION.

       DATA DIVISION.
       WORKING-STORAGE SECTION.
      *
      *    WS00 - MISCELLANEOUS WORK FIELDS
      *
       01  FILLER                      PIC X(32) VALUE
           'BEGIN WS00 MISC WORK FIELDS     '.
       01  WS00-INTERFACE-KEY          PIC X(32)       VALUE SPACES.
       01  WS00-QMGR                   PIC X(48)       VALUE SPACES.
       01  WS00-DATA-FORMAT            PIC X(8)        VALUE 'MQSTR   '.
       01  WS00-MSGLENGTH              PIC S9(9) COMP  VALUE ZERO.
991001 01  WS00-MQ-REASON-SIGNED       PIC S9(4)       VALUE ZERO.
       01  WS00-TYPE-OF-ERROR-CODE     PIC 99          VALUE ZERO.
           88 WS00-NO-ERRORS-FOUND                     VALUE ZERO.
       01  WS00-FIRST-TIME-SW          PIC X           VALUE 'Y'.
           88 WS00-FIRST-TIME                          VALUE 'Y'.
       01  WS00-MSGBUFFER.
           05  WS00-MSGBUFFER-ARRAY    PIC X     OCCURS 64000 TIMES.

990120 01  WS00-RETURN-MESSAGE.
991001     05  WS00-MQ-REASON          PIC 9(4).
990120     05  FILLER                  PIC X        VALUE ','.
990120     05  WS00-RETURN-TEXT        PIC X(45).

      *
      *    WS01 - LINKAGE FIELDS
      *
       01  FILLER                      PIC X(32)    VALUE
           'BEGIN WS01 LINKAGE FIELDS       '.
       01  WS01-ENVIRONMENT            PIC X        VALUE ' '.
       01  WS01-QUEUE-MANAGER          PIC X(48)    VALUE ' '.
       01  WS01-QUEUE-NAME             PIC X(48)    VALUE ' '.
       01  WS01-MESSAGE-LENGTH         PIC S9(9)    VALUE +0 COMP.
       01  WS01-MESSAGE-PERSISTENCE    PIC X        VALUE ' '.
       01  WS01-BATCH-PROCESS-CD       PIC X        VALUE ' '.
       01  WS01-MESSAGE-ID             PIC X(24)    VALUE ' '.
       01  WS01-CORRELATION-ID         PIC X(24)    VALUE ' '.
       01  WS01-MESSAGE-DATA           PIC X(64000) VALUE ' '.


      *
      *    WS02 - ERROR MESSAGES
      *
       01  FILLER                      PIC X(32) VALUE
           'BEGIN WS02 ERROR MESSAGE TABLE  '.
       01  WS02-ERROR-MESSAGE-TABLE.
990120     05 FILLER                   PIC X(45) VALUE
01            'NON-NUMERIC VALUE PASSED IN MESSAGE LENGTH   '.
990120     05 FILLER                   PIC X(45) VALUE
02            'VALUE OTHER THAN N OR Y FOUND IN PERSISTENCE '.
990120     05 FILLER                   PIC X(45) VALUE
03            'MQSERIES QUEUE CONNECT ERROR,CHECK Q MGR NAME'.
990120     05 FILLER                   PIC X(45) VALUE
04            'MQSERIES QUEUE OPEN ERROR, CHECK QUEUE NAME  '.
990120     05 FILLER                   PIC X(45) VALUE
05            'MQSERIES QUEUE PUT ERROR                     '.
990120     05 FILLER                   PIC X(45) VALUE
06            'MQSERIES QUEUE COMMIT ERROR                  '.
990120     05 FILLER                   PIC X(45) VALUE
07            'MQSERIES QUEUE CLOSE ERROR                   '.
990120     05 FILLER                   PIC X(45) VALUE
08            'MQSERIES QUEUE DISCONNECT ERROR              '.
990120     05 FILLER                   PIC X(45) VALUE
09            'VALUE OTHER THAN P,C,B,X IN BATCH PROCESS CD '.
990120     05 FILLER                   PIC X(45) VALUE
10            'VALUE OTHER THAN B,C,I FOUND IN ENVIRON FIELD'.
990120     05 FILLER                   PIC X(45) VALUE
11            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
12            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
13            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
14            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
15            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
16            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
17            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
18            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
19            '-->NOT CURRENTLY USED<--                     '.
990120     05 FILLER                   PIC X(45) VALUE
20            '-->NOT CURRENTLY USED<--                     '.
       01  FILLER REDEFINES WS02-ERROR-MESSAGE-TABLE.
990120     05 WS02-ERROR-MESSAGE       PIC X(45)
              OCCURS 20 TIMES
              INDEXED BY WS02-ERROR-INDEX.

      *
      *    WS03 - API FIELDS - FOR MQ/SERIES CALLS
      *
       01  FILLER                      PIC X(32) VALUE
           'BEGIN WS03 API FIELDS           '.
       01  WS03-HCONN                  PIC S9(9) BINARY VALUE 0.
       01  WS03-HOBJ                   PIC S9(9) BINARY VALUE 0.
       01  WS03-OPENOPTIONS            PIC S9(9) BINARY.
       01  WS03-COMPCODE               PIC S9(9) BINARY.
       01  WS03-REASON                 PIC S9(9) BINARY.

      *
      *    MQ/SERIES - API CONTROL BLOCKS
      *
       01  FILLER                      PIC X(32) VALUE
           'BEGIN MQM-OBJECT-DESCRIPTOR     '.
       01  MQM-OBJECT-DESCRIPTOR.
           COPY CMQODV.

       01  FILLER                      PIC X(32) VALUE
           'BEGIN MQM-MESSAGE-DESCRIPTOR    '.
       01  MQM-MESSAGE-DESCRIPTOR.
           COPY CMQMDV.

       01  FILLER                      PIC X(32) VALUE
           'BEGIN MQM-PUT-MESSAGE-OPTIONS   '.
       01  MQM-PUT-MESSAGE-OPTIONS.
           COPY CMQPMOV.

       01  FILLER                      PIC X(32) VALUE
           'BEGIN MQM-CONSTANTS             '.
       01  MQM-CONSTANTS.
           COPY CMQV SUPPRESS.


       LINKAGE SECTION.

       01  LS1-LINKAGE-AREA.
           05  LS1-ENVIRONMENT         PIC X.
           05  LS1-QUEUE-MANAGER       PIC X(48).
           05  LS1-QUEUE-NAME          PIC X(48).
           05  LS1-MESSAGE-LENGTH      PIC 9(9).
           05  LS1-PERSISTENCE-IND     PIC X.
           05  LS1-BATCH-PROCESS-CD    PIC X.
           05  LS1-MESSAGE-ID          PIC X(24).
           05  LS1-CORRELATION-ID      PIC X(24).
           05  LS1-MESSAGE-DATA        PIC X(64000).
           05  LS1-RETURN-CODE         PIC 99.
           05  LS1-RETURN-MESSAGE      PIC X(50).


       PROCEDURE DIVISION.

       MODULE-OVERVIEW.
           ENTRY 'IMPUTB28' USING LS1-LINKAGE-AREA.

           PERFORM A000-INITIALIZATION
              THRU A000-INITIALIZATION-EXIT.

      *****************************************************************
      * NOTE1: IF WE'RE RUNNING IN BATCH MODE AND THE BATCH PROCESS
      *-->     CODE IS SET TO 'C', THEN THE PROGRAM PERFORMS A
      *        ROUTINE TO COMMIT THE PENDING MESSAGES TO THE QUEUE,
      *        BYPASSING THE PUT PROCESSING. IN THIS CASE, IT ASSUMES
      *        THE CALLING PROGRAM IS AT END-OF-FILE OR AT A CHECKPOINT
      *        AND DOES NOT HAVE A MESSAGE TO PUT WITH THIS CALL.
      *-->     IF THE CODE IS SET TO 'B', THE PROGRAM PERFORMS
      *        A BACKOUT CALL, AGAIN BYPASSING THE PUT LOGIC.
      *-->     IF THE CODE IS SET TO 'P', THE PROGRAM PERFORMS
      *        THE PUT CALL ONLY WITH NO COMMIT.
      *-->     IF THE CODE IS SET TO 'X', THE PROGRAM PERFORMS
      *        THE PUT CALL AND THEN COMMITS THE MESSAGE.
      * NOTE2: MQ/SERIES COMMITS ARE NOT SUPPORTED IN CICS OR IMS.
      *****************************************************************

           IF WS00-NO-ERRORS-FOUND
               IF WS01-ENVIRONMENT = 'B'
                   EVALUATE WS01-BATCH-PROCESS-CD
                     WHEN 'P'
                         PERFORM D000-PROCESS-MESSAGES
                            THRU D000-PROCESS-MESSAGES-EXIT
                     WHEN 'C'
                         PERFORM N300-QUEUE-COMMIT
                            THRU N300-QUEUE-COMMIT-EXIT
                     WHEN 'B'
                         PERFORM N350-QUEUE-BACKOUT
                            THRU N350-QUEUE-BACKOUT-EXIT
                     WHEN 'X'
                         PERFORM D000-PROCESS-MESSAGES
                            THRU D000-PROCESS-MESSAGES-EXIT
000120* IF THE PUT FAILS, WE DON'T WANT TO MASK THE MQ REASON CODE
000120* BY PERFORMING THE COMMIT CALL, SO WE BYPASS THE COMMIT.
000120                   IF WS00-NO-ERRORS-FOUND
                             PERFORM N300-QUEUE-COMMIT
                                THRU N300-QUEUE-COMMIT-EXIT
000120                   END-IF
                   END-EVALUATE
               ELSE
                   PERFORM D000-PROCESS-MESSAGES
                      THRU D000-PROCESS-MESSAGES-EXIT
               END-IF
           END-IF.

CLUSTR     IF WS00-NO-ERRORS-FOUND
CLUSTR         IF WS01-ENVIRONMENT = 'B'
CLUSTR             CONTINUE
      *****************************************************************
      * NOTE1: IF WE'RE RUNNING IN BATCH MODE WE NEVER PERFORM THE
      *        A300 TERMINATION ROUTINE TO CLOSE THE QUEUE CONNECTION
      *        AND DISCONNECT FROM THE QUEUE MANAGER.
      *        THE REASON: DURING NORMAL RECORD PROCESSING, WE DON'T
      *        WANT TO CLOSE/DISCONNECT SINCE THE EXPECTATION IS THAT
      *        THERE WILL BE MORE MESSAGES COMING.  IF THE CALLING BATCH
      *        PROGRAM SENDS A PROCESS CODE OF 'C', IT MAY BE JUST
      *        DOING A COMMIT AS PART OF CHECKPOINT PROCESSING AND,
      *        AGAIN, WE'LL HAVE MORE RECORDS TO PROCESS.  IF THERE
      *        ARE NO MORE RECORDS/MESSAGES, THEN THE PROGRAM WILL END
      *        NORMALLY AND AN IMPLICIT COMMIT/CLOSE/DISCONNECT WILL
      *        OCCUR.  IF THE CALLING BATCH PROGRAM SENDS A BATCH
      *        PROCESS CODE OF 'B' FOR ROLLBACK, IT STILL MAY CONTINUE
      *        TO SEND MESSAGES, BUT, IF NOT, THE IMPLICIT CLOSE AND
      *        DISCONNECT WILL AGAIN BE INVOKED.
      *
      *        BOTTOM LINE: WHEN A BATCH PROGAM INVOKES THIS CALLED
      *        ROUTINE:
      *          CONNECT AND OPEN CALLS ARE PERFORMED ONCE
      *          A PUT IS PERFORMED FOR EACH RECORD PASSED
      *          COMMIT IS ISSUED IF SO INSTRUCTED
      *          ROLLBACK IS ISSUED IF SO INSTRUCTED
      *          CLOSE AND DISCONNNECT ARE NEVER EXPLICITLY PERFORMED
      *****************************************************************
CLUSTR         ELSE
CLUSTR             PERFORM A300-TERMINATION
CLUSTR                THRU A300-TERMINATION-EXIT
CLUSTR         END-IF
CLUSTR     END-IF.

           MOVE WS00-TYPE-OF-ERROR-CODE TO LS1-RETURN-CODE.
           IF WS00-NO-ERRORS-FOUND
               MOVE SPACES TO LS1-RETURN-MESSAGE
           ELSE
               SET WS02-ERROR-INDEX TO WS00-TYPE-OF-ERROR-CODE
               MOVE WS02-ERROR-MESSAGE (WS02-ERROR-INDEX)
990120           TO WS00-RETURN-TEXT
991001         MOVE WS03-REASON TO WS00-MQ-REASON-SIGNED
991001         MOVE WS00-MQ-REASON-SIGNED TO WS00-MQ-REASON
990120         MOVE WS00-RETURN-MESSAGE TO LS1-RETURN-MESSAGE
           END-IF.

       MODULE-OVERVIEW-EXIT.
           GOBACK.


       A000-INITIALIZATION.
      *****************************************************************
      * THIS PARAGRAPH:
      * (1) VALIDATES LINKAGE PARAMETERS.
      * (2) SETS UP MQSERIES VARIABLES.
      *****************************************************************

           MOVE ZERO TO WS00-TYPE-OF-ERROR-CODE.

           IF (LS1-ENVIRONMENT = 'B') OR
              (LS1-ENVIRONMENT = 'C') OR
              (LS1-ENVIRONMENT = 'I')
               MOVE LS1-ENVIRONMENT TO WS01-ENVIRONMENT
           ELSE
               MOVE 10 TO WS00-TYPE-OF-ERROR-CODE
               GO TO A000-INITIALIZATION-EXIT
           END-IF.

CLUSTR     MOVE SPACES TO WS01-QUEUE-MANAGER.
           MOVE LS1-QUEUE-NAME TO WS01-QUEUE-NAME.
           IF LS1-MESSAGE-LENGTH NUMERIC
               MOVE LS1-MESSAGE-LENGTH TO WS01-MESSAGE-LENGTH
           ELSE
               MOVE 01 TO WS00-TYPE-OF-ERROR-CODE
               GO TO A000-INITIALIZATION-EXIT
           END-IF.

           IF (LS1-PERSISTENCE-IND = 'N') OR
              (LS1-PERSISTENCE-IND = 'Y')
               MOVE LS1-PERSISTENCE-IND TO WS01-MESSAGE-PERSISTENCE
           ELSE
               MOVE 02 TO WS00-TYPE-OF-ERROR-CODE
               GO TO A000-INITIALIZATION-EXIT
           END-IF.

           IF WS01-ENVIRONMENT = 'B'
               IF (LS1-BATCH-PROCESS-CD = 'P') OR
                  (LS1-BATCH-PROCESS-CD = 'C') OR
                  (LS1-BATCH-PROCESS-CD = 'B') OR
                  (LS1-BATCH-PROCESS-CD = 'X')
                   MOVE LS1-BATCH-PROCESS-CD TO WS01-BATCH-PROCESS-CD
               ELSE
                   MOVE 09 TO WS00-TYPE-OF-ERROR-CODE
                   GO TO A000-INITIALIZATION-EXIT
               END-IF
           END-IF.

           MOVE LS1-MESSAGE-ID     TO WS01-MESSAGE-ID.
           MOVE LS1-CORRELATION-ID TO WS01-CORRELATION-ID.

           MOVE LS1-MESSAGE-DATA TO WS01-MESSAGE-DATA.
      *
TEMP  *    DISPLAY ' '.
TEMP  *    DISPLAY 'IN PROGRAM IMP00028, PARAGRAPH A000'.
TEMP  *    DISPLAY '--> LINKAGE FIELDS AFTER MOVES TO WS01...'.
TEMP  *    DISPLAY '  WS01-ENVIRONMENT------ ' WS01-ENVIRONMENT.
TEMP  *    DISPLAY '  WS01-QUEUE-MANAGER---- ' WS01-QUEUE-MANAGER.
TEMP  *    DISPLAY '  WS01-QUEUE-NAME------- ' WS01-QUEUE-NAME.
TEMP  *    DISPLAY '  WS01-MESSAGE-LENGTH--- ' WS01-MESSAGE-LENGTH.
TEMP  *    DISPLAY '  WS01-MESSAGE-PERSIST-- ' WS01-MESSAGE-PERSISTENCE.
TEMP  *    DISPLAY '  WS01-BATCH-PROCESS-CD- ' WS01-BATCH-PROCESS-CD.
TEMP  *    DISPLAY '  WS01-MESSAGE-ID------- ' WS01-MESSAGE-ID.
TEMP  *    DISPLAY '  WS01-CORRELATION-ID--- ' WS01-CORRELATION-ID.
TEMP  *    DISPLAY '  WS01-MESSAGE-DATA----- ' WS01-MESSAGE-DATA.
TEMP  *    DISPLAY ' '.

************************************************************************
********** SETUP MQSERIES VARIABLES AND PERFORM MQSERIES PREP
************************************************************************

           IF WS01-MESSAGE-PERSISTENCE = 'Y'
               MOVE MQPER-PERSISTENT     TO MQMD-PERSISTENCE
           ELSE
               MOVE MQPER-NOT-PERSISTENT TO MQMD-PERSISTENCE
           END-IF.

************************************************************************
********** CONNECT TO THE QUEUE MANAGER
********** NOTE1: CICS APPLICATIONS DO NOT NEED TO CONNECT TO THE QUEUE
**********        MANAGER, BECAUSE CICS ITSELF IS CONNECTED. THE CALL
**********        IS SKIPPED HERE, BECAUSE IN A CICS ENVIRONMENT
**********        THE CALL IS ALWAYS SUCCESSFUL.  THIS MEANS THAT THE
**********        RETURN CODE MAY NOT REFLECT THE TRUE STATE OF THE
**********        CONNECTION TO THE QUEUE MANAGER.
********** NOTE2: BATCH APPLICATIONS WILL MORE THAN LIKELY INVOKE THIS
**********        PROGRAM AS THEY LOOP THROUGH THEIR INPUT FILE(S).
**********        AS SUCH, I SET UP A FIRST TIME SWITCH TO AVOID
**********        REPETITIVE CONNECTION AND OPEN CALLS TO THE QUEUE
**********        MANAGER.  THE SWITCH IS RESET AFTER THE QUEUE
**********        CONNECTION IS SUCCESFULLY OPENED.
************************************************************************
           IF WS01-ENVIRONMENT = 'C'
               CONTINUE
           ELSE
               IF WS01-ENVIRONMENT = 'I'
                   PERFORM N100-CONNECT-QUEUE
                      THRU N100-CONNECT-QUEUE-EXIT
               ELSE
CLUSTR             IF (WS01-ENVIRONMENT = 'B') AND (WS00-FIRST-TIME)
                       PERFORM N100-CONNECT-QUEUE
                          THRU N100-CONNECT-QUEUE-EXIT
                   END-IF
               END-IF
           END-IF.

           IF WS00-NO-ERRORS-FOUND
               CONTINUE
           ELSE
               GO TO A000-INITIALIZATION-EXIT
           END-IF.

************************************************************************
********** OPEN THE QUEUE CONNECTION.  WHEN RUNNING IN BATCH MODE, ON
********** THE FIRST SUCCESSFUL OPEN, I TURN OFF THE FIRST TIME SWITCH.
************************************************************************
           IF (WS01-ENVIRONMENT = 'C') OR (WS01-ENVIRONMENT = 'I')
               PERFORM N200-OPEN-QUEUE
                  THRU N200-OPEN-QUEUE-EXIT
           ELSE
               IF (WS01-ENVIRONMENT = 'B') AND (WS00-FIRST-TIME)
                   MOVE 'N' TO WS00-FIRST-TIME-SW
                   PERFORM N200-OPEN-QUEUE
                      THRU N200-OPEN-QUEUE-EXIT
               END-IF
           END-IF.

           IF WS00-NO-ERRORS-FOUND
               CONTINUE
           ELSE
               GO TO A000-INITIALIZATION-EXIT
           END-IF.

      *
      *    SETUP MQSERIES VARIABLES
      *    3) SET PMO OPTIONS
      *
           COMPUTE MQPMO-OPTIONS = MQPMO-SYNCPOINT.

       A000-INITIALIZATION-EXIT.
           EXIT.



       A300-TERMINATION.
************************************************************************
********** CLOSE THE QUEUE CONNECTION
************************************************************************
           PERFORM N400-CLOSE-QUEUE
              THRU N400-CLOSE-QUEUE-EXIT.

           IF WS00-NO-ERRORS-FOUND
               CONTINUE
           ELSE
               GO TO A300-TERMINATION-EXIT
           END-IF.

************************************************************************
********** DISCONNECT FROM THE QUEUE MANAGER
********** NOTE: CICS APPLICATIONS DO NOT NEED TO DISCONNECT FROM THE
**********       QUEUE MANAGER, BECAUSE CICS ITSELF IS CONTINUALLY
**********       CONNECTED.
**********       THE CALL IS SKIPPED HERE, BECAUSE IN A CICS ENVIRONMENT
**********       THE CALL IS ALWAYS SUCCESSFUL.  THIS MEANS THAT THE
**********       RETURN CODE MAY NOT REFLECT THE TRUE STATE OF THE
**********       DIS-CONNECTION TO THE QUEUE MANAGER.
************************************************************************
           IF WS01-ENVIRONMENT = 'C'
               CONTINUE
           ELSE
               PERFORM N500-DISCONNECT-QUEUE
                  THRU N500-DISCONNECT-QUEUE-EXIT
           END-IF.

           IF WS00-NO-ERRORS-FOUND
               CONTINUE
           ELSE
               GO TO A300-TERMINATION-EXIT
           END-IF.

       A300-TERMINATION-EXIT.
           EXIT.


       D000-PROCESS-MESSAGES.
           MOVE WS01-MESSAGE-DATA TO WS00-MSGBUFFER.
           MOVE WS01-MESSAGE-LENGTH TO WS00-MSGLENGTH.
      *
      *    SETUP THE MESSAGE DESCRIPTOR (MQMD)
      *

           IF WS01-MESSAGE-ID = SPACES
               MOVE MQMI-NONE TO MQMD-MSGID
           ELSE
               MOVE WS01-MESSAGE-ID TO MQMD-MSGID
           END-IF.

           IF WS01-CORRELATION-ID = SPACES
               MOVE MQCI-NONE TO MQMD-CORRELID
           ELSE
               MOVE WS01-CORRELATION-ID TO MQMD-CORRELID
           END-IF.

           MOVE WS00-DATA-FORMAT TO MQMD-FORMAT.

           CALL 'MQPUT' USING WS03-HCONN
                              WS03-HOBJ
                              MQMD
                              MQPMO
                              WS00-MSGLENGTH
                              WS00-MSGBUFFER
                              WS03-COMPCODE
                              WS03-REASON.

           IF WS03-COMPCODE NOT = MQCC-OK
               MOVE 05 TO WS00-TYPE-OF-ERROR-CODE
               GO TO D000-PROCESS-MESSAGES-EXIT
TEMP       ELSE
TEMP           DISPLAY 'MQPUT SUCCESSFUL'
           END-IF.


       D000-PROCESS-MESSAGES-EXIT.
           EXIT.




       N100-CONNECT-QUEUE.
      *
      *    CONNECT TO THE QUEUE MANAGER
      *

           MOVE WS01-QUEUE-MANAGER TO WS00-QMGR.
CLUSTR     DISPLAY 'WS00-QMGR = ' WS00-QMGR.

           CALL 'MQCONN' USING WS00-QMGR
                               WS03-HCONN
                               WS03-COMPCODE
                               WS03-REASON.

           IF (WS03-COMPCODE NOT = MQCC-OK)
               MOVE 03 TO WS00-TYPE-OF-ERROR-CODE
TEMP           DISPLAY ' '
TEMP           DISPLAY '/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/'
TEMP           DISPLAY 'IN IMP00028 MQCONN NOT SUCCESSFUL'
TEMP           DISPLAY '--- WS03-COMPCODE = ' WS03-COMPCODE
TEMP           DISPLAY '--- WS03-REASON   = ' WS03-REASON
TEMP           DISPLAY '/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/'
TEMP           DISPLAY ' '
TEMP       ELSE
TEMP           DISPLAY 'MQCONN SUCCESSFUL'
           END-IF.

       N100-CONNECT-QUEUE-EXIT.
           EXIT.


       N200-OPEN-QUEUE.
      *
      *    OPEN THE QUEUE FOR PUT PROCESSING
      *
           COMPUTE WS03-OPENOPTIONS = MQOO-OUTPUT +
CLUSTR                                MQOO-BIND-NOT-FIXED.

CLUSTR     MOVE SPACES TO MQOD-OBJECTQMGRNAME
           MOVE WS01-QUEUE-NAME TO MQOD-OBJECTNAME.

           CALL 'MQOPEN' USING WS03-HCONN
                               MQOD
                               WS03-OPENOPTIONS
                               WS03-HOBJ
                               WS03-COMPCODE
                               WS03-REASON.

           IF (WS03-COMPCODE NOT = MQCC-OK)
               MOVE 04 TO WS00-TYPE-OF-ERROR-CODE
TEMP       ELSE
TEMP           DISPLAY 'MQOPEN SUCCESSFUL'
           END-IF.

       N200-OPEN-QUEUE-EXIT.
           EXIT.


       N300-QUEUE-COMMIT.
************************************************************************
********** ONLY PERFORM THE COMMIT CALL IF WE'RE RUNNING IN A BATCH
********** ENVIRONMENT, SINCE MQCMIT IS NOT SUPPORTED IN IMS/DC OR CICS.
********** ALSO, THE CALLING BATCH PROGRAM MUST SPECIFY A BATCH PROCESS
********** CODE OF 'C' OR 'X' IN ORDER TO COMMIT.  THE CALLING PROGRAM
********** WOULD LIKELY SEND A 'C' WHEN IT ENCOUNTERS END-OF-FILE,
********** BUT IT MAY ALSO CONTAIN CHECKPOINT LOGIC.  IT MAY CHOOSE TO
********** SEND AN 'X' TO COMMIT WITH EACH MESSAGE PUT.
************************************************************************
           CALL 'MQCMIT' USING WS03-HCONN
                               WS03-COMPCODE
                               WS03-REASON.

           IF (WS03-COMPCODE NOT = MQCC-OK)
               MOVE 06 TO WS00-TYPE-OF-ERROR-CODE
TEMP       ELSE
TEMP           DISPLAY 'MQCMIT SUCCESSFUL'
           END-IF.

       N300-QUEUE-COMMIT-EXIT.
           EXIT.


       N350-QUEUE-BACKOUT.
           CALL 'MQBACK' USING WS03-HCONN
                               WS03-COMPCODE
                               WS03-REASON.

           IF (WS03-COMPCODE NOT = MQCC-OK)
               MOVE 10 TO WS00-TYPE-OF-ERROR-CODE
TEMP       ELSE
TEMP           DISPLAY 'MQBACK SUCCESSFUL'
           END-IF.

       N350-QUEUE-BACKOUT-EXIT.
           EXIT.


       N400-CLOSE-QUEUE.
      ******************************************************************
      * CLOSE THE QUEUE
      ******************************************************************
           CALL 'MQCLOSE' USING WS03-HCONN
                                WS03-HOBJ
                                MQCO-NONE
                                WS03-COMPCODE
                                WS03-REASON.

           IF (WS03-COMPCODE NOT = MQCC-OK)
               MOVE 07 TO WS00-TYPE-OF-ERROR-CODE
TEMP       ELSE
TEMP           DISPLAY 'MQCLOSE SUCCESSFUL'
           END-IF.

       N400-CLOSE-QUEUE-EXIT.
           EXIT.


       N500-DISCONNECT-QUEUE.
      ******************************************************************
      * DISCONNECT FROM THE QUEUE MANAGER
      ******************************************************************
           CALL 'MQDISC' USING WS03-HCONN
                               WS03-COMPCODE
                               WS03-REASON.

           IF (WS03-COMPCODE NOT = MQCC-OK)
               MOVE 08 TO WS00-TYPE-OF-ERROR-CODE
TEMP       ELSE
TEMP           DISPLAY 'MQDISC SUCCESSFUL'
           END-IF.

       N500-DISCONNECT-QUEUE-EXIT.
           EXIT.


The queue is defined on both target queue managers with Bind option of not fixed:

Code:

DEFINE QLOCAL ('LARRY.TESTQ2') +
* CRDATE (2003-11-13) +
* CRTIME (15.14.56) +
* ALTDATE (2003-11-13) +
* ALTTIME (15.14.56) +
       DESCR(' ') +
       PUT(ENABLED) +
       DEFPRTY(0) +
       DEFPSIST(NO) +
       SCOPE(QMGR) +
       GET(ENABLED) +
       MAXDEPTH(5000) +
       MAXMSGL(4194304) +
       SHARE +
       DEFSOPT(SHARED) +
       MSGDLVSQ(PRIORITY) +
       HARDENBO +
       USAGE(NORMAL) +
       NOTRIGGER +
       TRIGTYPE(FIRST) +
       TRIGDPTH(1) +
       TRIGMPRI(0) +
       TRIGDATA(' ') +
       PROCESS(' ') +
       INITQ(' ') +
       RETINTVL(999999999) +
       BOTHRESH(0) +
       BOQNAME(' ') +
       QDEPTHHI(80) +
       QDEPTHLO(20) +
       QDPMAXEV(ENABLED) +
       QDPHIEV(DISABLED) +
       QDPLOEV(DISABLED) +
       QSVCINT(999999999) +
       QSVCIEV(NONE) +
       DISTL(NO) +
       NPMCLASS(NORMAL) +
       CLUSTER('DAASSTT') +
       CLUSNL(' ') +
       DEFBIND(NOTFIXED) +
       REPLACE


If I PUT inhibit this Q on my QM3, all the messages do flow to QM2, so I know MQ can make the connection there. He just always seems to favor QM3 as a destination.

Also, I'm running MQ 5.3 on all platforms, have applied CSD06 on the Windows machines (which host QM2 and QM3).
_________________
Cheers,
Larry
MQ Certifiable
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Sep 29, 2004 5:31 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

The code and queues look OK.

That fact that the messages go to QM2 when you PUT_INHIBIT the queue on QM3 eliminates the app coding QM3 in the OnjectQueueManager filed. If it was doing that, the messages would still try and go to QM3 even after you PUT_INHIBITED the queue (DLQ or S.C.T.Q. until the Q was available can't remember(to lazy/tired to test this now)).

So its either the BIND_ON_OPEN, or a custom CLuster WorkLoad Exit someone snuck in on you, or a bug.

Hate to say it, it may be time to fire up strmqtrc -t all -t detail, run your tests, and send the trace files to IBM. Since you are 5.3 CSD06, I would expect clustering to work pretty well, and not like you are describing. Unless there is a detail you are leving out, it looks like something is not working properly.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
RatherBeGolfing
PostPosted: Thu Sep 30, 2004 4:52 am    Post subject: Reply with quote

Centurion

Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA

Peter, and all others who participated in this chat - Thank you!
I'm sure that I'm using Bind NotFixed everywhere I need to and I'm also positive there are no workload exits in our environment.

The only other detail I could mention is that I have other queue managers participating in this same cluster. I chose 2 of the others, created another clustered Q on each of them and reran my PUT program from z/OS. They performed the round-robin-dance perfectly. So, perhaps it is environmentally related to the 2 QMgrs I really want to PUT to. These 2 QMgrs that don't work, share their servers with WMQI (V2.1) software - perhaps something in there or it's CSD04 updates affected MQ's behaviour??

Anyway, I'll open a PMR with IBM and post the results here.

Thanks again!
_________________
Cheers,
Larry
MQ Certifiable
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Sep 30, 2004 4:54 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

We have one of our clusters at 5.3 CSD04 on Windows running WMQI 2.1 CSD05 brokers as well. No problems.

Interested to hear the results of the PMR.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Nigelg
PostPosted: Thu Sep 30, 2004 11:58 pm    Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

I have one other thought. Is it possible that the qmgr that the msgs will not go to is suspended from the cluster? This would cause the msgs to go the other qmgr unless the dest queue is PUT inhibited.
You can check this by the mqsc command DIS CLUSQMGR(QMGR) ALL. THe SUSPEND attribute will be set to YES if it is suspended.
You can resume the qmgr in the cluster with the RESUME QMGR mqsc command.
Back to top
View user's profile Send private message
RatherBeGolfing
PostPosted: Fri Oct 01, 2004 9:10 am    Post subject: Reply with quote

Centurion

Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA

None of the QMgrs involved in this testing are suspended from the cluster. Thanks for the thought though....
_________________
Cheers,
Larry
MQ Certifiable
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri Oct 01, 2004 10:26 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

The destination QM may not say SUSPENDED, but the sending QM may incorrectly think it is in its repository.

You will have to use the amqrfdm utility to parse the data in the sending QM's repository queue. It is possible if the destination QM at 1 time was suspended, that fact may be incorrectly left over in places.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
RatherBeGolfing
PostPosted: Fri Oct 01, 2004 11:14 am    Post subject: Reply with quote

Centurion

Joined: 12 Nov 2002
Posts: 118
Location: Syracuse, NY, USA

Is there a z/OS equivalent command for the amqrfdm utility? My sending QMGR is MQ V5.3.1 on z/OS.
_________________
Cheers,
Larry
MQ Certifiable
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Sat Oct 02, 2004 12:10 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

Larry, I don't know. I have to assume IBM does have something to look at the cluster info on a z/OS QM. Whether its that program or something else.....

If you opened a PMR, I'm sure you'll learn how (let us know!).
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum Index » Clustering » Workload balancing not working as expected
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.