| Author |
Message
|
| divyadam |
 Posted: Wed Oct 03, 2018 7:21 am Post subject: Handling corrupted System Objects |
 |
|
Novice
Joined: 03 Oct 2018
Posts: 22
|
My Qmanager went down due to network issues and corrupted "SYSTEM!PROTECTION!ERROR!QUEUE" and now my Qmanager wont come online. i have enabled trace on my installation and i got this
MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668275 26363.1 : MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668279 26363.1 : MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668295 26363.1 : MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668312 26363.1 : 0x0010: 4f4e2e45 52524f52 2e515545 55452020 |ON.ERROR.QUEUE |
11:49:07.668312 26363.1 : 0x0050: 4f544543 54494f4e 21455252 4f522151 |OTECTION!ERROR!Q|
11:49:07.668356 26363.1 : aotAddEntries: StrucId(LQH ) Objname(SYSTEM.PROTECTION.ERROR.QUEUE ) RealName(queues/SYSTEM!PROTECTION!ERROR!QUEUE) Flags(512) bReuse(FALSE)
11:49:07.668452 26363.1 : OSName(/MQHA/PL07EV/data/qmgrs/PL07EV/queues/SYSTEM!PROTECTION!ERROR!QUEUE)
11:49:07.673982 26363.1 : ErrorCode :- 40406109 Numeric Insert1 :- 00000000 (0) Numeric Insert2 :- 00000000 (0)
I am assuming that the object got corrupted and i need to delete the file under queue manager to get pass the validation and start my Qmanager and then deal with corrupted object.
need help with deleting the physical file cause when tried to remove the file this is what i get
rm -f "SYSTEM!PROTECTION!ERROR!QUEUE"
bash: !PROTECTION!ERROR!QUEUE": event not found |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Oct 03, 2018 7:27 am Post subject: Re: Handling corrupted System Objects |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| divyadam wrote: |
| I am assuming that the object got corrupted and i need to delete the file under queue manager to get pass the validation and start my Qmanager and then deal with corrupted object. |
Why are you assuming that? Have you already tried the repair option and that's failed?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| divyadam |
| Posted: Wed Oct 03, 2018 7:31 am Post subject: |
|
|
Novice
Joined: 03 Oct 2018
Posts: 22
|
Tried -f option and -c option this is what i get
strmqm -c XXXXXX
WebSphere MQ queue manager 'XXXXXX' starting.
The queue manager is associated with installation 'Installation6'.
AMQ7047: An unexpected error was encountered by a command. Reason code is 0. |
|
| Back to top |
|
|
| divyadam |
| Posted: Wed Oct 03, 2018 7:34 am Post subject: |
|
|
Novice
Joined: 03 Oct 2018
Posts: 22
|
Also couple other steps i took
- cleared semaphores and shared Memory on the Linux server
- cleared logs by cold starting the Qmanager
- rebooted the hosting server multiple times just to make sure lock files and Semaphores/shared memory is flushed.
nothing helps! |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 03, 2018 7:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| divyadam wrote: |
| AMQ7047: An unexpected error was encountered by a command. Reason code is 0. |
What's the probe id associated with this failure? Is it the same for the -f attempt?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| divyadam |
| Posted: Wed Oct 03, 2018 8:58 am Post subject: |
|
|
Novice
Joined: 03 Oct 2018
Posts: 22
|
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Wed October 03 2018 08:30:36 PDT |
| UTC Time :- 1538580636.783169 |
| UTC Time Offset :- -420 (PST) |
| Host Name :- xxx-xxxxxx-xxx-xx |
| Operating System :- Linux 2.6.32-754.3.5.el6.x86_64 |
| PIDS :- 5724H7251 |
| LVLS :- 8.0.0.10 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| O/S Registered :- 1 |
| Data Path :- /var/mqm |
| Installation Path :- /opt/apps/mqm/80010 |
| Installation Name :- Installation6 (6) |
| License Type :- Production |
| Probe Id :- AO124040 |
| Application Name :- MQM |
| Component :- aotAddEntry |
| SCCS Info :- /build/slot1/p800_P/src/lib/lqm/amqaotca.c, |
| Line Number :- 1637 |
| Build Date :- Jun 7 2018 |
| Build Level :- LAIT25591-210852 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 6800 (mqm) |
| Real UserID :- 6800 (mqm) |
| Program Name :- amqzxma0 |
| Arguments :- -m XXXXXX -c -u mqm |
| Addressing mode :- 64-bit |
| LANG :- en_US |
| Process :- 26610 |
| Process(Thread) :- 26610 |
| Thread :- 1 ECMain |
| QueueManager :- XXXXXX |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 2 |
| ConnId(2) QM :- 2 |
| ConnId(3) QM-P :- 2 |
| ConnId(4) App :- 2 |
| Last HQC :- 2.5.5-17888000 |
| Last HSHMEMB :- 2.11.11-2070272 |
| Last ObjectName :- |
| Major Errorcode :- STOP |
| Minor Errorcode :- OK |
| Probe Type :- HALT6109 |
| Probe Severity :- 1 |
| Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+ |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 03, 2018 9:51 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Open a PMR with IBM.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 03, 2018 9:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I think you need a Unix forum rather than an MQ one. That FDC is indicating a file system problem resetting the object, which I rather suspect is the same sort of problem you got trying to delete the file.
So on the plus side, it turns out you were right to try and delete the file manually.
On the down side, it sounds like your disc is having serious issues.
I'm assuming this isn't mounted disc for a multi-instance queue manager? If it is you could try tooling associated with that. If not, find a Unix forum and ask your question about deleting the file again there.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 03, 2018 10:26 am Post subject: Re: Handling corrupted System Objects |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| divyadam wrote: |
| My Qmanager went down due to network issues and corrupted |
Has this qmgr functioned correctly before?
Why do you presume that a network issue caused this? At qmgr restart, no network is required.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| divyadam |
| Posted: Wed Oct 03, 2018 8:37 pm Post subject: |
|
|
Novice
Joined: 03 Oct 2018
Posts: 22
|
This is a mounted disk with multi instance. We run 2 other Qmanagers along with the one with issues. Other 2 came up clean.
What tools I can use to fix the disk issues. I can ask to run file check to identify any disk corruption or to flush any blocks on the disk.
Let me know what else I can do to resolve the issue.
As a last resort I am planning to rebuild the qmanager with latest backup. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 03, 2018 9:06 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Does the other qmgr with issues get the same error?
What did IBM say when you opened a PMR?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| LJM |
| Posted: Thu Oct 04, 2018 4:01 am Post subject: Re: Handling corrupted System Objects |
|
|
Novice
Joined: 05 Jul 2018
Posts: 22
|
| divyadam wrote: |
rm -f "SYSTEM!PROTECTION!ERROR!QUEUE"
bash: !PROTECTION!ERROR!QUEUE": event not found |
delete using the INODE
ls -i
find . -inum 'your file inode' -exec rm -i {} \; |
|
| Back to top |
|
|
| divyadam |
| Posted: Thu Oct 04, 2018 9:13 am Post subject: |
|
|
Novice
Joined: 03 Oct 2018
Posts: 22
|
mqm@xxx-xxxxxxx-xxx-xx$ find . -inum 149
./SYSTEM!PROTECTION!ERROR!QUEUE
mqm@xxx-xxxxxxx-xxx-xx$ rm -i ./SYSTEM!PROTECTION!ERROR!QUEUE
bash: !PROTECTION!ERROR!QUEUE: event not found |
|
| Back to top |
|
|
|
|
