ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Handling corrupted System Objects

Post new topic  Reply to topic
 Handling corrupted System Objects « View previous topic :: View next topic » 
Author Message
divyadam
PostPosted: Wed Oct 03, 2018 7:21 am    Post subject: Handling corrupted System Objects Reply with quote

Novice

Joined: 03 Oct 2018
Posts: 22

My Qmanager went down due to network issues and corrupted "SYSTEM!PROTECTION!ERROR!QUEUE" and now my Qmanager wont come online. i have enabled trace on my installation and i got this

MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668275 26363.1 : MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668279 26363.1 : MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668295 26363.1 : MtxName: SYSTEM.PROTECTION.ERROR.QUEUE Id: 2193
11:49:07.668312 26363.1 : 0x0010: 4f4e2e45 52524f52 2e515545 55452020 |ON.ERROR.QUEUE |
11:49:07.668312 26363.1 : 0x0050: 4f544543 54494f4e 21455252 4f522151 |OTECTION!ERROR!Q|
11:49:07.668356 26363.1 : aotAddEntries: StrucId(LQH ) Objname(SYSTEM.PROTECTION.ERROR.QUEUE ) RealName(queues/SYSTEM!PROTECTION!ERROR!QUEUE) Flags(512) bReuse(FALSE)
11:49:07.668452 26363.1 : OSName(/MQHA/PL07EV/data/qmgrs/PL07EV/queues/SYSTEM!PROTECTION!ERROR!QUEUE)
11:49:07.673982 26363.1 : ErrorCode :- 40406109 Numeric Insert1 :- 00000000 (0) Numeric Insert2 :- 00000000 (0)

I am assuming that the object got corrupted and i need to delete the file under queue manager to get pass the validation and start my Qmanager and then deal with corrupted object.

need help with deleting the physical file cause when tried to remove the file this is what i get
rm -f "SYSTEM!PROTECTION!ERROR!QUEUE"
bash: !PROTECTION!ERROR!QUEUE": event not found
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 03, 2018 7:27 am    Post subject: Re: Handling corrupted System Objects Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

divyadam wrote:
I am assuming that the object got corrupted and i need to delete the file under queue manager to get pass the validation and start my Qmanager and then deal with corrupted object.


Why are you assuming that? Have you already tried the repair option and that's failed?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
divyadam
PostPosted: Wed Oct 03, 2018 7:31 am    Post subject: Reply with quote

Novice

Joined: 03 Oct 2018
Posts: 22

Tried -f option and -c option this is what i get

strmqm -c XXXXXX
WebSphere MQ queue manager 'XXXXXX' starting.
The queue manager is associated with installation 'Installation6'.
AMQ7047: An unexpected error was encountered by a command. Reason code is 0.
Back to top
View user's profile Send private message
divyadam
PostPosted: Wed Oct 03, 2018 7:34 am    Post subject: Reply with quote

Novice

Joined: 03 Oct 2018
Posts: 22

Also couple other steps i took

- cleared semaphores and shared Memory on the Linux server
- cleared logs by cold starting the Qmanager
- rebooted the hosting server multiple times just to make sure lock files and Semaphores/shared memory is flushed.

nothing helps!
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 03, 2018 7:47 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

divyadam wrote:
AMQ7047: An unexpected error was encountered by a command. Reason code is 0.


What's the probe id associated with this failure? Is it the same for the -f attempt?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
divyadam
PostPosted: Wed Oct 03, 2018 8:58 am    Post subject: Reply with quote

Novice

Joined: 03 Oct 2018
Posts: 22

+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Wed October 03 2018 08:30:36 PDT |
| UTC Time :- 1538580636.783169 |
| UTC Time Offset :- -420 (PST) |
| Host Name :- xxx-xxxxxx-xxx-xx |
| Operating System :- Linux 2.6.32-754.3.5.el6.x86_64 |
| PIDS :- 5724H7251 |
| LVLS :- 8.0.0.10 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| O/S Registered :- 1 |
| Data Path :- /var/mqm |
| Installation Path :- /opt/apps/mqm/80010 |
| Installation Name :- Installation6 (6) |
| License Type :- Production |
| Probe Id :- AO124040 |
| Application Name :- MQM |
| Component :- aotAddEntry |
| SCCS Info :- /build/slot1/p800_P/src/lib/lqm/amqaotca.c, |
| Line Number :- 1637 |
| Build Date :- Jun 7 2018 |
| Build Level :- LAIT25591-210852 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 6800 (mqm) |
| Real UserID :- 6800 (mqm) |
| Program Name :- amqzxma0 |
| Arguments :- -m XXXXXX -c -u mqm |
| Addressing mode :- 64-bit |
| LANG :- en_US |
| Process :- 26610 |
| Process(Thread) :- 26610 |
| Thread :- 1 ECMain |
| QueueManager :- XXXXXX |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 2 |
| ConnId(2) QM :- 2 |
| ConnId(3) QM-P :- 2 |
| ConnId(4) App :- 2 |
| Last HQC :- 2.5.5-17888000 |
| Last HSHMEMB :- 2.11.11-2070272 |
| Last ObjectName :- |
| Major Errorcode :- STOP |
| Minor Errorcode :- OK |
| Probe Type :- HALT6109 |
| Probe Severity :- 1 |
| Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Oct 03, 2018 9:51 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Open a PMR with IBM.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 03, 2018 9:52 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

I think you need a Unix forum rather than an MQ one. That FDC is indicating a file system problem resetting the object, which I rather suspect is the same sort of problem you got trying to delete the file.

So on the plus side, it turns out you were right to try and delete the file manually.

On the down side, it sounds like your disc is having serious issues.

I'm assuming this isn't mounted disc for a multi-instance queue manager? If it is you could try tooling associated with that. If not, find a Unix forum and ask your question about deleting the file again there.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Oct 03, 2018 10:26 am    Post subject: Re: Handling corrupted System Objects Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

divyadam wrote:
My Qmanager went down due to network issues and corrupted

Has this qmgr functioned correctly before?

Why do you presume that a network issue caused this? At qmgr restart, no network is required.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
divyadam
PostPosted: Wed Oct 03, 2018 8:37 pm    Post subject: Reply with quote

Novice

Joined: 03 Oct 2018
Posts: 22

This is a mounted disk with multi instance. We run 2 other Qmanagers along with the one with issues. Other 2 came up clean.
What tools I can use to fix the disk issues. I can ask to run file check to identify any disk corruption or to flush any blocks on the disk.
Let me know what else I can do to resolve the issue.
As a last resort I am planning to rebuild the qmanager with latest backup.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Oct 03, 2018 9:06 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Does the other qmgr with issues get the same error?

What did IBM say when you opened a PMR?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
LJM
PostPosted: Thu Oct 04, 2018 4:01 am    Post subject: Re: Handling corrupted System Objects Reply with quote

Novice

Joined: 05 Jul 2018
Posts: 22

divyadam wrote:

rm -f "SYSTEM!PROTECTION!ERROR!QUEUE"
bash: !PROTECTION!ERROR!QUEUE": event not found


delete using the INODE

ls -i
find . -inum 'your file inode' -exec rm -i {} \;
Back to top
View user's profile Send private message
divyadam
PostPosted: Thu Oct 04, 2018 9:13 am    Post subject: Reply with quote

Novice

Joined: 03 Oct 2018
Posts: 22

mqm@xxx-xxxxxxx-xxx-xx$ find . -inum 149
./SYSTEM!PROTECTION!ERROR!QUEUE
mqm@xxx-xxxxxxx-xxx-xx$ rm -i ./SYSTEM!PROTECTION!ERROR!QUEUE
bash: !PROTECTION!ERROR!QUEUE: event not found
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » Handling corrupted System Objects
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.