| Author |
Message
|
| ganesh |
 Posted: Sun Mar 08, 2020 8:25 pm Post subject: z-linux vm unreachable after IIB fixpack |
 |
|
Master
Joined: 18 Jul 2010
Posts: 294
|
Hi,
We have IIB installed on z-linux vm and we recently applied a fixpack to the IIB runtime (10.0.0.6 to 10.0.0.15), after installing the fix pack we have 2 z-linux vm's become unreachable on the network.
The IIB queue managers nor the nodes were down at the time but were unreachable because the VM was not reachable, has anyone else faced a similar situation?
Eventually the admins had to restart the VM(I am hoping they did that from some VM management tool) and then we were able to ssh to the vm.
linux version -- Linux version 4.4.114-94.14-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Sat Mar 21, 2020 7:27 am Post subject: Re: z-linux vm unreachable. |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| ganesh wrote: |
Hi,
We have IIB installed on z-linux vm and we recently applied a fixpack to the IIB runtime (10.0.0.6 to 10.0.0.15), after installing the fix pack we have 2 z-linux vm's become unreachable on the network. |
Please be a bit more precise.
You have multiple z/VM guest images running z/Linux, yes?
Did you apply the fixpack to IIB to ONE of the z/Linux guests? Both of the z/Linux images?
What do you mean by "unreachable?" What did you try? Telnet? TCP ping?
What precisely is unreachable? Is the entire z/VM LPAR image unreachable? Or, is the z/Linux guest unreachable? Or, is the IIB unreachable?
| ganesh wrote: |
| Eventually the admins had to restart the VM(I am hoping they did that from some VM management tool) and then we were able to ssh to the vm. |
How precisely did they restart the VM? Re-IPL the host? Re-IPL (re-boot) the guest z/Linux?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ganesh |
| Posted: Sat Mar 21, 2020 7:25 pm Post subject: Re: z-linux vm unreachable. |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
| Quote: |
You have multiple z/VM guest images running z/Linux, yes?
|
yes.
| Quote: |
Did you apply the fixpack to IIB to ONE of the z/Linux guests? Both of the z/Linux images?
|
To all the IIB nodes (4) on 4 zlinux guests.
| Quote: |
What do you mean by "unreachable?" What did you try? Telnet? TCP ping?
|
Unreachable via putty and neither the http and https ports were reachable on the network ( tried by invoking web services listening on http and https ports). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Mar 22, 2020 6:26 am Post subject: Re: z-linux vm unreachable after IIB fixpack |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| ganesh wrote: |
| [neither] The IIB queue managers nor the nodes were down at the time ... |
So, you applied maintenance to live (running) systems? “Results are unpredictable” is the expected outcome.
The recommended maintenance practice is to take the system out-of-service before maintenance is applied. Once successfully applied, the system should be tested, then restarted.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ganesh |
| Posted: Sun Mar 22, 2020 8:48 am Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
| Quote: |
[neither] The IIB queue managers nor the nodes were down at the time ...
So, you applied maintenance to live (running) systems? “Results are unpredictable” is the expected outcome.
|
The complete statement was as below.
The IIB queue managers nor the nodes were down at the time but were unreachable because the VM was not reachable.
All the steps recommended by IBM for applying fixpack were followed. The nodes were taken out of F5 and the nodes were bounced later as per recommendation of IBM.
After applying the fix pack the nodes were running as expected and we were able to test them but after an hour one node could not be reached and that's when the restart of the guest had to be done, this reoccurred on a second node the next day and the same process was followed to bring it back up. Since then we have not seen this issue. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Mar 22, 2020 10:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| ganesh wrote: |
The complete statement was as below.
The IIB queue managers nor the nodes were down at the time ... |
I'm a bit confused as to the meaning of this statement. Likely, it's a language issue. Either or. Neither nor.
Does the statement mean that IIB and the IIB queue managers and some kind of nodes were all up? Or, does this mean that some were down, while others were up.
The word 'nor' in common English usage implies that what precedes the 'nor' is tied to what follows the 'nor.' Thus, and for clarity, neither __A__ nor __B__ were down implies that A and B were both up - and not either one or the other was up.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ganesh |
| Posted: Sun Mar 22, 2020 3:50 pm Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
| Neither the queue manager nor the IIB node were down when the z-linux host was inaccessible but they did go down when the guest was restarted and i could confirm this from the IIB logs once the guest came back up. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Mar 22, 2020 4:12 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
I remember we had a problem like this a couple of years ago. No IIB upgrade was in play, but we lost ethernet connectivity to all our z/Linux virtual servers. Something to do with some system ID locking out and access to OSA cards being lost if I remember correctly. There was Hiper Socket connectivity the entire time between the virtual servers and to/from z/OS. But ethernet connectivity to the zLinux servers was N/A until they fixed it.
ganesh, does your shop have other z/Linux virtual servers? Were they ALL impacted the same way at the same time?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ganesh |
| Posted: Sun Mar 22, 2020 4:21 pm Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
Thanks for your reply Peter, there are other z -linux virtual servers and they were not impacted.
We have 4 z-linux guests for IIB in production and only 2 of them were impacted and the other have not faced the same issue so far. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Mar 22, 2020 4:49 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
Connectivity losses, HIPER-Sockets or NIC, should have been logged as errors in the guest VM.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ganesh |
| Posted: Sun Mar 22, 2020 4:57 pm Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
Thanks, we have 3 PMR's open with 1 IBM to figure out why this happened, one with SUSE LINUX team, one with z/vm and the other with IIB support team.
The issue last occurred 3 weeks ago and since then we have not had the issue.
I will update the thread if we find out the root cause. |
|
| Back to top |
|
|
| ganesh |
| Posted: Thu Apr 02, 2020 9:28 am Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
This happened again last week when i was trying to stop the IIB node, the z-linux guest crashed. We had the crash logs on the linux vm. IIB took a look at the logs and concluded that this was due to a cifs mount set up on the z-linux guest. IBM has asked us to upgrade to the latest linux kernel which has a fix for the issue.
Yet to upgrade kernel, will be updating in the next couple of weeks. |
|
| Back to top |
|
|
|
|
