| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| How do/would you get a channel audit log? |
« View previous topic :: View next topic » |
| Author |
Message
|
| Jeff.VT |
 Posted: Thu Jul 12, 2018 8:48 am Post subject: How do/would you get a channel audit log? |
 |
|
Acolyte
Joined: 02 Mar 2017
Posts: 71
|
I'm not a developer, I'm in Operations / Support. My developer has been trying to code a channel exit logger similar to MA0Z (but for MQ9) for a few weeks now and the trouble they're having is making me rethink the entire idea.
They don't typically code things for IBM MQ, and I'm getting worried about how stable it would be in unforeseen circumstances in the future.
I don't know how most people use IBM MQ, but what I support is middleware. I have say 100 inside endpoints that all talk to eachother and to say 50 external endpoints. It's just a patchwork of nonsense a lot of the time. I'd guess about 2 million or so messages flow through these 40 or so queue managers set up around the world on a given day.
I might be asked detailed information about any one of them - their format, the route they took through the system, where they ended up, who sent them, how long they took to get there, if they queued at all, etc.
I also have a team of 10 people who are human and make mistakes with this rather complicated system.
So having my endpoint applications log a message just isn't good enough for me. I need a log of the messages being sent over each of my external channels (inbound and outbound).
I need to know what channel a message came in on before it hit the main message routing system. I need to know what channel a given message left out of.
And I don't see any other way to do this other than a channel exit.
* Is there something I'm missing?
* Does anybody else have this problem? How did you resolve it? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Jul 12, 2018 9:05 am Post subject: Re: How do/would you get a channel audit log? |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Jeff.VT wrote: |
I'm not a developer, I'm in Operations / Support.
I need to know what channel a message came in on before it hit the main message routing system. I need to know what channel a given message left out of. |
Why do you need to know this? Is this a request/demand of internal/external auditors? Something else?
Exits, MQ exits specifically, are complicated - an advanced task - beyond the usual MQ app developer.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 12, 2018 9:07 am Post subject: Re: How do/would you get a channel audit log? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Jeff.VT wrote: |
| * Is there something I'm missing? |
Yes - this is an insane requirement.
If it's not, ask the network people how they track the same information you're trying to for each HTTP session made over their network. If they don't laugh in your face, leverage their solution.
| Jeff.VT wrote: |
| * Does anybody else have this problem? |
No - because I laugh at people's faces when they ask me. As an MQ admin / operator / support person why do I care about the format of the messages and what would I be expected to do if one of them is "wrong"? What does the route a message took matter so long as it arrived? Why does "how long they took to get there" matter in a messaging system which is deliberately designed to be asynchronous? Likewise so what if they queued at all?
Obviously you need to be monitoring for queue buildup, especially in transmission queues, monitoring channel status (for instance going into RETRY) and local queues with no open handles on them (i.e. not being serviced). For what it's worth, your system doesn't sound as complicated as some I've seen.
| Jeff.VT wrote: |
| How did you resolve it? |
If they didn't leave when I laughed, I asked them exactly what use cases the information would be put to and why the monitoring regime (which I outline above) wouldn't meet it. No-one's yet come up with a use case that wasn't to detect application failure, to deal with a problem with the application design, or wasn't resolved by switching on statistics for certain specific worries.
You're right to be worried about exits. Because they run in line with the MCA (in the case of a channel exit), a problem with the exit will cause the MCA to crash. A more insidious problem in the scenario you describe is this exit is trying to capture a ton of data and log it somewhere. This means you've got I/O in the channel process which will have a fairly serious impact on channel performance. Woe betide you if you get a serious amount of disc latency or the disc actually fills up.
Again, think about doing this by adding an exit to the switches, routers and load balancers in the network to track HTTP traffic; think about how much mayhem would result if all of that started lagging. Also under what circumstances the data's used - there's a reason network people don't log data going over the wire but connect WireShark if asked.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 12, 2018 9:12 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I'd also worry about the security aspects. This theoretical log will contain a contact admin amount of data about your MQ topology, who's using it and what messages are being sent across it. A bad actor gets any kind of access to that and it's not going to be a hard task to reverse engineer it into a series of exploits.
Also if the channel exit has to decrypt the payload data then that's more latency in your channel process.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Thu Jul 12, 2018 9:58 am Post subject: Re: How do/would you get a channel audit log? |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
| Jeff.VT wrote: |
| I'm not a developer, I'm in Operations / Support. My developer has been trying to code a channel exit logger similar to MA0Z (but for MQ9) for a few weeks now and the trouble they're having is making me rethink the entire idea. |
Are these distributed queue managers (i.e. Unix, Linux, Windows or OS/400)? If so, why didn't you look into a free trial of MQ Auditor to see if it fits your needs? It does everything you list plus a whole lot more.
MQ Auditor is only $299 per queue manager and I bet your developers have wasted far more money in work time than that.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| tczielke |
| Posted: Thu Jul 12, 2018 12:06 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
You should review the Application Activity Trace. It can audit the data you are looking for.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jul 12, 2018 5:34 pm Post subject: Re: How do/would you get a channel audit log? |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Vitor wrote: |
| Jeff.VT wrote: |
| * Is there something I'm missing? |
Yes - this is an insane requirement.
If it's not, ask the network people how they track the same information you're trying to for each HTTP session made over their network. If they don't laugh in your face, leverage their solution.
| Jeff.VT wrote: |
| * Does anybody else have this problem? |
No - because I laugh at people's faces when they ask me. As an MQ admin / operator / support person why do I care about the format of the messages and what would I be expected to do if one of them is "wrong"? What does the route a message took matter so long as it arrived? Why does "how long they took to get there" matter in a messaging system which is deliberately designed to be asynchronous? Likewise so what if they queued at all?
Obviously you need to be monitoring for queue buildup, especially in transmission queues, monitoring channel status (for instance going into RETRY) and local queues with no open handles on them (i.e. not being serviced). For what it's worth, your system doesn't sound as complicated as some I've seen.
| Jeff.VT wrote: |
| How did you resolve it? |
If they didn't leave when I laughed, I asked them exactly what use cases the information would be put to and why the monitoring regime (which I outline above) wouldn't meet it. No-one's yet come up with a use case that wasn't to detect application failure, to deal with a problem with the application design, or wasn't resolved by switching on statistics for certain specific worries.
You're right to be worried about exits. Because they run in line with the MCA (in the case of a channel exit), a problem with the exit will cause the MCA to crash. A more insidious problem in the scenario you describe is this exit is trying to capture a ton of data and log it somewhere. This means you've got I/O in the channel process which will have a fairly serious impact on channel performance. Woe betide you if you get a serious amount of disc latency or the disc actually fills up.
Again, think about doing this by adding an exit to the switches, routers and load balancers in the network to track HTTP traffic; think about how much mayhem would result if all of that started lagging. Also under what circumstances the data's used - there's a reason network people don't log data going over the wire but connect WireShark if asked. |
Huh?
No I don't think its an insane requirement.
No I wouldn't laugh at someone asking these questions.
Its completely reasonable to have access to data proving how long the MQ piece took and what happened in that layer. What's insane is saying every app at every hop should record every detail to the same level of detail, and expect every app person to know how to decipher this data, and actually do it.
When your stuck in the middle and guilty until proven innocent, having your own data to show what's happening is not insane or laughable.
Honestly, your response sounds like someone locked in the ivory tower of architecture where everything works perfectly on the Visio and in the PowerPoint, with no idea of what its like on the front lines. That, or your shop is the only one in the world where apps accept responsibility for every hiccup.
| Quote: |
| Why does "how long they took to get there" matter in a messaging system which is deliberately designed to be asynchronous? Likewise so what if they queued at all? |
Really?!?!? C'mon. Just because you use MQ doesn't mean every expectation of a timely delivery between App A and App B goes out the window.
Every byte of every message sent 24x7x365 in every environment, along with every detail of every MQ API, does not need to be captured and stored for years for some sort of transaction tracing thru MQ to be valuable.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 13, 2018 5:48 am Post subject: Re: How do/would you get a channel audit log? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| When your stuck in the middle and guilty until proven innocent, having your own data to show what's happening is not insane or laughable. |
Which is what monitoring is for.
| PeterPotkay wrote: |
| Honestly, your response sounds like someone locked in the ivory tower of architecture where everything works perfectly on the Visio and in the PowerPoint, with no idea of what its like on the front lines. |
I'm in the middle of the front lines. The road between support and architecture is length of one serious incident.
| PeterPotkay wrote: |
| That, or your shop is the only one in the world where apps accept responsibility for every hiccup. |
There's no such place.
What I refuse to get involved with is proving that one specific message arrived at application B when sent from application A, which then leads to be proving that application B read it off, which then leads me to determining why application B didn't process it but didn't log the error, which then leads me to telling application B how to fix their error handling and application A what's wrong with their message.
This is exactly the position taken by network teams when told that web service calls are disappearing into applications, and with most transmission problems. No one seriously expects the network team to log every single packet that goes across the wire (it's insane) so it's unreasonable (and insane) to expect MQ to do it.
| PeterPotkay wrote: |
| Just because you use MQ doesn't mean every expectation of a timely delivery between App A and App B goes out the window. |
Define "timely". MQ assures delivery, it doesn't guarantee it. And I'm not going to tune MQ because application A and application B are using MQ for a sub second request/response transaction that should have been identified at design time as a candidate for a synchronous web service. But wasn't because the application is all "we use MQ because we use MQ" and then stares doe eyed at you when the business starts whining.
| PeterPotkay wrote: |
| Every byte of every message sent 24x7x365 in every environment, along with every detail of every MQ API, does not need to be captured and stored for years for some sort of transaction tracing thru MQ to be valuable. |
If you want to track transactions, track business transactions. We have a requirement to track data from one application to another and we use Business Process Monitoring to do it. Some links are HTTP, some are MQ and that much more use than a raw dump of everything going across the channels.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Jul 15, 2018 4:57 pm Post subject: Re: How do/would you get a channel audit log? |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Jeff.VT wrote: |
| I'm not a developer, I'm in Operations / Support. My developer has been trying to code a channel exit logger similar to MA0Z (but for MQ9) for a few weeks now and the trouble they're having is making me rethink the entire idea. |
SupportPac MA0Z was withdrawn by IBM a few years ago, so there is no support for it and the source code is unavailable. I regularly get private requests for the source code or support for MQ versions after 7.0, but I have to politely decline. Last time I experimented with it on later versions it seemed to work OK, but don't trust me....
_________________
Glenn |
|
| Back to top |
|
|
| hughson |
| Posted: Sun Jul 15, 2018 10:02 pm Post subject: Re: How do/would you get a channel audit log? |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| gbaddeley wrote: |
| Jeff.VT wrote: |
| I'm not a developer, I'm in Operations / Support. My developer has been trying to code a channel exit logger similar to MA0Z (but for MQ9) for a few weeks now and the trouble they're having is making me rethink the entire idea. |
SupportPac MA0Z was withdrawn by IBM a few years ago, so there is no support for it and the source code is unavailable. I regularly get private requests for the source code or support for MQ versions after 7.0, but I have to politely decline. Last time I experimented with it on later versions it seemed to work OK, but don't trust me.... |
Surprised it hadn't been githubbed like everything else!
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Jul 16, 2018 5:18 pm Post subject: Re: How do/would you get a channel audit log? |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| hughson wrote: |
| Surprised it hadn't been githubbed like everything else! |
Yeah, that would be nice. There is nothing strictly IBM proprietary in the code, although it does use various reverse-engineered fields in the TSH and other structures in the channel traffic, which are subject to undisclosed change from version to version.
_________________
Glenn |
|
| Back to top |
|
|
| Jeff.VT |
| Posted: Fri Aug 03, 2018 10:09 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017
Posts: 71
|
The questions as to why... Because I'm middleware... and because my customers are in the business of making money.
I have hundreds of endpoints around the world in dozens of queue managers that all need to talk to eachother in dozens of formats and message types.
One endpoint might need to send a Basketball message to 3rdParty:Zebra out of Spain. Who would then begin its chain around 3 different companies and governments until it finally lands at its destination.
The same endpoint, right after that, might send a Football message to 3rdParty:Elephant out of Zimbabwe. Same thing here - it passes from company to company, routing around the world until it hits its destination.
I have a system that takes the Zebra message and sends it to the remote queue for Zebra messages. And can recognize the Elephant messages and send them to Elephant queues.
But what if I accidentally route a Zebra message to Elephant? Or what if the queue manager goes down and I lose something? Or what if the message was stuck in some weird loop. Or what if the Zebra sender channel was down for 45 minutes on May 12th at 8:23 am? What if Zebra messages are only useful for 30 seconds, and without it, my client has to pay a customer $10,000?
I don't see that it's an outlandish ask to have a record of when the message left my queue manager for good.
It allows me to say, "Hey... You know you sent this message to me at 8:20:00.001. I see it along each step along the way, and it finally left all systems that I have any control over at 8:20:00.005 (e.g. It was sent across my external sender channel going into another company's queue manager). It was sent to Zebra. If you want to know what happened after that, you're going to need to call Zebra."
Otherwise it's a bug hunt with everybody pointing fingers at everybody else.
So unless you have a solution that would let me tell my customer when a message left my environment for good, PeterPotkay... Because I think if I laughed in their face, I'd get fired. |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Aug 03, 2018 10:34 am Post subject: Re: How do/would you get a channel audit log? |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
How long are you given to answer these audit-style questions?
You could potentially answer them using Linear Logging and dmpmqlog, but it isn't quick or easy.
(Messages do have to be Persistent, or else they are not written to the logs).
If you have to answer such questions quickly, then a Message Exit plus your own database would be needed.
What is going wrong with your devloper's project.. is it just that this is a bigger job than was anticipated? |
|
| Back to top |
|
|
| Jeff.VT |
| Posted: Fri Aug 03, 2018 10:46 am Post subject: Re: How do/would you get a channel audit log? |
|
|
Acolyte
Joined: 02 Mar 2017
Posts: 71
|
| mvic wrote: |
How long are you given to answer these audit-style questions?
You could potentially answer them using Linear Logging and dmpmqlog, but it isn't quick or easy.
(Messages do have to be Persistent, or else they are not written to the logs). |
We process probably 100 million messages or so per day. So linear logging might be a bit... fun. I dunno. I'm trying to be as uncomplicated as I can be. At the moment, if I accidentally don't save off the Ma0z exit logs, nothing really happens. They roll over and I'm fine. But adding a whole new maintenance procedure just seems excessive.
We currently save other logs for 45 days. And it's a client so it's usually, "Government XYZ didn't get message 8901354871724123 on 2/1/2005 and is threatening to charge me a huge fine (which I'm then going to charge you) unless I can prove to them you sent it."
And now it's time for me to play find the needle in a stack of needles. It doesn't happen often, but often enough that I'd like an auditor of some kind. Also, we use it for troubleshooting purposes - so we can see the exact message that was sent over the channel in hex. Maybe they aren't sending control characters perfectly, but it doesn't show up on the end product in a noticeable way... that kind of thing.
I dunno. If it were just me, I would probably go to linear logging, and just set something up to consume the log into a big database somewhere. But it's not just me, and I need to make sure if I get hit a bus my guys can understand what's going on too.
| mvic wrote: |
| If you have to answer such questions quickly, then a Message Exit plus your own database would be needed. |
That'd be ideal, I agree.
| mvic wrote: |
| What is going wrong with your devloper's project.. is it just that this is a bigger job than was anticipated? |
Not sure the best way to put this. My developers don't typically code for IBM MQ - it's out of their comfort zone, and they have assigned somebody to me that seems on the new side.
TBH If I'd known a channel exit could blow up my queue manager completely, I probably would never have asked at all. It's frustrating because Ma0z was perfect - it did exactly what we needed.
Last edited by Jeff.VT on Fri Aug 03, 2018 10:49 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 03, 2018 10:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Jeff.VT wrote: |
| PeterPotkay... Because I think if I laughed in their face, I'd get fired. |
In fairness, I think @PeterPotkay is more in your camp on this; I was the one who suggested laughing (and I've never been fired for it - maybe I'm just lucky).
I understand the topology you're describing but I stand by my advice; find out how the network people monitor traffic (and they must by definition be monitoring the same traffic you are) and how they're solving for this.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2, 3 Next |
Page 1 of 3 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
