| Author |
Message
|
| paustin_ours |
 Posted: Tue Aug 23, 2016 4:15 am Post subject: |
 |
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
I am talking about folks in hursley L3 who have looked at our code multiple times in its entirety and have never once said anything.
we have file reads and file writes which take a long time but when we do the same via perl, it flies and IBM never said don't do that. Our scenario is more like a file router where we need to send parts of files to different end points.
again we used to do this via file input, file read, file out put nodes but they were slow. Our files are in the 10+GB minimum
we then make java system call from esql that calls perl and it flies.
it has been working thus far and way faster than inbuilt nodes.
Like i said, we have never once had IBM raise any concern.
there are other things like long running flows which folks like yourself in the forum would probably scream but again IBM hasn't said anything at all. There are flows where we don't commit for hours, we sleep and loop back.
To be honest, sometimes i get frustrated that IBM agrees too much to what we are doing and doesn't necessarily slap our wrists when they should. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Aug 23, 2016 4:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
Which "IBM"?
Broker L2 support? The PMR process will only complain about your code if it is shown to be the problem behind the PMR.
A team of GBS consultants? They're not paid to approve your code, they're paid to do whatever their contract says.
A Software services consultant? They will tell you that this is not a recommended approach for dealing with files, but if you complain that "we do it all the time and it works fine", then almost certainly they won't push the issue. |
Name names, or at least the area within IBM; or outside IBM if it's that kind of GBS engagement. Also post the developerWorks link or other documentation they used to support this position. How many times is "many"?
| mqjeff wrote: |
| Calling Perl and C and etc. from ESQL is fine and dandy. Unless it's doing things that are already handled directly by a Broker built-in node. Then it's a waste of time and coding. So, not a recommended practice... (at least by me) |
This kind of renaming is a great way of handling this problem. So is staging the file in another directory. IIB choses to do it that way and does it that way out of the box, a box which I might add you're paying a very great amount of money for. So even if they've not "raised any concerns", you're wasting that feature of the FileOutput node and have created 3 classes of components (Java, Perl & C) that you need to maintain, along with the interactions in memory between them and the ESQL.
| mqjeff wrote: |
| Again, when it comes to exchanging data using files, the FileOutput node will handle everything you need to protect the file from being processed before it's complete. So writing your own code to do it is, as I said, a waste of time and code. |
Or in summary -
| mqjeff wrote: |
| And in general, exchanging data using files is a practice that has not been a good idea since anything else was available - roughly around the time of the invention and availability of HTTP servers. Which was, what? 1992? |
Down boy. Breathe. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 23, 2016 4:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Like I said, L3 support in Hursley will not comment on your code unless it's a cause of the issue in the PMR you've raised.
Again, transferring data using files is a bad idea.
If your flow was slow breaking up 10GB files, perhaps it's because you were transferring 10GB files. There are several techniques in Broker to improve the performance of handling large files.
For example, using the features of the FileInput node that only read parts of the file at a time. Or using the features that control when and how sections of the file are actually parsed.
And keeping up with FixPacks.
Given the speed improvements you have seen using Perl, it's not clear why you would use IIB in the first place.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Tue Aug 23, 2016 4:43 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| Quote: |
| Given the speed improvements you have seen using Perl, it's not clear why you would use IIB in the first place. |
It won't be the first time a product is used for purposes it is not meant for but i wish IBM would tell us that. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 23, 2016 4:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| paustin_ours wrote: |
| I am talking about folks in hursley L3 who have looked at our code multiple times in its entirety and have never once said anything. |
They won't. L3 are focused entirely on product support. If you were to raise a PMR along the lines of "what do you think about this code", the PMR would be referred to GBS and closed.
| paustin_ours wrote: |
| To be honest, sometimes i get frustrated that IBM agrees too much to what we are doing and doesn't necessarily slap our wrists when they should. |
And that's why we asked what you meant by "IBM". It's not a singular entity. L3 (or any part of the PMR chain) will not comment on code unless the code is directly responsible for the reported problem.
Has "IBM" ever done a Health Check on your system? Which isn't done via a PMR?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Sun Aug 28, 2016 1:03 pm Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| I am not talking about a support call or a PMR here. I am talking about a recurring call we have with folks from Hursley where we discuss our pain points in general. Again IBM [folks from Hursley who we talk] have looked at our code in its entirety multiple times and never have raised any of the concerns that you guys on the forum would identify or suggest in a heartbeat. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Aug 28, 2016 10:49 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| paustin_ours wrote: |
It won't be the first time a product is used for purposes it is not meant for but i wish IBM would tell us that. |
It is not IBM's place to do that. Your Architect should do that.
I have one system that I support that because of the tightwad management some essential bits of H/W and S/W were not bought. One of these was a proper SNMP Management system.
We ended up using IIB to receive SNMP Traps and Polls and do all the processing of them and if the rules were violated we send an SMS Message to the support team.
Some other bits of kit on the site have such poor SNMP Implementations that the ONLY time they send a trap is when the CPU loading hits 100% for more than a second. In the end we had to use MQ (triggering) to fire up a C++ application to send a PING to these devices (As a keep alive) and process the responses before sending them to the SNMP Flow in IIB
IIB was not designed for this sort of thing. Yes it can do it.
It is not pretty but it does work.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Mon Aug 29, 2016 3:25 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| Quote: |
| It is not IBM's place to do that. Your Architect should do that. |
More often than not, Architects are not experts in ESB implementation/Design or many times might be completely new to it. what is the point of having these calls with Hursley if IBM doesn't stop us from doing something that is not right or at least raise a concern.
if the implementation is not up to the mark, it is the product that gets blamed and I will leave it up to you to guess who is it at a loss at that point. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 29, 2016 4:23 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Again.
L3 support for IIB will not do any review of your code unless it is a direct cause of the issue you are working.
If the calls you are talking about are with some other part of IBM, then they may or may not be able to provide you with a proper review of your code or your implementation, because it's outside the scope of your job.
Would you perform a review of the financial processes at your company? Is that part of your job?
If you want IBM to do a review of your implementation and discuss how it can be improved, then you need to work with the right people at IBM - this is usually a Software Services consultant or a GBS consultant. To find out how and who to talk to, discuss this with your IBM Sales representative.
If you discuss this with your architect and first/second line management, and make ti sound like something other than "our architect is an idiot and we need to get data to prove this", then you may be able to get a consultant to come in.
But if you're happy with the way your implementation works, and it has not caused you to spend many many hours working on support calls, then you don't need or probably want, to have this discussion.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Mon Aug 29, 2016 5:11 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
|
| Back to top |
|
|
|
|
