BPI systems horror of June 6
By Chemrock
Having made a comment elsewhere that in deference to some sensitive accounts in BPI, the systems horror of June 6 need to be openly explained, it’s a matter of need to do right that I share some thoughts on what the Senate inquiry revealed. It is also to oblige a request from a TSH member I hold in high respect.
Let’s understand some terminology first :
- Banking system — this is the host system where the bank’s core accounting system is.
- ATM system — Banks operate their own proprietary ATM system. This system interfaces the banking system where customers’ accounts and info are kept.
- Transaction input — Transactions are input to the banking system via several ways — by users accessing the banking system, direct feeds from POS devises like ATM, or some straight through processing from other applications, or by importing a file of transaction data generated by other apps. Once transactions are entered, it updates the banking system.
- Realtime input — Means the accounts are updated immediately when the transaction input is complete.
- Batch input — Means the postings get done on completion of entering the entire batch of transactions. This is normally for input via importing a file, or downloading transactions from another system.
- End of day (EOD) run — This is a routine that is run to end the day’s processing. The system will carry out a series of processes in a pre-determined order. There is no user interference, it’s all systems-generated. It will process stuff like interest computation, foreign exchange revaluation, systems cleaning, reports generation, etc.
- Production environment — when the systems are running real data mode.
My comments here are based purely on what was explained in the Senate inquiry and I write from the point of view of a banking operations guy, not a systems geek. Technically it was not even a systems glitch but human error. A programmer made a backup file of ATM transaction data for the period Apr 27 to May 2 and inadvertently this file ended up as a batch input to the end of day run for June 6. The result on Jun 7 is all customers who had ATM transactions for that period saw their withdrawals and deposits duplicated. Now I am not a systems guy, and even if I were so, without the intimate knowledge of the BPI system, it is not possible to make fair comment if the process thus explained was possible or the programmer is being thrown under the bus. However, the Senate inquiry failed to pursue further logical questions that would have helped provide more clarity. Let me share these points here.
Every system lives with a zero day threat. It means users and developers of a system understand there is the possibility of vulnerabilities due to a flaw which no one has yet experienced nor detected. It’s the geeks saying they are not perfect. Hackers that discover such flaws attack the system before developers knew the existence of those vulnerabilities. These are known as zero day attacks. However, for systems that have been running for years, they are robust and remotely unlikely to kick up problems in daily routine operations. Problems are normally encountered during upgrades or installation of patches. As BPI officials explained, there was no tweaking of the systems on Jun 6, so let’s have more confidence in the machines and systems in the bank.
The programmer’s error screams two fundamental breaches of internal control in the IT Department. Firstly, no programmer should ever be allowed to interfere with the systems operator running his routine. (This was actually what happened in the 2016 elections when a Smartmatic programmer performed a simple and harmless code enhancement whilst the system was running live). Secondly, programmers are never, ever, allowed to do any systems work in a live production environment. These two restrictions are like commandments # One and # Two in any computer room. There was a serious lapse of line management oversight. It seems a similar play as in the RCBC case of money laundering re Bangladesh Central Bank, higher ups are protecting their own posteriors.
The bank has a proprietary ATM system that manages the deposits and withdrawals by customers. As explained by BPI, the ATM system and the host banking system are separate. ATM transactions are recorded and posted by batch at about 8 pm daily into the banking system when customers’ accounts are updated. That means ATM transactions do not update customer accounts in realtime. So obviously there has to be some means of shadow posting, or an application to ‘put on hold’ ATM transactions in the banking system, otherwise, how can the bank track customers’ intra day balances. Another point to note is BPI explained the ATM system is a closed system, meaning it does not interface with other systems, thus it is closed to outside forced entry. This is incorrect, because their ATM system obviously interfaces with the accounting system, and it is also connected to the interbank ATM network Bancnet, as well as international card association networks like Cirrus, Mastercard, Visa etc.
ATM transactions are posted at 8 pm. What I don’t understand about this is banking hour is up to 4.30 pm which is the end of check clearing and the time when banks determine their cash positions and do the necessary overnight borrowing to meet liquidity compliance or invest excess funds. It only makes sense if ATM transactions posted at 8 pm are considered as next day transactions. However, if it were so, then they are not included in the EOD run but next day’s run. In other words, it does not wait for the EOD run routine to process. It is entered as a batch entry and the transactions are updated immediately as Jun 7 transactions. And if this is right, then on Jun 7, after the error was discovered, they could have used the EOD June 6 backup which would have excluded the ATM batch of Apr 27 to May 2. Recovery would have been faster that way than manually reversing those duplicated entries which totaled more than 2 million transactions.
So was the programmer posting a batch of ATM transactions dated Apr 27 to May 2 on Jun 7 run time (on the evening of Jun 6)? She cannot be that dumb. Even if she were dumb, a programmer certainly cannot have user access to the banking system to enter the batch. That’s standard internal control protocol. So the real question is, if she was’nt doing a batch data entry, how did the file of 7 days ATM transactions get posted into the banking system?
Now if I was wrong and the ATM batches indeed needed the EOD run to execute the postings of all the 7 days transactions, then I’m wondering if their system has a safety net. Normally, before EOD, there ought to be a temporary back up.The reason is the EOD run is a long process and it is on auto pilot. Operators can go home or go for their breaks. If a problem occurs during EOD, the bank has the option to go back to the pre-EOD backup and do a rerun. So if they realised their problem on Jun 7 morning, they can reload Jun 6 pre-EOD backup, delete the transactions of the ATM batch for Apr 27 to May 2 then run EOD of Jun 6.
If the Apr 27 to May 2 batch of transactions was input to the banking system, then the system seriously requires some stress testing for allowing back-dated transactions to get through.
BPI explained that the Apr 27 to May 2 transactions were copied into a file because there was a request for reconciliation by someone. There was a moment of poor audio in the Senate inquiry video so I don’t know if it was a customer, network operator, or card association request. I find this strange because reconciliation is a very routine check in banking. It is easily served by reference to hard copy reports that are printed daily or monthly. Some good system may allow onscreen viewing of historical records. I have no idea why the programmer deemed it necessary to resort to copying from a live environment. Another nagging thought is reconciliation is the purview of a middle office operations control unit, not a programmer’s function, unless it relates to network issues.
Not surprisingly, social media went wild on Jun 7 with equally wild mention of friends of friends seeing million peso deposits or withdrawals in their accounts. BPI reiterated that those duplicated entries relate to ATM transactions which are low value types, thrashing those social media claims as fakes. That’s not 100% correct. Withdrawals are certainly limited value per transaction, and deposits too are unlikely to be large. Who in Philippines will stand in front of an ATM to deposit a bagful of 100,000 pesos? However, there is no value limit on fund transfers to accounts within BPI.
I’m not suggesting something nefarious going on. It’s only wondering thoughts from a curious mind. Public cynicism and speculation is to be expected as we have simply seen too much bank system and management failures in Philippines. It happened in Citibank, Banco Filipino, BDO, RCBC, BPI, now BDO yet again, and several others. The elephants in BPI are some contentious current accounts and if events are as what officials explained, there was no revisionism of accounting entries. It’s ridiculous to even think that an Ayala-owned bank will accede to a Svengali request to attempt an audacious juggling of the systems to rewrite entries in some specific accounts, putting at risk the credibility of the bank and the entire national banking industry.
Let’s cut the BPI folks some slack. They did their darn best to get the systems back running. Manually reversing more than 2 million transactions in 3 days is no mean task. Of course all fees duplicated, or interest charges as a result of error balances, would be reversed, with apologies. Customers whose checks bounced should have their dignity restored as their payees understand the situation by now. I’m just looking out for customers who try to claim for lost opportunities because they had no access to their funds at critical moments. Wonder what the law for such situation is like here.
My summation of the Senate inquiry is that the BPI official explanation seemed aboveboard, but it did not adequately cover the technical question of how the old ATM transactions got into the banking system. A description of the run sequence is all it takes. It was glossed over although I don’t think intentionally, but it left me hanging.
Let me conclude with this thought. It’s something I touched on in a book on banking operations which I’m in the process of writing. Advancement in technology is greatly felt in banking, especially in the dealing room which is always the first to embrace the latest innovation. Algorithmic trading is gaining momentum in banks and we are seeing a lot of personnel displacements. It’s not a case of robotics replacing humans, it’s simply software. And we are not talking blue collar factory workers being replaced by robots. It’s high level jobs of bank dealers and traders being displaced. The attrition rate in some banks are close to 40% because algorithmic trading is more efficient and cost less. Now it’s true that technological advancements take some jobs away, but they in turn create new jobs. Whilst dealing rooms are seeing less dealers and traders, they are being replaced by software engineers. These brilliant guys seat next to dealing staff watching the screens, trying to best understand the internal and external parameters that drive the trading game, They try to evaluate the decision making thought processes of the humans. The purpose being to dynamically tweak the parameter settings in the software for optimum performance. So there are guys fiddling in the production environment with systems that deal with $ millions per transaction, compared to BPI’s ATM transactions of $1,000 each.
Comments
67 Responses to “BPI systems horror of June 6”Trackbacks
Check out what others are saying...-
[…] via BPI systems horror of June 6 — The Society of Honor: the Philippines […]
The Society of Honor by Joe America 
Many many thank yous for this and all your well researched articles Chemrock!😊👍
Re: Jobs taken away by AI.
ATMs are the perfect example that Automation do not take away all of the jobs.
ATMs have been around since the 80s, but still there are tellers.
Right on that observation. But governments want to move away from cash to digital cashless societies. India has withdrawn I think 80% of their cash in circulation. Many countries are cutting back on the higher value printed notes.
We are not far behind. (sarcasm)
waiting for Edgar’s, Gian’s or Irineo’s inputs for the system’s side.
“Every system lives with a zero day threat. It means users and developers of a system understand there is the possibility of vulnerabilities due to a flaw which no one has yet experienced nor detected. It’s the geeks saying they are not perfect.”
I never understood why they don’t beta test this first before rolling it out and waiting for complaints, or hacks. Similar to our old adage of training to deploy vs. deploying to train. Proper Planning Prevents Piss Poor Performance. If the movie industry can do https://en.wikipedia.org/wiki/Test_screening , why not them?
Nonetheless, a great read, chemp!
Lance,
All proper software do undergo rigid assurance test sequence covering 4 basic areas — unit testing, integration testing, system testing, and acceptance testing.
Within each area there are loads of subset tests using various tools and technigues. For example, on systems testing, they probably conduct the following : —
Graphical user interface testing
Usability testing
Software performance testing
Compatibility testing
Exception handling
Load testing
Volume testing
Stress testing
Security testing
Scalability testing
Sanity testing
Smoke testing
Exploratory testing
Ad hoc testing
Regression testing
Installation testing
Maintenance testing[clarification needed]
Recovery testing and failover testing.
Accessibility
(info from Wikipedia)
Then follows in beta mode.
Thanks, chemp! I’m not really too keen on this IT stuff, most of what I know comes from Showtime’s series “Dark Net” and they did a whole episode on Zero Day Exploits industry,
as I understood it, after beta, what they can’t catch thru that process, they open up to hackers who make a living ‘penetrating’, ‘exploiting’ their wares, then companies pay said hackers, but the savvy ones skip the corporate bounty for zero days, and instead sell to those offering more money, namely gov’t entities.
I do not think this ATM stuff would happen over here. It’s so much part of the infrastructure that any issue will get resolved immediately. But we did have some sales issue in Wells Fargo awhile back, with bankers/associates/clerks opening up accounts and credit cards without customer knowledge. Turns out Wells Fargo employees were under pressure to open accounts, hence all the fraud.
I just got a notice from my bank that my account minimum just went up. I’d have to pay a fee if it’s under the minimum. I’m not really a principled guy 😉 , but that feels like a shake down to me, hence I’ll be closing my account asap and moving my money to another bank, who’ll not up their minimum willy nilly… seems bad banking practice to me.
Looking forward to more articles like this, chemp! Learning lots!
They probably did. But the setup was wrong. They could’ve pointed the programs to the wrong file.
There was no software change. It was the batch file that was picked up in error by the software. Batch files, like all computer files, go by file names. The questions to ask are: (1) why was the backed up file given the same name? (2) why does the software not check for transactions that are already in the system with the same ATM transaction#, timestamp, etc.?
(1) If these batch files run on IBM job control cards, production operators can override the logical filenames and point to physical filenames.
Hard to comment on this as the true details are missing… but a programmer is definitely not allowed to do anything in production, that is indeed a rule. About batch jobs – why was an old batch re-run? Why where there no safeguards against this like “batch numbers” with a respective status?
What were the contents of the said batch, if it was not ATM withdrawals? Why no safeguards against transactions with dates that old? These are just questions from a NON-banking IT expert.
In unmarked batches like scanned documents (this is something I have done elsewhere) there are standard operating procedures that entail manually marking the batch as DONE and filing it away to prevent accidental re-scanning. Who audits the SOPs of the bank, what audit regulations exist?
*******
1. First question: Why was an extract required from the Apr 27 – May 2 period?
1.1. If as stated, it was for reconciliation, then why could not reconciliation be carried out via an onscreen inquiry? Is it possible that BPI’s system is so backward that there is no such capability?
1.2. Assuming no such capability, should not the extract have been in (a) paper report form, (b) Excel form, or (c) sub-transaction file form rather than (d) a full transaction file form?
2. Second question: How and why was the extract file used to update account balances? Was it merged with the Jun 6 daily transaction file? Or was a special separate posting done? In either case, this would have required some extra step(s) in the daily routine of the operations side.
3. Also, in either case, and as a corollary to 1.2, this brings up a question as to the nature of the problem.
Since a transaction file form was required, is it possible that some transactions from the Apr 27 – May 2 period were not posted? This scenario changes the nature of the problem from one of reconciliation to misposting (or, rather, non-posting).
This scenario would answer the question of why an extract in transaction file format was required. It would explain the time gap between May 2 and June 6. It would also explain why the extract was merged (?) with the daily transactions of Jun 6.
The basic error then points to the fact that the extract file contained more data – in terms of format as well as content — than was necessary.
This scenario begs many questions. Such as:
2.1. Would not unposted transactions pose reconciliation problems for each day from Apr 27 – May 2?
2.2. If a programmer was required to do the extract, would not control totals be required as, say, to the number of transactions and their values?
2.3. Would not testing, at all levels, be done to verify that the extract program was in perfect working order before it was versioned into production?
2.4. Would not a visual check of the extract be mandatory to verify the accounts, the dates, and the amounts?
3. Third question: If the scenario of misposting (or non-posting) is incorrect, how could the extract file be “inadvertently” used to update account balances?
3.1. And why, as noted, doesn’t the system check for duplicate updates? It is an easy validation to check duplicates by account, amount, date and time (to the seconds at least).
4. Fourth question: Why blame the programmer? Why not the analyst? The testers? The operators? And management?
It would seem that there may have been a communication error, a problem definition error, and a solution error. And, certainly, there was an insufficiency of controls and verification at all levels.
*****
Edgar/ Irineo
So many questions indeed.
You raised a good point — what happened in the May 2 to Jun 6 gap?
I think I can answer that and perhaps it explains they way BPI process their ATM transactions.
The ATM transactions do not post directly into the banking system in realtime. Perhaps they perform in realtime what is termed a shadow posting into the banking system so that info on customers ATM transactions are reflected in the current or savings accounts in realtime. Shadow postings are erased when real postings are effected.
.
Each evening, operator action is required to download ATM transactions into a file that is uploaded into the banking system. The EOD run contains a function to pickup this ATM file and process it and complete the postings. So what must have happened is that on Jun 6, instead of loading the day’s ATM download file, they uploaded the Apr 27-May 2 file. That is the only logical explanation I can think off.
If that is the case, Irineo’s question is valid. Why did the process allow old dated transactions to go through. Perhaps this is a zero day event.
In general, DDA processing is already a well-automated process. Errors such as double-postings are the most basic catch of the editing and validating programs in the system in both real-time and batch portions of the DDA system. If export & import files are involved, timing validation and file-generations control would be also provided for in the design. Additionally a history log-file is probably used.
My DDAccount allows me only to withdraw a maximum of $300/day. I suspect then it is “$300-available-balance-date-driven” transaction and involves an export-import file between the ATM-subsystem & the mainframe DDA system.
As an application programmer, I most likely am no longer involved after one week of parallel testing of the DDA system. (fixes and minor enhancements) 🙂
*******
Therefore, the application programmer is not SOLELY at fault… as BPI would have people believe.
*****
I think there are too many keys available to just anybody.
And this is why I believe the Senate inquiry was all for show. That or they thought they are smart enough to ask the correct questions. I don’t think anybody in the Senate could ask the proper questions. They should’ve have deferred to the BSP.
I myself would wonder the following:
1) Normally you don’t keep the ATM transaction files too long. Once they are posted to the core bank system, you should remove it and archive it somewhere. No need to keep it in a system and consume storage resources. So why was it still there?
2) The error was blamed on an over-zealous programmer who wanted to make the process more efficient. What exactly were they trying to make more efficient? And how did she make the error?
3) The obvious question: how did she bypass the most fundamental of IT management principles, which is never let a development programmer have direct access to the production system?
4) Let us not forget that BPI outsourced their data center operations to IBM a few years ago. What role did they play in this?
3)
Indeed we are just making guesses because the Senate inquiry drew a blank. Why don’t they let the regulatory guys do their job. Why don’t they hear it from BSP first? It’s the same as the Bangladesh Central Bank case. Why did’nt they let AMLC do their job first?
Most of these Congressional hearings look like early election campaigning to me. Just a show of politicians being politicians, trying hard to impress on the electorate that they care about the issues of the day, all by stretching the meaning of the clause “in aid of legislation,” and thereby stepping on the mandates of dedicated agencies. Frankly, the reactionary, impulsive nature of these hearings irritate me. The Senate or House calling yet another investigation on the latest controversy is like an adolescent screaming on Twitter at the latest trending hashtag.
Superb point. It’s almost as if there is no real marching agenda, even to do important things like pass laws to cut through the crap of Manila traffic, or resolve the rice situation, or confirm who is an ally and who is not demonstrating behavior that is in Philippine interest. People are waiting for Federalism or figuring out how to tap dance through the death penalty while getting re-elected, rather than solving problems.
*******
Has any Congressional hearing ferreted the truth? Binay? Mamasapano? Matobato? Lascañas?
I know of one Congressional hearing series that produced untruths and landed a senator in jail.
*****
I think the Binay hearing did, but the truth seems to have dissolved in the court processes. It is almost magical. He was recently made a dean at the U. of Makati. It is almost comical.
He chairs the school where you learn to pick locks and pick pockets.
Oh, you mean public finance?
Having spent time in banking IT myself, and remembering how even the most technical people spend more time on processes than actual technical work, I would say that, at least, the blame should not fall upon any single person (i.e. the programmer). That’s one of this blogpost’s main implications, and should be a major takeaway of any reader.
Such catastrophic incidents should always be seen as a failure of processes, a lot of which are designed particularly to safeguard against human error. For example, to be specific, the programmer having access to the production environment is a failure of security processes; someone else is also accountable for not properly managing production environment access.
By the way, is the ‘programmer’ being blamed actually that, a developer? I’m thinking it might just be a generic term being used by the bank (or the Senate) to refer to IT persons. Lay people don’t distinguish between IT operations and IT development; they’re all programmers to them. (Sorry if this has been clarified, I’m not familiar with the details of the hearing. I didn’t bother because I really didn’t expect such technical affairs to be investigated properly in a place like the Senate.)
This entire mess certainly has cost BPI a lot in terms of reputation, and in operational expenses related to remediation (overtime work for tellers, IT, PR). But as far as I know, actual financial loss suffered by customers is nil. Opportunity costs due to the system downtime is the only real concern for most, but as the writer implies, verifying and quantifying such losses is a murky affair. I suggest that if funds availability is critical to you, then you should diversify—split your deposits between at least two major banks, as I do. Your eggs are too important to be placed in just one conglomerate’s basket.
Thank you for sharing your thoughts DJ.
I believe there was no clarification of ‘programmer’ at the Senate hearing. I too agree with you that the person concerned is most likely a systems operator and not a development programmer. If we are right on this, it is telling on the resource parties’ perception of senators at hearings that they don’t bother to explain this critical difference to them.
About splitting your money with different banks, you are absolutely right. A lot of people do not understand the legal relationship of depositors with banks. It is simply a debtor-creditor relationship. A depositor is simply an unsecured creditor to a bank. Come crunch time, a bank failure, depositors rank pari pasu with all other unsecured creditors. During liquidity or solvency crunch, govt authorities give depositors lower priority over other institutions considered too dangerous to fail — these are other financial institutions that will be severely affected by the problematic bank’s failure, because it will cause financial problems to cascade in the market. So govts will save these institutions first before depositors. The moral of the story — spread your deposits to various banks where you get protection of deposit insurance. I think the coverage in Phils is 500,000 pesos.
*******
Occam’s Razor would point to an Operator error rather than a Programmer error.
1. But this would mean that the entire explanation of BPI was a cover-up.
2. It would also mean that the system allows an operator to select an input batch from the ATM subsystem other than the day’s transactions.
3.Add to these the fact that the erroneous input batch consisted of several days’ transactions that (a) were more than one month old and (b) that had already been processed, then one is wielding an Occam’s Razor that has been severely blunted.
*****
I don’t know if we can go so far as to say cover-up, but the explanation at the senate was inadequate for any of us here to any meaningful opinion. But senators were satisfied so what the heck.
Yes that file containing the old transactions is problematic. Apparently they must be using the production environment to store kiv files. That’s akin to allowing smokers to go 5m vicinity of a gasoline tanker.
*******
Cover-up: “an attempt to prevent people discovering the truth about a serious mistake or crime.”
*****
i find this whole incident perplexing. I am simply not familiar with back office banking processes. I’ve never had need to know which I suspect is the case for many people.
For me the questions that are important are the following:
1 : Was this a stuff up or was it ‘deliberate’ ? I pose this question because if it was deliberate BPI would be very keen to suppress all public knowledge of this failure in it’s computer security processes.
2: If it was deliberate, then who did it for what purpose ? Could, for example, a major competitor have plotted to benefit from BPI’s embarrassment ?
3 : If this incident was a stuff up in BPI, it does not speak highly of their training & security processes.
4 : I did not follow the Philippines Senate hearings on this. But I wonder why the hell elected politicians are getting involved in this issue.
Is it a national governance issue ? No.
Is it an issue of a bank acting in an illegal or corrupt manner ? No.
Is BPI as a bank or company accountable to the Senate in any specific way ? No
Do Senators have any specific skill or experience in the Banking Industry ? No
I suggest that Senators are not doing the job they are elected to do and wasting their time.
A bit dopey.
*******
The penultimate paragraph hits the mark.
*****
1. Too late to suppress info because it affected millions of depositors. Your ‘deliberate’ is cheeky and it opens yet another conspiracy theory. Could it be they create a huge fiasco, but still containable, and in the process of rectifying the error made know publicly, undercover of the massive rectification work, they sneak in and do their targetted intention, whatever that may be….There;s no end to this haha.
2. Competitor plotting is out of the question. You seem unfamiliar with some contentious accounts in BPI?
3. BPI is a 185 year old institution. I have some respect for that. As Nherrera pointed out,it’s more likely a procedural mis-step than a process weakness.
4. All agree say AYE.
AYE
Anecdotal comment: Once upon a time in a College of Public Admionistration, it was customary before the start of the academic year for the Dean and some Senior Faculty to have an ecclesia, to face new and old students for a dialogue. The Dean was asked point blank: “Sir, In a job interview, I was asked: ‘So you have Master’s degree in Public Administration, Tell me where or on what are you good at? Frankly I did not know how to answer sir.’
The Dean hesitated and thought for a moment: “You should have told him, YOU ARE GOOD AT EVERYTHING.” Then proceeded with his long discourse. Any person or government agency by their utterances, their record of performance really demonstrates WHERE or WHAT they are good at.
A Congress of the people may be good at REMEDIAL (by investigation) legislation because (it is an admission) it is not good at ORIGINAL Legislation. Where and when a branch of government is good at for any ulterior purposes can only be IMPERCEPTIBLE to unwary (natutulog sa pancitan). Simon says: If Congress is good at corruption and at anomalous funding of development projects then expect Congress to do an excellent job investigating those things for purposes (kuno) of remedial legislation. Come to think of it now, the Dean did not say what Simon said.
Split monies among major local banks? Who else has not been in negative news about them, if RCBC, BPI, and BDO have been? That leaves Metro, PNB, Landbank, Security, EWB, AUB, Union Bank and who else untainted (?) to my knowledge.
Relating to BDO, I had been advised, by a local BDO branch manager, against using offsite BDO ATM machines. Because, allegedly, offsite ATM machines are the ones targeted by ATM hackers who are generally “foreigners”. Maybe, not just BDO’s😖. Thank goodness for Senior Priority Lanes.
Metrobank — auditors qualified the 1999 audit report — ,the bank mis-represented they P&L overstating it by 2.9 trillion pesos.
Bank of Commerce, Bohol — 2006 Branch employees withdrew 1 mm pesos from a depositors account using forged signatures.
Equitable Bank (now part of BDO) — mis-represented profits by over 4 billion pesos (can’t remember which year)
PNB,Cagayan de Oro City branch — 1995 employees involved in checkclearing fraud.
PNB — The original version of bank heist — Martial Law marcos swipe the bank clean.
Coconut Planters — Remember the Senate inquiry into allegations that P30 billion in UCPB assets had been squandered since the Estrada years, prompting the bank to secure rehabilitation funding from state-run Philippine Deposit Insurance Corp.?
How many went unreported? I know of one personally.BDO in NCR. 2015/2016 Branch executive took money from depositors accounts. Went unnoticed for a long while and bank did notgo public.So many affected depositors never knew what happened until the day they happened to update their passbooks. My wife’s account was affected.
Wow in the negative! Thanks for the info, Chemrock.
Just wanted to add — I always use ATMs at bank premises, for the security guard and if problem arises, the bank can attend to it. One day the machine swallowed my card at first attempt. I approached the bank staff and surprise, they opened the ATM, took out my card, and after some verification with issuing bank, they returned my card. Considering mine is a foreign card, I sure appreciated the branch’s response. By the way, it was BDO.
Yes, Chemrock, that’s exactly the rationale given to me for favoring onsite (at bank premises) ATMs. Still, I’d rather go in and be served, usually seated and as a priority. Onsite BDO ATM of my local branch usually has a long line as does the 2 tellers-minded counter, from 8:30 AM to 5:30 PM. There is one offsite BDO ATM at a grocery store, also, usually with a line. To date, BDO is the only commercial bank in my town. There are 5 rural banks; generally look empty inside.
Most IT programming work in banking are “fixes” in coding and plugging SQL queries that are not done in production environment because of a possible catastrophic result. Most banking systems are still coded in COBOL and had been developed years ago. Updates are “programmed” into the system when new banking policies are rolled out for implementation.
“A programmer made a backup file of ATM transaction data for the period Apr 27 to May 2 and inadvertently this file ended up as a batch input to the end of day run for June 6.”
Based on the excerpt above from chemrock’s article, it might be a network administrator who made an error as it is her/his task to configure server back-ups and run batch reconciliation.
Below is an example of IT hierarchy in most financial organizations:
https://opsdog.com/industries/information-technology/information-technology-organization-chart
Thanks JP,
To further clarify.
The network/ systems admin configures the backup servers.
The Computer operator does the backup.
I know this because I did this for a few years.(backup)
The higher ups just used “programmer” for some reason.
Thanks, Karl.
At that point, is it just a matter of activating preconfigured commands?
Yes JP,as far as system backups are concerned.
The reconciliation request from someone meant that something did not add up in his/her End of Month report, so there was a need to extract backups for dates April 27 to May2,and something got mixed up after that.
It may have been a programmer who did the recovery from backup.
A DIFFERENT PERSPECTIVE
First note the recent two international events in a First World Setting:
* The fire in a residential building in London
* The collision on a non-stormy day between a US Destroyer and a Container Ship
I wrote this to illustrate that the combination of technical and procedural components of a complex activity can go haywire in a setting other than from a Third World setting.
I really don’t know how the error propagates in a Banking System such as BPI, but let me assume that there are at least two main components: the technical and the procedural. Let me assume farther that the propagation of the error is multiplicative from the technical to the procedural. I then offer the attached table as illustrative.
In the table I used 50 components of the technical part and 3 (group of) components on the procedural part. I also used the quality level as indicated in the table.
The System Failure Level is not absolute but only RELATIVE. But note the impact of the quality of the procedural part on the System Failure:
* In one case, 4 out of a thousand;
* In the other case, 3 out of a hundred.
Comment: Thus, I agree with the observation made that we should be fair to the technical staff; the procedural part which may be in 3 levels from the lowest staff level to top management may have to account more than we may be led to believe.
One other matter not resilved by the hearing is the migration to EMV to replace all the current machines using magnetic striped cards. There was a question of security if one day its security will be breached, it was left hanging, or the answer satisfied the senator/s.I know it is a cat and mouse game this security stuff.
You are referring to the swipe cards vs smart cards?
Most cards are hybrid nowadays. Due to different technologies in countries, hybrid cards are better for the cardholders. You may have a million $ card limit but your pin based smart card is useless in countries where merchants still use the swipe or signature-based system.
Thanks Chemrock.
I hope one day you will write an article on this if the issue is still relevant.
http://business.inquirer.net/231392/bsp-orders-banks-issue-emv-cards-clients-june-30
Here is one article that covers the emv migration deadline.
This is the kind of writing I am most proud of in the Society of Honor. I bestow the Michael Lewis Award for Best Financial Writing to chemrock for this and his previous pieces on banking, IT and economics. 🙂
It is beyond the reach of trolls and morons, and pushes mankind and the public intellect forward.
Mabuhay ang matalinong Pilipino.
*******
Hear! Hear!
*****
Haha thanks Andrew. Glad you like it.
Mark Twain: “I have been complimented many times and they always embarrass me; I always feel that they have not said enough.”
Yes, chempo — it is embarrassing; they have not said enough. I want to weigh in; I hope it adds something to fill the bucket to the brim. 🙂
🙂 🙂 🙂
a short say: this piece is indeed a metamorphic rock of service and benefits to TSOH contributors, learners and readers.
“The elephants in BPI are some contentious current accounts and if events are as what officials explained, there was no revisionism of accounting entries. It’s ridiculous to even think that an Ayala-owned bank will accede to a Svengali request to attempt an audacious juggling of the systems to rewrite entries in some specific accounts, putting at risk the credibility of the bank and the entire national banking industry.”
Ahh, this must be about the conspiracy theory making its round involving a certain Svengali and his bank account(s). Is BPI, in your opinion, beyond reproach? So you are saying BPI is not amenable to what was asked of Central Bank during the Marcos era? Is it because it is a private entity?
I have no idea if there was a Marcos-central bank deja vu. The fact remains that Ayalas had damning info on those accounts and knowing where their political affiliation obviously is,it is good if they maintain their professionalism separating business from politics. The bank’s public stand on the matter has been legally and morally impeccable. Then again perhaps Pnoy admin already had some info earlier, because those huge transactions definitely ended up in reports to AMLC. That was at a time when the Svengali was still viewed as an ally in a troublesome electorate. When the mammoth propaganda machinery got into full swing, it was too late to use this info as a political weapon because it may backfire.
There is a story that still needs to be told. Someday. when an honourable admin is installed at Malacanang, AMLC officials ought to be held to account on why matters were not investigated. If indeed those huge transactions transpired, I’m sure it ended up in AMLC records. Big big question is what happened? It points to direlection of duty in Pnoy admin.
Yes. There is a possible tangled web that needs unraveling under the right circumstances. Story of Philippines’ political cycles. It is one of those cultural/political quirks that I am partial to as long as hard evidences such as financial forensics could justify a litigation.
If I could recall correctly, BPI on the same day issued new vacant IT positions which indicated that those directly responsible for the ATM software were fired.
*******
This clue is a more probable cause of the fiasco.
Rather than pointing to a general programmer or operator error on the main system, it localizes the error to the source of duplicate postings — the ATM subsystem. Simply put, the ATM subsystem erroneously passed on to the main system old processed transactions.
It could still be a “programmer” error, but a programmer working with the ATM subsystem.
The observations on levels of testing and kinds of testing would still apply.
But did not some of the transactions involve millions? And isn’t there a limit on the amount that can be transacted daily via ATM? Apr 27 – May 2 is 6 days. If the daily limit is P50K, the max in 6 days would be P300K.
There would still be the problem though that the system does not check for duplicate transactions.
***
By the way, how does the shadow posting work? Surely not by separate forms of transactions — that is, a shadow transaction in real time and a separate final transaction at batch time.
I would think there is just one transaction that is posted to the accounts in real time but is tagged with a “pending” status. Then the EOD run will update the status to “completed” or “final” .
In the first scenario, duplicate postings might occur. In the second, duplicate postings might be avoided.
There is the possibility of a shadow accounts database as well.
*****
Clinton talked about 3 dimensional chess. First has to push his policy agenda or get done what he intended to do; secondly also politics are important, get majorities, manage the utangs, damage the opponents… ; and lastly the communications with the public, educate, feed your voters red meat and manage voters’ expectations… .
I expect that the BPI management prepared the senate hearing with these 3 objectives in mind. We know too little of what they really wanted to achieve on all 3 levels. BPI objectives of the hearing “regain public confidence” ?, “make the senators look important so they will back off”?, “educate the public some very basics”? I got the feeling they achieved most of their goals.
Haha you are sharp.
The trick to throw off senators — give them the rigamorole on systems technicalities, make it sound simple, throw them off track. For Important stuff — never ask, dont tell.
Thanks for posting this Chemrock. I had read about the BPI snafu previously and it brought back memories of my 40+ years in IT where I had spent many a night firefighting and where every minute the problem remained unsolved was costing the company a lot of real $’s in lost business. After recovering from the immediate problem, a lot of hours would also be spent on post analysis and designing process fixes.
As I understand from your article, a backup file was inadvertently picked up by the production process and duplicate transactions ended up being posted. One thing has never changed — automated processes will always be prone to human error. Human intervention will always be required to deal with special cases or abnormal conditions, as in this case where a programmer (or maybe a database engineer) had to copy the batch file for a one-off reconciliation process. The goal within all IT organizations is to continue refining processes so that changes and human interventions do not affect the regular production environment.
Given what we know, I would first look at the job scheduling software which points to the files that are to be processed. Why did the batch job pick up a file that had already been processed? Normally batch jobs rename the processed files so the filename indicates if it is new or old. Other job schedulers transfer them out of an incoming folder to a processed folder. Did the programmer who copied the file forget to change the name or transfer it to a different folder?
The second place I would look at would be the validation process that normally precedes the actual processing of transactions. Was there a built-in check for duplicate transactions given the ATM transaction numbers, transaction timestamps, ATM machine number, etc.?
It is unfortunate that this issue happened within the BPI IT organization. I am certain there have been several past processing issues that went unnoticed because of IT heroes who have burned the midnight oil to fix the problems before it got to a code-blue status. This one unfortunately slipped through. One thing is sure, the BPI processes will be improved so the same issue will not happen again.
Thanks for your input Sal.
You just added more questions that need to be asked. BPI explanation glossed over the details which left people like many here with loads of questions.
Chemrock
Thank you again for another brilliant article. You never fail to meet our expectations, we learn a lot from your expertise. More, more!
I have an ATM / current account with BPI, am careful to keep just the maintaining balance and just deposit the exact funding for the PDC I issued for the current month. I maintain another account for my emergency fund – with a passbook. The dates of the account opening were not the same so I was not able to arrange for an automatic transfer of fund to the C/A from the S/A every time my check is presented for clearing. I had a scare when I was not able to deposit in time the check funding for May 28 (a Sunday). I made the deposit a day late, on May 30 due to a personal crisis I am undergoing; when I inquired, there was no record yet of any check deposited against my checking account. I wonder if this is included in the glitch, anyway the developer did not contact me re possible bounced check. When I inquired again, the bank said it was cleared June 8, curiouser, curiouser…
Yes, I try to spread my hard earned pesos keeping in mind the 500K limit that PDIC has set.
The Land Bank of the Philippines although being the depository of the government of the Philippines is also stating in their policies seen in passbook that depositors are also insured up to 500K. I was under impression that being the government depository bank, LBP will not fail, ever; that is unless the government goes bankrupt, so it is the safest bank to keep your money in excess of the amount set by PDIC.
I remember PNB enjoying the same impression as to stability until it was made known that it could go bankrupt because of the too many behest loans granted to the Marcos cronies and the way the first family treated PAL as their private mode of transportation within and outside of the country. The behest loans were later shouldered by the government to be paid out of the taxpayers’ hard earned money. Lucio Tan (rumored to be refusing to turn over to the Marcoses what they are trying to recover, he being one of the famous cronies) of PAL acquired it, he maintained the name PNB and Allied Bank was swallowed.
I sincerely hope LBP will not suffer the same fate under this administration.
It is good that your SA is not linked to your CA so a loss of ATM card carries lower risk. Small inconvenience to pay to transfer funds for checks issued.
The check drawn on May 28 by you was presented late and cleared on Jun 6 has no mystery. The drawee took his time to deposit it.
Yes LBP is a safer bet. In banking terms, as a Filipino, your govt institutions are considered risk free. Foreigners view it differently of course.
PNB was a crying shame. How Marcos screwed taxpayers the more learned knows. 16,5m are clueless. Today many still consider Imee a heroine for giving her middle finger to Congress.