House of Cards: a view from the ivory tower
![SteemImg](https://steemitimages.com/640x0/http://www.steemimg.com/images/2016/11/26/pic178811.jpg)
I work for a global investment bank, in the upper reaches of a vast office tower just like this one. It's one of those Wall Street companies that people love to target during election campaigns. And you know what: conspiracy theories aside, there are good reasons to be concerned about these rulers of the financial markets.
This post will be the first in a series chronicling my adventures within the secretive & byzantine labyrinth of finance. I'll be referring to my employer as Big Bank throughout, obscuring certain identifying details, and speaking in generalities. This is mainly out of practicality; I need to keep paying the bills and have no desire to get on the wrong side of non-disclosure agreements (whoever says free speech isn't dead has obviously never worked for a large corporation).
In writing this I aim not to make my employer look bad, but to sound the alarm and raise awareness. The world's financial system is teetering on the edge of a cliff, and sanity must prevail before it falls off.
My duties for Big Bank
I'm a member of the front office IT team. That means I look after the software systems used by the bank's traders and clients. As opposed to the middle office or back office, which own different systems related to trade settlement and risk management. My job is split pretty evenly between tech support and software development duties. I'll start by giving you a brief overview of each, and then we'll go into more detail in future posts.
1. On the front lines of tech support
The role is referred to as front line support for good reason: at times it can feel like making a courageous last stand in the middle of a war zone, standing your ground while facing down hoards of angry traders, impatient salespeople, and stressed out clients.
My team is basically the first point of contact when IT systems malfunction (which happens with disturbing regularity). At least two or three people monitor the support mailbox at all times, 24 hours a day in rotating shifts. Response time is critical; if client facing trading systems go down and aren't brought back online quickly, the bank can lose millions of dollars from lost business opportunities, not to mention reputational impact with clients who depend on reliable service. System malfunctions are also carefully monitored by government regulators and in extreme cases can result in large fines levied against the bank.
During support shifts I don my detective hat. When issues occur, my goal is to solve the case and ensure as little business disruption as possible. A typical incident might go something like this:
- E-mail comes in from a disgruntled trader: "hey all the prices are stale in my liquidity GUI. What gives!?"
- I walk over to the trader's desk, get him to show me the problem.
- I go back to my desk and fruitlessly hunt around in log files looking for cryptic error messages that might be related.
- After pulling info from the logs, I read through a support wiki with sparse documentation that has nothing useful whatsoever about this mysterious problem I've never seen before.
- I open a chat window with someone on the pricing team: "do you have any idea what this error means? I've never seen it before".
- Response comes back: "hmm, no idea. Try asking my counterpart in London, he's the expert".
- Of course London guys won't be in the office for 5 more hours. Meanwhile trader e-mails me again: "what's taking so long? I've gotta be ready for a BOJ (Bank of Japan) announcement in 10 minutes!"
- Me: "umm... let me try restarting the server software". I proceed to restart the process in question, praying it won't break something else and / or get me yelled at.
- A few minutes later, trader happily calls me up: "it worked, you're a genius!"
- I feel good, even though I have no idea what the problem was or why restarting fixed it.
- I send a follow up mail to the development team that made the software: "we had this incident today, would you mind taking a look and letting me know what the root cause was?"
- Development team never mails me back.
![SteemImg](https://steemitimages.com/640x0/http://www.steemimg.com/images/2016/11/26/pic31e936.jpg)
This right here illustrates the whole problem with tech support in a nutshell. We have a bewilderingly vast array of systems to support, many of which require specialized business knowledge to operate. The support teams are understaffed and undertrained, with most knowledge concentrated amongst a few senior people scattered around the globe. If I was a client, I'd be scared witless if I knew just how little the people that are supposed to keep these multi-million dollar software systems running actually know about them.
The problems are two-fold:
- Entropy - as time goes on, systems get increasingly chaotic. Documentation that might have been perfectly well organized at product launch quickly grows outdated. New features introduce more complexity, and more bugs. Longstanding problems don't get fixed properly due to budget constraints, and hastily improvised patches keep things running but at the expense of maintainability.
- Knowledge transfer (or lack thereof) - in an age of reduced profits and increased regulation, layoffs are common. And every time someone gets axed, they take all their specialized systems knowledge, sometimes built up over many years, with them. Training is minimal, done in informal hour-long sessions that are barely sufficient to scratch the surface. Often the only help I get is "here's a system diagram and a wiki page that may or may not be accurate; good luck!".
2. Financial software development
My days off from support shifts are known as "dev days". This is the part of the job that is actually fun and rewarding. Big Bank has created a lot of custom in-house development tools & frameworks to make the lives of developers easier and provide a consistent strategic direction for the firm. There's always something new to learn and some interesting project to sink my teeth into.
The first thing to know about a bank's financial software is that it's not just a single computer program that runs on your PC, like Microsoft Word. It truly is a software system, made up of hundreds of individual programs (yes, hundreds) that all communicate with each other, passing data back & forth to accomplish some larger purpose.
For example, consider the simple action of a client using electronic trading software to buy some Japanese yen (JPY) with US dollars (USD). Behind the scenes, this triggers a hugely complicated data flow within Big Bank to "book" the trade and settle the cash flows it represents. In highly simplified form, the system diagram for this action might look something like this:
Note: this is a hypothetical example whipped up for illustrative purposes only. It is not a depiction of real trading systems at any company I have worked for.
Each box in that diagram is a separate computer program, most of which run on the bank's servers. As if this isn't complicated enough, different teams own different parts of the system, with their own programming styles, logging formats, and interfaces for the programs to talk to other programs. This is represented by the color coding in the diagram:
![]() | The pricing system runs complicated mathematical algorithms on trader input to produce quotes for various financial products |
![]() | Human facing graphical interface programs |
![]() | Trade booking system (I work mostly on these components) |
![]() | The settlement system handles cash flows, trade confirmations, and client account management |
![]() | Databases for storing trade information & miscellaneous info needed to settle the trades |
But wait! There's more...
Legacy Software
Really old bits of the system that no longer work so well and have fallen into disrepair are known as "legacy software". Just hearing that term and thinking of all the trouble it causes is enough to make me shudder. It puts me in mind of a big ball of tangled, rusty piping, much patched over with duct tape and leaking water from a dozen places. Not something you want to go near if you can help it, for fear that the slightest touch will cause the whole thing to fall apart.
This is legacy software given physical form. The stuff of nightmares...
I once worked for a firm where the low-level plumbing in the back office settlement systems dated back to the early 1990s. And sections of it were even older, written in FORTRAN (an ancient programming language that nobody in their right mind would use for business software today). All the original developers were gone, and nobody really understood how the hundreds of thousands of lines of code worked. And this was a vital system at the heart of the bank's business, without which the firm could not function on a day-to-day basis.
God forbid it should ever malfunction, because nobody would know how to fix it. Before I left the firm for greener pastures, there was an attempt to start an initiative to replace this aging monster with something more modern. But it never got off the ground because it was estimated to require several years of effort, hiring a whole new team of expert developers, and carry a price tag in the tens of millions of dollars.
Sadly, this is not an uncommon problem for a bank to have. Pretty much every firm I've worked for has been saddled with legacy software of some form or another, and dealing with it consumes a significant portion of the IT budget. The finance industry as a whole is paying a high price as it struggles to modernize and keep up with the latest technology.
Beyond human comprehension
Now you can begin to understand why maintaining and providing tech support for Big Bank's software is such a challenge. By their very nature, financial systems are brittle and break easily, despite attempts to build in redundancy and safeguards (which in turn just makes the system even more complicated and difficult to understand, like a vicious feedback loop).
These days there is much talk at Big Bank about using machine learning algorithms and adaptive AI to tame the complexity, thereby taking away a lot of the support burden from humans. Imagine programs that can self-diagnose and heal themselves, or are able to judge the severity of issues, only turning to their human masters for help in the most extreme circumstances.
With vast, sprawling software systems on the verge of spiraling beyond the ability of humans to comprehend them, more and more duties will be handed over to machines. Already, Big Bank's systems are too big for any one person to understand in their entirety; efforts to trace data flow from one end of the system to the other often require coordination across multiple teams of engineers. Travel down this path far enough, and it's not impossible to imagine a day when my job gets completely replaced by an artificial intelligence.
Would you want this to be your new boss?
Anyone who's seen The Matrix, or Terminator, should fear that day. Skynet may not be that far-fetched after all.
What can we do to avert disaster?
Well, setting aside the issue of malevolent AI for the time being, it's clear that the present state of affairs cannot continue. Sooner or later, the world's financial system will implode in spectacular fashion unless drastic changes are made.
The answer may be blockchain technology, which has generated much excitement in financial circles. This radical new technology (upon which Steemit itself is based!) has the potential to cut clean through the rusted pipes of legacy software, replacing the whole cancerous system with something much cleaner and purer. But that's a topic for a future blog post.
I'll stop here, ending on a positive note with my blockchain teaser that all is not lost, that a solution could be at hand if we but have the courage to reach for it.
Next time, we will journey deeper into the financial mire and discuss one beast that bank employees fear above all others: the dreaded RIF (Reduction In Force). So stay tuned and grab your weed-whacker, because there's plenty more adventuring to come!
Links for further study
Good explanation on the differences between front office, middle office, and back office: http://www.investopedia.com/terms/f/frontoffice.asp
Discussion & examples of legacy software: https://en.wikipedia.org/wiki/Legacy_system
A starting point for researching the types of financial software systems I work on: https://en.wikipedia.org/wiki/Electronic_trading_platform
For more posts about cryptocurrency, finance, travels in Japan, and my journey to escape corporate slavery, please follow me: @cryptomancer
Image credits: the first image of the skyscraper is a photo I took on my iPhone. The trade flow system diagram is my own invention, made with Microsoft PowerPoint & incorporating some people pictures from a clip art package I purchased and own distribution rights to.
All other images in this post are taken from Pixabay and used under Creative Commons CC0 .
With family coming in, I had this on my laptop screen (closed) until I could read it! MAGNIFICANT! Not only do your descriptions give the reader understanding, but your choice of photos seal the feeling. I particularly love the one you provide regarding the 'legacy software!'
Not only well written IMO, cryptomancer, but the word 'informative' is inadequate.
I only ask that please, please, do not put yourself in jeopardy to bring us this - unless you are prepared for any consequences from "Big Bank" (or any other such entity, who may also believe this could be them). As much as I appreciate your candidness, I fear for your safety in sharing this with us. "Big Banks" can decipher small posts, no matter how much one 'blacks out' certain parts. Please be careful.
Thanks for your concern, it is appreciated. I believe I have taken adequate steps to keep myself covered, and will have to hope it's enough.
I spent almost as much time preparing the right pictures as I did writing the article. It sounds like they conveyed the intended feelings, so that's good feedback to get, thanks! :-)
Nice reading this kind of insider perspective...def something i don't hear much of in the academic ivory tower!
Thanks, glad you enjoyed the perspective. Sometimes there's just no substitute for getting down in the trenches and getting your hands dirty to see the state things are really in.
Absolutely!
Informative, and quite frankly the best post I've read on steemit in months.
Thank you.
You're welcome, glad you liked it.
I remember that recent DASH video from some crypto conference or another with the slide that says:
My Boss is a blockchain.
And it is a great boss!
Or something like that. Probably not a thing to change jobs just yet, but something that might be interesting to look forward?
Also as someone who worked tech support for a while, I can relate. It wasn't as stressful, but I was doing support for other systems that used our backup and disaster recovery software. Sometimes the systems that it had to interface were really esoteric systems, but the biggest perk was, that there were usually other engineers on the other side.
I think we're pretty far away from some of the more fanciful "company on a blockchain" visions. Things like DAOs for decentralized governance need more time to mature. But I think it's a fantastic vision that would allow people to move around freely, essentially working wherever they want instead of having to live in big cities to get the best tech jobs. Note this would be different from handing over key business functionalities to an AI; at the end of the day humans are still the decision makers with blockchain acting as a facilitating communications tool.
I find one big problem is sometimes it's hard just to know who to talk to when stuff goes wrong. When there's a problem with some esoteric system you've never heard of before, how do you find out who the engineers on the other side are? And what if they're all asleep on the other side of the world?
Yep, the humans make decisions, but the blockchain allocates the money, based on their decision. And it is something that is happening today.
I am talking about what I am doing for DASH today, but blogging and Steem blockchain is similar. :-)
Anyway. I was on the end, where people were calling me with their problems, thus our team was awakre 24/7/365. Some times those calls were just to justify the corporate fees, and whatnot, but at others it was something weird happening, or people testing the software in their environment to decide if it is going to work there.
So no such problems there, but that was about as lucky as you could be as support engineer. :-)
Very nice reading!
Thanks, I'm happy you found it so.
This is a fantastic and very well detailed article of what it is like wearing multiple hats in a IT system where balancing all of the different systems is like twirling plates on sticks.
It is exactly how I feel every day. Most tech calls can be solved with a "When was the last time you shut down you computer then turned it back on - reboot?"
There are days where you think you will get to work on a development project you want to work on but get stuck overwhelmed in B.S. because the techs are too busy.
Legacy software is kicking my butt right now. I need to get it moved to 64 bit and feel like a gopher trying to tunnel my way through some kinda way.
Thank you for writing - I will be sharing.
Thanks, the resteem is much appreciated! I think a lot of people in the tech industry can relate to these experiences. Good luck with your legacy software migration - I hope it works out OK!
Oh, this is the worst. I can't tell you how many times I'd been looking forward to a productive dev day only to have it ruined by some massive support problem that requires "all hands on deck" to get resolved. When I give project stakeholders an estimate of how long it will take to develop something, I always add at least 50% to the estimate to make up for time lost due to support issues.
My life - I have a project I am working on that is awesome - software for a small yet vital division. But my life the past 2 months has been re-writes of sql pulls because of managers who know nothing about their programs and their original requests were screwed. Then of course, we are doing big migrations and techs, my backup are all busy.
My project is 40 days overdue and looks like January right now. I hate that.
Yikes, sounds like a tough situation to be in, especially with the end of the year coming up. I find there is often some friction between business users / managers that don't understand the tech side, and programmers who don't necessarily understand the business side. Hard to meet in the middle and hash things out to come up with a reasonable understanding on requirements. Good luck getting caught up!
Great post, do you think the house of cards will fall while you are still at working age or will it all move over to Blockchain and AI before a catastrophic event happens?
I think it could go either way. We're at a tipping point right now. People inside the banks are aware of the magnitude of the problem, but there's not a firm consensus yet of how to fix it. The very fact that firms are experimenting with new technologies and radically different methodologies is encouraging, but the process is still in its infancy and change does not come easily to these conservative companies. It takes time to steer a ship this massive away from the icebergs. What worries me is that it's been 8 years since the 2008 financial crisis, and it feels like depressingly little has changed since then. What should have been a wake up call is now merely a historical footnote. I just hope it doesn't take another such event, or something even worse, to really light the fires of change.
When the death star blows up and the janitor is killed no one feels bad that he was just doing his job to feed his family by supporting the empire to keep their killer weapon aimed at people. You should find another job and let the system rot.
That's a great analogy. I think sometimes people who vilify banks forget about the ranks of ordinary people who work at them just trying to make ends meet to support their families. These people don't necessarily care about ideology, they just want to survive. Darth Vader and the Emperor are the ones deserving of all the hate, not the maintenance crews trapped into following orders who take pride in a job well done at the end of the day.
Ultimately, instead of getting a new job I would like to quit being a corporate slave altogether, and retire to live a life of freedom doing what I want instead of what my boss wants. Hopefully someday my cryptocurrency investments will allow me to take that giant leap forward and leave the rat race behind for good. But I'm not there yet.
There's an intermediate step of working somewhere that doesn't spend all it's time and effort literally enslaving the entire planet. You seem like a good guy. Stop doing things that help others do bad shit for a paycheck and get a new line of work. You are the banking equivalent of the dipshit cops at standing rock.
Thanks for sharing your experiences mate. Really insightful and interesting read. Look forward to the next part in this series..
Thanks, glad you enjoyed it and I hope the next part will not disappoint.
Great stuff. Tired as hell right now but skimmed through it. Resteem'd so I can read it more in-depth tomorrow! Steem on, brother!
I appreciate your support as always, and thanks very much for the resteem. We're all in this together!