> “The amount has been multiplied by 100, instead of being divided by 100.”
I'm very interested to know if there is any coverage of automated tests at all for this system? This seems like the kind of mistake that would be picked up immediately with a bare minimum test suite.
The error wasn't made in any of the 'important' paths that dealt with real winnings or real money (which I assume are heavily tested) but 'only' on the notification on the mobile app. Apparently if you logged into your account on their website you would see the correct amount. So I'm guessing the bug in this case might have been something as simple as typo in a string formatting template, that wasn't tested.
I just want to make sure I'm understanding, this means when they say somebody expected to get x euros, really they won (x/100)/100 euros? Once to get it back to what it was before math, a second time to do the math correctly?
This is one of those cultural transitions which is difficult for people on the other side to understand; it belongs to the forgotten era of personal honor. These days one would simply lie on TV, or hire a PR firm to do that for you, and of course put the blame on the lowest individual that can be found. Repeat when more mistakes are made, because low level employees are disposable accountability shields. (See, for example, the UK post office/Fujitsu Horizon scandal)
But it used to be the case that leaders were expected to take responsibility for the culture and systems underneath them, rather than just taking as much of a salary as the business will bear from it.
That is, if a low level employee makes a significant money-handling mistake on this scale, that's a systems failure. There should be checks and testing and a software development culture which makes this kind of error unlikely. This is what was lost with "move fast and break things". After all, it's only other people's money.
(edit: it seems not to have been an actual money-handling error, but a notification error. Still fairly serious in terms of angry customers)
In the public enquiry for the Horizon scandal we saw a pattern that happens over and over where extremely well paid and supposedly high performing executives suddenly remember they were idiots and had no idea about anything so it's not their fault.
I believe we should impose a statutory burden on boards for such outfits to establish that their executives are not idiots, so that when inevitably this happens again the executive is obliged to agree either they are an idiot who couldn't have known they were doing awful things but, they also lied over and over in this mandatory paperwork - and so they go to jail for lying, or, they weren't an idiot after all, they're guilty.
We use this same approach for drink driving here. You can let us take the sample, proving you were hopelessly drunk, then you get banned for drink driving, or, you can refuse the sample, we can't prove you were drunk but you refused and the same penalty applies for refusal.
I'm sure some people will protest - why should I have to put myself at jeopardy, surely I should be allowed to do crimes without consequence? OK, well lets try it out, if every CEO job goes vacant once such a rule comes into effect I guess we'll have discovered every single one of you is a crook, which is good to know. But I suspect instead we'll have no trouble finding candidates and this was simply mispriced - we won't fix the criminality but now those responsible will sometimes end up behind bars at least.
Simplified, but if CEO's want to make the big bucks when their developers create some great new feature, they should also stop making the big bucks (resign) when their developers mess up.
Because CEOs should take responsibility for the public errors of companies? If they had had proper testing procedures in place, this mistake would have been caught long before it reached the public. Clearly they did not, and the buck stops at the CEO.
In a dream world? Working for N month without pay (including giving up any bonuses etc.) and making it your personal responsibility to make sure the issues that led to this problem are actually fixed.
I am not asking the CEO to be familiar with the details of programming or with whatever other technical minutiae a company is forced to deal; they can – however – demand insurance that their systems are properly tested, even if they don't understand the details of those procedures.
CEOs are the public faces of companies; with which comes higher pay but more responsibility. Conversely, employees shouldn't be sacked for honest mistakes, since they are supposed to be faceless. If an employee makes a mistake at a company, and that company does not have procedures in place to protect against that mistake; then it's the company's problem, not the employee's. And when it's the company's problem, it becomes the CEO's.
The CEO should set the organisational culture. I would argue that - alongside the fact this isn't the first issue of this kind by the sounds of things - the fact that a serious error like this could be caused by a developer making a typo reflects very badly on the culture of the organisation.
Quality Assurance is the responsibility of management. In PRINCE2 e.g. it is stipulated to be organized by the steering group.
It other words: Management (steering group or CxO level) should ensure there is a QA organization to handle this. I don't know if this was the case here or if somebody just 'tested in production'.
Resignation definitely does not seem like the right direction, and is likely a panic reaction.
That said. The ceo is responsible regardless of their knowledge - in tech heavy companies, like this apparently is, they'd usually appoint a trusted cto for things like this.
But failure always run up the chain of command, and ofcause it should. It is after all why CEOs are paid well. They are responsible.
The entire justification that has always been given for CEOs deserving astronomical salaries is "because they take responsibility when things go wrong", hence the term "golden parachute". We should be more upset that this isn't the norm anymore, because it's proof that CEOs don't deserve their salaries.
"Figure out who won what" is kind of the core business function of a lottery, obviously upper management is responsible for ensuring that happens correctly. Not necessarily through direct involvement, but through ensuring the right people and processes are in place.
If they aren't responsible for that, what else would a lottery CEO be responsible for?
> “It is during this conversion that a manual error has been made in the code that is entered into our game engine,” the company said in a statement. “The amount has been multiplied by 100, instead of being divided by 100.”
That sounds like a problem a CEO should lead the charge to address, not resign from. I wonder if there are still deep issues.
I've always thought leaders resigning in the face of a problem, were doing a greater disservice. Bit of a cop out. Kind of like "Yeah I made a mess, but anyway - bye".
That's when you need to rise to it and resolve the problem. 'Accepting' responsibility might please hordes of blame seekers, but it's immature and short sighted. Resolving whatever the problem is would better serve the situation.
You don't hear the phrase "pour encorager les autres" very often these days, but it's one of the classic justifications for punishment: not just for the punished, but also for everyone else in a similar position. Sometimes called "skin in the game" these days. If there's no penalty for failure, there's no incentive to pay attention to the routine operations of the business.
Conversely the naval tradition of the Captain being the last person to leave a ship in distress embodies the idea of the responsibility of staying to deal with the problem. See recently the Costa Concordia incident.
Accepting responsibility and leaving is a cop-out?
That's totally opposite to my impression. She took responsibility, left and the company has now shown externally and internally that these kinds of errors have large consequences.
Also: This was not the first sign of mismanagement and bad culture at Norsk Tipping.
I get that it's sort of an accepting of responsibility if you accept we made a grave error -> the person who made (was responsible for) the grave error must be fired; I am ultimately responsible for everything -> I must be fired.
But it's not an accepting of responsibility in the sense of we made a grave error -> it must be made right. Someone getting fired doesn't help the people who were improperly notified. It likely doesn't help prevent improper notifications in the future.
Taking responsibility could mean setting up a quality control process for future changes as well as validating all current the notification strings are appropriate and accurate. And sending an apology to the affected people.
That said, it is perhaps a time for reflection when considering this:
> The company acknowledged it has experienced “a number of technical problems” over the past year.
If you're responsible and things continue to go wrong, perhaps someone else would do a better job, and the best choice is to let someone else have a try.
The article mentions – but does not elaborate – that this is not the first time the company have been scrutinised publicly. The fact that it had to attend a meeting with the ministry on a Saturday suggests the severity it was considered by the government. So I am pretty confident, that it was the minister who decided – during that meeting – that she had to resign. Heads must roll, as they say.
Kind of a ansilary question. Why are CEOs paid so handsomely if they dont't assume any risk for what their company does?
You wanna be in charge? You want to apply pressure that pushes for growth over quality? You're responsible.
We seem to live in a world where CEOs are considered both gurus heroically pulling their company along and blamelesz victims of the wills of the organization whenever there's a failure. They get the best of both worlds, compensated like the first and treated like the second between gigs.
That's a rather limited view of the responsibilities of the leader of a company. A CEO must ensure a company works properly. It's a bit of a Wall Street view: I'll have the money without the responsibilities, please.
> “The amount has been multiplied by 100, instead of being divided by 100.”
I'm very interested to know if there is any coverage of automated tests at all for this system? This seems like the kind of mistake that would be picked up immediately with a bare minimum test suite.
The error wasn't made in any of the 'important' paths that dealt with real winnings or real money (which I assume are heavily tested) but 'only' on the notification on the mobile app. Apparently if you logged into your account on their website you would see the correct amount. So I'm guessing the bug in this case might have been something as simple as typo in a string formatting template, that wasn't tested.
Somewhat sarcastically, I could see this being TDD, with a typo in the test for this :).
Definition of testing straight into the production, "what could go wrong"
Also many life-changing amounts in the same batch should prob trigger some alarms.
I just want to make sure I'm understanding, this means when they say somebody expected to get x euros, really they won (x/100)/100 euros? Once to get it back to what it was before math, a second time to do the math correctly?
That was my understanding as well. So the person who thought they had won £138,000 did really get £13.80.
relevant Reno 911 clip
https://www.youtube.com/watch?v=hpwesfKJ1AI
Test failures for the greater good
Why did the CEO leave because some dev made a typo? That wasn't necessarily caused by some error in the strategy of the company...
This is one of those cultural transitions which is difficult for people on the other side to understand; it belongs to the forgotten era of personal honor. These days one would simply lie on TV, or hire a PR firm to do that for you, and of course put the blame on the lowest individual that can be found. Repeat when more mistakes are made, because low level employees are disposable accountability shields. (See, for example, the UK post office/Fujitsu Horizon scandal)
But it used to be the case that leaders were expected to take responsibility for the culture and systems underneath them, rather than just taking as much of a salary as the business will bear from it.
That is, if a low level employee makes a significant money-handling mistake on this scale, that's a systems failure. There should be checks and testing and a software development culture which makes this kind of error unlikely. This is what was lost with "move fast and break things". After all, it's only other people's money.
(edit: it seems not to have been an actual money-handling error, but a notification error. Still fairly serious in terms of angry customers)
In the public enquiry for the Horizon scandal we saw a pattern that happens over and over where extremely well paid and supposedly high performing executives suddenly remember they were idiots and had no idea about anything so it's not their fault.
I believe we should impose a statutory burden on boards for such outfits to establish that their executives are not idiots, so that when inevitably this happens again the executive is obliged to agree either they are an idiot who couldn't have known they were doing awful things but, they also lied over and over in this mandatory paperwork - and so they go to jail for lying, or, they weren't an idiot after all, they're guilty.
We use this same approach for drink driving here. You can let us take the sample, proving you were hopelessly drunk, then you get banned for drink driving, or, you can refuse the sample, we can't prove you were drunk but you refused and the same penalty applies for refusal.
I'm sure some people will protest - why should I have to put myself at jeopardy, surely I should be allowed to do crimes without consequence? OK, well lets try it out, if every CEO job goes vacant once such a rule comes into effect I guess we'll have discovered every single one of you is a crook, which is good to know. But I suspect instead we'll have no trouble finding candidates and this was simply mispriced - we won't fix the criminality but now those responsible will sometimes end up behind bars at least.
Simplified, but if CEO's want to make the big bucks when their developers create some great new feature, they should also stop making the big bucks (resign) when their developers mess up.
Because CEOs should take responsibility for the public errors of companies? If they had had proper testing procedures in place, this mistake would have been caught long before it reached the public. Clearly they did not, and the buck stops at the CEO.
But how is taking responsability just disappearing after you messed up?
What does taking responsibility look like to you?
In a dream world? Working for N month without pay (including giving up any bonuses etc.) and making it your personal responsibility to make sure the issues that led to this problem are actually fixed.
And a pony? A sparkly one?
> If they had had proper testing procedures in place, this mistake would have been caught long before it reached the public
I don't know a CEO outside of smaller strictly-tech companies who would have any familiarity or direct involvement with testing procedures.
This is a weird head to roll for a developer's typo.
I am not asking the CEO to be familiar with the details of programming or with whatever other technical minutiae a company is forced to deal; they can – however – demand insurance that their systems are properly tested, even if they don't understand the details of those procedures.
CEOs are the public faces of companies; with which comes higher pay but more responsibility. Conversely, employees shouldn't be sacked for honest mistakes, since they are supposed to be faceless. If an employee makes a mistake at a company, and that company does not have procedures in place to protect against that mistake; then it's the company's problem, not the employee's. And when it's the company's problem, it becomes the CEO's.
The CEO should set the organisational culture. I would argue that - alongside the fact this isn't the first issue of this kind by the sounds of things - the fact that a serious error like this could be caused by a developer making a typo reflects very badly on the culture of the organisation.
Quality Assurance is the responsibility of management. In PRINCE2 e.g. it is stipulated to be organized by the steering group.
It other words: Management (steering group or CxO level) should ensure there is a QA organization to handle this. I don't know if this was the case here or if somebody just 'tested in production'.
Resignation definitely does not seem like the right direction, and is likely a panic reaction.
That said. The ceo is responsible regardless of their knowledge - in tech heavy companies, like this apparently is, they'd usually appoint a trusted cto for things like this.
But failure always run up the chain of command, and ofcause it should. It is after all why CEOs are paid well. They are responsible.
The entire justification that has always been given for CEOs deserving astronomical salaries is "because they take responsibility when things go wrong", hence the term "golden parachute". We should be more upset that this isn't the norm anymore, because it's proof that CEOs don't deserve their salaries.
"Figure out who won what" is kind of the core business function of a lottery, obviously upper management is responsible for ensuring that happens correctly. Not necessarily through direct involvement, but through ensuring the right people and processes are in place.
If they aren't responsible for that, what else would a lottery CEO be responsible for?
The winnings were calculated and paid out correctly. The error was just in the string formatting in some App notification.
[dead]
It's the CEO's job to organize their company in such a way that a dev's fat finger does not lead to such large-sized consequences.
The "strategy of the company" must include correct prize computations, and it failed.
CEO puts a lot of pressure in delivery
CEO doesn't want to invest the resources to match the delivery demand
Quality goes down, safety and security standards are lowered
Also known as the Okta-effect
The Norway lottery is not a private business, it’s a state-owned gambling monopoly.
Hence the CEO is practically a political appointee. And in politics, you want to see heads roll. Probably the Prime Minister told her to resign.
State owned lottery. So politicians get blamed, and the CEO will be fired as the politicians need to show they are 'tackling the issue'.
That was always implicitly in the job description for any public sector CEO role.
It won't affect the ceo's job prospects in any way, they will pop up in a new gig soon enough.
Norsk Tipping has recently gotten fines (46MNok and 36MNok) from the gaming authorities for other issues:
https://www.vg.no/sport/i/KMAxP4/varsler-gigantbot-til-norsk...
https://www.nrk.no/innlandet/norsk-tipping-far-bot-pa-36-mil...
It’s the latest error in a rather long line of errors, it’s not only this one
I had the same question.
> “It is during this conversion that a manual error has been made in the code that is entered into our game engine,” the company said in a statement. “The amount has been multiplied by 100, instead of being divided by 100.”
That sounds like a problem a CEO should lead the charge to address, not resign from. I wonder if there are still deep issues.
I've always thought leaders resigning in the face of a problem, were doing a greater disservice. Bit of a cop out. Kind of like "Yeah I made a mess, but anyway - bye". That's when you need to rise to it and resolve the problem. 'Accepting' responsibility might please hordes of blame seekers, but it's immature and short sighted. Resolving whatever the problem is would better serve the situation.
It can get much worse: https://voltairefoundation.wordpress.com/2020/05/20/pour-enc...
You don't hear the phrase "pour encorager les autres" very often these days, but it's one of the classic justifications for punishment: not just for the punished, but also for everyone else in a similar position. Sometimes called "skin in the game" these days. If there's no penalty for failure, there's no incentive to pay attention to the routine operations of the business.
Conversely the naval tradition of the Captain being the last person to leave a ship in distress embodies the idea of the responsibility of staying to deal with the problem. See recently the Costa Concordia incident.
Accepting responsibility and leaving is a cop-out?
That's totally opposite to my impression. She took responsibility, left and the company has now shown externally and internally that these kinds of errors have large consequences.
Also: This was not the first sign of mismanagement and bad culture at Norsk Tipping.
I get that it's sort of an accepting of responsibility if you accept we made a grave error -> the person who made (was responsible for) the grave error must be fired; I am ultimately responsible for everything -> I must be fired.
But it's not an accepting of responsibility in the sense of we made a grave error -> it must be made right. Someone getting fired doesn't help the people who were improperly notified. It likely doesn't help prevent improper notifications in the future.
Taking responsibility could mean setting up a quality control process for future changes as well as validating all current the notification strings are appropriate and accurate. And sending an apology to the affected people.
That said, it is perhaps a time for reflection when considering this:
> The company acknowledged it has experienced “a number of technical problems” over the past year.
If you're responsible and things continue to go wrong, perhaps someone else would do a better job, and the best choice is to let someone else have a try.
The article mentions – but does not elaborate – that this is not the first time the company have been scrutinised publicly. The fact that it had to attend a meeting with the ministry on a Saturday suggests the severity it was considered by the government. So I am pretty confident, that it was the minister who decided – during that meeting – that she had to resign. Heads must roll, as they say.
Kind of a ansilary question. Why are CEOs paid so handsomely if they dont't assume any risk for what their company does?
You wanna be in charge? You want to apply pressure that pushes for growth over quality? You're responsible.
We seem to live in a world where CEOs are considered both gurus heroically pulling their company along and blamelesz victims of the wills of the organization whenever there's a failure. They get the best of both worlds, compensated like the first and treated like the second between gigs.
That's a rather limited view of the responsibilities of the leader of a company. A CEO must ensure a company works properly. It's a bit of a Wall Street view: I'll have the money without the responsibilities, please.
[flagged]