Database update paralyzes Toyota production
3 min readWhat was supposed to be routine maintenance, but instead there was a two-day shutdown of 14 automobile plants.
On August 28, Toyota’s factories shut down, reducing global production by a third at a company known for producing under tight deadlines.
The cause? A database maintenance error that caused the database to exceed available disk space on both the primary server and the backup.
Because the database wasn’t working, Toyota could no longer make the automated orders needed to produce 13,500 cars per day. And for two days, 25 production lines across the 14 factories sat empty as the “disruption” meant new car parts could not be ordered from suppliers.
That might not have been a problem—until you combine that with the company’s famous “just-in-time” approach, a careful balancing act in which you don’t get the goods or raw materials you need until you need them.
A story of “Just in Time”
Just In Time (JIT) can be a great ally for the right company – it maximizes inventory turnover, which in turn brings many benefits to sales and costs.
With precise and surgical handling, storage space requirements and associated costs, such as labor and electricity costs, can represent a fraction of your daily or weekly production.
But if the balance tips one way or the other, you can either have too many raw materials and nowhere to store them, or there are no raw materials and machines and employees remain idle.
The latter appears to have been the case at Toyota, where, according to calculations by Reuters a third of global production came to a standstill. The news agency also suspects that the JIT structure will make it difficult to catch up with the two-day backlog.
An extremely efficient production line operating at near maximum capacity every day doesn’t leave much room for productivity improvements, meaning Toyota could be struggling with late deliveries to customers for some time to come.
Seiji Sugiura, an analyst at Tokai Tokyo Research Institute, told US News: “Production has been at full capacity, so there is little additional headroom for production.”
Toyota: Not enough hard drive space
So what happened on the maintenance side?
Toyota said: “The system disruption was caused by the unavailability of several servers that process spare parts orders.
“As far as the circumstances are concerned, regular maintenance work was carried out on August 27, the day before the incident occurred. During maintenance, data accumulated in the database was deleted and organized, and an error occurred due to insufficient disk space, causing the system to fail.
“As these servers were running on the same system, a similar error occurred in the backup function and a switchover could not be made. This led to an interruption in the operation of the domestic plant.
“The system was restored after the data was transferred to a larger capacity server on August 29, and the facilities resumed operations the following day.
“We would like to inform that we have found the above cause. Countermeasures were also taken by repeating and reviewing the situation.
“We would also like to reiterate that the system disruption was not caused by a cyber attack and apologize to everyone involved for any inconvenience this may have caused.
The bottom line
It is a chilling example of how a small part of the system, in this case routine maintenance of a database, brought the entire productivity of thousands of workers and 14 factories to a standstill, causing disruption to suppliers and customers and even a… temporary slump on the stock market led.
As the world moves toward the benefits of automation and artificial intelligence (AI), this is a reminder that the more parts are connected, the worse a domino effect can be.
Still, the system works well for them 99% of the time, so the counterargument is an ode to efficiency in good times.
And although a company can only be as strong as its weakest component, at least this gap in the system has been patched.
Toyota added:
“Going forward, we will review our maintenance procedures and increase our efforts to prevent a recurrence so that we can deliver as many vehicles as possible to our customers.”
The best platfroms to buy Bitcoin by debit or credit card 2023
