MySQL 8: Successful upgrade of millions of databases, and unmatched customer experience

We are happy to announce that we’ve successfully upgraded our servers to MySQL 8, and that roughly 3 million databases are now using it at full steam. More importantly, this massive transition required zero effort on the behalf of webmasters and site owners! Our unique approach to this task allowed us to provide customers with all the benefits of the newest MySQL version, and at the same time mitigate the high risk of updating such a vital underlying website technology.

While many web hosts would change their MySQL version and let customers deal with the consequences, that just doesn’t align with our values. We took it upon ourselves to bring the benefits of MySQL 8 to all our clients, and prepared extensively in order to make the adoption painless and risk-free for them. That meant testing and evaluating seemingly countless application configurations (Joomla, Drupal, WordPress and other CMSs, and all their extensions and themes that come in different setups) to foresee the tons of possible post-upgrade problems – and then eliminate them proactively and swiftly without any work or hassle for the clients. 

Let’s dive into how we made that happen.

The Challenge: MySQL8 upgrade is risky and creates heavy workload for the website owners and webmasters

Upgrading a website’s MySQL is always risky: these big version updates bring with them truly innovative features, but it also means that existing applications and web setups may not be compatible.  Because of that, it usually takes years until they get largely adopted. For example, the first version of MySQL 8 premiered in 2018 and a year after its release it only had a 17% global adoption rate. 

The decision about if and when to switch to a newer MySQL version requires the webmaster to weigh potential problems that might occur against the benefits it brings, and invest time and money in fixing post-upgrade problems. This creates a huge burden for website owners and webmasters.

That’s where web hosting companies might be of huge help – or not – depending on how they manage their servers and the level of service they provide to their customers. Among our competitors there are two popular approaches to such large-scale upgrades: either enable the new MySQL version on only the new servers you launch, and if current clients want the new version they need to relocate to the new servers; or put both the old and new MySQL’s on one server, which would increase the resource usage of the server and might ultimately deteriorate the performance of the sites hosted on it. 

Neither of these options is ideal though. In both cases, the burden of risk still falls on the website owner or webmaster, as they would need to do the work to make their sites compatible with the new MySQL. The likely outcome is that many would just not upgrade, thus missing out on performance, security and other benefits. What’s worse, many clients would continue using an older version of MySQL for so long that it would become vulnerable and unsupported by the official developers. This would put their websites at serious risk. Not a great plan. 

SiteGround’s unique approach: millions of databases automatically updated to MySQL8 without hassle for the clients

At SiteGround our philosophy is to ensure that all our customers can benefit from the latest technologies, and with the least possible hassle: no work, no risk, no extra investment, just take it out of the box and use it. When we offer a new software like MySQL 8 that comes with multiple benefits, we make sure all clients can safely and easily enjoy it. 

This is why our approach to this upgrade was different. We didn’t leave it to our clients to figure out how to upgrade to MySQL8; we did it for them and took on all the work to eliminate incompatibilities and resolve post-upgrade problems. That upgrade process included evaluating the level of risk, researching all possible incompatibilities and upgrade errors, automatically fixing problems, and manually verifying the results. As a result, millions of sites are now using MySQL 8 without any extra headaches.

Extensive research and multiple dry runs to reduce upgrade fail rate 

We began by extensively researching the incompatibilities of MySQL 5.7 (the previously used version of MySQL). The goal was to discover which issues could act as potential blockers to updating a database to MySQL 8. 

Next, we copied and isolated hundreds of servers and started dry-run upgrades. The objective was to identify problematic software and setups that break after the transition, find solutions for them in advance, and document all results carefully. We used all that data to minimize the upgrade fail rate and to automate the aftermath problem resolution.

Automated and quick fixes of broken sites

Building on the extensive research and testing, our engineers developed a smart system that essentially served as a skilled mechanic for your website. It would run a compatibility check with MySQL 8 based on your specific application configuration and database structure. If it detected incompatibilities, it would patch them on the spot.

As a result, after the upgrade this automatic system helped us to immediately remove errors and broken queries for thousands of websites, and it spared us tons of manual work, in addition to resolution wait time for the clients.

Manual fix of 7518 websites by our experts (free of charge for the clients)

The automated checks did a tremendous job, but we went even further. After a server was upgraded, our technicians would open pretty much all websites hosted on it to make sure they were fully functional. Of those, they identified 7518 websites with issues, which they then fixed manually after the migration in due time. 

804 sites provided with a time extension (at our expense)

All our automated and manual checks left us with an impressively low number of sites that remained incompatible with the new MySQL version — below 0.001% of all. For these clients we provided a custom server setup where they would be able to use MySQL 5.7 for two additional months allowing them enough time to address significant query incompatibilities and to get ready for the new version. We believe each of our users should be given the option to receive the best service, so at this time we’ve absorbed the cost of maintaining the old version on additional servers. 

Efficient server upgrade schedule paid off in fast problem resolution

One of the internal challenges during the process was to schedule the servers for upgrade so that we would have had enough people to not only address potential issues, but also complete the upgrade successfully in the shortest possible time frame. Based on our statistics from the dry runs, we could predict how many websites would have had issues per server, and therefore knew how many servers to schedule and how many technicians to staff so that we could address problems quickly. At some point, our process was so efficient that it allowed us to successfully upgrade as many as 180,000 websites in 24 hours! 

Timely and honest communication with customers

Those of you who’ve been customers for a while surely already know how much we value transparent and clear communication when it comes to events that impact your websites. This upgrade to MySQL8 was no exception. We notified each client at least seven days prior to the transition, informing them of the day and hours we would upgrade their websites (always non-business hours for their respective region). 

The Summary 

Our top priority in this upgrade process was that all sites would be fully functional as usual after the upgrade – no problems, just benefits from the newer MySQL. The outcome? We had a 99.99%+ success rate of the migration process! 

• With solid planning, preparation and precise execution, we successfully upgraded roughly 3 million databases and made them fully compatible with MySQL 8 without any work for our clients. 
• It took us only 63 migration days to achieve this for all websites hosted on our platform, totally free of charge, all while making sure that every single client website had been thoroughly taken care of. Well, let’s not forget the month or two we invested in preparation work,including research, dry runs, and automations.
• A special team of 34 people worked for a total of 1228 hours on the actual upgrade, completing the process with utmost efficiency and professionalism – it took them an average of seven minutes per server to switch and resolve issues. 

It was an enormous task on a staggering scale, especially considering the high bar we set for ourselves. But in the end all the hard work on our part paid off in absolutely no work for our clients. And that in itself, for us, is the end goal that justifies all the effort.