Kristen Evans | 05 February 2018
Today, the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC) voted to accept Vitess as the 16th hosted project, alongside Kubernetes, Prometheus, OpenTracing, Fluentd, Linkerd, gRPC, CoreDNS, containerd, rkt, CNI, Envoy, Jaeger, Notary, TUF and Rook. Vitess has been accepted as an incubation-level project, under the CNCF Graduation Criteria v1.0.
The TOC’s Graduation Criteria provides every CNCF project an associated maturity level of either inception, incubating or graduated. As an incubating project, Vitess must document it is being used successfully in production by at least three independent end users, have a healthy number of committers and demonstrate a substantial ongoing flow of contributions.
Originally created as an internal solution by YouTube to handle scaling for massive amounts of storage, Vitess is a database orchestration system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vitess allows application code and database queries to remain agnostic to the distribution of data onto multiple shards. With Vitess, organizations can even split and merge shards as needs grow, with an atomic cutover step that takes only a few seconds. Companies like BetterCloud, Flipkart, Quiz of Kings, Slack, Square Cash, Stitch Labs and YouTube are using Vitess across various stages of production and deployment. Organizations including Booking.com, GitHub, HubSpot, Slack, and Square are also active contributors to the project.
“Slack is in the midst of a major migration of the MySQL infrastructure at the core of our service, driven by the need for an architecture that scales to meet the growing demands of our largest customers and features under the pressure to maintain a stable and performant service that executes billions of MySQL transactions per hour, said Michael Demmer, Senior Staff Engineer at Slack. “We needed a solution that would offer a familiar full featured SQL interface, and wanted to continue to use MySQL as the backing store to maintain our operations knowledge and comfort level. Vitess is a natural choice for this purpose and has served us well so far.”
Vitess has been under development since 2010. The earliest version was a connection proxy that helped to buy some headroom, but over time the features evolved, while the tools and servers grew to be more efficient, fault tolerant, and manageable. This iterative journey led to what Vitess has become today: a distributed, cloud-based storage solution that exhibits some of the best properties of a relational database.
“Faced with rapid organic and internal growth at YouTube, we had to come up with something that would leap ahead of the curve instead of just fighting fires. When we finally built the initial feature list for Vitess, it was obvious that we were addressing problems that are common to all growing organizations,” said Sugu Sougoumarane, CTO at PlanetScale Data and Co-creator of Vitess. “Our collaboration with Kubernetes over the last two years means anyone can now run Vitess the way YouTube does: dynamically scaled and scheduled in a container cluster. We’re excited to work with the CNCF to expand the capabilities of Vitess even further.”
Main features:
- Combines important MySQL features with the scalability of a NoSQL database
- Enables MySQL to run in the cloud
- Cloud-native functionality such as support for automatic failover/recovery, replication and rolling upgrades
- Vertical and horizontal sharding support, and virtually seamless dynamic re-sharding
- Multiple sharding schemes, with the ability to plug-in custom ones
- Query routing, rewriting and sanitization, blacklisting, streaming, and de-duping
- Master management tools (handles reparenting)
- Performance analysis tools
Technical Specs:
- Backend components implemented in Go
- Continuously tested against Ubuntu 14.04 (Trusty) and Debian 8 (Jessie). Other Linux distributions should work as well
- Supports MySQL 5.6, MariaDB 10.0, and any newer versions
- VTGate server is the main entry point applications use to connect to Vitess
- Supports data backups to either a network mount (e.g. NFS) or to a blob store
“To build successful cloud native applications, developers have to consider new ways of managing storage,” said Chris Aniszczyk, COO of Cloud Native Computing Foundation and Vice President of Developer Relations at The Linux Foundation. “Vitess is a fantastic example of bridging the existing popular MySQL database and scaling it in a cloud native fashion.”
The open source Vitess project runs best in a containerized environment. With Kubernetes and Google Kubernetes Engine as the container cluster manager, it’s now a lot easier to get started. In this environment, Vitess provides a MySQL storage layer with improved reliability, scalability, and manageability. Kubernetes handles scheduling onto nodes in a compute cluster, actively manages workloads on those nodes, and groups containers comprising an application for easy management and discovery. Also with Kubernetes, Vitess even becomes agnostic to the underlying choice of cloud platform, providing cloud portability with no vendor lock-in.
Notable Milestones:
- 105 contributors
- 5,413 GitHub stars
- 15 releases
- 13,733 commits
- 707 forks
“A big advantage of cloud solutions is easy horizontal scalability, but when it comes to stateful systems such as databases, the problem is that application complexity and operational overhead tend to scale up along with the number of instances,” said Brian Grant, TOC representative and project sponsor. “A cloud-native solution like Vitess abstracts the complexity of horizontal scalability from both application developers and database operators. We’re thrilled to welcome Vitess as a CNCF project as we continue to enable more classes of workloads in cloud-native environments.”
As a CNCF hosted project, Vitess is part of a neutral cloud native foundation aligned with its technical interests, as well as the larger Linux Foundation, which provide the project with with project governance, marketing support and community outreach.
For more on Vitess, read DZone’s Launching Vitess: How to Run Youtube’s MySQL Sharding Engine, or watch the the recording of the Vitess beyond YouTube session from Percona Live Open Source Database Conference 2017. You can also read Vitess’ guide to getting started on Kubernetes.