Manu: A Cloud Native Vector Database Management System
With the development of learning-based embedding models, embedding vectors are widely used for analyzing and searching unstructured data. As vector collections exceed billion-scale, fully managed and horizontally scalable vector databases are necessary. In the past three years, through interaction with our 1200+ industry users, we have sketched a vision for the features that next-generation vector databases should have, which include long-term evolvability, tunable consistency, good elasticity, and high performance. We present Manu, a cloud native vector database that implements these features. It is difficult to integrate all these features if we follow traditional DBMS design rules. As most vector data applications do not require complex data models and strong data consistency, our design philosophy is to relax the data model and consistency constraints in exchange for the aforementioned features. Specifically, Manu firstly exposes the write-ahead log (WAL) and binlog as backbone services. Secondly, write components are designed as log publishers while all read-only analytic and search components are designed as independent subscribers to the log services. Finally, we utilize multi-version concurrency control (MVCC) and a delta consistency model to simplify the communication and cooperation among the system components. These designs achieve a low coupling among the system components, which is essential for elasticity and evolution. We also extensively optimize Manu for performance and usability with hardware-aware implementations and support for complex search semantics. Manu has been used for many applications, including, but not limited to, recommendation, multimedia, language, medicine and security. We evaluated Manu in three typical application scenarios to demonstrate its efficiency, elasticity, and scalability.
|EI Accession Number|
Concurrency control ; Database systems ; Elasticity ; Embeddings ; Semantics
|ESI Classification Code|
Database Systems:723.3 ; Artificial Intelligence:723.4 ; Algebra:921.1
Cited Times [WOS]:0
|Document Type||Journal Article|
|Department||Southern University of Science and Technology|
2.Southern University of Science and Technology,China
3.Technical University of Munich,Germany
Guo，Rentong,Luan，Xiaofan,Xiang，Long,et al. Manu: A Cloud Native Vector Database Management System[J]. Proceedings of the VLDB Endowment,2022,15(12):3548-3561.
Guo，Rentong.,Luan，Xiaofan.,Xiang，Long.,Yan，Xiao.,Yi，Xiaomeng.,...&Xie，Charles.(2022).Manu: A Cloud Native Vector Database Management System.Proceedings of the VLDB Endowment,15(12),3548-3561.
Guo，Rentong,et al."Manu: A Cloud Native Vector Database Management System".Proceedings of the VLDB Endowment 15.12(2022):3548-3561.
|Files in This Item:||There are no files associated with this item.|
|Recommend this item|
|Export to Endnote|
|Export to Excel|
|Export to Csv|
|Similar articles in Google Scholar|
|Similar articles in Baidu Scholar|
|Similar articles in Bing Scholar|
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.