Zilliz announces key contributions to Milvus 2.1, the leading open source vector database for structured and unstructured data

Zilliz announces key contributions to Milvus 2.1, the leading open source vector database for structured and unstructured data

SAN FRANCISCO – (COMMERCIAL LINE) – Zilliz, whose founders created the Milvus open source project, today announced major contributions to Milvus version 2.1. The added functionality further bridges the gap between data pools, removes data silos, and offers performance and availability improvements to address common developer concerns. Milvus is one of the most advanced vector databases in the world, capable of handling huge amounts of structured and unstructured data to accelerate the development of next generation data fabrics.

Milvus is a phased open source project within the LF AI & Data Foundation, designed for scalable similarity research and used by a wide range of companies across all industries. It embraces distributed architecture and can easily scale as data volumes and workloads increase. Highly scalable, reliable and exceptionally fast, Milvus supports DML operations (add, delete, update) and near real-time search for vectors on a trillion byte scale.

With this 2.1 update, Milvus sees a significant improvement in its performance, reducing search latency on a million-scale dataset to five milliseconds, further simplifying deployment and operational workflow.

Fill in the gaps and improve performance

Machine learning produces large pools of scalar and vector data on a daily basis. With the introduction of more scalar data types, Milvus 2.1 is bridging this critical gap between data pools.

“Data silos can now be better integrated and connected, enabling companies to fully exploit the potential of their data,” said Xiaofan James Luan, Milvus Project Manager, who also serves as director of engineering at Zilliz. “When it comes to unstructured data, the solutions offered by incumbents in the industry tend to be additional features or tools in a legacy database management system, while Milvus is designed around unstructured data from day one and now. offers more integrated features to unlock more powerful and integrated data processing ”.

Zilliz’s contributions to version 2.1 include in particular:

  • An overall increase in performance including reduced latency; Significantly improved throughput for small NQ application scenarios, such as reverse image search and intelligent chatbot; support for multiple memory replicas for small tables to increase throughput; and 2x the increase in search performance.
  • Improved scalar data processing which adds Varchar to supported data types and supports the creation of indexes for scalar data, taking hybrid search to a more intuitive level.
  • Production level improvements and increased availability, with clearer monitoring metrics for observability, simpler and more diverse deployment options, including integrated Milvus for easy deployment and Ansible for offline deployment, integration that supports Kafka as log storage, and advanced security that supports password protection and TLS connection.
  • A developer-friendly ecosystem under construction that includes more tutorials for building real-world applications, linking Milvus with the Towhee open source vector data ETL framework; and that adds Feder, an open source tool that helps Milvus users select the most suitable index for their application scenario by viewing the vector similarity search process.

In addition to the listed integration and security features, Milvus will provide more functionality essential to modern security mechanisms, including ACLs (Access Control Lists) and advanced encryption methods.

Commitment to open-source ecosystems

“As a data infrastructure for unstructured data, Milvus is revolutionary because it processes vector embeds and not just strings. Going forward, Zilliz, the company founded by the creators of Milvus, seeks to build an ecosystem of solutions around Milvus and some of the projects that will contribute to this have already emerged, including Towhee, our open source vector ETL framework and Feder, an interactive visualization tool for unstructured data. With Milvus 2.1 and the new demos, users can see how these products can come together to solve a variety of problems involving unstructured data, ”added Luan.

Zilliz is committed to the developer community and will continue to contribute to open source projects such as Milvus. The company’s technology has broad applications ranging from drug discovery, computer vision, recommendation engines, chatbots, and more.

About Zilliz

Zilliz is a leading production-ready AI vector database company. Created by the engineers behind Milvus, the world’s most popular open source vector database, Zilliz is on a mission to unleash insights into data with AI. The company develops next-generation database technologies to help organizations quickly build AI / ML applications and unlock the potential of unstructured data. By removing the burden of complex data infrastructure management from its users, Zilliz is committed to bringing the power of AI to every business, every organization and every individual.

Based in San Francisco, Zilliz is backed by a number of prestigious investors, including Hillhouse Capital, Aramco’s Prosperity7 Ventures, Temasek’s Pavilion Capital, 5Y Capital, Yunqi Partners, Trustbridge Partners and others. Zilliz technologies and products help over 1000 organizations around the world to easily build AI applications in various scenarios, including computer vision, image retrieval, video analytics, NLP, recommendation engines, targeted ads, custom search, chatbots smart devices, fraud detection, network security, drug discovery, and more. Find out more at zilliz.com or follow @zilliz_universo.

Leave a Comment

Your email address will not be published.