Databricks Announced the Public Preview of Mosaic AI Agent Framework and Agent Evaluation 

Databricks announced the public preview of the Mosaic AI Agent Framework and Agent Evaluation during the Data + AI Summit 2024. These innovative tools aim to assist developers in building and deploying high-quality Agentic and Retrieval Augmented Generation (RAG) applications on the Databricks Data Intelligence Platform.

Challenges in Building High-Quality Generative AI Applications

Creating a proof of concept for generative AI applications is relatively straightforward. However, delivering a high-quality application that meets the rigorous standards required for customer-facing solutions takes time and effort. Developers often struggle with:

Choosing the right metrics to evaluate application quality.

Efficiently collecting human feedback to measure quality.

Identifying the root causes of quality issues.

Rapidly iterating to improve application quality before deploying to production.

Introducing Mosaic AI Agent Framework and Agent Evaluation

The Mosaic AI Agent Framework and Agent Evaluation address these challenges through several key capabilities:

Human Feedback Integration: Agent Evaluation allows developers to define high-quality responses for their generative AI applications by inviting subject matter experts across their organization to review and provide feedback, even if they are not Databricks users. This process helps in gathering diverse perspectives and insights to refine the application.

Comprehensive Evaluation Metrics: Developed in collaboration with Mosaic Research, Agent Evaluation offers a suite of metrics to measure application quality. These metrics include accuracy, hallucination, harmfulness, and helpfulness. The system automatically logs responses and feedback to an evaluation table, facilitating quick analysis and identifying potential quality issues. AI judges, calibrated using expert feedback, evaluate responses to pinpoint the root causes of problems.

End-to-End Development Workflow: Integrated with MLflow, the Agent Framework allows developers to log and evaluate generative AI applications using standard MLflow APIs. This integration supports seamless transitions from development to production, with continuous feedback loops to enhance application quality.

App Lifecycle Management: The Agent Framework provides a simplified SDK for managing the lifecycle of agentic applications, from permissions management to deployment with Mosaic AI Model Serving. This comprehensive management system ensures that applications remain scalable and maintain high quality throughout their lifecycle.

Building a High-Quality RAG Agent

To illustrate the capabilities of the Mosaic AI Agent Framework, Databricks provided an example of building a high-quality RAG application. This example involves creating a simple RAG application that retrieves relevant chunks from a pre-created vector index and summarizes them in response to queries. The process includes connecting to the vector search index, setting the index into a LangChain retriever, and leveraging MLflow to enable traces and deploy the application. This workflow demonstrates the ease with which developers can build, evaluate, and improve generative AI applications using the Mosaic AI tools.

Real-World Applications and Testimonials

Several companies have successfully implemented the Mosaic AI Agent Framework to enhance their generative AI solutions. For instance, Corning used the framework to build an AI research assistant that indexes hundreds of thousands of documents, significantly improving retrieval speed, response quality, and accuracy. Lippert leveraged the framework to evaluate the results of their generative AI applications, ensuring data accuracy and control. FordDirect integrated the framework to create a unified chatbot for their dealerships, facilitating better performance assessment and customer engagement.

Pricing and Next Steps

The pricing for Agent Evaluation is based on judge requests, while Mosaic AI Model Serving is priced according to Mosaic AI Model Serving rates. Databricks encourages customers to try the Mosaic AI Agent Framework and Agent Evaluation by accessing various resources such as the Agent Framework documentation, demo notebooks, and the Generative AI Cookbook. These resources provide detailed guidance on building production-quality generative AI applications from proof of concept to deployment.

In conclusion, Databricks’ announcement of the Mosaic AI Agent Framework and Agent Evaluation represents a significant advancement in generative AI. These tools provide developers with the necessary capabilities to efficiently build, evaluate, and deploy high-quality generative AI applications. By addressing common challenges and offering comprehensive support, Databricks empowers developers to create innovative solutions that meet the highest quality and performance standards.