Mozilla.ai has developed 'Otari,' a system that allows for centralized management of multiple AI models.
Otari: Own Your AI Stack | AI Gateway & Hosted Platform
https://otari.ai/
Introducing Otari: The Open-Source LLM Control Plane
https://blog.mozilla.ai/introducing-otari-the-open-source-llm-control-plane/
Mozilla.ai was founded with the aim of being 'a company that develops AI while ensuring openness, transparency, and accountability,' amidst a growing trend of AI development companies keeping their data private. Mozilla.ai believes that developers should be able to build open, flexible, and self-controllable AI systems, and Otari was developed with this in mind.
Otari is a control plane for centralized management of LLM infrastructure. It helps developers and engineering teams centrally manage request routing, cost management, governance, deployment, and reliability across multiple LLM providers.
Developers can access the following features from a single platform:
- Route requests to multiple LLM providers through a single endpoint.
- Manage LLM usage costs through budget management functions, usage visualization, and automatic budget limits.
- Centralize management of API keys, workspaces, and access permissions to strengthen governance.
- Supports both cloud-hosted and self-hosted models.
- Service reliability is improved through 'intelligent routing,' which automatically distributes requests to the most suitable LLM, and 'automatic failover,' which automatically switches to another LLM in the event of a failure.
- Build AI agent-compatible applications using 'Agent Harnesses,' which is equipped with tools and orchestration functions for AI agent development.
According to Mozilla.ai, Otari aims to reduce the time developers spend on complex infrastructure operations and allow them to focus on developing AI applications by providing a centralized environment for managing LLM infrastructure. Below is the Otari dashboard.
Regarding the reasons for developing Otari, Mozilla.ai explained, 'While working with developers and organizations that were adopting LLMs, we witnessed the same infrastructure challenges recurring many times. Each team was independently building their own routing logic, managing provider keys, tracking usage and budgeting, and implementing governance in each application. We believe that these should be managed in a dedicated control plane that sits between the application and the LLM provider, and we wanted to provide a single place for developers to manage and operate their infrastructure.'
Otari is an open-source project, and its repository is hosted on GitHub.
GitHub - mozilla-ai/otari: Open-source, OpenAI-compatible LLM gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking. · GitHub
https://github.com/mozilla-ai/otari
Related Posts:
in AI, Posted by log1e_dh