Skip to content

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

License

Notifications You must be signed in to change notification settings

RoiTabach/fm-leaderboarder

 
 

Repository files navigation

FM-Leaderboard-er

Create your own private LLM leaderboard! 📊

Introduction

There's no one-fit-all leaderboard. FM-Leaderboard-er will allow you to find the best LLM for your own business use case based on your own tasks, prompts, and data.

Features:

  1. Tasks - Example notebooks for common tasks like Summarization, Classification, and RAG (coming soon).
  2. Models - Amazon Bedrock, OpenAI, any API (with a code integration).
  3. Metrics - Built-in metrics per task + custom metrics (via a code integration).
  4. Latency - Latency metric per model
  5. Cost - comparison.
  6. Prompt - You could compare several prompts across one model

Getting Started

Prerequisits

  1. AWS account with Amazon Bedrock access to selected models.

Installation

  1. Clone the repository:
    git clone https://github.com/aws-samples/fm-leaderboarder.git
    

Usage

To get started, open the example-1 notebook and follow the instructions provided.

Architecture

Coming soon.

Dependency on third party libraries and services

This code can interact with the OpenAI service which has terms published here and pricing described here. You should be familiar with the pricing and confirm that your use case complies with the terms before proceeding.

This repository makes use of aws/fmeval Foundation Model Evaluations Library. Please review any license terms applicable to the dataset with your legal team and confirm that your use case complies with the terms before proceeding.

Security

See CONTRIBUTING for more information.

Contributing

Contributions to FM-Leaderboarder are welcome! Please refer to the CONTRIBUTING.md file for guidelines on how to contribute.

Contributors

License

This project is licensed under the Apache-2.0 License.

About

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 51.7%
  • Jupyter Notebook 47.4%
  • CSS 0.9%