Skip to content

Commit

Permalink
Merge branch 'eosphoros-ai:main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
Shen9C authored Dec 10, 2023
2 parents e9f4698 + b8dc9cf commit 9e6167d
Show file tree
Hide file tree
Showing 609 changed files with 3,305 additions and 4,300 deletions.
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ sdist/

var/
wheels/
models/
/models/
# Soft link
models
/models
plugins/

pip-wheel-metadata/
Expand Down
14 changes: 0 additions & 14 deletions .plugin_env.template

This file was deleted.

24 changes: 18 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
</p>


[**简体中文**](README.zh.md) | [**Discord**](https://discord.gg/7uQnPuveTY) | [**Documents**](https://db-gpt.readthedocs.io/en/latest/) | [**Wechat**](https://github.com/eosphoros-ai/DB-GPT/blob/main/README.zh.md#%E8%81%94%E7%B3%BB%E6%88%91%E4%BB%AC) | [**Community**](https://github.com/eosphoros-ai/community)
[**简体中文**](README.zh.md) | [**Discord**](https://discord.gg/7uQnPuveTY) | [**Documents**](https://docs.dbgpt.site) | [**Wechat**](https://github.com/eosphoros-ai/DB-GPT/blob/main/README.zh.md#%E8%81%94%E7%B3%BB%E6%88%91%E4%BB%AC) | [**Community**](https://github.com/eosphoros-ai/community)
</div>

## What is DB-GPT?
Expand Down Expand Up @@ -196,7 +196,7 @@ The core capabilities primarily consist of the following components:
- [ ] Redis
- [ ] MongoDB
- [ ] HBase
- [ ] Doris
- [x] Doris
- [ ] DB2
- [ ] Couchbase
- [ ] Elasticsearch
Expand All @@ -205,9 +205,9 @@ The core capabilities primarily consist of the following components:
- [ ] StarRocks

### Multi-Models And vLLM
- [x] [Cluster Deployment](https://db-gpt.readthedocs.io/en/latest/getting_started/install/cluster/vms/index.html)
- [x] [Cluster Deployment](https://docs.dbgpt.site/docs/installation/model_service/cluster)
- [x] [Fastchat Support](https://github.com/lm-sys/FastChat)
- [x] [vLLM Support](https://db-gpt.readthedocs.io/en/latest/getting_started/install/llm/vllm/vllm.html)
- [x] [vLLM Support](https://docs.dbgpt.site/docs/installation/advanced_usage/vLLM_inference)
- [ ] Cloud-native environment and support for Ray environment
- [ ] Service Registry(eg:nacos)
- [ ] Compatibility with OpenAI's interfaces
Expand All @@ -223,7 +223,7 @@ The core capabilities primarily consist of the following components:
- [ ] Integration of multi-agents and visualization capabilities, defining LLM+Vis new standards

### Cost and Observability
- [x] [debugging](https://db-gpt.readthedocs.io/en/latest/getting_started/observability.html)
- [x] [debugging](https://docs.dbgpt.site/docs/application_manual/advanced_tutorial/debugging)
- [ ] Observability
- [ ] cost & budgets

Expand All @@ -247,7 +247,19 @@ As of October 10, 2023, through the fine-tuning of an open-source model with 13
[More Information about Text2SQL finetune](https://github.com/eosphoros-ai/DB-GPT-Hub)

## Licence

The MIT License (MIT)

## Citation
Please consider citing our project if you find it useful:

```bibtex
@software{db-gpt,
author = {DB-GPT Team},
title = {{DB-GPT}},
url = {https://github.com/eosphoros-ai/DB-GPT},
year = {2023}
}
```

## Contact Information
We are working on building a community, if you have any ideas for building the community, feel free to contact us.
Expand Down
44 changes: 25 additions & 19 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
</a>
</p>

[**English**](README.md)|[**Discord**](https://discord.gg/7uQnPuveTY)|[**文档**](https://db-gpt.readthedocs.io/projects/db-gpt-docs-zh-cn/zh_CN/latest/)|[**微信**](https://github.com/csunny/DB-GPT/blob/main/README.zh.md#%E8%81%94%E7%B3%BB%E6%88%91%E4%BB%AC)|[**社区**](https://github.com/eosphoros-ai/community)
[**English**](README.md)|[**Discord**](https://discord.gg/7uQnPuveTY)|[**文档**](https://www.yuque.com/eosphoros/dbgpt-docs/bex30nsv60ru0fmx)|[**微信**](https://github.com/csunny/DB-GPT/blob/main/README.zh.md#%E8%81%94%E7%B3%BB%E6%88%91%E4%BB%AC)|[**社区**](https://github.com/eosphoros-ai/community)
</div>

## DB-GPT 是什么?
Expand Down Expand Up @@ -59,10 +59,6 @@ DB-GPT是一个开源的数据库领域大模型框架。目的是构建大模
![excel](https://github.com/eosphoros-ai/DB-GPT/assets/13723926/3044e83b-a71e-41fe-a1e2-98e479e0ab59)

#### 根据自然语言对话生成分析图表
<p align="left">
<img src="./assets/chat_excel/chat_excel_6.png" width="800px" />
</p>

<p align="left">
<img src="./assets/dashboard.png" width="800px" />
</p>
Expand All @@ -79,14 +75,14 @@ DB-GPT是一个开源的数据库领域大模型框架。目的是构建大模
- [**源码安装**](https://www.yuque.com/eosphoros/dbgpt-docs/urh3fcx8tu0s9xmb)
- [**Docker安装**](https://www.yuque.com/eosphoros/dbgpt-docs/glf87qg4xxcyrp89)
- [**Docker Compose安装**](https://www.yuque.com/eosphoros/dbgpt-docs/wwdu11e0v5nkfzin)
- [**产品使用手册**](https://www.yuque.com/eosphoros/dbgpt-docs/tkspdd0tcy2vlnu4)
- [**数据对话**](https://www.yuque.com/eosphoros/dbgpt-docs/gd9hbhi1dextqgbz)
- [**使用手册**](https://www.yuque.com/eosphoros/dbgpt-docs/tkspdd0tcy2vlnu4)
- [**知识库**](https://www.yuque.com/eosphoros/dbgpt-docs/ycyz3d9b62fccqxh)
- [**ChatExcel**](https://www.yuque.com/eosphoros/dbgpt-docs/prugoype0xd2g4bb)
- [**数据对话**](https://www.yuque.com/eosphoros/dbgpt-docs/gd9hbhi1dextqgbz)
- [**Excel对话**](https://www.yuque.com/eosphoros/dbgpt-docs/prugoype0xd2g4bb)
- [**数据库对话**](https://www.yuque.com/eosphoros/dbgpt-docs/wswpv3zcm2c9snmg)
- [**报表分析**](https://www.yuque.com/eosphoros/dbgpt-docs/vsv49p33eg4p5xc1)
- [**插件**](https://www.yuque.com/eosphoros/dbgpt-docs/pom41m7oqtdd57hm)
- [**如何部署模型服务**](https://www.yuque.com/eosphoros/dbgpt-docs/vubxiv9cqed5mc6o)
- [**模型服务部署**](https://www.yuque.com/eosphoros/dbgpt-docs/vubxiv9cqed5mc6o)
- [**单机部署**](https://www.yuque.com/eosphoros/dbgpt-docs/kwg1ed88lu5fgawb)
- [**集群部署**](https://www.yuque.com/eosphoros/dbgpt-docs/gmbp9619ytyn2v1s)
- [**vLLM**](https://www.yuque.com/eosphoros/dbgpt-docs/bhy9igdvanx1uluf)
Expand All @@ -106,7 +102,7 @@ DB-GPT是一个开源的数据库领域大模型框架。目的是构建大模

围绕大语言模型、Text2SQL数据集、LoRA/QLoRA/Pturning等微调方法构建的自动化微调轻量框架, 让TextSQL微调像流水线一样方便。详见: [DB-GPT-Hub](https://github.com/eosphoros-ai/DB-GPT-Hub)

- **Data-Driven Multi-Agents&Plugins**
- **数据驱动的Agents插件**

支持自定义插件执行任务,原生支持Auto-GPT插件模型,Agents协议采用Agent Protocol标准

Expand Down Expand Up @@ -145,7 +141,7 @@ DB-GPT是一个开源的数据库领域大模型框架。目的是构建大模

- **GBI**:生成式BI是DB-GPT项目的核心能力之一,为构建企业报表分析、业务洞察提供基础的数智化技术保障。

- **Fine-tune框架**: 模型微调是任何一个企业在垂直、细分领域落地不可或缺的能力,DB-GPT提供了完整的微调框架,实现与DB-GPT项目的无缝打通,在最近的微调中,基于spider的准确率已经做到了82.5%
- **微调框架**: 模型微调是任何一个企业在垂直、细分领域落地不可或缺的能力,DB-GPT提供了完整的微调框架,实现与DB-GPT项目的无缝打通,在最近的微调中,基于spider的准确率已经做到了82.5%

- **数据驱动的Multi-Agents框架**: DB-GPT提供了数据驱动的自进化微调框架,目标是可以持续基于数据做决策与执行。

Expand All @@ -167,8 +163,7 @@ DB-GPT是一个开源的数据库领域大模型框架。目的是构建大模

🌐 [AutoDL镜像](https://www.codewithgpu.com/i/eosphoros-ai/DB-GPT/dbgpt)

🌐 [阿里云镜像](http://dbgpt.site/web/#/p/dc4bb97e0bc15302dbf3a5d5571142dd)

🌐 [小程序云部署](https://www.yuque.com/eosphoros/dbgpt-docs/ek12ly8k661tbyn8)

### 多语言切换

Expand All @@ -178,7 +173,7 @@ DB-GPT是一个开源的数据库领域大模型框架。目的是构建大模

### 多模型使用

[使用指南](https://db-gpt.readthedocs.io/projects/db-gpt-docs-zh-cn/zh_CN/latest/modules/llms.html)
[使用指南](https://www.yuque.com/eosphoros/dbgpt-docs/huzgcf2abzvqy8uv)

# 贡献
> 提交代码前请先执行 `black .`
Expand Down Expand Up @@ -234,7 +229,7 @@ The MIT License (MIT)
- [ ] Redis
- [ ] MongoDB
- [ ] HBase
- [ ] Doris
- [x] Doris
- [ ] DB2
- [ ] Couchbase
- [ ] Elasticsearch
Expand All @@ -243,13 +238,12 @@ The MIT License (MIT)


### 多模型管理与推理优化
- [x] [集群部署](https://db-gpt.readthedocs.io/en/latest/getting_started/install/cluster/vms/index.html)
- [x] [集群部署](https://www.yuque.com/eosphoros/dbgpt-docs/gmbp9619ytyn2v1s)
- [x] [fastchat支持](https://github.com/lm-sys/FastChat)
- [x] [fastchat支持](https://github.com/lm-sys/FastChat)
- [x] [vLLM 支持](https://db-gpt.readthedocs.io/en/latest/getting_started/install/llm/vllm/vllm.html)
- [x] [vLLM 支持](https://www.yuque.com/eosphoros/dbgpt-docs/bhy9igdvanx1uluf)
- [x] 上层接口兼容Openai
- [ ] 云原生环境与Ray环境支持
- [ ] 注册中心引入nacos
- [ ] 上层接口兼容Openai
- [ ] Embedding模型扩充,优化

### Agents与插件市场
Expand Down Expand Up @@ -292,6 +286,18 @@ The MIT License (MIT)

[More Information about Text2SQL finetune](https://github.com/eosphoros-ai/DB-GPT-Hub)

## 引用
如果您觉得我们的项目有用,请考虑引用我们的项目:

```bibtex
@software{db-gpt,
author = {DB-GPT Team},
title = {{DB-GPT}},
url = {https://github.com/eosphoros-ai/DB-GPT},
year = {2023}
}
```

## 联系我们

<p align="center">
Expand Down
Binary file removed assets/DB-GPT-Product.jpg
Binary file not shown.
Binary file removed assets/chatSQL.png
Binary file not shown.
Binary file removed assets/chat_agent/agent_hub_1.png
Binary file not shown.
Binary file removed assets/chat_agent/agent_hub_2.png
Binary file not shown.
Binary file removed assets/chat_agent/chat_agent0.png
Binary file not shown.
Binary file removed assets/chat_agent/chat_agent1.png
Binary file not shown.
Binary file removed assets/chat_agent/chat_agent2.png
Binary file not shown.
Binary file removed assets/chat_dashboard/chat_dashboard_1.png
Binary file not shown.
Binary file removed assets/chat_dashboard/chat_dashboard_2.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_1.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_2.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_3.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_4.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_5.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_6.png
Binary file not shown.
Binary file removed assets/chat_excel/chat_excel_7.png
Binary file not shown.
Binary file removed assets/chat_knowledge.png
Binary file not shown.
Binary file removed assets/chat_knowledge_zh.png
Binary file not shown.
Binary file removed assets/chatdata.png
Binary file not shown.
Binary file removed assets/chatdb.png
Binary file not shown.
Binary file removed assets/ks.png
Binary file not shown.
Binary file removed assets/llm/dbgpt_llm.jpg
Binary file not shown.
Binary file modified assets/wechat.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions dbgpt/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
from dbgpt.component import SystemApp, BaseComponent


__ALL__ = ["SystemApp", "BaseComponent"]

_CORE_LIBS = ["core", "rag", "model", "agent", "datasource", "vis", "storage", "train"]
_SERVE_LIBS = ["serve"]
_LIBS = _CORE_LIBS + _SERVE_LIBS


def __getattr__(name: str):
# Lazy load
import importlib

if name in _LIBS:
return importlib.import_module("." + name, __name__)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
4 changes: 4 additions & 0 deletions dbgpt/_private/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
"""This is a private module.
You should not import anything from this module.
"""
4 changes: 1 addition & 3 deletions pilot/common/chat_util.py → dbgpt/_private/chat_util.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
import asyncio
from typing import Coroutine, List, Any

from starlette.responses import StreamingResponse

from pilot.scene.base_chat import BaseChat
from pilot.scene.chat_factory import ChatFactory
from dbgpt.app.scene import BaseChat, ChatFactory

chat_factory = ChatFactory()

Expand Down
6 changes: 3 additions & 3 deletions pilot/configs/config.py → dbgpt/_private/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
import os
from typing import List, Optional, TYPE_CHECKING

from pilot.singleton import Singleton
from dbgpt.util.singleton import Singleton

if TYPE_CHECKING:
from auto_gpt_plugin_template import AutoGPTPluginTemplate
from pilot.component import SystemApp
from dbgpt.component import SystemApp


class Config(metaclass=Singleton):
Expand Down Expand Up @@ -120,7 +120,7 @@ def __init__(self) -> None:
)
self.speak_mode = False

from pilot.prompts.prompt_registry import PromptTemplateRegistry
from dbgpt.core._private.prompt_registry import PromptTemplateRegistry

self.prompt_template_registry = PromptTemplateRegistry()
### Related configuration of built-in commands
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from pydantic import Field, BaseModel
from dbgpt._private.pydantic import Field, BaseModel

DEFAULT_CONTEXT_WINDOW = 3900
DEFAULT_NUM_OUTPUTS = 256
Expand Down
33 changes: 33 additions & 0 deletions dbgpt/_private/pydantic.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import pydantic

if pydantic.VERSION.startswith("1."):
PYDANTIC_VERSION = 1
from pydantic import (
BaseModel,
Extra,
Field,
NonNegativeFloat,
NonNegativeInt,
PositiveFloat,
PositiveInt,
ValidationError,
root_validator,
validator,
PrivateAttr,
)
else:
PYDANTIC_VERSION = 2
# pydantic 2.x
from pydantic.v1 import (
BaseModel,
Extra,
Field,
NonNegativeFloat,
NonNegativeInt,
PositiveFloat,
PositiveInt,
ValidationError,
root_validator,
validator,
PrivateAttr,
)
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@

import requests

from pilot.base_modules.agent.commands.command_mange import command
from pilot.configs.config import Config
from dbgpt.agent.commands.command_mange import command
from dbgpt._private.config import Config

CFG = Config()

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
import requests
from PIL import Image

from pilot.base_modules.agent.commands.command_mange import command
from pilot.configs.config import Config
from dbgpt.agent.commands.command_mange import command
from dbgpt._private.config import Config

logger = logging.getLogger(__name__)
CFG = Config()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from .exception_not_commands import NotCommands
from .generator import PluginPromptGenerator

from pilot.configs.config import Config
from dbgpt._private.config import Config


def _resolve_pathlike_command_args(command_args):
Expand Down Expand Up @@ -36,7 +36,7 @@ def execute_ai_response_json(
Returns:
"""
from pilot.speech.say import say_text
from dbgpt.util.speech.say import say_text

cfg = Config()

Expand Down
Original file line number Diff line number Diff line change
@@ -1,20 +1,17 @@
import functools
import importlib
import inspect
import time
import json
import logging
import xml.etree.ElementTree as ET
import pandas as pd

from pilot.common.json_utils import serialize
from dbgpt.util.json_utils import serialize
from datetime import datetime
from typing import Any, Callable, Optional, List
from pydantic import BaseModel
from pilot.base_modules.agent.common.schema import Status, ApiTagType
from pilot.base_modules.agent.commands.command import execute_command
from pilot.base_modules.agent.commands.generator import PluginPromptGenerator
from pilot.common.string_utils import extract_content_open_ending, extract_content
from dbgpt._private.pydantic import BaseModel
from dbgpt.agent.common.schema import Status
from dbgpt.agent.commands.command import execute_command
from dbgpt.util.string_utils import extract_content_open_ending, extract_content

# Unique identifier for auto-gpt commands
AUTO_GPT_COMMAND_IDENTIFIER = "auto_gpt_command"
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from pandas import DataFrame

from pilot.base_modules.agent.commands.command_mange import command
from dbgpt.agent.commands.command_mange import command
import pandas as pd
import uuid
import os
Expand All @@ -11,14 +11,15 @@
import matplotlib.pyplot as plt
import matplotlib.ticker as mtick
from matplotlib.font_manager import FontManager
from pilot.common.string_utils import is_scientific_notation
from dbgpt.util.string_utils import is_scientific_notation
from dbgpt.configs.model_config import PILOT_PATH

import logging

logger = logging.getLogger(__name__)


static_message_img_path = os.path.join(os.getcwd(), "message/img")
static_message_img_path = os.path.join(PILOT_PATH, "message/img")


def data_pre_classification(df: DataFrame):
Expand Down
Loading

0 comments on commit 9e6167d

Please sign in to comment.