Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat : Integrated Configuration Center, Registry Component #4

Open
wants to merge 141 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
b196d00
combine logging for kitex and hertz
smx-Morgan Jul 23, 2024
878aba7
combine logging for kitex and hertz V2
smx-Morgan Jul 23, 2024
e1e40e6
combine logging for kitex and hertz V2
smx-Morgan Jul 23, 2024
ae5509f
combine metrics
smx-Morgan Jul 23, 2024
a0fc887
otel V2
smx-Morgan Jul 27, 2024
446d328
logging down
smx-Morgan Jul 28, 2024
4685ebb
logging V2
smx-Morgan Jul 30, 2024
9e71cff
Merge pull request #1 from smx-Morgan/mainV3
smx-Morgan Jul 30, 2024
b3729e2
propagtor V1
smx-Morgan Jul 31, 2024
6597632
propagtor V2
smx-Morgan Aug 1, 2024
89c2ad3
fix
smx-Morgan Aug 1, 2024
f5b2114
可观测组件整合 V2
smx-Morgan Aug 2, 2024
c55b0a3
抽象otel和promth共性逻辑
smx-Morgan Aug 4, 2024
327a8f9
抽象otel和promth共性逻辑 update
smx-Morgan Aug 4, 2024
af34e39
Merge pull request #2 from smx-Morgan/mainV3
smx-Morgan Aug 4, 2024
55b9056
fix bug
smx-Morgan Aug 4, 2024
830ef57
fix Reference Path
smx-Morgan Aug 4, 2024
a0801d1
add promProvider
smx-Morgan Aug 8, 2024
bd21d3f
add labelControl
smx-Morgan Aug 12, 2024
8b63a53
fix bugs
smx-Morgan Aug 12, 2024
0e8ecb7
fix format
smx-Morgan Aug 12, 2024
458b34a
fix CI
smx-Morgan Aug 15, 2024
446e393
fix labelControl name, fix promProvider,promTracer
smx-Morgan Aug 16, 2024
d5dbc22
fix labelControl name, fix promProvider,promTracer
smx-Morgan Aug 16, 2024
9cfcc20
move Measure build to promProvider
smx-Morgan Aug 17, 2024
7b61315
Unified Label Standard
smx-Morgan Aug 19, 2024
7430b81
fix CI
smx-Morgan Aug 19, 2024
2961603
fix CI bugs
smx-Morgan Aug 19, 2024
29621e2
fix CI bugs
smx-Morgan Aug 19, 2024
611aefc
fix CI bug
smx-Morgan Aug 20, 2024
afa535b
fix remove registry from tracer
smx-Morgan Aug 20, 2024
a406cd2
fix CI gofumpt
smx-Morgan Aug 20, 2024
7770544
fix change name to telemetry
smx-Morgan Aug 20, 2024
bbedf5e
fix remove promtracer
smx-Morgan Aug 21, 2024
991e025
fix tracer
smx-Morgan Aug 26, 2024
f1ad26b
fix file_name
smx-Morgan Aug 26, 2024
a9c2581
remove labelcontrol
smx-Morgan Aug 27, 2024
c3889ae
Standardized code
smx-Morgan Aug 30, 2024
e022803
add recorder for retry
smx-Morgan Aug 30, 2024
5a96cf2
update test
smx-Morgan Sep 2, 2024
57a8f2d
fix Data Race
smx-Morgan Sep 2, 2024
0e50950
fix Data Race
smx-Morgan Sep 2, 2024
95901c1
remove promProvider test add example
smx-Morgan Sep 2, 2024
be0e2ea
fix format
smx-Morgan Sep 2, 2024
2358778
move go.mod to telemetry
smx-Morgan Sep 3, 2024
d6cfb97
update makefile
smx-Morgan Sep 3, 2024
0a83f00
update makefile, update file name
smx-Morgan Sep 3, 2024
b6d3a8a
Integrate service discovery components, configuration center
smx-Morgan Sep 3, 2024
88d62bf
Integrate service discovery components, configuration center
smx-Morgan Sep 3, 2024
3de654b
update makefile,update labels
smx-Morgan Sep 4, 2024
1383aba
update pr Request Check
smx-Morgan Sep 4, 2024
3c59ee5
update measure structure
smx-Morgan Sep 7, 2024
971f968
update measure structure
smx-Morgan Sep 8, 2024
f662adb
update formate
smx-Morgan Sep 9, 2024
0eccb48
fix test
smx-Morgan Sep 11, 2024
a7655f8
fix consul and kitexhertz
smx-Morgan Sep 11, 2024
7ea5238
integration config
smx-Morgan Sep 12, 2024
09b99e7
integration config
smx-Morgan Sep 12, 2024
963e743
add go mod to root directory
smx-Morgan Sep 12, 2024
fdedc6f
fix nacos
smx-Morgan Sep 13, 2024
bb83542
fix name, add resolver option to nacos
smx-Morgan Sep 13, 2024
23c28df
feat add resolver_option for servicecomb
smx-Morgan Sep 13, 2024
ced4246
update TelemetryProvider
smx-Morgan Sep 16, 2024
634b01d
update Code Formatting
smx-Morgan Sep 16, 2024
91f7ee3
update build name func
smx-Morgan Sep 16, 2024
7f07ace
Optimize logical structure
smx-Morgan Sep 17, 2024
11f1826
fix nacos weight <= 0 bug
smx-Morgan Sep 19, 2024
54b55b6
Support simultaneous monitoring of HTTP and RPC
smx-Morgan Sep 19, 2024
1685629
update pr from Aug to Sep
smx-Morgan Sep 22, 2024
8d527a0
remove measure form promProvider
smx-Morgan Sep 26, 2024
a3d2728
update labels
smx-Morgan Sep 29, 2024
3ccb78c
otel log component compatible with Kitex
smx-Morgan Sep 30, 2024
ca362bd
Optimize names
smx-Morgan Oct 2, 2024
266ad88
fix header
smx-Morgan Oct 3, 2024
ad4c2db
fix ci
smx-Morgan Oct 3, 2024
fb7777f
fix ci
smx-Morgan Oct 3, 2024
3e8575c
fix pr check
smx-Morgan Oct 3, 2024
235ec84
fix pr check
smx-Morgan Oct 3, 2024
bc37466
fix path
smx-Morgan Oct 3, 2024
cf65cef
rename common
smx-Morgan Oct 8, 2024
9d2204d
replace logging to hlog
smx-Morgan Oct 8, 2024
644816b
Replace logging with hlog
smx-Morgan Oct 8, 2024
d311634
fix
smx-Morgan Oct 8, 2024
0ebc361
fix
smx-Morgan Oct 8, 2024
8fa3353
fix log && Optimize promProvider
smx-Morgan Oct 8, 2024
9c72b1b
update example for promProvider
smx-Morgan Oct 8, 2024
1e6d032
update go work and fix CI
smx-Morgan Oct 8, 2024
42937e7
fix log
smx-Morgan Oct 8, 2024
5697728
Fix duplicate naming for prometheus
smx-Morgan Oct 8, 2024
3580afa
fix ci
smx-Morgan Oct 8, 2024
cd07dd7
update server
smx-Morgan Oct 9, 2024
133c326
update to metadata, support shell
smx-Morgan Oct 11, 2024
0e71ce3
update CI
smx-Morgan Oct 11, 2024
278a6e0
add examples and support shell
smx-Morgan Oct 12, 2024
77c616b
fix bugs remove the Local dependencies
smx-Morgan Oct 12, 2024
b935878
fix CI
smx-Morgan Oct 12, 2024
b6d6c17
fix CI
smx-Morgan Oct 12, 2024
94d3594
rename common
smx-Morgan Oct 14, 2024
e8f946c
support soft connection
smx-Morgan Oct 14, 2024
332f535
support soft connection
smx-Morgan Oct 14, 2024
30d1253
fix mod bug
smx-Morgan Oct 14, 2024
6f2a8d2
fix ci & add counter to otelkitex
smx-Morgan Oct 14, 2024
1dba93f
fix unit test ci
smx-Morgan Oct 14, 2024
4631375
fix unit test ci
smx-Morgan Oct 14, 2024
72bfeb2
fix unit test ci
smx-Morgan Oct 14, 2024
17d2dc7
fix CI
smx-Morgan Oct 14, 2024
cc58949
Merge branch 'main' into master-registry
smx-Morgan Oct 15, 2024
0f55a69
Merge branch 'main' into master-registry
smx-Morgan Oct 15, 2024
e35af73
fix CI
smx-Morgan Oct 15, 2024
5bfe831
update zook test
smx-Morgan Oct 15, 2024
fdd5916
update format
smx-Morgan Oct 15, 2024
b040c9e
fix ci
smx-Morgan Oct 15, 2024
f61af5b
fix ci
smx-Morgan Oct 15, 2024
69d1dee
fix ci
smx-Morgan Oct 15, 2024
6799030
fix ci
smx-Morgan Oct 15, 2024
378d328
fix ci
smx-Morgan Oct 15, 2024
cdc652a
fix ci
smx-Morgan Oct 15, 2024
ec973fd
fix test
smx-Morgan Oct 15, 2024
2cdda8d
fix test
smx-Morgan Oct 15, 2024
1cdde32
fix test
smx-Morgan Oct 15, 2024
fd86ef2
fix test
smx-Morgan Oct 15, 2024
d10fdee
fix services test
smx-Morgan Oct 15, 2024
d56ef7e
fix services test
smx-Morgan Oct 16, 2024
0a2f2cf
Transfer the parser to config component path
smx-Morgan Oct 22, 2024
a84e5e7
change registry version
smx-Morgan Oct 22, 2024
d1b833e
change registry version
smx-Morgan Oct 22, 2024
be768d2
fix nacos_test
smx-Morgan Oct 22, 2024
5990c03
fix nacos_test
smx-Morgan Oct 22, 2024
b5824a5
fix nacos_test
smx-Morgan Oct 22, 2024
f871765
fix nacos_test
smx-Morgan Oct 22, 2024
fd5be0e
fix nacos_test
smx-Morgan Oct 22, 2024
1c26eb3
fix nacos_test
smx-Morgan Oct 22, 2024
e701114
fix nacos_test
smx-Morgan Oct 22, 2024
9a75880
fix nacos_test
smx-Morgan Oct 22, 2024
d890c90
fix
smx-Morgan Oct 22, 2024
78ee80b
fix nacos registry test
smx-Morgan Oct 22, 2024
1f71f86
update zook
smx-Morgan Oct 22, 2024
684ddf8
update nacos registry's name
smx-Morgan Oct 22, 2024
36b591d
update nacos_test registry's name
smx-Morgan Oct 22, 2024
97b4275
fix etcd config
smx-Morgan Oct 22, 2024
a16af31
update file config
smx-Morgan Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 2 additions & 1 deletion .github/workflows/pr-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version: "1.21"


- name: Golangci Lint
# https://golangci-lint.run/
Expand Down
43 changes: 43 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,49 @@ on: [ push, pull_request ]
jobs:
unit-benchmark-test:
runs-on: ubuntu-latest

services:
consul:
image: 'hashicorp/consul:latest'
ports:
- '8500:8500'
polaris:
image: 'polarismesh/polaris-server:latest'
ports:
- '8091:8091'
zookeeper:
image: 'zookeeper:3.7.0'
ports:
- '2181:2181'
servicecomb:
image: 'servicecomb/service-center:latest'
ports:
- '30100:30100'
etcd:
image: bitnami/etcd:latest
ports:
- "2379:2379"
- "2380:2380"
env:
ALLOW_NONE_AUTHENTICATION: yes
ETCD_ADVERTISE_CLIENT_URLS: http://etcd:2379
nacos:
image: 'nacos/nacos-server:2.0.3'
ports:
- '8848:8848'
- '9848:9848'
env:
MODE: standalone
eureka:
image: 'xdockerh/eureka-server:latest'
ports:
- "8761:8761"
redis:
image: redis:latest
ports:
- '6379:6379'


steps:
- uses: actions/checkout@v4
- name: Set up Go
Expand Down
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@
# Dependency directories (remove the comment below to include it)
# vendor/

# Go workspace file
go.work
go.work.sum

# env file
.env

# the result of the go build
output*
output/*
Expand Down
2 changes: 1 addition & 1 deletion .licenserc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ header:
- '**/*.s'
paths-ignore:
- example/prom/promWithkitex/kitex_gen/**

- registry/**/kitex_gen/**
comment: on-failure
292 changes: 292 additions & 0 deletions config/apollo/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,292 @@
# config-apollo (*This is a community driven project*)

[中文](README_CN.md)

Apollo as config centre.

## How to use?

### Basic

#### Server

```go
package main

import (
"context"
"log"
"net"

cwserver "github.com/cloudwego-contrib/cwgo-pkg/config/apollo/server"

"github.com/cloudwego-contrib/cwgo-pkg/config/apollo/apollo"
"github.com/cloudwego-contrib/cwgo-pkg/config/apollo/utils"
"github.com/cloudwego/kitex-examples/kitex_gen/api"
"github.com/cloudwego/kitex-examples/kitex_gen/api/echo"
"github.com/cloudwego/kitex/pkg/klog"
"github.com/cloudwego/kitex/pkg/rpcinfo"
"github.com/cloudwego/kitex/server"
)

// Customed by user
type configLog struct{}

func (cl *configLog) Apply(opt *utils.Options) {
fn := func(cp *apollo.ConfigParam) {
klog.Infof("apollo config %v", cp)
}
opt.ApolloCustomFunctions = append(opt.ApolloCustomFunctions, fn)
}

var _ api.Echo = &EchoImpl{}

// EchoImpl implements the last service interface defined in the IDL.
type EchoImpl struct{}

// Echo implements the Echo interface.
func (s *EchoImpl) Echo(ctx context.Context, req *api.Request) (resp *api.Response, err error) {
klog.Info("echo called")
return &api.Response{Message: req.Message}, nil
}

func main() {
klog.SetLevel(klog.LevelDebug)
apolloClient, err := apollo.NewClient(apollo.Options{})
if err != nil {
panic(err)
}
serviceName := "ServiceName" // server-side service name
addr, err := net.ResolveTCPAddr("tcp", "localhost:8899")
if err != nil {
panic(err)
}

cl := &configLog{}

svr := echo.NewServer(
new(EchoImpl),
server.WithServerBasicInfo(&rpcinfo.EndpointBasicInfo{ServiceName: serviceName}),
server.WithSuite(cwserver.NewSuite(serviceName, apolloClient, cl)),
server.WithServiceAddr(addr),
)
if err := svr.Run(); err != nil {
log.Println("server stopped with cwerror:", err)
} else {
log.Println("server stopped")
}
}
```

#### Client

```go
package main

import (
"context"
"log"
"time"

"github.com/cloudwego-contrib/cwgo-pkg/config/apollo/apollo"
cwclient "github.com/cloudwego-contrib/cwgo-pkg/config/apollo/client"

"github.com/cloudwego/kitex-examples/kitex_gen/api"
"github.com/cloudwego/kitex-examples/kitex_gen/api/echo"
"github.com/cloudwego/kitex/client"
"github.com/cloudwego/kitex/pkg/klog"

"github.com/cloudwego-contrib/cwgo-pkg/config/apollo/utils"
)

// Customed by user
type configLog struct{}

func (cl *configLog) Apply(opt *utils.Options) {
fn := func(cp *apollo.ConfigParam) {
klog.Infof("apollo config %v", cp)
}
opt.ApolloCustomFunctions = append(opt.ApolloCustomFunctions, fn)
}

func main() {
klog.SetLevel(klog.LevelDebug)
apolloClient, err := apollo.NewClient(apollo.Options{})
if err != nil {
panic(err)
}
cl := &configLog{}

serviceName := "ServiceName" // your server-side service name
clientName := "ClientName" // your client-side service name
client, err := echo.NewClient(
serviceName,
client.WithHostPorts("localhost:8899"),
client.WithSuite(cwclient.NewSuite(serviceName, clientName, apolloClient, cl)),
)
if err != nil {
log.Fatal(err)
}
for {
req := &api.Request{Message: "my request"}
resp, err := client.Echo(context.Background(), req)
if err != nil {
klog.Errorf("take request cwerror: %v", err)
} else {
klog.Infof("receive response %v", resp)
}
time.Sleep(time.Second * 10)
}
}
```

### Apollo Configuration

Initialize the client according to the parameters in Options. After establishing the link, the suite will subscribe to the corresponding configuration based on 'AppId', 'NameSpace', 'ServiceName', or 'ClientName' and dynamically update its own policy. Please refer to the following ·Options· variables for specific parameters.

The configured format supports' json 'by default, and can be customized using the function [SetParser] for format parsing. In' NewSuite ', the field function in the instance that implements the' Option 'interface is used to modify the format of the subscription function.
####

#### CustomFunction

Allow users to use instances of custom implementation Option interfaces to customize Apollo parameters

#### Options Variable

| 参数 | 变量默认值 | 作用 |
| :-------------- | :-------------------------------------------: | ------------------------------------------------------------ |
| ConfigServerURL | 127.0.0.1:8080 | apollo config service address |
| AppID | KitexApp | appid of apollo (Uniqueness constraint / Length limit of 32 characters) |
| ClientKeyFormat | {{.ClientServiceName}}.{{.ServerServiceName}} | Using the go [template](https://pkg.go.dev/text/template) syntax to render and generate the corresponding ID, using two metadata: `ClientServiceName` and `ServiceName` (Length limit of 128 characters) |
| ServerKeyFormat | {{.ServerServiceName}} | Using the go [template](https://pkg.go.dev/text/template) Syntax rendering generates corresponding IDs, using 'ServiceName' as a single metadata (Length limit of 128 characters) |
| Cluster | default | Using default values, users can assign values as needed (Length limit of 32 characters) |

#### Governance Policy

> The namespace in the following example uses fixed policy values, with default values for APPID and Cluster. The service name is ServiceName and the client name is ClientName

##### Rate Limit Category=limit
> Currently, current limiting only supports the server side, so ClientServiceName is empty.

[JSON Schema](https://github.com/cloudwego/kitex/blob/develop/pkg/limiter/item_limiter.go#L33)

|Variable|Introduction|
|----|----|
|connection_limit| Maximum concurrent connections |
|qps_limit| Maximum request number every 100ms |
Example:
```json
namespace: `limit`
key: `ServiceName`

{
"connection_limit": 100,
"qps_limit": 2000
}
```

Note:

- The granularity of the current limit configuration is server global, regardless of client or method.
- Not configured or value is 0 means not enabled.
- connection_limit and qps_limit can be configured independently, e.g. connection_limit = 100, qps_limit = 0

##### Retry Policy Category=retry
[JSON Schema](https://github.com/cloudwego/kitex/blob/develop/pkg/retry/policy.go#L63)

|Variable|Introduction|
|----|----|
|type| 0: failure_policy 1: backup_policy|
|failure_policy.backoff_policy| Can only be set one of `fixed` `none` `random` |

Example:
```json
namespace: `retry`
key: `ClientName.ServiceName`
{
"*": {
"enable": true,
"type": 0,
"failure_policy": {
"stop_policy": {
"max_retry_times": 3,
"max_duration_ms": 2000,
"cb_policy": {
"error_rate": 0.3
}
},
"backoff_policy": {
"backoff_type": "fixed",
"cfg_items": {
"fix_ms": 50
}
},
"retry_same_node": false
}
},
"echo": {
"enable": true,
"type": 1,
"backup_policy": {
"retry_delay_ms": 100,
"retry_same_node": false,
"stop_policy": {
"max_retry_times": 2,
"max_duration_ms": 300,
"cb_policy": {
"error_rate": 0.2
}
}
}
}
}
```
Note: retry.Container has built-in support for specifying the default configuration using the `*` wildcard (see the [getRetryer](https://github.com/cloudwego/kitex/blob/v0.5.1/pkg/retry/retryer.go#L240) method for details).

##### RPC Timeout Category=rpc_timeout

[JSON Schema](https://github.com/cloudwego/kitex/blob/develop/pkg/rpctimeout/item_rpc_timeout.go#L42)

Example:
```json
namespace: `rpc_timeout`
key: `ClientName.ServiceName`
{
"*": {
"conn_timeout_ms": 100,
"rpc_timeout_ms": 3000
},
"echo": {
"conn_timeout_ms": 50,
"rpc_timeout_ms": 1000
}
}
```
Note: The circuit breaker implementation of kitex does not currently support changing the global default configuration (see [initServiceCB](https://github.com/cloudwego/kitex/blob/v0.5.1/pkg/circuitbreak/cbsuite.go#L195) for details).

##### Circuit Break: Category=circuit_break

[JSON Schema](https://github.com/cloudwego/kitex/blob/develop/pkg/circuitbreak/item_circuit_breaker.go#L30)

|Variable|Introduction|
|----|----|
|min_sample| Minimum statistical sample number|
Example:
```json
The echo method uses the following configuration (0.3, 100) and other methods use the global default configuration (0.5, 200)
namespace: `circuit_break`
key: `ClientName.ServiceName`
{
"echo": {
"enable": true,
"err_rate": 0.3,
"min_sample": 100
}
}
```
### More Info

Refer to [example](https://github.com/kitex-contrib/examples/config/apollo) for more usage.

maintained by: [xz](https://github.com/XZ0730)

Loading
Loading