Skip to content

Commit

Permalink
Update to v0.2 (#2)
Browse files Browse the repository at this point in the history
* update go version and dependencies

* bump version

* get full data for threshold based policies without aggregation to avoid internal server error

* Specify supported condition types in README

* fix typo

* update deps

* update dependencies

* clarify execution period in calculation comments

* move link in README

* update deps

* update Go version and GitHub action

* update deps

* update actions

* update go version

* update default duration to 12h

* update readme

* update deps

* add summary flag and update readme

* update goreleaser

* update readme
  • Loading branch information
hanneshayashi authored Nov 28, 2024
1 parent f189fec commit 22242ec
Show file tree
Hide file tree
Showing 6 changed files with 126 additions and 101 deletions.
10 changes: 5 additions & 5 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,21 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4.1.7
uses: actions/checkout@v4.2.2
- name: Unshallow
run: git fetch --prune --unshallow
- name: Set up Go
uses: actions/setup-go@v5.0.2
uses: actions/setup-go@v5.1.0
with:
go-version: 1.23
go-version: 1.23.3
- name: Import GPG key
id: import_gpg
uses: crazy-max/ghaction-import-gpg@v6.1.0
uses: crazy-max/ghaction-import-gpg@v6.2.0
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.PASSPHRASE }}
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6.0.0
uses: goreleaser/goreleaser-action@v6.1.0
with:
version: latest
args: release --clean
Expand Down
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# IMPORTANT NOTE

Google has postponed the launch to April 2026 and has implemented a preview function of the estimated cost when editing an alerting policy. This estimate is considered much more accurate because `appe` can currently only use the total number of time series in the estimation period (30 days). In reality, this may fluctuate causing `appe` to produce an over-estimation.
As of this time, Google has not made the underlying data for their calculation available via any metric or API. If this changes in the future, `appe` will be updated to produce more accurate results.
Google has postponed the launch to April 2026 and has implemented a preview function of the estimated cost when editing an alerting policy. This estimate is uses a very similar calculation to `appe` but only looks at the last few minutes / hours (the exact value is not known and seems to vary slightly). This will generally result in lower estimates than `appe` but you can change the length of the window `appe` goes back using the `-d` flag (eg. `-d 5m` or `-d 4h`) to get the same or similar results. In order to more closely match the results of the preview functionality, `appe`'s default has been changed to `12h` for now. This will likely change again if or when Google updates the built-in preview functionality again.
Please note that both the preview function and `appe`'s default are not 100% accurate.

# appe - Alerting Policy Price Estimator

Expand Down Expand Up @@ -36,8 +36,6 @@ We recommend that you assign the following two roles for full compatibility:
## Installation
The easiest way to get and install `appe` is to download one of the pre-compiled binaries from the [releases](https://github.com/doitintl/gcp-tool-appe/releases). `appe` is a self-contained binary without any dependencies and can be run from anywhere. You do not need to download any runtime and there is no need for an installer.

See also https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev.

### macOS
Since the binary is not signed with an Apple Developer Certificate, your Mac will likely report it as untrustworthy.
There are two ways to deal with this:
Expand All @@ -62,6 +60,8 @@ If you are running `appe` locally, the easiest way to set up ADC is to use [gclo
gcloud auth application-default login
```

See also https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev.

## Usage
Using `appe` is fairly straightforward

Expand Down Expand Up @@ -107,10 +107,16 @@ You can also specify multiple organizations:
```
Note that you will need to specify the `--recursive` or `-r` flag to also scan subfolders.

### Supported Condition Types
The following condition types are supported by `appe`:
- Monitoring Query Language (MQL) via [projects.timeSeries.query](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/query)
- Threshold and Absence via [projects.timeSeries.list](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/list)
- Prometheus Query Language (PromQL / PQL) via [projects.location.prometheus.api.v1.query_range](https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.location.prometheus.api.v1/query_range)

### All Flags
```
-c, --csvOut string Path to a CSV file to redirect output to. If this is not set, human-readable output will be given on stdout.
-d, --duration duration The delta from now to go back in time for query. Default is 30 days. (default 720h0m0s)
-d, --duration duration The delta from now to go back in time for query. Default is 12 hours. (default 12h0m0s)
-e, --excludeFolder strings One or more folders to exclude. Separated by ",".
-f, --folder strings One or more folders to scan. Use the "-r" flag to scan recursively. Separated by ",".
-h, --help help for appe
Expand All @@ -120,6 +126,7 @@ Note that you will need to specify the `--recursive` or `-r` flag to also scan s
-p, --project strings One or more projects to scan. Separated by ",".
-q, --quotaProject string A quota or billing project. Useful if you don't have the serviceusage.services.use permission in the target project.
-r, --recursive If parent should be scanned recursively. If this is not set, only projects at the root of the folder or organization will be scanned. (default false)
-s, --summary Whether the output should just be a summary (sum of all scanned policies) (default false)
-t, --testPermissions If the application should verify that the user has the necessary permissions before processing a project. (default false)
--threads int Number of threads to use to process folders, projects and policies in parallel. (default 4)
-v, --version version for appe
Expand Down
8 changes: 4 additions & 4 deletions cmd/monitoring.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ func processAlertPolicy(
policyOut.Error = err.Error()
break
}
// 60 (seconds) * 60 (minutes) * 24 (hours) * 30 (days) / 30 (step) * 0.35 (price) / 1000000 (per 1M) = 0.03024 (price per time series)
// 60 (seconds) * 60 (minutes) * 24 (hours) * 30 (days) / 30 (execution period) * 0.35 (price) / 1000000 (per 1M) = 0.03024 (price per time series)
policyOut.Price += 0.03024
policyOut.TimeSeries++
}
Expand Down Expand Up @@ -188,7 +188,7 @@ func processAlertPolicy(
}
// 60 (seconds) * 60 (minutes) * 24 (hours) * 30 (days) = 2592000
// 2592000 * 0.35 (price) / 1000000 (per 1M) =
// 0.9072 / step * time series = price for all time series with this condition
// 0.9072 / execution period * time series = price for all time series with this condition
policyOut.Price += 0.9072 / float64(seconds) * float64(len(pqlResp.Data.Result))
policyOut.TimeSeries += len(pqlResp.Data.Result)
}
Expand Down Expand Up @@ -216,7 +216,7 @@ func processAlertPolicy(
if len(aggregations) > 1 {
tsReq.SecondaryAggregation = aggregations[1]
}
if tsReq.Aggregation.GetCrossSeriesReducer().String() == "REDUCE_COUNT_FALSE" || tsReq.SecondaryAggregation.GetCrossSeriesReducer().String() == "REDUCE_COUNT_FALSE" {
if tsReq.Aggregation == nil || tsReq.Aggregation.GetCrossSeriesReducer().String() == "REDUCE_COUNT_FALSE" || tsReq.SecondaryAggregation.GetCrossSeriesReducer().String() == "REDUCE_COUNT_FALSE" {
tsReq.View = monitoringpb.ListTimeSeriesRequest_FULL
}
tsIt := metricClient.ListTimeSeries(ctx, tsReq)
Expand All @@ -229,7 +229,7 @@ func processAlertPolicy(
policyOut.Error = err.Error()
break
}
// 60 (seconds) * 60 (minutes) * 24 (hours) * 30 (days) / 30 (step) * 0.35 (price) / 1000000 (per 1M) = 0.03024 (price per time series)
// 60 (seconds) * 60 (minutes) * 24 (hours) * 30 (days) / 30 (execution period) * 0.35 (price) / 1000000 (per 1M) = 0.03024 (price per time series)
policyOut.Price += 0.03024
policyOut.TimeSeries++
}
Expand Down
24 changes: 21 additions & 3 deletions cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import (

// rootCmd represents the base command when called without any subcommands
var rootCmd = &cobra.Command{
Version: "0.1",
Version: "0.2",
Use: "appe",
Short: "Alerting Policy Price Estimator",
Long: `Scans for alerting policies in the specified projects, folder or orgs and approximates their cost by executing the queries defined in them against the monitoring API`,
Expand Down Expand Up @@ -90,6 +90,10 @@ func Execute() {
if err != nil {
log.Fatalln(err)
}
summary, err := rootCmd.Flags().GetBool("summary")
if err != nil {
log.Fatalln(err)
}
quotaProject, err := rootCmd.Flags().GetString("quotaProject")
if err != nil {
log.Fatalln(err)
Expand Down Expand Up @@ -182,7 +186,7 @@ func Execute() {

// If one or more individual policies should be analyzed, we need to first get them from the API.
// We then put them directly on the policiesIn channel, which will be processes by threads that are spawned below.
// Finally, we will close teh projectsIn channel once done, because the policiesIn channel will be closed automatically.
// Finally, we will close the projectsIn channel once done, because the policiesIn channel will be closed automatically.
if lenPol > 0 {
if lenPol > int(threads) {
threads = int64(lenPol)
Expand Down Expand Up @@ -274,6 +278,18 @@ func Execute() {
csvWriter.Flush()
}
// Otherwise, the application will just output to stdout
} else if summary {
policiesSum := 0
conditionsSum := 0
timeSeriesSum := 0
priceSum := 0.0
for policy := range policiesOut {
policiesSum++
conditionsSum += policy.Conditions
timeSeriesSum += policy.TimeSeries
priceSum += policy.Price
}
log.Printf("Summary: You have %d policies with a combined total of %d conditions and %d time series. It will cost approximately $%f\n", policiesSum, conditionsSum, timeSeriesSum, priceSum)
} else {
for policy := range policiesOut {
log.Printf("Alerting Policy %s (%s) has %d condition(s) and %d time series. It will cost approximately $%f\n", policy.DisplayName, policy.Name, policy.Conditions, policy.TimeSeries, policy.Price)
Expand All @@ -291,13 +307,15 @@ func init() {
rootCmd.Flags().StringSliceP("excludeFolder", "e", nil, "One or more folders to exclude. Separated by \",\".")
rootCmd.Flags().BoolP("testPermissions", "t", false, "If the application should verify that the user has the necessary permissions before processing a project. (default false)")
rootCmd.Flags().BoolP("includeDisabled", "i", false, "If the application should also include disabled policies. (default false)")
rootCmd.Flags().BoolP("summary", "s", false, "Whether the output should just be a summary (sum of all scanned policies) (default false)")
rootCmd.Flags().BoolP("recursive", "r", false, "If parent should be scanned recursively. If this is not set, only projects at the root of the folder or organization will be scanned. (default false)")
rootCmd.Flags().Int64("threads", 4, "Number of threads to use to process folders, projects and policies in parallel.")
rootCmd.Flags().DurationP("duration", "d", 24*time.Hour*30, "The delta from now to go back in time for query. Default is 30 days.")
rootCmd.Flags().DurationP("duration", "d", 12*time.Hour, "The delta from now to go back in time for query. Default is 12 hours.")
rootCmd.MarkFlagsOneRequired("policy", "project", "folder", "organization")
rootCmd.MarkFlagsMutuallyExclusive("policy", "project", "recursive")
rootCmd.MarkFlagsMutuallyExclusive("policy", "testPermissions")
rootCmd.MarkFlagsMutuallyExclusive("policy", "includeDisabled")
rootCmd.MarkFlagsMutuallyExclusive("policy", "project", "excludeFolder")
rootCmd.MarkFlagsMutuallyExclusive("policy", "project", "folder", "organization")
rootCmd.MarkFlagsMutuallyExclusive("csvOut", "summary")
}
56 changes: 28 additions & 28 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,47 +3,47 @@ module github.com/doitintl/gcp-tool-appe
go 1.23

require (
cloud.google.com/go/iam v1.2.0
cloud.google.com/go/monitoring v1.21.0
cloud.google.com/go/resourcemanager v1.10.0
cloud.google.com/go/iam v1.2.2
cloud.google.com/go/monitoring v1.21.2
cloud.google.com/go/resourcemanager v1.10.2
github.com/spf13/cobra v1.8.1
google.golang.org/api v0.194.0
google.golang.org/protobuf v1.34.2
google.golang.org/api v0.209.0
google.golang.org/protobuf v1.35.2
)

require (
cloud.google.com/go v0.115.1 // indirect
cloud.google.com/go/auth v0.9.1 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
cloud.google.com/go/compute/metadata v0.5.0 // indirect
cloud.google.com/go/longrunning v0.6.0 // indirect
cloud.google.com/go v0.116.0 // indirect
cloud.google.com/go/auth v0.11.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
cloud.google.com/go/compute/metadata v0.5.2 // indirect
cloud.google.com/go/longrunning v0.6.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.13.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/spf13/pflag v1.0.5 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 // indirect
go.opentelemetry.io/otel v1.28.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/net v0.28.0 // indirect
golang.org/x/oauth2 v0.22.0 // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/sys v0.24.0 // indirect
golang.org/x/text v0.17.0 // indirect
golang.org/x/time v0.6.0 // indirect
google.golang.org/genproto v0.0.0-20240822170219-fc7c04adadcd // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240822170219-fc7c04adadcd // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240822170219-fc7c04adadcd // indirect
google.golang.org/grpc v1.65.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.57.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.57.0 // indirect
go.opentelemetry.io/otel v1.32.0 // indirect
go.opentelemetry.io/otel/metric v1.32.0 // indirect
go.opentelemetry.io/otel/trace v1.32.0 // indirect
golang.org/x/crypto v0.29.0 // indirect
golang.org/x/net v0.31.0 // indirect
golang.org/x/oauth2 v0.24.0 // indirect
golang.org/x/sync v0.9.0 // indirect
golang.org/x/sys v0.27.0 // indirect
golang.org/x/text v0.20.0 // indirect
golang.org/x/time v0.8.0 // indirect
google.golang.org/genproto v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/grpc v1.68.0 // indirect
)
Loading

0 comments on commit 22242ec

Please sign in to comment.