Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: replace with faster JSON encoding #1871

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion cmd/zb/helper.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package main

import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
Expand All @@ -15,6 +14,7 @@ import (
"time"

"github.com/google/uuid"
jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
imeta "github.com/opencontainers/image-spec/specs-go"
ispec "github.com/opencontainers/image-spec/specs-go/v1"
Expand All @@ -24,6 +24,8 @@ import (
"zotregistry.io/zot/pkg/common"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

func makeHTTPGetRequest(url string, resultPtr interface{}, client *resty.Client) error {
resp, err := client.R().Get(url)
if err != nil {
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -514,3 +514,5 @@ require (
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)

replace github.com/aquasecurity/trivy v0.45.1 => github.com/project-zot/trivy v0.45.1-disable-analyzers
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -396,8 +396,6 @@ github.com/aquasecurity/table v1.8.0/go.mod h1:eqOmvjjB7AhXFgFqpJUEE/ietg7RrMSJZ
github.com/aquasecurity/testdocker v0.0.0-20230111101738-e741bda259da h1:pj/adfN0Wbzc0H8YkI1nX5K92wOU5/1/1TRuuc0y5Nw=
github.com/aquasecurity/tml v0.6.1 h1:y2ZlGSfrhnn7t4ZJ/0rotuH+v5Jgv6BDDO5jB6A9gwo=
github.com/aquasecurity/tml v0.6.1/go.mod h1:OnYMWY5lvI9ejU7yH9LCberWaaTBW7hBFsITiIMY2yY=
github.com/aquasecurity/trivy v0.45.1 h1:JjkrawgNpVUV6mxtFX635I3MhzDqmGkze46SnygkYN0=
github.com/aquasecurity/trivy v0.45.1/go.mod h1:3cawI6q9o32pPGXhuGEIWWwUCSMAzRk/FhsEdA4eW1k=
github.com/aquasecurity/trivy-db v0.0.0-20230831170347-f732860d4917 h1:MQd7h7yUyA8UlUzhjNMzpUX0NpD7jfxmRfSKwp/Ji3E=
github.com/aquasecurity/trivy-db v0.0.0-20230831170347-f732860d4917/go.mod h1:WJ5Qnk5ZNGWvks07GOZe2IOsuXrPfSC5c8hYGOGfrsU=
github.com/aquasecurity/trivy-java-db v0.0.0-20230209231723-7cddb1406728 h1:0eS+V7SXHgqoT99tV1mtMW6HL4HdoB9qGLMCb1fZp8A=
Expand Down Expand Up @@ -1438,6 +1436,8 @@ github.com/proglottis/gpgme v0.1.3 h1:Crxx0oz4LKB3QXc5Ea0J19K/3ICfy3ftr5exgUK1AU
github.com/proglottis/gpgme v0.1.3/go.mod h1:fPbW/EZ0LvwQtH8Hy7eixhp1eF3G39dtx7GUN+0Gmy0=
github.com/project-zot/mockoidc v0.0.0-20230307111146-f607b4b5fb97 h1:V6z9y0Yx2sQs4WSKx79mgkKJWwjbu/lHQg1yza5bmQE=
github.com/project-zot/mockoidc v0.0.0-20230307111146-f607b4b5fb97/go.mod h1:46X30UrCsiwicZcg5L098Pyilaj94AO39mvS5PEyPn8=
github.com/project-zot/trivy v0.45.1-disable-analyzers h1:mrCqhX8rbYdM8WUXW6xledo5kZ0kVHiUnGkAMeniYcQ=
github.com/project-zot/trivy v0.45.1-disable-analyzers/go.mod h1:3cawI6q9o32pPGXhuGEIWWwUCSMAzRk/FhsEdA4eW1k=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
Expand Down
3 changes: 2 additions & 1 deletion golangcilint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ linters-settings:
checks: argument,case,condition,operation,return,assign
ignored-numbers: 10,64
gomoddirectives:
replace-allow-list: []
replace-allow-list:
- github.com/aquasecurity/trivy
issues:
exclude-rules:
- path: pkg/extensions/search/schema.resolvers.go
Expand Down
4 changes: 3 additions & 1 deletion pkg/api/config/config.go
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
package config

import (
"encoding/json"
"os"
"time"

jsoniter "github.com/json-iterator/go"
distspec "github.com/opencontainers/distribution-spec/specs-go"

extconf "zotregistry.io/zot/pkg/extensions/config"
storageConstants "zotregistry.io/zot/pkg/storage/constants"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

var (
Commit string //nolint: gochecknoglobals
ReleaseTag string //nolint: gochecknoglobals
Expand Down
3 changes: 2 additions & 1 deletion pkg/api/routes.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ package api

import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
Expand Down Expand Up @@ -53,6 +52,8 @@ import (
"zotregistry.io/zot/pkg/test/inject"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

type RouteHandler struct {
c *Controller
}
Expand Down
4 changes: 3 additions & 1 deletion pkg/cli/client/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
Expand All @@ -16,13 +15,16 @@
"sync"
"time"

jsoniter "github.com/json-iterator/go"
ispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/sigstore/cosign/v2/pkg/oci/remote"

zerr "zotregistry.io/zot/errors"
"zotregistry.io/zot/pkg/common"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

Check failure on line 26 in pkg/cli/client/client.go

View workflow job for this annotation

GitHub Actions / Run zot with extensions tests

json already declared through import of package json ("encoding/json")

Check failure on line 26 in pkg/cli/client/client.go

View workflow job for this annotation

GitHub Actions / Run zot with extensions tests

json already declared through import of package json ("encoding/json")

Check failure on line 26 in pkg/cli/client/client.go

View workflow job for this annotation

GitHub Actions / lint

json already declared through import of package json ("encoding/json")

Check failure on line 26 in pkg/cli/client/client.go

View workflow job for this annotation

GitHub Actions / lint

json already declared through import of package json ("encoding/json")

Check failure on line 26 in pkg/cli/client/client.go

View workflow job for this annotation

GitHub Actions / lint

json already declared through import of package json ("encoding/json")

Check failure on line 26 in pkg/cli/client/client.go

View workflow job for this annotation

GitHub Actions / lint

json already declared through import of package json ("encoding/json")

var (
httpClientsMap = make(map[string]*http.Client) //nolint: gochecknoglobals
httpClientLock sync.Mutex //nolint: gochecknoglobals
Expand Down
2 changes: 1 addition & 1 deletion pkg/cli/server/extensions_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1031,7 +1031,7 @@ func TestServeSearchEnabledCVE(t *testing.T) {

// The default config handling logic will convert the 1h interval to a 2h interval
substring := "\"Search\":{\"Enable\":true,\"CVE\":{\"UpdateInterval\":7200000000000,\"Trivy\":" +
"{\"DBRepository\":\"ghcr.io/aquasecurity/trivy-db\",\"JavaDBRepository\":\"ghcr.io/aquasecurity/trivy-java-db\"}}}"
"{\"DBRepository\":\"ghcr.io/aquasecurity/trivy-db\",\"JavaDBRepository\":\"\"}}}"

found, err := ReadLogFileAndSearchString(logPath, substring, readLogFileTimeout)

Expand Down
7 changes: 4 additions & 3 deletions pkg/cli/server/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -551,9 +551,10 @@ func applyDefaultValues(config *config.Config, viperInstance *viper.Viper, log z

if config.Extensions.Search.CVE.Trivy.JavaDBRepository == "" {
defaultJavaDBDownloadURL := "ghcr.io/aquasecurity/trivy-java-db"
log.Info().Str("trivyJavaDownloadURL", defaultJavaDBDownloadURL).
Msg("Config: using default Trivy Java DB download URL.")
config.Extensions.Search.CVE.Trivy.JavaDBRepository = defaultJavaDBDownloadURL
log.Info().Str("trivyJavaDownloadURL", "").
Str("suggestedTrivyJavaDownloadURL", defaultJavaDBDownloadURL).
Msg("Config: Trivy DB javaDBRepository configuration key was not provided, " +
"will skip scanning java packages")
}
}
}
Expand Down
5 changes: 4 additions & 1 deletion pkg/common/common.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
package common

import (
"encoding/json"
"errors"
"fmt"
"io/fs"
Expand All @@ -10,8 +9,12 @@ import (
"syscall"
"time"
"unicode/utf8"

jsoniter "github.com/json-iterator/go"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

const (
httpTimeout = 5 * time.Minute
certsPath = "/etc/containers/certs.d"
Expand Down
1 change: 0 additions & 1 deletion pkg/common/http_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"io"
"net/http"
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/extension_mgmt.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,19 @@
package extensions

import (
"encoding/json"
"net/http"

"github.com/gorilla/mux"
jsoniter "github.com/json-iterator/go"

"zotregistry.io/zot/pkg/api/config"
"zotregistry.io/zot/pkg/api/constants"
zcommon "zotregistry.io/zot/pkg/common"
"zotregistry.io/zot/pkg/log"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

type HTPasswd struct {
Path string `json:"path,omitempty"`
}
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/lint/lint.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
package lint

import (
"encoding/json"
"fmt"

jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"

Expand All @@ -16,6 +16,8 @@ import (
storageTypes "zotregistry.io/zot/pkg/storage/types"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

type Linter struct {
config *config.LintConfig
log log.Logger
Expand Down
5 changes: 4 additions & 1 deletion pkg/extensions/monitoring/minimal_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,17 @@ package monitoring
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"net/http"
"time"

jsoniter "github.com/json-iterator/go"

"zotregistry.io/zot/pkg/log"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

const (
httpTimeout = 1 * time.Minute
)
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/search/convert/metadb.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@

import (
"context"
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"time"

"github.com/99designs/gqlgen/graphql"
jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/vektah/gqlparser/v2/gqlerror"
Expand All @@ -24,6 +24,8 @@
mTypes "zotregistry.io/zot/pkg/meta/types"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

Check failure on line 27 in pkg/extensions/search/convert/metadb.go

View workflow job for this annotation

GitHub Actions / Run zot with extensions tests

json already declared through import of package json ("encoding/json")

type SkipQGLField struct {
Vulnerabilities bool
}
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/search/cve/cve.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
package cveinfo

import (
"encoding/json"
"sort"
"strings"
"time"

jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"

Expand All @@ -17,6 +17,8 @@ import (
"zotregistry.io/zot/pkg/storage"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

type CveInfo interface {
GetImageListForCVE(repo, cveID string) ([]cvemodel.TagInfo, error)
GetImageListWithCVEFixed(repo, cveID string) ([]cvemodel.TagInfo, error)
Expand Down
13 changes: 12 additions & 1 deletion pkg/extensions/search/cve/trivy/scanner.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

import (
"context"
"encoding/json"
"fmt"
"os"
"path"
Expand All @@ -12,11 +11,13 @@
dbTypes "github.com/aquasecurity/trivy-db/pkg/types"
"github.com/aquasecurity/trivy/pkg/commands/artifact"
"github.com/aquasecurity/trivy/pkg/commands/operation"
"github.com/aquasecurity/trivy/pkg/fanal/analyzer"
fanalTypes "github.com/aquasecurity/trivy/pkg/fanal/types"
"github.com/aquasecurity/trivy/pkg/flag"
"github.com/aquasecurity/trivy/pkg/javadb"
"github.com/aquasecurity/trivy/pkg/types"
regTypes "github.com/google/go-containerregistry/pkg/v1/types"
jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"
_ "modernc.org/sqlite"
Expand All @@ -31,15 +32,25 @@
"zotregistry.io/zot/pkg/storage"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

Check failure on line 35 in pkg/extensions/search/cve/trivy/scanner.go

View workflow job for this annotation

GitHub Actions / Run zot with extensions tests

json already declared through import of package json ("encoding/json")

const cacheSize = 1000000

// getNewScanOptions sets trivy configuration values for our scans and returns them as
// a trivy Options structure.
func getNewScanOptions(dir, dbRepository, javaDBRepository string) *flag.Options {
disabledAnalizers := []analyzer.Type{}
if javaDBRepository == "" {
disabledAnalizers = append(disabledAnalizers, analyzer.TypeJar)
disabledAnalizers = append(disabledAnalizers, analyzer.TypePom)
disabledAnalizers = append(disabledAnalizers, analyzer.TypeGradleLock)
}

scanOptions := flag.Options{
GlobalOptions: flag.GlobalOptions{
CacheDir: dir,
},
DisabledAnalyzers: disabledAnalizers,
ScanOptions: flag.ScanOptions{
Scanners: types.Scanners{types.VulnerabilityScanner},
OfflineScan: true,
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/search/resolver.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@

import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"

"github.com/99designs/gqlgen/graphql"
jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/vektah/gqlparser/v2/gqlerror"
Expand All @@ -30,6 +30,8 @@
"zotregistry.io/zot/pkg/storage"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

Check failure on line 33 in pkg/extensions/search/resolver.go

View workflow job for this annotation

GitHub Actions / Run zot with extensions tests

json already declared through import of package json ("encoding/json")

// THIS CODE IS A STARTING POINT ONLY. IT WILL NOT BE UPDATED WITH SCHEMA CHANGES.

const (
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/sync/local.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
package sync

import (
"encoding/json"
"errors"
"fmt"
"os"
Expand All @@ -13,6 +12,7 @@
"time"

"github.com/containers/image/v5/types"
jsoniter "github.com/json-iterator/go"
"github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"

Expand All @@ -28,6 +28,8 @@
storageTypes "zotregistry.io/zot/pkg/storage/types"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

Check failure on line 31 in pkg/extensions/sync/local.go

View workflow job for this annotation

GitHub Actions / Running privileged tests on Linux

json already declared through import of package json ("encoding/json")

type LocalRegistry struct {
storeController storage.StoreController
tempStorage OciLayoutStorage
Expand Down
4 changes: 3 additions & 1 deletion pkg/extensions/sync/references/oci.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ package references

import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"

jsoniter "github.com/json-iterator/go"
godigest "github.com/opencontainers/go-digest"
ispec "github.com/opencontainers/image-spec/specs-go/v1"

Expand All @@ -24,6 +24,8 @@ import (
storageTypes "zotregistry.io/zot/pkg/storage/types"
)

var json = jsoniter.ConfigCompatibleWithStandardLibrary //nolint: gochecknoglobals // to replace standard lib

type OciReferences struct {
client *client.Client
storeController storage.StoreController
Expand Down
Loading
Loading