Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve our existing Grafana dashboard #397

Open
vnugent opened this issue Mar 13, 2024 · 2 comments
Open

Improve our existing Grafana dashboard #397

vnugent opened this issue Mar 13, 2024 · 2 comments
Labels

Comments

@vnugent
Copy link
Contributor

vnugent commented Mar 13, 2024

I'm looking for an experienced DevOps/infra engineer to help improve our Grafana cloud dashboard. We've already connected Granana with our K8s cluster.

  1. Create a "single pane of glass" view containing essential metrics such as API requests, frontend request (not sure how to integerate with next.js)
  2. Set up email or Discord notification alerts
@vnugent vnugent converted this from a draft issue Mar 13, 2024
@enapupe
Copy link
Contributor

enapupe commented Mar 18, 2024

It seems we could push vercel logs to our grafana dash using the same loki endpoint we have configured already, however this requires Pro plan on vercel, I'm not sure we got it. Also, the log format vercel pushes doesn't seem to be accepted by loki, so we'd have to add a middleman to transform it, maybe a basic lambda function running on vercel itself?

https://logs-prod-006.grafana.net
    basicAuth:
      username: "<some user id>"
      password:  <some token here>

Regarding vercel TRACES, we must add an environment variable pointing open telemetry metrics to our grafana agent, however, our agent is not exposed publicly and I haven't considered the drawbacks of exposing it so I'm not 100% sure this is the way forward nor sure we even want to capture otel traces.

OTEL_EXPORTER_OTLP_ENDPOINT=http://grafana-k8s-monitoring-grafana-agent.default.svc.cluster.local:4318

also follow the instructions on https://vercel.com/docs/observability/otel-overview

pnpm i @opentelemetry/api @vercel/otel
nextConfig.experimental = { instrumentationHook: true };

Create an instrumentation.ts file in the root of your project and add the following code to initialize and configure OTEL using @vercel/otel.

import { registerOTel } from '@vercel/otel';
 
export function register() {
  registerOTel({ serviceName: 'your-project-name' });
}
// NOTE: You can replace `your-project-name` with the actual name of your project

@vnugent
Copy link
Contributor Author

vnugent commented Mar 20, 2024

@enapupe we're on Vercel Pro plan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: General backlog
Development

No branches or pull requests

2 participants