Add replay via Github API command

README.md

@@ -1,15 +1,18 @@
-# gosmee - A webhook forwarder/relayer
+# gosmee - A webhook forwarder/relayer and replayer
 
 Gosmee is a webhook relayer that can be easily run anywhere.
+It can act as well as a replayer using the GitHub API for GitHub Hooks.
 
 ## Description
 
 Gosmee enables you to relay webhooks from either itself (as a server) or from <https://smee.io> to your local laptop or an infrastructure that is not publicly available from the internet.
 
-gosmee let you easily expose the service on your local network (like a web service on [localhost](https://en.wikipedia.org/wiki/Localhost)) or behind a VPN, allowing a public service (such as GitHub) push webhooks into it.
+Gosmee let you easily expose the service on your local network (like a web service on [localhost](https://en.wikipedia.org/wiki/Localhost)) or behind a VPN, allowing a public service (such as GitHub) push webhooks into it.
 
 For instance, if you configure your GitHub Webhook to direct to a <https://smee.io/> URL or where gosmee server is listening, you can then use the gosmee client on your local notebook to obtain the events from the server and forward them to the local service, thereby establishing a connection between the GitHub webhook and your local service on your workstation.
 
+Alternatively if you don't want to use a relay server and use GitHub you can replay the hooks deliveries via the GitHub API.
+
 ### Diagram
 
 For the people who rather prefer to understand on how it works with a small
@@ -46,7 +49,7 @@ yay -S gosmee-bin
 docker run ghcr.io/chmouel/gosmee:latest
 ```
 
-## GO
+### GO
 
 ```shell
 go install -v github.com/chmouel/gosmee@latest
@@ -84,9 +87,17 @@ nix flake check # runs tests
 
 System Service example file for macOS and Linux is available in the [misc](./misc) directory.
 
+### Kubernetes
+
+You can expose an internal kubernetes deployment or service with gosmee by using [this file](./misc/kubernetes-deployment.yaml).
+
+Adjust the `SMEE_URL` in there to your endpoint and the `http://deployment.name.namespace.name:PORT_OF_SERVICE` URL is the Kubernetes internal URL of your deployment running on your cluster, for example:
+
+   <http://service.namespace:8080>
+
 ### Shell completion
 
-Shell completion is available with:
+Shell completions is available for gosmee:
 
 ```shell
 # BASH
@@ -154,7 +165,7 @@ With `/new` you can easily generate a random ID, ie:
 http://localhost:3333/NqybHcEi
 ```
 
-### Caddy
+#### Caddy
 
 [Caddy](https://caddyserver.com/) is the best way to run gosmee server, you just need this:
 
@@ -166,7 +177,7 @@ https://webhook.mydomain {
 
 It will automatically configure a letsencrypt certificate for you
 
-### Nginx
+#### Nginx
 
 Running gosmee server behind nginx may require some configuration to work properly.
 Here is a `proxy_pass location` to a locally running gosmee server on port localhost:3333:
@@ -181,15 +192,58 @@ Here is a `proxy_pass location` to a locally running gosmee server on port local
     }
 ```
 
-There is maybe some errors appearing some time with nginx with long running connections.
+There is maybe some errors appearing some time with nginx with long running connections. Help is welcome to help debug this.
 
-### Kubernetes
+## Replay
 
-You can expose an internal kubernetes deployment or service with gosmee by using [this file](./misc/kubernetes-deployment.yaml).
+Alternatively if you don't want to use a relay server and use GitHub you can
+replay the hooks deliveries via the GitHub API. Compared to the relay server
+method this is more reliable and you don't have to worry about the relay server
+being down. The downside is that it only works with GitHub and you need to have
+a GitHub token with the `repo` scope.
 
-Adjust the `SMEE_URL` in there to your endpoint and the `http://deployment.name.namespace.name:PORT_OF_SERVICE` URL is the Kubernetes internal URL of your deployment running on your cluster, for example:
+It currently only supports Repository and not yet organisation hooks.
 
-   <http://service.namespace:8080>
+You will need to know the Hook ID of the webhook you want to replay, you can
+get it with the `--hook-id` command:
+
+```shell
+goplay replay --github-token=$GITHUB_TOKEN --list-hooks org/repo
+```
+
+This will list all the hooks for the repository and their ID. When you grab the
+appropriate you can start to listen to the events and replay them on a local
+server:
+
+```shell
+goplay replay --github-token=$GITHUB_TOKEN org/repo HOOK_ID http://localhost:8080
+```
+
+This will listen to all **new** events and replay them on <http://localhost:8080>.
+
+You can also replay all the events that have been previously received by the
+hook from a date time. The date is is in UTC and in the format of
+`2023-12-19T12:31:12` and it will replay all the events from that date to now:
+
+```shell
+goplay replay --time-since=2023-12-19T09:00:00 --github-token=$GITHUB_TOKEN org/repo HOOK_ID http://localhost:8080
+```
+
+To make it easier to know the date you can use the `--list-deliveries` command
+to list all the deliveries and their date:
+
+```shell
+goplay replay --github-token=$GITHUB_TOKEN --list-deliveries org/repo HOOK_ID
+```
+
+>[!NOTE]
+>`gosmee replay` does not support paging yet, and list only the last
+>100 deliveries. So if you specify a date that is older than the last 100
+>deliveries it will not work.
+
+>[!NOTE]
+>When the token gets rate limited, gosmee will be just failing and do not at the
+>moment do anything to manage this.
 
 ## Thanks
 

go.mod

@@ -4,6 +4,7 @@ go 1.20
 
 require (
 	github.com/go-chi/chi/v5 v5.0.10
+	github.com/google/go-github/v57 v57.0.0
 	github.com/mattn/go-isatty v0.0.20
 	github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
 	github.com/mitchellh/mapstructure v1.5.0
@@ -18,7 +19,8 @@ require (
 require (
 	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/google/go-cmp v0.5.9 // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
+	github.com/google/go-querystring v1.1.0 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/xrash/smetrics v0.0.0-20231213231151-1d8dd44e695e // indirect

go.sum

@@ -5,8 +5,13 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/go-chi/chi/v5 v5.0.10 h1:rLz5avzKpjqxrYwXNfmjkrYYXOyLJd37pz53UFHC6vk=
 github.com/go-chi/chi/v5 v5.0.10/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
-github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
-github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
+github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/go-github/v57 v57.0.0 h1:L+Y3UPTY8ALM8x+TV0lg+IEBI+upibemtBD8Q9u7zHs=
+github.com/google/go-github/v57 v57.0.0/go.mod h1:s0omdnye0hvK/ecLvpsGfJMiRt85PimQh4oygmLIxHw=
+github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=
+github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=
 github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
 github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
 github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
@@ -45,6 +50,7 @@ golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
 golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gopkg.in/cenkalti/backoff.v1 v1.1.0 h1:Arh75ttbsvlpVA7WtVpH4u9h6Zl46xuptxqLxPiSo4Y=
 gopkg.in/cenkalti/backoff.v1 v1.1.0/go.mod h1:J6Vskwqd+OMVJl8C33mmtxTBs2gyzfv7UDAkHu8BrjI=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

gosmee/app.go

@@ -21,17 +21,43 @@ var zshCompletion []byte
 //go:embed templates/bash_completion.bash
 var bashCompletion []byte
 
+func getLogger(c *cli.Context) (*slog.Logger, bool, error) {
+	nocolor := c.Bool("nocolor")
+	w := os.Stdout
+	var logger *slog.Logger
+	switch c.String("output") {
+	case "json":
+		logger = slog.New(slog.NewJSONHandler(os.Stdout, nil))
+		nocolor = true
+	case "pretty":
+		logger = slog.New(tint.NewHandler(w, &tint.Options{
+			TimeFormat: time.RFC1123,
+			NoColor:    !isatty.IsTerminal(w.Fd()),
+		}))
+	default:
+		return nil, false, fmt.Errorf("invalid output format %s", c.String("output"))
+	}
+	return logger, nocolor, nil
+}
+
 func makeapp() *cli.App {
 	app := &cli.App{
 		Name:  "gosmee",
 		Usage: "Forward SMEE url from an external endpoint to a local service",
-		UsageText: `gosmee can forward webhook from https://smee.io or from
-itself to a local service. The server is the one from where the webhook
-points to. The client runs on your laptop or behind a non publically
-accessible endpoint and forward request to your local service`,
+		UsageText: `Gosmee can help you reroute webhooks either from https://smee.io or its own server to a local service.
+Where the server is the source of the webhook, and the client, which you run on your laptop or behind a
+non-publicly accessible endpoint, forward those requests to your local service.`,
 		EnableBashCompletion: true,
 		Version:              strings.TrimSpace(string(Version)),
 		Commands: []*cli.Command{
+			{
+				Name:  "replay",
+				Usage: "Replay payloads from GitHub",
+				Action: func(c *cli.Context) error {
+					return replay(c)
+				},
+				Flags: append(commonFlags, replayFlags...),
+			},
 			{
 				Name:  "server",
 				Usage: "Make gosmee a relay server from your external webhook",
@@ -41,63 +67,16 @@ accessible endpoint and forward request to your local service`,
 					}
 					return serve(c)
 				},
-				Flags: []cli.Flag{
-					&cli.StringFlag{
-						Name:  "public-url",
-						Usage: "Public URL to show to user, useful when you are behind a proxy.",
-					},
-					&cli.IntFlag{
-						Name:    "port",
-						Aliases: []string{"p"},
-						Value:   defaultServerPort,
-						Usage:   "Port to listen on",
-					},
-					&cli.BoolFlag{
-						Name:  "auto-cert",
-						Value: false,
-						Usage: "Automatically generate letsencrypt certs",
-					},
-					&cli.StringFlag{
-						Name:  "footer",
-						Usage: "An HTML string to show in footer for copyright and author",
-					},
-					&cli.StringFlag{
-						Name:    "address",
-						Aliases: []string{"a"},
-						Value:   defaultServerAddress,
-						Usage:   "Address to listen on",
-					},
-					&cli.StringFlag{
-						Name:    "tls-cert",
-						Usage:   "TLS certificate file",
-						EnvVars: []string{"GOSMEE_TLS_CERT"},
-					},
-					&cli.StringFlag{
-						Name:    "tls-key",
-						Usage:   "TLS key file",
-						EnvVars: []string{"GOSMEE_TLS_KEY"},
-					},
-				},
+				Flags: serverFlags,
 			},
 			{
 				Name:      "client",
 				UsageText: "gosmee [command options] SMEE_URL LOCAL_SERVICE_URL",
 				Usage:     "Make a client from the relay server to your local service",
 				Action: func(c *cli.Context) error {
-					nocolor := c.Bool("nocolor")
-					w := os.Stdout
-					var logger *slog.Logger
-					switch c.String("output") {
-					case "json":
-						logger = slog.New(slog.NewJSONHandler(os.Stdout, nil))
-						nocolor = true
-					case "pretty":
-						logger = slog.New(tint.NewHandler(w, &tint.Options{
-							TimeFormat: time.RFC1123,
-							NoColor:    !isatty.IsTerminal(w.Fd()),
-						}))
-					default:
-						return fmt.Errorf("invalid output format %s", c.String("output"))
+					logger, nocolor, err := getLogger(c)
+					if err != nil {
+						return err
 					}
 
 					var smeeURL, targetURL string
@@ -127,67 +106,22 @@ accessible endpoint and forward request to your local service`,
 						decorate = false
 					}
 					cfg := goSmee{
-						smeeURL:           smeeURL,
-						targetURL:         targetURL,
-						saveDir:           c.String("saveDir"),
-						noReplay:          c.Bool("noReplay"),
-						decorate:          decorate,
-						ignoreEvents:      c.StringSlice("ignore-event"),
-						channel:           c.String("channel"),
-						targetCnxTimeout:  c.Int("target-connection-timeout"),
-						insecureTLSVerify: c.Bool("insecure-skip-tls-verify"),
-						logger:            logger,
+						replayDataOpts: &replayDataOpts{
+							smeeURL:           smeeURL,
+							targetURL:         targetURL,
+							saveDir:           c.String("saveDir"),
+							noReplay:          c.Bool("noReplay"),
+							decorate:          decorate,
+							ignoreEvents:      c.StringSlice("ignore-event"),
+							targetCnxTimeout:  c.Int("target-connection-timeout"),
+							insecureTLSVerify: c.Bool("insecure-skip-tls-verify"),
+						},
+						logger:  logger,
+						channel: c.String("channel"),
 					}
-					err := cfg.clientSetup()
-					return err
-				},
-				Flags: []cli.Flag{
-					&cli.StringFlag{
-						Name:    "channel",
-						Aliases: []string{"c"},
-						Usage:   "gosmee channel to listen, only useful when you are not use smee.io",
-						Value:   smeeChannel,
-					},
-					&cli.StringFlag{
-						Name:    "output",
-						Usage:   `Output format, one of "json", "pretty"`,
-						Value:   "pretty",
-						Aliases: []string{"o"},
-					},
-					&cli.StringSliceFlag{
-						Name:    "ignore-event",
-						Aliases: []string{"I"},
-						Usage:   "Ignore these events",
-					},
-					&cli.StringFlag{
-						Name:    "saveDir",
-						Usage:   "Save payloads to `DIR` populated with shell scripts to replay easily.",
-						Aliases: []string{"s"},
-						EnvVars: []string{"GOSMEE_SAVEDIR"},
-					},
-					&cli.IntFlag{
-						Name:    "target-connection-timeout",
-						Usage:   "How long to wait when forwarding the request to the service",
-						EnvVars: []string{"GOSMEE_TARGET_TIMEOUT"},
-						Value:   defaultTimeout,
-					},
-					&cli.BoolFlag{
-						Name:    "noReplay",
-						Usage:   "Do not replay payloads",
-						Aliases: []string{"n"},
-						Value:   false,
-					},
-					&cli.BoolFlag{
-						Name:    "nocolor",
-						Usage:   "Disable color output, automatically disabled when non tty",
-						EnvVars: []string{"NO_COLOR"},
-					},
-					&cli.BoolFlag{
-						Name:  "insecure-skip-tls-verify",
-						Value: false,
-						Usage: "If true, the target server's certificate will not be checked for validity. This will make your HTTPS connections insecure",
-					},
+					return cfg.clientSetup()
 				},
+				Flags: append(commonFlags, clientFlags...),
 			},
 			{
 				Name:  "completion",