Allow multiple sopsFiles for easier common secrets sharing across multiple configuration #417

vdbe · 2023-10-08T08:13:19Z

Allow multiple sopsFiles for easier secret sharing across multiple configurations.

For example you have files

secrets/common.yaml

hashed_password: ....
vpn_user: ...
vpn_password: ...
vpn_cert: | ...

secrets/<system>/default.yaml

vpn_user: ...
vpn_password: ...

with sops.defaultSopsFiles = [ ./secrets/common.yaml ./secrets/<system>/default.yaml ]; in the configuration for <system>.

This would allow overriding of shared secrets across hosts without manually checking and changing the required sopsFile per secret.

vdbe · 2023-10-08T08:14:36Z

Current changes are just a rough draft

Mic92 · 2023-10-08T09:52:15Z

Only question is how you order priority between those in case keys overlap? Otherwise it would make sense to throw an error so that users don't accidentally shadow secrets.

vdbe · 2023-10-08T10:03:54Z

My idea was to have the files in increasing priority,
this way you can have a default common file and append system/user specific files.

In other words search the list of sops files in reverse order and use the first found secret.

In the example above vpn_user and vpn_password in secrets/common.yaml would be shadowed by vpn_user and vpn_password in secrets/<system>/default.yaml.

Mic92 · 2023-10-08T10:19:11Z

It's just hard to enforce an order in lists if you those lists get defined in different nixos modules.

Mic92 · 2023-10-08T10:22:03Z

Ah. I think one can use lib.mkBefore for host files.

Mic92 · 2023-10-08T10:23:31Z

modules/sops/default.nix

+        description = ''
+          Sops file the secret is loaded from.
+        '';
+      };
      sopsFileHash = mkOption {


We should deprecated this one than.

Deprecate sopsFiles? Just added that this PR, do you mean deprecate sopsFile?

We shouldn't really need both sopsFileHash and sopsFilesHash, no?

They could be point to the same value.

Mic92 · 2023-10-08T10:23:56Z

modules/sops/default.nix

@@ -172,6 +189,13 @@ in {
        Default sops file used for all secrets.
      '';
    };


And this one with a warning.

vdbe

Something like this?

vdbe · 2023-10-14T10:58:15Z

modules/sops/default.nix

-      sopsFile = lib.mkOptionDefault cfg.defaultSopsFile;
-      sopsFileHash = mkOptionDefault (optionalString cfg.validateSopsFiles "${builtins.hashFile "sha256" config.sopsFile}");
+      sopsFile = mkOptionDefault cfg.defaultSopsFile;
+      sopsFiles = if options.sopsFile.isDefined then warn "`sops.secrets.<name>.sopsFile` is being deprecated, use `sops.secrets.<name>.sopsFiles` instead" [ config.sopsFile ] else (lib.mkOptionDefault cfg.defaultSopsFiles);


defaultSopsFile/sopsFile takes priority over defaultSopsFiles

Can we not add defaultSopsFile to the new defaultSopsFiles list?

Would it not be better to keep defaultSopsFile and defaultSopsFiles separate and place sopsFile before sopsFiles per secret?
If you merge in default you would need to remove sopsFile
otherwise sopsFiles would become [ sopsFile defaultSopsFile ] ++ defaultSopsFiles.

The only way I was able to implement this feels however a bit hacky

config = mkMerge [{ sopsFile = mkOptionDefault cfg.defaultSopsFile; sopsFiles = mkOptionDefault cfg.defaultSopsFiles; sopsFilesHash = mkOptionDefault (optionals cfg.validateSopsFiles (forEach config.sopsFiles (builtins.hashFile "sha256"))); } { sopsFiles = mkIf (config.sopsFile != null) (mkOverride options.sopsFile.highestPrio (mkBefore [config.sopsFile])); }];

The effect of this is that when both sopsFile and sopsFiles are defined (and not null) with different priorities only one will make it into config.sopsFiles

sopsFile or sopsFiles also take priority over defaultSopsFile and defaultSopsFiles

vdbe · 2023-10-14T11:00:49Z

pkgs/sops-install-secrets/main.go

+	for i := len(files) - 1; i >= 0; i-- {
+		err := app.validateSopsFile(secret, &files[i])
+		if err == nil {
+			// Found valid sopsFile
+			break
+		} else if i == 0 {
+			// No valid sopsFile found in sopsFiles
+			return err
+		}
+	}


The error "secret %s defined the format of %s as %s, but it was specified as %s in %s before" from validateSopsFile is ignored

Mhm. I would build a new error message that would give you all files where the secret has been looked for instead.

vdbe · 2023-10-14T11:06:02Z

pkgs/sops-install-secrets/main.go

+			default:
+				return fmt.Errorf("Secret of type %s in %s is not supported", s.Format, s.SopsFiles[i])
+			}
+		}
 		switch s.Format {


Assuming all sopsFiles have the same format

Mic92 · 2023-10-24T13:40:35Z

pkgs/sops-install-secrets/main.go

@@ -393,40 +400,40 @@ func lookupKeysGroup() (int, error) {
  return 0, fmt.Errorf("Can't find group 'keys' nor 'nogroup' (%w).", err2)
 }

-func (app *appContext) loadSopsFile(s *secret) (*secretFile, error) {
+func (app *appContext) loadSopsFile(s *secret, sopsFileIndex int) (*secretFile, error) {


Can we not pass in the sopsFile as a pointer here rather than using an index?

Mic92 · 2023-10-24T13:41:10Z

pkgs/sops-install-secrets/main.go

-		if err != nil {
-			return err
+	files := []secretFile {};
+	for index, sopsFile := range secret.SopsFiles {


You already have you reference here.

gabevenberg · 2024-05-31T23:02:14Z

is this dead in the water? is there another way to share secrets among hosts without copying secrets between individual hosts secrets.yaml files?

Mic92 reviewed Oct 8, 2023

View reviewed changes

modules/sops/default.nix

@@ -172,6 +189,13 @@ in {

Default sops file used for all secrets.

'';

};

Copy link

Owner

Mic92 Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this one with a warning.

vdbe commented Oct 14, 2023

View reviewed changes

Mic92 reviewed Oct 24, 2023

View reviewed changes

vdbe force-pushed the master branch from addbe58 to e9ff03f Compare November 12, 2023 18:38

vdbe marked this pull request as ready for review November 12, 2023 18:55

vdbe requested a review from Mic92 December 3, 2023 10:35

vdbe added 4 commits February 2, 2024 13:33

feat(shadowing): golang

262a222

feat(shadowing): nixos module

8ee9a71

feat(shadowing): home manager module

0068d61

test(shadowing): sops-install-secrets

c623385

vdbe force-pushed the master branch from 707e856 to c623385 Compare February 2, 2024 12:51

fabiosouzadev approved these changes Aug 14, 2024

View reviewed changes

bjornfor mentioned this pull request Sep 16, 2024

How to take secrets from multiple files? #622

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow multiple sopsFiles for easier common secrets sharing across multiple configuration #417

Allow multiple sopsFiles for easier common secrets sharing across multiple configuration #417

vdbe commented Oct 8, 2023

vdbe commented Oct 8, 2023

Mic92 commented Oct 8, 2023

vdbe commented Oct 8, 2023

Mic92 commented Oct 8, 2023

Mic92 commented Oct 8, 2023

Mic92 Oct 8, 2023

vdbe Oct 8, 2023

Mic92 Oct 10, 2023

Mic92 Oct 10, 2023

Mic92 Oct 8, 2023

vdbe left a comment

vdbe Oct 14, 2023

Mic92 Oct 24, 2023

vdbe Oct 28, 2023 •

edited

Loading

vdbe Oct 14, 2023

Mic92 Oct 24, 2023

vdbe Oct 14, 2023

Mic92 Oct 24, 2023

Mic92 Oct 24, 2023

gabevenberg commented May 31, 2024

Allow multiple sopsFiles for easier common secrets sharing across multiple configuration #417

Are you sure you want to change the base?

Allow multiple sopsFiles for easier common secrets sharing across multiple configuration #417

Conversation

vdbe commented Oct 8, 2023

vdbe commented Oct 8, 2023

Mic92 commented Oct 8, 2023

vdbe commented Oct 8, 2023

Mic92 commented Oct 8, 2023

Mic92 commented Oct 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdbe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdbe Oct 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabevenberg commented May 31, 2024

vdbe Oct 28, 2023 •

edited

Loading