[2019 02 12] Log: Ongoing improvements' discussions

UPDATE: This has effectively been processed into the main SaltStack Formulas project.

---

This is a (temporary) page to capture ongoing discussions had about improvements that can be made to SaltStack Formulas. The idea is for this content to be processed into separate (permanent) pages for planning and decision-making purposes.

---

Slack

Person	Time	Message
Eric	10:29	Hi. there seems to have a lot of recent and interesting discussions, here or on github, about formulas, so I’ll give my 2 cents too. (edited) I think the main thing missing is versioning, and the corresponding tool to manage your formulas dependencies. I’m not sure the actual way to use it with GitFS is ideal because you need to do it in the master config file Puppet/Chef have tools like librarian-puppet which lets you have a "requirements" file containing your list of formulas and constraints about versions. The 'librarian' tool reads this requirements and generate a .lock file containing all the dependencies trees You commit the requirements and .lock in your repository, push it and deploy it, using the tool to install dependencies listed in the .lock, with each exact version. You are then sure to have the exact same version between your dev, staging and prod environment That’s today my main 'problem' with how formulas are actually managed and used. (edited) (I don’t really know about SPM, doc is sparse but I’m interested to read more about it, if you can point me somewhere) (edited)
Daniel Wallace	10:43	We could build that lock file into spm
Eric	10:51	When I read about SPM https://docs.saltstack.com/en/latest/topics/spm/ it seems more targeted to package formulas than managing versioning and deployment
Javier Bértoli	10:56	@daks, same happened to me when I 1st moved from puppet/chef to salt: Wanted a way to specify formulas' versions and didn’t find one. Read SPM but didn’t see the right tool, so I begun forking the formulas I use, and stick to my own copies. It’s not solution for regular users, though 😐
Eric	11:00	At the moment that’s what we do, vendor it in our repo. We think about using GitFS with our own forks
Daniel Wallace	11:06	Spm doesn’t do it right now, but it could. And that is probably what we should use.
Javier Bértoli	11:18	+ > At the moment that’s what we do, vendor it in our repo. We think about using GitFS with our own forks My solution is 'somewhat' convoluted, but let’s me have my own versions while contributing and all. What I did is: 1. Have my own gitolite local server 2. Fork formula_X from saltstack-formulas to my own github site (netmanagers) 3. git clone formula_X from netmanagers 4. Add formula_X’s empty repo to gitolite 5. Add a remote tracking branch to upstream (saltstack-formulas) 6. Add a remote tracking branch to gitolite ('locals') 7. Use gitfs pulling from my gitolite server All this is a few commands the 1st time. Then, with that setup, I can just: a. git remote update to get upstream changes b. git push to locals to get my local formulas updated and used by my salt servers c. git push to netmanagers to submit a PR 🙂
Javier Bértoli	11:20	not the best solution, but works nice. eager to get a better one, tough 😄
Imran Iqbal	11:22	@Javier Bértoli I’ve had something similar, except using the excellent [GitBucket](https://gitbucket.github.io/) instead of Gitolite. GitBucket GitBucket - A Git platform GitBucket is a Git platform powered by Scala with easy installation, high extensibility & GitHub API compatibility And salt-formula can be used to do steps 1-7 pretty much automatically, via. pillar(!) configuration. Hopefully, this rigmarole can be avoided if we can get the versioning and SPM working.
Eric	11:43	but yeah, coming from a puppet ecosystem, I find the maturity level of formulas' ecosystem, perfectible (edited) the other good and bad point is the centralization of most formulas on saltstack-formulas organization. Good because no more abandoned formulas Bad because no one is really responsible of code, no roadmap or specific need which drives development
Imran Iqbal	11:46	Well, that’s what all these recent discussions have been about. Trying to put together that roadmap.
Eric	11:47	ok great! Are those discussions happening here or elsewhere?
Imran Iqbal	11:48	Just GitHub and Slack so far. Collecting the plans in the main project: https://github.com/orgs/saltstack-formulas/projects/1.
Dominik Bessler	20:56	Could using SDB and specifically the yaml sdb module be a Solution for the Pillar Problematic If I have this, I can do a config.get instead of a pillar.get and get the same values. It also seems to me that config.get will go through pillar / config / grains and see if the value is there
Imran Iqbal	21:00	I have no familiarity with SDB so far, so I can’t offer anything about that.
Dominik Bessler	21:01	Untitled cat /etc/salt/minion.d/sdb.conf yaml: driver: yaml files: - /etc/salt/sdb/file.yaml merge: strategy: smart data: sdb://yaml/data Collapse
Imran Iqbal	21:02	One issue I’ve had with config.get is that there can be limitations in specific cases. For example, only one postgres.port value can be set and that’s no good if you need to host multiple PostgreSQL servers on one machine. That may have no relation to this issue at all. I’m just mentioning that config.get wasn’t enough in that specific situation.
Dominik Bessler	21:03	ok
Imran Iqbal	21:03	But thanks for sharing the idea. Hopefully, someone else with more familiarity with SDB can comment.
Dominik Bessler	21:04	I have been working on a few methods that we can use for external data instead of using pillar
Imran Iqbal	21:04	Where was most of the data being stored?
Dominik Bessler	21:04	we are using Pillarstack https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.stack.html I was also able to change that module so that it works as SDB Module by adding a few lines
Imran Iqbal	21:07	I don’t believe we can specifically select Pillarstack across all formulas; we have to allow for any choices to be made. But in terms of these YAML files, where do they end up?
Dominik Bessler	21:09	The Yaml file for the SDB Yaml Module can be anywhere I think like on each minion I think I may need to create something written with examples
Imran Iqbal	21:10	Sounds like an idea worth considering. Yes, that would be useful for evaluation purposes.
Dominik Bessler	22:29	Here is an example on how the Pillarstack Module can be used as SDB Module https://github.com/claudekenni/pillarstack_sdb/blob/master/SDB_Readme.md SDB_Readme.md PillarStack_SDB Module =========== This is an Idea on how this Pillar Module could be used as a SDB Module. The basic premise is that the pillar data will distributed via the file state to the minions Show more claudekenni/pillarstack_sdbAdded by GitHub
Daniel Wallace	22:29	neat
Imran Iqbal	22:30	Thanks, I’ll add that to the plans so that it can be evaluated. @gtmanfred Any thoughts about this idea?
Daniel Wallace	22:31	sdb is a great idea
Dominik Bessler	22:31	Most of the other Ideas I had were quite specific to our Environment so I can’t really share that
Daniel Wallace	22:31	since it offloads the load from the master to the minion as well
Imran Iqbal	22:32	Would this be alongside or instead of TOFS?
Daniel Wallace	22:32	alongside basically, you would just include your stack in your fileroots or other fileserver backend which is basically what TOFS is anyway
Imran Iqbal	22:33	I’m sure @aboe is familiar with Pillarstack. He’d have a good idea of how this would work alongside what we’ve currently planned. Thanks for this effort @Dominik Bessler!
Dominik Bessler	22:34	np The good part about this Module is that you can easily adjust it via adding files / folders and the configuration
Imran Iqbal	22:37	Are you familiar with the TOFS pattern? Have you tried integrating the two methods together?
Dominik Bessler	22:39	I only just read about it today while poking around in here. Currently I’m tasked with getting rid of our extensive pillar data so I will likely look into how I can integrate these
Imran Iqbal	22:41	It would be great if you could report back your findings here.
Dominik Bessler	22:41	will do
Imran Iqbal	22:41	Fantastic, appreciate it.