[ADD] clouder_template_red_october: File encyption/decryption #187
Conversation
| RUN go install github.com/cloudflare/redoctober | ||
|
|
||
| EXPOSE 8080 8081 | ||
| ENV RO_CERTS=/var/lib/redoctober/data/server.crt \ |
There was a problem hiding this comment.
Do you have an example of a template file that is filled by Python variables when deploying the service? I need to be able to pass a cert into this file, from the CA.
A bit lower, I actually need to edit the Dockerfile too. There is an addr option there that will need to be dynamic based on the port we assign in the XML.
I have a similar requirement during the generation of the CA, in which I need to generate a configuration file based on some data that is set in the model (CN, OU, etc).
CFSSL then needs to be run against that config file in order to generate the root CA cert, which is then used as part of the Dockerfile CMD.
There was a problem hiding this comment.
I usually replace them in deploy_post hook https://github.com/clouder-community/clouder/blob/0.9.0/clouder_template_odoo/template.py#L49
I suggest you use this method for now. It'd be great to avoid using deploy_post for this purpose so we can make generic docker images which can be used without Clouder, but that's not the point of this PR.
I'm working on it, I need to make such generic image for the Clouder docker compose so let's wait until I figure the best way to do it.
In the case you describe, I still believe most of the operation will stay in deploy_post.
There was a problem hiding this comment.
A template language like Jinja2 would be seriously badass here if that's similar to what you're thinking. We could even use the pre-existing odoo.tools.safe_eval to allow us a safe code context that we control, allowing for some nice possibilities.
Or better yet, the Jinja2 sandbox like in mail - https://github.com/odoo/odoo/blob/226b28e090deb12ca9a204fdcedc20c0359aad85/addons/mail/models/mail_template.py#L59
Current coverage is 31.54% (diff: 44.44%)@@ master #187 diff @@
==========================================
Files 73 74 +1
Lines 5653 5732 +79
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 1753 1808 +55
- Misses 3900 3924 +24
Partials 0 0
|
There was a problem hiding this comment.
Ok finished review.
Don't you think this template shall be merged with the cfssl template ? With your proposition we'll have
openssl
cfssl-data
cfssl-exec
redoctober-data
redoctober-exec
Each deployed service increased the time needed for oneclick installation. I believe we shall only have
openssl
redoctober-data
redoctober-exec
with redoctober filling the purpose of cfssl.
EDIT : Forget what I said with merge redoctober and cfssl. I was sure that redoctober required cfssl in his container but it's not the case.
| model="clouder.image.port" | ||
| > | ||
| <field name="template_id" ref="image_template_redoctober_exec" /> | ||
| <field name="name">redoctober-http</field> |
There was a problem hiding this comment.
If you need proxy, you shall use http tag
| FROM clouder/base:3.4 | ||
| MAINTAINER Dave Lasley <[email protected]> | ||
|
|
||
| CMD tail -f /dev/null |
| FROM clouder/base:3.4 | ||
| MAINTAINER Dave Lasley <[email protected]> | ||
|
|
||
| CMD tail -f /dev/null |
| RUN go install github.com/cloudflare/redoctober | ||
|
|
||
| EXPOSE 8080 8081 | ||
| ENV RO_CERTS=/var/lib/redoctober/data/server.crt \ |
There was a problem hiding this comment.
I usually replace them in deploy_post hook https://github.com/clouder-community/clouder/blob/0.9.0/clouder_template_odoo/template.py#L49
I suggest you use this method for now. It'd be great to avoid using deploy_post for this purpose so we can make generic docker images which can be used without Clouder, but that's not the point of this PR.
I'm working on it, I need to make such generic image for the Clouder docker compose so let's wait until I figure the best way to do it.
In the case you describe, I still believe most of the operation will stay in deploy_post.
|
Just to be sure, redoctober is about file/data secure storage am I right ? If yes, we had a discussion about https://www.vaultproject.io/ which looked very good to me. Can you describe what is the exact purpose for redoctober ? Shall it replace https://www.vaultproject.io/ and if yes can you explain why we shall use redoctober instead ? |
Red October and CFSSL serve different purposes. Red October is file encryption, and CFSSL is a certificate authority. CFSSL is required to secure our internal communications, such as Logstash=>Elastic. Red October is going to handle encryption/decryption of private keys generated by CFSSL, Openssl, OpenSSH, etc. OpenSSL container is required in order to get some certificate information that I was otherwise unable to obtain from the other services. As I understand it, Docker is built upon the premise of modular design & the combining of logical services is not the way to go. Technically we could just install OpenSSL on every container that requires it, but I feel like that will just increase the deploy time exponentially vs. the one container.
I included some of this our email thread before I noticed the question here. Just so it's all public too, here's the breakdown from the email. Let me know if it doesn't clarify. Vaultproject duplicates a lot of functionality that the CFSSL CA does, without actually providing a CA. It does technically offer more than Red October though, specifically the storage of the encrypted data vs simply key management. On the flip side, Red October is more secure. It has another layer of decryption rights called delegation, which allows you to delegate decryption rights to different RO Vaults. This means that the RO Vault itself is actually portable, and able to be distributed amongst multiple Vaults. In our context, this means we can add superusers to decrypt the data & template those users across customer vaults. TBH both are about as state of the art in terms of design, for the most part it’s just the question of feature duplication. I think a lot of Vault project’s appeal (at least for me) is their website. I still need to do a side-by-side comparison though, and I’m somewhat thinking of making a Vaultproject connector simply for the hell of it. Edit: Just saw your edit. Too late, I explained anyways! 🚀 |
|
Thanks @lasley for clarification, looks like the way to go for me ! |
* Remove data and source from base * Proxy compat * Roadmap notes * Switch user to red october * Fix dockerfile * Bugfixes
lasley
force-pushed
the
feature/master/clouder_crypto
branch
from
e05639f to
be1aaaa
Compare
|
I'm planning on doing a few tests with this later today & will report back on whether we can merge. |
| <field name="template_ids" | ||
| eval="[(4, [ref('image_template_redoctober_data')])]" | ||
| /> | ||
| <field name="parent_from">clouder/base:3.4</field> |
There was a problem hiding this comment.
After inheriting from base, I am getting a build failed due to no command. Looking at the base, this makes sense (because there isn't).
Do we have another Docker layer with just a cat or tail in it that we can use instead?
2016-12-29 23:22:31,950 9723 INFO clouder odoo.addons.clouder.models.model: command : docker build --pull -t dev-red_october-data-20161229.232231 /tmp/dev-red_october-data-20161229.232231
2016-12-29 23:22:32,002 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Sending build context to Docker daemon 2.048 kB
2016-12-29 23:22:32,106 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Step 1 : FROM clouder/base:3.4
2016-12-29 23:22:33,218 9723 INFO ? werkzeug: 10.0.2.2 - - [29/Dec/2016 23:22:33] "GET /web/static/src/img/spin.png HTTP/1.1" 200 -
2016-12-29 23:22:33,273 9723 INFO clouder odoo.addons.clouder.models.model: stdout : 3.4: Pulling from clouder/base
2016-12-29 23:22:33,282 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Digest: sha256:393853d46770a169871d54e6462d3cce1ad2d553ab1522939206b5b389be99e2
2016-12-29 23:22:33,286 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Status: Image is up to date for clouder/base:3.4
2016-12-29 23:22:33,290 9723 INFO clouder odoo.addons.clouder.models.model: stdout : ---> 2921143ce024
2016-12-29 23:22:33,295 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Step 2 : MAINTAINER [email protected]
2016-12-29 23:22:33,298 9723 INFO clouder odoo.addons.clouder.models.model: stdout : ---> Using cache
2016-12-29 23:22:33,303 9723 INFO clouder odoo.addons.clouder.models.model: stdout : ---> 411c56dc580b
2016-12-29 23:22:33,307 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Step 3 : VOLUME data
2016-12-29 23:22:33,736 9723 INFO clouder odoo.addons.clouder.models.model: stdout : ---> Running in 14a3ba11c900
2016-12-29 23:22:33,997 9723 INFO clouder odoo.addons.clouder.models.model: stdout : ---> 12fcb9fb3e12
2016-12-29 23:22:34,066 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Removing intermediate container 14a3ba11c900
2016-12-29 23:22:34,091 9723 INFO clouder odoo.addons.clouder.models.model: stdout : Successfully built 12fcb9fb3e12
2016-12-29 23:22:34,096 9723 INFO clouder odoo.addons.clouder.models.model: connect: ssh dev.localhost
2016-12-29 23:22:34,100 9723 INFO clouder odoo.addons.clouder.models.model:
2016-12-29 23:22:34,104 9723 INFO clouder odoo.addons.clouder.models.model: host : 192.168.69.108
2016-12-29 23:22:34,109 9723 INFO clouder odoo.addons.clouder.models.model: command : rm -rf /tmp/dev-red_october-data-20161229.232231
2016-12-29 23:22:34,150 9723 INFO clouder odoo.addons.clouder.models.model: connect: ssh dev.localhost
2016-12-29 23:22:34,155 9723 INFO clouder odoo.addons.clouder.models.model:
2016-12-29 23:22:34,159 9723 INFO clouder odoo.addons.clouder.models.model: host : 192.168.69.108
2016-12-29 23:22:34,163 9723 INFO clouder odoo.addons.clouder.models.model: command : docker run -d -t --restart=always --name dev-red_october-data dev-red_october-data-20161229.232231
2016-12-29 23:22:34,215 9723 INFO clouder odoo.addons.clouder.models.model: stderr : docker: Error response from daemon: No command specified.
2016-12-29 23:22:34,221 9723 INFO clouder odoo.addons.clouder.models.model: stderr : See 'docker run --help'.
There was a problem hiding this comment.
Ouch, ok I was sure we had a CMD in the inherited base image.
I made a direct commit to resolve it, can you try again with the latest image from Docker hub ? clouder-community/clouder@8695902 https://hub.docker.com/r/clouder/base/builds/bzb6vgezh9o8qk55ycqhedc/
There was a problem hiding this comment.
Woot looks like we're in business thanks! Few more tweaks here and we should be good, I'll let you know
There was a problem hiding this comment.
As a sidenote, I shall remove all tail -f I made in the data images, they are all replaced by the CMD cat I added in the base image
| MAINTAINER Dave Lasley <[email protected]> | ||
|
|
||
| ARG RO_CERTPASSWD="password" | ||
| ARG RO_COMMONNAME="localhost" |
There was a problem hiding this comment.
Do we have a way to specify build args programmatically? I'd like to make this common-name & password something more meaningful
There was a problem hiding this comment.
You should use environment for this. In Clouder deployment, you can specify them by overriding an hook, like this example in mysql template :
https://github.com/clouder-community/clouder/blob/master/clouder_template_mysql/template.py#L53
Note that the use case seems to be the same, you probably want to store them in Clouder as options, like the root password of mysql.
lasley
force-pushed
the
feature/master/clouder_crypto
branch
from
345744d to
d14d4cd
Compare
| <field name="local_port">8080</field> | ||
| </record> | ||
|
|
||
| <record id="image_port_redoctober_metrics" |
There was a problem hiding this comment.
When deploying this from clouder, I'm getting an error when binding to 8081. This isn't happening when I build and run the image on its own, so it has to be something I'm doing wrong in the config here or there's something I don't know about on that port. Any clues where to start?
Relevant Docker logs:
2017/01/02 19:52:25 core.init success: path=/var/lib/redoctober/data/diskrecord.json
2017/01/02 19:52:25 metrics.init start: addr=localhost:8081
2017/01/02 19:52:25 http.serve start: addr=:8080
2017/01/02 19:52:25 listen tcp 198.23.230.10:8081: bind: cannot assign requested address
Without Clouder:
root@odoo-dev-10:/media/sf_Repos/clouder/clouder_template_red_october/images/exec# docker run -t lasley/redoctober-exec
Generating RSA private key, 2048 bit long modulus
....+++
....................................................................................................................+++
e is 65537 (0x10001)
writing RSA key
Signature ok
subject=/C=US/ST=California/L=Everywhere/CN=localhost
Getting Private key
Generated default certificates for RedOctobeer at /var/lib/redoctober/data/server.crt and /var/lib/redoctober/data/server.pem
2017/01/02 19:56:24 core.init success: path=/var/lib/redoctober/data/diskrecord.json
2017/01/02 19:56:24 metrics.init start: addr=localhost:8081
2017/01/02 19:56:24 http.serve start: addr=:8080
I tried removing this port definition just for the hell of it, but that had no effect.
There was a problem hiding this comment.
It seems the problem is located inside your container.
listen tcp 198.23.230.10:8081 I suspect this is not the port which can't be allocated, it's the IP adress. Probably 198.23.230.10 can be used by the node but not by the container, which has probably another IP like 172.17.. more often used by Docker. You probably have a setting somewhere which should be set to 0.0.0.0.
There was a problem hiding this comment.
Woot good call! No idea what that IP is either, it doesn't match with any of my infrastructure (both internal and external facing).
2d241b1a6c52 dev-red_october-exec-20170102.203532 "/usr/lib/go/src/gith" 24 seconds ago Up 18 seconds 0.0.0.0:30012->8080/tcp, 0.0.0.0:30011->8081/tcp dev-red_october-exec
|
Alright this works! I'm not sure if the proxy part is though. Am I supposed to do something more than just setting the name as |
|
So redoctober has a web interface... That's really good to know. You'll need to add link to proxy/dns with https://github.com/clouder-community/clouder/blob/0.9.0/clouder_template_odoo/template.xml#L315, and allow base creation in the application. Then try to create a base on this service, with the url you want to access it. |
Yeah totally - I'm using the JSON API via the same port, but it comes with a rudimentary interface based on the JSON API. Makes it really freaking easy to poke around |
|
@YannickB Hmm ok so I don't think I've tried to create a Base manually before. I created #193 because the view was basically broken, but now that I have things entered I'm getting stuck on some dependencies. I think I have everything figured out except for the |
|
@lasley Application with children shall not have default_image_id, I'm wondering if I didn't let a wrong constraint behind. Can you provide a link to this constraint ? |
|
Oh ok. This function is used when you didn't first created a service. This way you can directly create a base, it will create the service behind for the customer. Useful when you want to create a service each time you create a base for a customer, so he have his own service (no multi-base mode) TBH, recently I only followed the process first create manually the service, then create the base and fill the service_id field so this function is not called. You should do the same for now until we debug it. |
|
Ok I see, so there's just a missing deploy step if you create it the one direction. I already had a service, so that was a pretty easy fix. Hmmm I must still be missing something though, now I'm getting a missing port error. I can replicate it by Reloading the proxy, which jives in the traceback: I think everything is having this trouble though, here's Gitlab Edit: maybe not Gitlab actually, but something during the Odoo one click that isn't generating an exception, but in the logs: And looking back on this, I think we maybe are missing some proxy stuff in Elasticsearch. Oh well, we'll find out soon enough 😆 Oh and for good measure, relevant |
|
Hum I guess we're not entering this condition : https://github.com/clouder-community/clouder/blob/master/clouder_template_proxy/template.py#L253 Can you output |
|
|
|
And this makes sense because the service doesn't seem to have any exposed ports:
|
|
That's strange since you specified the port in https://github.com/clouder-community/clouder/pull/187/files#diff-067c1eb14f6f8af45482f279b63141a4R7 But I see what the problem is. You have to expose it, either local or internet in order for the port to be inherited in service. Eg. https://github.com/clouder-community/clouder/blob/master/clouder_template_odoo/template.xml#L122 |
|
Damn still no go. It looks like Let's Encrypt is failing on the proxy before this (my dev doesn't have a public port 80). Think it's something to do with that? Should I maybe try a Base for something else that's known good (and what would that be)? |
|
Hum, to be honest the local expose status wasn't much tested until now, can you try with internet exposed ? The local shall be used only for container to container interactions, so it'll not publish port in the container itself. |
This is a WIP, I have a question that I'll ask inline. ReadMe below for info on this template.
Clouder Template - Red October
This module provides a Clouder Template for Red Octover.
Red October is a cryptographically-secure implementation of the two-person rule
to protect sensitive data. From a technical perspective, Red October is a
software-based encryption and decryption server. The server can be used to
encrypt a payload in such a way that no one individual can decrypt it. The
encryption of the payload is cryptographically tied to the credentials of the
authorized users.
Authorized persons can delegate their credentials to the server for a period of
time. The server can decrypt any previously-encrypted payloads as long as the
appropriate number of people have delegated their credentials to the server.
This architecture allows Red October to act as a convenient decryption service.
Other systems, including CloudFlare’s build system, can use it for decryption
and users can delegate their credentials to the server via a simple web interface.
All communication with Red October is encrypted with TLS,
ensuring that passwords are not sent in the clear.
Read More on CloudFlare's Blog
Browse Red October on Github