Set up staging.tunapanda.org (essential software)

[limikael]

This includes Learning Locker and the Swag portal (or equivalent).

The provisioning repo contains Ansible roles (a role is a set of instructions for setting up a service automatically) that can be used to deploy important services. I propose that it's worth some extra effort up-front to create Ansible roles for everything we need on staging.tunapanda rather than configuring it manually, since once we have the setup automated with Ansible, we can easily re-deploy, deploy to a local server for dev and testing, etc.

A while ago I started (but never completed) work on roles for swag, learninglocker, and composer, on which learninglocker depends. These are part of a very old branch, and will need to be merged to a newer branch before further work can be done. For now I'm just linking to them as examples.

[usernamenumber]

I've merged the beginnings of support for a swag dev environment into the master branch of the provisioning repo. If you follow these instructions, you should end up with a server that has swag and learninglocker installed. _However_, there are still problems that need to be worked out.

I can split these out into their own issues if needed, but hopefully we will be able to deal with them here:

_Swagportal Crash_
Upon visiting the swagportal page, I get this error:

**** swagportal.dev.swag.tunapanda.org **** 

No such controller  

#0 /usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(145): WebDispatcher->loadController('main') #1 /usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(94): WebDispatcher->dispatchComponents(Array) #2 /usr/local/tunapanda/data/swagportal/index.php(8): WebDispatcher->dispatch() #3 {main} 

I did some research on this "minimvc" I'd never heard of and found that @limikael wrote it! Nice! :) Hopefully having the author on-hand will make this easy to troubleshoot?

_Learninglocker Still Needs Manual Config_
Learninglocker seems to work, but it looks like turning a mongodb dump into a templateable text file is more difficult than it is for mysql, so there's no pre-configuration on it, and PHP doesn't seem to be able to send email (e.g. for account verification) properly, even though there is a working mail server on the machine. Currently you must manually do the following:

1. Fill in the registration form (you're taken there by default). This creates a user with admin rights, which for some reason still requires email vaidation.
2. Use the mongo learninglocker and manually update your user record, setting verified to true (sorry, I don't remember the exact command, though this reference is helpful).

At this point you should have admin rights on a working(???) instance of learninglocker.

...which actually leads to one more TODO:
_Test Learninglocker_
I don't have time right now, but could we test LL by configuring swag to use it and then just running the swag reporter tests? @limikael?

[limikael]

Hm... maybe it is something with the .htaccess

Will try it tomorrow!

// Micke

On Tue, May 5, 2015 at 8:23 PM, Brad Smith [email protected] wrote:

I've merged the beginnings of support for a swag dev environment into the
master branch of the provisioning repo. If you follow these instructions
https://github.com/tunapanda/swag/wiki/Setting-up-a-Swag-dev-environment-with-Vagrant,
you should end up with a server that has swag and learninglocker installed.
However, there are still problems that need to be worked out.

I can split these out into their own issues if needed, but hopefully we
will be able to deal with them here:

Swagportal Crash
Upon visiting the swagportal page, I get this error:

**** swagportal.dev.swag.tunapanda.org ****

No such controller

#0 /usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(145): WebDispatcher->loadController('main') #1 /usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(94): WebDispatcher->dispatchComponents(Array) #2 /usr/local/tunapanda/data/swagportal/index.php(8): WebDispatcher->dispatch() #3 {main}

I did some research on this "minimvc" I'd never heard of and found that
@limikael https://github.com/limikael wrote it
https://github.com/limikael/minimvc! Nice! :) Hopefully having the
author on-hand will make this easy to troubleshoot?

Learninglocker Still Needs Manual Config
Learninglocker seems to work, but it looks like turning a mongodb dump
into a templateable text file is more difficult than it is for mysql, so
there's no pre-configuration on it, and PHP doesn't seem to be able to
send email (e.g. for account verification) properly, even though there is a
working mail server on the machine. Currently you must manually do the
following:

1. Fill in the registration form (you're taken there by default). This
   creates a user with admin rights, which for some reason still requires
   email vaidation.
2. Use the mongo learninglocker and manually update your user record,
   setting verified to true (sorry, I don't remember the exact command,
   though this reference
   http://docs.mongodb.org/manual/reference/sql-comparison/ is helpful).

At this point you should have admin rights on a working(???) instance of
learninglocker.

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

Ah! That's probably it. Since a bunch of other things we deploy use nginx I've been deploying things under it instead of Apache, which means adapting htaccess files to nginx directives. Don't have time to check right now, but if that's the issue it should be pretty simple for me to fix.

[usernamenumber]

It looks like the htaccess files just translate /foo to index.php?foo if /foo doesn't exist, correct? I've got what should be an equivalent nginx rule in there, so maybe there's something else going on. I'll investigate further when I get a chance, but if get a chance to look, let me know if you spot anything obviously wrong.

[limikael]

Yep will take a look... But actually, /too is not translated to
index.php?too, it is just translated to index.php, and then I use a little
mechanism inside to check the actual path... But I'm pretty sure that this
mechanism is apache specific, now it under nginx? So I need to fix my
little library to support nginx then?

// Micke

On Wednesday, May 6, 2015, Brad Smith [email protected] wrote:

It looks like the htaccess files just translate /foo to index.php?foo if
/foo doesn't exist, correct? I've got what should be an equivalent nginx
rule in there, so maybe there's something else going on. I'll investigate
further when I get a chance, but if get a chance to look, let me know if
you spot anything obviously wrong.

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[limikael]

Aha... Found it...

There was another error, but there was also an error in the error
reporting, so the error was difficult to detect... I have fixed the error
with the error reporting, so the actual error shows now:

**** swagportal.dev.swag.tunapanda.org ****

syntax error, unexpected '$' in
/usr/local/tunapanda/data/swagportal/src/controllers/../../config.ini on
line 3

#0 [internal function]: SystemUtil::exception_error_handler(2, 'syntax
error, u...', '/usr/local/tuna...', 37, Array) #1
/usr/local/tunapanda/data/swagportal/src/controllers/MainController.php(37):
parse_ini_file('/usr/local/tuna...') #2
/usr/local/tunapanda/data/swagportal/src/controllers/MainController.php(25):
MainController->loadConfig() #3
/usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(110):
MainController->MainController() #4
/usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(147):
WebDispatcher->loadController('main') #5
/usr/local/tunapanda/data/swagportal/extern/minimvc/src/dispatcher/WebDispatcher.php(95):
WebDispatcher->dispatchComponents(Array) #6
/usr/local/tunapanda/data/swagportal/index.php(8):
WebDispatcher->dispatch() #7 {main}

This is because I changed the format of the .ini file... It was a .php file
before, but I changed it into being an .ini file. It should look like this:

https://github.com/tunapanda/swag/blob/master/config.template.ini

So there must be a template somewhere that gets copied during the
provisioning, this template needs to be changed also... Where is this
template and how would I do that?

// Micke

On Wed, May 6, 2015 at 7:18 AM, Mikael Lindqvist [email protected]
wrote:

Yep will take a look... But actually, /too is not translated to
index.php?too, it is just translated to index.php, and then I use a little
mechanism inside to check the actual path... But I'm pretty sure that this
mechanism is apache specific, now it under nginx? So I need to fix my
little library to support nginx then?

// Micke

On Wednesday, May 6, 2015, Brad Smith [email protected] wrote:

It looks like the htaccess files just translate /foo to index.php?foo if
/foo doesn't exist, correct? I've got what should be an equivalent nginx
rule in there, so maybe there's something else going on. I'll investigate
further when I get a chance, but if get a chance to look, let me know if
you spot anything obviously wrong.

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

I'm still finishing up some testing before I commit the change, but yes, one you fixed the error reporting I found and fixed the template bug right away! :)

With the code I've got, portal now goes to a login page, but that actually leads to another problem: it looks like the php pam module has been deprecated, with no replacement I've been able to find, so auth doesn't work. There are probably workarounds, but the easiest might be to just go ahead and implement social media login as discussed in the other thread. Thoughts?

[limikael]

Great that it worked!

But we need both... We need some kind of pam auth for classroom servers, and social login for when we deploy on the web...

[usernamenumber]

Yes, we definitely need both. Sorry, I wrote that literally as I was falling asleep, and I meant to say this would be a short term solution, just so people can log in and start hacking asap.

For local auth, there are a few ways we could do it. For example, nginx has a pam module, so we could offload authenticating local users there. This has the disadvantage to using a pop-up dialogue instead of an html form to prompt for un/pw (though based on my reading this might be fixable), but would have the advantage of making it easy to write authentication code for pretty much anything. All the code really has to do is check whether or not the server has set an environment variable with the name of the authenticated user and then treat that user as logged in. Having something that can be easily adapted to other services would be a big win for me, since there are a bunch of other things we'll want to run in other contexts that will also need to authenticate, and we really don't want to have to manage a separate userbase for each.

Anyway, for now, since the social auth thing seems almost trivial to set up, I'm thinking it will be the quickest way to produce a working dev environment, but I agree that we also need a solution for local auth.

[limikael]

I also created a thing here:

https://github.com/tunapanda/swag/blob/master/config.template.ini

In the .ini file it is possible to set a fixed user... That is the absolutely simplest...

We do need pam authentication also, and I would rather rely on php than the webserver... Is there no way to actually do pam authentication in php that is not considered obsolete?

// Micke

[usernamenumber]

Not that I've been able to find, but that seems really odd to me so maybe I've just missed something. If you can find something that works easily in Ubuntu Trusty, great!

[limikael]

Ok will look for something like that...

// Micke

On Thu, May 7, 2015 at 3:06 PM, Brad Smith [email protected] wrote:

Not that I've been able to find, but that seems really odd to me, so maybe
I've just missed something. If you can find something that works easily in
Ubuntu Trusty, though, great!

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

btw, the tests completed successfully and I pushed the changes, so if you do a git pull and reprovision a swag_dev vm, you should end up with a working (but not configured) learninglocker, and working (but not able to log in) swagportal. Progress! :D

[limikael]

Yep! It worked!

Starting to like vagrant actually!

Will now look for a way to get pam_auth to work...

What I did before was just "apt-get install php5-pam-auth" but this is what
is deprecated? This one:

http://packages.ubuntu.com/search?keywords=php5-auth-pam

Noticing that there is no Trusty there, so guess so...

So what I can do now is to ssh into the provisioned machine and hack
around, right?

// Micke

On Thu, May 7, 2015 at 3:59 PM, Brad Smith [email protected] wrote:

btw, the tests completed successfully and I pushed the changes, so if you
do a git pull and reprovision a swag_dev vm, you should end up with a
working (but not configured) learninglocker, and working (but not able to
log in) swagportal. Progress! :D

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[limikael]

Hi,

Got pam working I think... At least to the point where it said

Authentication failure (in pam_authenticate)

When I tried to login. Remind me again please which users are there so I
can try if login actually works as well... Maybe there is something else
that needs to be configured too that allows php to use pam...

I did the following steps:

• sudo apt-get install libpam0g-dev php5-dev php-pear
• sudo pecl install pam
• I added "extension=pam.so" to /etc/php5/fpm/php.ini
• sudo service php5-fpm restart

Is this standard enough and are those the kind of things you can/should
typically do as part of a provision script such as ours?

// Micke

On Fri, May 8, 2015 at 11:21 AM, Mikael Lindqvist [email protected]
wrote:

Yep! It worked!

Starting to like vagrant actually!

Will now look for a way to get pam_auth to work...

What I did before was just "apt-get install php5-pam-auth" but this is
what is deprecated? This one:

http://packages.ubuntu.com/search?keywords=php5-auth-pam

Noticing that there is no Trusty there, so guess so...

So what I can do now is to ssh into the provisioned machine and hack
around, right?

// Micke

On Thu, May 7, 2015 at 3:59 PM, Brad Smith [email protected]
wrote:

btw, the tests completed successfully and I pushed the changes, so if you
do a git pull and reprovision a swag_dev vm, you should end up with a
working (but not configured) learninglocker, and working (but not able to
log in) swagportal. Progress! :D

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

Yeah, those all look like things I can automate. :) I set it up manually and now I'm getting Authentication failure (in pam_authenticate) too, though. I think maybe pam its self has to be configured more. Here's what I see in the logs:

May  8 11:51:00 vagrant-ubuntu-trusty-64 ool www: pam_unix(php:auth): authentication failure; logname= uid=33 euid=33 tty= ruser= rhost=172.28.128.1  user=teacher
May  8 11:51:28 vagrant-ubuntu-trusty-64 unix_chkpwd[29992]: check pass; user unknown
May  8 11:51:28 vagrant-ubuntu-trusty-64 unix_chkpwd[29992]: password check failed for user (teacher)

and yet if I run su - teacher and enter the same password it works. I may not have time to investigate today, but when I do I'll also at least add the automation steps we've figured out so far.

[limikael]

Ok will investigate as well... Btw, what is the password for the teacher user? (is it public?)

[usernamenumber]

Well, it shouldn't be, but it's currently revealed in a comment in one of the config files, soo... ;)

Until I implement something that lets (forces?) you to specify a custom pw, it's Tunapanda2.

[usernamenumber]

You can also set your own pw for any user in /etc/passwd with passwd USERNAME (you probably knew that, but just in case)

[limikael]

Of course... I feel stupid now... :s :)

On Fri, May 8, 2015 at 6:03 PM, Brad Smith [email protected] wrote:

You can also set your own pw for any user in /etc/passwd with passwd
USERNAME (you probably knew that, but just in case)

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

Nah, it takes a bit to get used to the fact that Vagrant gives you a more-or-less disposable machine where you don't have to worry so much about messing around with things. But once you do, it can be really cool. :)

[limikael]

I read up a bit here:

http://svn.php.net/viewvc/pecl/pam/trunk/README?view=markup

Where it says that any program that want to authenticate needs to be able
to read /etc/shadow... :( This makes sense actually, how would it otherwise
work? And when I did:

sudo chmod 755 /etc/shadow

Then it worked... When I think about it, actually that's what I did before
to get it to work also... So... I don't think we want to have our systems
like that, right?

So plan b... What about if we create an external command line program with
the suid bit that does the authentication, and then the webserver calls
that? I can create that command line program in php also...

// Micke

On Sat, May 9, 2015 at 1:11 AM, Brad Smith [email protected] wrote:

Nah, it takes a bit to get used to the fact that Vagrant gives you a
more-or-less disposable machine where you don't have to worry so much about
messing around with things. But once you do, it can be really cool. :)

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

Actually, I think in the short-term I'm in favor of just doing the chmod as part of the provisioning. These are just disposable dev boxes and I'd rather get to the point where people can actually start developing on them sooner. In the mid/long-term, x2go is working on a login gateway that can CrUD accounts across an arbitrary set of sources through a single interface, which would mitigate the need for using PAM at all.

I also have some good news about learninglocker! I just pushed an update that pre-populates its db with a user. On a VM it will be [email protected] with the pw Tunapanda2 (see also: this issue).

It was supposed to also create an LRS, but that doesn't seem to have happened, so that's the next thing to investigate.

[limikael]

Cool!

Then I'll consider it that we have a solution for that for now...

Let me know if I can help out with the automation scripts, would like to
learn about that!

And I'll go work on the facebook integration now...

// Micke

On Sat, May 9, 2015 at 4:14 PM, Brad Smith [email protected] wrote:

Actually, I think in the short-term I'm in favor of just doing the chmod
as part of the provisioning. These are just disposable dev boxes and I'd
rather get to the point where people can actually start developing on them
sooner. In the mid/long-term, x2go is working on a login gateway that can
CrUD accounts across an arbitrary set of sources through a single
interface, which would mitigate the need for using PAM at all.

I also have some good news about learninglocker! I just pushed an update
that pre-populates its db with a user. On a VM it will be
[email protected] with the pw Tunapanda2 (see also: this
issue tunapanda/provision#59).

It was supposed to also create an LRS, but that doesn't seem to have
happened, so that's the next thing to investigate.

—
Reply to this email directly or view it on GitHub
tunapanda/wikonnect#5 (comment).

[usernamenumber]

It looks like the dev environment setup is now working smoothly except for still requiring one manual setup step for Learning Locker, which is covered in #8. Would anyone object to closing this issue now?

[limikael]

Do it! I'll create a separate issue for automating the LearningLocker setup...

[limikael]

tunapanda/wikonnect#9