django compressor: is it doing anything? besides 500 crashing Django? #397

btbonval · 2015-02-14T21:16:58Z

While running a local VM, I got an error about some compress key missing and that I needed to run manage.py compress. This is a new issue, but it looks like compress has been around for some time.

The error might have been because I was running a database dumped from the beta system, but that assumes compress uses the database to store stateful information.

It sounds like compress is used to glob together js and css files in a tight format. That's great! But I've never run it manually before.

Does it run automatically?
Does it integrate with collectstatic so that compressed static files are actually uploaded? [Yes]
Does it integrate with collectstatic so that uncompressed static files are not actually uploaded? [No, all files are uploaded regardless of compress]
Are there any uncompressed files which ought to be compressed?

The text was updated successfully, but these errors were encountered:

btbonval · 2015-02-14T21:23:01Z

You have offline compression enabled, but key BLAH is missing from the offline manifest. You may need to run "python manage.py compress".

This error raises its head again after changing an environment variable in foreman.

yourcelf · 2015-02-15T15:20:28Z

django-compressor can run in two modes -- "offline" mode where you compress via a management command, and "online" mode where it compares a hash of the {% compress %}{% endcompress %} block contents to a key in the cache and recompiles the assets if needed. I presume we're using online mode. In that case, it'll use the cache to store an association between each compress block in the source and a file. Compiled assets are stored in static/CACHE, and are stored using the defualt file storage (which we have set to be S3). Checking the source on prod, it's clear that css and js are both being compressed there.

Additionally, by default, compressor doesn't minify and concatenate assets if DEBUG == True. If you want to test it locally, you can either set DEBUG = False or override the default with COMPRESS_ENABLED = True in settings.

Again based on casual perusal of the production source, it doesn't look like there are obviously compressible assets that we're missing at the moment.

yourcelf · 2015-02-15T15:25:12Z

Following up after reading more carefully: we might be using it in "offline mode" in prod, in which case the management command needs to be run. I'm not sure which key it's missing -- could be something for managing its connection to AWS or to another cache server (redis?) somewhere.

btbonval · 2015-02-15T16:45:45Z

@yourcelf I'm surprised you didn't bump into this issue, because you're also using a beta system database dump and the same branch of git code. Very confusing.

The prod.py settings enable both online and offline caching. When I disabled offline compression, I got errors about memcache. At first I thought it was just a fluke.

Now I bet that's the problem. I probably need to fix the memcache problem so that online mode works and the system doesn't need to fall back to offline mode. I bet that's the difference in our configurations.

yourcelf · 2015-02-15T17:19:46Z

I have memcache running on my system, and you probably didn't on your VM? Could explain my lack of issues.

btbonval · 2015-02-15T17:21:56Z

Yeah I thought I installed it, but maybe I only installed memcached-dev as a prereq and forgot memcached itself. We used to have automation to take care of this in the repo, but it got deleted (and thus was left unmaintained) after the switch to Heroku.

btbonval · 2015-02-15T20:50:02Z

I only installed memcached-dev but forgot memcached. It's now installed and running, but the error persists.

20:46:52 web.1  | Internal Server Error: /note/harvard/cs-50-114/1st-lecture
20:46:52 web.1  | Traceback (most recent call last):
20:46:52 web.1  |   File "/home/vagrant/karmanotes-web/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 138, in get_response
...
20:46:52 web.1  |   File "/home/vagrant/karmanotes-web/local/lib/python2.7/site-packages/django/template/base.py", line 844, in render_node
20:46:52 web.1  |     return node.render(context)
20:46:52 web.1  |   File "/home/vagrant/karmanotes-web/local/lib/python2.7/site-packages/compressor/templatetags/compress.py", line 147, in render
20:46:52 web.1  |     return self.render_compressed(context, self.kind, self.mode, forced=forced)
20:46:52 web.1  |   File "/home/vagrant/karmanotes-web/local/lib/python2.7/site-packages/compressor/templatetags/compress.py", line 88, in render_compressed
20:46:52 web.1  |     cached_offline = self.render_offline(context, forced=forced)
20:46:52 web.1  |   File "/home/vagrant/karmanotes-web/local/lib/python2.7/site-packages/compressor/templatetags/compress.py", line 72, in render_offline
20:46:52 web.1  |     'You may need to run "python manage.py compress".' % key)
20:46:52 web.1  | OfflineGenerationError: You have offline compression enabled but key "95213aa91f496f23f83a46b52e763231" is missing from offline manifest. You may need to run "python manage.py compress".
20:46:52 web.1  | ERROR:django.request:Internal Server Error: /note/harvard/cs-50-114/1st-lecture

I have run python manage.py compress at least 12 times now. I'd love to know why that key in particular is being sought. I can't find that key mentioned on the file system and I don't see any obvious place for it to be stored in the database.

btbonval · 2015-02-15T20:53:46Z

I disabled offline compression in prod.py. Now the page renders. That's really strange. My assumption that online was checked first, failed, and then defaulted to offline appears incorrect.

In any case, this does not effect the production systems and I appear to have found a workaround for my use for the time being. Good enough for closing this ticket.

btbonval · 2015-02-16T21:07:44Z

ehh turns out django-compress could potentially be very useful for dealing with CDN problems in #396.

At present, I do not see anything happening with compress. All the js and css appear to be posted in the S3 bucket as-is, even though COMPRESS_ENABLED = True is set in prod.py and run on my VM, beta, and prod.

So ... the question is reiterated: does it do anything? And can we make it, because that would mesh well with our CDN.

btbonval · 2015-02-16T21:11:48Z

This configuration appears good:
https://github.com/FinalsClub/karmaworld/blob/note-editing/karmaworld/settings/prod.py#L147-L151

The default storage env var is confirmed to be S3 boto on staging and my VM. The cloudfront domain on both is also correct from the environment.

btbonval · 2015-02-16T21:18:12Z

found it!

compress is pushing into the bucket under /static/CACHE/[js|css]/. I see fresh files there.

Django has been instructed to prepend /static to all file locations into and out of the static location.

karmaworld/karmaworld/settings/prod.py

Lines 111 to 114 in 3fb9188

    
           # Put static files in the folder 'static' in our S3 bucket. 
        
           # This is so they have the same path as they do when served 
        
           # locally for development. 
        
           AWS_LOCATION = 'static'

It looks like these files are working correctly with compress:

karmaworld/karmaworld/templates/base.html

Lines 11 to 28 in 3fb9188

    
           {% compress css %} 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/fontface.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/foundation.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/jquery-ui.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/font-awesome.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/opentip.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/ajax_select.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/jquery.dataTables.css"> 
        
             <link rel="stylesheet" type="text/css" media="all" href="{{ STATIC_URL }}css/global.css"> 
        
           {% endcompress %} 
        
           {% compress js %} 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/modernizr.js"></script> 
        
           {% endcompress %} 
        
           <!-- block pagestyle --> 
        
           {% block pagestyle %} 
        
           {% endblock %} 
        
           <!-- end block pagestyle -->

As taken from http://karmanotes-beta.herokuapp.com/

  <link rel="shortcut icon" href="//d2ry5v6uki5dz0.cloudfront.net/static/img/favicon.ico">
  <link rel="stylesheet" href="https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/9de46941e995.css" type="text/css" media="all" />
  <script type="text/javascript" src="https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/a52a868564de.js"></script>

  <!-- block pagestyle -->

  <link rel="stylesheet" href="https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/ac272f84448a.css" type="text/css" media="all" />

  <!-- end block pagestyle -->

btbonval · 2015-02-16T21:27:52Z

So that answers the question of "Does it integrate with collectstatic so that compressed static files are actually uploaded?". Yeah, actually it does.

It appears that "Does it integrate with collectstatic so that uncompressed static files are not actually uploaded?" is not possible. All static files are uploaded whether or not they are handled by compress. So note-detail.js is uploaded even though the only call to it is inside compress.

Here's where note-detail.js is wrapped by compress (the only appearance of note-detail.js in the codebase):

karmaworld/karmaworld/templates/notes/note_base.html

Lines 31 to 37 in 3fb9188

    
           {% compress js %} 
        
             <script src="{{ STATIC_URL }}js/setup-ajax.js"></script> 
        
             <script src="{{ STATIC_URL }}js/note-detail.js" ></script> 
        
             <script src="{{ STATIC_URL }}js/pxem.jQuery.js"></script> 
        
             <script src="{{ STATIC_URL }}js/marked.js" ></script> 
        
             <script src="{{ STATIC_URL }}js/annotator-full.min.js"></script> 
        
           {% endcompress %}

I disabled beta's cloudfront for testing. I see attempts to download the hash file names, and no attempts to download e.g. note-detail.js. This is the expected behavior, but I was not seeing it earlier.

Waiting for beta's cloudfront to come back online to test again.

btbonval · 2015-02-16T21:44:22Z

okay, cloudfront is running again, and once again I am seeing note-detail.js in the network tab of the console and in the source code view.

Is FireFox translating the hash codes into files even in the debug areas?

btbonval · 2015-02-16T21:45:47Z

answer: no. At the top of the source I'm looking at is still calling CACHE hash filenames from base.html.

btbonval · 2015-02-16T21:48:04Z

Blah this is inconsistent and annoying. The topmost base.html template is working with compress just fine, while the note_base.html template seems to be skipping compress entirely.

btbonval · 2015-02-16T21:50:22Z

Also jQuery seems to be missing.

btbonval · 2015-02-16T21:55:11Z

jQuery should be compressed after the footer.

karmaworld/karmaworld/templates/base.html

Lines 45 to 71 in 3fb9188

    
           <!-- end include footer--> 
        
           {% compress js %} 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/jquery-2.1.0.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/jquery-ui-1.10.4.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/fastclick.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/opentip-jquery.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/jquery.cookie.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/foundation.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/foundation.joyride.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/foundation.tab.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/foundation.alert.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/foundation.topbar.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/foundation.reveal.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}ajax_select/js/ajax_select.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}ajax_select_cascade/js/ajax_select_cascade.js"></script> 
        
             <script type="text/javascript" src="{{ STATIC_URL }}js/jquery.dataTables.js"></script> 
        
           {% endcompress %} 
        
           <script type="text/javascript" src="//use.typekit.net/pnc8diw.js"></script> 
        
           <script type="text/javascript">try{Typekit.load();}catch(e){}</script> 
        
           <script> 
        
             $(function() { 
        
               $(document).foundation(); 
        
             }); 
        
           </script> 
        
           <!-- block pagescripts -->

In the page source I'm looking at:

  <!-- end include footer-->

  <script type="text/javascript" src="https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/b96ad19ec98b.js"></script>

  <script type="text/javascript" src="//use.typekit.net/pnc8diw.js"></script>
  <script type="text/javascript">try{Typekit.load();}catch(e){}</script>
  <script>
    $(function() {
      $(document).foundation();
    });
  </script>
  <!-- block pagescripts -->

It looks like jQuery is crammed in there, but either it is not loaded at the right time or it is not in the correct namespace. I assume this is a problem with compress.
https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/b96ad19ec98b.js

btbonval · 2015-02-16T21:59:04Z

These problems do not appear on http://karmanotes-beta.herokuapp.com/note/harvard/cs-50-114/1st-lecture. It appears that jquery is crammed in the same place and even using the same cloudfront hash; that did not get updated. So it looks like missing jquery is a separate issue entirely from compress.

 <!-- end include footer-->

  <script type="text/javascript" src="https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/b96ad19ec98b.js"></script>

  <script type="text/javascript" src="//use.typekit.net/pnc8diw.js"></script>
  <script type="text/javascript">try{Typekit.load();}catch(e){}</script>
  <script>
    $(function() {
      $(document).foundation();
    });
  </script>
  <!-- block pagescripts -->

btbonval · 2015-02-16T22:02:51Z

forked jquery issue to #400 as it now appears unrelated to compress.

btbonval · 2015-02-16T23:10:06Z

ah, some of the weirdness with compress might be due to manifest.json. While everything else is versioned, manifest.json is statically hosted but is not versioned.
http://stackoverflow.com/questions/15044399/django-compressor-cloudfront-and-manifest-json

It seems like the front end never uses manifest.json, and the backend can get around caching by accessing the S3 bucket directly using the default storage.

btbonval · 2015-02-16T23:25:10Z

darnit. S3 and Cloudfront appear to match for manifest.json, so my theory seems to be wrong.

http://karma-beta.s3.amazonaws.com/static/CACHE/manifest.json

{
  "442c18c2a05bb4d3fd13ee627a5c7417": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/febe1bc0c6e9.css\" type=\"text/css\" media=\"all\" />",
  "8814067983406d53beedaae5667ace44": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/9baa4331d306.js\"></script>",
  "97b36458b6db8b3e32dc5a8723c35913": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/cc9da0afb61c.css\" type=\"text/css\" media=\"all\" />",
  "a7faab5496fe386820911dbc1b2d1d16": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/a52a868564de.js\"></script>",
  "bcb3f4d38cd2f812acfa34b2b280c967": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/b96ad19ec98b.js\"></script>",
  "82d94a4224b72da20caf3a1be3a6d54d": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/ac272f84448a.css\" type=\"text/css\" media=\"all\" />",
  "efdec72c75dd83fb57fc636dc8cd4040": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/1c9098619b8b.js\"></script>",
  "86d6e666af9057070c90cfd1c6827b4b": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/c65cf523ca3b.css\" type=\"text/css\" media=\"all\" />",
  "d4b49aaae233b114a3a190733be902b5": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/a0cebb0c3c74.js\"></script>",
  "0b7b58e7f7597cee10588304e025d150": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/7c08411ba945.css\" type=\"text/css\" media=\"all\" />",
  "88e86e7fea3c44b253cf997fad40758d": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/db6299073c6f.css\" type=\"text/css\" media=\"all\" />",
  "d71b198ed04ac08b1724ed508ff86911": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/fc79bd7d88e1.js\"></script>"
}

https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/manifest.json

{
  "442c18c2a05bb4d3fd13ee627a5c7417": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/febe1bc0c6e9.css\" type=\"text/css\" media=\"all\" />",
  "8814067983406d53beedaae5667ace44": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/9baa4331d306.js\"></script>",
  "97b36458b6db8b3e32dc5a8723c35913": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/cc9da0afb61c.css\" type=\"text/css\" media=\"all\" />",
  "a7faab5496fe386820911dbc1b2d1d16": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/a52a868564de.js\"></script>",
  "bcb3f4d38cd2f812acfa34b2b280c967": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/b96ad19ec98b.js\"></script>",
  "82d94a4224b72da20caf3a1be3a6d54d": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/ac272f84448a.css\" type=\"text/css\" media=\"all\" />",
  "efdec72c75dd83fb57fc636dc8cd4040": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/1c9098619b8b.js\"></script>",
  "86d6e666af9057070c90cfd1c6827b4b": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/c65cf523ca3b.css\" type=\"text/css\" media=\"all\" />",
  "d4b49aaae233b114a3a190733be902b5": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/a0cebb0c3c74.js\"></script>",
  "0b7b58e7f7597cee10588304e025d150": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/7c08411ba945.css\" type=\"text/css\" media=\"all\" />",
  "88e86e7fea3c44b253cf997fad40758d": "<link rel=\"stylesheet\" href=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/css/db6299073c6f.css\" type=\"text/css\" media=\"all\" />",
  "d71b198ed04ac08b1724ed508ff86911": "<script type=\"text/javascript\" src=\"https://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/js/fc79bd7d88e1.js\"></script>"
}

btbonval · 2015-02-16T23:28:21Z

I just noticed the key changed. Earlier I pasted this:

20:46:52 web.1  | OfflineGenerationError: You have offline compression enabled but key "95213aa91f496f23f83a46b52e763231" is missing from offline manifest. You may need to run "python manage.py compress".

now I'm getting this:

22:57:31 web.1  | OfflineGenerationError: You have offline compression enabled but key "9eaf52d2274436d3f29aa9ad5398c55e" is missing from offline manifest. You may need to run "python manage.py compress".

It looks like I'll need to run foreman run python manage.py compress every time I change branches or change the code? If so, this needs to be part of deploy code if it isn't already.

btbonval · 2015-02-16T23:29:22Z

nope, didn't help. Alright, so the key has changed, but running foreman run python manage.py compress doesn't fix the problem. Perhaps that plus collectstatic are required in tandem?

btbonval · 2015-02-18T03:59:53Z

PS after all the compress running and such, I just tested that link from my local server. Still 500 with the error: 03:58:25 web.1 | OfflineGenerationError: You have offline compression enabled but key "9eaf52d2274436d3f29aa9ad5398c55e" is missing from offline manifest. You may need to run "python manage.py compress".

Looking at the manifest above, it certainly is missing. I'm not sure how many times I need to run manage.py compress to get that key in there :(

btbonval · 2015-02-18T04:00:53Z

Just to confirm I'm cleanly in the branch:

(karmanotes-web)vagrant@vagrant-ubuntu-trusty-64:/vagrant$ git status
On branch note-editing
Untracked files:
  (use "git add <file>..." to include in what will be committed)

    .vagrant/
    Vagrantfile

nothing added to commit but untracked files present (use "git add" to track)
(karmanotes-web)vagrant@vagrant-ubuntu-trusty-64:/vagrant$ git log | head
commit 4736b4b0994b07623ed5562090ddec9c8b3e2f4e
Author: Charlie DeTar <[email protected]>
Date:   Sat Feb 14 16:27:17 2015 -0700

    Updates to sanitizer; all-in with inline HTML

     - Remove iframe and "static hosting" as a strategy for showing a note.
       Only show inline HTML.
     - Add a "format preserving" sanitizer that does XSS filtering and prep
       for inline HTML, but doesn't remove visual markup.

yourcelf · 2015-02-18T04:02:03Z

the key the error references is not in the manifest.json I generated locally.

FWIW, I run django-compressor in 'online' mode for all of my production sites. The only time the performance is any different is when generating files for the first time with an empty cache.

Did you figure out the train you were following earlier of checking whether the contents of manifest.json can change if the filename doesn't? Was that just cloudfront, or was it s3 too?

btbonval · 2015-02-18T04:06:28Z

The manifest.json I pasted above is what is currently on S3, after running compress and so forth. It turns out that it is the same manifest.json on CloudFront, so nothing has changed.

I can confirm that I am pushing code up to S3 because it is hosting the new .js files and I see current timestamps. e.g. /static/CACHE/manifest.json is showing me a timestamp of about 9-10 minuts ago.

Here's the S3 version (10 minutes old):
http://karma-beta.s3.amazonaws.com/static/CACHE/manifest.json

Here's the CloudFront version:
http://d2ry5v6uki5dz0.cloudfront.net/static/CACHE/manifest.json

btbonval · 2015-02-18T04:11:42Z

I wonder if I can inject a printout to determine what is the context that is generating this missing key.
https://github.com/django-compressor/django-compressor/blob/369065c6a7070b9395d4bbfd6cb61da3ed59b62f/compressor/templatetags/compress.py#L66

yourcelf · 2015-02-18T04:18:14Z

My best guess: when executing manage.py compress, django-compressor is encountering and swallowing an error when trying to parse the files to compress. This is resulting in a file not getting added to the manifest. Then when running with COMPRESS_OFFLINE = True, it's looking for a file that should be there that doesn't exist.

This would explain both the missing key errors and the fun error that I got above. My error shows up if COMPRESS_OFFLINE = False, and your error shows up if COMPRESS_ONLINE = True (I can confirm that's the case locally).

btbonval · 2015-02-18T04:20:50Z

Well I output the context, but man, that's a lot of noise and very little signal. I think I might pdb this and step through what it's doing to the context to figure out what is actually being hashed.

yourcelf · 2015-02-18T04:21:57Z

Switching the wysiwyg to reference the non-minified version of its source file eliminates my TemplateSyntaxError.

btbonval · 2015-02-18T04:24:01Z

pdb had no output :(

I was hoping to dive into this interactively to determine just what is missing.

btbonval · 2015-02-18T04:28:21Z

this is the chunk that is being choked on:

04:27:01 web.1  |     <link rel="stylesheet" type="text/css" media="all" href="//d2ry5v6uki5dz0.cloudfront.net/static/css/note_course_pages.css">
04:27:01 web.1  |     <link rel="stylesheet" type="text/css" media="all" href="//d2ry5v6uki5dz0.cloudfront.net/static/css/annotator.min.css" />
04:27:01 web.1  |     <link href="//d2ry5v6uki5dz0.cloudfront.net/static/wysihtml5/toolbar.css" type="text/css" media="all" rel="stylesheet" />

wysiwyg is definitely in there, and it's the only thing "new" to this code base in general. It sounds like you found the problem, although I wish I understood why.

yourcelf · 2015-02-18T04:30:43Z

Though it's not really fixed. Commenting out the wysihtml5 js doesn't fix the problem when running with COMPRESS_OFFLINE = True -- manage.py compress runs as expected; new manifest and files generated, but still missing the manifest.

I think the TemplateSyntaxError is just bugs in the jsmin parser... but the missing manifest entry is a different problem.

yourcelf · 2015-02-18T04:31:25Z

(ps I upgraded django-compressor locally to 1.4; but no improvement)

yourcelf · 2015-02-18T04:31:51Z

turning in for the night. Good luck. This one's not cracking easily.

btbonval · 2015-02-18T04:32:47Z

@yourcelf good night, thanks for your help. I would have fumbled around with the wrong project, so you saved me going down the wrong path.

yourcelf · 2015-02-18T04:35:13Z

There is this: django-compressor/django-compressor#443

btbonval · 2015-02-18T04:35:58Z

Given the chunk of three CSS files that are causing the missing key error, I noticed that they are all referencing cloudfront. I figured I'd see what's actually there.

http://d2ry5v6uki5dz0.cloudfront.net/static/wysihtml5/toolbar.css is a blank file.

The other two have file contents.

btbonval · 2015-02-18T04:45:57Z

toolbar.css is empty in the repo.
https://github.com/FinalsClub/karmaworld/blob/note-editing/karmaworld/apps/wysihtml5/static/wysihtml5/toolbar.css

I don't know if it is supposed to be empty, but I'm going to remove it from the {compress} to see if it fixes the compress error.

This is apparently where the empty file is coming from: {{ note_edit_form.media.css }}, so I'll comment it out.
https://github.com/FinalsClub/karmaworld/blob/note-editing/karmaworld/templates/notes/note_base.html#L9-L15

btbonval · 2015-02-18T04:54:43Z

woo! moved past that error. new one.

04:54:20 web.1  | OfflineGenerationError: You have offline compression enabled but key "1ea81454b4926d592f670f0172e68ef4" is missing from offline manifest. You may need to run "python manage.py compress". Also forced False and context 
04:54:20 web.1  |     <script src="//d2ry5v6uki5dz0.cloudfront.net/static/js/setup-ajax.js"></script>
04:54:20 web.1  |     <script src="//d2ry5v6uki5dz0.cloudfront.net/static/js/note-detail.js" ></script>
04:54:20 web.1  |     <script src="//d2ry5v6uki5dz0.cloudfront.net/static/js/pxem.jQuery.js"></script>
04:54:20 web.1  |     <script src="//d2ry5v6uki5dz0.cloudfront.net/static/js/marked.js" ></script>
04:54:20 web.1  |     <script src="//d2ry5v6uki5dz0.cloudfront.net/static/js/annotator-full.min.js"></script>
04:54:20 web.1  |     <script type="text/javascript" src="//d2ry5v6uki5dz0.cloudfront.net/static/wysihtml5/wysihtml-0.4.17/dist/wysihtml5x-toolbar.min.js"></script>
04:54:20 web.1  | <script type="text/javascript" src="//d2ry5v6uki5dz0.cloudfront.net/static/wysihtml5/wysihtml-0.4.17/parser_rules/advanced_and_extended.js"></script>
04:54:20 web.1  | <script type="text/javascript" src="//d2ry5v6uki5dz0.cloudfront.net/static/wysihtml5/init.js"></script>
04:54:20 web.1  |

btbonval · 2015-02-18T04:58:52Z

not a single empty file in that collection.

btbonval · 2015-02-18T20:37:33Z

All three wysihtml5 includes are coming from {{ note_edit_form.media.js }}
https://github.com/FinalsClub/karmaworld/blob/note-editing/karmaworld/templates/notes/note_base.html#L38

I'll comment that out just to see if I can avoid a 500 error with compress.

yourcelf · 2015-02-18T20:40:06Z

Try commenting the CSS out too; it's also handled by compress.

btbonval · 2015-02-18T20:41:43Z

you mean this?
https://github.com/FinalsClub/karmaworld/blob/note-editing/karmaworld/templates/notes/note_base.html#L13

I commented that out back in this comment: #397 (comment)

yourcelf · 2015-02-18T20:43:33Z

yup, that one. :)

btbonval · 2015-02-18T20:44:51Z

Alright commenting out both media includes allows the page to load without a 500 error from compress.

Perhaps we should move those media includes outside of compress so that they are available.

The CSS file was empty, so I could understand why that might cause an issue. The three JS files look alright to me although I didn't look closely.

btbonval · 2015-02-18T20:56:51Z

Alright I pulled those media tags out of the compress tags. They are being included in the source (albeit directly from static and not inside a compress hash file) and the server isn't throwing a 500.

Visiting /note/harvard/reason-and-faith-in-the-west/resumepdf-5-27-43979, I am seeing an error on the console:

TypeError: m.getComputedStyle(...) is null wysihtml5x-toolbar.min.js:5

I think this is one of the examples @yourcelf had showed me interactively, but with the error, I don't see anything where the note should be. Unsure if that is due to the changes made with how compress works (and thus should stay in this ticket) or if it is worthy of its own ticket.

btbonval · 2015-02-18T21:03:14Z

Alright, I think this issue is long and worked around. No reason to keep it open.

I created a ticket to follow up with django-compressor and wysihtml5. #401

I created a ticket to troubleshoot the wysihtml5 typeerror that doesn't belong in this ticket. #402

btbonval added enhancement question DevOps labels Feb 14, 2015

btbonval mentioned this issue Feb 15, 2015

testing data migrations of editable notes feature #396

Closed

btbonval closed this as completed Feb 15, 2015

btbonval reopened this Feb 16, 2015

btbonval mentioned this issue Feb 16, 2015

jquery not found when running note-editing branch in prod-like environment #400

Closed

btbonval mentioned this issue Feb 18, 2015

drop offline caching from django-compressor to support compressing dynamic includes #401

Open

btbonval closed this as completed Feb 18, 2015

btbonval added a commit that referenced this issue Feb 18, 2015

pulling wysihtml5 out of compress to stop key errors for #397

14fb885

btbonval added a commit that referenced this issue Feb 27, 2015

pulling wysihtml5 out of compress to stop key errors for #397

ccee821

django compressor: is it doing anything? besides 500 crashing Django? #397

django compressor: is it doing anything? besides 500 crashing Django? #397

Comments

btbonval commented Feb 14, 2015

btbonval commented Feb 14, 2015

yourcelf commented Feb 15, 2015

yourcelf commented Feb 15, 2015

btbonval commented Feb 15, 2015

yourcelf commented Feb 15, 2015

btbonval commented Feb 15, 2015

btbonval commented Feb 15, 2015

btbonval commented Feb 15, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 16, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

yourcelf commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

yourcelf commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015

btbonval commented Feb 18, 2015