Feature Request: Delay transform for first view, but queue

[meredevelopment]

I would like to see an option to return the original image if a transform doesn't exist, but still queue/create the transforms for next time it's requested.

I believe it would behave a bit like the existing 'noop' setting.

Use case: We have a large site which gets 300>600 page views a minute. When new assets are uploaded, the first time they are viewed the users have to wait 10-20 seconds for the transforms to run. When a lot of images are put live at once, this can make page loads take even longer! We have Workers setup to handle the Queue, which is fine. This option would allow us to deliver pages quicker to everyone.

Please let me know if you have questions or work-arounds. Thanks!

[satcreative]

I agree. This would definitely benefit my projects that are using Imager. + 1 from me.

[cookie10codes]

Following :-) +1

[wblommaert]

Would love to see something like this. Some clients are photographers and have a lot of images on a single entry which causes timeouts all over the place when rendering responsive transforms for each of those.

I need to add that Preparse fields are not an option anymore since there is a lot of existing content. That cannot be easily rerendered without causing timeouts galore. Having this feature request's functionality would ease the burden on the web server.

[aelvan]

Although I see the allure of a feature like this, I've yet to see/come up with a solution that would fit into Imager without coming with major drawbacks that would either make it hard to use, not scale, or be likely to generate a ton of support. But I'm open to suggestions, so let's look at the options:

---

The option (let's call it 1) that @meredevelopment suggests - outputting the original image if the transform hasn't been generated - is not a viable solution. In addition to the fact that returning original images that could be several megabytes each would be really bad, I'm guessing it would break most sites design if images would randomly be displayed to the user without any cropping or effects applied to them. Unless you used intrinsic ratios, object-fit and overflow hidden on all containers that contain images, things would look crap and/or layouts would break. Also, if a site uses template caching (which it should), the original image url could be cached, which would mean that it wouldn't just be the first request that receives the original image, but all requests until the cache is broken.

So yeah, I don't really see this being a viable solution, except in a few edge cases where everything aligns.

---

Another option (2) is implementing it in a similar way to what the native Craft transforms do (the behaviour is toggled on/off with generateTransformsBeforePageLoad in the general config). What happens when that's set to false is that when a transform is run, Craft 1) adds a job to the queue that will (eventually) create the transform, 2) a url for a generate transform controller action is output in the markup instead of the url for the transformed image, and 3) a piece of javascript is injected into the page to kick off the queue (because that normally wouldn't happen unless someone logged into the control panel, or you have a cron job or daemon set up to trigger/listen to the queue).

Obviously this will fail if you load the markup with ajax, or in a way that keeps the javascript from running - the queue will not be started, and the controller action will not run.

The controller action is made in a way that's not really scalable; a separate PHP process is opened for each image request, and it waits (using sleep()) and checks every second (with a database query) to see if the transform has been created. If it hasn't been after 30 seconds, the controller tries to force-generate it. In a scenario where you actually have a ton of images that needs to be generated, and/or a high-traffic site, the result would be that a lot of visitors will get pages without images, and your server will have to handle a ton of PHP requests.

P&T has some secret sauce somewhere that keeps template caching from working if there are pending image transforms within a cached block. Which is great, except it means that template caching is disabled at the same time as your site is busy doing image transforms. Also, any front-side caching (Cloudflare, Varnish, etc) you have, wouldn't know about these url's and would cache them, meaning a controller action would be hit on every request instead of your images.

Although this solution has some major short-comings, it's probably still the best plug'n'play solution, since it doesn't require any additional setup on the server. I'd need to figure out how to store the information about the transforms that are being generated; Craft is keeping track of it in the database, but Imager doesn't use the database at all at the moment, and I would prefer not to.

---

One option (3) that's already in Imager, is off-loading the transforms to an external service. At the moment there're only support for Imgix (which is an awesome service), but the next major version of Imager will have support for custom transformers, so you could write an interface to whatever service you'd like to use. For high-traffic sites, this is the only viable solution in my opinion. Having the webserver generate images is just a really, really bad idea, neither PHP, nor most web servers, are even remotely optimised to generate image transforms. I understand that people want to save the money/reduce the number of services they use, though... (even though, Imgix is so worth it).

---

Next option (4) is to treat your web server like an external service, using something like Glide. Like with Imgix, a url with the necessary transform parameters would be generated, and the url would transparently generate the transform, or return an existing image. Behind the scenes there'd be a controller action that picks up the parameters, checks if the transform already exists, and either returns the image or creates it.

This is a tempting solution, but requires that the web server is properly configured, or you'd risk running a PHP request on every image request. It might be best to implement this as a separate custom transforme since it'd have to support a subset of what Imager's Craft transformer currently does (it wouldn't work with external storages for instance).

Also, it still requires your web server to generate all those transforms the first time. If you have a page with 50 images, there'll be 50 PHP requests (pr user request), that'll compete for the resources on the server.

---

And there's probably an option (5) that involves outputting placeholders instead of the actual transforms, creating queue jobs for the transform, and having some advanced javascript injected into the page that checks to see if the actual images have been generated, and update them. Hacky and fragile.

---

So yeah, that are some of the options, and as I said, I don't think any of them are especially good. And all of them will most likely come with a ton of extra support requests, so there's that too... But, open to suggestions! Hopefully (and most likely) there're someone smarter than me out there. :)

Btw, the next version of Imager will have functionality to auto-generate transforms on file upload (like Imager Pretransform) or on element save (like Preparse Field, but a lot easier to setup), and console commands and element actions to batch generate transforms. This will probably ease the pain somewhat, but... if a user hits a cold cache the delay will be the same as before.

[JoshCoady]

Ultimately, the reason there is no good solution is because the requested feature is more of a workaround and does not actually address the real issue -- if you want a fast site, then the images should already be generated before the first view. If you are waiting for the first view to trigger the image resize, then you are already too late -- nothing will make that faster than sites that pregenerate images (and pages). Prime your caches, dont let your users do it for you :P