Comment notifications held in queue and being sent out in small batches

[bobinoz]

Hi everybody

I have installed Comment Mail and tested it for a few days now and noticed comments are being held in the queued (pending) notifications area with the message 'n/a; awaiting processing'. They are getting sent, there just seems to be a long delay.

After I had installed and activated Comment Mail, I started answering comments on various pages with different numbers of subscribers. It sent out 76 emails immediately but then after that it started holding them in the queue for processing. What it seems to be doing since then is sending out little batches of emails every now and then at around 10 or 20 minute intervals. The most emails it seems to send out from the queue any one time is 21. Sometimes it is as few as 5. The maximum gap between sending them appears to be 30 minutes.

In the space of about seven hours, it has managed to send out all 363 emails in the queue which is now empty as it's late at night here and new comments have stopped coming in. If it were a busy day though, I can imagine the queue just building up until eventually it could take seven or eight hours before catching up with itself. I am using SparkPost to deliver email, they do not have any hourly limits. As far as I'm aware, they just have a daily limit of 10,000 emails.

I've checked the settings in Comment Mail → Config. Options → Queue Processor Adjustments, but cannot make changes as that's only available in the pro version. However, the settings seem good to me, if I am reading it correctly it should send 100 emails at a time and do that every five minutes.

Any idea why it isn't able to do that? Any help you can offer me with this would be much appreciated, many thanks, Bob

[raamdev]

@bobinoz Thanks for opening a separate issue for this.

@jaswsinc Anything come to mind for this?

[jaswrks]

@bobinoz Thank you for that detailed report.

Debug Instructions

I'd like to ask if you can download this updated file and upload it via FTP, allowing it to override your existing copy of: wp-content/plugins/comment-mail/src/includes/classes/QueueProcessor.php

With that new file in place, you should then find that Comment Mail records each queue processing event by leaving behind a log file here: wp-content/cm-debug.log. If you can let Comment Mail run for half a day (or a full day), then upload your cm-debug.log and share it with us here, that would really help.

Once you have collected the contents of cm-debug.log you can delete that file and/or revert back to the original copy of wp-content/plugins/comment-mail/src/includes/classes/QueueProcessor.php

[bobinoz]

Okay, I have uploaded the PHP file to the path you have suggested after renaming the existing queueprocessor.php file by putting an X in front of it. I did this through my cPanel as I do not have an FTP client installed on this PC. Hope that is okay.

How do I find the wp-content/cm-debug.log though? If I go to www.mywebsite.com/wp-content/cm-debug.log I get a 404. So how can I access the log? As you've probably guessed, I'm not good at the techie stuff.

[aptharsia]

@jaswsinc Uploaded the file and after a 17 hour timespan, 20 emails were sent out (despite over a hundred in queue). I looked for the cm-debug.log file and it's not in my wp-content folder.

[jaswrks]

@aptharsia If the cm-debug.log file does not exist at all, it could be that your web server does not have permission to create and write to that log file. To help things out, you can log in via FTP and create wp-content/cm-debug.log and set file permissions to 666 please.

[bobinoz]

So I uploaded the PHP file through cPanel as mentioned above, and as also mentioned, I couldn't find where to view the log. Then I saw your reply to aptharsia and createdwp-content/cm-debug.log and set file permissions to 666, again doing that through cPanel.

Then I tried to answer a comment live on my website and got the following error page...

The www.bobinoz.com page isn’t working

www.bobinoz.com is currently unable to handle this request.
HTTP ERROR 500

So I have deleted the debug PHP file and reinstated the original QueueProcessor.php, so at this stage I am not able to provide you with any debug information. If you can tell me how to fix this, I'll have another go.

[aptharsia]

@jaswsinc Created the file and had a batch of 15 go out but nothing was written to the file. Set to 666.

[bobinoz]

@jaswsinc

Just a quick update; as I thought, this problem just builds and builds. I now have 693 comments held in the queue, that represents a 12 hour backlog. As people keep commenting it could get to a stage where it simply cannot catch up with itself.

Thought you might like to also know that whilst most of them say...

n/a; awaiting processing

Under the Holding Until section, a few of them are now saying...

Nov 19, 2016 4:53 am
(11 hours) ~ part of a digest

[bobinoz]

@jaswsinc Okay, I think this is a significant development. As I just mentioned, I was staring at a 693 comment queue that would have taken 12 hours to clear. As a test, I decided to turn off SparkPost which I can do via the plug-in I am using and instead send emails through my own hosts server.

The reason I use SparkPost is because I often exceed my hosts limits, but I do know it's okay to go through them now and then. Anyway, the result has been significantly different.

Through this server 77 emails got delivered from this queue every 30 minutes on the dot. Regular as clockwork. Within five hours the whole queue was gone compared to the 12 hours it would have taken with the dribs and drabs sending through SparkPost mentioned above in my first post.

This arrangement is perfect for my hosts, because they have a 400 email an hour limit, with this plug-in only sending 154 emails and hour, I won't be exceeding that.

Anyway, the point is obvious, maybe this problem isn't with Comment Mail, maybe it with the email server.

[aptharsia]

I don't have SparkPost. My host says my email server is limited to 500 emails an hour but I don't hit that per hour.

[raamdev]

@bobinoz Can you link us to the SparkPost plugin you were using?

[aptharsia]

Ok, here's a strange occurrence. When I make a comment as admin, small batches of notifications are sent. Tested it 4 times and each time about 14 notifications are sent out.

Edit: Sent out all pending comments by making comment as admin (worked on any post.) That suppose to happen?

[jaswrks]

@bobinoz Thank you for the follow-ups. Very helpful.

Based on the details you've given here, I would recommend that you bump the Max Execution Time setting in Comment Mail from 30 to 300. That is, assuming the following are true:

• You have Comment Mail Pro; i.e., you have the ability to alter that setting.

• And, your installation of PHP has a max_execution_time >= 300.

  If your max_execution_time in PHP is lower than 300 (e.g., maybe 120), set Comment Mail to whatever that value is so that Comment Mail is able to take advantage.

  Note: The default value of 30 is conservative. It matches the default value for PHP. If your installation of PHP allows more than 30, tell Comment Mail about this and then each queue runner will be able to process more outgoing mail in a shorter amount of time.

---

You can find out what max_execution_time is set to in your installation of PHP by creating a PHP file that contains this bit of code. Open that file in a browser and review the configuration settings. Look for the max_execution_time setting.

Create: /phpinfo.php

<?php
phpinfo();

[jaswrks]

@raamdev See my previous post. I propose that we bump the default value of 30 to a default value that matches the current max_execution_time in PHP, not to exceed a 120 second default maximum. Then let site owners adjust it from there on their own.

I say that because from the reports coming in here, it almost sounds like the issue could be attributed to theme/plugin overhead on some sites. For example, imagine a site that is running 25 other plugins, one of those creates a bottleneck, and then Comment Mail's Queue Processor runs quietly behind-the-scenes via WP Cron. If the entire WordPress installation is bogged down, Comment Mail will be bogged down also. In such a scenario, 30 may only be enough time to process 5-10 emails. That's an example of a very inefficient PHP installation, but it happens.

Another example would be a case where the underlying mail server (e.g., SparkPost plugin, WP Mail SMTP configuration, etc) have been configured to use an external SMTP server that responds slowly at times, or responds slowly all the time. Here again, 30 might be too conservative.

Another example would be a very slow MySQL connection; e.g., database bogged down.

[jaswrks]

@aptharsia writes...

Ok, here's a strange occurrence. When I make a comment as admin, small batches of notifications are sent. Tested it 4 times and each time about 14 notifications are sent out.

That sounds like the expected behavior to me. When you leave a comment as admin it is instantly approved and Comment Mail's Queue Processor runs in real-time at that moment. When this happens, by default, Comment Mail will process up to 5 queued emails in real-time, based on your configuration here.

However, if emails only seem to go out whenever someone leaves a comment, that could be an indication that only real-time processing is taking place, and that the CRON-driven Queue Processor that runs every 5 minutes behind-the-scenes is not doing its job. In other words, that could be an indication that Comment Mail's Queue Processor has been unable to run via CRON for some reason. You can inspect the CRONs on your WP installation with this plugin and do a review. https://wordpress.org/plugins/wp-crontrol/

[aptharsia]

@jaswsinc I have moderated comments so I'm the only one that has instant approval. Other than when I comment nothing is sent. I have WP Crontol installed and this is what the queue processor says, "_cron_comment_mail_queue_processor None 2016-11-19 08:18:53 (58 seconds) 5 minutes"

It always says "Next Run" is 58 seconds but I don't think it ever does it until I comment. There's nothing about Comment Mail in the WP-Cron Schedules.

[jaswrks]

@aptharsia @bobinoz Regarding the cm-debug.log file. Since neither of you were able to get this working I went back through and did a review. It seems that I made a mistake with the URL that I posted earlier. I have just updated the download link in my original post to the correct version of QueueProcessor.php that creates that log file.

In summary...

Very sorry. If you can both go back and follow these instructions again it should work for you now; i.e., you should get the cm-debug.log file that you can share with us.

[jaswrks]

@aptharsia Thank you. If can keep tabs on this and report back again after a few hours I'd really appreciate it. Note also that WP CRON jobs require that your site has traffic; i.e., a CRON should run whether or not anyone leaves a comment, but WP CRON jobs do require that someone is periodically visiting some page on the site. If you see a CRON job that is seemingly stuck, hit the front page of your site a few times and it should then run as expected. That's just how WordPress works.

[aptharsia]

@jaswsinc I've got about 20k+ visitors a day so don't think that's the problem. I've reuploaded the file so will let you know if the log reports anything. Thank you.

[aptharsia]

The cm-debug file so far...just a snippet but all of them similar (these aren't being processed themselves, I'm still leaving comment as admin at various times to get emails sent out):

Version: 161118
Is CRON: false
Start Time: Sat, 19 Nov 16 13:38:06 +0000
Time Limit:
Max Limit: 25
Max Time: 10
Delay: 0
Total Entries: 2
End Time: Sat, 19 Nov 16 13:38:07 +0000
Deleted Entry Counter: 2
Processed Entry Counter: 2

Version: 161118
Is CRON: false
Start Time: Sat, 19 Nov 16 15:08:46 +0000
Time Limit:
Max Limit: 25
Max Time: 10
Delay: 0
Total Entries: 25
End Time: Sat, 19 Nov 16 15:08:51 +0000
Deleted Entry Counter: 19
Processed Entry Counter: 19

Version: 161118
Is CRON: false
Start Time: Sat, 19 Nov 16 15:10:00 +0000
Time Limit:
Max Limit: 25
Max Time: 10
Delay: 0
Total Entries: 4
End Time: Sat, 19 Nov 16 15:10:02 +0000
Deleted Entry Counter: 4
Processed Entry Counter: 4

Version: 161118
Is CRON: false
Start Time: Sat, 19 Nov 16 16:10:28 +0000
Time Limit:
Max Limit: 25
Max Time: 10
Delay: 0
Total Entries: 25
End Time: Sat, 19 Nov 16 16:10:33 +0000
Deleted Entry Counter: 16
Processed Entry Counter: 19

[jaswrks]

@aptharsia Thank you very much for providing that. So just to confirm, none of the entries in your cm-debug.log file have Is CRON: true. Is that correct?

[aptharsia]

All say FALSE, no TRUE.

[bobinoz]

@raamdev - Sure, the SparkPost plugin I have been using is...

https://wordpress.org/plugins/sparkpost/

[bobinoz]

@jaswsinc I don't have the pro version, so unable to make those changes. My queue process is on the default settings, I posted an image in my first comment on this thread, right at the top.

[bobinoz]

@jaswsinc I'm afraid pretty much the same thing happened when I downloaded your new PHP file. I uploaded this file through cPanel to the correct location and renamed the old Queue Processor by placing an x in front of it.

Uploading the new Queue Processor file did automatically create the debug log file with 666 permissions, but when I tried to answer a comment I again saw an error page, 501. So again I had to undo the changes and revert back to the original Queue Processor. My web host does have quite a few security settings, so maybe that's why I cannot get it to work?

[jaswrks]

@aptharsia writes...

All say FALSE, no TRUE.

Thank you. So that explains what the problem is in your case. The Comment Mail Queue Processor is not running via WP Cron on your installation. This gives us a research path now. We can work to identify what may prevent the Comment Mail CRON job from running in some scenarios.

Would you mind telling us who your hosting provider is?

[jaswrks]

@bobinoz writes...

I don't have the pro version, so unable to make those changes. My queue process is on the default settings, I posted an image in my first comment on this thread, right at the top.

Copy that.

when I tried to answer a comment I again saw an error page, 501.

You could try setting permissions to 644, or even 600 and see if that makes a difference. Otherwise, I'd contact your hosting provider and see if they can shed light. If it's not a permissions issue, then, as you noted, it could be a security feature on that web host I'm unaware of.

[aptharsia]

@jaswsinc Host is BigScoots. I'm on a VPS, not shared hosting.

[aptharsia]

Get this message when I manually run the queue processor in WP Crontrol:
The DISABLE_WP_CRON constant is set to true. WP-Cron spawning is disabled.

Not sure if that is what is doing it or not.

[bobinoz]

Hi guys

After nearly 2 years of this plug-in working fine, Comment Mail has stopped working again, and it would appear this problem started after I updated to WordPress 4.9.7 on 5 July. It does process four or five comments in real-time when I answer a comment, but it now never sends out comment notifications via the Cron.

I'm posting this here because as has been mentioned in the comment just above this one, when I manually run the Cron in WP-Cron Events I also get the message...

The DISABLE_WP_CRON constant is set to true. WP-Cron spawning is disabled.

In cPanel though, I have a Cron set for every 30 minutes that ends with...

public_html/wp-cron.php >/dev/null

So for the last three weeks I have been manually processing the queue, and that's been a real pain. Desperate to get this fixed if I can, and I'm happy to upgrade to pro if that will fix it. I am still using SparkPost, but even when I turn it off, the same thing happens.

Can anyone help me with this?

Many thanks, Bob