Duplicator version update breaks it
-
Backups are failing after updating Duplicator version. Problem is that it can’t reach the AJAX endpoint (loopback) and local control is failing. (Backups hang.) I disabled all other plugins including security. I deactivated Cloudflare DNS. I tried all debugging steps in the knowledge base. Please fix quickly, this is messing up my deadlines.
The page I need help with: [log in to see the link]
-
By the way I have Dreamhost and they did something to verify the loopback was working:
To check the current state, I ran a live test against the exact endpoint
Duplicator uses:
curl -s -o /dev/null -w “HTTP %{http_code}\n”
“https://www.learnwithmikemossey.com/wp-admin/admin-ajax.php?action=duplicator_process_worker&duplicator_kprobe=test“
HTTP 200
That 200 response means the loopback endpoint is responding correctly
right now.Here’s more information from tech support at Dreamhost: (note that I tested with all other plugins disabled)
WordPress 7.1
PHP 8.3.30
Duplicator 5.0.2
I also checked the previous Duplicator logs and confirmed that you were
already using DupArchive, so there is no need to switch the archive
engine.
ARCHIVE Type=DUP Mode=DupArchive
I then tested the same admin ajax endpoint that Duplicator uses for its
loopback request.
The direct request completed successfully.
AJAX_HTTP=200
TTFB=1.805659s
TOTAL=1.805945s
AJAX_BODY=ok
The same test performed through the WordPress HTTP API also completed
successfully.
WP_LOOPBACK=PASS
HTTP=200
TIME=1.683s
BYTES=2
I also checked the PHP FastCGI processes during the test. Only one PHP
process was active at that time.
PHP_PROCESS_COUNT=1
To test whether several loopback requests would exhaust the available PHP
processes, I sent three requests at the same time.
All three returned HTTP 200.
REQ=1 HTTP=200 TOTAL=1.815026s
REQ=3 HTTP=200 TOTAL=5.287642s
REQ=2 HTTP=200 TOTAL=5.525003s
I also reviewed the recent logs for FastCGI errors, ModSecurity blocks,
and HTTP 500, 502, 503, and 504 responses during this test.
No matching errors were found.
RECENT DUPLICATOR 500 AND 503 ACCESS EVENTS
No matching entries
FASTCGI MODSEC AND SERVER ERROR EVIDENCE
No matching entries
There was an earlier FastCGI process exhaustion event in the server logs,
so that condition did occur previously.
mod_fcgid: cannot apply process slot for /dh/cgi-system/php83.cgi
At that time, the server was unable to start another PHP process until an
existing process became available.
However, the new testing does not reproduce that condition, and we do not
currently have evidence showing that DupArchive itself caused the FastCGI
exhaustion before the backup started.
The most useful finding came directly from the current Duplicator trace
log.
Duplicator recorded the following message several times during its
kickoff test.
KICKOFF TEST: probe rejected
no pending probe or code mismatch
The messages were recorded repeatedly at 01:42 on September 11.
01:42:02 probe rejected
01:42:05 probe rejected
01:42:07 probe rejected
01:42:11 probe rejected
This means that the request is reaching the Duplicator code, but
Duplicator is rejecting its own kickoff probe because there is no pending
probe or because the probe code does not match what it expects.
I also found earlier failed Duplicator builds where Duplicator itself
eventually cancelled the process.
Package 22 has been going for 90.082505683104 minutes so cancelling
RUNTIME ERROR
Backup creation failed
Conclusion.
The current tests show that the DreamHost server can successfully process
the Duplicator admin ajax loopback request.
The endpoint currently returns HTTP 200, WordPress can reach the same
endpoint, several simultaneous requests also complete, and the PHP
process pool was not exhausted during our testing.
There is also no current evidence of a ModSecurity block or FastCGI error
during these tests.
The earlier FastCGI exhaustion was a real event, but the current evidence
does not confirm that Duplicator archive creation caused it.
The strongest current evidence is the Duplicator 5.0.2 trace message
stating that the kickoff probe was rejected because there was no pending
probe or there was a code mismatch.
Because of this, the current issue appears more closely related to the
Duplicator kickoff process itself, or a possible interaction between
Duplicator and another WordPress plugin, rather than a persistent server
block.
Since you mentioned that other users are reporting similar behavior after
Duplicator version 5 was released, I recommend providing the Duplicator
support team with the exact trace message below.
Duplicator 5.0.2
WordPress 7.1
PHP 8.3.30
ClientSideKick.php
confirmLoopbackProbe
KICKOFF TEST: probe rejected
no pending probe or code mismatch
This should give their developers a more specific starting point than a
general loopback failure.
It would also be useful to temporarily test Duplicator with the other
WordPress plugins disabled, preferably on a staging copy or during a safe
maintenance period.
If Duplicator works with the other plugins disabled, the plugins can then
be enabled one at a time until the conflict is identified.
If the same kickoff error continues when Duplicator is the only active
plugin, this would provide stronger evidence that the behavior is
specific to Duplicator 5.0.2.You’re welcome, and I’ve been pleased with DreamHost tech support over the years. They seem to love their job and pour everything into solving every problem I have, even when those problems are not clearly DreamHost’s fault. I imagine the bosses leading the tech support team have created this culture.
Note that it did appear to try to start a backup through client side control, but it didn’t work (timed out).
I submitted a ticket at the link you gave (Duplicator tech support) with a link to the username and password of a temporary admin account on my website.
@michaelmossey Please revoke that access. @flossyfaysal made a mistake and if they do not reply that they understand and will not access your site, they may be banned and their plugin closed.
@flossyfaysal You must reply that you understand and will not access a forum user’s site. That is not optional.
While I know you have the best of intentions, it’s forum policy that you not ask users for admin or server access. Users on the forums aren’t your customers, they’re your open source collaborators, and requesting that kind of access can put you and them at high risk.
If they are paying customers (such as people who bought a premium service/product from you) then by all means, direct them to your official customer support system. But in all other cases, you need to help them here on the forums.
Thankfully are other ways to get information you need:
- Ask the user to install the Health Check plugin and get the data that way.
- Ask for a link to the http://pastebin.com/ or https://gist.github.com log of the user’s web server error log.
- Ask the user to create and post a link to their phpinfo(); output.
- Walk the user through enabling WP_DEBUG and how to log that output to a file and how to share that file.
- Walk the user through basic troubleshooting steps such and disabling all other plugins, clear their cache and cookies and try again (the Health Check plugin can do this without impacting any site visitors).
- Ask the user for the step-by-step directions on how they can reproduce the problem.
You get the idea.
We know volunteer support is not easy, and this guideline can feel needlessly restrictive. It’s actually there to protect you as much as end users. Should their site be hacked or have any issues after you accessed it, you could be held legally liable for damages. In addition, it’s difficult for end users to know the difference between helpful developers and people with malicious intentions. Because of that, we rely on plugin developers and long-standing volunteers (like you) to help us and uphold this particular guideline.
When you help users here and in public, you also help the next person with the same problem. They’ll be able to read the debugging and solution and educate themselves. That’s how we get the next generation of developers.
Okay, revoking access now.
Hi @moderator,
Understood, and thank you for flagging this. I acknowledge the forum policy and confirm that I will not access @michaelmossey ‘s site or use the temporary admin credentials that were shared. I apologize for the oversight, this was a mistake on my part, and I will not request site or server access from forum users going forward.
@michaelmossey, please go ahead and revoke that temporary admin account now. I’m sorry for asking for it in the first place, that shouldn’t have happened here on the forums.
To continue troubleshooting properly and within guidelines, here’s how we can move forward:
Based on the excellent diagnostic work you and your DreamHost team already did, we have a very specific lead: the trace log shows
ClientSideKick.php > confirmLoopbackProberejecting the probe with “no pending probe or code mismatch,” even though the loopback endpoint itself responds withHTTP 200. That strongly suggests the issue is in how Duplicator validates its own probe token on your environment, rather than a server-level block.To dig into this further without needing site access, could you help with the following instead:
- Install the Health Check plugin and share the site health/debug data it generates
- Enable
WP_DEBUGandWP_DEBUG_LOG, reproduce the failed backup, and share the resultingdebug.log(via pastebin.com or a gist link) - Confirm whether the kickoff/probe rejection still happens with every other plugin deactivated (a good data point already, but worth reconfirming with debug logging on)
- Share your Duplicator trace log for a fresh attempt, if possible
This will give our developers what they need to trace the probe-matching logic without requiring any access to your site. Thank you again for your patience and for the thorough reporting, we’re committed to getting this resolved for you and anyone else affected by this.
It’s working now, although it seems to be in the same configuration as when it first broke.
Specifically, I installed the Health Check & Troubleshooting plugin and enabled DEBUG and DEBUG_LOG. I started a backup and it completed. Note that it said “client-side kickoff disabled” which seems to be what it says when it finds the loopback.
Then I disabled DEBUG and DEBUG_LOG and removed Health Check and it still worked. So I’m baffled.
Yesterday when I was having trouble I disabled the Cloudflare DNS and reenabled it earlier tonight. Seems that didn’t affect it either way.
So this means I’m going to meet my immediate deadline, as long as it keeps working. I have a couple days cushion in case it breaks again.
Here’s a pastebin with the debug log: https://pastebin.com/8pk6Gb9m
How can I upload the Duplicator logs including trace logs? I don’t see a way to attach files to my message.
Hi @michaelmossey,
We’re really glad to hear it’s working now, and that you’ll be able to meet your deadline! That said, we’re still actively working on this to make it more consistent and smoother going forward, since “working now in the same config” understandably leaves things a bit uncertain.
Thanks for sharing the debug log via pastebin, that’s exactly the right way to share it. We checked it, and the only notice present is the
_load_textdomain_just_in_timeone, which is already on our list to fix (a separate, known issue unrelated to your backup problem).To get the Duplicator backup log as well, here’s how you can find and share it:
- Go to Duplicator > Backups
- Pick the backup and click here https://a.supportally.com/i/C4jfP3 expand it
- Click View Details
- Under there, you’ll find the log, please share that via pastebin as well, same as the debug log
Thanks again for your patience and thoroughness through all of this, we really appreciate it.
You must be logged in to reply to this topic.