What happened? Constant time-outs???

Resolved braunsch
(@braunsch)

2 years, 7 months ago

This WAS such a great plug-in – but what happened?
I never had a single problem with it, but suddenly:
– It can’t calculate the size of the backup many a time
– It warns me that my database is too big (it’s a whooping 12MB)
– pretty much every task takes 2 or 3 times to complete – if at all
(looking at the file system, backup files (pclzip…tmp) all of a sudden stop growing – and then I have to wait forever until it re-tries)
– Uploading files to Google Drive stalls at least three times until it manages to get my file uploaded (if at all), with WPvivid losing network connectivity every so many seconds – (“Network: Retrying”)
– If I manage to get a backup processed it may take me well more than an hour.

As a result of all this, my scheduled backups now end in error – not a good situation. I now need to do backups manually, can’t store them on Google drive anymore, and have to tend to it until I am lucky that it gets done. Making a backup should not be this hard.

What happened to this plugin that worked just so well?

(some technical data: My site is hosted at Dremhost (shared hosting), my database is 12MB, the whole site takes about 850MB (with the zipped backup file being about 600 MB).

The page I need help with: [log in to see the link]

Viewing 10 replies - 1 through 10 (of 10 total)

Plugin Support tonyrobins
(@tonyrobins)

2 years, 7 months ago

Hello @braunsch

Thanks for reaching out to us.

We haven’t made any changes to the plugin recently, only some minor improvements. Our guess is that it is because your website is growing bigger and the files are larger.

1. It can’t calculate the size of the backup many a time. It warns me that my database is too big (it’s a whooping 12MB)

The warning message will pop up when the plugin detects that your files or db is large. Apparently, it’s your files.

You can simply ignore it and click ‘OK’ to continue the backup. Or you can go to WPvivid settings > General Settings > Uncheck the option ‘Calculate the size of files, folder and database before backing up’ so that it won’t appear any more.

2. Pretty much every task takes 2 or 3 times to complete – if at all

This looks like a timeout issue. We would recommend you to increase PHP max execution time to 1800s on your server side. Then go to WPvivid settings > Advanced Settings > Change the ‘PHP script execution timeout for backup’ to the same value.

3. Uploading files to Google Drive stalls at least three times until it manages to get my file uploaded (if at all), with WPvivid losing network connectivity every so many seconds – (“Network: Retrying”)

This looks like a network problem. Please note that the plugin uses Google’s standard API for uploading files, the upload speed depends on your internet speed.

4. If I manage to get a backup processed it may take me well more than an hour.

The speed up backup process, you can go to WPvivid settings > General Settings > Uncheck the option ‘Merge all the backup files into single package when a backup completes.’

I hope this helps, please feel free to let us know if you need any further information.

Kind Regards,

Thread Starter braunsch
(@braunsch)

2 years, 7 months ago

Hello Tony –

Thank you for the detailed reply.

Indeed, it may be that my site has grown to the point of outgrowing the standard parameters of your plugin, So I changed some of the parameters – and then waited for an unattended backup to happen – to see if it really works again or not.

– I kept the “merge backup files into one” option – because the other one is a mess (as the program does appear not place all the files belonging to a singe backup into a separate folder for each backup.
– I enabled optimization mode for shared hosting
– Changed access mode to PDO – which sped up things big time.
– Compress files every 1024 GB as I want only one file
– backup timeout 300 sec
– memory limit 1024 MB.

The backup made it all the way to the end – but it took over 90 minutes due to several timeouts on the way. The following activities timed out:

[2020-08-24 00:04:51][notice]Prepare to zip files. file: http://www.baylands.org_wpvivid-5f4303d23e4c5_2020-08-23-17-03_backup_plugin.zip
[2020-08-24 00:13:11][notice]Prepare to zip files. file: http://www.baylands.org_wpvivid-5f4303d23e4c5_2020-08-23-17-03_backup_plugin.zip
[2020-08-24 00:27:04][notice]Prepare to zip files. file: http://www.baylands.org_wpvivid-5f4303d23e4c5_2020-08-23-17-03_backup_uploads.zip
[2020-08-24 00:43:39][notice]Prepare to zip files. file: http://www.baylands.org_wpvivid-5f4303d23e4c5_2020-08-23-17-03_backup_core.zip
[2020-08-24 01:05:44][notice]Uploading http://www.baylands.org_wpvivid-5f4303d23e4c5_2020-08-23-17-03_backup_all.zip Total size: 662.55 MB Uploaded: 334.00 MB speed:10223.62kb/s

This time, I did not see if the size was calculated or not – as this was a scheduled backup.

Item #2 (time-outs):

This looks like a timeout issue. We would recommend you to increase PHP max execution time to 1800s on your server side. Then go to WPvivid settings > Advanced Settings > Change the ‘PHP script execution timeout for backup’ to the same value.

I don’t think that the timeout setting is the problem: I have been looking at file sizes “live” on the file system while these processes were running (on other occasions, not this particular run because it was unattended). What happens is that it starts writing files (to the local directory). They start growing – and then the growth suddenly stops. Only then I can start looking at the time and see that the timeout limit starts counting. It just stops, and it does not matter if I tell it to wait for 5, 10 or 15 minutes – before it re-starts that piece of the job.
When it re-starts, it deletes the previous file and starts at zero again. If it runs without problems it’s usually done quickly: Backup_plugin, in the last (and successful) attempt, took 2 minutes, 5 seconds for 201 MB – far from the time limit of 5 minutes I gave it.
In the end:

Backup_plugins took 23 minutes (including two timeouts) for the 201 MB.

Backup_uploads took 16 minutes (including one timeout) for 617 MB

Backup_core took 21 minutes (including one timeout) for 42 MB

Later came the upload – which has one thing that baffles me a bit:

[2020-08-24 01:09:36][notice]Not responding for a long time.
[2020-08-24 01:13:56][notice]Add resume event succeeded.. arg1:1598231696 arg2:wpvivid_resume_schedule_event arg3:wpvivid-5f4303d23e4c5
[2020-08-24 01:13:56][error]Task timed out.
[2020-08-24 01:32:34][notice]Start upload.
[2020-08-24 01:32:35][notice]Check upload folder wpvivid_backup

Apparently the network connection was lost once (detected at 01:13:56). But, why does it take almost 20 minutes before it starts uploading again?

While the disconnected network (not between me and Google, but between Dreamhost and Google) is not in your or my control, I am surprised by the many problems that are occurring locally while compressing files.

Thanks a lot for your help!

Plugin Support tonyrobins
(@tonyrobins)

2 years, 7 months ago

Hey @braunsch

Thanks for your update.

First, it is recommended to uncheck the ‘Merge’ option, because the merge will take more time to compress files.

Next, since it’s with shared hosting that has a timeout limit, it is recommended to ‘compress files every’ 100-200Mb. Because when you set it to 1024GB, there will be only one zip, then in a timeout situation, the plugin will delete the failed backup files(together with temporary files and junk files) then try to backup the entire zip again, which will take more time, and may even cause the backup fail to complete eventually.

But when you set it to 100-200Mb, there will be multiple parts of files. When backing up part1 times out, the plugin will delete and retry part1 only, rather than the entire zip, which will save you great time.

Finally, you said:

It just stops, and it does not matter if I tell it to wait for 5, 10 or 15 minutes – before it re-starts that piece of the job.

This is because the timeout on your server side(which is short) and the timeout you set for backups in WPvivid settings(which is longer) are different. In that case, the process will continue to wait for the longer timeout to be reached, though the scripts have already been terminated by your server.

Therefore, we would recommend you increase the php max execution time on your server side to at least 900s, and set ‘PHP script execution timeout for backup’ to the same in WPvivid settings, so that you will not wait for the time difference.

After changing the above settings, please try to run a backup again.

Thanks,

All the best,

Plugin Support tonyrobins
(@tonyrobins)

2 years, 7 months ago

Sorry, I almost forgot your last question:

Apparently the network connection was lost once (detected at 01:13:56). But, why does it take almost 20 minutes before it starts uploading again?

The plugin’s retry mechanism works by adding scheduled tasks to your queue, so our guess is that this may be because the scheduled tasks were not triggered on time.

All the best,

Thread Starter braunsch
(@braunsch)

2 years, 7 months ago

Thanks for your thorough replies. I took a moment for more experiments.

First, it is recommended to uncheck the ‘Merge’ option, because the merge will take more time to compress files.

I intentionally keep this because I want only one file. I have tried it – and the resulting many files are just a mess since they aren’t in the folder. The merge is not what is causing the backup time to be so long.

Next, since it’s with shared hosting that has a timeout limit, it is recommended to ‘compress files every’ 100-200Mb. Because when you set it to 1024GB, there will be only one zip, then in a timeout situation, the plugin will delete the failed backup files(together with temporary files and junk files) then try to backup the entire zip again, which will take more time, and may even cause the backup fail to complete eventually.

But when you set it to 100-200Mb, there will be multiple parts of files. When backing up part1 times out, the plugin will delete and retry part1 only, rather than the entire zip, which will save you great time.

Same problem: If I use this I will get dozens of files, which I don’t want.

The largest part of the excessive time is not the actual work being done (it probably would get an entire backup done in less than 20 minutes, if there weren’t all these timeout problems.
20 minutes is fine, and it does not matter if it’s 18 or 22 minutes. 90 minutes with five timeouts is not.

Therefore, we would recommend you increase the php max execution time on your server side to at least 900s, and set ‘PHP script execution timeout for backup’ to the same in WPvivid settings, so that you will not wait for the time difference.

Since wait time after stalled processes is the biggest problem I have, I upped my max execution time more carefully. 300 seconds on both sides, PHP and WPvivid Backup. Yet: After 155 seconds into the job, at the 48% complete mark (in the middle of processing uploads), the job apparently stalled (no more file growth). So, timeout at 300 seconds can not be the problem here. There must be something else that makes this not processing correctly.

Thanks for your great help so far!

Plugin Support tonyrobins
(@tonyrobins)

2 years, 7 months ago

Hey @braunsch

Thanks for your update.

1. Theoretically, the merge will take the same amount of time again to compress the 6 backups parts(db, core, uploads…) into one. But you can still keep it.

2.

After 155 seconds into the job, at the 48% complete mark (in the middle of processing uploads), the job apparently stalled (no more file growth).

If so, there must be some other settings or applications on your server that will terminate php scripts in 155 seconds, you can contact your hosting provider to confirm what the settings are.

Thanks,

All the best,

Thread Starter braunsch
(@braunsch)

2 years, 7 months ago

Just got a reply from Dreamhost. Their reply: “Unfortunately, your user’s processes are being killed for reaching the memory limit”.

My memory settings are: 512 MB in phprc, in wp-config and in wp-vivid.

I told the plugin to break files after 128 MB. Yet, the problem did not go away: this time it stalled three times:
– after 1.2 MB of backing up the themes (4.5 MB total)
– after 18 MB of the 2nd chunk of plugins (plugins are ~220 MB in total)
– after 43 MB of backing up the core (which should be ~45 MB).

In between, there were five “full-size” files (128MB) that ran through without any problems.

Never during the entire process did the WP memory usage plugin report more than 98.5MB of memory used.

Any idea how I can go about figuring this out?

Thanks a lot!

Plugin Support tonyrobins
(@tonyrobins)

2 years, 7 months ago

Hi @braunsch

Thanks for your update.

It does not seem to be a memory issue.

It’s normal that the backup completed after 3 retries. We recommend you set the max execution time in the php.ini file on the server side to 900 seconds, so that the number of retries will be reduced and the backup process will be faster.

All the best,

Thread Starter braunsch
(@braunsch)

2 years, 6 months ago

So – I think the problem is now reasonably resolved: After discussions with Dreamhost, I ended up reducing the amount of memory for the installation – from 512MB to 384MB, with a 180 second time-out. That in fact did enhance stability.
Now backups run end-to-end in a good 20 minutes.

Thanks for your help!

Plugin Support tonyrobins
(@tonyrobins)

2 years, 6 months ago

Hi @braunsch

No problem. Glad the problem is resolved now.

All the best,

Viewing 10 replies - 1 through 10 (of 10 total)

The topic ‘What happened? Constant time-outs???’ is closed to new replies.

What happened? Constant time-outs???

Topic Tags

Views