Support » Plugin: BackWPup - WordPress Backup Plugin » Backup archive larger than 5GB cannot be uploaded to S3 service

  • shibasr79

    (@shibasr79)


    When BackWPup is updated to version 3.6. 10, forwarding to S3 services will fail.

    The log when the job failed is as follows:

    [16-Jul-2019 09:17:17] Backup archive created.
    [16-Jul-2019 09:17:17] Archive size is 5.63 GB.
    [16-Jul-2019 09:17:17] 79639 Files with 5.96 GB in Archive.
    [16-Jul-2019 09:17:18] 1. Trying to send backup file to S3 Service…
    [16-Jul-2019 09:17:19] Connected to S3 Bucket “xxxxxxxxx” in ap-northeast-1
    [16-Jul-2019 09:17:19] Checking for not aborted multipart Uploads…
    [16-Jul-2019 09:17:19] Starting upload to S3 Service…
    [16-Jul-2019 09:22:02] ERROR: Cannot transfer backup to S3! (0)
    [16-Jul-2019 09:22:03] One old log deleted
    [16-Jul-2019 09:22:03] ERROR: Job has ended with errors in 3076 seconds. You must resolve the errors for correct execution.

    The environment running BackWPup plugin is as follows:

    • WordPress version: 5.2.2
    • BackWPup version: 3.6.10
    • PHP version: 7.1.29 (64bit)
    • MySQL version: 10.1.23-MariaDB

    Looking at the BackWPup source code, it appears that there seems to be a problem with the MultipartUpload argument of the completeMultipartUpload method.
    The argument uses the results of the listParts method, which can only get 1,000 parts at a time.

    Multipart Upload Overview

    Multipart Upload Listings

    You can list the parts of a specific multipart upload or all in-progress multipart uploads. The list parts operation returns the parts information that you have uploaded for a specific multipart upload. For each list parts request, Amazon S3 returns the parts information for the specified multipart upload, up to a maximum of 1,000 parts. If there are more than 1,000 parts in the multipart upload, you must send a series of list part requests to retrieve all the parts.

    The size of each part seems to be 5 MB, so multipart uploads can only upload 5 MB * 1000 = 5 GB.

    • This topic was modified 5 months ago by shibasr79.
    • This topic was modified 5 months ago by shibasr79.
Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Support duongcuong96

    (@duongcuong96)

    Hi @shibasr79
    hmm, I don’t think so, we have a client which backup successfully more than 30 GB each backup archive to Amazon S3.
    But I asked our devs about your case, will back to you when I have something new!
    Thank you!

    shibasr79

    (@shibasr79)

    Hi.

    Thanks for the reply.
    When I rollback to version 3.6.9, it works OK. So I will keep using this version for the time being.

    I’ll be waiting for something new.

    I am having the same issue (same error code and file size > 5GB). The backups start but dont finish and the files that do end up in s3 are corrupted (presumably incomplete). There are some size discrepancies too (S3 shows tar is 4.9GB, BackWPUP shows 5.24 / 5.37 (see below).

    I’ve been running BackWPUp for ages, no changes to server etc.

    Relevant part of my log looks like this

    
    [22-Jul-2019 05:05:16] Backup archive created.
    [22-Jul-2019 05:05:16] Archive size is 5.24 GB.
    [22-Jul-2019 05:05:16] 21460 Files with 5.37 GB in Archive.
    [22-Jul-2019 05:05:16] Restart after 6 seconds.
    [22-Jul-2019 05:05:17] 1. Trying to send backup file to S3 Service …
    [22-Jul-2019 05:05:17] Connected to S3 Bucket "***" in us-east-1
    [22-Jul-2019 05:05:17] Checking for not aborted multipart Uploads …
    [22-Jul-2019 05:05:17] Starting upload to S3 Service …
    [22-Jul-2019 05:05:44] Restart after 28 seconds.
    [22-Jul-2019 05:06:13] Restart after 28 seconds.
    [22-Jul-2019 05:06:41] Restart after 28 seconds.
    [22-Jul-2019 05:07:09] Restart after 28 seconds.
    [22-Jul-2019 05:07:37] Restart after 28 seconds.
    [22-Jul-2019 05:08:07] Restart after 30 seconds.
    [22-Jul-2019 05:08:35] Restart after 28 seconds.
    [22-Jul-2019 05:09:16] Restart after 41 seconds.
    [22-Jul-2019 05:09:46] Restart after 30 seconds.
    [22-Jul-2019 05:10:15] Restart after 28 seconds.
    [22-Jul-2019 05:10:42] Restart after 27 seconds.
    [22-Jul-2019 05:11:09] Restart after 28 seconds.
    [22-Jul-2019 05:11:54] Restart after 44 seconds.
    [22-Jul-2019 05:12:23] Restart after 28 seconds.
    [22-Jul-2019 05:12:25] ERROR: Cannot transfer backup to S3! (0) 
    [22-Jul-2019 05:12:25] One file deleted on S3 Bucket.
    [22-Jul-2019 05:12:25] One old log deleted
    [22-Jul-2019 05:12:25] ERROR: Job has ended with errors in 745 seconds. You must resolve the errors for correct execution.
    

    Any new ideas?

    Also, reverting to 3.6.9 resolves this for me too, must be something in AWS changes from v3.6.10

    Thanks for any help

    Plugin Support duongcuong96

    (@duongcuong96)

    @nikonratm
    yes, it was confirmed issue, our devs currently investing in and have a solution soon.

    • This reply was modified 4 months, 3 weeks ago by duongcuong96.

    I just wanted to confirm that this has also become a problem for me after a recent plugin update. My backup size is 6.29GB. It’s been >5GB for a while now, but only began failing to upload after the update.

Viewing 6 replies - 1 through 6 (of 6 total)
  • You must be logged in to reply to this topic.