Backup archive larger than 5GB cannot be uploaded to S3 service

  • Resolved shibasr79

    (@shibasr79)


    5 years, 9 months ago

    When BackWPup is updated to version 3.6. 10, forwarding to S3 services will fail.

    The log when the job failed is as follows:

    [16-Jul-2019 09:17:17] Backup archive created.
    [16-Jul-2019 09:17:17] Archive size is 5.63 GB.
    [16-Jul-2019 09:17:17] 79639 Files with 5.96 GB in Archive.
    [16-Jul-2019 09:17:18] 1. Trying to send backup file to S3 Service…
    [16-Jul-2019 09:17:19] Connected to S3 Bucket “xxxxxxxxx” in ap-northeast-1
    [16-Jul-2019 09:17:19] Checking for not aborted multipart Uploads…
    [16-Jul-2019 09:17:19] Starting upload to S3 Service…
    [16-Jul-2019 09:22:02] ERROR: Cannot transfer backup to S3! (0)
    [16-Jul-2019 09:22:03] One old log deleted
    [16-Jul-2019 09:22:03] ERROR: Job has ended with errors in 3076 seconds. You must resolve the errors for correct execution.

    The environment running BackWPup plugin is as follows:

    • WordPress version: 5.2.2
    • BackWPup version: 3.6.10
    • PHP version: 7.1.29 (64bit)
    • MySQL version: 10.1.23-MariaDB

    Looking at the BackWPup source code, it appears that there seems to be a problem with the MultipartUpload argument of the completeMultipartUpload method.
    The argument uses the results of the listParts method, which can only get 1,000 parts at a time.

    Multipart Upload Overview

    Multipart Upload Listings

    You can list the parts of a specific multipart upload or all in-progress multipart uploads. The list parts operation returns the parts information that you have uploaded for a specific multipart upload. For each list parts request, Amazon S3 returns the parts information for the specified multipart upload, up to a maximum of 1,000 parts. If there are more than 1,000 parts in the multipart upload, you must send a series of list part requests to retrieve all the parts.

    The size of each part seems to be 5 MB, so multipart uploads can only upload 5 MB * 1000 = 5 GB.

    • This topic was modified 5 years, 9 months ago by shibasr79.
    • This topic was modified 5 years, 9 months ago by shibasr79.
Viewing 15 replies - 1 through 15 (of 20 total)
1 2
  • Plugin Support happyAnt

    (@duongcuong96)

    5 years, 9 months ago

    Hi @shibasr79
    hmm, I don’t think so, we have a client which backup successfully more than 30 GB each backup archive to Amazon S3.
    But I asked our devs about your case, will back to you when I have something new!
    Thank you!

    Thread Starter shibasr79

    (@shibasr79)

    5 years, 9 months ago

    Hi.

    Thanks for the reply.
    When I rollback to version 3.6.9, it works OK. So I will keep using this version for the time being.

    I’ll be waiting for something new.

    Michael

    (@nikonratm)

    5 years, 9 months ago

    I am having the same issue (same error code and file size > 5GB). The backups start but dont finish and the files that do end up in s3 are corrupted (presumably incomplete). There are some size discrepancies too (S3 shows tar is 4.9GB, BackWPUP shows 5.24 / 5.37 (see below).

    I’ve been running BackWPUp for ages, no changes to server etc.

    Relevant part of my log looks like this

    
[22-Jul-2019 05:05:16] Backup archive created.
[22-Jul-2019 05:05:16] Archive size is 5.24 GB.
[22-Jul-2019 05:05:16] 21460 Files with 5.37 GB in Archive.
[22-Jul-2019 05:05:16] Restart after 6 seconds.
[22-Jul-2019 05:05:17] 1. Trying to send backup file to S3 Service …
[22-Jul-2019 05:05:17] Connected to S3 Bucket "***" in us-east-1
[22-Jul-2019 05:05:17] Checking for not aborted multipart Uploads …
[22-Jul-2019 05:05:17] Starting upload to S3 Service …
[22-Jul-2019 05:05:44] Restart after 28 seconds.
[22-Jul-2019 05:06:13] Restart after 28 seconds.
[22-Jul-2019 05:06:41] Restart after 28 seconds.
[22-Jul-2019 05:07:09] Restart after 28 seconds.
[22-Jul-2019 05:07:37] Restart after 28 seconds.
[22-Jul-2019 05:08:07] Restart after 30 seconds.
[22-Jul-2019 05:08:35] Restart after 28 seconds.
[22-Jul-2019 05:09:16] Restart after 41 seconds.
[22-Jul-2019 05:09:46] Restart after 30 seconds.
[22-Jul-2019 05:10:15] Restart after 28 seconds.
[22-Jul-2019 05:10:42] Restart after 27 seconds.
[22-Jul-2019 05:11:09] Restart after 28 seconds.
[22-Jul-2019 05:11:54] Restart after 44 seconds.
[22-Jul-2019 05:12:23] Restart after 28 seconds.
[22-Jul-2019 05:12:25] ERROR: Cannot transfer backup to S3! (0) 
[22-Jul-2019 05:12:25] One file deleted on S3 Bucket.
[22-Jul-2019 05:12:25] One old log deleted
[22-Jul-2019 05:12:25] ERROR: Job has ended with errors in 745 seconds. You must resolve the errors for correct execution.

    Any new ideas?

    Michael

    (@nikonratm)

    5 years, 9 months ago

    Also, reverting to 3.6.9 resolves this for me too, must be something in AWS changes from v3.6.10

    Thanks for any help

    Plugin Support happyAnt

    (@duongcuong96)

    5 years, 8 months ago

    @nikonratm
    yes, it was confirmed issue, our devs currently investing in and have a solution soon.

    • This reply was modified 5 years, 8 months ago by happyAnt.
    Scott Carpenter

    (@squatto)

    5 years, 7 months ago

    I just wanted to confirm that this has also become a problem for me after a recent plugin update. My backup size is 6.29GB. It’s been >5GB for a while now, but only began failing to upload after the update.

    Plugin Support happyAnt

    (@duongcuong96)

    5 years, 3 months ago

    Hi @squatto
    Please update the plugin to the latest 3.7
    Thanks!

    Michael

    (@nikonratm)

    5 years, 2 months ago

    Was this bug re-introduced recently? Im suddenly having this again where I cant upload and log just shows:
    ERROR: Cannot transfer backup to S3! (0)

    I’m running 3.7.0

    Michael

    (@nikonratm)

    4 years, 9 months ago

    Im having this on multiple installs. What gives?

    Plugin Support happyAnt

    (@duongcuong96)

    4 years, 9 months ago

    hello @nikonratm
    could you please help me install this version instead: https://www.dropbox.com/s/hcd4p5tr37m1ivc/backwpup-1474.zip?dl=0
    Thank you!

    Plugin Support happyAnt

    (@duongcuong96)

    4 years, 9 months ago

    @nikonratm
    it is the beta version, please help me test and give feedback. Thank you so much <3

    Michael

    (@nikonratm)

    4 years, 9 months ago

    Thanks for your reply. I get the same error unfortunately

    
[15-Jul-2020 08:38:57] Backup archive created.
[15-Jul-2020 08:38:57] Archive size is 4.88 GB.
[15-Jul-2020 08:38:57] 12823 Files with 5.04 GB in Archive.
[15-Jul-2020 08:38:57] Restart after 12 seconds.
[15-Jul-2020 08:38:58] 1. Trying to send backup file to S3 Service …
[15-Jul-2020 08:38:58] Connected to S3 Bucket "***" in us-east-1
[15-Jul-2020 08:38:58] Checking for not aborted multipart Uploads …
[15-Jul-2020 08:38:58] Starting upload to S3 Service …
[15-Jul-2020 08:39:27] Restart after 29 seconds.
[15-Jul-2020 08:39:58] Restart after 31 seconds.
[15-Jul-2020 08:40:26] Restart after 28 seconds.
[15-Jul-2020 08:40:59] Restart after 33 seconds.
[15-Jul-2020 08:41:27] Restart after 28 seconds.
[15-Jul-2020 08:41:55] Restart after 28 seconds.
[15-Jul-2020 08:42:23] Restart after 28 seconds.
[15-Jul-2020 08:42:52] Restart after 29 seconds.
[15-Jul-2020 08:43:20] Restart after 28 seconds.
[15-Jul-2020 08:43:48] Restart after 28 seconds.
[15-Jul-2020 08:44:16] Restart after 28 seconds.
[15-Jul-2020 08:44:30] ERROR: Cannot transfer backup to S3! (0) 
[15-Jul-2020 08:44:30] One file deleted on S3 Bucket.
[15-Jul-2020 08:44:30] One old log deleted
[15-Jul-2020 08:44:30] ERROR: Job has ended with errors in 682 seconds. You must resolve the errors for correct execution.
    cncuwp

    (@cncuwp)

    4 years, 8 months ago

    Any news on this issue? We are encountering it as well.

    federicoquattrolinee

    (@federicoquattrolinee)

    4 years, 7 months ago

    Same issue here, plugin version 3.7.1. Files bigger than 5Gb are truncated down to 4,9 Gb in the S3 bucket.

    Might this be the problem?

    VERSION 3.6.10
    Release Date: July 8, 2019
    – Removed: S3 multipart upload checkbox, now in destination definition

    federicoquattrolinee

    (@federicoquattrolinee)

    4 years, 7 months ago

    P.s. I tested with version 3.6.9 and it works

Viewing 15 replies - 1 through 15 (of 20 total)
1 2
  • The topic ‘Backup archive larger than 5GB cannot be uploaded to S3 service’ is closed to new replies.