WordPress.org

Ready to get started?Download WordPress

Forums

CSV Importer
[Plugin: CVS Importer] Large CSV files taking 100x longer than normal (5 posts)

  1. netflow
    Member
    Posted 3 years ago #

    By large I mean 800KB or so and we're looking at processing times upwards of 45-60 minutes. Inside each CSV is 400 posts, it uploads it nice and quick and then the server just gets slammed.

    I running the blog on a quad-core xeon 2.8ghz machine with 2GB of ram and my load averages are around 3.0 2.0 2.0 with mysql sucking up around 80% of the cpu power the entire time the import is going on.

    However if I take a smaller file, around 200kb with 100 posts it'll get finished in 20-30 seconds no problems. Even if I import the smaller file while the larger one is going on it still completes in around 30 seconds.

    So it seems like there is a critical mass for the CSV that when reach melts the plugin into a pile of goo, slowing the importing process to a crawl. I know it's not the number of posts because i've taken a CSV file with 1400 posts that was only 200kb and it finished in 30 seconds or so which leads me to believe it's the size of the file that is causing this mayhem

  2. netflow
    Member
    Posted 3 years ago #

    I did some more digging and noticed mysql was always taking up 128MB of system memory when it was working which led me to thinking that maybe this wasn't enough to handle larger files. So I increased my key_buffer_size from the default of 128M up with 512M in the my.cnf and importing 800KB files worked like a charm. gonna see how big I can go now :)

  3. netflow
    Member
    Posted 3 years ago #

    So far so good..

    Here are some benchmarks / stress tests
    I've imported a 1.6MB file with 800 posts in 64 seconds.
    I've imported a 2.5MB file with 1200 posts in 103 seconds
    I've imported a 3MB file with 1500 posts 133.03 seconds.
    I've imported a 5MB file with 2500 posts 203.54 seconds.
    I've imported an 8.4MB file with 4000 posts in 324.42 seconds.
    I've imported a 10MB file with 5000 posts in 410.57 seconds.

    I stopped after the last one as it seems that with 512 MB of key cache available this thing can handle just about anything

    Although the weird thing is that after it's finished it reports 1599 posts imported even though there's only 800 new posts on the blogm or 4999 when I import 2000, etc etc

    With all the larger tests my Proc size stays around 469MB and my Resident Mem Size around 100MB

  4. netflow
    Member
    Posted 3 years ago #

    I've scaled it down and can happily report that with 384M for my key_buffer size I'm able to import 6000 posts at once as long as they are going straight to drafts.. If they aren't going to drafts the whole thing hangs.

  5. dvkob
    Member
    Plugin Author

    Posted 3 years ago #

    netflow: Thanks for the report. If you take a look at the plugin's source, you'll see that the plugin handles drafts exactly the same way it handles published posts. I think this means one of two things: either you have other plugins that cause WordPress to hang on import or there's some logic in WordPress itself that causes it to hang.

    It would be nice if you could zip up and upload the files you're using for testing somewhere - I'd be interested in running these tests on my machine, too.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic