Support » Plugin: Really Simple CSV Importer » Amazing! But…

  • Resolved Humiges

    (@humiges)


    Hi,

    first of all, thank yo so much for such a great plugin.

    I have a question in regards of its speed… we are transferring over 500k articles wrote by 1000s of authors.

    We split the database in pieces, but it still takes about 10hours to do only 20k articles.

    Please, is there any way this can be speed up?

    Thanks a lot

    (we are using to latest wordpress/buddypress and have plesk dedicated server)

    • This topic was modified 2 years, 4 months ago by Humiges.
Viewing 6 replies - 1 through 6 (of 6 total)
  • Dear @hissy,
    Just an idea… maybe removing the “update” = “check if the post exists” would speed it up? We don’t need that… we just need to add everything and all is new and until it’s done the site is down.

    Can anyone help me to stop the update and check if exists? I’m more than happy yo donate your efforts if you know the solution :))

    Thanks so much

    @humiges,

    As the name suggests, this is a simple importer. Your situation is not simple. Maybe this is not the right tool for the job.

    If, as you say, the site is down until the data can be imported, I recommend hiring a developer to build a custom script that does just what you need without the checks and features this plugin needs to work effectively in a wide variety of situations.

    Dear @cdillon27

    this plugin actually works much better than any other CSV importers we tried… here is a simple reason for it… only this one allows us to use our own ID.
    Believe me, we tried sooo many other CSV importers.

    So, the option can be to just switch off the “update” = “check if the post exists” … possibly by removing part of the code… I just don’t want to mess around, so I thought I would ask.

    It really works like a charm… just painfully slow and since we keep using this… it would take around 10days to do it… which is bad because we need some time for testing.

    Thanks a lot

    PS: we are importing those data:
    post_id
    post_title
    post_author
    post_date
    post_type
    post_status
    post_content
    post_tags

    • This reply was modified 2 years, 3 months ago by Humiges.
    • This reply was modified 2 years, 3 months ago by Humiges.
    Plugin Author Takuro Hishikawa

    (@hissy)

    @humiges This plugin is a simple extension of core importer class in WordPress. Its architecture is also simple, so not suitable for importing a large number of posts. I think core importer class should be support queue system to import huge contents, but I don’t have a plan to support it in this plugin. I’m sorry but search another importer plugin.

    Thank you for your kind reply @hissy,

    Once again, thank you for this simple plugin… it really works like a charm for me more than any other CSV and more complex importers.

    Anyway, I have found out way about it.

    So, here is a trick, please keep backing up, backing up… this is simple way to do it on NON live sites, because it crashes your site and makes it unaccessible!

    Anyway, so… Frist, back up!!! This is trick for a bit advance users who know phpMyAdmin!

    So, I opened the phpMyAdmin and:

    A, copy the _posts to _posts_all

    B, empty whole _posts (this will obviously destroy current site – but you have it in _posts_all)

    C, import your CSV (finally, work super fast :))

    D, Merge the uploaded _posts with _posts_all by
    INSERT IGNORE INTO YourPrefix_posts_all SELECT * FROM YourPrefix_posts

    …you have now all in _posts_all

    E, Empty _posts and repeat B-D as many times you wish

    When you are done… remove empty _posts and rename the _posts_all to _posts

    Hope it makes sense
    Good luck 🙂

    My suggestion to those looking to import very large amounts of data via csv is to do it locally (If you are setup on a local machine using wamp or xamp, etc). You can increase your server resources in the php.ini file to avoid all of the time out errors on upload. Then export your site and upload it back your web host.

    I just uploaded 50k rows of csv data in one go and it completed in about 2.5 hours. This page has more info on maxing out your settings to avoid time out errors. https://docs.appthemes.com/support/how-to-import-very-large-files/

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Amazing! But…’ is closed to new replies.