server overload when importing

  • john smith

    (@hippypink)


    10 years, 4 months ago

    I am only importing about 1,000 at a time. Each listing has about 30 fields, with 7 custom taxonomies (in some cases, 5 or 6 terms per taxonomy).

    My hosting company says its generating 300,000,000 (300 million) reads, and 22 database connections, so they disabled my site for taking over the server–its not even live yet.

    Im using IE
    —————
    Table rows read hit 301582810 > 300000000
    Current load: 23:40:45
    ———————

    P.S. Have you thought about adding ability to add some sort of wait command, or batching, to be able to upload large uploads on shared hosts?

    http://wordpress.org/plugins/really-simple-csv-importer/

Viewing 4 replies - 1 through 4 (of 4 total)
  • Thread Starter john smith

    (@hippypink)

    10 years, 4 months ago

    They also sent this statement that was running at the time:

    STATE: Creating tmp table
    TIME: 0
    COMMAND: Query
    INFO: SELECT t.*, tt.* FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN (example’) AND tr.object_id IN (34057) ORDER BY t.name ASC

    Plugin Author Takuro Hishikawa

    (@hissy)

    10 years, 4 months ago

    Interesting, why such a lot of read queries..I’d like to do test case of importing many taxonomy. But, I think importing 1000 posts in single run is quite a heavy job for shared hosts. You should divide your csv file (eg. each file has 100 posts). I don’t have any plans to add ability to queue import job, because this plugin extends built-in importer of WordPress which has not queueing functions.

    Thread Starter john smith

    (@hippypink)

    10 years, 4 months ago

    This particular database is 11,000, so dividing into 100 blocks prob going to take awhile 🙂

    Interesting about the queuing stuff. Maybe I will develop one someday. I have another project that needs several hundred thousand posts imported 🙁

    Glad to see you are supporting your plugin well!

    Plugin Author Takuro Hishikawa

    (@hissy)

    10 years, 4 months ago

    Thanks a lot. In my project, I had been imported several thousands posts, but in that time, our WordPress were hosted on AWS and changed larger instance while importing. I think wp_get_object_terms function is too heavy, so I will try to reduce using it. Perhaps we need more lightweight API 😉

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘server overload when importing’ is closed to new replies.