Release memory while adding posts programatically

  • mattm184

    (@mattm184)


    15 years, 6 months ago

    I am trying to add almost 100,000 posts programatically to WP about once a day. This is working pretty well, but the process usually crashes with a PHP out of memory exception. I think WP is not releasing all the Post objects it creates.

    Is there a way to force this? Is there a way to unload and reload the entire WP environment periodically to force WP to release all its data structures?

    I am somewhat new to using WP like this, can anyone think of any other reason why this might be a bad idea?

    Thanks

Viewing 3 replies - 1 through 3 (of 3 total)
  • Xephan

    (@xephan)

    15 years, 6 months ago

    I’ll recommend you break down that 100,000 process into smaller runs and loop through them externally. Not sure how your script is being ran so can’t really give a more specific example.

    Breaking it down would not only get around the memory issue, it’s also less painful if something goes wrong and you need to rerun/undo something. Of course, that’s assuming you’re somehow checking if the insertions are successful 😀

    Thread Starter mattm184

    (@mattm184)

    15 years, 6 months ago

    That is what I am trying to do, but I do not know how to unload the wordpress modules.

    I have a non-WP database. I select * from a table and iterate over all the results. Each row gets its own WP post. selecting all and iterating over the entire table does no use up too much memory.

    Forcing an unload of the wp environment would be a really easy way to chunk up this task.

    rarwar

    (@rarwar)

    14 years, 7 months ago

    This is bit late but here is what worked for me…

    During the insert process there are a few WordPress variables that get quite bloated, so once you have inserted say, a 1000 posts, you can reset them:

    global $wpdb, $wp_actions;
$wpdb->queries = array();
$wp_actions = array();
wp_cache_flush();

    These were the main culprits in my case…and releasing them freed a good chunk of memory

    In my case for a bulk insert of 6000~ posts it was originally taking upto 300 MB, but after releasing the memory after every 1000 inserted posts memory usage never went above 50 MB.

    ps. You can use var_dump(get_defined_vars()); to have a look at whats taking up memory in your case and free variables appropriately.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Release memory while adding posts programatically’ is closed to new replies.