WordPress.org

Ready to get started?Download WordPress

Forums

CSV Importer
Using a unique field to prevent duplicates (6 posts)

  1. Jason Judge
    Member
    Posted 1 year ago #

    Often you want to import posts from another system, and use the unique key from the external records to ensure no duplicates are loaded in WP.

    For example, WooCommerce has the "sku" field that is unique across all products. If you try and load a second product with the same SKU, it will update the original instead.

    This CSV loader could benefit greatly from being able to do that. At the moment, if loading fails, you have no way to start that load again without duplicating what has already loaded, and that can leave a real mess in the database.

    http://wordpress.org/extend/plugins/csv-importer/

  2. Jason Judge
    Member
    Posted 1 year ago #

    Even if the import does not do updates of posts that already exist (according to the defined "key" field), which would be awsome, the ability to at least skip over posts that have already been loaded would be ideal.

    Perhaps if you add a filter to this plugin, that fires on each record before it is created (being given both the core post fields *and* any additional custom fields) then the check for duplicates can be handled in an external plugin.

  3. Jason Judge
    Member
    Posted 1 year ago #

    Around line 217 of csv_importer.php I have added the marked code (untested):

    foreach ($csv->connect() as $csv_data) {
                $csv_data = apply_filters('csv_importer_validate', $csv_data); // Call up validation filters
                // Only create post if validation filters did not throw away data
                if (!empty($csv_data) && $post_id = $this->create_post($csv_data, $options)) {
                    $imported++;
                    $comments += $this->add_comments($post_id, $csv_data);
                    $this->create_custom_fields($post_id, $csv_data);
                } else {
                    $skipped++;
                }
            }

    That allows me to create a custom filter to sanitize any of the data, and also to validate it. In the filter, by returning FALSE instead of the data array, the row can be skipped.

    I would probably also pass in the record number, so that can be logged in any validation errors that are subsequently logged. Also pass in $this so that the $this->log['error'][] can be used to send the error or warnings (aka notices) to.

  4. Jason Judge
    Member
    Posted 1 year ago #

    Now tested, 213 of csv_importer.php:

    $skipped = 0;
            $imported = 0;
            $comments = 0;
            foreach ($csv->connect() as $csv_data) {
                $csv_data = apply_filters('csv_importer_validate', $csv_data, $this, $skipped + $imported + 1);
                if (!empty($csv_data) && $post_id = $this->create_post($csv_data, $options)) {
                    $imported++;
                    $comments += $this->add_comments($post_id, $csv_data);
                    $this->create_custom_fields($post_id, $csv_data);
                } else {
                    $skipped++;
                }
            }

    That filter adds all sorts of possibilities. I am using it in my theme functions.php like this:

    add_filter('csv_importer_validate',  'my_csv_importer_validate', 10, 3);
    
        function csv_importer_validate($csv_data, $importer, $row_number)
        {
            // In here I check if the item has already been imported,
            // based on a "key" custom field. If it has, then I raise a
            // a warning:
            // $importer->log['notice'][] = 'Post key ' . $csv_data['key'] . ' already exists';
            // then return false, and that skips the row. Otherwise I return
            // $csv_data and the row loads as normal.
        }

    So, an this filter be added to this plugin? Please? Pretty please?

  5. alexros
    Member
    Posted 1 year ago #

    Hello judgej, I tried to do as you did (input your codes to 213 of csv_importer.php - input your code to 217 of csv_importer.php - and also put your function to functions.php) But it writes me Fatal error: Call to a member function connect() on a non-object in Z:\home\lab.lab\www\wp-content\plugins\csv-importer\csv_importer.php on line 217

    Can you please clarify on this point please. Does it works for you? Thank you in advance.

  6. Jason Judge
    Member
    Posted 1 year ago #

    You do replace the existing loop with the new code, don't you? The loop is already in the plugin around line 213, and the additional code introduces the filter, the condition (import or skip).

    I have put a copy of my modified version of this plugin here:

    http://academe.co.uk/2012/10/wordpress-csv-importer-plugin-skipping-duplicates/

    Hopefully that will help you.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic

Tags

No tags yet.