Support » Developing with WordPress » For each loop does not always do all iterations

  • daanburozero

    (@daanburozero)


    Currently, I have the following problem:
    I have created a WordPress environment that sends personalized emails to subscribers based on their preferences. This has worked for quite some time but for a couple of months, we are experiencing some inconsistencies. These inconsistencies are as followed:

    Once in a while, the foreach loop for sending the emails stops in the middle of its execution. For example, we have a newsletter with 4000 subscribers. Once in a while, the program randomly stops its sending procedure at around 2500 emails. When this happens, there are literally no signs of any errors and there is also nothing to be seen in the debug log.

    I have tried the following things to fix the issue:
    – Different sender; we switched from Sendgrid to SMTPeter (Dutch SMTP service)
    – Delays; we have tried whether placing a wait after x number of emails would have any impact because there might be too many requests per minute, but this was not the case.
    – Disable plugins; For 5 weeks we thought we had found the problem. WordFence seemed to be the problem, unfortunately, the send function stopped again last week and this did not appear to be causing the problems. Just to show how unstable it really is. It can go well for 5 weeks and then not for 2 weeks.
    – Rewriting of functions
    – Logging, we write values ​​to a txt file after every important step to keep track of where the send function stops. This is just to see which users have received an email and which still need to receive it so that we can continue sending it from there.
    – Debug log, the annoying thing is that even when we have the wp_debug on, nothing comes up that indicates a cause of crashing.

    To schedule the sender I use the WP_Cron to run the task in the background. From there the following function is triggered;

    Below, the code I wrote in stripped format. I removed all the $message additions as this is just HTML with some variables of ACF for the email. I translated it so it’s easier to understand.

     
    <?php
    function send_email($edition_id, $post)
    {
      require_once('SMTPeter.php'); //Init SMTPeter Sender
    
      $myfile = fopen("log.txt", "a") or die("Unable to open file!"); //Open custom logfile
      $editionmeta = get_post_meta($edition_id); //Get data of edition
      $users = get_users();
    
      $args = array(
        'post_type' => 'articles',
        'post_status' => 'publish',
        'posts_per_page' => -1,
        'order' => 'asc',
        'meta_key' => 'position',
        'orderby' => 'meta_value_num',
        'meta_query' => array(
          array(
            'key'     => 'edition_id',
            'value'   => $edition_id,
            'compare' => 'LIKE',
          ),
        ),
      );
    
      $all_articles = new WP_Query($args); // Get all articles of edition
    
      $i = 0; // Counter users interrested in topic
      $j = 0; // Counter sent emails
    
      foreach ($users as $user) { //Loop over all users <---- This is the loop that not always finishes all itterations
    
        $topic_ids = get_field('topicselect_', 'user_' . $user->ID);
        $topic_id = $editionmeta['topic_id'][0];
    
        if (in_array($editionmeta['topic_id'][0], $topic_ids)) { // Check if user is interrested in topic. 
          $i++; // Counter interrested in topic +1.
    
          // Header info
          $headerid = $editionmeta['header_id'][0];
          $headerimage = get_field('header_image', $headerid);
          $headerimagesmall = get_field('header_image_small', $headerid);
    
          // Footer info
          $footerid = $editionmeta['footer_id'][0];
          $footer1 = get_field('footerblock_1', $footerid);
          $footer2 = get_field('footerblock_2', $footerid);
          $footer3 = get_field('footerblock_3', $footerid);
    
          $message = '*HTML header newsletter*'; // First piece of content email  
    
          if ($all_articles->have_posts()) :
            $articlecount = 0; // Set article count to check for empty newsletters
    
            while ($all_articles->have_posts()) : $all_articles->the_post();
              global $post;
              $art_categories = get_the_category($post->ID); // Get categories of article
              $user_categories = get_field('user_categories_', 'user_' . $user->ID); // Get categories user is interrested in
    
              $user_cats = array();
              foreach ($user_categories as $user_category) {
                $user_cats[] = $user_category->name; // right format for comparison
              }
    
              $art_cats = array();
              foreach ($art_categories as $art_category) {
                $art_cats[] = $art_category->name; // right format for comparison
              }
    
              $catcheck = array_intersect($user_cats, $art_cats); // Check if 1 of the article's categories matches one of a user's categories
    
              if (count($catcheck) > 0) { // The moment the array intersect count is greater than 0 (at least 1 category matches), the article is added to the newsletter.                     
                $message .= "*Content of article*"; // Append article to content of newsletter
                $articlecount++;
              }
    
            endwhile;
          endif;
    
          if ($articlecount > 0) { //As soon as the newsletter contains at least 1 article, it will be sent.
    
            $j++; //Sent email counter.
    
            $mailtitle = $editionmeta['mail_subject'][0]; // Title of the email
            $sender = new SMTPeter("*API Key*"); // Class SMTPeter sender
            $output = $sender->post("send", array(
              'recipients'    =>  $user->user_email, // The receiving email address
              'subject'       =>  $mailtitle,       // MIME's subject
              'from'          =>  "*Sender*",  // MIME's sending email address
              'html'          =>  $message,
              'replyto'       =>  "*Reply To*",
              'trackclicks'   =>  true,
              'trackopens'    =>  true,
              'trackbounces'  =>  true,
              'tags'          =>  array("$edition_id")
            ));
           error_log(print_r($output, TRUE));        
           fwrite($myfile, print_r($output, true));
          }
        }
      }
      fclose($myfile);
    }
    

    All I want to know is the following;
    Why can’t my code run the foreach completely, every time? I mean, it’s quite frustrating to see that it sometimes works like a charm, and the next time it could get stuck again.
    Some things I thought about but did not yet implement:

    – Rewrite parts of the function into separate functions. Retrieving the content and setting up the HTML for the newsletter could be done in a different function. Besides the fact that it would obviously be an improvement for cleaner code, I just wonder if this could actually be the problem.
    – Can a foreach crash due to a fwrite trying to write to a file that is already being written to? So does our log cause the function to not run properly? (Concurrency, but is this a thing in PHP with its workers?)
    – Could the entire sending process be written in a different way?

    Thanks in advance,

    Really looking forward to your feedback and findings

Viewing 10 replies - 1 through 10 (of 10 total)
  • Lucas Karpiuk

    (@karpstrucking)

    @daanburozero what’s your max_execution_time set to in PHP? it’s possible the process is just timing out before completed.

    you could rewrite this to make use of the WP Cron to break up the work over a series of jobs to increase reliability

    corrinarusso

    (@corrinarusso)

    Why can’t my code run the foreach completely, every time?

    ????????
    Bc this is not written for any kind of scale – and secondly does not accommodate for any data validation, network validation, expected latency etc, etc, etc, etc.

    imo, it’s time to bail on your home-grown email solution where you are putting in time and energy and resources to troubleshoot, maintain, expand, customize and migrate to a solution that removes all these time wasting variables for you – use tools provided that are best in class and save yourself the headache and instead you can focus on running your business.
    Her’es a few samples from the hundreds to choose from that would surely meet your requirements:
    https://mailchimp.com/
    https://www.constantcontact.com/ca
    https://www.sendinblue.com
    https://www.mailjet.com

    Moderator bcworkz

    (@bcworkz)

    In addition to what Corrina said, sending large quantities of email from your webserver is a good way to get your server blacklisted as a spam source. Even if you are able to send so many emails, a lot of intended recipients may never see the message. You should really be using a mass mailing service.

    MK

    (@mkarimzada)

    I agree with Bcworkz and Corrina there are lots of downsides sending emails this way, but not the issues about reliability and delivery of the emails. SMTPeter is a reliable SMTP service to use and their API is pretty powerful specially when you want to create emails from JSON data.

    However to troubleshoot your function, I would start with $users object. It’s obvious wp_debug wouldn’t work because you have if statements to check if values exists which prevents from code errors. You should look into debugging your data with dumping the data, not code errors.

    Also, try to improve the WP_Query $args. In meta_query array you are passing $edition_id as number and comparing it as string. Try using other comparisons and dump the articles.

    It worths mentioning that there is a core new enhancement ticket open about unexpected behavior of filtering users with certain dynamic meta_key/value.

    Also, I would clean up the code and use $wpdb and INNER JOIN couple of queries to get the results instead of looping over and over on users and posts.

    I hope this helps!

    corrinarusso

    (@corrinarusso)

    I would add that while it’s possible to work this way, it’s just not practical anymore. The services and APIs available to us now are too good, and very cheap.

    Customization bad.
    Configuration good.

    MK

    (@mkarimzada)

    @corrinarusso I don’t see much difference between SMTPeter and Mailchimp. Both are third party APIs.

    As far as your concern about validation and best practices, both APIs and their PHP class is similar SMTPeter and Mailchimp. You could take a look yourself.

    I think the question here is how to conditionally send the email based on available data in database, not which API you want to call to send the email. Also, there are lots of plugins with way more customization, and I think it’s very practical.

    corrinarusso

    (@corrinarusso)

    Agree.
    Pick a product that works well for your business and your business integration points and go for it.

    Use the API on the application to customize as much as you want. My point is that we don’t want to be doing this in a functions file or some other script to kick it off. We want to use the best tools in class that are available, and use best practices.

    Cloud tools like MailChimp etc etc will provide the functionality you want OOB almost all the time. The days of sitting down to code a new mousetrap are long gone. Creates far too much headache and overhead.

    MK

    (@mkarimzada)

    I think you misunderstood the point. He is using an API in his application and a cloud based mail service to send the emails. It’s not his web server/mail server. You could read more here.

    FYI If you want to send a custom mail using Mailchimp you have to do exactly the same by requiring Mailchimp PHP Class.

    The problem he has is with filtering the users based on available meta_value/meta_key and dynamically use their email address to send the email.

    Thread Starter daanburozero

    (@daanburozero)

    @mkarimzada you are totally right. Thank you for your advice! This will probably help me further. Whenever I find the issue, I’ll post it over here to maybe help others out. Once again, thank you for the replies!

    corrinarusso

    (@corrinarusso)

    @mkarimzada

    Oooooooooohhhhhhhhhhh.
    Lol – thanks for clarifying.
    That was a lot of conversation for being off-base the entire time! haha

Viewing 10 replies - 1 through 10 (of 10 total)
  • The topic ‘For each loop does not always do all iterations’ is closed to new replies.