• Resolved jeff9315

    (@jeff9315)


    I’ve got a memory problem that I hope you can help me with. As usual, I am SO grateful for your wonderful plugin and even more so for the fantastic support you provide here …

    I’ve got a page that keeps running into an out-of-memory issue. It’s the only page that has this issue. This page executes MLA 14 times … once for each category, with different sort and selection options, plus different headings. Currently, I’m up to 215 MB and I’m concerned that I’ll eventually hit my server’s max of 256 MB.

    I have a small PHP file that’s basically a single function invoked by a shortcode. Each time I invoke my shortcode, my function determines which set of MLA parameters to use based on the attachment-category. It then invokes MLA once.

    I wanted to make sure that you at least know about this memory issue, agree it’s caused by the number of MLA invocations, and can’t really do anything about it. None of the categories has more than about 20 items in them. And they are all fairly small PDF documents.

    I suppose I could break my page into 2 different pages so that I’m only invoking MLA half the number of times or I could have one page that lists the 14 categories and a link to 14 separate pages. Each page only containing the shortcode.

    Or maybe you can think of something else I can do.

    Thanks … Jeff

    Here’s the my PHP code:

    add_shortcode('list_docs', 'my_list_docs');
    
    /**
     * FUNCTIONS
     */
    
    /**
     * This action invokes MLA with all the standard parameters
     * Note that <<<EOV ... EOV uses the PHP "Heredoc" format.
     *   link: http://php.net/manual/en/language.types.string.php#language.types.string.syntax.heredoc
     *
     * Usage: [list_docs category="attachment category", query_value="value for the query using MLA syntax", itemwidth="item width", numberposts="numberposts" style ="style"]
     *
     * It invokes MLA.
     * All parameters are optional, although it's not overy useful if you don't specify at least category.
     * mla_itemwidth defaults to "45" (since that's my default for most pages).
     * The mla_style defaults to "sc-list" (since that's my default for most lists).
     */
    function my_list_docs( $atts, $content = null ) {
    	$a = shortcode_atts( array(
    		'category' => '',
    		'query_value' => '',
    		'item_width' => '45',
    		'numberposts' => '',
    		'style' => 'sc-list'
    	), $atts );
    	$category    = $a['category'];
    	$query_value = $a['query_value'];
    	$item_width  = $a['item_width'];
    	$numberposts = $a['numberposts'];
    	$style       = $a['style'];
    
      $my_shortcode = "";
    	$my_orderby = "";
    	$my_query = "";
    
      if ($numberposts <> "") { $numberposts = "numberposts=" . $numberposts; }
    
      $basic_mla_value = <<<EOV
    mla_gallery mla_style="$style" columns=1 attachment_category="$category" post_mime_type=application/pdf link=file mla_item_width="$item_width" $numberposts
    EOV;
    
    	switch ($category) {
    		case "Town Hall Statuses":
    		case "Town Hall Meeting Notes":
    		case "Town Hall Agendas":
    		case "Forms":
    		case "Contracts":
            $my_orderby = ' orderby=title order=ASC';
            break;
    		case "Event Flyers":
    		case "Board Agendas":
    		case "Board Minutes":
    		case "Financial Reports":
    			$my_orderby = ' orderby="title" order="DESC" ';
          break;
    		case "Governing Documents":
       		$my_orderby = ' orderby=meta_value meta_key="my-sort-order" order=ASC ';
            break;
    		case "Board Resumes":
    			$my_orderby = ' orderby=meta_value meta_key="my-sort-order" order=ASC ';
    			$my_query   = <<<EOV
    				meta_query="array( array( 'key' => 'YYYY-MM', 'value' => '$query_value', 'compare' => 'LIKE' ) )"
    EOV;
    			break;
    		case "Newsletter":
    			$my_orderby = ' orderby=meta_value meta_key="YYYY-MM" order=DESC ';
    			$my_query   = <<<EOV
    				meta_query="array( array( 'key' => 'YYYY-MM', 'value' => '$query_value', 'compare' => 'LIKE' ) )"
    EOV;
    			break;
    
    		default:
    			$my_orderby = ' orderby=title order=ASC ';
    	}
    
    $my_shortcode = "[" . $basic_mla_value . $my_orderby . $my_query . "]";
    return do_shortcode($my_shortcode);
    // return $my_shortcode;

    Here’s my invocation:

    <p><strong>Town Hall Statuses</strong></p><p>[list_docs category="Town Hall Statuses"]</p>
    <p><strong>Town Hall Meeting Notes</strong></p><p>[list_docs category="Town Hall Meeting Notes"]</p>
    <p><strong>Town Hall Agendas</strong></p><p>[list_docs category="Town Hall Agendas"]</p>
    
    <p><strong>Forms</strong></p><p>[list_docs category="Forms"]</p>
    <p><strong>Contracts</strong></p><p>[list_docs category="Contracts"]</p>
    
    <p><strong>Board Agendas</strong></p><p>[list_docs category="Board Agendas"]</p>
    <p><strong>Board Minutes</strong></p><p>[list_docs category="Board Minutes"]</p>
    
    <p><strong>Financial Reports</strong></p><p>[list_docs category="Financial Reports"]</p>
    <p><strong>Governing Documents</strong></p><p></strong></p><p>[list_docs category="Governing Documents"]</p>
    
    <p><strong>Event Flyers</strong></p><p>[list_docs category="Event Flyers" ]</p> 
    
    <p><strong>Board Resumes - 2016-03</strong></p><p></strong></p><p>[list_docs category="Board Resumes" query_value="2016-03"]</p>
    <p><strong>Board Resumes - 2015-12</strong></p><p>[list_docs category="Board Resumes" query_value="2015-12"]</p>
    <p><strong>Newsletter - 2015</strong></p><p>[list_docs category="Newsletter" query_value="2015"]</p>
    <p><strong>Newsletter - 2014</strong></p><p>[list_docs category="Newsletter" query_value="2014"]</p>

    I’ll eventually replace the paragraph headings with Spoilers so that only the category names show until the visitor expands the one they want.

    https://wordpress.org/plugins/media-library-assistant/

Viewing 10 replies - 1 through 10 (of 10 total)
  • Plugin Author David Lingren

    (@dglingren)

    Thanks for your kind words and interesting report/question. Thanks as well for all the details you provided and especially the PHP code and invocation text.

    I ran many tests to measure the memory use of [mla_gallery] shortcodes along the lines of those you are using. On my system, with the WordPress Twenty Twelve theme a page with ten shortcodes consumes 18MB – 22MB of memory. I do not know why your usage is so much higher, but it depends on all of the plugins you are using and your theme.

    I can see a gradual increase in memory use each time the shortcode runs. I believe this is because WordPress is caching the results of the query as well as the taxonomy terms and metadata values associated with each of the items returned by the main query. There is no easy way to disable this; the [mla_gallery] caching parameters actually make things worse.

    I suggest that you add a bit of debug logic to your PHP code to see how much [mla_gallery] is contributing to the overall usage. Try adding something like this to the bottom of your code:

    $my_shortcode = "[" . $basic_mla_value . $my_orderby . $my_query . "]";
    $memory_use = ' before mla_gallery_shortcode memory = ' . var_export( memory_get_usage( false ), true ) . ' , real_usage = ' . var_export( memory_get_usage( true ), true ) . '<br>';
    return $memory_use . do_shortcode($my_shortcode);

    With the additional $memory_use code in place you will see something like before mla_gallery_shortcode memory = 17468432 , real_usage = 17825792 above each of the gallery displays. That will give you a sense of usage before the first [mla_gallery] invocation and then before each subsequent invocation. You could instead write this information to the error log if you have access to that file.

    You may see a large jump after the first shortcode invocation. This is because the PHP code required to process the shortcode is not loaded until it is actually needed.

    I will leave this topic unresolved for now. Let me know if the above suggestions are helpful and what your results are. Thanks for your interest in the plugin.

    Thread Starter jeff9315

    (@jeff9315)

    Oh … I had forgotten to mention one detail … I do NOT have any issues when I view the page when I’m logged in as Admin. Only when I’m logged in as a normal user. I’m curious if you had tested under both scenarios.

    Adding your debugging code (which I am HUGELY appreciative of since I’ve been trying to figure out how to get that info), you can see that the memory looks fine in Admin mode — starts relatively small and doesn’t change much. However, when I’m logged in as a normal user, it starts out about the same. Then it jumps up significantly at Financial Reports (8th invocation) and then again at Board Resumes (11th invocation). But it’s MOSTLY the real_usage that’s jumping up.

    Logged in as Admin

    Town Hall Statuses (1st invocation)
    before mla_gallery_shortcode memory = 46275104 , real_usage = 48234496
    
    Newsletter – 2014 (last invocation ... #14)
    before mla_gallery_shortcode memory = 49596064 , real_usage = 51380224

    So, you can see when I’m logged in as Admin, there’s no major increase … Just 3 MB.

    Logged in as a normal user

    Town Hall Statuses
    before mla_gallery_shortcode memory = 45742816 , real_usage = 46923776
    
    Town Hall Meeting Notes
    before mla_gallery_shortcode memory = 49056440 , real_usage = 58720256
    
    Town Hall Agendas
    before mla_gallery_shortcode memory = 49963984 , real_usage = 60293120
    
    Forms
    before mla_gallery_shortcode memory = 50876200 , real_usage = 61079552
    
    Contracts
    before mla_gallery_shortcode memory = 51780496 , real_usage = 62128128
    
    Board Agendas
    before mla_gallery_shortcode memory = 51801136 , real_usage = 62128128
    
    Board Minutes
    before mla_gallery_shortcode memory = 54634768 , real_usage = 82837504
    
    Financial Reports
    before mla_gallery_shortcode memory = 66486536 , real_usage = 176685056
    
    Governing Documents
    before mla_gallery_shortcode memory = 70047352 , real_usage = 177209344
    
    Event Flyers
    before mla_gallery_shortcode memory = 75239336 , real_usage = 177733632
    
    Board Resumes – 2016-03
    before mla_gallery_shortcode memory = 87201072 , real_usage = 217579520
    
    Board Resumes – 2015-12
    before mla_gallery_shortcode memory = 89846168 , real_usage = 217841664
    
    Newsletter – 2015
    before mla_gallery_shortcode memory = 92476080 , real_usage = 218103808
    
    Newsletter – 2014
    before mla_gallery_shortcode memory = 101308224 , real_usage = 219152384
    Thread Starter jeff9315

    (@jeff9315)

    OK … I changed the order I’m executing them in to see if that tells us anything interesting … Board Resumes starts out the same as the previous run started. The 2nd Board Resumes jumps up by 30 MB. Governing Docs (11th invocation) jumps up a huge amount. And Newsletter 2015 (13th invocation) jumps up again.

    Board Resumes – 2016-03
    before mla_gallery_shortcode memory = 45743688 , real_usage = 46923776
    
    Board Resumes – 2015-12
    before mla_gallery_shortcode memory = 50933712 , real_usage = 78381056
    
    Financial Reports
    before mla_gallery_shortcode memory = 53592288 , real_usage = 81788928
    
    Town Hall Statuses
    before mla_gallery_shortcode memory = 57633232 , real_usage = 94896128
    
    Town Hall Meeting Notes
    before mla_gallery_shortcode memory = 58536696 , real_usage = 94896128
    
    Town Hall Agendas
    before mla_gallery_shortcode memory = 59449840 , real_usage = 94896128
    
    Forms
    before mla_gallery_shortcode memory = 60366400 , real_usage = 94896128
    
    Contracts
    before mla_gallery_shortcode memory = 61277592 , real_usage = 95158272
    
    Board Agendas
    before mla_gallery_shortcode memory = 61299336 , real_usage = 95158272
    
    Board Minutes
    before mla_gallery_shortcode memory = 63950488 , real_usage = 95420416
    
    Governing Documents
    before mla_gallery_shortcode memory = 75273216 , real_usage = 186122240
    
    Event Flyers
    before mla_gallery_shortcode memory = 80476072 , real_usage = 186646528
    
    Newsletter – 2015
    before mla_gallery_shortcode memory = 92513624 , real_usage = 223084544
    
    Newsletter – 2014
    before mla_gallery_shortcode memory = 101331360 , real_usage = 224133120
    Plugin Author David Lingren

    (@dglingren)

    Thanks for taking the time to add the debugging code, run your tests and post the results; very helpful!

    I have no idea why login credentials or user type would make a difference. It makes no difference at all on my system.

    The “Board Resumes”, “Governing Docs” and “Newsletters” shortcodes all require sorting by a custom field, and all but Governing Documents requires a custom field query as well. I suspect that the extra database work required may be driving up the memory usage, and that would be “real” usage more than “php” usage.

    The debug reporting is complicated by the “before shortcode” format. I would try modifying your code to show the increase required for each shortcode:

    $my_shortcode = "[" . $basic_mla_value . $my_orderby . $my_query . "]";
    $php_usage = memory_get_usage( false );
    $real_usage = memory_get_usage( true );
    $text =  do_shortcode($my_shortcode);
    $memory_use = ' mla_gallery_shortcode php_increase = ' . var_export( memory_get_usage( false ) - $php_usage, true ) . ' , real_increase = ' . var_export( memory_get_usage( true ) - $real_usage, true ) . '<br>';
    return $memory_use . $text;

    Then, I would try running a few experiments with just one shortcode on the page to see if the issue is specific to the type of shortcode, not a cumulative effect.

    Do you have access to PHPMyAdmin or another way of looking at your database? Can you say how large your “postmeta” table is?

    Thread Starter jeff9315

    (@jeff9315)

    OK … I modified the memory stats per your request. Plus I divided by (1024*1024) and rounded to the nearest 100th MB to make it easier to read.

    Here’s an interesting experiment … I used only the “Board Minutes” category. There are 12 documents in that category and it does NOT filter or sort by any custom fields. I increased the numberposts parameter from 1 to 19 and then didn’t specify numberposts at all for the last iteration. It looks like there is about a 3-5 MB increase of real memory for each iteration EVEN after it goes beyond 12. Then, for the final iteration (when I do NOT specify the numberposts), there is a huge jump of over 100 MB so that it needs 244 MB.

    [mla_gallery mla_style="sc-list" columns=1 attachment_category="Board Minutes" post_mime_type=application/pdf link=file mla_item_width="45" orderby="title" order="DESC" ] ...
    numberposts=01 ...
    php_increase ~ 3.17M, real_increase: 3.5M, real_memory: 48.25M
    
    numberposts=02 ...
    php_increase ~ 3.45M, real_increase: 3.25M, real_memory: 51.5M
    
    numberposts=03 ...
    php_increase ~ 2.56M, real_increase: 2.5M, real_memory: 54M
    
    numberposts=04 ...
    php_increase ~ 3.64M, real_increase: 4M, real_memory: 58M
    
    numberposts=05 ...
    php_increase ~ 3.43M, real_increase: 3.5M, real_memory: 61.5M
    
    numberposts=06 ...
    php_increase ~ 3.81M, real_increase: 3.75M, real_memory: 65.25M
    
    numberposts=07 ...
    php_increase ~ 5.2M, real_increase: 5.25M, real_memory: 70.5M
    
    numberposts=08 ...
    php_increase ~ 4.96M, real_increase: 5.75M, real_memory: 76.25M
    
    numberposts=09 ...
    php_increase ~ 5.09M, real_increase: 5.5M, real_memory: 81.75M
    
    numberposts=10 ...
    php_increase ~ 5.51M, real_increase: 5.75M, real_memory: 87.5M
    
    numberposts=11 ...
    php_increase ~ 5.93M, real_increase: 6.25M, real_memory: 93.75M
    
    numberposts=12 ...
    php_increase ~ 6.29M, real_increase: 6.5M, real_memory: 100.5M
    
    numberposts=13 ...
    php_increase ~ 5M, real_increase: 5.25M, real_memory: 105.75M
    
    numberposts=14 ...
    php_increase ~ 4.99M, real_increase: 5.25M, real_memory: 111M
    
    numberposts=15 ...
    php_increase ~ 4.98M, real_increase: 5.25M, real_memory: 116.25M
    
    numberposts=16 ...
    php_increase ~ 4.94M, real_increase: 5M, real_memory: 121.25M
    
    numberposts=17 ...
    php_increase ~ 4.94M, real_increase: 5.25M, real_memory: 126.5M
    
    numberposts=18 ...
    php_increase ~ 4.94M, real_increase: 5.25M, real_memory: 131.75M
    
    numberposts=19 ...
    php_increase ~ 4.94M, real_increase: 5.25M, real_memory: 137M
    
    numberposts NOT SPECIFIED!!!
    php_increase ~ 106.4M, real_increase: 107.5M, real_memory: 244.5M

    Then I ran it with ONLY one invocation on the page … without a numberposts and it’s still high but not as bad.

    php_increase ~ 13.77M , real_increase = 116.5M , real_memory = 161.25M

    I still want to see if I can look at the postmeta table per your request. I also want to try a completely clean install on the same host … no posts, pages or data except for the 12 “Board Minutes” and what’s necessary to run this test. And no plugins other than yours. But I need to see if I have enough space on the server for an extra WordPress installation.

    Thanks for looking into this!!!!
    Jeff

    Thread Starter jeff9315

    (@jeff9315)

    OK … The wp_postmeta table has 4810 rows and is 848 KB … according to WP Cleanup Optimizer.

    wp_postmeta* 4810 InnoDB 848.0 KB

    Plugin Author David Lingren

    (@dglingren)

    Thanks for your additional testing and results. I have sent a considerable amount of effort on this issue as well. Tracking down the source of memory use in this complex environment is quite difficult. I have only preliminary and incomplete results to report.

    Each time the shortcode is run the WordPress WP_Query class retrieves and caches the attachment objects that populate the gallery as well as all of the taxonomy terms and postmeta (custom field) elements for the attachments. In my tests the cache was working properly; invoking the shortcode multiple times and emptying the cache between invocations made memory use much worse, as you might expect.

    I also compared [mla_gallery] with [gallery], the WordPress shortcode. I attached 11 images to a page and invoked [mla_gallery] or [gallery] ten times with no parameters. In this series, both of the shortcodes had similar memory use increases. In fact, [mla_gallery] had smaller increases after the first invocation than [gallery] even though it made five additional database queries for some MLA options and templates. The “attachments” database queries made for the two shortcodes were identical.

    My current theory is that the memory increase is coming from repeated WP_Query invocations and occurs in WP_Query or something it calls in turn. I do not believe it comes from the [mla_gallery]/[gallery] code itself.

    In my tests, the additional memory required for each invocation is small, e.g., 56Kb, 63Kb or 101Kb. I have no idea why your increases are so dramatic. Are there other plugins on your site that may be involved?

    Any additional information you can supply would be most helpful. Thanks for your patience, persistence and understanding.

    Thread Starter jeff9315

    (@jeff9315)

    FIrst off David … I’d like to REALLY thank you for all your help on this. You really went above and beyond. I know I was really getting discouraged trying to find a solution but you helped me debug it.

    I finally determined that the plugin was “PP Content Teaser” that caused the issue. Or at least the interaction between that plugin and yours. “PP Content Teaser” is an addon to the “Press Permit” plugin. So I need to contact that developer to let him know the issue. For me, the good news is that although Press Permit is essential to my site the “PP Content Teaser” isn’t necessary at all (I just never got around to deactivating it).

    Deactivating “PP Content Teaser” dropped the memory footprint from 190 MB to below 70 MB. I haven’t tested it to see how low I can go. But looking at the memory debugging lines I put in, I suspect I can go to 50 MB.

    Thanks again … Jeff

    Plugin Author David Lingren

    (@dglingren)

    That is great news, thanks! Thanks for your kind words as well. This was a tough problem and I always learn something about WordPress, PHP or both by digging in to issues like this one.

    I will leave this topic resolved, but please update it or start a new topic if there’s anything else I can do to help you get the most out of MLA. Thanks again for working with me on this and for your interest in the plugin.

    I’ve put together some logic to suppress all PP Content Teaser activity for queries initiated by the mla_gallery shortcode. This will be included in my next plugin extension update.

Viewing 10 replies - 1 through 10 (of 10 total)
  • The topic ‘Memory Issue cause by number of invocations?’ is closed to new replies.