It appears that this issue was ultimately due to the fact that there is a separate memory limit for the WordPress Admin that is unaffected by the WP_MEMORY_LIMIT definition. I was able to resolve the issue by modifying line 96 of /wp-admin/admin.php as follows:
@ini_set( 'memory_limit', apply_filters( 'admin_memory_limit', '2048M' ) );
Typically this is adjusted on your server in the php.ini file as it’s a server limitation not a WP limitation.
We had the Memory Limit in the php.ini set to 4096M and the Memory Limit in the WordPress config set to 1024M (as is noted in the original post). As it turns out, any scripts running under the admin have to have the memory limit defined within the /wp-admin/admin.php. By default it is set to 256M. So in fact, this is a WordPress limitation, not a server limitation.
Why would you set the memory limit to 4GB and the WP limit to 1GB! You are limited by your web host as to how high you can set it and that is not the way to write the code to set it.
256M which is 256MB should be plenty of memory for just about any website. No web host is going to allow you 4Gb and 1GB of memory.
BTW, WP runs on top of the server OS and the server OS controls the resources of the server. So even though your WP plug-in may allow you to set a limit doesn’t mean the server OS will allow you to have it. So in reality the “Server” controls how much memory your application is allowed to use.
@gixxer188, The PHP Server limit comprises the upper limit of memory available to all PHP scripts together. As was stated in our original post, we are importing a 128MB file, which, during the import process checks for existing posts and does a lot of string parsing. This consumes a very large amount of memory when working with a file of this size and quickly exceeds the hard coded admin limit (admin_memory_limit) of 256MB.
If you want to only import one file at a time, then you would not need the PHP memory limit to exceed the limit of the admin_memory_limit. However, we are importing more than one file at a time. In this scenario, concurrent scripts running in the admin would push the limit past 256MB very quickly.
I resolved this issue because I discovered that this was reported as a bug by a Plugin developer and WordPress has acknowledged that it is a bug and will, in the future, add a configurable setting in the wp-config.php file that can modify the “admin_memory_limit”.
Additionally, we run our WordPress server on a dedicated machine with over 16GB of memory (mostly for caching). We have hundreds of authors, hundreds of thousands of posts and millions of users. When you work with a database as large as ours, WP can handle it, but often needs a few modifications.
@gixxer188, You are correct. I never asserted otherwise and in fact, that is the very reason why we have the server limit set higher than the WP limit as the server uses PHP for other functions other than just WordPress.
However, in this specific case, the problem was not the limit set at the server level, it was in fact the hard coded limit inside WordPress for the amount of memory that could be used by any script running inside the admin panel.
