one3rdnerd

All good points, thanks for the input, very useful. I guess in this situation this approach isn’t ideal.

I have solved it without this plugin now and I have adopted an approach that just puts everything in the traditional WP media folder structure.

For those that run into an issue like this with 7000+ folders and files, my approach was

1) create a CSV file of the content I was importing (along with image URLs). I used ScrapeStorm ( a standalone program).

2) Migrate the folder structure for the images via FTP or file manager to the new server or site folder.

3) Search and replace the old url path to the images to the new one.

4) Use WP All Import to import and map the new content along with the image import/upload.

One other issue I had that this solved – In the old folder structure there were 100s of images with the same name nested in different folders (hence the structure).

During the import with WP All Import I set the import and upload of images to rename the image after the post name + a number for multiples or duplicates.

All working great now so marking this topic as resolved and hopefully the above is useful to someone else who runs into my unique situation.

Cheers fellas, @alanp57 and @psmits1567 you have both been great.

Ah the host got back to me after investigating further and said the following

We have investigated the case in more details.

The folder scan process should work in account with less folders, but currently there are 7000+ folders in the images subfolder:
     0.000070 lstat("/home/customer/www/timothyb3.sg-host.com/public_html/wp-content/uploads/images/available/ballpythons/12cphpm1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
     0.000055 stat("/home/customer/www/timothyb3.sg-host.com/public_html/wp-content/uploads/images/available/ballpythons/12cphpm1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
     0.000146 sendto(4, "\254\0\0\0\3SELECT ID FROM wp_posts\nLEFT JOIN wp_postmeta AS pm ON pm.post_id = ID\nWHERE pm.meta_value = 'images/available/ballpythons/12cphpm1' \nand pm.meta_key = '_wp_attached_file'", 176, MSG_DONTWAIT, NULL, 0) = 176
     0.000100 poll([{fd=4, events=POLLIN|POLLERR|POLLHUP}], 1, 1471228928) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
     0.044216 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=35930, si_uid=99} ---
     0.000072 rt_sigaction(SIGTERM, NULL, {0x90ac00, ~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], SA_RESTORER|SA_SIGINFO, 0x7f1e60d9f600}, 8) = 0
     0.000104 rt_sigaction(SIGTERM, {SIG_DFL, [], SA_RESTORER|SA_SIGINFO, 0x7f1e60d9f600}, NULL, 8) = 0
     0.000042 rt_sigprocmask(SIG_UNBLOCK, [TERM], NULL, 8) = 0
     0.000054 getpid()                  = 35931
     0.000118 kill(35931, SIGTERM)      = 0
     0.000071 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=35931, si_uid=1415} ---
     0.015776 +++ killed by SIGTERM +++

I wonder if there is a way for the plugin to run in batches of 100 at a time with breaks inbetween? Would that resolve my issues?

This is from a legacy site that used a nested and deep media directory structure but the site is low traffic so I can’t really justify to the client upgrade hosting to solve something that is a one-off as part of the migration.

Is that available in the pro version?

Appreciate your input gents.

I just checked the plugin on another site, also on Siteground but using a legacy account with CPanel access and the main page loads with a folder tree in the top left on the main plugin page.

But on my other site with the issue, this just shows a spinning loading image.

So it has to be something unique to that WordPress instance or the hosting environment just weird they share most of the same plugins and are both hosted by the same company on the same package. The only distinction between the two hosting accounts is that one is old and has CPanel and one is new and has Sitegrounds new proprietary replacement for CPanel which is a result of the fallout after CPanel’s pricing hike.

Just adding some additional info if it helps.

Hi Alan,

Thanks for your reply.

Yes, I tried this and have worked with the host “Siteground” to increase everything and test again and still running into the same issue.

Siteground max execution time is 120 seconds, which is what I have it set at but the plugin just starts loading and doesn’t get anywhere until the 500 error or a 504 error.

what makes this plugin take longer to complete its action? Is it the number of new files and folders it’s looking for/finding?

Is there a way to process it in smaller batches?

Folder permissions are 755 @psmits1567

Thanks, I submitted a support message through that link.

I managed to resolve this. While it fixed it turning off Yoast, the issue was elsewhere with a piece of PHP so I resolved it without needing to mess with the plugins.

So you are aware, the code that was causing the issue while yoast was active was

function custom_genesis_page_builder_attributes_site_inner( $attributes ) {
     if ( class_exists( 'FLBuilderModel' ) && temp_is_builder_enabled() || FLBuilderModel::is_builder_enabled() ) {
	$attributes['role']     = 'main';
	$attributes['itemprop'] = 'mainContentOfPage';
	return $attributes;
}
}

I changed the above to the below and it fixed the issue. The above code worked fine for the last 2 years on maybe 50 different websites but now seems to throw that issue. 

function custom_genesis_page_builder_attributes_site_inner( $attributes ) {
     if ( class_exists( 'FLBuilderModel' ) && temp_is_builder_enabled() || FLBuilderModel::is_builder_enabled() ) {
	$attributes['role']     = 'main';
	$attributes['itemprop'] = 'mainContentOfPage';
	 }
	return $attributes;
}