gonazar

I’m also wondering if this would even help, since my memory is hitting cap would the child processes have enough resources to do anything? I definitely have room on the CPU as I don’t think it’s ever gone past 50% utilization. PHP only complained about not having enough child processes 7 minutes (19:10) after the server started to hang too (19:03 first 504 warning on access log).

I’m now looking at what is even using the memory as it appears to have some sort of leak. The reason I’m hitting memory cap is that over the course of two months my memory usage climbs from 50% up to 90%.

Supposing I just added more RAM, I feel like that would just raise the cap before it failed again.

Inspecting my current RAM usage I reported that 25% was going to mysql and another 22% was going to Ruby. That ruby process appears to have been running for quite a long time too.

root 22782 0.1 22.6 874212 639944 ? Sl Jul19 74:15 /opt/google-fluentd/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/google-fluentd –log /var/log/google-fluentd/google-fluentd.log –daemon /var/run/google-fluentd/google-fluentd.pid –under-supervisor

Just putting this here for the record. I don’t think I can pull historical RAM usage logs associated with scripts/applications. I’ll have to check it in a week or two to see if this increases at all.

I didn’t see much other than some of the requests in the access logs receiving 504 errors. Before and after the following block is normal GET requests HTTP response 200.

`96.48.118.65 – – [19/Aug/2021:19:03:45 -0700] “POST /wp-admin/async-upload.php HTTP/1.1” 504 247
172.69.34.250 – – [19/Aug/2021:19:10:45 -0700] “GET / HTTP/1.1” 408 221
108.162.215.134 – – [19/Aug/2021:19:11:20 -0700] “GET / HTTP/1.1” 408 221
172.69.35.45 – – [19/Aug/2021:19:11:19 -0700] “GET / HTTP/1.1” 408 221
172.69.35.159 – – [19/Aug/2021:19:12:18 -0700] “GET / HTTP/1.1” 408 221
108.162.215.134 – – [19/Aug/2021:19:12:18 -0700] “GET / HTTP/1.1” 408 221
108.162.215.112 – – [19/Aug/2021:19:12:18 -0700] “GET / HTTP/1.1” 408 221
172.69.34.128 – – [19/Aug/2021:19:12:18 -0700] “GET / HTTP/1.1” 408 221
172.69.35.45 – – [19/Aug/2021:19:16:51 -0700] “GET / HTTP/1.1” 408 221
66.249.65.197 – – [19/Aug/2021:19:07:23 -0700] “GET /robots.txt HTTP/1.1” 504 247
96.48.118.65 – – [19/Aug/2021:19:07:07 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 504 247
96.48.118.65 – – [19/Aug/2021:19:07:10 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 504 247
172.69.34.66 – – [19/Aug/2021:19:18:25 -0700] “GET /favicon.ico HTTP/1.1” 408 221
172.69.35.45 – – [19/Aug/2021:19:18:38 -0700] “GET / HTTP/1.1” 408 221
157.55.39.105 – – [19/Aug/2021:19:09:07 -0700] “GET /robots.txt HTTP/1.1” 504 247
172.69.35.45 – – [19/Aug/2021:19:22:03 -0700] “GET / HTTP/1.1” 408 221
172.69.34.66 – – [19/Aug/2021:19:21:38 -0700] “GET /favicon.ico HTTP/1.1” 408 221
172.69.35.61 – – [19/Aug/2021:19:22:03 -0700] “GET / HTTP/1.1” 408 221
127.0.0.1 – – [19/Aug/2021:19:09:56 -0700] “GET /server-status?auto HTTP/1.1” 200 1749
172.69.35.159 – – [19/Aug/2021:19:27:53 -0700] “GET / HTTP/1.1” 408 221
66.249.73.97 – – [19/Aug/2021:19:09:12 -0700] “GET /events/photo/2017-07-29/ HTTP/1.1” 504 247
127.0.0.1 – – [19/Aug/2021:19:29:10 -0700] “GET /server-status?auto HTTP/1.1” 200 1841
127.0.0.1 – – [19/Aug/2021:19:29:10 -0700] “GET /server-status?auto HTTP/1.1” 200 1841
96.48.118.65 – – [19/Aug/2021:19:29:09 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
96.48.118.65 – – [19/Aug/2021:19:09:40 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
96.48.118.65 – – [19/Aug/2021:19:29:09 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
96.48.118.65 – – [19/Aug/2021:19:29:09 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
96.48.118.65 – – [19/Aug/2021:19:29:09 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
96.48.118.65 – – [19/Aug/2021:19:29:09 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
96.48.118.65 – – [19/Aug/2021:19:29:09 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
207.46.13.154 – – [19/Aug/2021:19:09:19 -0700] “GET / HTTP/1.1” 500 –
216.244.66.243 – – [19/Aug/2021:19:29:08 -0700] “GET /robots.txt HTTP/1.1” 500 –
96.48.118.65 – – [19/Aug/2021:19:29:08 -0700] “GET /favicon.ico HTTP/1.1” 500 –
96.48.118.65 – – [19/Aug/2021:19:29:17 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 390
35.217.65.174 – – [19/Aug/2021:19:29:18 -0700] “POST /wp-admin/admin-ajax.php?action=as_async_request_queue_runner&nonce=ba60591476 HTTP/1.1” 200 –
96.48.118.65 – – [19/Aug/2021:19:29:32 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 98
96.48.118.65 – – [19/Aug/2021:19:29:47 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 98
96.48.118.65 – – [19/Aug/2021:19:30:02 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 98
127.0.0.1 – – [19/Aug/2021:19:30:10 -0700] “GET /server-status?auto HTTP/1.1” 200 1836
96.48.118.65 – – [19/Aug/2021:19:30:17 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 98
96.48.118.65 – – [19/Aug/2021:19:30:32 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 98
96.48.118.65 – – [19/Aug/2021:19:30:47 -0700] “POST /wp-admin/admin-ajax.php HTTP/1.1” 200 98
96.48.118.65 – – [19/Aug/2021:19:30:51 -0700] “GET / HTTP/1.1” 200 41346`

The php log complained about max children?

[19-Aug-2021 19:10:08] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 1 idle, and 9 total children
[19-Aug-2021 19:10:22] WARNING: [pool wordpress] server reached pm.max_children setting (10), consider raising it

And the apache error log I got these during the down period:

`[Thu Aug 19 19:08:46.381186 2021] [proxy_fcgi:error] [pid 11703:tid 140181818435328] (70007)The timeout specified has expired: [client 96.48.118.65:0] AH01075: Error dispatching request to : (polling), referer: https://members.makercube.ca/wp-admin/post-new.php
[Thu Aug 19 19:12:29.011356 2021] [proxy_fcgi:error] [pid 11703:tid 140181801649920] (70007)The timeout specified has expired: [client 96.48.118.65:0] AH01075: Error dispatching request to : (polling), referer: https://members.makercube.ca/wp-admin/post-new.php
[Thu Aug 19 19:12:29.011389 2021] [proxy_fcgi:error] [pid 4268:tid 140182053431040] (70007)The timeout specified has expired: [client 96.48.118.65:0] AH01075: Error dispatching request to : (polling), referer: https://events.makercube.ca/wp-admin/post.php?post=2696&action=edit
[Thu Aug 19 19:13:20.749579 2021] [proxy_fcgi:error] [pid 10396:tid 140181591832320] (70007)The timeout specified has expired: [client 66.249.65.197:0] AH01075: Error dispatching request to : (polling)
[Thu Aug 19 19:17:28.692991 2021] [proxy_fcgi:error] [pid 11703:tid 140181717722880] (70007)The timeout specified has expired: [client 157.55.39.105:0] AH01075: Error dispatching request to : (polling)
[Thu Aug 19 19:29:08.870082 2021] [proxy_fcgi:error] [pid 27766:tid 140182003074816] (70007)The timeout specified has expired: [client 66.249.73.97:0] AH01075: Error dispatching request to : (polling)
[Thu Aug 19 19:29:10.925553 2021] [proxy_fcgi:error] [pid 11703:tid 140181776471808] [client 207.46.13.154:0] AH01071: Got error ‘PHP message: PHP Fatal error: Uncaught Error: Call to undefined function wp_cache_get() in /opt/bitnami/apps/wordpress/htdocs/wp-includes/option.php:97\nStack trace:\n#0 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(53): get_option(‘WPLANG’)\n#1 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(137): get_locale()\n#2 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1278): determine_locale()\n#3 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1240): _get_path_to_translation_from_lang_dir(‘wp-optimize’)\n#4 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1210): _get_path_to_translation(‘wp-optimize’)\n#5 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1308): _load_textdomain_just_in_time(‘wp-optimize’)\n#6 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(177): get_translations_for_domain(‘wp-optimize’)\n#7 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(289): translate(‘User is logged …’, ‘wp-optimize’)\n#8 /opt/bitnami/apps/wordpress/htdocs/wp-content/…’
[Thu Aug 19 19:29:10.926572 2021] [proxy_fcgi:error] [pid 4268:tid 140182174000896] [client 216.244.66.243:0] AH01071: Got error ‘PHP message: PHP Fatal error: Uncaught Error: Call to undefined function wp_cache_get() in /opt/bitnami/apps/wordpress/htdocs/wp-includes/option.php:97\nStack trace:\n#0 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(53): get_option(‘WPLANG’)\n#1 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(137): get_locale()\n#2 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1278): determine_locale()\n#3 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1240): _get_path_to_translation_from_lang_dir(‘wp-optimize’)\n#4 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1210): _get_path_to_translation(‘wp-optimize’)\n#5 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1308): _load_textdomain_just_in_time(‘wp-optimize’)\n#6 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(177): get_translations_for_domain(‘wp-optimize’)\n#7 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(289): translate(‘User is logged …’, ‘wp-optimize’)\n#8 /opt/bitnami/apps/wordpress/htdocs/wp-content/…’
[Thu Aug 19 19:29:10.949346 2021] [proxy_fcgi:error] [pid 10396:tid 140181826828032] [client 96.48.118.65:0] AH01071: Got error ‘PHP message: PHP Fatal error: Uncaught Error: Call to undefined function wp_cache_get() in /opt/bitnami/apps/wordpress/htdocs/wp-includes/option.php:97\nStack trace:\n#0 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(53): get_option(‘WPLANG’)\n#1 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(137): get_locale()\n#2 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1278): determine_locale()\n#3 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1240): _get_path_to_translation_from_lang_dir(‘wp-optimize’)\n#4 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1210): _get_path_to_translation(‘wp-optimize’)\n#5 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(1308): _load_textdomain_just_in_time(‘wp-optimize’)\n#6 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(177): get_translations_for_domain(‘wp-optimize’)\n#7 /opt/bitnami/apps/wordpress/htdocs/wp-includes/l10n.php(289): translate(‘User is logged …’, ‘wp-optimize’)\n#8 /opt/bitnami/apps/wordpress/htdocs/wp-content/…’,`

I don’t think I can update my php, not easily anyways.

I’m running a bitnami stack deployment, which recommends for updating I have to migrate my entire site to a fresh deployment to update the stack dependencies safely.

I’m using a multisite which makes that really difficult so I’m leaning towards not bothering with the update.

Many of the errors in my initial posting were directly related to the wp-optimize plugin. I feel like that other error was just a coincidence.

I disabled the plugin for a few hours and I also got this error:

[Thu Mar 25 13:25:51.163512 2021] [proxy_fcgi:error] [pid 1995:tid 139858672436992] [client 172.70.35.20:46060] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Class 'getid3_handler' not found in /opt/bitnami/apps/wordpress/htdocs/wp-includes/ID3/module.audio.ac3.php:18\nStack trace:\n#0 {main}\n thrown in /opt/bitnami/apps/wordpress/htdocs/wp-includes/ID3/module.audio.ac3.php on line 18'

I don’t think it’s related, but since it’s also a php error I thought I would mention it just in case.

I used to have pagespeed but I already disabled it, I’m in the process of switching to wp-optimize.

I also had W3 cache installed but not activated. I’ve now deleted it but i’m still getting the errors.

I’ve also completely uninstalled wp-optimize and reinstalled it, and I also tried a settings wipe.

I looked at my wp-config and confirmed I have define('WP_CACHE', true); // WP-Optimize Cache at the top of the file.

So I’m not sure if I’ve tapped into my php logs just yet as i’m a bit confused whether it’s logging at all, logging in a file I can’t find or just dumping it into my apache logs.

In my apache logs I’ve found a lot of this:
[Mon Jan 11 06:34:54.288344 2021] [proxy_fcgi:error] [pid 9096:tid 140056198936320] [client 162.158.203.13:44448] AH01071: Got error 'PHP message: PHP Warning: Use of undefined constant \x91COOKIE_DOMAIN\x92 - assumed '\x91COOKIE_DOMAIN\x92' (this will throw an Error in a future version of PHP) in /opt/bitnami/apps/wordpress/htdocs/wp-config.php on line 83PHP message: PHP Warning: Use of undefined constant \x91HTTP_HOST\x92 - assumed '\x91HTTP_HOST\x92' (this will throw an Error in a future version of PHP) in /opt/bitnami/apps/wordpress/htdocs/wp-config.php on line 83PHP message: PHP Warning: Use of undefined constant TEST_COOKIE - assumed 'TEST_COOKIE' (this will throw an Error in a future version of PHP) in /opt/bitnami/apps/wordpress/htdocs/wp-includes/functions.php on line 7601PHP message: PHP Warning: Use of undefined constant COOKIE_DOMAIN - assumed 'COOKIE_DOMAIN' (this will throw an Error in a future version of PHP) in /opt/bitnami/apps/wordpress/htdocs/wp-includes/functions.php on line 7601'

This error accounted for 122,500 of the 125,000 lines in the log over the course of just one week. @__@
It appears to be some leftover debugging crap related to cookies. I adjusted the line in wp_config and removed the extra code in functions.php.

Despite that, I’m wondering whether or not this would dump data into the wp_options table. Other than just filling my error logs, is this actually the source of the issue with my database? I’ll have to leave it running for a bit to see if the problem persists.

Other than that line of code, the thee other most common errors in my log are a lot of:
RateController: drop deferred fetch of "XYZ" on shutdown
and
PageSpeed Serf fetch failure rate extremely high; only 0 of 1173 recent fetches fully successful; is fetching working?
and
[Sat Jan 16 23:14:48.762017 2021] [pagespeed:error] [pid 27045:tid 140056215721728] [mod_pagespeed 1.13.35.2-0 @27045] Slow ReadFile operation on file /opt/bitnami/apache2/var/cache/mod_pagespeed/!clean!time!: 78.661ms; configure SlowFileLatencyUs to change threshold\n