Support » Plugin: Redirection » How to Deal with Millions of Records in 404 Redirection Log

  • Resolved americanninja

    (@americanninja)


    The other day I noticed my WordPress database had grown to more than 1.4GB, which made no sense to me. Upon further investigation, I found that it was mostly due to the 404 error log in redirection having roughly 3.5 Million entries.

    I was able to luckily spot the DELETE ALL button (I suggest the owner of this plugin, move this button to the top). Not sure why its hided at the button, and as many people have posted on this forum, it seems most do not notice it. Luckily i did and it said me from going through page by page to delete these.

    Anyway, I cleared everything, and then this morning after checking the table again using WP Optimize, I saw that another 20k were added in (just 10 hrs later). Which means this will grow again quite rapidly.

    How can I resolve this? I don’t understand why this error log table is growing so quickly and why entries are being added. I did massively change the site from a previous Drupal site with different URL paths, however, that was more than a year or so ago, why am I still getting so many 404s. Is there something else that’s going wrong here?

    Thanks!

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Author John Godley

    (@johnny5)

    Have you configured your expiry options?

    https://redirection.me/support/options/

    why am I still getting so many 404s

    Please see https://redirection.me/support/tracking-404-errors/

    however, that was more than a year or so ago, why am I still getting so many 404s

    Google and other search engines won’t forget your old URLs for many, many years. I have a 12 yr old website which to this day still receives occasional probes from Googlebot after a shop page I briefly tested in the site over 11 yrs ago!

    You can try to create a redirect matching the old URLs with a “Do nothing” action. That will prevent the 404s from these URLs from being added to the log.

    Or you can do this with a match for any URL and Googlebot’s User Agent. Then monitor Google’s 404s on Google Search Console.

    Thanks for the posts and advice both of you! @johnny5 I did take a look at the expiry options. I have Redirect Logs set to “1 week” and 404 Logs set to “1 day”. And I just jumped over to see what’s in the 404s log and there are already 36,000 entries. It doesn’t seem to be exactly 1 day though, as I see records in with a date stamp of today on the 27th and the oldest on the 25th. But nothing before then. However, I haven’t cleared this table in the past week, so it does appear to be working, just not exactly one day (not sure how it calculates).

    Anyway, what’s strange is, I didn’t even touch these settings when I went to clear out the 404 logs when it had 3M records. So I would assume it was already set with 1 day expiry. So very strange, maybe some how my site got 3M hits within a 24 hr period to the dead links. I wouldn’t think so, but not sure what else would have caused it.

    Ill monitor the 404s log now to see if it sticks to clearing out the logs automatically within a 1-2 day delay.

    If you need to clear the redirection or the 404 logs out the WPS Cleaner plugin has an option for the redirection plugin for it. You can then purge both logs if needed.

    wbcorus

    (@wbcorus)

    I am having the same issue.

    I deleted all the logs, set the expiry date for both logs and redirects to 1 day. Today is the 4th afternoon and I still see logs from the 2nd in the morning.

    It seems that expiry date is being ignored.

    Plugin Author John Godley

    (@johnny5)

    The log is cleared using WP Cron, and will be slowly cleared so as not to affect performance of your site.

    Please start a new thread instead of posting on a 5 month old thread.

Viewing 6 replies - 1 through 6 (of 6 total)
  • You must be logged in to reply to this topic.