A FeedWordPress syndicated post filter that checks for duplicates before posting items from your feeds.
This is a filter for the FeedWordPress plugin. If you don't use FeedWordPress this plugin will not be useful to you.
I wrote this filter after seeing the same post in my database 32 times. Maybe you've had the same problem, and I hope this filter helps you solve it.
The filter works by hooking the "syndicated_post" action of FeedWordPress and the "save_post" action in the core of WordPress.
For each potential post from a feed, the plugin attempts to find an identical SHA1 hash of the first 1024 non-whitespace characters (stripped of HTML tags) of the post content. If it finds a match the post is skipped. If not, the post is processed and a hash is inserted into the post's metadata. (Stored as key _dpf.)
NOTE: No filter is going to be 100% accurate. This filter will stop most or some duplicates, but in all likelihood, will not stop all of them. If you want a filter that stops all duplicates, this isn't your solution. In my test installation I had 5 posted duplicates out of 125 syndicated posts (4% false negative) out of a corpus of about 300 duplicates. Most of those false negatives were due to slightly different HTML markup in the post content itself. So my approach isn't perfect, but it is "good enough" for me.