WordPress.org

Ready to get started?Download WordPress

Forums

RSS Aggregator (22 posts)

  1. kellan
    Member
    Posted 10 years ago #

    There are several high quality WPHacks which allow you to display one or more RSS feeds on your blog. However what I wanted to was to use WP as an RSS aggregator, periodically polling a number of RSS feeds, and inserting each new item as a new WP post. There might be an existing implementation of this idea, but I didn't find it, so I whipped up something quick this morning.
    Features:

    • tries to be smart about mapping DublinCore fields to WP fields
    • supports automatically categorizing new posts based on their originating feed
    • uses postmeta to keep track of previously seen posts

    Limitations:

    • Ignores mod_content, only works with RSS 1.0
    • Only supports inserts, not updates
    • Built against nightly build 2004-7-14, ymmv

    Download at: http://laughingmeme.org/code/wp-rss-aggregate.php.txt
    This is just a quick hack, but I thought I'd share. Among other things, there might be a couple of functions in there that would be useful to other hack authors.

  2. Alan
    Member
    Posted 10 years ago #

    That would be a neat hack for WP...if one doesn't exist already. Drupal has some of the infrastructure already in place for parts of what you'd like to do. It has a built in aggregator and php cron file...and it lists all the news items with a special link icon to "blog it"...thus filling out most of the form with what you need to start a post--including the quoted excerpt. It would be very cool to see something like this built into WP.

  3. kellan
    Member
    Posted 10 years ago #

    Hi Alan,
    The Drupal aggregator sounds more like a "reBlogging" system, where you're picking and choosing from a number of incomimg feeds.
    Removing the manual intervention was definitely key to what I was wanting to do.
    Thanks for the feedback.

  4. davidchait
    Member
    Posted 10 years ago #

    I had thought about something like this as an extension to CG-FeedRead (or part of the PowerPack), but wasn't sure there would be enough users to make the time investment working. At the moment, the CG-Amazon stuff is keeping me busy, as is my site, but I'll keep this idea on the backburner, see if I can find an interesting way to do this.
    -d

  5. Vohiyaar
    Member
    Posted 9 years ago #

    Kellan,
    Nice tool. Do you have a howto for setting this up?
    -Vohiyaar

  6. kellan
    Member
    Posted 9 years ago #

    Hi Vohiyaar,
    Here is a quick howto. The code leverages the WP libraries, but is essentially a stand alone application in that it isn't designed to be used as a WP hack, or WP plugin.
    While you don't have to place wp-rss-aggregate.php in your WordPress directory (and I don't) that is probably the simplest way to set it up (or you could place in in wp-admin)
    Open up wp-rss-aggregate.php in a text editor and you'll see a define, and 4 require_once's at the top of the file. Edit the define and first two requires to point at where you have MagpieRSS installed on your system. Edit the second to requires for where you have WordPress installed. (e.g. if you place wp-rss-aggregate.php in your WP folder you'd want the require to read: require_once './wp-config.php';)
    Now run it from your web browser, make sure it works, (no error messages), and that should aggregate any feeds you've configured in the feeds array. (e.g. http://example.org/wordpress/wp-rss-aggregate.php)
    Presumably you want the aggregation to happen automatically, so the next step is to addd a cron job that runs periodically requesting this page. To do this you'll need cron, understanding of crontabs, and a command line web browser (lynx, links, curl, wget, GET would all work). FeedOnFeeds has an example of this.

  7. Anonymous
    Unregistered
    Posted 9 years ago #

    Hello! I tried using the wp-rss-aggregate-script, but I keep getting SQL errors. The table name is missing from all the statements, so I guess the $wpdb->users calls are not working (does not generate any output). I know very little PHP, can anyone point me to what I am doing wrong?

  8. kellan
    Member
    Posted 9 years ago #

    manne, you have to use WP 1.3, grab a nightly build

  9. Anonymous
    Unregistered
    Posted 9 years ago #

    That did it! Thanks a bunch. :)

  10. Anonymous
    Unregistered
    Posted 9 years ago #

    I love this script. I use it now to get my links from de.icio.us and republish them in a separate wordpress blog. Works great. I changed the script slightly as so to hack up all the space separated tags from del.icio.us using explode() on the dc:subject-tag before they are inserted as categories in WordPress, works like a charm.
    One question though: I just tried to add a link to del.icio.us and then run the script again. Expected result would be that one new linkpost would be published, but nothing happened. A total of zero posts were added. How come?

  11. kellan
    Member
    Posted 9 years ago #

    Could be that Magpie still has a cached version of the del.icio.us RSS feed, and therefore didn't re-fetch. Default caching time is 1 hour.
    Ditto I wouldn't be surprised is Joshua has a lag between when you add an item to del.icio.us and when they show up in your RSS feed.

  12. klintron
    Member
    Posted 9 years ago #

    Could anyone tell me what to change in this script to keep it from adding new categories based on del.icio.us tags? I'd like to just dump all my del.icio.us bookmarks into one category.

  13. klintron
    Member
    Posted 9 years ago #

    A more major problem: how do I get the del.icio.us links to actually work? They end up in a "custom field" but not in the actual posts anywhere. What's up with that? My blog: http://www.klintron.com/brain

  14. kayaker
    Member
    Posted 9 years ago #

    Anyone know who the most recent "Anonymous - Unregistered" poster is? This post!

    I want to know how this all works for him/her. If there is any action on the forum anywhere else please post. I've got this thread RSS'd.

  15. plainsman
    Member
    Posted 9 years ago #

    Is there any further documentation for this script - particularly explanations of wp-agg.ini (how to use/config) and the exact place to add the various RSS feeds to pull in as posts in the blog.

    Thank you.

  16. edwinkort
    Member
    Posted 9 years ago #

    I tried this script, but after a few days I gave up. I can't figure it out. Can't find the wp-agg.ini file anywhere. Will give it another try in the next week or so.

  17. mirdin
    Member
    Posted 9 years ago #

    Does anyone have this working on WordPress 1.5? A short howto for 1.5 would be appreciated :)

  18. Ap0s7le
    Member
    Posted 9 years ago #

    mirdin,

    Simply upload the files as stated in the file itself.
    After you've done that, create a file named "wp-agg.ini" in the same directory. Format that file like so for each feed you're posting.

    <begin example>
    [Name of source just for personal reference, this is actually ignored]
    url = http://laughingmeme.org/index.rdf
    cats = LM
    </begin example>

    The cats are the category you want the post entered under, these can either exist or upon running of the script can be created. You can specify multiple categorys by seperating them with :

    Example: cat1:cat2:cat345

    Did this help any?

    later

  19. nickm
    Member
    Posted 9 years ago #

    The only issue I've had with this is when I collect a feed from another WP site, it aggregates fine, but adds a very incorrect date. For example, I ran it today which pulled in 10 articles, and promptly dated them 1969-12-31 at 3:59:59 pm.

    Any ideas on how to fix this?

  20. Nom
    Member
    Posted 9 years ago #

    Coincidence seeing this post - quoted it a couple of days ago from search results for 'lynx' and 'curl' words causing problems with admin access and errordoc and not saving problems - same symptoms on trouble board now, perhaps, so mentioned the post - podz problem had been 'curl', mine 'lynx', put together they came up with the cron terms quoted above. 'WGET' will also strangle a post. podz 'flagged' for hackers...

    Only reason to reminisce here, however, is that if aggregating automatically to posts are these same terms going to hang up your rss processing?

    http://wordpress.org/support/topic/37489

  21. nickm
    Member
    Posted 9 years ago #

    I've looked over the post you mention, but don't see a similarity... My issues is primarily with the date issue. I don't seem to have any problems with certain words.

  22. theodusian
    Member
    Posted 9 years ago #

    What you're looking for is CG-FeedRead. I already use this to bring posts from a Podcast and a Xanga blog into my church's website. It works very well and is incredibly customizable, but be sure to read the documentation.

Topic Closed

This topic has been closed to new replies.

About this Topic