RSS is a XML format (actually WordPress supports three different formats with the same purpose) that allows others to import your headlines to their pages. This is important for news aggregation sites, for example, which provide an overview on the news that are provided by a huge amount of pages. But it also allows other blog users to import news items that they find interesting enough. If you want to know more about RSS, you should take a look at What is RSS?, an article by Mark Pilgrim.
You can turn off RSS support by simply renaming and/or deleting the files b2rss.php, b2rss2.php and b2rdf.php from your WordPress directory.
Pingback is a feature that allows you to inform other bloggers that you referenced a blog entry in your blog. "Referencing" in the meaning of "I mentioned your URL in one of my blog entries, which can be found here". This, as well as TrackBack, allows some kind of virtual meshing between blogs.
Hope that helps.