Yes, from the index.php page of the template folder, delete the code <div id="featured-cats"> all the way through to it's closing </div> tag.
Be careful not to delete too much. Undo and try again if the site layout goes really wrong. The correct </div> tag to delete up to is the one just before the line <!--END FEATURED CATS-->. There should now be a white box where the featured categories used to be.
To remove this white box, open up style.css. Edit the settings for #more-posts, found by scrolling down the page till you encounter it or by tapping Ctrl/Cmd and F and typing in 'more-posts'.
When you find the element, change float:right; to float:left; and then change the width to width: 600px;. These should be the only CSS settings for more-posts.