I've been using both for several months now, with no apparent compatibility issues. Some functions seem to overlap (lock out functions for the most part) and that can make the logging a bit obscure, but I haven't seen any obvious problems.
Of the two, I have found Wordfence to be a little less robust in general. Going into the logs and blocking IP's for example, can sometimes cause the 'Wordfence Working' spinning wheel to go on forever with an accompanying high server load evident. Loading another page generally sets things right, but it has become a big enough problem that I no longer attempt to add IP's to the blocked list - just too sluggish and prone to going into some sort of (apparently) infinite loop.
When I have the time and interest one day I'm going to attempt to keep the main functions of the two plug-ins separate by configuration. WP-Security seems to be aimed at locking down susceptible files and Wordfence is more of a firewall. Carefully configuring to prevent unneeded overlap would probably make a cleaner installation.