While it's true that most sites will benefit from having a responsive site eventually. It can introduce a lot of headaches in maintenance - especially if you make site changes on a regular basis.
The best way to determine if your site would benefit is to check your analytics. Do you have Google analytics installed? If so, you can see how many of your visitors come on mobile phones or tablets. Log into the Analytics dashboard and go to Audience > Mobile > Overview and scroll down to see the percentage of visitors using desktop, mobile and tablets.
This will help you evaluate the ROI for the time it takes to convert and maintain and responsive website.