According to an article published on March 31, 2015 by Forbes Magazine, Google will start penalizing websites that are non-mobile friendly starting on April 21, 2015. You can view the article by clicking the link at the bottom of this post.
Essentially, what the article is telling you is that If your website is not a mobile responsive website, you will see a dramatic impact on where the site will show up on search engines such as Google, Yahoo and Bing.
With over 60% of searches taking place on mobile devices, the search engines want to ensure that they are providing their number 1 customers, searchers that is, with the most user friendly experience possible.
Your website should be working for you as a communication tool. With that being said, it is essential that you accommodate your visitors with a user friendly experience that displays optimally across all of the devices they are accessing your website from.