If you haven't been advised by your SEO provider or website developer yet, Google is to start using the page speed and load time for your mobile search in its search engine result pages and this could affect your SEO ranking.
Google is calling it the "Speed Update" and it has said it will only impact a small percentage of queries. It has said only pages that "deliver the slowest experience to users" will be impacted by this update. That means if you've not got your site mobile-friendly, you'll be pushed well down the list.
Be aware one problem you might have is with Google's PageSpeed Insights report in that as it only uses data from the Chrome Browser, it doesn't have enough data to reliably measure smaller sites. That means the speed portion of the report is or mightn't be unavailable for those users. Note that optimisation scores are still available but you might not be able to tell if your site's pages are deemed "slow".