Every few months a client sends me a screenshot of a speed test showing a low score and asks if their site is "getting penalized." Almost never. Speed matters, but it's one signal among many, and most sites lose more from thin content and bad structure than from being a second slower to load.
Still, it's worth getting right, because the parts of speed that hurt you the most are also the cheapest to fix.
What's actually worth your time
Image size is the single biggest offender I see. A photo taken on a modern phone can be 4-8MB. Uploaded straight to a website, uncompressed, that one image can add several seconds to a page load on a phone connection. Compressing images before upload, or letting your hosting platform do it automatically, usually fixes 70% of a slow site's problem in an afternoon.
After that, look at what's loading that doesn't need to. Plugins, tracking scripts, chat widgets, font libraries you added once and forgot about — each one is a small tax on load time, and they add up. I've audited sites running four different analytics tools nobody remembered installing.
What's usually not worth chasing
Getting a perfect 100 on a speed testing tool is a rabbit hole with diminishing returns. Beyond a certain point — roughly, a page that loads and becomes usable in under three seconds on mobile — you're optimizing for a number, not for users or rankings. I'd rather a client spend that time writing a better product description.
Speed is a threshold, not a leaderboard. Once you clear it, other things matter more.
A simple way to check where you stand
Google's own PageSpeed Insights tool will tell you your real score and, more usefully, exactly which specific files or images are slowing you down. Run it, fix the top two or three items it flags, and move on. Don't let it become a permanent to-do list — I've seen businesses spend weeks chasing a speed score while their actual pages sat half-written.