I use Android, I can't just deny permission and not use a feature. I would have to stop using Spotify alltogether.

It's not the same as giving permission to Google Maps to use my location because I know exactly why it wants that. With Spotify, "improving our services" is just not a good enough reason.