@kushal ...not a lawyer but I feel like you should be able to just walk away? Of course laws and practise are very different things and there are plenty of very arbitrary laws in Sweden that let cops harass to their hearts content if they don't like you.
To be clear there are also things you are forbidden from taking photos of (skyddsobjekt - think important infrastructure and stuff) but I'm guessing that's not the issue here