Dear Ipso Fatso,

«The one exception would be defending yourself from harm.
But that would mean someone else was already breaking the rules.»

EXACTLY!

But someone has already been breaking the rules by forcefully counting you in their group and subjecting you to its laws and regulations without first asking for your consent.

Assuming you never impinged unreasonably on the life of others,
never harmed another person,
never stole from another person,
others too may not impinge unreasonable on your life!
The fact that these particular others happened to form a group and called it by whatever name (such as "society", "the State" or "Australia"), makes no difference whatsoever in the rules that apply to them.

Hi IF,

There are always restrictions, "If you lived totally alone, you could do what you liked." No, not so, for example, you can't talk to another person. Even totally alone you are still restricted in what you can and can't do.

"You must never harm another person: ever!
You must never steal from another person: ever!"

Well, you have moved into the cave next door to me, and by doing so you have stole my amenity, you have cased me harm. I have to defend myself from harm, as you have broken the rule, my rule.