I agree with you here, and go much further to include any drug that is not a systemic risk. A systemic risk might be things like antibiotics, because antibiotic resistant bacteria are a systemic risk. My daughter's asthma inhaler and grandma's heart medication are certainly not. Let me buy these things off the shelf without anyone's permission. Hopefully once we get pot legalized everywhere for recreational use, we can move on to everything else.
Sure, there are difficult cases in this domain that require more complex moral reasoning. But as you say, for drugs that are simply an individual putting something into their own body without infringing on others, there is no justification for one group having monopoly authority to decide whether that's "safe".