If a lot of people have that as their hobby, and some are careless or malicious, people will be harmed. Now suppose that it's not simple to stop the offenders directly. Instead, restricting the sale of nuclear isotopes or cholera samples would probably be highly effective.
Yes, you should be allowed to do those things as long as you do it without harming others.