I found an even bigger flaw.
I have access control as a toggle on the notification shade, that takes you to the setting where you turn it on. All ok so far, but if you then do the same when it is active, you can turn it off without entering the password/pattern!
Protecting apps with a password...