No, there are an infinity of things that shouldn't have access to your secret keys. So you have to take a default deny approach, and ensure that only things that positively should have access to your secret keys do.