It only takes "relatively few" to be a huge problem. Most serious threats come from nation states and criminal gangs, and they definitely do have the ability and resources to train top models. Beyond that though, I would bet many of the nation states even have access to a version of OpenAI/Google/etc that allows them to do this stuff.