This would be even more difficult to achieve than previous attempts (e.g. in the Linux kernel [0]) due to the fact that an attacker needs to corrupt thousands of repositories that are guaranteed to be part of the training set.
Potential attackers would have two problems: 1) getting malicious checked into many repos and 2) making sure that these repos find their way into future deployed versions of GPT-3/Codex/CoPilot.
CoPilot generates enough vulnerable code as-is [1], so the extra effort isn't even required.
Crafting might not be necessary. You might find a vulnerability in a commonly copiloted piece of code, and now you can exploit it in many projects. Better yet, those snippets cannot be updated even if Copilot improves, and there is nothing to file a CVE against either.
The number of people who never write a vuln normally but would write a vuln if they were using machine synthesized code has got to be fewer than ten people on the planet.
This worries me.