An existing implementation of that are Smart Cards. They contain a Public-Private key pair, and they will actually perform the crypto themselves (when plugged into a reader), so the private key never leaves the card.
Here in Portugal our new ID cards (Citizen Card) are already smart cards, so we can log in to e.g. our IRS using it, at least with browsers which support PKCS#11, like Firefox.
You need to ask for a new one, with its own key pair (and the old one will be revoked).
As for a possible attacker that found your card:
The card has a PIN and locks out after three failed tries. After locking out, you can only unlock it by going to an IRS office where they authenticate you using photo/fingerprint/etc and then using their machine and inserting an unblocking code that only you have. If you don't know the code, you need to ask for a new card.
Here in Portugal our new ID cards (Citizen Card) are already smart cards, so we can log in to e.g. our IRS using it, at least with browsers which support PKCS#11, like Firefox.