Someone tried this, I saw it one of the Reddit AI subs. They were training a local model on whatever they could find that was written before $cutoffDate.

Found the GitHub: https://github.com/haykgrigo3/TimeCapsuleLLM