fail2ban is based on the horribly flawed premise of correctly parsing arbitrary text logs from tons of different programs that wrote their log format without any concerns about parsing that output later. Not surprisingly there have been numerous vulnerabilities in fail2ban that let an attacker ban arbitrary hosts such as your DNS server, database, etc.
A better approach is something like pam_abl which is a pam module that will accomplish mostly the same thing but only for login attempts and without the crappy plain text log parsing.
https://www.google.com/search?q=fail2ban+dos+bug
A better approach is something like pam_abl which is a pam module that will accomplish mostly the same thing but only for login attempts and without the crappy plain text log parsing.