On Debian Trixie, Dovecot gets upgraded to version 2.4 which makes substantial changes to the configuration syntax and also to the log output. Unfortunately, the new log format for unsuccessful authentication might not match the rules defined in /etc/fail2ban/filter.d/dovecot.conf. This bug has been documented in the Debian bug tracking system as #1107903 but remains unaddressed as of today.

To mitigate this, i have acquired a more recent filter definition from fail2ban GIT:

repo=https://raw.githubusercontent.com/fail2ban/fail2ban
wget \
    -O /etc/fail2ban/filter.d/dovecot-new.conf \
    "$repo"/refs/heads/master/config/filter.d/dovecot.conf

I tested the filters, comparing the filter definition distributed by Debian Trixie with the upstream one:

fail2ban-regex /var/log/mail.log /etc/fail2ban/filter.d/dovecot.conf
fail2ban-regex /var/log/mail.log /etc/fail2ban/filter.d/dovecot-new.conf

I then applied these changed filters to the jail definitions in /etc/fail2ban/jail.d/local.conf:

[dovecot]
enabled = true
# This changes the filter:
filter = dovecot-new
# Note: The other settings might be different at your site:
maxretry = 3
logpath = /var/log/mail.log

[postfix]
enabled = true
filter = postfix-new
maxretry = 3
logpath = /var/log/mail.log

I restarted fail2ban.service and followed the fail2ban.log to see if it operates as expected (which it does):

systemctl restart fail2ban
tail -f /var/log/fail2ban.log