Zimbra: A network service error has occurred
Lately from time to time on one of my Zimbra servers I was getting A network service error has occurred error message. Restarting the Zimbra service was fixing the issue but it was time to find out why this is happening.
This is the the screen I’m getting and nothing in my log files:
Now my problem was that I’m using a nginx proxy to route the traffic to my server and I changed the proxy server IP address and forgot about the Zimbra DosFilter Throttling Mechanism
The denial-of-service filter or DoSFilter was added to the mailbox server in ZCS 8.0 to throttle clients sending a large number of requests over a very short period of time. The DoSFilter is applied to all requests for service, mailbox and admin.
My solution was to add my nginx ip to zimbraHttpThrottleSafeIPs
For Zimbra Servers 8.7 and above you need to add CIDR address
zmprov mcf zimbraHttpThrottleSafeIPs 10.1.2.3/32 zimbraHttpThrottleSafeIPs 192.168.4.0/24
or append
zmprov mcf +zimbraHttpThrottleSafeIPs 10.1.2.3/32 zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.4.0/24
For Zimbra Servers 8.6, 8.5 and previous you need to add each IP
zmprov mcf zimbraHttpThrottleSafeIPs 10.1.2.3 zimbraHttpThrottleSafeIPs 192.168.4.5
or append
zmprov mcf +zimbraHttpThrottleSafeIPs 10.1.2.3 zmprov mcf +zimbraHttpThrottleSafeIPs 10.1.2.50 zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.4.5 zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.4.6
Finally you need to restart the Zimbra Mailbox server
zmmailboxdctl restart
You are now done and should not have this issue anymore.
Additionally you can also adjust the DosFilter parameters
zmprov mcf zimbraHttpDosFilterDelayMillis 20 zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec 100
Also from 8.5 and above you can Block IPs on Repeated Failed Login
Set the DoS filter to block an IP after 5 failed login attempts for 25 minutes, you would do this:
zmprov mcf zimbraInvalidLoginFilterDelayInMinBetwnReqBeforeReinstating 25 zmprov mcf zimbraInvalidLoginFilterMaxFailedLogin 5 zmmailboxdctl restart
More info on Zimbra Wiki: https://wiki.zimbra.com/wiki/DoSFilter
Thank you Nikola Stojanoski.
Just an add guys. To check the current value of these variables, use the zmprov gcf command. Example, to check the valeu of the variable zimbraInvalidLoginFilterMaxFailedLogin, use the command: zmprov gcf zimbraInvalidLoginFilterMaxFailedLogin