1

i have an VPS with 8GB memory and running on Ubuntu 16.04.02 with Plesk Onyx.

Since a while regularly the system crashes on dovecot and/or postfix.

Dovecot version: 2.3.4.1-ubuntu16.04.19031416 Postfix version: 3.1.0-3ubuntu0.3

I can't find how to solve this.

This server mainly is used for e-mail (imap) and 1 small Wordpress website with not much traffic.

Sometimes it's enough to restart the dovecot or postfix service. But sometimes I have to restart my whole server

Error in mail.log from the start of the crash.

> Jun 18 11:55:30 h2648771 dovecot: imap-login: Error: master(imap):
> net_connect_unix(imap) failed: Cannot allocate memory
> (client-pid=29864, client-id=1, rip=188.207.4.90, created 0 msecs ago,
> received 0/4 bytes) Jun 18 11:55:30 h2648771 dovecot: imap-login:
> Error: master(imap): net_connect_unix(imap) failed: Cannot allocate
> memory (client-pid=29866, client-id=1, rip=188.207.4.90, created 0
> msecs ago, received 0/4 bytes) Jun 18 11:55:30 h2648771 dovecot:
> imap-login: Error: socketpair() failed: Cannot allocate memory:
> user=<info@springstof.eu>, method=PLAIN, rip=188.207.4.90,
> lip=85.214.19.175, TLS, session=<H9L2HJaLeX28zwRa> Jun 18 11:55:30
> h2648771 dovecot: imap-login: Error: net_connect_unix(anvil) failed:
> Cannot allocate memory Jun 18 11:55:30 h2648771 dovecot: master:
> Error: service(imap-login): command startup failed, throttling for 2
> secs Jun 18 11:55:30 h2648771 dovecot: imap: Error:
> net_connect_unix(/var/run/dovecot/stats-writer) failed: Cannot
> allocate memory Jun 18 11:55:30 h2648771 dovecot: imap: Error:
> net_connect_unix(/run/dovecot/auth-master) failed: Cannot allocate
> memory

etc/dovecot/dovecot.conf:

## Dovecot configuration file

# PLEASE DON'T EDIT ANYTHING IN THIS FILE! ANY CHANGES WILL BE LOST ON UPGRADE.
# Instead add your custom configuration to /etc/dovecot/conf.d/
# See the end of this file for details on customization.

# See example configuration files in /usr/share/doc/dovecot/example-config/
# See stock configuration files in /usr/share/doc/plesk-dovecot/dist-config/


# Space separated list of wanted authentication mechanisms.
# NOTE: See also disable_plaintext_auth setting.
auth_mechanisms = plain login digest-md5 cram-md5 apop

# Valid UID range should include popuser.
first_valid_uid = 30

# List of allowed characters in username. "'&" is added here to the default list.
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890&.-_@'

# Plesk Maildirs layout.
mail_home = /var/qmail/mailnames/%Ld/%Ln
mail_location = maildir:/var/qmail/mailnames/%Ld/%Ln/Maildir

# Enable Maildir++ quota support.
mail_plugins = $mail_plugins quota

plugin {
  # Take quota limits from maildirsize file. If it doesn't exist, don't enforce.
  quota = maildir:User quota
  # LDA/LMTP allows saving the last mail to bring user from under quota to
  # over quota, if the quota doesn't grow too high. Default is to allow as
  # long as quota will stay under 10% above the limit. Also allowed e.g. 10M.
  # 0 doesn't allow going over quota (compatibility with Courier-IMAP behavior).
  quota_grace = 0
}


service auth {
  # Allow userdb lookups for popuser.
  unix_listener auth-userdb {
    mode = 0600
    user = popuser
    group = popuser
  }
}


# PEM encoded X.509 SSL/TLS certificate and private key.
ssl_cert = </etc/dovecot/private/ssl-cert-and-key.pem
ssl_key =  </etc/dovecot/private/ssl-cert-and-key.pem


# Protocols support.

protocols = imap pop3

protocol imap {
  mail_plugins = $mail_plugins imap_quota
}

protocol pop3 {
  # POP3 UIDL format similar to the one used by Courier-IMAP.
  pop3_uidl_format = UID%u-%v
}

# Applicable workarounds for various client bugs.
imap_client_workarounds = delay-newmail
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh


# Default private namespace setup compatible with Courier-IMAP.

namespace inbox {
  separator = .
  prefix = INBOX.
  inbox = yes
}


# Logging verbosity and debugging.

# Log unsuccessful authentication attempts and the reasons why they failed.
#auth_verbose = yes

# In case of password mismatches, log the attempted password. Valid values are
# no, plain and sha1. sha1 can be useful for detecting brute force password
# attempts vs. user simply trying the same password over and over again.
#auth_verbose_passwords = no

# Even more verbose logging for debugging purposes.
#auth_debug = yes

# In case of password mismatches, log the passwords and used scheme so the
# problem can be debugged. Enabling this also enables auth_debug.
#auth_debug_passwords = yes

# Enable mail process debugging. This can help you figure out why Dovecot
# isn't finding your mails.
#mail_debug = yes

# Show protocol level SSL errors.
#verbose_ssl = yes


# Log format compatible with Plesk statistics collector.
mail_log_prefix = "service=%s, user=%u, ip=[%r]. "
pop3_logout_format = "rcvd=%i, sent=%o, top=%t/%p, retr=%r/%b, del=%d/%m, size=%s"
imap_logout_format = "rcvd=%i, sent=%o"

# Customizable part of configuration gets included below. The filenames are
# first sorted by their ASCII value and parsed in that order. The 00-prefixes
# in filenames are intended to make it easier to understand the ordering.
#
# Prefixes 10 through 20 and 90 through 95 (inclusive) are reserved for Plesk. 
# Please do not occupy them or edit files with these prefixes!
!include conf.d/*.conf

# vim:ts=2 sts=2 sw=2 et:

etc/postfix/main.cf:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file = /etc/postfix/postfix.pem
smtpd_tls_key_file = $smtpd_tls_cert_file
smtpd_use_tls = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = h1234567.stratoserver.net
alias_maps = hash:/etc/aliases, hash:/var/spool/postfix/plesk/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = localhost.stratoserver.net, localhost, localhost.localdomain
mynetworks = 
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_protocols = all
virtual_mailbox_domains = $virtual_mailbox_maps, hash:/var/spool/postfix/plesk/virtual_domains
virtual_alias_maps = $virtual_maps, hash:/var/spool/postfix/plesk/virtual
virtual_mailbox_maps = , hash:/var/spool/postfix/plesk/vmailbox
transport_maps = , hash:/var/spool/postfix/plesk/transport
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtp_use_tls = no
smtpd_timeout = 3600s
smtpd_proxy_timeout = 3600s
disable_vrfy_command = yes
smtpd_sender_restrictions = check_sender_access hash:/var/spool/postfix/plesk/blacklists, permit_sasl_authenticated
smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated
smtp_send_xforward_command = yes
smtpd_authorized_xforward_hosts = 127.0.0.0/8 [::1]/128
smtpd_sasl_auth_enable = yes
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
virtual_mailbox_base = /var/qmail/mailnames
virtual_uid_maps = static:30
virtual_gid_maps = static:31
smtpd_milters = , inet:127.0.0.1:12768
sender_dependent_default_transport_maps = hash:/var/spool/postfix/plesk/sdd_transport_maps
virtual_transport = plesk_virtual
plesk_virtual_destination_recipient_limit = 1
mailman_destination_recipient_limit = 1
message_size_limit = 1024000000
virtual_mailbox_limit = 0
recipient_canonical_maps = tcp:127.0.0.1:12346
recipient_canonical_classes = envelope_recipient,header_recipient
smtpd_tls_ciphers = medium
smtpd_tls_mandatory_ciphers = medium
tls_medium_cipherlist = HIGH:!aNULL:!MD5
smtpd_tls_mandatory_protocols = TLSv1 TLSv1.1 TLSv1.2
smtpd_tls_protocols = TLSv1 TLSv1.1 TLSv1.2
mynetworks_style = subnet
authorized_flush_users = 
authorized_mailq_users = 
inet_interfaces = all

Does any know how to solve this?

Output

cat /proc/user_beancounters

Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
  2648771:  kmemsize                 78343618            423485440  9223372036854775807  9223372036854775807                    0
            lockedpages                     0                   16              2097152              2097152                    0
            privvmpages               1138097              1401250  9223372036854775807  9223372036854775807                    0
            shmpages                    86247                86261  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       211                  330                  700                  700                    0
            physpages                 1052473              1520136              2097152              2097152                    0
            vmguarpages                     0                    0              2097152              2097152                    0
            oomguarpages               576172               606928              2097152              2097152                    0
            numtcpsock                     67                  214                  750                  750                    0
            numflock                      478                  489  9223372036854775807  9223372036854775807                    0
            numpty                          0                    0  9223372036854775807  9223372036854775807                    0
            numsiginfo                      0                   60  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                 5779952              9023696  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                 1097728              3635784  9223372036854775807  9223372036854775807                    0
            othersockbuf               871624              2229824  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0               296480  9223372036854775807  9223372036854775807                    0
            numothersock                  469                  916                  950                  950                    0
            dcachesize               41858232            380313547  9223372036854775807  9223372036854775807                    0
            numfile                      6049                 8635  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                     363                  365  9223372036854775807  9223372036854775807                    0
Ronny
  • 63
  • 7
  • Have you checked actual size of free memory at problem moment? Do you use 64bit version of Ubuntu? Also check `dmesg` output. – Anton Danilov Jun 20 '19 at 12:33
  • the server is running on average use om 30% Ram. Even after dovecot/postfix have crashed. So i don't think that is the problem Yes i use the 64 bit version - Exact: Linux 4.4.0-042stab138.1 op x86_64 And there are no specific log lines which seems to cause this .... – Ronny Jun 20 '19 at 12:50
  • Check output of `cat /proc/user_beancounters`. Maybe your issue related with limits of virtual server. – Anton Danilov Jun 20 '19 at 13:19
  • Nothing is near the limits. – Ronny Jun 20 '19 at 13:20
  • You should wait for a problem. Other way to troubleshoot is usage of `strace` to investigate syscalls. – Anton Danilov Jun 20 '19 at 14:04
  • i now see this: numothersock 776 950 950 950 so this has gone over the limit. how can i change that? – Ronny Jun 21 '19 at 12:53
  • i now see this: numothersock 776 950 950 950 1334 so this has gone over the limit. how can i change that? – Ronny Jun 21 '19 at 12:59
  • It should be done by the host system administrator, not the administrator of the VPS. Contact with the technical support of your hosting provider. – Anton Danilov Jun 21 '19 at 12:59
  • thx. i will contact them – Ronny Jun 21 '19 at 13:00

2 Answers2

4

This has happened to me before. The reason was that /tmp in ramdisk got filled by random/malevolent uploads. Configure your webserver to use a hard drive temporary upload directory. Also monitor used space in ramdisks, if any. Limit RAM used by php, if applicable.

Problems like these take long to solve. Good luck!

Zdenek
  • 240
  • 1
  • 4
1

My advice is to disable TLSv1 and heavily restrict your cipher list. (both inbound 'smtp' and submission 'smtpd')

If you absolutely require TLSv1 you should still heavily restrict your cipher list, if not you might be on the receiving end of automated hack attempts targetted at servers that support deprecated potentially vulnerable ciphers etc, and this could have a sizeable impact.

Why do this? It's quick to implement and in some scenarios could fix the issue, other advice above still good too.

p.s. In my opinion allowing 1 GB emails could also impact on memory utilisation, this value may be worth adjusting downwards:

message_size_limit=1024000000

Also (if running OpenVZ):

Further reading says change numothersock to "9223372036854775807" the same as the other counters.

Source:

https://www.cloudmin.com/node/35052

I don't know if numtcpsock would require the same change.

Allan Wallace
  • 271
  • 1
  • 4