Zabbix Server宕机报“__zbx_mem_malloc(): out of memory (requested 96 bytes)”

作者: 潇湘隐者  更新时间:2020-07-13 09:09:00  原文链接


早上登录Zabbix的时候,发现其提示 Zabbix server is not running: the information displayed may not be current ,检查Zabbix Server的状态为activating (auto-restart)而不是activating (running)态:,如下所示

# systemctl status zabbix-server

● zabbix-server.service - Zabbix Server

   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)

   Active: activating ( auto-restart ) (Result: exit-code) since Mon 2020-07-13 08:23:21 CST; 4s ago

  Process: 542511 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)

  Process: 542504 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)

  Main PID: 542506 (code=exited, status=0/SUCCESS)

Jul 13 08:23:21 xxxx systemd[1]: zabbix-server.service: Control process exited, code=exited status=1

Jul 13 08:23:21 xxxx systemd[1]: zabbix-server.service: Failed with result 'exit-code

检查Zabbix-Server的错误日志,发现其报如下错误:

542396:20200713:082055.295 __mem_malloc: skipped 0 asked 96 skip_min 18446744073709551615 skip_max 0
542396:20200713:082055.295 [file:dbconfig.c,line:95] __zbx_mem_malloc(): out of memory (requested 96 bytes)
542396:20200713:082055.295 [file:dbconfig.c,line:95] __zbx_mem_malloc(): please increase CacheSize configuration parameter

542395:20200713:082054.179 ****** Enabled features ******
542395:20200713:082054.179 SNMP monitoring:           YES
542395:20200713:082054.179 IPMI monitoring:           YES
542395:20200713:082054.179 Web monitoring:            YES
542395:20200713:082054.179 VMware monitoring:         YES
542395:20200713:082054.180 SMTP authentication:       YES
542395:20200713:082054.180 ODBC:                      YES
542395:20200713:082054.180 SSH support:               YES
542395:20200713:082054.180 IPv6 support:              YES
542395:20200713:082054.180 TLS support:               YES
542395:20200713:082054.180 ******************************
542395:20200713:082054.180 using configuration file: /etc/zabbix/zabbix_server.conf
542395:20200713:082054.190 current database version (mandatory/optional): 05000000/05000000
542395:20200713:082054.191 required mandatory version: 05000000
542395:20200713:082054.214 server #0 started [main process]
542396:20200713:082054.215 server #1 started [configuration syncer #1]
542396:20200713:082055.295 __mem_malloc: skipped 0 asked 96 skip_min 18446744073709551615 skip_max 0
542396:20200713:082055.295 [file:dbconfig.c,line:95] __zbx_mem_malloc(): out of memory (requested 96 bytes)
542396:20200713:082055.295 [file:dbconfig.c,line:95] __zbx_mem_malloc(): please increase CacheSize configuration parameter
542396:20200713:082055.295 === memory statistics for configuration cache ===
542396:20200713:082055.295 free chunks of size     80 bytes:        1
542396:20200713:082055.295 min chunk size:         80 bytes
542396:20200713:082055.295 max chunk size:         80 bytes
542396:20200713:082055.295 memory of total size 8388232 bytes fragmented into 74114 chunks
542396:20200713:082055.295 of those,         80 bytes are in        1 free chunks
542396:20200713:082055.295 of those,    7202344 bytes are in    74113 used chunks
542396:20200713:082055.295 ================================
542396:20200713:082055.295 === Backtrace: ===
542396:20200713:082055.297 16: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x3f) [0x56170ffb03c8]
542396:20200713:082055.297 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_malloc+0x180) [0x56170ffab414]
542396:20200713:082055.297 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1865aa) [0x56170ff735aa]
542396:20200713:082055.297 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1c831a) [0x56170ffb531a]
542396:20200713:082055.297 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashmap_set+0xf3) [0x56170ffb5791]
542396:20200713:082055.297 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1c510a) [0x56170ffb210a]
542396:20200713:082055.297 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1c52c1) [0x56170ffb22c1]
542396:20200713:082055.297 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_binary_heap_insert+0xbf) [0x56170ffb27b5]
542396:20200713:082055.297 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x191be0) [0x56170ff7ebe0]
542396:20200713:082055.297 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x1157) [0x56170ff7fddf]
542396:20200713:082055.297 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x116) [0x56170fe40ac2]
542396:20200713:082055.297 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x56170ffbf0cd]
542396:20200713:082055.297 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x9cb) [0x56170fe334c8]
542396:20200713:082055.297 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x2ff) [0x56170ffaffd4]
542396:20200713:082055.297 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x2f5) [0x56170fe32afb]
542396:20200713:082055.297 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f595fa966a3]
542396:20200713:082055.297 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x56170fe31bae]
542395:20200713:082055.301 One child process died (PID:542396,exitcode/signal:1). Exiting ...
542395:20200713:082055.304 syncing trend data...
542395:20200713:082055.304 syncing trend data done
542395:20200713:082055.304 Zabbix Server stopped. Zabbix 5.0.1 (revision c2a0b03480).

日志提示Zabbix内存溢出了, out of memory (requested 96 bytes)和please increase CacheSize configuration parameter。 需要增加CacheSize的大小,找到/etc/zabbix/zabbix_server.conf参数文件,检查发现CacheSize参数没有设置,使用默认的值,将其调整为2048M大小(新安装的Zabbix服务器,忘记调整一些参数设置了!),重启Zabbix Server服务,问题解决。

### Option: CacheSize
#       Size of configuration cache, in bytes.
#       Shared memory size for storing host, item and trigger data.
#
# Mandatory: no
# Range: 128K-64G
# Default:
# CacheSize=8M
CacheSize=2048M