1、现象描述：昨晚新上线一个服务，有人做了一个del操作，接着mysql主从全部挂机
如下：
主
Apr  6 22:56:58 wmappdb1 kernel: Out of memory: Kill process 32627 (mysqld) score 984 or sacrifice child
Apr  6 22:56:58 wmappdb1 kernel: Killed process 32627, UID 27, (mysqld) total-vm:54979084kB, anon-rss:31860436kB, file-rss:2044kB
Apr  6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: TCP connection to [10.249.1.37]:3306 failed !!!
Apr  6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Removing service [10.249.1.37]:3306 from VS [10.249.1.116]:3306
Apr  6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Executing [/usr/bin/killkeepalived.sh] for service [10.249.1.37]:3306 in


从：
Apr  6 22:56:58 wmappdb1 kernel: Out of memory: Kill process 32627 (mysqld) score 984 or sacrifice child
Apr  6 22:56:58 wmappdb1 kernel: Killed process 32627, UID 27, (mysqld) total-vm:54979084kB, anon-rss:31860436kB, file-rss:2044kB
Apr  6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: TCP connection to [10.249.1.37]:3306 failed !!!
Apr  6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Removing service [10.249.1.37]:3306 from VS [10.249.1.116]:3306
Apr  6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Executing [/usr/bin/killkeepalived.sh] for service [10.249.1.37]:3306 in 


2、问题处理过程
查看完以上log，发现linux系统发生了oom killer
原因分析
a、zabbix监控发现，当时load较高。
b、发现log中，swap等被用完。
Apr  6 22:39:17 wmappdb2 kernel: 12152 total pagecache pages
Apr  6 22:39:17 wmappdb2 kernel: 7320 pages in swap cache
Apr  6 22:39:17 wmappdb2 kernel: Swap cache stats: add 13734845, delete 13727525, find 192747597/194527644
Apr  6 22:39:17 wmappdb2 kernel: Free swap  = 0kB
Apr  6 22:39:17 wmappdb2 kernel: Total swap = 16416760kB
Apr  6 22:39:17 wmappdb2 kernel: 8388607 pages RAM
Apr  6 22:39:17 wmappdb2 kernel: 217229 pages reserved
Apr  6 22:39:17 wmappdb2 kernel: 7591 pages shared


c、分析mysql 中临时表的使用，发现有部分逻辑使用了临时表，且type=memory，需进一步优化。


4.解决办法
a、规划各程序的内存分配，控制各程序的总内存使用上限，避免内存耗尽。
b、消除程序中数据集较大的临时表(memory)，采用myisam表替代。
c、调整mysql的oom_score值（保留措施）