How To Fix Apache Error “(28)No space left on device” Couldn’t create the mpm-accept mutex

On some occasions apache server will just fail, and become crashed, failing to restart with an error message like

# tail /var/log/httpd/error_log

[Sun Dec 22 10:05:27.006998 2013] [core:emerg] [pid 15001:tid 140145945569216] (28)No space left on device: AH00023: Couldn't create the fcgid-proctbl mutex
[Sun Dec 22 10:05:38.000403 2013] [core:emerg] [pid 15018:tid 140368783902656] (28)No space left on device: AH00023: Couldn't create the proxy mutex
[Sun Dec 22 10:05:38.000456 2013] [proxy:crit] [pid 15018:tid 140368783902656] (28)No space left on device: AH02478: failed to create proxy mutex

If you see an error similar to the bellow, it could indicate that your server has run out of semaphores. To see how many semaphores are being used, type the following command

# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 38797312   apache     600        1
0x00000000 38731777   apache     600        1
0x00000000 38830082   apache     600        1
0x00000000 38961155   apache     600        1
0x00000000 38895620   apache     600        1
0x00000000 38993925   apache     600        1
0x00000000 39026694   apache     600        1
0x00000000 39059463   apache     600        1
0x00000000 39092232   apache     600        1
0x00000000 39125001   apache     600        1
0x00000000 39157770   apache     600        1
0x00000000 39288843   apache     600        1
0x00000000 39223308   apache     600        1
0x00000000 39321613   apache     600        1
0x00000000 39354382   apache     600        1
0x00000000 39387151   apache     600        1
0x00000000 39419920   apache     600        1
0x00000000 39452689   apache     600        1
0x00000000 39485458   apache     600        1

To fix it and get Apache server started again, we must clean the semaphores. Run the following command to flush them:

for whatever in `ipcs -s | awk '{print $2}'`; do ipcrm -s $whatever; done

If that command may not work on older server then you may need to do the following

# /etc/init.d/httpd stop
# ipcs -s | grep nobody | gawk '{ print $2 }' | xargs -n 1 ipcrm sem
# /etc/init.d/httpd start

Final, you may want to increase the semaphore limits on your server. You can do that by adding the following to the /etc/sysctl.conf

kernel.msgmni = 512
kernel.sem = 250 128000 32 512

After, type the following command to update new setting for kernel

# sysctl -p