ORA-27301: OS failure message: No space left on device

Got this just now :) on one of  11 node cluster instance,

SQL> startup
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
SQL> exit

 We have 50+ database running on this cluster .after searching detail  on this error found one usefully oracle doc , make value as per  document .

This is caused by Kernel parameter setting. Since we were running many databases on a server, we should check our semaphore.


In our case it's

$ /sbin/sysctl -a | grep sem
error: permission denied on key 'kernel.cad_pid'
kernel.sem = 250 524288 256 2048

reset value as per above document.

run sysctl and check.
[root@test ~]# sysctl kernel.sem
kernel.sem = 275        524288  256     2048
[root@test ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 32988817
kernel.shmmax = 67561097216
kernel.shmmni = 4096
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.ip_local_port_range = 1024 65500
kernel.msgmax = 8192
kernel.msgmni = 2878
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
kernel.shmmni = 4096
kernel.msgmnb = 65536
kernel.sysrq = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.ipv4.tcp_syncookies = 1
fs.aio-max-nr = 3145728
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.wmem_max = 1048576
kernel.sem = 275 524288 256 2048

Tried to start db again,
SQL> startup
ORACLE instance started.

Total System Global Area 1670221824 bytes
Fixed Size                  2227072 bytes
Variable Size            1241515136 bytes
Database Buffers          402653184 bytes
Redo Buffers               23826432 bytes
Database mounted.
Database opened.

wow all set :)