Today, one of the client database was down. Trying to start with
$ srvctl start instance -d TEST -i TEST3
PRCR-1013 : Failed to start resource ora.test.db
PRCR-1064 : Failed to start resource ora.test.db on node test-rac03
CRS-5017: The resource action "ora.test.db start" encountered the following
error:ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
CRS-2674: Start of 'ora.test.db' on 'test-rac03' failed
Tried through SQL plus,
SQL> startupORA-27154: post/wait create failedORA-27300: OS system dependent operation:semget failed with status: 28ORA-27301: OS failure message: No space left on deviceORA-27302: failure occurred at: sskgpcreates
checked in trace fileStarting ORACLE instance (normal)Errors in file /u01/app/oracle/TEST/diag/rdbms/test/TEST3/trace/TEST3_ora_10781.trc:ORA-27154: post/wait create failedORA-27300: OS system dependent operation:semget failed with status: 28ORA-27301: OS failure message: No space left on deviceORA-27302: failure occurred at: sskgpcreatesWed Nov 09 15:57:53 2011
From the trace filedbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)----- Error Stack Dump -----
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
$sudo su -
# /sbin/sysctl -a | grep semkernel.sem = 250 32000 100 142
Increased the semophores value to 180
This is a linux host, so I changed the value and re-initialize immediately.
cd /etc
vi sysctl.conf
look for kernel.sem
change the last value from 140 to 180
[root@npe-ora-rac03 etc]# /sbin/sysctl -p
net.ipv4.ip_forward = 0net.ipv4.conf.default.rp_filter = 1net.ipv4.conf.default.accept_source_route = 0kernel.sysrq = 1kernel.core_uses_pid = 1net.ipv4.tcp_syncookies = 1kernel.msgmnb = 65536kernel.msgmax = 8192kernel.shmmax = 4398046511104kernel.shmall = 1073741824fs.file-max = 6815744kernel.msgmni = 2878kernel.sem = 250 32000 100 180kernel.shmmni = 4096net.core.rmem_default = 1048576net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048576fs.aio-max-nr = 3145728net.ipv4.ip_local_port_range = 9000 65500vm.min_free_kbytes = 51200
srvctl start instance -d TEST -i TEST3
**Issue resolved**
