First off, editing sysctl.conf doesn't change the value until you either reboot, or execute:
sysctl -p
To have it reload the values.
You mention this is a 32-bit Linux. That puts constraints on how large you can set SHMMAX to and how big the Oracle SGA can be. See Installing Oracle9i on FC2 for more information about the limits you'll run into here. The largest generally useful setting is this:
kernel.shmmax=2147483648
And since the one you tried is >4GB that's why it failed altogether.
Many people seem to use some guide or Oracle's suggestions for a setting here as a magic number without actually considering whether the shared memory values really make sense for their system or not. I wrote the following little script to generate the settings for me on Linux. As written, it limits the shared memory block to 50% of total RAM, which might be light for your Oracle use; easy to adjust it to a higher percentage. I hate seeing people set this value to higher than the amount of RAM in their server.
#!/bin/bash
mem_bytes=`awk '/MemTotal:/ { printf "%0.f",$2 * 1024}' /proc/meminfo`
mem_max=`expr $mem_bytes / 2`
page_size=`getconf PAGE_SIZE`
shmall=`expr $mem_bytes / $page_size`
echo \# Maximum shared segment size in bytes
echo kernel.shmmax = $mem_max
echo \# Maximum number of shared memory segments in pages
echo kernel.shmall = $shmall
The output from this can get written right to the end of the sysctl.conf, run "sysctl -p", and you're off with a reasonable yet safe setting.
