I was working with 10.6, non-server edition. People have generally had more success with the Xserve OS X server versions. Many were quite old, and it appears that the process has gotten more difficult with more recent versions of OS X (at least Snow Leopard 10.5+). I first tried googling around, and I encountered a few references to this issue on various Mac forums. One of two things is generally meant by this, either using a Mac as the interface to a serial device (accomplished by running a terminal emulator program on the Mac), or using another machine to connect to the Mac over serial and accessing the shell provided by the Mac.
Setting up a Serial Console in Mac OS X Setting up a Serial Console in Mac OS X