While rpms and debs are available, for platforms besides i386 Linux you'll want to compile your own. It should be a simple matter of "./configure; make all". It's known to compile and work on Linux, NetBSD, Solaris, AIX, Compaq Tru64, HP-UX, IRIX, and Cygwin. Note that if you want libwrap (TCP wrappers) support, you'll need to run the configure script with the "--with-libwrap" option.
(Just as an aside, it takes about 0.6 seconds to compile Ostiary on my dual-Athlon Linux box, and over seven minutes to compile it on my Mac SE/30 NetBSD box. We've come a long way, baby.)
It's not that hard to set up ostiary. You need to set up the config file (an 'ostiary.cfg' file is installed with the rpm or "make install" that should give you a good start). You must assign a 'kill' password, and pick which port it will listen on (there's no default). Check the man pages - ostclient(1), ostiary.cfg(5), and ostiaryd(8) - for details.
If you want to set up Ostiary to start on boot, you'll need to do some research on your distribution and how it does that. There's a sample Red Hat startup script here (Thanks, Allan Peda!) that should get you most of the way there.
By default, ostiary will look for its config file in "/etc/ostiary.cfg", and a few other places. This can be overriden on the command line. It logs using syslog() with the LOG_AUTH facility code. (Typically you'll see its output in /var/log/authlog or /var/log/messages. You'll have to check how your syslog daemon is set up.) If you compiled with tcp wrappers, you'll probably need to update /etc/hosts.allow and /etc/hosts.deny.
One note on the config file - it's processed in order. If you want to debug the parsing, set the MIN_LOGLEVEL parameter at the top of the file. Also, if you're going to specify DEFAULT_UID and/or DEFAULT_GID, do so before the ACTION lines.
Next, you need to set up one or more command scripts to run, and choose passphrases for them. Currently ostiary limits itself to eight commands maximum (in addition to the 'kill' command). If you need more, you'll need to edit ost.h and recompile. (See here for a discussion of how Ostiary scales.)
When the scripts are run, they will get one argument - the IP address that sent the command to run them, in dotted-quad form. If ostiaryd is run as root, you can specify what uid/gid scripts will run as, either by default or on a script-by-script basis. If you don't specify, they'll run as a 'default' user - the "default default" is 'nobody'.
Since the scripts by default run as 'nobody', they probably won't have permissions to do anything interesting. You'll need to think about what permissions the scripts actually need. This is my way of forcing you to consider that. If you want a script to run as root, you must set it up to do so explicitly.
If ostiaryd isn't running as root, it will instead run them as the same uid/gid as ostiaryd itself, and attempting to set otherwise in the config file (either by default or in an ACTION line) will cause an error.
Passwords can be up to 63 characters, commands up to 127 characters. (You can edit ost.h and recompile if you need more; don't forget to update the clients, too, if you muck with the password length!) Note that the name of a command should include the full path. Passwords can contain just about any character except CR and LF. You can escape quotes by preceeding them with backslash, like so: '\"'. Be advised that you may have more trouble entering certain characters on different clients.
It turns out that sshd takes a while to get going. It needs to generate new 'ephemeral' keys, which can take a while - at least a couple minutes on my 68030 box. So killing and restarting sshd is not really practical in my case.
Fortunately, there's an alternative; sshd can be set up to use TCP Wrappers, which allow fine-grained control over who gets to access the service and who doesn't. So, one can set up a script that manipulates the /etc/hosts.allow file, adding and/or removing IPs in response to commands. There are two sample scripts in the 'samples' directory - one enables ssh for a specific IP the other locks things down and kills all current ssh sessions.
Of course, if you've got a faster machine, you might be able to afford killing and restarting sshd all the time.
Note that if you know that queries are going to be coming from a particular net range (e.g., my Palm connects to Sprint's network, so I know such queries will be coming from a particular netblock) you can have the script make sure that a command came from the correct neighborhood. You can also use tcp wrappers for this, if you enable wrappers support when compiling ostiaryd.
Sometimes you may need to source the full environment of a user to get a particular app to start or whatever. See the known issues section of the FAQs.