In order to understand how Ostiary works, you have to understand what a 'hash' is in computer-science terms. Essentially a hash is a process you apply to data to transform it into other data, with the following properties:
A hash can be used to generate a 'fingerprint' of a set of data. Imagine you want to prove to me that you know a piece of information, like a password. You could just tell me the password, but there are eavesdroppers around; you don't want them to hear it. You can generate a hash of the password, and tell me that. I generate a hash from my password, and compare them. If they match, I can be pretty certain you know the password, but an eavesdropper can't reverse the hash they overheard and recover the password.
(You may have spotted a problem with this system as described; an attacker could just give you the hash they overheard, without knowing the password. Don't worry, I'm just laying out some needed concepts at this point. I will hopefully address this problem to your satisfaction shortly.)
The SHA ("Secure Hash Algorithm") family is currently one of the most widely used type of hashes. They have proven to fulfill all of the properties above, and no known practical problems have surfaced - at least not yet. SHA256 takes arbitrary data and produces a 256-bit (32 byte) hash value.
A hash like SHA256 can be used in a simple way to confirm that a message wasn't accidentally garbled during transit. A sender can send the message, and append the SHA256 of the message. The receiver can get the message, compute the SHA256, and compare it to what was sent. If they match, the receiver can be pretty sure the message was transmitted correctly. Random changes are extremely unlikely to lead to a valid message+SHA256 combo. Used this way, the SHA-256 is nothing more than a glorified CRC.
But what if someone is deliberately trying to tamper with the message? They can alter the message, then compute a new hash and append that. The reciever has no way to detect this case with the simple process above.
HMAC is an algorithm to generate a Message Authentication Code to help prove that a message actually came from the person it claims to be from. The idea is that the HMAC isn't just a hash of the message, but the message and a secret key known to both the sender and reciever. So long as an attacker doesn't know that key, it's vanishingly unlikely that they will be able to generate a valid hash for a given message.
Fundamentally, Ostiary (for versions past 1.0) uses the basic ideas from Challenge Handshake Authentication Protocol (PPP's CHAP, RFC 1994), crossed with HMAC (Hashed Message Authentication Code, RFC 2104).
In CHAP, a server sends some data (a "challenge"), and expects the recipient to hash that challenge with a secret key and return that value (a "handshake"). If they match, access is granted. The type of hash is unspecified in CHAP; previous versions of Ostiary have used HMAC-MD5. Recent results have cast some doubt on MD5; its use in HMAC is still probably secure, but going forward I've chosen to use HMAC-SHA256. Essentially, the "message" that's authenticated by HMAC is the "challenge" sent by CHAP.
The actual algorithm used is as follows:
Note a few things about this. Ostiaryd sends and reads fixed-size messages, and then closes the connection. So long as things are coded correctly, a buffer overflow is basically impossible. And the password is hashed with a hash partially dependent on the current time - a command hash that works now will (almost certainly) not work even a second later. And the server won't accept a new connection until well after that window's past.
Even if an attacker records all the traffic between the client and the ostiaryd server, it won't do them much good. They can try dictionary attacks offline to try to guess the password. If they are in the middle and controlling communications, they can prevent the client's return message from arriving, or scramble it and lock out a valid user.
There's one disadvantage to a protocol this simple, however: there's no feedback on whether a command actually worked or not. If you want to know if, say, you've enabled ssh, you'll have to actually try to log in, or set up the 'enable' script to tell you via some other channel. On the bright side, you can tell if you're locked out; if you can connect, but don't receive a hash, then ostiaryd isn't talking to you.