How Ostiary Works

Table of Contents

  1. Introduction
  2. How Ostiary Works
    1. Hash Functions
    2. HMAC
    3. The Basic Algorithm
  3. Complications
  4. Getting Ostiary
  5. Installing Ostiary
  6. Alternatives to Ostiary
  7. FAQs

Hash Functions

In order to understand how Ostiary works, you have to understand what a 'hash' is in computer-science terms. Essentially a hash is a process you apply to data to transform it into other data, with the following properties:

It's easy to make sausage out of a pig; it's impossible (today, anyway) to reconstruct a pig from the sausage it was turned into. A hash performs a similar process on data. Any data - text files, image files... and passwords.

A hash can be used to generate a 'fingerprint' of a set of data. Imagine you want to prove to me that you know a piece of information, like a password. You could just tell me the password, but there are eavesdroppers around; you don't want them to hear it. You can generate a hash of the password, and tell me that. I generate a hash from my password, and compare them. If they match, I can be pretty certain you know the password, but an eavesdropper can't reverse the hash they overheard and recover the password.

(You may have spotted a problem with this system as described; an attacker could just give you the hash they overheard, without knowing the password. Don't worry, I'm just laying out some needed concepts at this point. I will hopefully address this problem to your satisfaction shortly.)

The SHA ("Secure Hash Algorithm") family is currently one of the most widely used type of hashes. They have proven to fulfill all of the properties above, and no known practical problems have surfaced - at least not yet. SHA256 takes arbitrary data and produces a 256-bit (32 byte) hash value.


A hash like SHA256 can be used in a simple way to confirm that a message wasn't accidentally garbled during transit. A sender can send the message, and append the SHA256 of the message. The receiver can get the message, compute the SHA256, and compare it to what was sent. If they match, the receiver can be pretty sure the message was transmitted correctly. Random changes are extremely unlikely to lead to a valid message+SHA256 combo. Used this way, the SHA-256 is nothing more than a glorified CRC.

But what if someone is deliberately trying to tamper with the message? They can alter the message, then compute a new hash and append that. The reciever has no way to detect this case with the simple process above.

HMAC is an algorithm to generate a Message Authentication Code to help prove that a message actually came from the person it claims to be from. The idea is that the HMAC isn't just a hash of the message, but the message and a secret key known to both the sender and reciever. So long as an attacker doesn't know that key, it's vanishingly unlikely that they will be able to generate a valid hash for a given message.

The Basic Algorithm

Fundamentally, Ostiary (for versions past 1.0) uses the basic ideas from Challenge Handshake Authentication Protocol (PPP's CHAP, RFC 1994), crossed with HMAC (Hashed Message Authentication Code, RFC 2104).

In CHAP, a server sends some data (a "challenge"), and expects the recipient to hash that challenge with a secret key and return that value (a "handshake"). If they match, access is granted. The type of hash is unspecified in CHAP; previous versions of Ostiary have used HMAC-MD5. Recent results have cast some doubt on MD5; its use in HMAC is still probably secure, but going forward I've chosen to use HMAC-SHA256. Essentially, the "message" that's authenticated by HMAC is the "challenge" sent by CHAP.

The actual algorithm used is as follows:

  1. Ostiaryd waits on a port for connections from remote machines. When one is received, ostiaryd checks to see if the ip address is locked out. (If compiled in, it will consult /etc/hosts.allow and hosts.deny, too.) If so, it drops the connection immediately.
  2. If the address is not locked out, ostiaryd sends a seed value. Currently this is a SHA256 hash (32 bytes) of the the current time, plus either the output of random() or (if available) bytes from /dev/urandom, plus the process' PID.
  3. The client takes the seed value and hashes it, HMAC-style, with the password the user provides.
  4. Ostiaryd then reads the response (32 bytes), and closes the connection.
  5. Now ostiaryd goes through a list of passwords, and hashes them (HMAC style) with the hash value it sent. It compares these new hashes with the response it received.
  6. If it finds a match, the command corresponding to that password is run. (E.g., it could start up sshd so you could log in remotely.) The ip address of the client is given as an argument to the command.
  7. If the hash does not match any of the listed hashes, the ip address that the bogus hash was sent from gets put on a list of bad addresses. If it exceeds a defined limit of bad connections, that address is locked out and no further communication is accepted.
  8. Now, ostiaryd sleeps a user-defined interval, at least one second.
  9. Finally, it jumps back to step 1.

Note a few things about this. Ostiaryd sends and reads fixed-size messages, and then closes the connection. So long as things are coded correctly, a buffer overflow is basically impossible. And the password is hashed with a hash partially dependent on the current time - a command hash that works now will (almost certainly) not work even a second later. And the server won't accept a new connection until well after that window's past.

Even if an attacker records all the traffic between the client and the ostiaryd server, it won't do them much good. They can try dictionary attacks offline to try to guess the password. If they are in the middle and controlling communications, they can prevent the client's return message from arriving, or scramble it and lock out a valid user.

There's one disadvantage to a protocol this simple, however: there's no feedback on whether a command actually worked or not. If you want to know if, say, you've enabled ssh, you'll have to actually try to log in, or set up the 'enable' script to tell you via some other channel. On the bright side, you can tell if you're locked out; if you can connect, but don't receive a hash, then ostiaryd isn't talking to you.