Discovering 'select'

Over the last couple weeks, Elad Bogomolny and I have worked on building an ssh-like program that allows a client to access a remote server and run bash commands on it. We weren’t totally sure where to take this–but it ended being a pretty awesome project for getting better at systems programming.

One of the unintended consequences of doing this project was learning about the select syscall in Linux. In this post, I’m going to cover the basics of why one might need to use it, and how.

Brief Summary

The ssh-like program we wrote has both a client and server component. The server simply waits for incoming connections, and when a client connects, the server forks. The forked child process becomes a bash processs, while the parent process enters a loop where it waits for bash commands to be sent over the network, and when bash commands come in, passes those to the child bash process. It then waits for the bash process to send back data, and when bash returns data, sends that data back over the network to the client.

Here’s some psuedo-code for the server process:

// server.c

accept() // accept connection from client

int result = fork()
if (result == 0) {
  // child process
  exec("bash");
} else {
  // parent process

  while(1) {
    data = read_from_network(); // blocks on the client
    write_data_to_bash_process(data);
    bash_result = read_from_child_process(); // blocks on bash process
    write_data_to_network(bash_result);
  }

}

And the client:

// client.c

connect_to_server();

while(1) {
  data = read_from_stdin(); // blocks on user input
  write_data_to_network(data);
  bash_result = read_from_network(); // blocks on server response
  print(bash_result);
}

Obviously, theres a lot more actually going on here, like encrypting the data, but this is a roughly how the program works.

A quick word on file descriptors

Before I go further, I want to briefly discuss the concept of “file descriptors”. In Unix-based operating systems, file descriptors are an abstraction that allow developers to interact with I/O in a consistent way, whether that I/O is reading/writing to a file, or reading/writing bytes over the network.

File descriptors are created through a number of system calls (“syscalls”). For instance, opening a file and opening up network connections both create file descriptors. Once you have access to a file descriptor, there are a number of syscalls that can operate on it, regardless of what the file descriptor is a reference to. For instance, the read and write syscalls are used, as you might guess, to read or write data to a particular file descriptor. As a developer, you don’t have to do anything differently based on what you are reading from or writing to–the OS takes care of it.

It’s also worth noting that file descriptors in Unix-based operating systems are each given a unique integer id. So, when you open a file, or make a network connection in C, the syscall for doing those operations will return a number to you that corresponds to the newly opened file descriptor.

Back to SSH!

Alright, so now that we know what file descriptors are, let’s go through an issue in the pseudocode I mention above, notably in this part:

data = read_from_network(); // blocks on the client
write_data_to_bash_process(data);
bash_result = read_from_child_process(); // blocks on bash process
write_data_to_network(bash_result);

What happens if we run a bash command that doesn’t return any output, like cd? The problem with this code is that if this is the case, we’ll block on the read_from_child_process() indefinitely, and no more commands will be able to be run.

Possible solutions?

We could put some kind of timeout on the read_from_child_process function call, and move on if the timeout is reached. This solution is problematic–either we set too short a timeout and this breaks bash commands that take a long time, or we set too long a timeout and commands that finish quickly, like cd, take way longer than they need to. It’s hard to distinguish between cases where the program is taking too long to run or whether it returns no output.

Another bad option would be to after every bash command, run a command that does return output, like echo. We could then chop off the result of echo when we return data to the client. This would work better than the previous solution, since every command will take the right amount of time. However, this breaks as soon as we want to do things like get the history of the shell, or get the return code of the previous command via echo $?.

What are some better options?

An option that doesn’t have the hacky compromises of the previous solutions is to instead of managing all of this in a single while loop, fork the process into two processes, one responsible for simply reading data out of the network socket and writing it to the bash process, and other responsible for reading bash responses and writing those back to the network.

To put this in pseudo-code:

// server.c
... // code to accept data from client

result = fork()
if (result == 0) {
  // child
  while (1) {
    data = read_from_network(); // blocks on the client
    write_data_to_bash_process(data);
  }

} else {
  // parent
  while (1) {
    bash_result = read_from_child_process(); // blocks on bash process
    write_data_to_network(bash_result);
  }
}

This has the drawback of not allowing us to block the client on responses from bash–the client will be able to continue sending data into the bash process before getting a response. This seems like a fine compromise–the actual ssh doesn’t even do this.

Enter: Select

While this is a pretty good solution, it does have the added cost of requiring another process or thread to be running on the server. Can we do better?

It turns out that Linux has a syscall to solve this very problem, select. select allows you to “watch” multiple file descriptors, and returns if any of them is ready to be read from. In our case, we have two fairly independent processes that each block on a file descriptor, for the bash reader, it blocks on bash, and for the network reader, it blocks on a network socket. Each process, when data is available on the file descriptor it is reading from, then executes some logic.

With select, we could instead of using multiple processes, in a single process pass both the file descriptor to the bash process and the file descriptor to the network socket to the call to select. When data is available on either of those file descriptors, select will return. We can then check which descriptor has data on it, and handle it appropriately. If we put this in a while loop, we no longer need to have multiple processes for handling this.

Basic Usage

The basic setup for using select is as follows:

/* select_example.c */

/*  These macros are used for managing a set of file descriptors */

fd_set read_file_descriptors; // object for managing sets of file descriptors

while (1) {
  FD_ZERO(&rfds);
  FD_SET(bash_fd, &read_file_descriptors);
  FD_SET(network_fd, &read_file_descriptors);
  select(highestFileDescriptor, &read_file_descriptors, NULL, NULL, NULL);
  if(FD_ISSET(bash_fd, &read_file_descriptors)) {
    ... // handle case when data is available to read from bash process
  }
  if(FD_ISSET(network_fd, &read_file_descriptors)) {
    ... // handle case when data is available to read from network
  }
}

Linux provides a number of macros that are helpful for setting up the call to select. The short of it is that you call FD_SET to add file descriptors that you care about to the list of read_file_descriptors, and then when you call select, you supply the file descriptor that is the highest number and the set of file descriptors you care to read from.

After select returns, select will have modified the state of the file descriptor in the set, and you can check it to see which ones are available for reading.

It’s worth also noting here that select can also be passed an fd_set of write file descriptors. If you pass these, select will return when those file descriptors are available for writing, rather than reading.

Conclusion

select is a very powerful syscall–the notion of being able to wait on multiple channels before executing some logic is useful in a bunch of other contexts besides the simple ssh client/server implementation I’m using here.

For instance, if you have a chat server, you can use select to manage multiple connections on a single process. Each connection will be represented by a different file descriptors, and through use of select, you can handle messages from each client when data is available to read from each of them, rather than having to have multiple threads that each block on a client.

If you’re interested in systems programming and want to learn more about select and other related syscalls, I’d recommend checking out this blog post, and reading the man page.

Thanks for reading!