System V IPC

Many variants of Unix these days support a set of inter-process communications methods, which are derived from Unix System V release 4, originating from AT&T Bell laboratories.

The fact that these resources are global to the system has two contradicting implications. On one hand, it means that if a process exits, the data it sent through a message queue, or placed in shared memory is still there, and can be collected by other processes. On the other hand, this also means that the programmer has to take care of freeing these resources, or they occupy system resources until the next reboot, or until being removed by hand.

I am going to make a statement here about these communications mechanisms, that might annoy some readers: System V IPC mechanisms are evil regarding their implementation, and should not be used unless there is a very good reason. One of the problem with these mechanism, is that one cannot use the select() (or its replacement, poll()) with them, and thus a process waiting for a message to be placed in a message queue, cannot be notified about messages coming via other resources (e.g. other message queues, pipes or sockets). In my opinion, this limitation is an oversight by the designers of these mechanisms. Had they used file descriptors to denote IPC resources (like they are used for pipes, sockets and files) life would be easier.

Another problem with System V IPC is their system-global nature. The total number of message queues that may live in the system, for example, is shared by all processes. Worse than that, the number of messages waiting in all messages queues is also limited globally. One process spewing many such messages will break all processes using message queues. The same goes for other such resources. There are various other limitations imposed by API (Application programming interface). For example, one may wait on a limited set of semaphores at the same time. If you want more than this, you have to split the waiting task, or re-design your application.

Having said that, there are still various applications where using system V IPC (we'll call it SysV IPC, for short) will save you a large amount of time. In these cases, you should go ahead and use these mechanism - just handle with care.

Before delving into the usage of the different System V IPC mechanisms, we will describe the security model used to limit access to these resources.

Each resource in SysV IPC may be either private or public. Private means that it may be accessed only by the process that created it, or by child processes of this process. Public means that it may be potentially accessed by any process in the system, except when access permission modes state otherwise.

SysV IPC resources may be protected using access mode permissions, much like files and directories are protected by the Unix system. Each such resource has an owning user and an owning group. Permission modes define if and how processes belonging to different users in the system may access this resource. Permissions may be set separately for the owning user, for users from the owning group, and everyone else. permissions may be set for reading the resource (e.g. reading messages from a message queue), or writing to the resource (e.g. sending a message on a queue, changing the value of a semaphore). A structure of type 'ipc_perm', which is defined as follows:

struct ipc_perm
{
  key_t  key;   /* key identifying the resource                     */
  ushort uid;   /* owner effective user ID and effective group ID   */
  ushort gid;
  ushort cuid;  /* creator effective user ID and effective group ID */
  ushort cgid;
  ushort mode;  /* access modes                                     */
  ushort seq;   /* sequence number                                  */
};

• key - the identifier of the resource this structure refers to.
• uid - effective user ID owning the resource.
• gid - effective group ID owning the resource.
• cuid - effective user ID that created the resource.
• cgid - effective group ID that created the resource.
• mode - access permission modes for the given resource. This is a bit field, with the lowest 9 bits denoting access flags, and are a bit-wise 'or' of the following (octal) values:
  • 0400 - owning user may read from this resource.
  • 0200 - owning user may write to this resource.
  • 0040 - owning group may read from this resource.
  • 0020 - owning group may write to this resource.
  • 0004 - every other user may read from this resource.
  • 0002 - every other user may write to this resource.
• seq - used to keep system-internal info about the resource. for further info, check your kernel's sources (you are working on a system with free access to its source code, right?).

Part of the SysV IPC API allows us to modify the access permissions for the resources. We will encounter them when discussing the different IPC methods.

Since SysV IPC resources live outside the scope of a single process, there is a need to manage them somehow - delete resources that were left by irresponsible processes (or process crashes); check the number of existing resources of each type (especially to find if the system-global limit was reached), etc. Two utilities were created for handling these jobs: 'ipcs' - to check usage of SysV IPC resources, and 'ipcrm' - to remove such resources.

Running 'ipcs' will show us statistics separately for each of the three resource types (shared memory segments, semaphore arrays and message queues). For each resource type, the command will show us some statistics for each resource that exists in the system. It will show its identifier, owner, size of resources it occupies in the system, and permission flags. We may give 'ipcs' a flag to ask it to show only resources of one type ('-m' for shared Memory segments, -q for message Queues and '-s' for Semaphore arrays). We may also use 'ipcs' with the '-l' flag to see the system enforced limits on these resources, or the '-u' flag to show us usage summary. Refer to the manual page of 'ipcs' for more information.

The 'ipcrm' command accepts a resource type ('shm', 'msg' or 'sem') and a resource ID, and removes the given resource from the system. We need to have the proper permissions in order to delete a resource.

One of the problems with pipes is that it is up to you, as a programmer, to establish the protocol. Now, usually this protocol is based on sending separate messages. With a stream taken from a pipe, it means you have to somehow parse the bytes, and separate them to packets. Another problem is that data sent via pipes always arrives in a FIFO order. This means that before you can read any part of the stream, you have to consume all the bytes sent before the piece you're looking for, and thus you need to construct your own queuing mechanism on which you place the data you just skipped, to be read later. If that's what you're interested at, this is a good time to get acquainted with message queues.

A message queue is a queue onto which messages can be placed. A message is composed of a message type (which is a number), and message data. A message queue can be either private, or public. If it is private, it can be accessed only by its creating process or child processes of that creator. If it's public, it can be accessed by any process that knows the queue's key. Several processes may write messages onto a message queue, or read messages from the queue. Messages may be read by type, and thus not have to be read in a FIFO order as is the case with pipes.

In order to use a message queue, it has to be created first. The msgget() system call is used to do just that. This system call accepts two parameters - a queue key, and flags. The key may be one of:

• IPC_PRIVATE - used to create a private message queue.
• a positive integer - used to create (or access) a publicly-accessible message queue.

The second parameter contains flags that control how the system call is to be processed. It may contain flags like IPC_CREAT or IPC_EXCL, which behave similar to O_CREAT and O_EXCL in the open() system call, and will be explained later, and it also contains access permission bits. The lowest 9 bits of the flags are used to define access permission for the queue, much like similar 9 bits are used to control access to files. the bits are separated into 3 groups - user, group and others. In each set, the first bit refers to read permission, the second bit - to write permission, and the third bit is ignored (no execute permission is relevant to message queues).

Lets see an example of a code that creates a private message queue:

#include <stdio.h>     /* standard I/O routines.            */
#include <sys/types.h> /* standard system data types.       */
#include <sys/ipc.h>   /* common system V IPC structures.   */
#include <sys/msg.h>   /* message-queue specific functions. */

/* create a private message queue, with access only to the owner. */
int queue_id = msgget(IPC_PRIVATE, 0600); /* <-- this is an octal number. */
if (queue_id == -1) {
    perror("msgget");
    exit(1);
}

A few notes about this code:

1. the system call returns an integer identifying the created queue. Later on we can use this key in order to access the queue for reading and writing messages.
2. The queue created belongs to the user whose process created the queue. Thus, since the permission bits are '0600', only processes run on behalf of this user will have access to the queue.

Before we go to writing messages to the queue or reading messages from it, we need to see how a message looks. The system defines a structure named 'msgbuf' for this purpose. Here is how it is defined:

struct msgbuf {
    long mtype;     /* message type, a positive number (cannot be zero). */
    char mtext[1];  /* message body array. usually larger than one byte. */
};

The message type part is rather obvious. But how do we deal with a message text that is only 1 byte long? Well, we actually may place a much larger text inside a message. For this, we allocate more memory for a msgbuf structure than sizeof(struct msgbuf). Lets see how we create an "hello world" message:

/* first, define the message string */
char* msg_text = "hello world";
/* allocate a message with enough space for length of string and */
/* one extra byte for the terminating null character.            */
struct msgbuf* msg =
        (struct msgbuf*)malloc(sizeof(struct msgbuf) + strlen(msg_text));
/* set the message type. for example - set it to '1'. */
msg->mtype = 1;
/* finally, place the "hello world" string inside the message. */
strcpy(msg->mtext, msg_text);

Few notes:

1. When allocating a space for a string, one always needs to allocate one extra byte for the null character terminating the string. In our case, we allocated strlen(msg_text) more than the size of "struct msgbuf", and didn't need to allocate an extra place for the null character, cause that's already contained in the msgbuf structure (the 1 byte of mtext there).
2. We don't need to place only text messages in a message. We may also place binary data. In that case, we could allocate space as large as the msgbuf struct plus the size of our binary data, minus one byte. Of-course then to copy the data to the message, we'll use a function such as memset(), and not strcpy().

Once we created the message queue, and a message structure, we can place it on the message queue, using the msgsnd() system call. This system call copies our message structure and places that as the last message on the queue. It takes the following parameters:

1. int msqid - id of message queue, as returned from the msgget() call.
2. struct msgbuf* msg - a pointer to a properly initializes message structure, such as the one we prepared in the previous section.
3. int msgsz - the size of the data part (mtext) of the message, in bytes.
4. int msgflg - flags specifying how to send the message. may be a logical "or" of the following:
   • IPC_NOWAIT - if the message cannot be sent immediately, without blocking the process, return '-1', and set errno to EAGAIN.
   to set no flags, use the value '0'.

int rc = msgsnd(queue_id, msg, strlen(msg_text)+1, 0);
if (rc == -1) {
    perror("msgsnd");
    exit(1);
}

Note that we used a message size one larger than the length of the string, since we're also sending the null character. msgsnd() assumes the data in the message to be an arbitrary sequence of bytes, so it cannot know we've got the null character there too, unless we state that explicitly.

We may use the system call msgrcv() In order to read a message from a message queue. This system call accepts the following list of parameters:

1. int msqid - id of the queue, as returned from msgget().
2. struct msgbuf* msg - a pointer to a pre-allocated msgbuf structure. It should generally be large enough to contain a message with some arbitrary data (see more below).
3. int msgsz - size of largest message text we wish to receive. Must NOT be larger than the amount of space we allocated for the message text in 'msg'.
4. int msgtyp - Type of message we wish to read. may be one of:
   • 0 - The first message on the queue will be returned.
   • a positive integer - the first message on the queue whose type (mtype) equals this integer (unless a certain flag is set in msgflg, see below).
   • a negative integer - the first message on the queue whose type is less than or equal to the absolute value of this integer.
5. int msgflg - a logical 'or' combination of any of the following flags:
   • IPC_NOWAIT - if there is no message on the queue matching what we want to read, return '-1', and set errno to ENOMSG.
   • MSG_EXCEPT - if the message type parameter is a positive integer, then return the first message whose type is NOT equal to the given integer.
   • MSG_NOERROR - If a message with a text part larger than 'msgsz' matches what we want to read, then truncate the text when copying the message to our msgbuf structure. If this flag is not set and the message text is too large, the system call returns '-1', and errno is set to E2BIG.

Lets then try to read our message from the message queue:

/* prepare a message structure large enough to read our "hello world". */
struct msgbuf* recv_msg =
     (struct msgbuf*)malloc(sizeof(struct msgbuf)+strlen("hello world"));
/* use msgrcv() to read the message. We agree to get any type, and thus */
/* use '0' in the message type parameter, and use no flags (0).         */
int rc = msgrcv(queue_id, recv_msg, strlen("hello world")+1, 0, 0);
if (rc == -1) {
    perror("msgrcv");
    exit(1);
}

A few notes:

1. If the message on the queue was larger than the size of "hello world" (plus one), we would get an error, and thus exit.
2. If there was no message on the queue, the msgrcv() call would have blocked our process until one of the following happens:
   • a suitable message was placed on the queue.
   • the queue was removed (and then errno would be set to EIDRM).
   • our process received a signal (and then errno would be set to EINTR.

As an example of using non-private message queues, we will show a program, named "queue_sender", that creates a message queue, and then starts sending messages with different priorities onto the queue. A second program, named "queue_reader", may be run that reads the messages from the queue, and does something with them (in our example - just prints their contents to standard output). The "queue_reader" is given a number on its command line, which is the priority of messages that it should read. By running several copies of this program simultaneously, we can achieve a basic level of concurrency. Such a mechanism may be used by a system in which several clients may be sending requests of different types, that need to be handled differently. The complete source code may be found in the public-queue directory.