STAGE 4 · THE PIPE MANIFOLD

The Pipe Manifold

A pipeline is not three steps in order. It is three processes running at once.

What you're wiring up

A pipe is a kernel-managed byte buffer with a write end and a read end, handed to you as two file descriptors. When you run cat log | grep err | wc -l, the shell is not running three commands in sequence and passing text along. It creates the pipes, forks all three children, cross-plugs their fd slots, and then gets out of the way while all three run simultaneously.

That concurrency is the point. It is why you can pipe a gigabyte through without a gigabyte of memory, and why an infinite producer like yes terminates the moment head has seen enough.

There is one bug on this stage that every person writing a shell hits, and it is worth hitting. End of file on a pipe does not mean the writer finished. It means every copy of the write end is closed. Fork duplicates descriptors, so one forgotten copy anywhere, including in the parent, and the reader waits forever.

# cat f | tr a-z A-Z | wc -l   two pipes, three children
pipe A: child1 stdout → child2 stdin
pipe B: child2 stdout → child3 stdin

# every copy of a write end must be closed
parent closes both ends, each child closes what it did not dup2

Assembly steps

[ 01 ]

Split the token stream on | into pipeline segments, each with its own argv and its own redirection list.

hint

Reuse the Stage 3 parser per segment. Precedence is simple: redirections bind inside a segment.

[ 02 ]

Get two-command pipelines working: one pipe(2), two forks, dup2 the write end onto child one's stdout and the read end onto child two's stdin.

hint

After the dup2s, close both pipe fds in both children and in the parent. Count the copies; any surviving write end is a hang.

[ 03 ]

Generalize to N stages with N-1 pipes, forking every child before waiting on any of them.

hint

A loop carrying prev_read_fd between iterations is cleaner than an array of every pipe.

[ 04 ]

Wait for every child, but set $? from the last segment only, per POSIX.

hint

waitpid each pid you collected; store only the final one's status.

[ 05 ]

Let builtins appear inside pipelines without wrecking the shell, as in pwd | wc -c.

hint

Simplest correct answer: run a builtin inside a pipeline in a forked child like anything else. It cannot mutate shell state there, and that matches real shells.

Go deeper (after it passes)

The Unix pipes philosophy section of DDIA Chapter 10 is literally this stage: a shell pipeline is single-machine MapReduce. Follow it into DDIA M10 — Batch: Your Own MapReduce.