What makes a node?
The individual Isilon nodes are generally commodity servers running Isilon's OneFS operating system, though there several things that are done to the nodes to make them differ from a white-box server. Firstly, every node has an Infiniband card in it and is connected to every other node in the cluster over that Infiniband network. Through this back-end network the nodes share everything with each other—the distributed pieces of the single big OneFS volume, metadata about the files that they're keeping, the contents of each others' cache, their hopes and secret dreams—wait, not that, sorry. The back-end network keeps every node in communication with every other node, though, and this tight communication is necessary because of how the Isilon system uses resources from as many nodes as possible for as many operations as it can. More on that in the next section, though.
The distributed OneFS file system is journaled, and every node keeps its copy of the file system journal on a 512MB NVRAM card. A "journaled file system" is a file system where every change that gets made to the file system is recorded in a log (called a journal) before it's actually changed on the disk, so that if something goes wrong while the change is being made to the disk—if there's a power failure, or the system bursts into flame, or an angry sysadmin spin-kicks the server Quantum Leap-style and it goes flying across the data center—then the change that was interrupted can be reconstructed and finished or re-done when the system returns to normal, probably without any corruption to the file system. Most common file systems these days (like NTFS, ext3, and HFS+) are journaled. Keeping the journal on a separate NVRAM card instead of integrated on the cluster's disks is done partially to add an additional layer of redundancy, and also so that when a new node is added into an Isilon cluster it can immediately get a copy of the file system journal and begin participating in the cluster's read and write workload.
It's all about the OneFS
The thing that makes an Isilon system an Isilon system, though, and not just a whole bunch of servers in a rack, is the operating system. That operating system, OneFS, started out in its 1.0 guise as FreeBSD with some extra stuff, and has since matured into a completely separate OS. It provides the coordination and clustering of resources between nodes, keeps track of where all the files are stored in the big cluster, controls how incoming writes from users are striped, keeps the amount of used space balanced between all of the nodes in the cluster, moves data around between different types of nodes in accordance with set system policies, and does all the other things necessary to keep the Isilon system operating as a single entity with a single big volume of storage.
Among the things that OneFS provides, there are two key areas to focus on—data protection, and IO distribution. These two things are linked pretty closely together, too, so let's dive in.
As stated above, all of the storage in all of the Isilon nodes is presented as a single big volume, but how exactly does that work? Each node has a certain number of disks in it, with the exact number varying by node type—the big near-line SATA nodes have more disks than the performance oriented nodes, for example—but they're not held together by any kind of RAID layout or by simple concatenation. Instead, the OneFS file system takes files, breaks them up into 8K blocks, and then stripes those blocks in 16-block units across the disks of as many nodes in the cluster as it's allowed to use, depending on the configured protection level. The striping goes "wide" (that is, across nodes) before it goes "deep" (across multiple disks in one node)—that is, OneFS tries to get data written to one disk in every node possible first, before then wrapping back and putting data on a second disk in a node. If the file is big enough that it stretches across one disk in every node, the striping will wrap around to the second disk in the first node and start again.
Files don't necessarily fit perfectly into 128KB pieces, though, and so when the Isilon system gets to the last piece of the file it's writing and finds it has less than 128KB left, it pads the chunk out to 128KB and commits that. Conversely, some files are a lot less than 128KB, and so they are by default mirrored instead of striped.
Tied into this picture of striping is the underlying reason why Isilon works so well for big files, like giant raw video files of blue cat people for Jim Cameron or thousands and thousands of multi-gigabyte iTunes movies: large enough files get striped across lots of disks inside lots of nodes, and access to those files is serviced by what could be a very large amount of distributed CPU and global cache.
You've been made redundant
Striping without any protection against failure does no one any good, though, because then all you're doing is adding more things that could break. So, OneFS incorporates a customizable amount of redundancy in how it does its striping, using either stripe element mirroring or a RAID-like distributed parity scheme with one or more parity stripe elements using FEC, or "Forward Error Correction."
Isilon writes out its FEC scheme in N+M notation, where "N" refers to the number of nodes in a stripe that will get actual data, and "M" refers to the number of nodes that contain parity data. "M" also signifies the total number of failures, both nodes and disks combined, that can occur without affecting the availability of the stripes. The minimum number of nodes in an Isilon cluster is three, so the minimum FEC protection level is 2+1—each stripe the cluster makes consists of two data chunks and one parity chunk, with each chunk going to a different node. Any node in the cluster can lose a disk and things are OK, or any node can die and things are OK. However, if you lose a disk in one node and a disk in a second node, it would be just as bad for the affected stripes as losing two disks in a RAID 5 set.
The picture above shows an extremely simplified three node system, with each node having two disks (such a configuration doesn't actually exist, but Visio is hard, so just go with it). The default protection on this system is set to 2+1, which is really the only protection level we can have other than mirroring. Each file gets broken up into 128KB elements and striped across the nodes. In this idealized system, it turns out that we only have six files to write, and each of those files is 256KB. So, we write files A, B, C, D, E, and F out to the system and it happily makes with the striping.
But, uh oh, node 1's disk explodes!
What does this mean for our data? File A has lost its parity element, and files B and C have both lost a data element. Nothing we can't replace, though, and in a real system with more disks, the cluster would begin recreating the affected files' protection with unused capacity. We don't have that here, though, so the sysadmin better hurry up and replace the failed drive before OH NO—
Node 1 now has no more disks left, but we're still OK—none of the files have lost any data that can't be recreated from parity. Of course, at this point we're at the limit of our protection, and any additional disk failures would result in lost data.