I already blogged about Gecko's frame continuations being a mistake. It's time to rant about another big layout design error: the uniformity of the Reflow() API.
During layout, frames (i.e. CSS boxes) call Reflow to lay out their children and then position those children in some manner. The core problem is that Reflow's signature is the same for every frame type. That means that in theory, table frames have to interact with their row(group) frames in exactly the same way that inline frames have to interact with their inline children, or XUL boxes have to interact with their box children, etc. But this is nonsense, since the invariants, dependencies and data that need to cross the parent/child boundary are entirely different in each of these cases.
So in practice we have various kinds of magic probing and/or magic fields of structures that are only used in certain cases. This inevitably leads to breakage and overcomplex code. Problems are particularly acute in line layout, where we really would like to work with a flattened list of inline leaf items, but thanks to Reflow we can't. The textrun work has moved us to a situation where we construct a flattened representation of at least the text of a paragraph for line breaking and text shaping, and then we pull data from that in a hokey way as we Reflow text frames. It's somewhat painful as the two traversal models fight each other in nsLineLayout. Of course the aforementioned frame continuations crash the party too.
What we really should do is admit that parent-child frame type pairs are actually tightly constrained, i.e. an inline frame can only have an inline parent or a block parent, a table row frame can only have a table rowgroup parent which can only have a table parent, etc. (These invariants already exist actually, although the code sometimes doesn't want to admit it.) Then we should toss Reflow over the side and give each frame type its own specialized layout interface. Only a few frame types (e.g. blocks and tables) would be able to occur in "unknown" contexts and offer a generic layout interface.
In fact, we could go further and specialize further the types of child frames or even get rid of a uniform nsIFrame tree altogether. There's no particular reason why it makes sense to have tables, table rowgroups, table rows and table cells all be the same kind of object as text frames or SVG frames. Sure, we want to be able to iterate over CSS box tree geometry and style somehow, but there's a whole lot of generality exposed in nsIFrame that doesn't really make sense everywhere.
As an aside, the conflict between textrun layout and inline layout is an example of a general problem I've seen with cleaning up code. You rework module A to make it nice and clean, but because modules B and C are quirky, you have to add complexity to module A to keep things working and shippable. Eventually you rework B and C too but A's already been contaminated, so B and C inherit some of the sins of their fathers. You can reduce the effect by enlarging the scope of rework, but that adds risk and schedule issues of course --- in the limit you rewrite everything and sink your entire project.