Are there any practical situations that can be represented using a modified definition of an edge - where each edge connects three vertices instead of two?
You're talking about $3$-uniform hypergraphs. They will arise in a range of contexts, some of which are:
These contexts are more areas of pure mathematics, and can be rephrased to not be $3$-uniform hypergraphs (however, I'm guessing all $3$-uniform hypergraph problems can be thus rephrased).
Combinatorial designs, such as Steiner triple systems, have applications in software testing. Hypergraphs in general arise in the study of complex networks, but I haven't seen the particular case of $3$-uniform hypergraphs arise.
However, we don't need applications for mathematics to be interesting; the following quote applies to mathematics as well as physics.