said by guppy_fish:
Verizon could care less what its users send/receive as being a tier one provider, it costs the same for one bit or one trillion GB, they have no peering charges
Peering charges have nothing to do with the stuff I was thinking about.
Unless Verizon can break the laws of physics, their costs would start increasing exponentially once they start having to stitch multiple 2M$ routers together to accommodate peak demand across network nodes. Having to do this at few strategic facilities is one thing but having to do it systematically network-wide would kick costs up a few notches.
The biggest single-chassis routers can handle about 5Tbps of non-blocking traffic. Once you need to go beyond that while maintaining close to non-blocking routing, things get a whole lot more expensive and that is definitely where Verizon or any other ISP with millions of subscribers would end up if everyone was trying to use 100+Mbps during peak hours, ignoring potential congestion at the DSLAM or node/CMTS level.
People keep saying that equipment gets cheaper but what they almost always neglect to mention is that density in routed Tbps/rack only doubles every ~5 years, which is much too slow to keep up with peak demand which increases by 50-60%/year which is about 10X over the same period. Since technological progress alone is insufficient to meet demand (about 5X too slow), how many identical switches or routers do you think you need to put together to double the capacity of a single one of the same while maintaining non-blocking performance? You need six of 'em... 2X the capacity = 6X the rack space using same-model equipment. The cost scaling is really horrible.
Not every ISP can afford (or is willing) to use the latest and biggest gear available so don't be surprised if there are more stories about smaller ISPs hitting their equipment's practical brick walls in the future or attempting to extract money from their transit providers to cover some upgrade costs.