Despite your impossible requirements, I'll scribble down my thoughts for other people in the future who aren't so hamstrung, based on my experiences doing this for Github.
Distributing data across a number of locations (be they partitions, machines, data centres) based on a hash is a dangerous undertaking, for two reasons:
- You're never going to get a balanced distribution of data based on your hash -- not necessarily because the hash isn't balanced (although that is a factor too) but more because the items you're storing aren't equal size. So you store two items, one 1kB in size the other 1GB in size. Already you're massively unbalanced. Try that a few times and all of a sudden you've got big imbalances.
- Once your hash-to-server algorithm is in place, you can't change the number of "buckets" (machines, partitions, whatever) to store your data in without a massive amount of pain. This is because the hash algorithm is used both to decide where to store stuff, and also where to find it again. If you change the number of servers, then the rule of "where stuff is" changes, and so some of the existing data is expected to be somewhere else. You end up either having a lengthy offline "rebalance" operation (each server searches for data that, in the new scheme, should be somewhere else, and moves it there) or you have to search for your data on all the fileservers (mmmm, inefficient).
On the other hand, having a lookup table for all your files makes these problems go away. When you say "no database", I'm betting you insert an implicit "SQL" before "database". However, there is a whole other world of databases out there that have nothing to do with SQL, and they are perfect for this situation. They're known as "key-value stores", and if you're dead keen on going ahead with building this boondoggle yourself, then I'd highly recommend using one (I've got experience with Redis, but they all seem pretty reasonable).
Ultimately, though, if you go ahead with the "all hashes, all the time" system and then hack around the problems inherent in it (there are solutions, just not real awesome ones) all you will end up with, at the end of the day, is a half-assed, botchy, non-feature-complete version of GlusterFS. If you need a large amount of storage, growable over time, distributed across multiple physical machines, in a single namespace, I really would recommend it over anything you can build yourself.