GIS Software Legacy
The previous high cost of ArcSDE and lack of a spatial datatype in SQL Server (until 2008), and Oracle until version 10, meant there was little choice but to store data in shapefiles for many organisations (and by tenderers to keep bid costs down).
The introduction of native spatial types in SQL Server meant almost instantly that ArcSDE went from a huge investment, to being included for free in ArcGIS, and the "bringing in to the fold" of spatial data in organisations.
Organisations using ArcGIS and SQL Server previously had three choices:
- Pay the 20k+ fee to purchase ArcSDE and store spatial data in "proper" SQL Server databases.
- Store spatial data in shapefiles / personal GDBs, and link to the rest of the organisational data in databases (or export these attributes to DBFs)
- Switch GIS vendors and store spatial data in a single database but in a format only accessible by the new GIS software
Once SQL Server had a native spatial type most vendors used this instead of their proprietary formats, meaning spatial data could suddenly be accessed by other applications. ESRI had to either reduce the cost of ArcSDE (which they did by integrating it into ArcGIS) and/or allow spatial data to be stored in the native database format.
In addition queries performed in ArcIMS on shapefiles meant associated with DBFs had to include all required fields and duplication as there was no option to create spatial views, or easily linking features with a back end database.
I agree with others that that until recently spatial data became a native database type it has long been ignored or kept separate by database administrators in organisations, and become the reponsibility of a GIS manager. The concepts of database design, normalisation, replication, security, and SQL views require an often very different, and specialised skillset and cannot easily be learned as you go along.
Explaining in a tender the requirement of large amount of time and effort to be spent on a data model, and cleaning / importing data into this model is often impossible. Often the project purchasers are coming from an analytical view of GIS and overlook the importance of structured data.