[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [kDev] Revocation, reification, logging and audit tree...
Hi Kirit and All,
Our data could be comprised of anything from freeform to structured.
So, we need a solution that can cope with both these extremes and
anywhere in between. Seeing that RDF type triples can get really fine
grained how do we represent them in our database? A triple store? So,
are you saying we could have a triple store for unstructured data and
metadata structured tables side by side?
What happens if the data starts off being unstructured and moves
towards being more structured as people add more data, perhaps using
the same terminology to describe things? Or vice versa, the data
becomes more unstructured as people make more cross relationships
between elements of structured data. We just don't know what's going to
happen with any given data or set of users.
Another way I look at your recommendation is to think of it as
optimisation. Optimisation of queries. So, you're saying set up a load
of tables based on the structure of the metadata/data and I'm saying
that these metadata structures could change as time progresses. But
what we're really talking about here is how we query the data. We want
to optimise the query. But the queries may change from day to day and
from user to user. So, we need to cope with new, unforeseen query
structures as well as expected query structures.
Our database, in response to an incoming query (external or internal to
the database), could dynamically create the tables necessary to fulfil
the query - to optimise performance. We would then cache the tables
(the structure and the data). The next time the same query came in we'd
either use the same table or refresh the data in it using the same
table structure. Apart from the initial table creation this would give
us better performance, wouldn't it?
And, for good measure, when importing XML, we'll create their
respective structured tables. And also allow users to pre-define tables
based on hunches about incoming queries. So, the table cache would hold
these and dynamically created tables and have rules about time to live
and first in first out, etc.
I can't believe any of this is original thought. Someone must have
looked into this. Any ideas who? Has it been implemented? Does it make
sense? Are there any alternatives?
In order to "think this through" I'm using methods that I'm familiar
with, such as open discussion, requirement descriptions and simple
demonstrations. But not much in the way of learnt database theory, I'm
afraid. The big question is: how would you think this through? What
methods would you use? Could you do a simple demonstration?
Joke: What advice did people give to the person who invented the wheel?
On 26 Oct 2003, at 12:33, Kirit Saelensminde wrote:
You're not really proposing to store a huge amount of disparate data
patterns in just a few tables are you? If you want a database
structure like that then maybe putting it into an SQL database isn't
what you really want - the performance is going to suck big time the
moment you put a few million records in there.