Is there a way to define a tensor product over a measure space(=index set) with a continuous measure for Hilbert spaces?
For the sum we have the notion of a direct integral, here.
"Continuous tensor products" have been applied in certain areas such as quantum stochastic processes and quantum field theory, see for example the following paper by Arveson. I think that the definitions in Vershik and Tsilevich are more transparent (in this article the continuous tensor product is mainly referred to as a "factorization").
The basic object which possesses a continuous tensor product structure is the Fock space. However, one can find in the references of Vershik examples of non-Fock factorizations, see for example the following talk. A further application is in the representation theory of current algebras.