[SciPy-dev] Ideas for scipy.sparse?
Fri Apr 11 14:26:09 CDT 2008
On Fri, Apr 11, 2008 at 12:47 PM, Brian Granger <email@example.com> wrote:
> 1) I need N-dimensional sparse arrays. Some of the storage formats in
> scipy.sparse (dok, coo, maybe lil) could be generalized to
> N-dimensions, but some work would have to be done.
To make this efficient, you'd probably need a lower-level
implementation of a hash-based container like DOK.
BTW, which applications use sparse N-dimensional arrays?
> 2) I need these things to be in numpy. I hate to start another
> "should this go into numpy or scipy" thread, but I actually do think
> there is a decent case for moving the core sparse arrays into numpy
> (not the solvers though). Please hear me out:
I don't see the need for or utility of sparse arrays in numpy. Numpy
does low-level manipulation of dense arrays/memory buffers.
> 3) I need sparse arrays that are implemented more in C. What do I
> mean by this. I am using cython for the performance critical parts of
> my package and there are certain things (indexing in tight loops for
> example) that I need to do in c. Because the current sparse array
> classes are written in pure python (with a few c++ routines underneath
> for format conversions), this is difficult. So...
All of the costly operations on CSR/CSC/COO matrices are done in C++.
Only lil_matrix and dok_matrix are pure-Python implementations.
Indexing sparse arrays inside a Python loop *is* slow, but there's not
much that can be done about it.
> I think it would be a very good idea to begin moving the sparse array
> classes to cython code. This would be a very nice approach because it
> could be done gradually, without breaking any of the API. The benefit
> is that we could improve the performance of the sparse array classes
> drammatically, while keeping things very maintainable.
I'm not aware of any performance problems with the existing backend to
scipy.sparse. What would you implement in cython that's not already
implemented in scipy.sparse.sparsetools?
> 3) That we begin to move their implementation to using Cython (as an
> aside, cython does play very well with templated C++ code). This
> could provide a much nicer way of tying into the c++ code than using
In the context of scipy.sparse, what's wrong with SWIG and C++? What
would be improved by using Cython?
> Alright, fire away :)
Nathan Bell firstname.lastname@example.org
More information about the Scipy-dev