A memory efficient implementation of the .mtx reading function

[gjeuken]

• Closes #
• Tests included or not required because: test_datasets.py already implemented

• Release notes not necessary because: This is a backend change

Pandas read_csv function is very memory intensive, and this makes loading data (especially large datasets from EBI Single Cell Expression Atlas) impossible on computers with 16gb of ram or less. The subsequent analysis of such datasets with scanpy, however, works well on such computers.

Loading the data into chunks, using the same pandas function, solves this problem.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.45%. Comparing base (9741ca6) to head (792b5e2).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3389   +/-   ##
=======================================
  Coverage   75.44%   75.45%           
=======================================
  Files         113      113           
  Lines       13250    13253    +3     
=======================================
+ Hits         9997    10000    +3     
  Misses       3253     3253           

Files with missing lines	Coverage Δ
src/scanpy/datasets/_ebi_expression_atlas.py	94.18% <100.00%> (+0.21%)	⬆️

[flying-sheep]

Good idea! some small notes:

[flying-sheep]

Suggested change

	mtx = sparse.csr_matrix(([0], ([0], [0])), shape=(m, n))
	mtx = sparse.csr_matrix((m, n), dtype=np.float64)

[flying-sheep]

This is probably slightly slower than necessary, since we know the chunks don‘t overlap and = needs to deal with actually summing up things. But I imagine it could also be pretty well optimized, so if the following is not faster, please just add a comment instead explaining that + is well-enough optimized.

The way csr_matrix((data, (i, j)), [shape]) works is that it first creates a coo_matrix, then converts it to csr.
I think the best way is probably:

1. build up data, i, j arrays in a loop
2. create a csr_matrix from the final arrays as the last step