Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch Buffers to memoryviews & remove extra copies/allocations #656

Merged
merged 31 commits into from
Mar 30, 2025
Merged
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
0c823e2
Switch `Buffer`s to `memoryview`s
jakirkham Nov 22, 2024
c690446
Merge zarr-developers/main into jakirkham/use_mv
jakirkham Mar 26, 2025
9ac8d8e
Add back `ZSTD_freeCCtx`
jakirkham Mar 26, 2025
0ade20c
Drop leftover `Buffer` from merge conflict
jakirkham Mar 26, 2025
13f8e50
Add minor comment
jakirkham Mar 26, 2025
bd1c401
Add trivial `try...finally...`s to cleanup diff
jakirkham Mar 26, 2025
64eed12
Use Cython `cimport`s for Python C API
jakirkham Mar 26, 2025
31a7446
Add news entry
jakirkham Mar 26, 2025
9069553
Move `cimport`s from `libc` up top
jakirkham Mar 26, 2025
1ab9983
Resize buffers without copying
jakirkham Mar 26, 2025
f91c283
Write directly to output array in VLen*
jakirkham Mar 26, 2025
d0a7721
Use `_mv` subscript name for typed-memoryivews
jakirkham Mar 26, 2025
7bc8022
Avoid excess copies in fletcher32
jakirkham Mar 26, 2025
4341ce3
Add news entry for better memory usage
jakirkham Mar 26, 2025
4baf1e1
Fix fletecher32's `cimport`s
jakirkham Mar 26, 2025
0ea019c
Fix blank lines to match
jakirkham Mar 26, 2025
2b871c7
Reassign `dest` with `dest_objptr`
jakirkham Mar 26, 2025
bdc7bc9
Fix `source_ptr` type
jakirkham Mar 26, 2025
0bfafed
Fix declaration order
jakirkham Mar 26, 2025
c17e314
Wrap `_PyBytes_Resize` for improved usability
jakirkham Mar 27, 2025
54fd4b2
Add `ensure_continguous_memoryview` function
jakirkham Mar 27, 2025
eac4ef1
Use `ensure_contiguous_memoryview`
jakirkham Mar 27, 2025
a77e8cf
Move `PyBytes_RESIZE` macro to `compat_ext`
jakirkham Mar 27, 2025
fb5ffac
Minimize diff by readding blank line after `try`
jakirkham Mar 27, 2025
9b289c6
Group `encv` with `value` args
jakirkham Mar 27, 2025
7d593b8
Organize `vlen`'s `imports`
jakirkham Mar 27, 2025
7d68d53
Use `ensure_contiguous_memoryview` with VLen
jakirkham Mar 27, 2025
b6d91ce
Space out input arg handling & checksum check
jakirkham Mar 27, 2025
2dd1cdf
Use `memcpy` to speedup copies in `fletcher32`
jakirkham Mar 27, 2025
b7bb7ef
In fletcher32's if output buffer, slice from input
jakirkham Mar 27, 2025
c83136a
Unwrap lines no longer needing wrapping
jakirkham Mar 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use Cython cimports for Python C API
jakirkham committed Mar 26, 2025
commit 64eed124e9b5aba32255b57904b2b5850b53e92d
17 changes: 9 additions & 8 deletions numcodecs/vlen.pyx
Original file line number Diff line number Diff line change
@@ -12,26 +12,27 @@ import numpy as np
from .abc import Codec
from .compat import ensure_contiguous_ndarray
from cpython.buffer cimport PyBuffer_IsContiguous
from cpython.bytearray cimport (
PyByteArray_AS_STRING,
PyByteArray_FromStringAndSize,
)
from cpython.bytes cimport (
PyBytes_AS_STRING,
PyBytes_GET_SIZE,
PyBytes_Check,
PyBytes_FromStringAndSize,
)
from cpython.memoryview cimport PyMemoryView_GET_BUFFER
from cpython.unicode cimport PyUnicode_AsUTF8String
from cpython.unicode cimport (
PyUnicode_AsUTF8String,
PyUnicode_Check,
PyUnicode_FromStringAndSize,
)
from libc.stdint cimport uint8_t
from libc.string cimport memcpy
from ._utils cimport store_le32, load_le32


cdef extern from "Python.h":
bytearray PyByteArray_FromStringAndSize(char *v, Py_ssize_t l)
char* PyByteArray_AS_STRING(object string)
object PyUnicode_FromStringAndSize(const char *u, Py_ssize_t size)
int PyUnicode_Check(object text)


# 4 bytes to store number of items
cdef Py_ssize_t HEADER_LENGTH = 4