Skip to content

Commit

Permalink
Fix: Hash @indices can grow larger than Int32::MAX bytes (#15347)
Browse files Browse the repository at this point in the history
`Hash` can theoretically hold up to `Int32::MAX` entries, but `@indices` grows faster. Using `Int32` maths to calculate the memory size for `@indices` limits capacity to `Int32::MAX // 4`. This patch uses `Int64` maths instead to enable bigger hash sizes.

Co-authored-by: Johannes Müller <[email protected]>
  • Loading branch information
ysbaddaden and straight-shoota authored Jan 18, 2025
1 parent 2b544f1 commit c4682db
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 2 deletions.
8 changes: 8 additions & 0 deletions spec/manual/hash_large_spec.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
require "spec"

it "creates Hash at maximum capacity" do
# we don't try to go as high as Int32::MAX because it would allocate 18GB of
# memory in total. This already tests for Int32 overflows while 'only' needing
# 4.5GB of memory.
Hash(Int32, Int32).new(initial_capacity: (Int32::MAX // 4) + 1)
end
6 changes: 4 additions & 2 deletions src/hash.cr
Original file line number Diff line number Diff line change
Expand Up @@ -236,12 +236,14 @@ class Hash(K, V)
# Translate initial capacity to the nearest power of 2, but keep it a minimum of 8.
if initial_capacity < 8
initial_entries_size = 8
elsif initial_capacity > 2**30
initial_entries_size = Int32::MAX
else
initial_entries_size = Math.pw2ceil(initial_capacity)
end

# Because we always keep indice_size >= entries_size * 2
initial_indices_size = initial_entries_size * 2
initial_indices_size = initial_entries_size.to_u64 * 2

@entries = malloc_entries(initial_entries_size)

Expand Down Expand Up @@ -830,7 +832,7 @@ class Hash(K, V)

# The actual number of bytes needed to allocate `@indices`.
private def indices_malloc_size(size)
size * @indices_bytesize
size.to_u64 * @indices_bytesize
end

# Reallocates `size` number of indices for `@indices`.
Expand Down

0 comments on commit c4682db

Please sign in to comment.