Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaking #946

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+241 −80
Open

Breaking #946

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cd83dfe
include DataType in CategoricalEltypes (#876)
tiemvanderdeure Mar 8, 2025
685efd7
Breaking: `DimVector` of `NamedTuple` is a `NamedTuple` `DimTable` (#…
rafaqz Mar 8, 2025
49746ef
Breaking: add `combine` method for `groupby` output, fixing `similar`…
rafaqz Mar 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/src/api/reference.md
View file
Original file line number Diff line number Diff line change
@@ -69,6 +69,7 @@ For transforming DimensionalData objects:

```@docs
groupby
combine
DimensionalData.DimGroupByArray
Bins
ranges
2 changes: 1 addition & 1 deletion src/DimensionalData.jl
View file
Original file line number Diff line number Diff line change
@@ -83,7 +83,7 @@ export dimnum, hasdim, hasselection, otherdims
export set, rebuild, reorder, modify, broadcast_dims, broadcast_dims!,
mergedims, unmergedims, maplayers

export groupby, seasons, months, hours, intervals, ranges
export groupby, combine, seasons, months, hours, intervals, ranges


export @d
2 changes: 1 addition & 1 deletion src/Lookups/lookup_arrays.jl
View file
Original file line number Diff line number Diff line change
@@ -468,7 +468,7 @@ abstract type AbstractCategorical{T,O} <: Aligned{T,O} end
order(lookup::AbstractCategorical) = lookup.order
metadata(lookup::AbstractCategorical) = lookup.metadata

const CategoricalEltypes = Union{AbstractChar,Symbol,AbstractString}
const CategoricalEltypes = Union{AbstractChar,Symbol,AbstractString,DataType}

function Adapt.adapt_structure(to, l::AbstractCategorical)
rebuild(l; data=Adapt.adapt(to, parent(l)), metadata=NoMetadata())
2 changes: 1 addition & 1 deletion src/array/methods.jl
View file
Original file line number Diff line number Diff line change
@@ -421,7 +421,7 @@ function _check_cat_lookups(D, ::Regular, lookups...)
@warn _cat_warn_string(D, "step sizes $(step(span(l))) and $s do not match")
return false
end
if !(lastval + s ≈ first(l))
if !(s isa Dates.AbstractTime) && !(lastval + s ≈ first(l))
@warn _cat_warn_string(D, "`Regular` lookups do not join with the correct step size: $(lastval) + $s ≈ $(first(l)) should hold")
return false
end
49 changes: 47 additions & 2 deletions src/groupby.jl
View file
Original file line number Diff line number Diff line change
@@ -247,9 +247,8 @@

Group some data along the time dimension:

```jldoctest groupby; setup = :(using Random; Random.seed!(123))

Check failure on line 250 in src/groupby.jl

GitHub Actions / build 

doctest failure in ~/work/DimensionalData.jl/DimensionalData.jl/src/groupby.jl:250-270

```jldoctest groupby; setup = :(using Random; Random.seed!(123))
julia> using DimensionalData, Dates
julia> A = rand(X(1:0.1:20), Y(1:20), Ti(DateTime(2000):Day(3):DateTime(2003)));

julia> groups = groupby(A, Ti => month) # Group by month
┌ 12-element DimGroupByArray{DimArray{Float64,3},1} ┐
├───────────────────────────────────────────────────┴───────────── dims ┐
  ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
├───────────────────────────────────────────────────────────── metadata ┤
  Dict{Symbol, Any} with 1 entry:
  :groupby => :Ti=>month
├─────────────────────────────────────────────────────────── group dims ┤
  ↓ X, → Y, ↗ Ti
└───────────────────────────────────────────────────────────────────────┘
  1  191×20×32 DimArray
  2  191×20×28 DimArray
  3  191×20×31 DimArray
  ⋮
 11  191×20×30 DimArray
 12  191×20×31 DimArray
```

Subexpression:

A = rand(X(1:0.1:20), Y(1:20), Ti(DateTime(2000):Day(3):DateTime(2003)));

Evaluated output:

ERROR: UndefVarError: `DateTime` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Hint: a global variable of this name also exists in Dates.
Stacktrace:
 [1] top-level scope
   @ none:1

Expected output:

using DimensionalData, Dates


  diff = 
    Warning: Diff output requires color.
    using DimensionalData, DatesERROR: UndefVarError: `DateTime` not defined in `Main`
    Suggestion: check for spelling errors or missing imports.
    Hint: a global variable of this name also exists in Dates.
    Stacktrace:
     [1] top-level scope
       @ none:1

Check failure on line 250 in src/groupby.jl

GitHub Actions / build 

doctest failure in ~/work/DimensionalData.jl/DimensionalData.jl/src/groupby.jl:250-270

```jldoctest groupby; setup = :(using Random; Random.seed!(123))
julia> using DimensionalData, Dates
julia> A = rand(X(1:0.1:20), Y(1:20), Ti(DateTime(2000):Day(3):DateTime(2003)));

julia> groups = groupby(A, Ti => month) # Group by month
┌ 12-element DimGroupByArray{DimArray{Float64,3},1} ┐
├───────────────────────────────────────────────────┴───────────── dims ┐
  ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
├───────────────────────────────────────────────────────────── metadata ┤
  Dict{Symbol, Any} with 1 entry:
  :groupby => :Ti=>month
├─────────────────────────────────────────────────────────── group dims ┤
  ↓ X, → Y, ↗ Ti
└───────────────────────────────────────────────────────────────────────┘
  1  191×20×32 DimArray
  2  191×20×28 DimArray
  3  191×20×31 DimArray
  ⋮
 11  191×20×30 DimArray
 12  191×20×31 DimArray
```

Subexpression:

groups = groupby(A, Ti => month) # Group by month

Evaluated output:

ERROR: UndefVarError: `A` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ none:1

Expected output:




  diff = 
    Warning: Diff output requires color.
    ERROR: UndefVarError: `A` not defined in `Main`
    Suggestion: check for spelling errors or missing imports.
    Stacktrace:
     [1] top-level scope
       @ none:1
julia> using DimensionalData, Dates

julia> A = rand(X(1:0.1:20), Y(1:20), Ti(DateTime(2000):Day(3):DateTime(2003)));

julia> groups = groupby(A, Ti => month) # Group by month
@@ -272,7 +271,7 @@

And take the mean:

```jldoctest groupby; setup = :(using Statistics)

Check failure on line 274 in src/groupby.jl

GitHub Actions / build 

doctest failure in ~/work/DimensionalData.jl/DimensionalData.jl/src/groupby.jl:274-291

```jldoctest groupby; setup = :(using Statistics)
julia> groupmeans = mean.(groups) # Take the monthly mean
┌ 12-element DimArray{Float64, 1} ┐
├─────────────────────────────────┴─────────────────────────────── dims ┐
  ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
├───────────────────────────────────────────────────────────── metadata ┤
  Dict{Symbol, Any} with 1 entry:
  :groupby => :Ti=>month
└───────────────────────────────────────────────────────────────────────┘
  1  0.500064
  2  0.499762
  3  0.500083
  4  0.499985
  ⋮
 10  0.500874
 11  0.498704
 12  0.50047
```

Subexpression:

groupmeans = mean.(groups) # Take the monthly mean

Evaluated output:

ERROR: UndefVarError: `groups` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ none:1

Expected output:

┌ 12-element DimArray{Float64, 1} ┐
├─────────────────────────────────┴─────────────────────────────── dims ┐
  ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
├───────────────────────────────────────────────────────────── metadata ┤
  Dict{Symbol, Any} with 1 entry:
  :groupby => :Ti=>month
└───────────────────────────────────────────────────────────────────────┘
  1  0.500064
  2  0.499762
  3  0.500083
  4  0.499985
  ⋮
 10  0.500874
 11  0.498704
 12  0.50047


  diff = 
    Warning: Diff output requires color.
    ┌ 12-element DimArray{Float64, 1} ┐
    ├─────────────────────────────────┴─────────────────────────────── dims ┐
      ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points
    ├───────────────────────────────────────────────────────────── metadata ┤
      Dict{Symbol, Any} with 1 entry:
      :groupby => :Ti=>month
    └───────────────────────────────────────────────────────────────────────┘
      1  0.500064
      2  0.499762
      3  0.500083
      4  0.499985
      ⋮
     10  0.500874
     11  0.498704
     12  0.50047ERROR: UndefVarError: `groups` not defined in `Main`
    Suggestion: check for spelling errors or missing imports.
    Stacktrace:
     [1] top-level scope
       @ none:1
julia> groupmeans = mean.(groups) # Take the monthly mean
┌ 12-element DimArray{Float64, 1} ┐
├─────────────────────────────────┴─────────────────────────────── dims ┐
@@ -295,13 +294,13 @@
`.-` rather than `-`. This is because the size of the arrays to not
match after application of `mean`.

```jldoctest groupby

Check failure on line 297 in src/groupby.jl

GitHub Actions / build 

doctest failure in ~/work/DimensionalData.jl/DimensionalData.jl/src/groupby.jl:297-299

```jldoctest groupby
julia> map(.-, groupby(A, Ti=>month), mean.(groupby(A, Ti=>month), dims=Ti));
```

Subexpression:

map(.-, groupby(A, Ti=>month), mean.(groupby(A, Ti=>month), dims=Ti));

Evaluated output:

ERROR: UndefVarError: `A` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ none:1

Expected output:




  diff = 
    Warning: Diff output requires color.
    ERROR: UndefVarError: `A` not defined in `Main`
    Suggestion: check for spelling errors or missing imports.
    Stacktrace:
     [1] top-level scope
       @ none:1
julia> map(.-, groupby(A, Ti=>month), mean.(groupby(A, Ti=>month), dims=Ti));
```

Or do something else with Y:

```jldoctest groupby

Check failure on line 303 in src/groupby.jl

GitHub Actions / build 

doctest failure in ~/work/DimensionalData.jl/DimensionalData.jl/src/groupby.jl:303-320

```jldoctest groupby
julia> groupmeans = mean.(groupby(A, Ti=>month, Y=>isodd))
┌ 12×2 DimArray{Float64, 2} ┐
├───────────────────────────┴────────────────────────────────────── dims ┐
  ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points,
  → Y  Sampled{Bool} [false, true] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────────── metadata ┤
  Dict{Symbol, Any} with 1 entry:
  :groupby => (:Ti=>month, :Y=>isodd)
└────────────────────────────────────────────────────────────────────────┘
  ↓ →  false         true
  1        0.499594     0.500533
  2        0.498145     0.501379
  ⋮
 10        0.501105     0.500644
 11        0.498606     0.498801
 12        0.501643     0.499298
```

Subexpression:

groupmeans = mean.(groupby(A, Ti=>month, Y=>isodd))

Evaluated output:

ERROR: UndefVarError: `A` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ none:1

Expected output:

┌ 12×2 DimArray{Float64, 2} ┐
├───────────────────────────┴────────────────────────────────────── dims ┐
  ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points,
  → Y  Sampled{Bool} [false, true] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────────── metadata ┤
  Dict{Symbol, Any} with 1 entry:
  :groupby => (:Ti=>month, :Y=>isodd)
└────────────────────────────────────────────────────────────────────────┘
  ↓ →  false         true
  1        0.499594     0.500533
  2        0.498145     0.501379
  ⋮
 10        0.501105     0.500644
 11        0.498606     0.498801
 12        0.501643     0.499298


  diff = 
    Warning: Diff output requires color.
    ┌ 12×2 DimArray{Float64, 2} ┐
    ├───────────────────────────┴────────────────────────────────────── dims ┐
      ↓ Ti Sampled{Int64} [1, 2, …, 11, 12] ForwardOrdered Irregular Points,
      → Y  Sampled{Bool} [false, true] ForwardOrdered Irregular Points
    ├────────────────────────────────────────────────────────────── metadata ┤
      Dict{Symbol, Any} with 1 entry:
      :groupby => (:Ti=>month, :Y=>isodd)
    └────────────────────────────────────────────────────────────────────────┘
      ↓ →  false         true
      1        0.499594     0.500533
      2        0.498145     0.501379
      ⋮
     10        0.501105     0.500644
     11        0.498606     0.498801
     12        0.501643     0.499298ERROR: UndefVarError: `A` not defined in `Main`
    Suggestion: check for spelling errors or missing imports.
    Stacktrace:
     [1] top-level scope
       @ none:1
julia> groupmeans = mean.(groupby(A, Ti=>month, Y=>isodd))
┌ 12×2 DimArray{Float64, 2} ┐
├───────────────────────────┴────────────────────────────────────── dims ┐
@@ -356,6 +355,7 @@
function _group_indices(dim::Dimension, f::Base.Callable; labels=nothing)
orig_lookup = lookup(dim)
k1 = f(first(orig_lookup))
# TODO: using a Dict here is a bit slow
indices_dict = Dict{typeof(k1),Vector{Int}}()
for (i, x) in enumerate(orig_lookup)
k = f(x)
@@ -447,11 +447,56 @@

Generate a `Vector` of `UnitRange` with length `step(A)`
"""
intervals(rng::AbstractRange) = IntervalSets.Interval{:closed,:open}.(rng, rng .+ step(rng))
intervals(rng::AbstractRange) =
IntervalSets.Interval{:closed,:open}.(rng, rng .+ step(rng))

"""
ranges(A::AbstractRange{<:Integer})

Generate a `Vector` of `UnitRange` with length `step(A)`
"""
ranges(rng::AbstractRange{<:Integer}) = map(x -> x:x+step(rng)-1, rng)

"""
combine(f::Function, gb::DimGroupByArray; dims=:)

Combine the `DimGroupByArray` using function `f` over the group dimensions.
Unlike broadcasting a reducing function over a `DimGroupByArray`, this function
always returns a new flattened `AbstractDimArray` even where not all dimensions
are reduced. It will also work over grouped `AbstractDimStack`.

If `dims` is given, it will combine only the dimensions in `dims`, the
others will be present in the final array. Note that all grouped dimensions
must be reduced and included in `dims`.

The reducing function `f` must also accept a `dims` keyword.

# Example

```jldoctest groupby
````
"""
function combine(f::Function, gb::DimGroupByArray{G}; dims=:) where G
targetdims = DD.commondims(first(gb), dims)
all(hasdim(first(gb), targetdims)) || throw(ArgumentError("dims must be a subset of the groupby dimensions"))
all(hasdim(targetdims, DD.dims(gb))) || throw(ArgumentError("grouped dimensions $(DD.basedims(gb)) must be included in dims"))
# This works for both arrays and stacks
# Combine the remaining dimensions after reduction and the group dimensions
destdims = (otherdims(DD.dims(first(gb)), dims)..., DD.dims(gb)...)
# Get the output eltype
T = Base.promote_op(f, G)
# Create a output array with the combined dimensions
dest = similar(first(gb), T, destdims)
for D in DimIndices(gb)
if all(hasdim(targetdims, DD.dims(first(gb))))
# Assigned reduced scalar to dest
dest[D...] = f(gb[D])
else
# Reduce with `f` and drop length 1 dimensions
xs = dropdims(f(gb[D]; dims); dims)
# Broadcast the reduced array to dest
broadcast_dims!(identity, view(dest, D...), xs)
end
end
return dest
end
26 changes: 12 additions & 14 deletions src/stack/indexing.jl
View file
Original file line number Diff line number Diff line change
@@ -150,6 +150,9 @@
end
end

@generated function _any_dimarray(v::Union{NamedTuple,Tuple})
any(T -> T <: AbstractDimArray, v.types)

Check warning on line 154 in src/stack/indexing.jl

Codecov / codecov/patch

src/stack/indexing.jl#L153-L154 

Added lines #L153 - L154 were not covered by tests
end

#### setindex ####
@propagate_inbounds Base.setindex!(s::AbstractDimStack, xs, I...; kw...) =
@@ -160,22 +163,17 @@
hassamedims(s) ? _map_setindex!(s, xs, i; kw...) : _setindex_mixed!(s, xs, i; kw...)
@propagate_inbounds Base.setindex!(s::AbstractDimStack, xs::NamedTuple, i::AbstractArray; kw...) =
hassamedims(s) ? _map_setindex!(s, xs, i; kw...) : _setindex_mixed!(s, xs, i; kw...)
@propagate_inbounds Base.setindex!(s::AbstractDimStack, xs::NamedTuple, i::DimensionIndsArrays; kw...) =

Check warning on line 166 in src/stack/indexing.jl

Codecov / codecov/patch

src/stack/indexing.jl#L166 

Added line #L166 was not covered by tests
_map_setindex!(s, xs, i; kw...)
@propagate_inbounds Base.setindex!(s::AbstractDimStack, xs::NamedTuple, I...; kw...) =
_map_setindex!(s, xs, I...; kw...)

@propagate_inbounds function Base.setindex!(
s::AbstractDimStack, xs::NamedTuple, I...; kw...
)
map((A, x) -> setindex!(A, x, I...; kw...), layers(s), xs)
end

_map_setindex!(s, xs, i; kw...) = map((A, x) -> setindex!(A, x, i...; kw...), layers(s), xs)
_map_setindex!(s, xs, i...; kw...) = map((A, x) -> setindex!(A, x, i...; kw...), layers(s), xs)

_setindex_mixed!(s::AbstractDimStack, x, i::AbstractArray) =
map(A -> setindex!(A, x, DimIndices(dims(s))[i]), layers(s))
_setindex_mixed!(s::AbstractDimStack, i::Integer) =
map(A -> setindex!(A, x, DimIndices(dims(s))[i]), layers(s))
function _setindex_mixed!(s::AbstractDimStack, x, i::Colon)
map(DimIndices(dims(s))) do D
map(A -> setindex!(A, D), x, layers(s))
function _setindex_mixed!(s::AbstractDimStack, xs::NamedTuple, i)
D = DimIndices(dims(s))[i]
map(layers(s), xs) do A, x
A[D] = x

Check warning on line 176 in src/stack/indexing.jl

Codecov / codecov/patch

src/stack/indexing.jl#L173-L176 

Added lines #L173 - L176 were not covered by tests
end
end

31 changes: 30 additions & 1 deletion src/stack/stack.jl
View file
Original file line number Diff line number Diff line change
@@ -153,7 +153,6 @@
Base.axes(s::AbstractDimStack) = map(first ∘ axes, dims(s))
Base.axes(s::AbstractDimStack, dims::DimOrDimType) = axes(s, dimnum(s, dims))
Base.axes(s::AbstractDimStack, dims::Integer) = axes(s)[dims]
Base.similar(s::AbstractDimStack, args...) = maplayers(A -> similar(A, args...), s)
Base.eltype(::AbstractDimStack{<:Any,T}) where T = T
Base.ndims(::AbstractDimStack{<:Any,<:Any,N}) where N = N
Base.CartesianIndices(s::AbstractDimStack) = CartesianIndices(dims(s))
@@ -197,6 +196,36 @@
@propagate_inbounds Base.iterate(st::AbstractDimStack, i) =
i > length(st) ? nothing : (st[DimIndices(st)[i]], i + 1)

Base.similar(s::AbstractDimStack) = similar(s, eltype(s))
Base.similar(s::AbstractDimStack, dims::Dimension...) = similar(s, dims)

Check warning on line 200 in src/stack/stack.jl

Codecov / codecov/patch

src/stack/stack.jl#L200 

Added line #L200 was not covered by tests
Base.similar(s::AbstractDimStack, ::Type{T},dims::Dimension...) where T =
similar(s, T, dims)
Base.similar(s::AbstractDimStack, dims::Tuple{Vararg{Dimension}}) =
similar(s, eltype(s), dims)
Base.similar(s::AbstractDimStack, ::Type{T}) where T =
similar(s, T, dims(s))
function Base.similar(s::AbstractDimStack, ::Type{T}, dims::Tuple) where T
# Any dims not in the stack are added to all layers
ods = otherdims(dims, DD.dims(s))
maplayers(s) do A
# Original layer dims are maintained, other dims are added
D = DD.commondims(dims, (DD.dims(A)..., ods...))
similar(A, T, D)
end
end
function Base.similar(s::AbstractDimStack, ::Type{T}, dims::Tuple) where T<:NamedTuple
ods = otherdims(dims, DD.dims(s))
maplayers(s, _nt_types(T)) do A, Tx
D = DD.commondims(dims, (DD.dims(A)..., ods...))
similar(A, Tx, D)
end
end

@generated function _nt_types(::Type{NamedTuple{K,T}}) where {K,T}
expr = Expr(:tuple, T.parameters...)
return :(NamedTuple{K}($expr))
end

# `merge` for AbstractDimStack and NamedTuple.
# One of the first three arguments must be an AbstractDimStack for dispatch to work.
Base.merge(s::AbstractDimStack) = s
Loading