Remove selector/space stuff #2458

mhauru · 2025-01-08T15:59:12Z

This PR aims to remove all use of the Selector/Gibbs ID/space mechanisms that the old Gibbs sampler relied on. Methods for functions like getspace will still be defined for compatibility with the current DPPL version, but will not do anything. The goal is that after this has been merged a new release of DPPL could be made that removes the same mechanisms from VarInfo without anything breaking on the Turing.jl side.

coveralls · 2025-01-08T16:36:32Z

Pull Request Test Coverage Report for Build 12714062015

Details

47 of 58 (81.03%) changed or added relevant lines in 9 files are covered.
121 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-7.8%) to 77.2%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/mcmc/mh.jl	13	15	86.67%
src/mcmc/hmc.jl	0	9	0.0%

Files with Coverage Reduction	New Missed Lines	%
src/mcmc/Inference.jl	1	86.5%
src/mcmc/repeat_sampler.jl	1	83.33%
src/mcmc/mh.jl	3	84.8%
src/mcmc/hmc.jl	116	0.0%

Totals
Change from base Build 12675082674:	-7.8%
Covered Lines:	1158
Relevant Lines:	1500

💛 - Coveralls

…an-up

codecov · 2025-01-08T18:42:59Z

Codecov Report

Attention: Patch coverage is 96.55172% with 2 lines in your changes missing coverage. Please review.

Please upload report for BASE (breaking@b97334b). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/mcmc/mh.jl	86.66%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             breaking    #2458   +/-   ##
===========================================
  Coverage            ?   85.01%           
===========================================
  Files               ?       21           
  Lines               ?     1495           
  Branches            ?        0           
===========================================
  Hits                ?     1271           
  Misses              ?      224           
  Partials            ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

penelopeysm · 2025-01-09T17:27:26Z

I feel like intuitively I'd have removed it from DPPL first, is there a particular reason you started here?

mhauru · 2025-01-10T09:54:57Z

I feel like intuitively I'd have removed it from DPPL first, is there a particular reason you started here?

My thinking was to first stop using a feature of a dependency, then remove that feature. It doesn't make much of a difference, but I thought it might help make incremental changes. For instance, when making changes to DPPL, I should be able to test that Turing.jl tests pass with the new DPPL, without modifying Turing.jl code.

penelopeysm

I suppose that the selector features in DPPL are almost purely motivated by this, so from that perspective it does make sense to me to remove this first. It's just a bit awkward with the testing machinery because you'd have to manually run the Turing testsuite with a dev'd version of DPPL..(?) Also, CI won't run the Turing integration tests.

Anyway, just some tiny tidyup suggestions below.

src/mcmc/particle_mcmc.jl

penelopeysm · 2025-01-10T11:28:24Z

src/mcmc/particle_mcmc.jl

+        unset_flag!(vi, vn, "del") # Reference particle parent
+        r = rand(trng, dist)
+        vi[vn] = DynamicPPL.tovec(r)
+        DynamicPPL.setgid!(vi, spl.selector, vn)


Is this line needed? (Specifically the setgid one)

Hmm, nice catch. I hope not. It shouldn't? I tried removing it, and a bunch of other calls to setgid! and updategid!, let's see what the test suite says.

src/mcmc/sghmc.jl

penelopeysm · 2025-01-10T11:45:06Z

src/mcmc/mh.jl

+            elseif length(proposals) == 1
+                # If we hit this block, check to see if it's
+                # a run-of-the-mill proposal or covariance
+                # matrix.
+                prop = proposal(s)
+
+                # Return early, we got a covariance matrix.
+                return new{typeof(prop)}(prop)


This feels confusing to me, it could probably be handled better with a new method, but maybe a case for another time too.

Is it the early return that confuses you? It confuses me. It's been there for at least ~5 years though, and I just left it be for now.

Yeah, that and the call to proposal(), which is a bit opaque. But it's definitely out of scope for this PR.

Co-authored-by: Penelope Yong <[email protected]>

penelopeysm

Happy to merge!

mhauru · 2025-01-13T11:44:26Z

Thanks @penelopeysm! Let's hold off for a while, since 0.36 hasn't even been released yet.

mhauru · 2025-02-14T17:21:39Z

Changed the base branch to breaking to collect multiple breaking changes into a single branch/PR.

* Bump minor version to v0.37 * Remove selector/space stuff (#2458) * Remove selector stuff from ESS * Remove selector stuff from MH * Remove selector stuff from HMC * Remove selector stuff from Emcee * Remove selector stuff from IS * Add missing getspace methods * Remove selector stuff for particle methods * Fix an HMC selector bug * Code style * Fix Emcee selector bug * Fix typo in ESS tests * Fix some constructor overwrites * Remove unnecessary tests * Remove selector stuff from SGHMC * Remove drop_space and other non-longer-necessary deprecation measures * Bump minor version 0.37. Add a HISTORY.md entry * Apply suggestions from code review Co-authored-by: Penelope Yong <[email protected]> * Remove unnecessary type parameters Co-authored-by: Penelope Yong <[email protected]> * Simplify constructors in particle_mcmc.jl * Remove calls to setgid and updategid --------- Co-authored-by: Penelope Yong <[email protected]> * Bump Mooncake compat to 0.4.95 * Support for DynamicPPL v0.35 (#2488) * Progress towards compat with DPPL v0.35 * More fixing of DPPL v0.35 stuff * Fix LogDensityFunction argument order * More minor bugfixes * [TEMP] Commit Manifest pointing to DynamicPPL#release-0.35 * remove LogDensityProblemsAD (#2490) * Remove LogDensityProblemsAD, part 1 * update Optimisation code to not use LogDensityProblemsAD * Fix field name change * Don't put chunksize=0 * Remove LogDensityProblemsAD dep * Improve OptimLogDensity docstring * Remove unneeded model argument to _optimize * Fix more tests * Remove essential/ad from the list of CI groups * Fix HMC function * more test fixes (#2491) * Remove LogDensityProblemsAD, part 1 * update Optimisation code to not use LogDensityProblemsAD * Fix field name change * Don't put chunksize=0 * Remove LogDensityProblemsAD dep * Improve OptimLogDensity docstring * Remove unneeded model argument to _optimize * Fix more tests * Remove essential/ad from the list of CI groups * Fix HMC function * More test fixes * Remove Manifest * More fixes for DynamicPPL 0.35 (#2494) * Remove test/dynamicppl/compiler.jl * Remove old regression tests * Remove vdemo2 * Fix last test * Add HISTORY.md entry about DPPL 0.35 * Allow ESS to sample variables with different symbols * Update a TODO note --------- Co-authored-by: Penelope Yong <[email protected]> * Fix call to DynamicPPL.initialize_parameters!! * Remove `Zygote` (#2505) * Remove `Zygote`; fix #2504 * Update test/test_utils/ad_utils.jl Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add HISTORY.md entry about removing support for Zygote --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Markus Hauru <[email protected]> Co-authored-by: Penelope Yong <[email protected]> Co-authored-by: Markus Hauru <[email protected]> * Fix a Gibbs test * Clean up exports (#2474) * Regroup exports by package * Export DynamicPPL.returned and DynamicPPL.prefix * Stop exporting @logprob_str and @prob_str * Remove DynamicPPL module export * Remove DynamicPPL.LogDensityFunction re-export * Remove BernoulliLogit, drop support for Distributions < 0.25.77 * Stop blanket re-exporting Libtask and Bijectors * Manually specify AbstractMCMC exports * Format Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Remove Bijectors.ordered export * Re-export LinearAlgebra.I * Replace Turing.Model -> DynamicPPL.Model * Format * Keep exporting LogDensityFunction * Add note in docs * Align Turing exports with docs API page * Fix things like `predict` on docs API page * Fix a Gibbs test * Format * Fix missing Bijectors import * Update docs/src/api.md Co-authored-by: Markus Hauru <[email protected]> * Update docs/src/api.md Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Clean up broken docs links, remove unneeded deps * Format * Add changelog entry for exports * Clean up exports in essential/Essential * Apply suggestions from code review Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Markus Hauru <[email protected]> Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Penelope Yong <[email protected]> Co-authored-by: Hong Ge <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

mhauru added 9 commits January 8, 2025 15:14

Remove selector stuff from ESS

c8b17ea

Remove selector stuff from MH

c62d35d

Remove selector stuff from HMC

2ac1d9f

Remove selector stuff from Emcee

e6e4e56

Remove selector stuff from IS

d1840aa

Add missing getspace methods

0bebd3e

Remove selector stuff for particle methods

eb90d4c

Fix an HMC selector bug

54df711

Code style

8af52ea

mhauru added 6 commits January 8, 2025 17:01

Fix Emcee selector bug

890bc29

Merge remote-tracking branch 'origin/master' into mhauru/selector-cle…

ef03e08

…an-up

Fix typo in ESS tests

8ec521e

Fix some constructor overwrites

ed3081c

Remove unnecessary tests

934ebef

Remove selector stuff from SGHMC

02f251d

mhauru added 2 commits January 9, 2025 15:04

Remove drop_space and other non-longer-necessary deprecation measures

6fde350

Bump minor version 0.37. Add a HISTORY.md entry

fb868fe

mhauru marked this pull request as ready for review January 9, 2025 17:05

mhauru requested a review from penelopeysm January 9, 2025 17:06

penelopeysm reviewed Jan 10, 2025

View reviewed changes

mhauru and others added 4 commits January 10, 2025 16:54

Apply suggestions from code review

091d114

Co-authored-by: Penelope Yong <[email protected]>

Remove unnecessary type parameters

9a0dee6

Co-authored-by: Penelope Yong <[email protected]>

Simplify constructors in particle_mcmc.jl

bd78c89

Remove calls to setgid and updategid

d6ba522

penelopeysm approved these changes Jan 13, 2025

View reviewed changes

mhauru mentioned this pull request Jan 21, 2025

Clean up exports #2474

Merged

mhauru changed the base branch from master to breaking February 14, 2025 17:20

Merge branch 'breaking' into mhauru/selector-clean-up

ab00808

mhauru merged commit 08e4c08 into breaking Feb 14, 2025
55 of 57 checks passed

mhauru deleted the mhauru/selector-clean-up branch February 14, 2025 17:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove selector/space stuff #2458

Remove selector/space stuff #2458

mhauru commented Jan 8, 2025

coveralls commented Jan 8, 2025 •

edited

Loading

codecov bot commented Jan 8, 2025 •

edited

Loading

penelopeysm commented Jan 9, 2025

mhauru commented Jan 10, 2025

penelopeysm left a comment •

edited

Loading

penelopeysm Jan 10, 2025 •

edited

Loading

mhauru Jan 10, 2025

penelopeysm Jan 10, 2025

mhauru Jan 10, 2025

penelopeysm Jan 10, 2025

penelopeysm left a comment

mhauru commented Jan 13, 2025

mhauru commented Feb 14, 2025

Remove selector/space stuff #2458

Remove selector/space stuff #2458

Conversation

mhauru commented Jan 8, 2025

coveralls commented Jan 8, 2025 • edited Loading

Pull Request Test Coverage Report for Build 12714062015

Details

💛 - Coveralls

codecov bot commented Jan 8, 2025 • edited Loading

Codecov Report

penelopeysm commented Jan 9, 2025

mhauru commented Jan 10, 2025

penelopeysm left a comment • edited Loading

Choose a reason for hiding this comment

penelopeysm Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

mhauru Jan 10, 2025

Choose a reason for hiding this comment

penelopeysm Jan 10, 2025

Choose a reason for hiding this comment

mhauru Jan 10, 2025

Choose a reason for hiding this comment

penelopeysm Jan 10, 2025

Choose a reason for hiding this comment

penelopeysm left a comment

Choose a reason for hiding this comment

mhauru commented Jan 13, 2025

mhauru commented Feb 14, 2025

coveralls commented Jan 8, 2025 •

edited

Loading

codecov bot commented Jan 8, 2025 •

edited

Loading

penelopeysm left a comment •

edited

Loading

penelopeysm Jan 10, 2025 •

edited

Loading