Lightweight soft topic assignment via temperature-scaled embedding distances

[pidefrem]

Feature request

Add a soft_clustering_temp parameter to transform() that computes soft topic distributions using temperature-scaled softmax on embedding distances to topic centroids:

# Hard assignment (current default)
topics, probs = topic_model.transform(new_docs)

# Soft assignment with temperature scaling
topics, probs = topic_model.transform(new_docs, soft_clustering_temp=0.5)
# probs is now a (n_docs, n_topics) matrix of soft assignments

Motivation

Getting soft (probabilistic) topic assignments currently requires either:

1. calculate_probabilities=True — expensive for large datasets and HDBSCAN-specific
2. approximate_distribution() — good but requires re-running the vectorizer

Users frequently ask about soft/overlapping topic assignments:

• #1419 — non-mutually exclusive topics
• #2017 — probability inconsistency
• #1613 — negative probabilities
• #1808 — probabilities NoneType
• #1962 — probabilities empty with zero-shot

The temperature-scaled approach is clustering-agnostic (works with any clustering model, not just HDBSCAN), lightweight (no vectorizer, no refit), and provides a smooth probability distribution over all topics.

Your contribution

I can submit a PR that adds soft_clustering_temp to transform(). Low temperature → sharper (near-hard) assignments; high temperature → softer distributions. Default is None — existing behavior unchanged.

I've already been prototyping this in my fork (working implementation with tests). Since this adds a new parameter, I'd value your steer on the API (name, whether it lives on transform() or a dedicated method) before I open the PR.

---