update pub

Author: zhenweilin

_publications/2024-02-02-decentralized_gradient_free.md

@@ -6,4 +6,4 @@ date: 2024-02-02
 paperurl: 'https://ojs.aaai.org/index.php/AAAI/article/view/29697'
 ---
 
-We consider decentralized gradient-free optimization of minimizing Lipschitz continuous functions that satisfy neither smoothness nor convexity assumption. We propose two novel gradient-free algorithms, the Decentralized Gradient-Free Method (DGFM) and its variant, the Decentralized Gradient-Free Method+ (DGFM+). Based on the techniques of randomized smoothing and gradient tracking, DGFM requires the computation of the zeroth-order oracle of a single sample in each iteration, making it less demanding in terms of computational resources for individual computing nodes. Theoretically, DGFM achieves a complexity of $\mathcal{O}(d^{3/2}\delta^{-1}\epsilon^{-4})$ for obtaining a $(\delta,\epsilon)$-Goldstein stationary point. DGFM+, an advanced version of DGFM, incorporates variance reduction to further improve the convergence behavior. It samples a mini-batch at each iteration and periodically draws a larger batch of data, which improves the complexity to $\mathcal{O}(d^{3/2}\delta^{-1}\epsilon^{-3})$. Moreover, experimental results underscore the empirical advantages of our proposed algorithms when applied to real-world datasets.
+We consider decentralized gradient-free optimization of minimizing Lipschitz continuous functions that satisfy neither smoothness nor convexity assumption. We propose two novel gradient-free algorithms, the Decentralized Gradient-Free Method (DGFM) and its variant, the Decentralized Gradient-Free Method+ (DGFM+). Based on the techniques of randomized smoothing and gradient tracking, DGFM requires the computation of the zeroth-order oracle of a single sample in each iteration, making it less demanding in terms of computational resources for individual computing nodes. Theoretically, DGFM achieves a complexity of O(d³/²δ⁻¹ε⁻⁴) for obtaining a (δ,ε)-Goldstein stationary point. DGFM+, an advanced version of DGFM, incorporates variance reduction to further improve the convergence behavior. It samples a mini-batch at each iteration and periodically draws a larger batch of data, which improves the complexity to O(d³/²δ⁻¹ε⁻³). Moreover, experimental results underscore the empirical advantages of our proposed algorithms when applied to real-world datasets.

_publications/2024-10-20-apdpro.md

@@ -6,4 +6,4 @@ date: 2024-10-20
 paperurl: 'https://openreview.net/forum?id=pG380vLYRU'
 ---
 
-In this paper, we introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints. Prior to our work, the best complexity bound was $\mathcal{O}(1/{\epsilon})$, regardless of the strong convexity of the constraint function. It is unclear whether the strong convexity assumption can enable even better convergence results. To address this issue, we have developed novel techniques to progressively estimate the strong convexity of the Lagrangian function. Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of $\mathcal{O}(1/\sqrt{\epsilon})$. This rate matches the complexity lower bound for strongly-convex-concave saddle point optimization and is therefore order-optimal. We show the superior performance of our methods in sparsity-inducing constrained optimization, notably Google’s personalized PageRank problem. Furthermore, we show that a restarted version of the proposed methods can effectively identify the optimal solution’s sparsity pattern within a finite number of steps, a result that appears to have independent significance.
+In this paper, we introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints. Prior to our work, the best complexity bound was O(ε⁻¹), regardless of the strong convexity of the constraint function. It is unclear whether the strong convexity assumption can enable even better convergence results. To address this issue, we have developed novel techniques to progressively estimate the strong convexity of the Lagrangian function. Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of O(ε⁻¹/²). This rate matches the complexity lower bound for strongly-convex-concave saddle point optimization and is therefore order-optimal. We show the superior performance of our methods in sparsity-inducing constrained optimization, notably Google’s personalized PageRank problem. Furthermore, we show that a restarted version of the proposed methods can effectively identify the optimal solution’s sparsity pattern within a finite number of steps, a result that appears to have independent significance.

_publications/2024-12-09-bundle-constraint.md

@@ -1,22 +1,21 @@
 ---
 title: "Uniformly Optimal and Parameter-free First-order Methods for Convex and Function-constrained Optimization"
 collection: publications
-category: Preprint
+category: preprint
 date: 2024-12-09
 paperurl: 'https://arxiv.org/pdf/2412.06319'
 ---
 
 This paper presents new first-order methods for achieving optimal oracle complexities in convex optimization with convex functional constraints. Oracle complexities are measured by the number of function
 and gradient evaluations. To achieve this, we enable first-order methods to utilize computational oracles
-for solving diagonal quadratic programs in subproblems. For problems where the optimal value $f^*$
-is
+for solving diagonal quadratic programs in subproblems. For problems where the optimal value f* is
 known, such as those in overparameterized models and feasibility problems, we propose an accelerated
-first-order method that incorporates a modified Polyak step size and Nesterov’s momentum. Notably, our
+first-order method that incorporates a modified Polyak step size and Nesterov's momentum. Notably, our
 method does not require knowledge of smoothness levels, Hölder continuity parameter of the gradient, or
-additional line search, yet achieves the optimal oracle complexity bound of $\mathcal{O}(\epsilon^{-\frac{2}{1+\rho}})$ under Hölder
-smoothness conditions. When $f^*$ is unknown, we reformulate the problem as finding the root of the optimal value function and develop inexact fixed-point iteration and secant method to compute $f^*$. These
+additional line search, yet achieves the optimal oracle complexity bound of O(ε⁻²/⁽¹⁺ᵖ⁾) under Hölder
+smoothness conditions. When f* is unknown, we reformulate the problem as finding the root of the optimal value function and develop inexact fixed-point iteration and secant method to compute f*. These
 root-finding subproblems are solved inexactly using first-order methods to a specified relative accuracy.
 We employ the accelerated prox-level (APL) method, which is proven to be uniformly optimal for convex optimization with simple constraints. Our analysis demonstrates that APL-based level-set methods
-also achieve the optimal oracle complexity of $\mathcal{O}(\epsilon^{-\frac{2}{1+\rho}})$ for convex function-constrained optimization, without requiring knowledge of any problem-specific structures. Through experiments on various
+also achieve the optimal oracle complexity of O(ε⁻²/⁽¹⁺ᵖ⁾) for convex function-constrained optimization, without requiring knowledge of any problem-specific structures. Through experiments on various
 tasks, we demonstrate the advantages of our methods over existing approaches in function-constrained
-optimization.
+optimization.