11th Applied Inverse Problems Conference
September 4 - 8, 2023 | Göttingen, Germany
Conference Agenda
Overview and details of the sessions of this conference. Please select a date or location to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).
|
|
|
Session Overview | |
| Location: VG0.111 |
| Date: Monday, 04/Sept/2023 | |
| 1:30pm - 3:30pm | MS25 1: Hyperparameter estimation in imaging inverse problems: recent advances on optimisation-based, learning and statistical approaches Location: VG0.111 Session Chair: Luca Calatroni Session Chair: Monica Pragliola |
|
|
Automatic Differentiation of Fixed-Point Algorithms for Structured Non-smooth Optimization Saarland University, Germany
A large class of non-smooth practical optimization problems can be written as minimization of a sum of smooth and partly smooth functions. We consider structured non-smooth optimization problems which also depend on a parameter vector and study the problem of differentiating its solution mapping with respect to the parameter which has far reaching applications in sensitivity analysis and parameter learning. We show that under partial smoothness and other mild assumptions, Automatic Differentiation (AD) of the sequence generated by proximal splitting algorithms converges to the derivative of the solution mapping. For a variant of automatic differentiation, which we call Fixed-Point Automatic Differentiation (FPAD), we remedy the memory overhead problem of the Reverse Mode AD and moreover provide faster convergence theoretically. We numerically illustrate the convergence and convergence rates of AD and FPAD on Lasso and Group Lasso problems and demonstrate the working of FPAD on prototypical practical image denoising problem by learning the regularization term.
Learning data-driven priors for image reconstruction: From bilevel optimisation to neural network-based unrolled schemes 1Queen Mary University of London, United Kingdom; 2Physikalisch-Technische Bundesanstalt, Germany; 3Humboldt-Universität zu Berlin; 4Technische Universität Berlin, Germany; 5Finden Ltd, Rutherford Appleton Laboratory, United Kingdom; 6Weierstrass Institute for Applied Analysis and Stochastics, Germany
Combining classical model-based variational methods for image reconstruction with deep learning techniques has attracted a significant amount of attention during the last years. The aim is to combine the interpretability and the reconstruction guarantees of a model-based method with the flexibility and the state-of-the-art reconstruction performance that the deep neural networks are capable of achieving. We introduce a general novel image reconstruction approach that achieves such a combination which we motivate by recent developments in deeply learned algorithm unrolling and data-driven regularisation as well as by bilevel optimisation schemes for regularisation parameter estimation. We consider a network consisting of two parts: The first part uses a highly expressive deep convolutional neural network (CNN) to estimate a spatially varying (and temporally varying for dynamic problems) regularisation parameter for a classical variational problem (e.g. Total Variation). The resulting parameter is fed to the second sub-network which unrolls a finite number of iterations of a method which solves the variational problem (e.g. PDHG). The overall network is then trained end-to-end in a supervised fashion. This results to an entirely interpretable algorithm since the “black-box” nature of the CNN is placed entirely on the regularisation parameter and not to the image itself. We prove consistency of the unrolled scheme by showing that, as the number of unrolled iterations tends to infinity, the unrolled energy functional used for the supervised learning $\Gamma$-converges to the corresponding functional that incorporates the exact solution map of the TV-minimization problem. We also provide a series of numerical examples that show the applicability of our approach: dynamic MRI reconstruction, quantitative MRI reconstruction, low-dose CT and dynamic image denoising.
Learned proximal operators in accelerated unfolded methods with pseudodifferential operators 1University of Bologna; 2University of Bath
In recent years, hybrid reconstruction frameworks has been proposed by unfolding iterative methods and learning a suitable pseudodifferential correction on the part that can provably not be handled by model-based methods. In particular, the inner hyperameters of the method are estimated by using supervised learning techniques.
In this talk, I will present a variant of this approach, where an accelerated iterative algorithm is unfolded and the proximal operator is replaced by a learned operators, as in the PnP framework. The numerical experiments on limited-angle CT achieve promising results.
Masked and Unmasked Principles for Automatic Parameter Selection in Variational Image Restoration for Poisson Noise Corruption 1University of Bologna, Italy; 2University of Naples Federico II, Italy
Due to the statistical nature of electromagnetic waves, Poisson noise is a widespread cause of data degradation in many inverse imaging problems. It arises whenever the acquired data is formed by counting the number of photons irradiated by a source and hitting the image domain. Poisson noise removal is a crucial issue typical in astronomical and medical imaging, where the scenarios are characterized by a low photon count. For the former case, this is related to the acquisition set-up, while in the latter it is desirable to irradiate the patient with lower electromagnetic doses in order to keep it safer. However, the weaker the light intensity, the stronger the Poisson noise degradation in the acquired images and the more difficult the reconstruction problem.
An effective model-based approach for reconstructing images corrupted by Poisson noise is the use of variational methods. Despite the successful results, their performance strongly depends on the selection of the regularization parameter that balances the effect of the regularization term and the data fidelity term. One of the most used approaches for choosing the parameter is the discrepancy principle proposed in [1] that relies on imposing that the data term is equal to its approximate expected values. It works well for mid- and high-photon counting scenarios but leads to poor results for low-count Poisson noise. The talk will address novel parameter
selection strategies that outperform the state-of-the-art discrepancy principles in [1], especially for low-count regime. The approaches are based on decreasing the approximation error in [1] by means of a suitable Montecarlo simulation [2], on applying a so-called Poisson whiteness principle [3] and on cleverly masking the data used for the parameter selection [4], respectively. Extensive experiments are presented which prove the effectiveness of the three novel methods.
[1] M. Bertero, P. Boccacci, G. Talenti, R. Zanella, L. Zanni. A discrepancy principle for Poisson data, Inverse Problems 26(10), 2010. [105004]
[2] F. Bevilacqua, A. Lanza, M. Pragliola, F. Sgallari. Nearly exact discrepancy principle for low-count Poisson image restoration, Journal of Imaging 8(1): 1-35, 2022.
[3] F. Bevilacqua, A. Lanza, M. Pragliola, F. Sgallari. Whiteness-based parameter selection for Poisson data in variational image processing, Applied Mathematical Modelling 117: 197-218, 2023.
[4] F. Bevilacqua, A. Lanza, M. Pragliola, F. Sgallari. Masked unbiased principles for parameter selection in variational image restoration under Poisson noise, Inverse Problems 39(3), 2023. [034002]
|
| Date: Tuesday, 05/Sept/2023 | |
| 4:00pm - 6:00pm | MS45 1: Optimal Transport meets Inverse Problems Location: VG0.111 Session Chair: Marcello Carioni Session Chair: Jan-F. Pietschmann Session Chair: Matthias Schlottbom |
|
|
Efficient adversarial regularization for inverse problems 1University of Bath, United Kingdom; 2University of Twente, Netherlands; 3KTH - Royal Institute of Technology, Sweden; 4University of Cambridge, United Kingdom
We propose a new optimal transport-based approach for learning end-to-end reconstruction operators using unpaired training data for ill-posed inverse problems. The key idea behind the proposed method is to minimize a weighted combination of the expected distortion in the measurement space and the Wasserstein-1 distance between the distributions of the reconstruction and the ground truth. The regularizer is parametrized by a deep neural network and learned simultaneously with an unrolled reconstruction operator in an adversarial training framework. The variational problem is then initialized with the output of the reconstruction network and solved iteratively till convergence. Notably, it takes significantly fewer iterations to converge as compared to variational methods, thanks to the excellent initialization obtained via the unrolled reconstruction operator. The resulting approach combines the computational efficiency of end-to-end unrolled reconstruction with the well-posedness and noise-stability guarantees of the variational setting. We demonstrate with the example of image reconstruction in X-ray computed tomography (CT) that our approach outperforms state-of-the-art unsupervised methods and that it outperforms or is at least on par with state-of-the-art supervised data-driven CT reconstruction approaches.
Data Driven Gradient Flows 1Universität Augsburg, Germany; 2UT Twente, Netherlands
We present a framework enabling variational data assimilation for gradient flows in general metric spaces, based on the minimizing movement (or Jordan-Kinderlehrer-Otto) approximation scheme. After discussing stability properties in the most general case, we specialise to the space of probability measures endowed with the Wasserstein distance. This setting covers many non-linear partial differential equations (PDEs), such as the porous medium equation or general drift-diffusion-aggregation equations, which can be treated by our methods independent of their respective properties (such as finite speed of propagation or blow-up). We then focus on the numerical implementation of our approach using an primal-dual algorithm. The strength of our approach lies in the fact that by simply changing the driving functional, a wide range of PDEs can be treated without the need to adopt the numerical scheme. We conclude by presenting detailed numerical examples.
The quadratic Wasserstein metric for inverse data matching 1The University of Texas at Austin, USA; 2Columbia University, USA; 3Cornell University, USA
This work characterizes, analytically and numerically, two major effects of the quadratic Wasserstein ($W_2$) distance as the measure of data discrepancy in computational solutions of inverse problems. First, we show, in the infinite-dimensional setup, that the $W_2$ distance has a smoothing effect on the inversion process, making it robust against high-frequency noise in the data but leading to a reduced resolution for the reconstructed objects at a given noise level. Second, we demonstrate that, for some finite-dimensional problems, the $W_2$ distance leads to optimization problems that have better convexity than the classical $L^2$ and $\dot{H}^{-1}$ distances, making it a more preferred distance to use when solving such inverse matching problems. This talk is based on the work [1].
[1] B. Engquist, K. Ren, Y. Yang. The quadratic Wasserstein metric for inverse data matching, Inverse Problems 36(5): 055001, 2020.
Quadratic regularization of optimal transport problems TU Braunschweig, Germany
In this talk we consider regularization of optimal transport problems with quadratic terms. We use the Kantorovich for of optimal transport and add a quadratic regularizer, which forces the transport plan to be a square integrable function instead of a general measure. We derive the dual problem and show strong duality and existence of primal and dual solutions to the regularized problem. Then we derive two algorithms to solve the dual problem of the regularized problem: A Gauss-Seidel method and a semismooth quasi-Newton method and investigate both methods numerically. Our experiments show that the methods perform well even for small regularization parameters. Quadratic regularization is of interest since the resulting optimal transport plans are sparse, i.e. they have a small support (which is not the case for the often used entropic regularization where the optimal transport plan always has full measure). Finally we briefly sketch an extension of the results to the more general case of regularization with so-called Young functions which unifies the entropic and the quadratic regularization.
|
| Date: Wednesday, 06/Sept/2023 | |
| 9:00am - 11:00am | MS45 2: Optimal Transport meets Inverse Problems Location: VG0.111 Session Chair: Marcello Carioni Session Chair: Jan-F. Pietschmann Session Chair: Matthias Schlottbom |
|
|
Inverse problems in imaging and information fusion via structured multimarginal optimal transport 1KTH Royal Institute of Technology; 2Georgia Institute of Technology; 3Aalto University; 4École polytechnique fédérale de Lausanne; 5Chalmers University of Technology and the University of Gothenburg
The optimal mass transport problem is a classical problem in mathematics, and dates back to 1781 and work by G. Monge where he formulated an optimization problem for minimizing the cost of transporting soil for construction of forts and roads. Historically the optimal mass transport problem has been widely used in economics in, e.g., planning and logistics, and was at the heart of the 1975 Nobel Memorial Prize in Economic Sciences. In the last two decades there has been a rapid development of theory and methods for optimal mass transport and the ideas have attracted considerable attention in several economic and engineering fields. These developments have led to a mature framework for optimal mass transport with computationally efficient algorithms that can be used to address problems in the many areas.
In this talk, we will consider optimization problems consisting of optimal transport costs
together with other functionals to address inverse problems in many domains, e.g., in medical imaging, radar imaging, and spectral estimation. This is a flexible framework and allows for incorporating forward models, specifying dynamics of the object and other dependencies. These problem can often be formulated as a multi-marginal optimal transport problem and we show how common problems, such as barycenter and tracking problems, can be seen as special cases of this. This naturally leads to consider structured optimal transport problems, which can be solved efficiently using customized methods inspired by the Sinkhorn iterations.
Wasserstein PDE G-CNN TU Eindhoven, Netherlands, The
PDE GCNNs are neural networks where each layer is seen as a set of PDE-solvers where geometrically meaningful PDE-coefficients become the layer’s trainable weights. In this talk, we present a contribution on building new layers that are based either on Wasserstein gradient flows or on normalizing measures that take inspiration from optimal transport maps. The tunable parameters are either connected to parameters of the gradient flow, or the transport maps, so the whole procedure can be interpreted as an inverse problem.
An Optimal Transport-based approach to Total-Variation regularization for the Diffusion MRI problem 1Karl-Franzens-Universität Graz, Austria; 2GE Global Research, Munich, Germany; 3Technische Universität Braunschweig, Germany; 4Technische Universität München, Germany
Diffusion Magnetic Resonance Imaging (dMRI) is a non-invasive imaging technique that draws structural information from the interaction between water molecules and biological tissues. Common ways of tackling the derived inverse problem include, among others, Diffusion Tensor Imaging (DTI), High Angular Resolution Diffusion Imaging (HARDI) and Diffusion Spectrum Imaging (DSI). However, these methods are structurally unable to recover the full diffusion distribution, only providing partial information about particle displacement. In our work, we introduce a Total-Variation (TV) regularization defined from an optimal transport perspective using 1-Wasserstein distances. Such a formulation produces a variational problem that can be handled by well-known algorithms enjoying good convergence properties, such as the primal-dual proximal method by Chambolle and Pock. It allows for the reconstruction of the complete diffusion spectrum from measured undersampled k/q space data.
A game-based approach to learn interaction rules for systems of rational agents 1University of Verona, Italy; 2Technical University of Munich, Germany; 3University of Göttingen, Germany
The modelling of the dynamic of a system of rational agents may take inspiration from various sources, depending on the particular application one has in mind. We can consider for example to model interactions via a Newtonian-like system, taking inspiration from physics, or via a game-based approach stemming from classical game theory or mean field games. In both cases, once we ensured the well-posedness of the proposed model, the model itself can be used as a tool to learn from real world observations, by means of learning (some) unknown components of it.
In [1], the authors study a class of spatially inhomogeneous evolutionary games to model the interactions between a finite number of agents: each agent evolves in space with a velocity which depends on a certain underlying mixed strategy, in turn evolving according to a replicator dynamic. In this talk we move from such a formulation, and introduce an entropic limiting version of it, which boils down to a purely spatial ODE. For a bounded set of pure strategies $U \subset \mathbb{R}^u$, $0 < \eta \in P(U)$ a probability measure on $U$, an ''entropic'' parameter $\varepsilon>0$, and maps $e \colon \mathbb{R}^d \times U \to \mathbb{R}$ and $J \colon \mathbb{R}^d \times U \times \mathbb{R}^d \to \mathbb{R}$, the $N$-agents system we consider is the following:
$$
\begin{aligned}
\partial_t x_i(t) &= v_i^J(x_1(t),\dots,x_N(t)) \quad \text{for } i = 1,\dots,N\\
v_i^J(x_1,\dots,x_N) &= \int_U e(x_i,u)\, \sigma_{i}^J(x_1,\ldots,x_N)(u)\,\,{d} \eta(u)\\
\sigma_{i}^J(x_1,\ldots,x_N) &= \frac{\exp\left(\tfrac{1}{\varepsilon N}\sum_{j=1}^N J(x_i,\cdot,x_j)\right)}{
\int_U \exp\left(\tfrac{1}{\varepsilon N}\sum_{j=1}^N J(x_i,v,x_j)\right)\,\,{d} \eta(v)}. \\
\end{aligned}
$$
We study the well-posedness and the mean field limit of such a system, and use it as the backbone of a learning procedure. In particular, we focus on the learnability of the interaction kernel $J$, all the rest given. Building on ideas of [3, 4, 5], we infer $J$ by penalizing the empirical mean squared error between observed velocities and predicted velocities, and also consider the choice of penalizing observed mixed strategies and predicted mixed strategies. We study the quality of the inferred kernel both as $N$ increases (i.e., as we have observations of an increasingly high number of agents) and in the limit of repeated observations with fixed $N$ (i.e., as we have repeated observations of the same number of agents). We show the effectiveness of the proposed inference on many different examples, from classical Newtonian systems to system modelling pedestrian dynamics.
[1] L. Ambrosio, M. Fornasier, M. Morandotti, G. Savaré. Spatially inhomogeneous evolutionary games, Communications on Pure and Applied Mathematics 74.7: 1353-1402, 2021.
[2] M. Bonafini, M. Fornasier, B. Schmitzer. Data-driven entropic spatially inhomogeneous evolutionary games, European Journal of Applied Mathematics 34.1: 106-159, 2023.
[3] M. Bongini, M. Fornasier, M. Hansen, M. Maggioni. Inferring interaction rules from observations of evolutive systems I: The variational approach, Mathematical Models and Methods in Applied Sciences 27.05: 909-951, 2016.
[4] F. Cucker, S. Smale. On the mathematical foundations of learning, Bulletin of the American mathematical society 39.1: 1-49, 2002.
[5] F. Lu, M. Maggioni, S. Tang. Learning interaction kernels in heterogeneous systems of agents from multiple trajectories, The Journal of Machine Learning Research 22.1: 1518-1584, 2021.
|
| Date: Thursday, 07/Sept/2023 | |
| 1:30pm - 3:30pm | MS16 1: Wave propagation and quantitative tomography Location: VG0.111 Session Chair: Leonidas Mindrinos Session Chair: Leopold Veselka |
|
|
Phase-contrast THz-CT for non-destructive testing 1Johann Radon Institue Linz, Austria; 2Johannes Kepler University Linz, Austria
In this talk, we consider the imaging problem of THz computed tomography (THz-CT), in particular for the non-destructive testing of extruded plastic profiles. We derive a general nonlinear mathematical model describing a full THz tomography experiment, and consider several approximations connecting THz tomography with standard computerized tomography and the Radon transform. The employed models are based on geometrical optics, and contain both the THz signal amplitude and the phase. We consider several reconstruction approaches using the corresponding phase-contrast sinograms, and compare them both qualitiatively and quantitatively on experimental data obtained from 3D printed plastic profiles which were scanned with a THz time-domain spectrometer in transmission geometry.
Diffraction tomography for a generalized incident beam wave University of Vienna, Austria
The mathematical imaging problem of diffraction tomography is an
inverse scattering technique used to find the material properties of an object. Here,
the object is exposed to a certain form of radiation and the scattered wave is
recorded. In conventional diffraction tomography, the incident wave is assumed
to be a monochromatic plane wave arriving from a fixed direction of propagation.
However, this plane wave excitation does not necessarily correspond to measurement
setups used in practice: There, the size of the emitting device is limited and therefore
cannot produce plane waves. Besides, it is common to emit focused beams to achieve
a better resolution in the far field. In this talk, I will present our recent results that
allow diffraction tomography to be applied to these realistic illumination scenarios.
We use a new forward model, that incorporates individually generated incident
fields. Based on this, a new reconstruction algorithm is developed.
Bias-free localizations in cryo-single molecule localization microscopy Johannes Kepler University Linz, Austria
Single molecule localization microscopy (SMLM) has the potential to resolve structural details of biological samples at the nanometer length scale. Compared to room temperature experiments, SMLM performed under cryogenic temperature achieves higher photon yields and, hence, higher localization precision. However, to fully exploit the resolution it is crucial to account for the anisotropic emission characteristics of fluorescence dipole emitters with fixed orientation. In this talk, I will present recent advances along this avenue.
Uncertainty-aware blob detection in astronomical imaging 1University of Vienna, Austria; 2Max-Planck Institut für Astronomie, Germany; 3Instituto de Astrofisica de Canarias, Spain; 4Universidad Central de Chile, Chile; 5Space Telescope Science Institute, USA; 6Johann Radon Institute for Computational and Applied Mathematics, Linz, Austria; 7Christian Doppler Laboratory for Mathematical Modeling and Simulation of Next Generations of Ultrasound Devices, Vienna, Austria
Blob detection, i. e. detection of blob-like shapes in an image, is a common problem in astronomy. A difficulty arises when the image of interest has to be recovered from noisy measurements, and thus comes with uncertainties. Formulating the reconstruction of the image as a Bayesian inverse problem, we propose an uncertainty-aware version of the classic Laplacian-of-Gaussians method for blob detection. It combines ideas from scale-space theory, statistics and variational regularization to identify salient blobs in uncertain images. The proposed method is illustrated on a problem from stellar dynamics: the identification of components in a stellar distribution recovered from integrated-light spectra. This talk is based on our recent preprint [1].
[1] F. Parzer, P. Jethwa, A. Boecker, M. Alfaro-Cuello, O. Scherzer, G. van de Ven. Uncertainty-Aware Blob Detection with an Application to Integrated-Light Stellar Population Recoveries, arXiv:2208.05881, 2022.
|
| 4:00pm - 6:00pm | MS16 2: Wave propagation and quantitative tomography Location: VG0.111 Session Chair: Leonidas Mindrinos Session Chair: Leopold Veselka |
|
|
Source Reconstruction from Partial Boundary Data in Radiative Transport Johann Radon Institute (RICAM), Austria
This talk concerns the source reconstruction problem in a transport problem through an absorbing and scattering medium from boundary measurement data on an arc of the boundary. The method, specific to two dimensional domains, relies on Bukgheim’s theory of A-analytic maps and it is joint work with A. Tamasan (UCF) and H. Fujiwara (Kyoto U).
Solving Cauchy problems using semi-discretization techniques and BIE Agricultural University of Athens, Greece
In this work we present a two-step method for the numerical solution of parabolic and hyperbolic Cauchy problems. Both problems are formulated in 2D and the proposed method is considered for the direct and the corresponding inverse problem. The main idea is to combine a semi-discretization with respect to the time variable with THE boundary integral equation method for the spatial variables. The time discretization reduces the problem to a sequence of elliptic stationary problems. The solution is represented using a single-layer ansatz and then we end up solving iteratively for the unknown boundary density functions. We solve the discretized problem on the boundary of the medium with the collocation method. Classical quadrature rules are applied for handling the kernel singularities. We present numerical results for different linear PDEs.
This is a joint work with R. Chapko (Ivan Franko University of Lviv, Ukraine)
and B. T. Johansson (Linköping University, Sweden).
Quantitative Parameter Reconstruction from Optical Coherence Tomographic Data 1University of Vienna, Austria; 2Medical University of Vienna, Austria
Optical Coherence Tomography (OCT), an imaging modality based on the interferometric measurement of back-scattered light, is known for its high-resolution images of biological tissues and its versatility in medical imaging. Especially in its main field of application, ophthalmology, the continuously increasing interest in OCT, aside from improving image quality, has driven the need for quantitative information, like optical properties, for a better medical diagnosis.
In this talk, we discuss the quantification of the refractive index, an optical property which describes the change of wavelength between different materials, from OCT data. The presented method is based on a Gaussian beam forward model, resembling the strongly focused laser light typically used within an OCT setup. Samples with layered structure are considered, meaning that the refractive index as a
function of depth is well approximated by a piece-wise constant function. For the reconstruction, a layer-by-layer method is presented where in every step the refractive index is obtained via a discretized $L^2$−minimization. The applicability of the proposed method is then verified by reconstructing refractive indices of layered media from both simulated and experimental OCT data.
Augmented total variation regularization in imaging inverse problems 1Mathematics Research Center, Academy of Athens, Athens, Greece; 2Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK; 3Institute of Communication and Computer Systems, National Technical University of Athens, Athens, Greece; 4Department of Mathematics, National Technical University of Athens, Athens, Greece
Total variation ($TV$) regularization has been extensively employed in inverse problems in imaging. In this talk, we propose a new method of $TV$ regularization for medical image reconstruction, which extends standard regularization approaches. Our novel method may be conceived as an augmented version of typical $TV$ regularization. Within this approach, a new monitoring variable, $\omega(x)$, is introduced via an additional term in the minimization functional. The integration in this term is performed with respect to the $TV$ measure, corresponding to the deviation of the image, $u(x)$. The dual function $\omega(x)$ is the integrand of the additional term, and its smoothing nature compensates, when necessary, for the abruptness of the $TV$ measure of the image. It is within this dual variable that the regularity is imposed via the minimization process itself. The main purpose of the dual variable is to control the behavior of $u(x)$, especially regarding its discontinuity properties. Our preliminary results indicate the fast convergence rate of our method, thus highlighting its promising potential. This research is partially supported by the Horizon Europe project SEPTON, under grant agreement 101094901.
|
| Date: Friday, 08/Sept/2023 | |
| 1:30pm - 3:30pm | MS13 1: Stochastic iterative methods for inverse problems Location: VG0.111 Session Chair: Tim Jahn |
|
|
Beating the Saturation of the Stochastic Gradient Descent for Linear Inverse Problems 1The Chinese University of Hong Kong; 2Rutgers University, United States of America
Stochastic gradient descent (SGD) is a promising method for solving large-scale inverse problems, due to its excellent scalability with respect to data size. The current mathematical theory in the lens of regularization theory predicts that SGD with a polynomially decaying stepsize schedule may suffer from an undesirable saturation phenomenon, i.e., the convergence rate does not further improve with the solution regularity index when it is beyond a certain range. In this talk, I will present our recent results on beating this saturation phenomenon:
(i) By using a small initial step size. We derive a refined convergence rate analysis of SGD, which shows that saturation does not occur if the initial stepsize of the schedule is sufficiently small.
(ii) By using Stochastic variance reduced gradient (SVRG), a popular variance reduction technique for SGD. We prove that, with a suitable constant step size schedule, SVRG can achieve an optimal convergence rate in terms of the noise level (under suitable regularity conditions), which means the saturation does not occur.
Early stopping of untrained convolutional networks 1University of Bonn, Germany; 2Chinese University of Hong Kong
In recent years new regularisation methods based on neural networks have shown promising performance for the solution of ill-posed problems, e.g., in imaging science. Due to the non-linearity of the networks, these methods often lack profound theoretical justification. In this talk we rigorously discuss convergence for an untrained convolutional network. Untrained networks are particulary attractive for applications, since they do not require any training data. Its regularising property is solely based on the architecture of the network. Because of this, appropriate early stopping is essential for the success of the method. We show that the discrepancy principle is an adequate method for early stopping here, as it yields minimax optimal convergence rates.
Stochastic mirror descent method for linear ill-posed problems in Banach spaces The Australian National University, Australia
Consider linear ill-posed problems governed by the system $A_i x = y_i$ for $i= 1,\cdots, p$, where each $A_i$ is a bounded linear operator from a Banach space $X$ to a Hilbert space $Y_i$. In case p is huge, solving the problem by an iterative regularization method using the whole information at each iteration step can be very expensive, due to the huge amount of memory and excessive computational load per iteration. To solve such large-scale ill-posed systems efficiently, we develop a stochastic mirror descent method which uses only a small portion of equations randomly selected at each iteration steps and incorporates convex regularization terms into the algorithm design. Therefore, our method scales very well with the problem size and has the capability of capturing features of sought solutions. The convergence property of the method depends crucially on the choice of
step-sizes. We consider various rules for choosing step-sizes and obtain convergence results under a priori stopping rules. Furthermore, we establish an order optimal convergence rate result when the sought solution satisfies a benchmark source condition. Various numerical simulations are reported to test the performance of the method. This is a joint work with Xiliang Lu and Liuying Zhang.
Early stopping for spectral filter estimators regularized by projection Paris 1 Panthéon-Sorbonne University, France
When using iterative algorithms such as gradient descent, a classical problem is the choice of the number of iterations to perform in practice. This is a crucial question since the number of iterations determines the final statistical performance of the resulting estimator.
The main purpose of the present talk is to design such data-criven stopping rules called "early stopping rules" (ESR) that will answer the above question not only for gradient descent but also for the broader class of spectral filter estimators among which ridge regression for instance.
Compared to previous works in this direction, the present contribution focuses on the computational issue raised by the use of spectral filter estimators in the context of a huge amount of data. In particular this requires the additional use of regularization by projection techniques for efficiently computing (approximations to) the spectral filter estimators.
In this talk we develop a theoretical analysis of the behavior of these projection-based spectral filter (PSF) estimators. Oracle inequalities also quantify the performance of the data-driven early stopping rule applied to these PSF estimators.
|
| 4:00pm - 6:00pm | MS13 2: Stochastic iterative methods for inverse problems Location: VG0.111 Session Chair: Tim Jahn |
|
|
From inexact optimization to learning via gradient concentration Bocconi University Milano, Italy
Optimization in machine learning typically deals with the minimization of empirical objectives defined by training data. The ultimate goal of learning, however, is to minimize the error on future data (test error), for which the training data provides only partial information. In this view, the optimization problems that are practically feasible are based on inexact quantities that are stochastic in nature. In this paper, we show how probabilistic results, specifically gradient concentration, can be combined with results from inexact optimization to derive sharp test error guarantees. By considering unconstrained objectives, we highlight the implicit regularization properties of optimization for learning.
Principal component analysis in infinite dimensions Universität Bielefeld, Germany
In high-dimensional settings, principal component analysis (PCA) reveals some unexpected phenomena, ranging from eigenvector inconsistency to eigenvalue (upward) bias. While such high-dimensional phenomena are now well understood in the spiked covariance model, the goal of this talk is to present some extensions for the case of PCA in infinite dimensions. As an application, we present bounds for the prediction error of spectral regularization estimators in the overparametrized regime.
Learning Linear Operators TU Braunschweig, Germany
We consider the problem of learning a linear operator $\theta$ between two Hilbert spaces from empirical observations, which we interpret as least squares regression in infinite dimensions. We show that this goal can be reformulated as an inverse problem for $\theta$ with the undesirable feature that its forward operator is generally non-compact (even if $\theta$ is assumed to be compact or of p-Schatten class). However, we prove that, in terms of spectral properties and regularisation theory, this inverse problem is equivalent to the known compact inverse problem associated with scalar response regression. Our framework allows for the elegant derivation of dimension-free rates for generic learning algorithms under Hölder-type source conditions. The proofs rely on the combination of techniques from kernel regression with recent results on concentration of measure for sub-exponential Hilbertian random variables. The obtained rates hold for a variety of practically-relevant scenarios in functional regression as well as nonlinear regression with operator-valued kernels and match those of classical kernel regression with scalar response.
SGD for select inverse problems in Banach spaces 1University College London; 2The Chinese University of Hong Kong,
In this work we present a mathematical framework and analysis for SGD in Banach spaces for select linear and non-linear inverse problems. Analysis
in the Banach space setting presents unique challenges, requiring novel mathematical tools. This is achieved by combining insights from Hilbert space theory with approaches from modern optimisation. The developed theory and algorithms open doors for a wide range of applications, and we present some future challenges and directions.
|
Contact and Legal Notice · Contact Address:  Privacy Statement · Conference: AIP 2023 |
Conference Software: ConfTool Pro 2.8.101+TC © 2001–2024 by Dr. H. Weinreich, Hamburg, Germany |
