Conference Agenda

Classical optimization theory cannot explain the success of optimization in machine learning. We motivate the assumption for the cost function to be the realization of a random function, discuss machine learning phenomena and heuristics explained by the random function framework and outline the benefits and future of this Bayesian approach to optimization.

The goal of this work is to justify the application of regression models for counterfactual reasoning in the sense of Judea Pearl. Let $X$ and $Y$ be two random variables with joint distribution $\pi$, and assume that $\pi$ can be described with an additive noise model $$ Y := \tilde{f}(X) + \tilde{U}, $$ where $\tilde{f}$ is a continuous function and $\tilde{U}$ is a random variable with mean value $E[\tilde{U}] = 0$ that is independent of $X$. Generally, it is not feasible to reason about the effects of external interventions or counterfactuals solely from the joint distribution $\pi$. Previous work, however, shows that, in most cases, there exists no additive noise model of the form $X := \tilde{g}(Y) + \tilde{V}$ describing $\pi$, establishing an asymmetry that allows us to identify $X$ as the cause of $Y$ and to predict effects from external interventions, such as setting $X$ to a specific value $x$. In my work, I further demonstrate that the joint distribution $\pi$ uniquely determines the function $ \tilde{f}$. I then argue that this also uniquely determines all counterfactual probabilities.

For context, humans reason in terms of counterfactuals, meaning they consider how events might have unfolded under different circumstances without experiencing those alternative realities. For instance, an economist might judge, "If the government had reduced the tax rate by 5%, we would have seen at least a 2% increase in economic growth," without observing the scenario in which the government reduced the tax rate. Since this capability is fundamental to understanding the past and reasoning about ethical notions such as responsibility, blame, fairness, or harm, it is also desirable to augment current machine learning models with counterfactual reasoning. In this case, the economist might use this counterfactual statement to attribute responsibility for a current economic crisis to the government.

Sticking with the above example, $X$ could denote the tax rate and $Y$ could denote economic growth to illustrate the main ideas behind counterfactual reasoning and my work. Following Pearl's framework, I assume that the situation of interest can be captured by a functional causal model. In this case, the influence of $X$ on $Y$ can be represented by the equation: $$ Y := f(X, U), $$ where $f$ is the causal mechanism and $U$ is a random variable with a distribution $\pi_{U}$, representing hidden or unmodeled influences.

Suppose we observe an economic growth of 0% and a tax rate of 30%. The distribution of $U$ can be updated according to the evidence $Y = 0$ and $X = 30$. This updated distribution for the error term $U$ and the modified model $$ Y := f(25, U) $$ then allows us to derive a distribution for the economic growth that would have taken place if the government had reduced the tax rate by 5%. This approach lets us quantify our degree of belief that the government is responsible for the observed recession.

In practice, this approach to counterfactual reasoning requires fitting a regression model $Y := \tilde{f}(X, \tilde{U})$ and providing justification as to why this regression model is the correct causal explanation for the observed distribution $\pi$. However, it is known that neither the causal direction nor the causal mechanism is generally uniquely determined by the joint distribution $\pi$. Therefore, in my work, I assume that the true functional causal model is also an additive noise model as well. I hope this assumption could be justified by Occam's razor in future studies. In our example, this means I assume that $$ Y := f(X) + U $$ for a function $f$ and a real-valued error term $U$ with mean value $E[U] = 0$ that is independent of $X$.

Further, assume that the distributions of $X$ and $U$ are given by strictly positive densities and that $f \in L^2(\pi_X)$ is continuous and square integrable over $\pi_X$. In this case, I am able to prove that the function $f$ is also uniquely determined by the joint distribution $\pi$ of $X$ and $Y$. From this fact, I further conclude that the functions $f$ and $\tilde{f}$ coincide. In particular, this means that the assumed regression model $Y := \tilde{f} (X) + \tilde{U}$ is the correct causal model, which can then be used to compute counterfactual probabilities. I therefore propose this result as a step towards using regression techniques from machine learning for data-based counterfactual reasoning.

Analogue to the well-known Langevin Monte Carlo method we provide a method to sample from a target distribution \(\pi\) by simulating a solution of a stochastic differential equation. Hereby, the stochastic differential equation is driven by a general Lévy process which - other than in the case of Langevin Monte Carlo - allows for non-smooth targets. Our method will be fully explored in the particular setting of target distributions supported on the half-line (0,∞) and a compound Poisson driving noise. Several illustrative examples demonstrate the usefullness of this method.