Information Geometry Optimization

You are handed a function $f:X\to \mathbb{R}$ and told: “minimize it. You may query $f(x)$ for any $x$ you choose, but you get nothing else — no gradients, no Hessians, no structure.” This is black-box optimization: hyperparameter tuning of a model, calibration of a simulator, policy search in reinforcement learning, a physics or chemistry experiment that takes minutes per evaluation. The budget of calls is precious; the landscape may be non-convex, non-smooth, noisy, high-dimensional, and rotated in ways you don't know.

If you have no gradients, the oldest robust idea is randomized search: keep a distribution $P_{\theta }$ over candidate solutions, sample a batch, evaluate the batch, and move the distribution so that good regions become more likely and bad regions less likely. Iterate. The meta-algorithm is tiny:

Everything hangs on that one box. The request for a rigorous answer is: given the current distribution $P_{\theta }$ and a batch of evaluations, what is the right direction to move $\theta$, and why that one and not another?

Naive answers fail in instructive ways. Gradient ascent on the expected fitness ${\mathbb{E}}_{x\sim P_{\theta }}[f(x)]$ w.r.t. a raw coordinate $\theta$ (so-called vanilla gradient) depends on how you wrote down the parameters — mean-and-variance or mean-and-standard-deviation describe the very same Gaussians but give different search trajectories. Worse, plain fitness expectations are hostage to the absolute scale of $f$, so rescaling $f\mapsto f^3$ changes the algorithm's whole behavior even though the ranking of points is identical. The field had found partial fixes heuristically (rank-based weights, covariance-adapting search distributions), but the picture was fragmented until information-geometric optimization (IGO) [Ollivier, Arnold, Auger & Hansen, JMLR 2017] unified it: IGO makes box 4 fully canonical by combining two old ideas — rank-based selection and the natural gradient — and showing that everything else follows from demanding invariance.

The two questions IGO answers, once and for all:

• What objective should $\theta$ climb? Not $f$ itself, but an adaptive, quantile-transformed version of $f$ that only cares about relative rank among the current batch. (Section 2.)
• In what geometry should it climb? Not the Euclidean geometry of your coordinate system, but the Fisher information geometry of the family of distributions $P_{\theta }$ — where distance is measured in how much the sampling distribution actually changes. (Sections 3–4.)

Fix a family $\{P_{\theta }{\}}_{\theta \in \Theta }$ of distributions on $X$ — for now imagine Gaussians on ${\mathbb{R}}^d$ with $\theta =(\mu ,C)$, or Bernoulli measures on $\{0,1{\}}^d$ with $\theta =({\theta }_1,\dots ,{\theta }_d)$. The optimization state is the distribution: ${\theta }^t$ encodes “where we currently believe good solutions live.” We want a deterministic rule — the IGO flow — that says how ${\theta }^t$ should evolve in continuous time, and then a sampling-based Euler discretization of it, the IGO algorithm.

2.1 Step one: replace $f$ by its adaptive quantile transform

A transfer of $f$ onto parameter space by plain expectation, $F(\theta )={\mathbb{E}}_{P_{\theta }}f$, is fragile: it changes under monotone rescaling of $f$, and is dominated by extreme values. IGO instead scores each point by where it falls in the distribution of $f$-values under the current distribution. Define the quality

where $w:[0,1]\to \mathbb{R}$ is any non-increasing selection scheme. The argument of $w$ is the $f$-quantile of $x$: the probability that a random sample from the current distribution is strictly better. Typical choices of $w$: truncation $w(u)=\frac{1}{q}{1}_{u\le q}$ (keep the top $q$ fraction, weight them equally), or soft log weights like xNES's $u_i\propto \max (0,\ln (\frac{\lambda }{2}+1)-\ln i)$ (smoothly favor the best). Because $W$ only looks at quantiles, it is invariant under any strictly increasing transform of $f$: $f$, $f^3$, or $f\times |f{|}^{-2/3}$ all produce the same $W$. (One technicality, which matters on discrete search spaces: if $f$ has atoms, the probability $\mathrm{Pr}(f(x^{\prime })<f(x))$ no longer pins down the quantile — many $x$'s tie — and the paper defines $W$ as the average of $w$ over the interval $[\mathrm{Pr}(f(x^{\prime })<f(x)),\mathrm{Pr}(f(x^{\prime })\le f(x))]$. For continuous $f$ the two endpoints coincide and the formula above is recovered.)

2.2 Step two: climb the expected quality with the natural gradient

Now transfer $W$ to parameter space. At time $t$ we freeze the transform at ${\theta }^t$ and define the parametrized objective

and ask: which direction $d\theta$ increases $g_t$ the most relative to how much the distribution actually changes? Holding the metric question for Section 3, suppose we use the natural gradient ${\tilde{\nabla }}_{\theta }g=I(\theta {)}^{-1}{\nabla }_{\theta }g$ (Fisher matrix $I$, defined next section). Two algebra moves turn this into something computable from samples only. First, the log-derivative trick (${\nabla }_{\theta }{P}_{\theta }=P_{\theta }{\nabla }_{\theta }\ln P_{\theta }$):

Second, the identity $\mathbb{E}[{\nabla }_{\theta }\ln P_{\theta }]=0$ lets us cancel any constant baseline in $W$ — so only the relative quality matters, not its absolute level. The flow is then [Ollivier et al. 2017, Def. 4]:

Reading the flow is pleasant: the update is a weighted average of intrinsic moves ${\tilde{\nabla }}_{\theta }\ln P_{\theta }(x)$ — each sample pulls the parameters a little toward “making $x$ more likely”, and the pull is weighted by how good $x$ is relative to the current batch. It is literally a gradient ascent on the weighted log-likelihood of the good points:

2.3 From flow to algorithm: Euler step + Monte Carlo

Two discretizations turn the flow into code. Replace the integral by a sample average over $\lambda$ points drawn from $P_{{\theta }^t}$, and replace the exact quantile $W$ by the empirical one: rank the batch, and give the $i$-th best point the weight

Note that nothing forces ${\int }_0^{1}w=1$ or ${\sum }_i{\hat{w}}_i=1$: multiplying $w$ by a constant just multiplies the whole update by that constant, i.e. rescales the effective learning rate — a pure convention to be absorbed into $\eta$. (The linear-function analysis in Section 5.4 deliberately uses the unnormalized indicator $w(u)=1_{u\le q_0}$.) Stepping the flow by an Euler step of size $\eta$ (the learning rate; the paper writes $\delta t$) gives the IGO algorithm [Def. 5]:

That's the whole algorithm. Everything else in this document is about why this is the right direction, how different families $P_{\theta }$ specialize it into famous algorithms, and what can be proven about it.

There is a rigorous consistency statement behind the Monte Carlo step: as $\lambda \to \infty$, the sampled update converges almost surely to one Euler step of the flow [Thm. 6]. This is not a trivial law-of-large-numbers fact, because the weights ${\hat{w}}_i$ depend on the whole batch through the ranks — the proof is a Glivenko–Cantelli argument. It means: any rank-based update of this form is secretly following a well-defined continuous-time flow.

Why did we smuggle the Fisher matrix $I(\theta )$ into the update, and what does it buy? The one-paragraph answer: parameter space is not Euclidean. Two different values of $\theta$ can describe nearly identical or wildly different sampling distributions; how “far apart” two $\theta$'s are should be measured by how different the distributions are, not by a coordinate distance. Information geometry gives the canonical ruler for that: the Fisher metric [Rao 1945; Amari 1998].

3.1 The local Taylor expansion of KL divergence

The Kullback–Leibler divergence between two nearby members of the family, expanded in $\delta \theta$, has no linear term and a quadratic term governed by $I(\theta )$:

So $I(\theta )$ is (twice) the Hessian of KL divergence at $\theta$, and it turns $\Theta$ into a Riemannian manifold: the distance between nearby distributions is $\sqrt{2\mathrm{KL}}$. Crucially, KL divergence depends only on the distributions $P_{\theta }$, not on how we named them — so this metric is intrinsic, identical under any reparametrization of the family.

The natural gradient is simply the gradient as measured by this metric — the steepest-ascent direction when “small move” means “small distribution change” (the derivation is the standard one: the steepest ascent of $g$ under a metric $A$ is $A^{-1}\nabla g$, because $g(y+\epsilon z)=g(y)+\epsilon ⟨\nabla g,z{⟩}_A+O({\epsilon }^2)$):

3.2 Why the Fisher metric is the right ruler for search

Three distinct arguments converge on Fisher geometry as the natural metric for optimizer updates:

• The shortest path uphill: gradient ascent under a metric $A$ realizes the maximal increase of $g$ per unit of $A$-distance [Ollivier et al. 2017, §2.1]. Taking $A=I(\theta )$ makes “distance” measure how much the sampling distribution changes — which is the only distance that matters for a search that lives in distribution space.
• Minimal change, maximal diversity retention: a step of the IGO flow of size $\eta$ is the movement of $P_{\theta }$ that increases the weighted log-likelihood the most subject to staying within KL distance proportional to ${\eta }^2$ [Thm. 10; the update is the argmax of $\int \ln P_{\theta }(1+\eta W)P_{{\theta }^t}$ minus an KL penalty]. A consequence: IGO provably maximizes the gain per unit of information spent, which keeps the distribution spread out — it does not greedily collapse onto the current best point. The paper shows the speed of the flow is bounded by the standard deviation of $w$ (Prop. 20), and quantifies the step size as an amount of Fisher information (bits) injected per iteration [§C.2, Cor. 21].
• Invariance: only the natural gradient transforms covariantly under reparametrization — Section 4.

There is a classical way to see the last point that is worth carrying around: the one-step optimality

i.e. the natural gradient step is exactly the KL-penalized maximizer (a proximal view of natural gradient — the same structure that later showed up in TRPO-style trust-region RL and in mirror descent). So IGO's flow is the infinitesimal version of “improve the distribution as much as possible without moving too far in KL”.

Here is the deeper reason IGO insists on quantiles and natural gradients: both are invariance requirements. An optimizer is a mapping from problems to trajectories; if two presentations of the “same” problem differ only by a rename that carries no information — different coordinate axes, different units, different parameter encoding, cubed fitness values — the algorithm should behave identically. IGO attains the maximum possible set of such invariances: reparametrization of the family, affine re-coordinatization of the search space, and strictly increasing transformation of $f$ [Props. 17–19].

4.1 Invariance under reparametrization of $\theta$ (Proposition 18)

Consider the family of one-dimensional Gaussians. You may parametrize it as $(\mu ,\sigma )$ (mean, standard deviation), as $(\mu ,{\sigma }^2)$ (variance), or as $(\mu ,\ln \sigma )$ (log-scale) — the same distributions, different numberings. A plain gradient ascent of $g(\theta )$ in these three charts runs three different algorithms: the direction ${\nabla }_{\theta }g$ changes with the chart because the metric “unit length” means different things in each chart.

The natural gradient does not suffer this: under a change of chart $\varphi =\varphi (\theta )$ with Jacobian $J=\partial \varphi /\partial \theta$, one has $I_{\varphi }=J^{-\mathsf{T}}{I}_{\theta }{J}^{-1}$ and ${\nabla }_{\varphi }g=J^{-\mathsf{T}}{\nabla }_{\theta }g$, so

which is exactly the transformation rule for a vector field under a change of coordinates: the same tangent direction on the manifold of distributions, re-expressed in the new chart. Natural gradient ascent is a coordinate-free differential equation on the statistical manifold; vanilla gradient ascent is not.

4.2 Invariance under affine changes of the search space (Proposition 19)

If you rotate, translate or scale the coordinate system of $X$ — solve $f(Ax+b)$ instead of $f(x)$ — a Gaussian family transforms along: $N(\mu ,A{A}^{\mathsf{T}})$ maps to Gaussians again. The IGO flow is invariant under exactly those transformations of the problem that the family can absorb (for Gaussians: all affine maps). This is the theoretical engine behind the well-known practical fact that CMA-ES is indifferent to coordinate rotations and scalings — performance transfers from one ill-conditioned problem to the whole equivalence class. (For the finite-$\eta$ algorithms this invariance holds approximately; the flow — the $\eta \to 0$ limit — is exactly invariant.)

4.3 Invariance under monotone transforms of $f$ (Proposition 17)

Quantile-based selection (Section 2.1) makes the quality $W_{\theta }^f$ depend only on the order of $f$-values, so replacing $f$ by $\phi \circ f$ for any strictly increasing $\phi$ leaves every step unchanged. Why this matters: it is what makes IGO “feel” like a second-order method on rugged functions — a cliff and a gentle slope impose the same pressure if they induce the same ordering — and it is exactly the property earlier natural-gradient methods (NES, xNES) had to bolt on heuristically with “fitness shaping”; IGO gets it from the construction [§2.2.2].

The choice of family is the one genuinely free design decision. The workhorse is the Gaussian family on ${\mathbb{R}}^d$, $P_{\theta }=\mathcal{N}(\mu ,C)$ with $C=A{A}^{\mathsf{T}}$. IGO on Gaussians turns out to be the exponential natural evolution strategy (xNES) [Glasmachers, Schaul, Sun, Wierstra & Schmidhuber, GECCO 2010], and a version of CMA-ES falls out of the same framework — the paper's central “unifying picture” claim [§5.2]. This is where the abstract recipe becomes concrete and visibly powerful, so the derivation is worth doing carefully — $d=1$ first, then the general matrix case.

5.1 One-dimensional Gaussians: the natural-gradient machinery in miniature

Take the family $\mathcal{N}(\mu ,{\sigma }^2)$ and the “log-scale” chart $(\mu ,\tilde{\sigma })$ with $\tilde{\sigma }=\ln \sigma$. The log-density and its scores, with $s=(x-\mu )/\sigma$ the standardized deviation, are

and the Fisher matrix is the constant-block matrix $I=\mathrm{diag}(1/{\sigma }^2,2)$. So the natural gradient is trivial to write: the $I^{-1}$ contraction absorbs the $(s/\sigma )$ into $\sigma \cdot s$, and the $\tilde{\sigma }$-block contributes a plain factor $\frac{1}{2}$.

Notice what the Fisher contraction bought: the mean step uses the standardized deviations $s_i$, i.e. the update is proportional to $\sigma$ (a move of size $\sigma$ in $x$-space), and the variance step is multiplicative (it rescales $\sigma$, never crosses past zero). Both properties are exactly what a scale-invariant search needs. Also note the sign structure: selected points with $|s_i|<1$ (close to $\mu$) push $\sigma$ down; points with $|s_i|>1$ push it up — the distribution widens along directions where the best points are spread out, and tightens where they cluster. That sentence is the entire secret of covariance adaptation.

5.2 The general case: Gaussians with a full covariance matrix

Now let $P_{\theta }=\mathcal{N}(\mu ,C)$ with $C=A{A}^{\mathsf{T}}$ and $A$ an invertible square root (say Cholesky). Write $s=A^{-1}(x-\mu )$, so that $\ell =-\frac{d}{2}\ln 2\pi -\ln |\det A|-\frac{1}{2}\Vert s{\Vert }^2$. Two gradient computations, keeping track of the matrix structure:

(for the second line: $\mathrm{d}\ln |\det A|=\mathrm{tr}(A^{-1}\mathrm{d}A)$ and the quadratic-form variation $\mathrm{d}\frac{1}{2}\Vert s{\Vert }^2=-\mathrm{tr}(A^{-1}\mathrm{d}As{s}^{\mathsf{T}})$.) The Fisher matrix splits into a mean block $I_{\mu }=\mathbb{E}[\frac{\partial \ell }{\partial \mu }{\frac{\partial \ell }{\partial \mu }}^{\mathsf{T}}]=C^{-1}$ and a shape block, and its inverse contracts the updates into two “natural” statistics of the standardized samples:

The key trick (xNES's “natural coordinate system”): because every sample is standardized by the current $A$, the Fisher matrix in these local coordinates is the identity — the natural gradient is just the weighted sum, with no matrix inversion at all. The exponential update keeps $C=A{A}^{\mathsf{T}}$ positive definite for any symmetric $G_M$ and makes the update multiplicative and hence invariant under linear re-coordinatizations of the search space. Splitting $A=\sigma B$ with $\det B=1$ reproduces xNES's three-chart version — a global step size $\sigma$ and a shape matrix $B$:

5.3 Why this behaves like a second-order method (and where CMA-ES fits)

Watch what $A$ does on an ill-conditioned, rotated quadratic. Samples spread along the valley get large $|s|$ in the valley direction; the covariance ellipse rotates to cover the valley and tightens across it — the search distribution literally learns the local metric of the function. This is the same effect that makes quasi-Newton methods (BFGS & co.) fast, obtained with function values only, through the magic of the Fisher contraction: the natural gradient on a Gaussian family acts like a Hessian pre-conditioner [§4]. It is also exactly why evolution strategies, despite having no gradient, converge linearly (geometrically) on well-scaled problems.

CMA-ES — the workhorse of black-box continuous optimization since the 1990s — turns out, in hindsight, to be the same machinery [§5.2]: its mean update is precisely the weighted mean-step above; its covariance update is the first-order expansion $\exp (\frac{{\eta }_A}{2}{G}_M)\approx I+\frac{{\eta }_A}{2}{G}_M$ (the additive rank-one $+$ rank-$\mu$ update), with two additions that matter in practice: an evolution path (an exponential moving average of the mean steps, giving the rank-one term correlated directions across generations) and cumulative step-size adaptation (CSA) for $\sigma$. The xNES authors put it sharply: “we deduced update rules very similar to the diverse rules found in CMA-ES from the single principle of natural gradient ascent,” retroactively justifying CMA-ES's heuristics [§4.4]. Conversely, the IGO theory explains why CMA's rank-based weights work: for large populations they follow the same gradient flow as IGO (Theorem 6).

There is also a fast endpoint of the IGO-Gaussian flow worth knowing: taking the full maximum-likelihood step (learning rate “1”, i.e. re-estimating $\mu$ and $C$ directly from the weighted sample) yields EMNA (Estimation of Multivariate Normal Algorithm) — and the IGO analysis shows exactly why it underperforms: a full-size ML step loses diversity too fast (it is the $\eta \to \infty$ limit of a minimal-change process). Small steps are not a fudge; they are the mechanism that keeps the search exploring [§2.2.4, 5.2].

5.4 A completely solved case: IGO on a linear function (and the “select less than half” law)

Some flows can be integrated in closed form, and the linear case is the Rosetta stone of evolutionary computation. Minimize $f(x)=x_1$ with truncation selection keeping the top $q_0$ fraction (weights $1_{u\le q_0}$). Because of invariance, this one case explains all linear functions. Using the 1-D flow equations above ($Z\sim \mathcal{N}(0,1)$, $a={\Phi }^{-1}(q_0)$, $\phi ,\Phi$ the standard normal density/cdf):

Read the laws. If you keep fewer than half of the samples ($q_0<\frac{1}{2}$), then $a<0$, $\alpha >0$: the step size grows exponentially while the mean streaks downhill at the same exponential rate — the search is expanding, confident, and correct. If you keep more than half ($q_0\ge \frac{1}{2}$), then $\alpha \le 0$: $\sigma$ collapses toward zero and the mean stalls — premature convergence, visible here as a theorem, not a rumor. This is the information-geometric restatement of the classic “success-pressure” laws of evolution strategies (the 1/5-rule's cousin).

5.5 The flagship in action (interactive)

The abstract's headline claim is that IGO “recover[s] versions of known algorithms”. Each famous method is just the same update (Section 2.3) applied to a different family $P_{\theta }$. Two more cases are worth seeing mathematically before the zoo table, because the algebra is short and illuminating.

6.1 Bernoulli → PBIL and cGA (with one line of algebra)

Search space $X=\{0,1{\}}^d$, family of Bernoulli measures $P_{\theta }(x)={\prod }_i{\theta }_i^{x_i}(1-{\theta }_i{)}^{1-x_i}$. The score w.r.t. ${\theta }_i$ is $x_i/{\theta }_i-(1-x_i)/(1-{\theta }_i)$, and the (diagonal) Fisher entry is $1/({\theta }_i(1-{\theta }_i))$. Contracting with $I^{-1}$ cancels the denominators:

Each bit probability drifts toward the weighted average of the selected bits — exactly the population-based incremental learning (PBIL) update [Baluja 1994; Harik, Lobo & Goldberg 1999]. With a population of two and weights $+\frac{1}{2},-\frac{1}{2}$ this becomes the compact genetic algorithm (cGA), which compares best vs. worst sample. The un-pleasant $1/(\theta (1-\theta ))$ denominators in the vanilla score are precisely what the natural gradient cancels — equivalently, IGO on Bernoulli is vanilla gradient ascent in the logit chart $\ln \frac{\theta }{1-\theta }$, where the geometry of a binomial family is flat. That one-sentence observation is the entire reason IGO doesn't struggle near the boundary ${\theta }_i\in \{0,1\}$.

6.2 Exponential families: derivative-free updates and the e-geometry

If $P_{\theta }(x)\propto \exp ({\sum }_i{\theta }_i{T}_i(x))$ (Gaussians and Bernoulli measures are both examples), the score and the Fisher matrix collapse into covariances of the sufficient statistics $T$:

The second line is the deep one: in the “mean-parameter” chart the IGO flow needs no gradients and no Fisher matrix — just sample covariances of $T$ with the quality $W$ (computable by Monte Carlo or Gibbs sampling). This is the duality of natural parameters $\theta$ and expectation parameters $\bar{T}$ under the e-/m- flat geometry of Amari & Nagaoka. It is also the direct route to IGO on restricted Boltzmann machines (marginal of an exponential family): the resulting algorithm handles dependent bits and — because RBM distributions are multimodal and IGO keeps diversity high (minimal-change property, Section 3.2) — can reach several valleys of the fitness landscape in one run [App. B].

6.3 The zoo

IGO's payoff relative to hand-crafted heuristics is that its objects — a continuous-time flow, exact invariances, and explicit approximations — are amenable to mathematical analysis. The main results, stated honestly:

• Consistency of sampling (Thm. 6). As $\lambda \to \infty$, the sample-based update converges almost surely to one Euler step of the IGO flow — despite the weights depending on the whole batch through ranks.
• Monotone quantile improvement (Prop. 7). For the truncation scheme $w(u)=1_{u\le q}$, along the flow the $q$-quantile of $f$ (the “median of the best” in the $q$-sense) never worsens, and strictly improves unless the flow is stationary. This is the honest replacement for a guarantee: the flow pushes the whole distribution across the $q$ level set, not just a point.
• Finite step sizes of $\eta \le 1$ are still descent. [Akimoto & Ollivier 2013] extend quantile/objective improvement to nonzero Euler steps: step sizes not larger than a natural unit guarantee monotone improvement of the expected utility in the IGO-ML setting.
• Finite-sample rates on quadratics. [Uchida, Shirakawa & Akimoto 2019] analyze IGO with an isotropic Gaussian family on convex quadratic functions with finite populations: with enough samples per generation, the algorithm converges linearly (exponentially in the number of generations) — matching the empirical linear-convergence of evolution strategies, and confirming that the geometric design (not luck) is what makes them fast.
• Information accounting (Prop. 20 & Cor. 21). The speed of the flow is bounded by the standard deviation of the selection scheme $w$, and one Euler step of size $\eta$ injects at most a fixed number of “bits” of Fisher information into the distribution — the formal statement of “minimal change, maximal diversity”.
• Noise (Prop. 22). Noisy evaluations fit the framework as-is: IGO on a noisy $f$ is equivalent to IGO on a noiseless function defined on an enlarged space $X\times \Omega$, so all the clean-case theorems transfer.

IGO is one application of a bigger idea — natural gradient descent — that recurs across optimization, machine learning and reinforcement learning whenever the object being optimized is a probability distribution.

• Machine learning. Amari's original message [Amari 1998]: gradient descent on the parameters of a model is coordinate-dependent; gradient descent on the distribution the model represents (Fisher metric) converges isotropically in function space. The dual-coordinate structure of exponential families (natural vs. expectation parameters, e- and m- flatness) is the same toolkit IGO's exponential-family results use [Amari & Nagaoka 2000].
• Reinforcement learning. Kakade's natural policy gradient [Kakade 2002] and the trust-region methods that followed (TRPO's KL-constrained policy step, PPO's clipped surrogate) are exactly the “improve the distribution subject to a KL budget” move of Section 3.2. Evolution strategies applied to RL (e.g. OpenAI-ES 2017, a mean-only Gaussian NES) are IGO with the smallest family that still works, scaled to parallelized supercomputers and settings where gradients are unavailable or unreliable.
• Optimization theory. The proximal view (maximize utility minus KL penalty) connects IGO to mirror descent and to the cross-entropy/EDA family; CMA-ES remains the field's most refined practical instance, and the IGO lens turns its heuristics into a coherent design history (evolution path and CSA are the surviving engineering extras).
• Frontiers. The template invites new families: heavy-tailed NES for faster global exploration, IGO flows over curved search spaces (spheres — von Mises–Fisher-type families — and rotation groups; Jacimović 2026), and surrogate-assisted IGO analyses [Akimoto 2024].

References & further reading

1. Y. Ollivier, L. Arnold, A. Auger, N. Hansen (2017). Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles. JMLR 18(18):1–65. jmlr.org/papers/v18/14-467.html · arXiv:1106.3708.
2. T. Glasmachers, T. Schaul, Y. Sun, D. Wierstra, J. Schmidhuber (2010). Exponential Natural Evolution Strategies. GECCO 2010. doi:10.1145/1830483.1830557
3. D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, J. Schmidhuber (2014). Natural Evolution Strategies. JMLR 15(1):949–980.
4. S. Amari (1998). Natural Gradient Works Efficiently in Learning. Neural Computation 10(2):251–276.
5. S. Amari, H. Nagaoka (2000). Methods of Information Geometry. AMS Translations of Mathematical Monographs 191.
6. N. Hansen, A. Ostermeier (2001). Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9(2):159–195.
7. Y. Akimoto, Y. Ollivier (2013). Objective Improvement in Information-Geometric Optimization. FOGA 2013. doi:10.1145/2460239.2460241
8. K. Uchida, S. Shirakawa, Y. Akimoto (2020). Finite-Sample Analysis of Information Geometric Optimization With Isotropic Gaussian Distribution on Convex Quadratic Functions. IEEE Trans. Evol. Comput. 24(6):1035–1049.
9. P.-T. de Boer, D. Kroese, S. Mannor, R. Rubinstein (2005). A Tutorial on the Cross-Entropy Method. Annals of Operations Research 134(1):5–36.
10. S. Baluja (1994). Population-Based Incremental Learning. CMU-CS-94-163. · G. Harik, F. Lobo, D. Goldberg (1999). The Compact Genetic Algorithm. IEEE TEVC 3(4).
11. S. Kakade (2002). A Natural Policy Gradient. NeurIPS 14.
12. T. Salimans, J. Ho, X. Chen, S. Sidor, I. Sutskever (2017). Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv:1703.03864.
13. V. Jacimović (2026). Information-Geometric Optimization on Spheres. arXiv:2606.07588.