Analysis API

get_params(x) -> NamedTuple

Extract nominal parameter values as a sorted NamedTuple.

Dispatches on the type of x:

• Physics: returns x.params sorted by key.
• Experiment: merges x.params with the parameters of its physics module.
• Wrapper: applies the alias translation from Wrapper to the keys.
• NamedTuple of modules: merges parameters from all modules via safe_merge, raising an error on conflicts.

Arguments

• x: a Newtrinos.Physics, Newtrinos.Experiment, Wrapper, or NamedTuple of such objects.

Returns

A sorted NamedTuple of parameter name → nominal value.

get_priors(x) -> NamedTuple

Extract prior distributions as a sorted NamedTuple.

Dispatches on the type of x:

• Physics: returns x.priors sorted by key.
• Experiment: merges x.priors with priors from its physics module.
• Wrapper: applies the alias translation from Wrapper to the keys.
• NamedTuple of modules: merges priors from all modules via safe_merge, raising an error on conflicts.

Arguments

• x: a Newtrinos.Physics, Newtrinos.Experiment, Wrapper, or NamedTuple of such objects.

Returns

A sorted NamedTuple of parameter name → prior distribution.

condition(priors, conditional_vars, params) -> NamedTuple

Fix a subset of parameters to specific values by replacing their prior distributions with constants (ConstValueDist).

Two dispatch methods:

Array form (conditionalvars is an array) — fixes each symbol in `conditionalvarsto its value inparams`:

condition(priors, [:theta23, :dm31_sq], params)

Dict form (conditional_vars is a dict) — fixes each key to the paired value, or to params[key] when the value is nothing:

condition(priors, Dict(:theta23 => pi/4, :dm31_sq => nothing), params)

Arguments

• priors::NamedTuple: prior distributions to modify.
• conditional_vars: AbstractArray{Symbol} or AbstractDict{Symbol} of parameters to fix.
• params::NamedTuple: fallback parameter values used when a dict entry is nothing or when using the array form.

Returns

The modified priors NamedTuple with the specified parameters frozen.

safe_merge(nt_list::NamedTuple...) -> NamedTuple

Merge NamedTuples, asserting that any key shared by two or more inputs has the same value in all of them.

Raises an error if a key conflict is detected. The result is sorted alphabetically by key.

Arguments

• nt_list::NamedTuple...: one or more NamedTuples to merge.

Returns

A sorted NamedTuple containing all key-value pairs from the inputs.

Examples

a = (x = 1.0, y = 2.0)
b = (y = 2.0, z = 3.0)
safe_merge(a, b)   # (x = 1.0, y = 2.0, z = 3.0)

correlated_priors_vars(priors, vars, dist) -> (corr_prior, other_prior)

Replace the independent priors for vars with a single correlated multivariate distribution, splitting the prior NamedTuple into two parts.

Useful when a set of parameters has a known covariance structure (e.g. from an external fit) that cannot be captured by independent marginals.

Arguments

• priors::NamedTuple: the full prior NamedTuple.
• vars::Union{AbstractArray, Tuple}: parameter names to correlate.
• dist::Distributions.Distribution: a multivariate distribution over vars (must match the length of vars).

Returns

A 2-tuple (corr_prior, other_prior) where:

• corr_prior: a callable returning a ReshapedDist over vars.
• other_prior: a distprod of the remaining (uncorrelated) priors.

generate_asimov_data(experiment, params) -> data
generate_asimov_data(experiments::NamedTuple, params) -> NamedTuple

Generate Asimov (expected-value) data from the experiment forward model.

Evaluates mean(forward_model(params)). For Poisson-distributed observables the mean is rounded to the nearest integer so that the data type matches what a real run would produce.

Arguments

• experiment: a single Newtrinos.Experiment or a NamedTuple of them.
• params::NamedTuple: parameter values at which to evaluate the forward model.

Returns

• Single-experiment form: the expected-value data array (or NamedTuple of arrays).
• Multi-experiment form: a NamedTuple with the same keys as experiments.

generate_toy_data(experiment, params) -> data
generate_toy_data(experiments::NamedTuple, params) -> NamedTuple

Generate random toy data by sampling from the experiment forward model.

Calls experiment.forward_model(params) to obtain the predictive distribution and draws one sample from it.

Arguments

• experiment: a single Newtrinos.Experiment or a NamedTuple of them.
• params::NamedTuple: parameter values at which to evaluate the forward model.

Returns

• Single-experiment form: one sample from the forward model distribution.
• Multi-experiment form: a NamedTuple with the same keys as experiments, each entry holding one sample.

make_prior_samples(posterior, nsamples::Int=10_000) -> NamedTuple

Initialize importance samples by using a standard normal as the proposal distribution.

Transforms posterior to the standard-normal space via BAT's PriorToNormal reparameterization, then generates nsamples samples by calling importance_sampling with the transformed posterior and the standard normal $\mathcal{N}(0, I)$ as the proposal distribution.

Arguments

• posterior: a BAT posterior measure.
• nsamples::Int: number of importance samples to draw (default: 10_000).

Returns

A NamedTuple with fields:

• approx_dist: the standard normal proposal used.
• samples_p: DensitySampleVector in the transformed (normal) space.
• samples_user: samples back-transformed to the original parameter space.

make_init_samples(posterior, nseeds::Int=10, nsamples::Int=10_000) -> NamedTuple

Initialize importance samples by fitting a mixture of local Gaussian approximations at posterior modes.

Finds nseeds approximate modes via L-BFGS optimization (using Sobol-sequence starting points), fits a local_MGVI_approx at each mode, and combines them into a MixtureModel. The mixture model weights are calculated as

\[\begin{aligned} \log \tilde{w}_i &= \log p (\mu_i) - \log q (\mu_i) \\ \tilde{w}_i &= \exp\bigl(\log \tilde{w}_i - \max(\log \tilde{w})\bigr) \\ w_i &= \frac{\tilde{w}_i}{\sum_j \tilde{w}_j} \end{aligned}\]

where $\mu_i$ is the mode of the $i$-th component. Importance samples are then drawn from the mixture via importance_sampling.

Arguments

• posterior: a BAT posterior measure.
• nseeds::Int: number of mode-finding restarts (default: 10).
• nsamples::Int: number of importance samples to draw (default: 10_000).

Returns

A NamedTuple with fields:

• approx_dist::MixtureModel: the weighted Gaussian mixture proposal.
• samples_p: DensitySampleVector in the transformed space.
• samples_user: samples back-transformed to the original parameter space.

make_init_samples(posterior, seed_points::DataFrame, nsamples::Int=10_000) -> NamedTuple

Initialize importance samples from user-supplied seed points given as a DataFrame.

Like the integer-seed form but uses rows of seed_points as starting points for mode-finding instead of Sobol samples. Each row is projected into the transformed parameter space and L-BFGS is run from there.

Arguments

• posterior: a BAT posterior measure.
• seed_points::DataFrame: each row provides the starting parameter values. Must contain columns Darkdim_radius, ca1, ca2, ca3.
• nsamples::Int: number of importance samples to draw (default: 10_000).

Returns

A NamedTuple with fields:

• approx_dist::MixtureModel: the weighted Gaussian mixture proposal.
• samples_p: DensitySampleVector in the transformed space.
• samples_user: samples back-transformed to the original parameter space.

whack_a_mole(posterior, init_samples, n_whack=100) -> NamedTuple

Refine an importance sampling approximation by iteratively adding Gaussian components at high-weight sample points.

At each of n_whack iterations:

1. The highest-weight sample in the current mixture is identified.
2. A local_MGVI_approx is fitted at that point.
3. The new component is added to the mixture with weight proportional to its posterior density at its mode relative to the mixture density.
4. New samples are drawn from the new component and importance-reweighted.

This "whack-a-mole" strategy adaptively targets undersampled regions of the posterior. Use whack_many_moles for a parallelized variant with convergence criteria.

Arguments

• posterior: a BAT posterior measure.
• init_samples: initialization NamedTuple as returned by make_init_samples or make_prior_samples.
• n_whack::Int: number of refinement iterations (default: 100).

Returns

A NamedTuple with fields:

• approx_dist::MixtureModel: the refined Gaussian mixture proposal.
• samples_p: accumulated DensitySampleVector in the transformed space.
• samples_user: samples back-transformed to the original parameter space.

whack_many_moles(posterior, init_samples;
                 target_efficiency=Inf, target_samplesize=Inf,
                 maxiter=100, n_parallel=Threads.nthreads(),
                 cache_dir=nothing) -> NamedTuple

Parallelized adaptive importance sampling with convergence criteria.

Like whack_a_mole but processes n_parallel high-weight sample points simultaneously per each iteration using Threads.@threads. Iteration stops when any of the following conditions is met:

• Effective sample size (ESS) exceeds target_samplesize.
• Efficiency (ESS / total samples) exceeds target_efficiency.
• Number of iterations reaches maxiter.

Intermediate results can be saved to disk after each iteration for fault tolerance.

Arguments

• posterior: a BAT posterior measure.
• init_samples: initialization NamedTuple as returned by make_init_samples or make_prior_samples.
• target_efficiency::Real: stop when ESS / n_samples exceeds this value (default: Inf, i.e. no efficiency target).
• target_samplesize::Real: stop when ESS exceeds this value (default: Inf, i.e. no ESS target).
• maxiter::Int: maximum number of refinement iterations (default: 100).
• n_parallel::Int: number of new components to fit per iteration (default: Threads.nthreads()).
• cache_dir::Union{String,Nothing}: if provided, saves a JLD2 checkpoint after each iteration to this directory. (Default: nothing).

Returns

A NamedTuple with fields:

• approx_dist::MixtureModel: the refined Gaussian mixture proposal.
• samples_p: accumulated DensitySampleVector in the transformed space.
• samples_user: samples back-transformed to the original parameter space.

importance_sampling(pstr, approx_dist, nsamples) -> DensitySampleVector

Draw importance-weighted samples from pstr using approx_dist as the proposal.

Samples nsamples points from approx_dist, evaluates $\log p$ (pstr) and $\log q$ (approx_dist) at each point, and computes normalized importance weights. The evaluation of $\log p$ is parallelized over threads. The weights are calculated as:

\[\log \tilde{w}_i = \log p - \log q w_i = \exp\!\bigl( \log \tilde{w}_i - \max(\log \tilde{w}_i) \bigr)\]

with the $\max(\log \tilde{w}_i)$ substracted in the exponent such that $\max_i w_i = 1$ for numerical stability.

Arguments

• pstr: BAT posterior (typically already transformed via PriorToNormal).
• approx_dist: proposal distribution supporting bat_sample and logdensityof.
• nsamples::Int: number of samples to draw from approx_dist.

Returns

A DensitySampleVector with samples from approx_dist, their respective $\log p$, and their importance weights.

local_MGVI_approx(pstr, θ_sel) -> MvNormal

Fit a local Gaussian approximation to pstr at the selected point θ_sel.

Uses the MGVI (Metric Gaussian Variational Inference) approach: the covariance is

\[\Sigma = \bigl(J^\top\, F(\theta)\, J + I\bigr)^{-1}\]

where $F(\theta)$ is the Fisher information matrix of the likelihood model evaluated at $\theta$, and $J = \partial_\theta(\texttt{flat\_params} \circ m)$ is the Jacobian of the parameter reparameterization. The result is regularized with a Cholesky decomposition via PositiveFactorizations.

Arguments

• pstr: BAT posterior with a likelihood field exposing the forward model k.
• θ_sel: parameter vector (in the transformed space) at which to centre the Gaussian.

Returns

A MvNormal distribution centred at θ_sel with covariance $\Sigma$.

generate_likelihood(experiments[, observed]) -> likelihood

Construct a joint likelihood from a NamedTuple of configured experiments.

Combines all experiment forward models into a product distribution and pairs it with the observed data. The returned object supports DensityInterface.logdensityof(likelihood, params).

Arguments

• experiments::NamedTuple: named collection of configured Newtrinos.Experiment objects.
• observed (optional): NamedTuple of observed data arrays matching experiments. Defaults to the observed data stored in each experiment's assets.observed field.

Returns

A likelihood object compatible with DensityInterface.logdensityof.

Examples

experiments = (deepcore = Newtrinos.deepcore.configure(),
               dayabay  = Newtrinos.dayabay.configure())
llh     = generate_likelihood(experiments)
logdensityof(llh, params)

get_observed(experiments::NamedTuple) -> NamedTuple

Extract the observed data from each experiment in a NamedTuple of experiments.

Arguments

• experiments::NamedTuple: named collection of Newtrinos.Experiment objects.

Returns

A NamedTuple with the same keys as experiments, each entry holding the corresponding experiment.assets.observed data.

get_fwd_model(experiments::NamedTuple) -> Function

Compose the forward models of all experiments into a single joint model.

Builds a NamedTuple of per-experiment forward models and composes them with ffanout and distprod so that calling the result with a parameter NamedTuple returns a joint product distribution over all experiments.

Arguments

• experiments::NamedTuple: named collection of Newtrinos.Experiment objects.

Returns

A callable params -> joint_distribution.

find_mle(likelihood, prior, params; adsel=AutoPolyesterForwardDiff())
    -> (llh, log_posterior, result)

Find the Maximum Likelihood Estimator (MLE) using L-BFGS optimization via BAT.jl.

Parameters whose prior is a ConstValueDist are held fixed during optimization. If the optimizer raises an ArgumentError (e.g. due to a degenerate starting point), returns (NaN, NaN, NaN-filled NamedTuple) instead of propagating the error.

Arguments

• likelihood: a likelihood object supporting DensityInterface.logdensityof.
• prior: a distprod-style prior NamedTuple or distribution.
• params::NamedTuple: starting parameter values for the optimizer.
• adsel: AD backend selector (default: AutoPolyesterForwardDiff()).

Returns

A 3-tuple (llh, log_posterior, result) where:

• llh::Float64: log-likelihood at the optimum.
• log_posterior::Float64: log-posterior at the optimum.
• result::NamedTuple: optimized parameter values.

find_mle_cached(likelihood, prior, params, cache_dir)
    -> (llh, log_posterior, result)

Like find_mle but caches results to disk using content hashing.

A hash of (prior, params) is computed with ContentHashes.hash; if a matching .jld2 file exists in cache_dir the optimization is skipped and the cached result is returned. Otherwise the result is computed and saved.

Arguments

• likelihood: a likelihood object supporting DensityInterface.logdensityof.
• prior: prior distribution (same as find_mle).
• params::NamedTuple: starting parameter values.
• cache_dir::Union{String,Nothing}: directory for cached .jld2 files. Pass nothing to disable caching.

Returns

A 3-tuple (llh, log_posterior, result) — see find_mle.

profile(likelihood, priors, vars_to_scan, params;
        cache_dir=nothing, map_func=nothing) -> NewtrinosResult

Run a profile likelihood scan over a parameter grid.

Creates grid points defined by vars_to_scan with generate_scanpoints. At each grid point, all nuisance parameters are optimized with _profile , i.e. via find_mle_cached. If all non-scanned parameters have fixed (Number) priors, falls back to scan automatically. Also collects meta data of the profile process and attaches it the returned object.

Arguments

• likelihood: a likelihood object supporting DensityInterface.logdensityof.
• priors::NamedTuple: prior distributions for all parameters.
• vars_to_scan::OrderedDict{Symbol,Int}: parameters to scan and grid sizes.
• params::NamedTuple: starting values for nuisance parameter optimization.
• cache_dir::Union{String,Nothing}: directory for caching MLE results (created if absent). Pass nothing to disable (this is the default).
• map_func: custom mapping function for parallelism (e.g. pmap for distributed workers). Defaults to Threads.@threads.

Returns

A NewtrinosResult NewtrinosResult(axes, values, meta) with the scan grid axes, per-point profiling results, and meta data.

scan(likelihood, priors, vars_to_scan, params;
     gradient_map=false) -> NewtrinosResult

Run a simple likelihood scan on a parameter grid (NO nuisance optimization).

Generates grid with _generate_grid via vars_to_scan and then evaluates logdensityof(likelihood, params) at each grid point without optimizing over nuisance parameters. Faster than profile but does not account for nuisance parameter variations. Also collects meta data of the scan process and attaches it the returned object.

Arguments

• likelihood: a likelihood object supporting DensityInterface.logdensityof.
• priors::NamedTuple: prior distributions used to place grid points at quantiles.
• vars_to_scan::OrderedDict{Symbol,Int}: parameters to scan and grid sizes.
• params::NamedTuple: nominal values; non-scanned parameters are held fixed.
• gradient_map::Bool: default=false; if true, also evaluates ForwardDiff.gradient at each grid point and includes per-parameter gradient arrays in the result.

Returns

A NewtrinosResult NewtrinosResult(axes, values, meta) with the scan grid axes, per-point scan results, and meta data.

bestfit(result::NewtrinosResult) -> NamedTuple

Extract the best-fit parameter values from a scan or profile result.

Finds the grid point with the highest log_posterior and returns both the scanned parameter values (from result.axes) and all optimized nuisance parameter values at that point.

Arguments

• result::NewtrinosResult: output of scan or profile.

Returns

A NamedTuple containing all parameter values at the best-fit point, including the scanned axis values, nuisance parameters, llh, and log_posterior.

generate_scanpoints(vars_to_scan, priors) -> (values, scanpoints)

Build a grid of fixed-parameter priors for a profile likelihood scan.

For each variable in vars_to_scan, grid points are placed at evenly-spaced quantiles of its prior distribution. All combinations are formed via a Cartesian product. At each grid point, the prior for the scanned variables is replaced with a ConstValueDist fixing them to the grid values.

Arguments

• vars_to_scan::OrderedDict{Symbol,Int}: maps each parameter name to the desired number of evenly spaced grid points within the prior range.
• priors::NamedTuple: prior distributions for all parameters.

Returns

A 2-tuple (values, scanpoints) where:

• values: vector of grid-point vectors (one per scanned parameter).
• scanpoints: array of distprod priors, one per grid point.

assemble_profile_results(opt_results, result_size) -> NamedTuple

Collect the 'raw' per-grid-point optimization results and structure them into flat arrays within a NamedTuple.

Stacks the (llh, log_posterior, params) 3-tuples returned by find_mle_cached into a NamedTuple of arrays with shape result_size.

Arguments

• opt_results: iterable of (llh, log_posterior, NamedTuple) 3-tuples.
• result_size: shape of the output arrays (matching the scan grid).

Returns

A NamedTuple with one array-entry per parameter (values across the grid) plus :llh and :log_posterior arrays.

Example

opt_results = [
    (-10.0, -12.0, (a=1.0, b=2.0)), # llh, log_posterior, params output of find_mle
    (-5.0,  -7.0,  (a=3.0, b=4.0))
]
res = Newtrinos.assemble_profile_results(opt_results, (2,))
# returns res = (a = [1.0, 3.0], b = [2.0, 4.0], llh = [-10.0, -5.0], log_posterior = [-12.0, -7.0])

add_meta!(meta::Dict)

Populate a metadata dictionary with execution environment information. When Newtrinos is loaded from the julia package index, it marks the git-repo information as missing.

Adds the following keys in-place:

• "hostname": result of gethostname().
• "username": from ENV["USER"] or ENV["USERNAME"].
• "date": current date-time formatted as "yyyy-mm-dd HH:MM:SS".
• "repo": path to the Newtrinos.jl root.
• "package_version": loaded version of the Newtrinos.jl package.
• "commit_hash": current HEAD commit hash.
• "repo_clean": true if the repository has no uncommitted changes.

Called automatically by profile and scan before returning their NewtrinosResult.

Arguments

• meta::Dict: dictionary to populate in-place.

Returns

nothing (mutates meta).

plot!(ax, result::NewtrinosResult;
      max_llh=maximum(result.values.log_posterior),
      levels=1 .- 2*ccdf(Normal(), 1:3),
      label=nothing, color=:blue, linestyle=:solid,
      cmap=:Blues, filled=false, edge=true,
      transform_x=identity, transform_y=identity) -> Axis

Plot a NewtrinosResult into an existing Makie Axis.

Dispatches on the dimensionality of result.axes:

• 1D: plots -2Δllh as a line with horizontal dotted lines at the requested confidence levels.
• 2D: plots confidence contours via contour! and optionally a filled version via contourf!.

Confidence levels are converted to -2Δllh thresholds using the appropriate chi-squared distribution (Chisq(1) for 1D, Chisq(2) for 2D).

Arguments

• ax::Axis: the Makie axis to plot into.
• result::NewtrinosResult: scan or profile result to visualize.
• max_llh: reference log-posterior value for computing -2Δllh; defaults to the maximum in result.values.log_posterior.
• levels: confidence levels to draw contours at; defaults to 1σ, 2σ, 3σ Gaussian intervals.
• label: legend label attached to the line or contour; default nothing.
• color: line/contour color; default :blue.
• linestyle: line style; default :solid.
• cmap: colormap used for the filled contours (2D only); default :Blues.
• filled::Bool: if true, draw filled contours via contourf! (2D only); default false.
• edge::Bool: if true, draw contour edges via contour! (2D only); default true.
• transform_x: function applied to the first axis before plotting; default identity.
• transform_y: function applied to the second axis before plotting; default identity.

Returns

The modified Axis ax.

plot(result::NewtrinosResult; kwargs...) -> Figure

Convenience wrapper that creates a new [Figure] and [Axis] and calls plot! with all kwargs. Returns the Figure.

sort_nt(nt::NamedTuple) -> NamedTuple

Return a copy of a named tuple nt with keys sorted alphabetically.

Used internally by safe_merge, get_params, and get_priors to ensure a canonical key ordering across modules.

Arguments

• nt::NamedTuple: input NamedTuple.

Returns

A NamedTuple with the same entries as nt, sorted alphabetically by key name.

_generate_grid(vars_to_scan, priors) -> (vars, values, mesh)

Construct the Cartesian grid used by generate_scanpoints and scan.

Grid points for each variable are placed at evenly-spaced quantiles of its prior distribution. All combinations are formed via IterTools.product.

Arguments

• vars_to_scan::OrderedDict{Symbol,Int}: maps parameter names to grid sizes.
• priors::NamedTuple: prior distributions used to compute quantiles.

Returns

A 3-tuple (vars, values, mesh) where:

• vars: Vector{Symbol} of scanned parameter names.
• values: vector of grid-point vectors (one per variable).
• mesh: Cartesian product array of grid-point tuples.

Examples

    priors = (a=Uniform(0.0, 2.0), b=Uniform(-1.0, 1.0))
    vars_to_scan = OrderedDict(:a => 3, :b => 4)
    vars, values, mesh = Newtrinos._generate_grid(vars_to_scan, priors)
    # => mesh is a 3x4 grid of evenly spaced points from within the prior range of a and b 

_profile(likelihood, scanpoints, params, cache_dir; map_func=nothing)
    -> NamedTuple

Execute find_mle_cached at every point in scanpoints, collect the results, and assemble them into a NamedTuple of result arrays with assemble_profile_results.

Uses Threads.@threads by default. Pass a custom map_func (e.g. pmap) to override the parallelism strategy.

Arguments

• likelihood: a likelihood object supporting DensityInterface.logdensityof.
• scanpoints: array of prior objects, one per grid point.
• params::NamedTuple: starting values for nuisance optimization.
• cache_dir::Union{String,Nothing}: passed to find_mle_cached.
• map_func: optional custom mapping function; default=nothing uses threaded loops.

Returns

A NamedTuple of result arrays as produced by assemble_profile_results.