Data–MC Comparison¶
This page presents Data–MC comparison plots using the burn sample: experimental runs whose run numbers end in 0, corresponding to approximately 10% of the full IceCube dataset. All histograms in the upper panels show rates normalized to the burn livetime, while the summary tables underneath report MC expectations rescaled to the full 12 years of livetime.
Unless stated otherwise, all plots use the final cut selection as defined in Sensitivity Optimization.
Weighting and Flux Assumptions¶
Lightning Tracks is primarily designed for point-source analyses. In this regime, the background for the test statistic is obtained from data scrambling, so the sensitivity is largely insensitive to detailed mis-modeling of the atmospheric background flux. What matters most for point sources is the signal modeling: in particular, that there is no significant discrepancy between the baseline and Snowstorm systematics NuGen sets used as signal in Csky. The Data–MC plots on this page therefore serve as sanity checks on both background and signal, rather than as a full diffuse-style flux fit.
For the atmospheric neutrino flux we use models from Nuflux. Multiple flux parameterizations can be selected in the interactive plots via the Atmospheric ν Flux Model selector (see the Nuflux documentation for details). This allows one to explore how different flux assumptions impact the distributions. We do not model the self-veto effect, so under-fluctuations of starting events in the southern sky, especially at high energies, are expected and should not be over-interpreted.
Atmospheric muons (CORSIKA) are weighted using SimWeights with the GlobalFitGST flux model. In practice, the absolute normalization of this flux does not match the data perfectly. Since we do not perform any dedicated fit to the muon background, we apply a single global normalization factor of 1.5 to the CORSIKA weights. This factor is motivated by regions where atmospheric muons clearly dominate (throughgoing tracks in the southern sky) and is consistent with the data–MC mismatch already observed at pre-cut and even unfiltered Level-2 stages. It should therefore be viewed as a crude global normalization adjustment of the muon flux model, rather than a fit to the burn sample or to specific low-energy bins. No additional energy-dependent tuning or shape adjustment is applied.
The astrophysical neutrino flux assumptions differ by topology:
- Starting tracks (MESE-like diffuse fit):
$$ \Phi_{\nu}^{\mathrm{astro}} = \frac{2.06 \times 10^{-18}}{2} \left(\frac{E}{10^5~\mathrm{GeV}}\right)^{-2.46}~\mathrm{GeV}^{-1}\mathrm{cm}^{-2}\mathrm{s}^{-1}\mathrm{sr}^{-1} $$
- Throughgoing tracks (northern-track diffuse fit):
$$ \Phi_{\nu}^{\mathrm{astro}} = \frac{1.44 \times 10^{-18}}{2} \left(\frac{E}{10^5~\mathrm{GeV}}\right)^{-2.2}~\mathrm{GeV}^{-1}\mathrm{cm}^{-2}\mathrm{s}^{-1}\mathrm{sr}^{-1} $$
These astrophysical components are also used “out of the box”: we do not attempt to fit or re-optimize them to the burn sample.
Uncertainties and Ratio Plot¶
Conceptually, the error treatment is simple: data are treated as Poisson counts, MC uncertainties are obtained from the sum of squared event weights in each bin, and both are propagated to the Data/MC ratio using standard Gaussian error propagation.
Each Data–MC figure consists of two main panels:
-
Top panel – burn-normalized rates: shows the individual MC components (atmospheric \(\mu\), atmospheric \(\nu\), astrophysical \(\nu\)), their sum, and the burn data with statistical error bars.
-
Ratio panel – Data/MC: shows the bin-wise ratio of burn data to the total MC expectation, with uncertainties obtained via standard error propagation.
The statistical errors shown in these panels are computed as follows:
-
Burn data (experimental) Data are histogrammed without weights, yielding counts \(N_j^{\mathrm{data}}\) in bin \(j\). The statistical uncertainty is treated as Poisson: $$ \delta N_j^{\mathrm{data}} = \sqrt{N_j^{\mathrm{data}}}. $$ When converting to rates using the burn livetime \(T_{\mathrm{burn}}\), $$ R_j^{\mathrm{data}} = \frac{N_j^{\mathrm{data}}}{T_{\mathrm{burn}}}, \qquad \delta R_j^{\mathrm{data}} = \frac{\sqrt{N_j^{\mathrm{data}}}}{T_{\mathrm{burn}}}. $$ These rate errors are the vertical error bars on the data points in the top panel.
-
MC components (weighted) For each MC component (CORSIKA, atmospheric \(\nu\), astrophysical \(\nu\)) and each bin, we sum the event weights \(w_i\) to obtain the expected contribution $$ H_j = \sum_{i \in \text{bin } j} w_i, $$ and estimate the statistical variance as the sum of squared weights, $$ \left(\delta H_j\right)^2 = \sum_{i \in \text{bin } j} w_i^2. $$ This is the standard treatment for weighted Monte Carlo. Contributions from different components are added in bin \(j\), and their uncertainties are combined in quadrature, so that the total MC expectation $$ H_j^{\mathrm{MC}} = H_j^{\mathrm{CORSIKA}} + H_j^{\mathrm{atmo}} + H_j^{\mathrm{astro}} $$ has an uncertainty $$ \delta H_j^{\mathrm{MC}} = \sqrt{ \left(\delta H_j^{\mathrm{CORSIKA}}\right)^2 + \left(\delta H_j^{\mathrm{atmo}}\right)^2 + \left(\delta H_j^{\mathrm{astro}}\right)^2 }. $$ In the top panel, the grey band around the total MC curve corresponds to \(H_j^{\mathrm{MC}} \pm \delta H_j^{\mathrm{MC}}\) (normalized to the burn livetime).
-
Data/MC ratio For each bin in which both data and MC are defined and non-zero, we form the ratio $$ \mathcal{R}_j = \frac{R_j^{\mathrm{data}}}{R_j^{\mathrm{MC}}}, $$ where \(R_j^{\mathrm{MC}} = H_j^{\mathrm{MC}} / T_{\mathrm{burn}}\). The relative uncertainty on \(\mathcal{R}_j\) is obtained by standard error propagation assuming independent errors on data and MC: $$ \left(\frac{\delta \mathcal{R}_j}{\mathcal{R}_j}\right)^2 = \left(\frac{\delta R_j^{\mathrm{data}}}{R_j^{\mathrm{data}}}\right)^2 + \left(\frac{\delta R_j^{\mathrm{MC}}}{R_j^{\mathrm{MC}}}\right)^2. $$ The ratio panel shows \(\mathcal{R}_j \pm \delta \mathcal{R}_j\) together with a reference line at Data/MC = 1.
Finally, the summary table in the lower part of each figure integrates over all bins, rescales the MC expectations to the full 12-year livetime, and reports total counts, rates, and flavor/topology breakdowns. It also lists the unweighted size of the CORSIKA sample, which provides a direct indication of where limited muon statistics make the MC band and the ratio plot particularly fragile.
Energy Variable Distributions¶
Energy distributions show Data–MC comparisons for reconstructed energy-related variables across different sky regions. Plots are available for both baseline NuGen and Snowstorm systematics sets, and for different atmospheric neutrino flux models.
Angular Variable Distributions¶
Angular distributions (zenith, declination, azimuth) show Data–MC agreement as a function of arrival direction and reconstructed energy. By scanning energy ranges and flux models, one can check that no glaring mismodeling appears in specific angular–energy regions.
Interpretation and Takeaways¶
Across all distributions, the dominant source of Data–MC discrepancy is the atmospheric background modeling—for both muons and atmospheric neutrinos. The limitations of the CORSIKA dataset are immediately visible: statistics are poor at low and high energies, and although this is well known, it remains an unavoidable constraint given the computational cost of simulating atmospheric muons at scale.
Even for neutrinos, where statistics are not the limiting factor, we observe that the choice of atmospheric flux model has an enormous impact on the resulting distributions. Most models (with the exception of the SIBYLL-2.1–based GaisserH3a parameterizations) produce low-energy fluxes that clearly disagree with data. The fluxes used here are only valid in specific energy ranges (see the Nuflux documentation), which makes it difficult to assess the quality of the selection itself, because the uncertainty is dominated by flux modeling, not by reconstruction or cuts. More aggressive tuning—fitting atmospheric-flux parameters to the data—would inevitably improve agreement, but only because the MC is then forced to match the data under the chosen flux-model prior; it does not resolve the underlying question of what the true atmospheric flux is. This is, of course, the subject of many dedicated diffuse analyses.
This problem is fundamentally hard:
- the cosmic-ray primary flux is uncertain and time-variable (seasonal modulation + solar cycle),
- hadronic interaction models differ substantially in their cascade predictions,
- and the prompt component remains poorly constrained.
As a result, large deviations in certain regions are fully expected and should not be interpreted as reconstruction or selection failures.
It would still be beneficial to treat these poorly constrained variables (e.g., the overall flux normalization and the cosmic-ray spectral shape) as nuisance parameters. This is the approach taken in diffuse analyses aiming to constrain the astrophysical neutrino flux. It would also allow us to evaluate the Data–MC agreement for additional observables relevant to point-source analyses, especially the distribution of angular-error estimates. Unfortunately, we do not currently have the human resources to perform such a fit.
Signal Modeling Appears Reliable¶
The key validation target for Lightning Tracks is not background prediction but signal modeling, since the point-source analyses we intend to support use scrambled data to estimate the background, making them largely insensitive to atmospheric-flux modeling uncertainties. In this context, the most meaningful comparison is between the baseline and systematics NuGen sets—and they are virtually indistinguishable in every plot shown here. Differences, where visible at all, are tiny and consistent with statistical fluctuations from finite Monte Carlo.
This leads to the primary conclusion:
There is no evidence of signal mis-modeling.
Baseline and Snowstorm NuGen are essentially indistinguishable across all observables. Any discrepancies in Data–MC are dominated by atmospheric-flux modeling uncertainties rather than by reconstruction or selection performance.
However, to reiterate, we cannot yet evaluate the more relevant distributions for point-source purposes without performing a basic diffuse fit of the atmospheric background.