Daily Papers

Sparse Autoencoders (SAEs) have emerged as a promising approach to decompose the activations of Large Language Models (LLMs) into human-interpretable latents. In this paper, we pose two questions. First, to what extent do SAEs extract monosemantic and interpretable latents? Second, to what extent does varying the sparsity or the size of the SAE affect monosemanticity / interpretability? By investigating these questions in the context of a simple first-letter identification task where we have complete access to ground truth labels for all tokens in the vocabulary, we are able to provide more detail than prior investigations. Critically, we identify a problematic form of feature-splitting we call feature absorption where seemingly monosemantic latents fail to fire in cases where they clearly should. Our investigation suggests that varying SAE size or sparsity is insufficient to solve this issue, and that there are deeper conceptual issues in need of resolution.

Understanding the internal representations of large language models is crucial for ensuring their reliability and safety, with sparse autoencoders (SAEs) emerging as a promising interpretability approach. However, current SAE training methods face feature absorption, where features (or neurons) are absorbed into each other to minimize L_1 penalty, making it difficult to consistently identify and analyze model behaviors. We introduce Adaptive Temporal Masking (ATM), a novel training approach that dynamically adjusts feature selection by tracking activation magnitudes, frequencies, and reconstruction contributions to compute importance scores that evolve over time. ATM applies a probabilistic masking mechanism based on statistical thresholding of these importance scores, creating a more natural feature selection process. Through extensive experiments on the Gemma-2-2b model, we demonstrate that ATM achieves substantially lower absorption scores compared to existing methods like TopK and JumpReLU SAEs, while maintaining excellent reconstruction quality. These results establish ATM as a principled solution for learning stable, interpretable features in neural networks, providing a foundation for more reliable model analysis.

Sparse autoencoders (SAEs) are a technique for sparse decomposition of neural network activations into human-interpretable features. However, current SAEs suffer from feature absorption, where specialized features capture instances of general features creating representation holes, and feature composition, where independent features merge into composite representations. In this work, we introduce Orthogonal SAE (OrtSAE), a novel approach aimed to mitigate these issues by enforcing orthogonality between the learned features. By implementing a new training procedure that penalizes high pairwise cosine similarity between SAE features, OrtSAE promotes the development of disentangled features while scaling linearly with the SAE size, avoiding significant computational overhead. We train OrtSAE across different models and layers and compare it with other methods. We find that OrtSAE discovers 9% more distinct features, reduces feature absorption (by 65%) and composition (by 15%), improves performance on spurious correlation removal (+6%), and achieves on-par performance for other downstream tasks compared to traditional SAEs.

Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting neural networks by extracting the concepts represented in their activations. However, choosing the size of the SAE dictionary (i.e. number of learned concepts) creates a tension: as dictionary size increases to capture more relevant concepts, sparsity incentivizes features to be split or absorbed into more specific features, leaving high-level features missing or warped. We introduce Matryoshka SAEs, a novel variant that addresses these issues by simultaneously training multiple nested dictionaries of increasing size, forcing the smaller dictionaries to independently reconstruct the inputs without using the larger dictionaries. This organizes features hierarchically - the smaller dictionaries learn general concepts, while the larger dictionaries learn more specific concepts, without incentive to absorb the high-level features. We train Matryoshka SAEs on Gemma-2-2B and TinyStories and find superior performance on sparse probing and targeted concept erasure tasks, more disentangled concept representations, and reduced feature absorption. While there is a minor tradeoff with reconstruction performance, we believe Matryoshka SAEs are a superior alternative for practical tasks, as they enable training arbitrarily large SAEs while retaining interpretable features at different levels of abstraction.

Neutrino self-interactions beyond the standard model have profound implications in astrophysics and cosmology. In this work, we study an uncharted scenario in which one of the three neutrino species has a mass much smaller than the temperature of the cosmic neutrino background. This results in a relativistic component that significantly broadens the absorption feature on the astrophysical neutrino spectra, in contrast to the sharply peaked absorption expected in the extensively studied scenarios assuming a fully nonrelativistic cosmic neutrino background. By solving the Boltzmann equations for neutrino absorption and regeneration, we demonstrate that this mechanism provides novel sensitivity to sub-keV mediator masses, well below the traditional sim 1--100 MeV range. Future observations of the diffuse supernova neutrino background with Hyper-Kamiokande could probe coupling strengths down to g sim 10^{-8}, surpassing existing constraints by orders of magnitude. These findings open new directions for discoveries and offer crucial insights into the interplay between neutrinos and the dark sector.

Young exoplanets provide a critical link between understanding planet formation and atmospheric evolution. Direct imaging spectroscopy allows us to infer the properties of young, wide orbit, giant planets with high signal-to-noise. This allows us to compare this young population to exoplanets characterized with transmission spectroscopy, which has indirectly revealed the presence of clouds, photochemistry, and a diversity of atmospheric compositions. Direct detections have also been made for brown dwarfs, but direct studies of young giant planets in the mid-infrared were not possible prior to JWST. With two exoplanets around a solar type star, the YSES-1 system is an ideal laboratory for studying this early phase of exoplanet evolution. We report the first direct observations of silicate clouds in the atmosphere of the exoplanet YSES-1 c through its 9-11 micron absorption feature, and the first circumplanetary disk silicate emission around its sibling planet, YSES-1 b. The clouds of YSES-1 c are composed of either amorphous iron-enriched pyroxene or a combination of amorphous MgSiO3 and Mg2SiO4, with particle sizes of less than or equal to 0.1 micron at 1 millibar of pressure. We attribute the emission from the disk around YSES-1 b to be from submicron olivine dust grains, which may have formed through collisions of planet-forming bodies in the disk.

The James Webb Space Telescope (JWST) has uncovered low-luminosity active galactic nuclei (AGNs) at high redshifts of zgtrsim 4-7, powered by accreting black holes (BHs) with masses of sim 10^{6-8}~M_odot. One remarkable distinction of these JWST-identified AGNs, compared to their low-redshift counterparts, is that at least sim 20% of them present Halpha and/or Hbeta absorption, which must be associated with extremely dense (gtrsim 10^9~{rm cm}^{-3}) gas in the broad-line region or its immediate surroundings. These Balmer absorption features unavoidably imply the presence of a Balmer break caused by the same dense gas. In this Letter, we quantitatively demonstrate that a Balmer break can form in AGN spectra without stellar components, when the accretion disk is heavily embedded in dense neutral gas clumps with densities of sim 10^{9-11}~{rm cm}^{-3}, where hydrogen atoms are collisionally excited to the n=2 states and effectively absorb the AGN continuum at the bluer side of the Balmer limit. The non-stellar origin of a Balmer break offers a potential solution to the large stellar masses and densities inferred for little red dots (LRDs) when assuming that their continuum is primarily due to stellar light. Our calculations indicate that the observed Balmer absorption blueshifted by a few hundreds {rm km~s}^{-1} suggests the presence of dense outflows in the nucleus at rates exceeding the Eddington value. Other spectral features such as higher equivalent widths of broad Halpha emission and presence of OI lines observed in high-redshift AGNs including LRDs align with the predicted signatures of a dense super-Eddington accretion disk.

Recent JWST observations with NIRCam and MIRI of the ultra-short-period super-Earth 55 Cancri e indicate a possible volatile atmosphere surrounding the planet. Previous analysis of the NIRCam spectra suggested potential absorption features from CO2 or CO and significant sub-weekly variability. The MIRI low-resolution spectrum does not contain substantial features but was found to be consistent with effective heat redistribution models. In this work, we computed a grid of over 25000 self-consistent 1D forward models incorporating H-N-O-C-S-P-Si-Ti equilibrium chemistry and assessed plausible atmospheric compositions based on the current JWST data. Despite exhaustive analysis, the composition and properties of the atmosphere remain elusive. While our results statistically favour a global, hydrogen-free, nitrogen-dominated atmosphere enriched in PO and CO2, various alternative compositions, including H2O-,CO-, PH3-, or Si-bearing remain viable explanations. Unconstrained heat redistribution efficiency and absolute NIRCam flux are among the largest sources of uncertainty in our analysis. We also find that the heat redistribution factor and surface pressure are highly degenerate with atmospheric composition, and that these parameters cannot be independently constrained using current JWST observations. Furthermore, we show that the observed variability may arise from dynamic interactions between the atmosphere and an underlying magma ocean, driving rapid shifts in atmospheric chemistry and thermal emission. Our results highlight the importance of using self-consistent forward models when analysing novel JWST spectra with limited signal-to-noise ratios -- such as those of 55 Cancri e -- as it allows for a more comprehensive evaluation of potential atmospheric scenarios while also being less sensitive to subtle spectral differences than retrievals...

The field of exocomets has been built around the unmatched number of detections made in the circumstellar disc of the archetypal star Beta Pictoris. An exocomet detection in spectroscopy is identified by variable atomic absorption features in a stellar spectrum, associated with transiting gas in and trailing an exocomet coma. This paper presents the largest spectroscopic search for exocomet transits to date, which overcomes the limitations of biased samples of stars with debris discs, and instead looks through the approx7500 stars in the HARPS archive for signs of exocomets in the CaII doublet (H:396.847nm and K:393.366nm). The search resulted in 155 candidate stars, which after filtering for false positives (e.g. binaries, stellar activity, etc.), were cut down to 22 stars. These 22 stars are classified into Tier1, 2, and 3 exocomet candidates, reflecting the confidence level of their exocomet detection. Our two best candidates (Tier1: Beta Pictoris, HD172555) and four lower confidence candidates (Tier2: Gl1, HIP5158, HD94771, HR1996) are discussed, yielding a detection rate of 0.03% (Tier1 only) and 0.1% (Tier1 & 2) in the HARPS sample. Both Tier1 stars are known exocomet host stars. These two young A-type stars correspond to 0.4% of all A-types in the sample, suggesting that detecting signs of exocomet transits using CaII is more likely around young A-type stars. Reanalysing a past HARPS study, we found no evidence to support the previously claimed four exocomet detections, indicating either that those detections are not robust or that we are only sensitive to the strongest signals.

TOI-836 is a ~2-3 Gyr K dwarf with an inner super Earth (R=1.7 R_oplus, P=3.8 d) and an outer mini Neptune (R=2.6 R_oplus, P=8.6 d). JWST/NIRSpec 2.8--5.2 mum transmission spectra are flat for both planets. We present Keck/NIRSPEC observations of escaping helium for super-Earth b, which shows no excess absorption in the 1083 nm triplet to deep limits (<0.2%), and mini-Neptune c, which shows strong (0.7%) excess absorption in both visits. These results demonstrate that planet c retains at least some primordial atmosphere, while planet b is consistent with having lost its entire primordial envelope. Self-consistent 1D radiative-hydrodynamic models of planet c reveal that the helium excess absorption signal is highly sensitive to metallicity: its equivalent width collapses by a factor of 13 as metallicity increases from 10x to 100x solar, and by a further factor of 12 as it increases to 200x solar. The observed equivalent width is 88\% the model prediction for 100x metallicity, suggesting an atmospheric metallicity similar to K2-18b and TOI-270d, the first two mini-Neptunes with detected absorption features in JWST transmission spectra. We highlight the helium triplet as a potentially powerful probe of atmospheric composition, with complementary strengths and weaknesses to atmospheric retrievals. The main strength is its extreme sensitivity to metallicity in the scientifically significant range of 10--200x solar, and the main weakness is the enormous model uncertainties in outflow suppression and confinement mechanisms, such as magnetic fields and stellar winds, which can suppress the signal by at least a factor of ~several.