Paper List
-
Developing the PsyCogMetrics™ AI Lab to Evaluate Large Language Models and Advance Cognitive Science
This paper addresses the critical gap between sophisticated LLM evaluation needs and the lack of accessible, scientifically rigorous platforms that in...
-
Equivalence of approximation by networks of single- and multi-spike neurons
This paper resolves the fundamental question of whether single-spike spiking neural networks (SNNs) are inherently less expressive than multi-spike SN...
-
The neuroscience of transformers
提出了Transformer架构与皮层柱微环路之间的新颖计算映射，连接了现代AI与神经科学。
-
Framing local structural identifiability and observability in terms of parameter-state symmetries
This paper addresses the core challenge of systematically determining which parameters and states in a mechanistic ODE model can be uniquely inferred ...
-
Leveraging Phytolith Research using Artificial Intelligence
This paper addresses the critical bottleneck in phytolith research by automating the labor-intensive manual microscopy process through a multimodal AI...
-
Neural network-based encoding in free-viewing fMRI with gaze-aware models
This paper addresses the core challenge of building computationally efficient and ecologically valid brain encoding models for naturalistic vision by ...
-
Scalable DNA Ternary Full Adder Enabled by a Competitive Blocking Circuit
This paper addresses the core bottleneck of carry information attenuation and limited computational scale in DNA binary adders by introducing a scalab...
-
ELISA: An Interpretable Hybrid Generative AI Agent for Expression-Grounded Discovery in Single-Cell Genomics
This paper addresses the critical bottleneck of translating high-dimensional single-cell transcriptomic data into interpretable biological hypotheses ...
A Standardized Framework for Evaluating Gene Expression Generative Models
University of Cambridge | Wellcome Sanger Institute | Sapienza University of Rome | ISTI-CNR
30秒速读
IN SHORT: This paper addresses the critical lack of standardized evaluation protocols for single-cell gene expression generative models, where inconsistent metric implementations and computation spaces make cross-study comparisons impossible.
核心创新
- Methodology Introduces GGE, the first unified Python framework with explicit computation space parameterization (raw, PCA, DEG-restricted) for standardized evaluation of generative models.
- Methodology Proposes perturbation-effect correlation metric that measures direction and magnitude of perturbation responses rather than raw expression correlation, focusing evaluation on biologically relevant signals.
- Methodology Demonstrates that Wasserstein distance values vary by nearly an order of magnitude (17.2 to 104.3) depending solely on computation space, quantifying the standardization problem.
主要结论
- Metric values vary substantially with implementation choices: W₂ distance ranges from 17.2 (PCA-25) to 104.3 (raw space) on identical data, highlighting critical need for standardization.
- DEG selection strategy affects correlation metrics: top-20 DEG selection yields Pearson correlation of 0.614±0.066 vs strict threshold selection (lfc>1, p<0.01) yielding 0.506±0.217 on Norman dataset.
- Perturbation-effect correlation in DEG space provides biologically meaningful evaluation, focusing on genes that actually respond to perturbations rather than steady-state background expression.
摘要: The rapid development of generative models for single-cell gene expression data has created an urgent need for standardised evaluation frameworks. Current evaluation practices suffer from inconsistent metric implementations, incomparable hyperparameter choices, and a lack of biologically-grounded metrics. We present Generated Genetic Expression Evaluator (GGE), an open-source Python framework that addresses these challenges by providing a comprehensive suite of distributional metrics with explicit computation space options and biologically-motivated evaluation through differentially expressed gene (DEG)-focused analysis and perturbation-effect correlation, enabling standardized reporting and reproducible benchmarking. Through extensive analysis of the single-cell generative modeling literature, we identify that no standardized evaluation protocol exists. Methods report incomparable metrics computed in different spaces with different hyperparameters. We demonstrate that metric values vary substantially depending on implementation choices, highlighting the critical need for standardization. GGE enables fair comparison across generative approaches and accelerates progress in perturbation response prediction, cellular identity modeling, and counterfactual inference.