GenoTEX

GenoTEX: A Benchmark for Evaluating LLM-Based Exploration of Gene Expression Data in Alignment with Bioinformaticians

Haoyang Liu, Haohan Wang

https://arxiv.org/abs/2406.15341 

https://arxiv.org/abs/2402.12391

https://github.com/Liu-Hy/GenoTex

Paving a New Era of AI-Driven Science: Benchmark and Agent Solutions to Automate End-to-End Scientific Discovery

🔭 Imagine AI agents working as scientists, handling every step of research from picking datasets to creating hypotheses, just like bioinformaticians do. These agents change how AI is used, taking on detailed analyses and result interpretation—tasks usually done by human experts.

🚀 Building upon this innovative concept, we are excited to unveil GenoTEX, a transformative benchmark for automating scientific discovery processes in genomics, focusing on gene expression data. GenoTEX is designed not merely as a dataset but as a comprehensive suite to evaluate and enhance AI-driven methods in genomics data analysis. By establishing this benchmark, we aim to pioneer advancements where our AI agents—our GenoAgents—emulate bioinformaticians to perform comprehensive, end-to-end scientific investigations.

🧬 About GenoTEX:
GenoTEX covers the full cycle of scientific research, from selecting and preprocessing data to performing detailed statistical analysis. The benchmark follows a standardized pipeline, checked by expert bioinformaticians, to ensure the AI-generated results are accurate and reliable. Importantly, the datasets in GenoTEX are manually curated, involving 181 relevant datasets with 163 successfully preprocessed, showcasing over 71,669 lines of manually written code for analysis. This benchmark provides extensive annotated code and results for 1,146 gene identification problems—both unconditional (82) and conditional (1,064)—which serve as a robust foundation for developing and evaluating AI methods in genomics.

🤖 Introducing GenoAgents:
Alongside GenoTEX, we introduce GenoAgents, a suite of AI-driven agents crafted to tackle the nuanced tasks traditionally reserved for human experts. These agents are designed with capabilities that include context-aware planning, iterative correction, and domain expertise integration, simulating a collaborative environment similar to a team of human researchers. Our evaluation shows that GenoAgents can automate the process of gene expression data analysis with good overall accuracy, presenting a promising method for future research in genomics.

Our work highlight the potential of LLM-based approaches to significantly reduce the labor-intensive aspects of gene data analysis. We invite the scientific and tech community to explore GenoTEX and participate in evolving this benchmark. Your insights and contributions are crucial as we strive to refine AI applications in real-world scientific contexts.

Benchmark arXiv link: https://lnkd.in/gkZiNehN
previous workshop version: https://lnkd.in/gXzXcEX6
GitHub link: https://lnkd.in/gnrMni3u