Monday, January 6, 2025

human scRNA aging data

There are some human single cell aging data,

https://www.nature.com/articles/s41586-024-07606-7

https://www.nature.com/articles/s43587-024-00640-0

https://www.nature.com/articles/s41467-020-17876-0?fromPaywallRec=false

https://pmc.ncbi.nlm.nih.gov/articles/PMC10306289/#_ad93_

Human PBMC scRNA-seq–based aging clocks reveal ribosome to inflammation balance as a single-cell aging hallmark and super longevity

Hongming Zhu ^1,^†, Jiawei Chen ^2,^†, Kangping Liu ^2,^†, Lei Gao ^1,³, Haiyan Wu ¹, Liangliang Ma ⁴, Jieru Zhou ⁴, Zhongmin Liu ^1,^*, Jing-Dong J Han ^2,^*

Saturday, January 4, 2025

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

https://arxiv.org/abs/2412.14135

[Submitted on 18 Dec 2024]

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, Xipeng Qiu

OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-level performances on many challanging tasks that require strong reasoning this http URL has claimed that the main techinique behinds o1 is the reinforcement learining. Recent works use alternative approaches like knowledge distillation to imitate o1's reasoning style, but their effectiveness is limited by the capability ceiling of the teacher model. Therefore, this paper analyzes the roadmap to achieving o1 from the perspective of reinforcement learning, focusing on four key components: policy initialization, reward design, search, and learning. Policy initialization enables models to develop human-like reasoning behaviors, equipping them with the ability to effectively explore solution spaces for complex problems. Reward design provides dense and effective signals via reward shaping or reward modeling, which is the guidance for both search and learning. Search plays a crucial role in generating high-quality solutions during both training and testing phases, which can produce better solutions with more computation. Learning utilizes the data generated by search for improving policy, which can achieve the better performance with more parameters and more searched data. Existing open-source projects that attempt to reproduce o1 can be seem as a part or a variant of our roadmap. Collectively, these components underscore how learning and search drive o1's advancement, making meaningful contributions to the development of LLM.

AI model editing techniques

chatgpt output, un-edited. Some wrong links and erorrs are apparent.

1. Fine-Tuning

Definition: Adjusting pre-trained models by retraining them on a specific dataset to tailor them to a particular task or domain.
References:
- Fine-Tuning Language Models
- Full and Parameter-Efficient Fine-Tuning

2. Prompt Engineering

Definition: Crafting specific inputs (prompts) to guide the behavior of large language models without altering their parameters.
References:
- Prompting for NLP
- Chain-of-Thought Prompting

3. Model Editing via Retrieval-Augmented Generation (RAG)

Definition: Integrating external databases or retrieval systems to improve or adapt the model's outputs without direct parameter changes.
References:
- RAG Explained
- Dynamic Knowledge Updates

4. Knowledge Injection

Definition: Incorporating domain-specific knowledge into a model post-training.
References:
- Knowledge-Integrated Language Models
- Knowledge Graph Integration

5. Soft Prompt Tuning

Definition: Learning a set of prompt tokens that adjust the behavior of pre-trained models without altering core weights.
References:
- Soft Prompt Tuning Explained
- Prompt-Based Learning

6. Modular Transfer Learning

Definition: Dividing models into modules (e.g., embeddings, encoders, decoders) and only updating or replacing specific components.
References:
- Modular Learning for NLP
- Parameter-Efficient Transfer Learning

7. Dynamic Reweighting

Definition: Adjusting the influence of certain parts of the model during inference based on specific tasks or inputs.
References:
- Sparsity-Based Optimization
- Attention Mechanisms

8. Model Surgery

Definition: Directly modifying neural network weights, layers, or architectures post-training.
References:
- MEMIT: Model Surgery Techniques
- Layerwise Editing

9. Continual Learning

Definition: Allowing a model to learn new information over time without forgetting prior knowledge.
References:
- Elastic Weight Consolidation
- Continual Learning Overview

10. Gradient Editing

Definition: Directly modifying gradients during training to induce specific behaviors or rectify known issues.
References:
- Gradient Editing Explained
- Debugging Neural Models

11. Reinforcement Learning from Human Feedback (RLHF)

Definition: Using human evaluations to fine-tune models, particularly for aligning AI with desired ethical or stylistic outcomes.
References:
- RLHF Overview
- OpenAI's Use of RLHF

12. Model Patching

Definition: Adding or replacing specific components in a model with updated or improved modules.
References:
- Knowledge Patch
- Error Correction Patching

13. Parameter-Free Updating

Definition: Techniques like black-box optimization or external decision systems that modify behavior without changing core parameters.
References:
- Black-Box Optimization
- Decision-Based Editing

14. Memory Editing

Definition: Directly modifying or updating specific "memories" in a model, allowing it to adjust responses to certain inputs or queries without retraining.
Techniques:
- MEMIT (Model Editing Made Informed by Targeting)
- ROME (Rank-One Model Editing)
References:

15. Multi-Modal Model Editing

Definition: Modifying models trained on multi-modal data (e.g., text and images) for domain-specific applications.
References:
- Multi-Modal Learning
- Vision-Language Consistency

16. Federated Learning Adjustments

Definition: Decentralized learning where updates are based on data from multiple users without directly sharing datasets.
References:
- Federated Learning Overview
- Privacy-Preserving Federated Learning

17. Meta-Learning (Learning to Learn)

Definition: Training models to quickly adapt to new tasks with minimal data by leveraging meta-learning algorithms.
References:
- MAML: Model-Agnostic Meta-Learning
- Meta-Learning Overview

This version now includes Memory Editing with references to emerging techniques like MEMIT and ROME for direct manipulation of model-specific knowledge.

Friday, January 3, 2025

editable neural networks in health science

Meng, K., Bau, D., Andonian, A. & Belinkov, Y. Locating and editing factual associations in GPT.

Adv. Neural Inf. Process. Syst. (2022). at

<https://proceedings.neurips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-

Abstract-Conference.html>

26. Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y. & Bau, D. Mass-editing memory in a

transformer. in International Conference on Learning Representations (arxiv.org, 2023). at

<https://arxiv.org/abs/2210.07229>

27. Mitchell, E., Lin, C., Bosselut, A., Manning, C. D. & Finn, C. Memory-Based Model Editing at

Scale. in Proceedings of the 39th International Conference on Machine Learning (eds. Chaudhuri,

K., Jegelka, S., Song, L., Szepesvari, C., Niu, G. & Sabato, S.) 162, 15817–15831 (PMLR, 17--23

Jul 2022).

28. Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y. & Ghassemi, M. Aging with GRACE:

Lifelong Model Editing with Discrete Key-Value Adaptors. in Advances in Neural Information

Processing Systems (2023). at <https://arxiv.org/abs/2211.11031>

29. Mitchell, E., Lin, C., Bosselut, A., Finn, C. & Manning, C. Fast model editing at scale. in

International Conference on Learning Representations (arxiv.org, 2022). at

<https://arxiv.org/abs/2110.11309>

30. Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S. & Babenko, A. Editable Neural Networks. in

International Conference on Learning Representations (2020). at <http://arxiv.org/abs/2004.00345>

31. De Cao, N., Aziz, W. & Titov, I. Editing Factual Knowledge in Language Models. in Proceedings of

the 2021 Conference on Empirical Methods in Natural Language Processing 6491–6506

(Association for Computational Linguistics, 2021).

32. Zhong, Z., Wu, Z., Manning, C. D., Potts, C. & Chen, D. MQuAKE: Assessing Knowledge Editing

in Language Models via Multi-Hop Questions. arXiv [cs.CL] (2023). at

<http://arxiv.org/abs/2305.14795>

33. Cohen, R., Biran, E., Yoran, O., Globerson, A. & Geva, M. Evaluating the ripple effects of

knowledge editing in language models. Trans. Assoc. Comput. Linguist. 12, 283–298 (2023).

De Cao, N., Aziz, W. & Titov, I. Editing Factual Knowledge in Language Models. arXiv [cs.CL]

(2021). at <http://arxiv.org/abs/2104.08164>

35. Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y. & Bau, D. Mass-Editing Memory in a

Transformer. arXiv [cs.CL] (2022). at <http://arxiv.org/abs/2210.07229>

36. Mitchell, E., Lin, C., Bosselut, A., Finn, C. & Manning, C. D. Fast Model Editing at Scale. arXiv

[cs.LG] (2021). at <http://arxiv.org/abs/2110.11309>

37. Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y. & Ghassemi, M. Aging with GRACE:

Lifelong Model Editing with Key-Value Adaptors. (2022). at

<https://openreview.net/pdf?id=ngCT1EelZk>

Language Models: Problems, Methods, and Opportunities. arXiv [cs.CL] (2023). at

<http://arxiv.org/abs/2305.13172>

41. Hase, P., Hofweber, T., Zhou, X., Stengel-Eskin, E. & Bansal, M. Fundamental problems with model

editing: How should rational belief revision work in LLMs? arXiv [cs.CL] (2024). at

<https://scholar.google.com/citations?view_op=view_citation&hl=en&citation_for_view=FO90FgM

AAAAJ:M3ejUd6NZC8C>

42. Cheng, S., Tian, B., Liu, Q., Chen, X., Wang, Y., Chen, H. & Zhang, N. Can We Edit Multimodal

Large Language Models? in Proceedings of the 2023 Conference on Empirical Methods in Natural

Language Processing (eds. Bouamor, H., Pino, J. & Bali, K.) 13877–13888 (Association for

Computational Linguistics, 2023).

Here are the URLs for the specified papers:

1. **Locating and editing factual associations in GPT**

Meng, K., Bau, D., Andonian, A. & Belinkov, Y. (2022).

[Link to Paper](https://proceedings.neurips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html)

2. **Mass-editing memory in a transformer**

Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y. & Bau, D. (2023).

[Link to Paper](https://arxiv.org/abs/2210.07229)

3. **Memory-Based Model Editing at Scale**

Mitchell, E., Lin, C., Bosselut, A., Manning, C. D. & Finn, C. (2022).

[Link to Paper](https://proceedings.mlr.press/v162/mitchell22a.html)

4. **Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors**

Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y. & Ghassemi, M. (2023).

[Link to Paper](https://arxiv.org/abs/2211.11031)

5. **Fast model editing at scale**

Mitchell, E., Lin, C., Bosselut, A., Finn, C. & Manning, C. D. (2022).

[Link to Paper](https://arxiv.org/abs/2110.11309)

6. **Editable Neural Networks**

Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S. & Babenko, A. (2020).

[Link to Paper](http://arxiv.org/abs/2004.00345)

7. **Editing Factual Knowledge in Language Models**

De Cao, N., Aziz, W. & Titov, I. (2021).

[Link to Paper](http://arxiv.org/abs/2104.08164)

8. **MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions**

Zhong, Z., Wu, Z., Manning, C. D., Potts, C. & Chen, D. (2023).

[Link to Paper](http://arxiv.org/abs/2305.14795)

9. **Evaluating the ripple effects of knowledge editing in language models**

Cohen, R., Biran, E., Yoran, O., Globerson, A. & Geva, M. (2023).

[Link to Paper](https://transacl.org/ojs/index.php/tacl/article/view/3736)

10. **Language Models: Problems, Methods, and Opportunities**

(2023).

[Link to Paper](http://arxiv.org/abs/2305.13172)

11. **Fundamental problems with model editing: How should rational belief revision work in LLMs?**

Hase, P., Hofweber, T., Zhou, X., Stengel-Eskin, E. & Bansal, M. (2024).

[Link to Paper](https://scholar.google.com/citations?view_op=view_citation&hl=en&citation_for_view=FO90FgMAAAAAJ:M3ejUd6NZC8C)

12. **Can We Edit Multimodal Large Language Models?**

Cheng, S., Tian, B., Liu, Q., Chen, X., Wang, Y., Chen, H. & Zhang, N. (2023).

[Link to Paper](https://arxiv.org/abs/2305.14795)

Citations:

[1] https://proceedings.neurips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b

Thursday, January 2, 2025

Autonomous AI-Driven Drug Discovery

A Framework for Autonomous AI-Driven Drug Discovery

Douglas W Selinger, , Eleni Stylianou, , ,

doi: https://doi.org/10.1101/2024.12.17.629024

https://www.biorxiv.org/content/10.1101/2024.12.17.629024v2