Gather Gen AI most useful tools, materials, publications and reports
| Report | Link | Date | Institution |
|---|---|---|---|
| Stanford AI index Report 2023 | Link | Stanford | |
| Sparks of Artificial General Intelligence: Early experiments with GPT-4 | Link | Microsoft | |
| A Survey of Large Language Models | Link | April 2023 | Renmin University, China & University of Montreal, Canada |
| Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond | Link | Amazon & many others | |
| A Cookbook of Self-Supervised Learning | Link | Meta & many others | |
| Let’s Verify Step by Step | Link | May 2023 | OpenAI |
| A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering | Link | May 2023 | Kyung Hee University and many |
| A Comprehensive Survey on Segment Anything Model for Vision and Beyond | Link | May 2023 | Hong Kong University of Science and Technology and many |
| On the Design Fundamentals of Diffusion Models: A Survey | Link | June 2023 | Durham University |
| Open LLM Leaderboard | Link | Update in real time | Huggingface |
| A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering | Link | May 2023 | JKyung Hee University and many |
| A Survey on Multimodal Large Language Models | Link | June 2023 | CST and many |
| Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey | Link | July 2023 | IIT |
| A Survey on Evaluation of Large Language Models | Link | July 2023 | Jilin University |
| Challenges and Applications of Large Language Models | Link | July 2023 | UCL and many |
| A Survey of Large Language Models in Medicine: Principles, Applications, and Challenges | Link | Sep 2023 | University of Oxford & Many |
| Large Language Models in Finance: A Survey | Link | Sep 2023 | Columbia University & Many |
| A Survey on Video Diffusion Models | Link | Oct 2023 | Fudan University & Many |
| Learn From Model Beyond Fine-Tuning: A Survey | Link | Oct 2023 | Wuhan University & Many |
| A Survey on Multimodal Large Language Models for Autonomous Driving | Link | Nov 2023 | Purdue University & Many |
| Green Edge AI: A Contemporary Survey | Link | Dec 2023 | Nanjing University |
| Efficient Large Language Models: A Survey | Link | Dec 2023 | Ohio State U |
| Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives | Link | Dec 2023 | Tsinghua University |
| Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code | Link | Dec 2023 | Shanghai Jiao Tong Universit, Ant Group |
| A Survey Of Reinforcement Learning From Human Feedback | Link | Dec 2023 | MCML Munich, Germany |
| Diffusion Models, Image Super-Resolution And Everything: A Survey | Link | Jan 2024 | Many |
| Exploring Large Language Model based Intelligent Agents: definitions, methods, and prospects | Link | Jan 2024 | The Chinese University of Hong Kong & Many |
| AI Alignment: A Comprehensive Survey | Link | Jan 2024 | Peking University & Many |
| Large Language Models for Robotics:Opportunities, Challenges, and Perspectives | Link | Jan 2024 | Many |
Alpaca Open Source Code Stanford March 2023
Dolly Open Source Code Databricks March 2023 Note: OK to use commercially
Vicuna Open Source Code UC Berkeley, CMU, Stanford, and UC San Diego March 2023
ChatPDF March 2023
Bard Google March 2023
Langchain Community Effort March 2023
Microsoft 365 Copilot Microsoft March 2023
AutoGPT Community Effort April 2023
Grounded SAM IDEA April 2023
DeepSpeed Chat Microsoft April 2023
AgentGPT Community Effort April 2023
MiniGPT King Abdullah University of Science and Technology April 2023
DeepFloyd IF Stability.ai April 2023
Open Llama Berkeley May 2023
SoftVC VITS Singing Voice Conversion Community May 2023
Falcon Tii May 2023
FinGPT Columbia University June 2023
UltraLM Tsinghua University June 2023
ChatLaw Peking University June 2023
LMFlow HK University of Science and Technology June 2023
GPT Store OpenAI Jan 2024
COS597G Understanding Large Language Models Princeton 2022
CS324 Large Language Models Stanford 2023
ChatGPT, LangChain and DS Courses Deeplearning.ai Jun 2023
Large Multimodal Models: Notes on CVPR 2023 Tutorial Microsoft Jun 2023
OpenAI Cookbook
Llama Index
PrivateGPT
Llama.cpp
petals
FlexGen
Flowise
Candle
ChatGPT Next Web
1. Data is still king - LLMs are great but if you don't have quality clean data you won’t go far.
2. Smaller models can be just as good as larger general models at specific tasks. And cheaper!
3. Fine-tuning is becoming cheaper.
4. Evaluation of LLMs is very hard - feels very subjective still.
5. Managed APIs are expensive.
6. "Traditional" ML isn't going anywhere.
7. Memory matters - for both serving and training.
8. Information retrieval w/ vector databases is becoming standard pattern.
9. Start w/ prompt engineering and push that to its limits before fine-tuning w/ smaller models.
10. Use agents/chains only when necessary. They are unruly.
11. Latency is critical for a good user experience.
12. Privacy is critical.
1. Prompt vs fine-tuning: The reliability of prompt engineering is still not enough, it is sensitive to certain prompts, and supervised fine-tuning (sft) remains a stable and efficient method.
2. The ability of open-source models and the gap with GPT-4 still lies in the complexity of the base model. Although the answer styles can be similar, the professional content and reasoning capabilities differ greatly. A better base model is still the key. Llama2 is about to be released and may be commercialized. The evaluation of the abilities of pretrained models is mainly based on a large set of tasks.
3. sft vs ppo: training and using ppo is still difficult, ppo can indeed improve results. There is a lot of academic research, but there are not many commercial applications yet. sft, when combined with good data, can replace ppo in most cases. Fine-tuning indeed improves tool usage and comprehensive summary response capabilities for specific domains.
4. Key points for sft data: diversity, not only the content and perspective of the problems should be diverse, but also the style of questioning. In terms of answers, not only the accuracy and truthfulness of the content are important, but also the format and style should be as expected (for example, clear and organized).
5. Regarding the link between pretraining and fine-tuning, when there is less data in the fine-tuning dataset, you can consider adding some pretraining datasets to increase stability. There are also many stages added between pretraining and fine-tuning, namely continuous pretraining. At this stage, you mainly train the domain you are focusing on, which will increase the recognition of this domain, such as training a dedicated coder, doing data and code mixed data training in pretraining, and doing coding code data training in continuous pretraining.
6. OpenLlama pre-training computing power: hundreds of GPUs, 1-2 months.
7. LangChain _ Vec DB is actually retrieval & tool use.
8. Fine-tuning vs Vec DB, fine-tuning is more about understanding large amounts of information, Vec DB is more about specific data details. There is no conflict between the two. Fine-tuning can consider turning the data retrieved by Vec DB into fine-tuning example data.
9. The optimization of the combination of fine-tuning and Vec DB, by generating related question keywords or sql through llm to retrieve relevant data from the database or knowledge base, then let llm summarize, and then put the entire process into the fine-tuning training dataset, will greatly enhance the effect. The problem of keyword matching can be solved by collecting a dataset of a few thousand examples, which will not be too difficult to teach llm this ability. Keyword matching is crucial for improving the tool use capability of fine-tuning. Pretrained models are not tools, they need to collect a large amount of data for fine-tuning to know that this task needs to search, and that task needs to use a mathematical model.
10. When keyword matching encounters more complex structured problems, llm can generate sql or python to solve it.
11. Many people are doing natural language processing to do data science, that is, llm generates sql to query relational databases, and this scenario can also be fine-tuned.
12. The reasoning, decision-making, and error correction of llm still depend on the base model capability. Generally, there is still a part of such data in the pretraining data of the model, but it has not been specialized. If you are biased towards traditional decision-making problems, you can strengthen this ability by specifically doing rewards and labels in fine-tuning.
13.Multimodal is a key direction. The open-source model community is starting to find that multimodal is not that difficult. The current mainstream method is to add multimodal during fine-tuning (not during pretraining). If you have computing power and data, fine-tuning with multimodal is better, it does not lose information. If it is a picture-to-text method, information will be lost. Diffusion models are still mainstream for generating images, because they have fewer computations and parameters, but autoregressive models have also been proven to have good image generation capabilities (palm e: first tokenize the picture and then use llm to generate the picture), but if the hardware improves further, there may be multimodal generation models that use autoregressive models, the model is simpler.
14. The long-tail problem (medical, autonomous driving and other critical scenarios) can be improved by collecting data similar to long-tail through simulated scenarios, usually data from accidents. For non-real-time scenarios like medical, it may still require human oversight.
1. For SFT, the quality and diversity of data are key (domain & task), as well as the way of training. Of course, having more data is good when the quality can be assured, but less is more when it can't. Task level generation will affect examples. Task level diversity is important, defining the intention of the instruction. The model's ability has basically been learned during pre-training, alignment is about teaching it the format of communication with the user. During FT, even if you only have a small amount of useful data (like 10), it is best to increase the data to 1,000 (a certain base number), which can give the model a good signal to learn from.
2. Basic observations of LLama 2: lack of code, logical reasoning, mathematical skills, multilingual and multimodal abilities, overdoing the alignment, the ecosystem should form quickly (many efforts are in progress). The 30B has stronger task generalization, the 7B is still relatively weak, and no noticeable progress has been observed in this regard from LLama 1 to 2.
3. The SFT results of LLama 2 are very good, Reject Sampling is helpful, which involves having the previous model generate a sample, ranking them, and then feeding the good ones back.
4. Continuous training in coding may require a guiding process supervision, which makes it easy for the model to learn. Unseen domains are mainly added to the model through additional training, not ft (ft is more about adjusting alignment format). Better base models may require less data for alignment. Direct additional training: code training is easier to fit, the loss will be lower; and additional code training will not affect the model's original natural language ability, if the data is not enough, just go through it a few more times.
5. User query distribution is not in line with the benchmarks we usually use, many companies have user query distribution, which has already created a barrier. During the annotation process, people need to act according to the assumed persona, then the data will be good.
6. Regarding the improvement of LLama 2's reasoning-intensive ability, llama's pre-training and ft did not target this improvement. 2T tokens may still be far from enough, it could be 4T, 5T, or the important data could be viewed several times. There is still a lot of room in LLama 2, saturation has not been seen. And it actually has a lot of knowledge in it, but this knowledge has not been played out, which is a problem of alignment.
7. Data mixsure, continuous training. When continuous training, it is best to mix general data with your domain data for training, which will reduce the probability of destroying the original ability of the model. LLaMA's web data mixture is already excellent, using methods like DoReMi or DRO (Distributional Robust Optimization) does not fill gaps, but simply takes care of the worse ones, not the expected mixture result.
8. SFT, rlhf comparison and combination. SFT is a subset and prelude to rlhf. There are no case-by-case comparative studies on these two yet.
9. MoE increases model capacity without changing latency. In the impossible situation of infinitely increasing model size, it is a way to increase capacity. Adding MoE initially will lose performance, so you need to first train to the original level.
10. One way to deploy Llama 2 into applications, which (llama 2 ) lacks mathematical ability, is to make it an agent, and when necessary, call a model with strong mathematical ability to solve problems.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent