Hello this is Zhang Di, An AI devloper at ShangHai AI Lab, and PhD students of Fudan Univ.
Former Full-time ML developer of Alibaba .Inc
Former M.Eng of USTC Robotics Lab and Internship at Ant Group, MIT Han Lab.
Hello this is Zhang Di, An AI devloper at ShangHai AI Lab, and PhD students of Fudan Univ.
Former Full-time ML developer of Alibaba .Inc
Former M.Eng of USTC Robotics Lab and Internship at Ant Group, MIT Han Lab.
Good afternoon,
I have refactored the code for MathBlackBox to be a Jupyter notebook. It requires you put in your own API key, by default it uses DeepSeek V2 Coder, although I cannot guarantee functionality because I ran out of API credits. Any OpenAI compatible API will work.
LLaMA_3B 2.ipynb.zip
I hope someone tries it and sees if it works properly.
The method seems very powerful but expensive. I'm wondering how is it compared with self-consistency under similar computational budgets?
小模型由于性能问题,很多时候没法对任务进行正确评估,这个你们是怎么解决的?
The work in the paper seems very powerful and can make significant progress on the 8B open-source model. Then, when using this method on a more powerful model and taking advantage of better instruction-following ability and reasoning ability, can relatively breakthrough mathematical reasoning results be achieved?
Hi! 😊 Your MathBlackBox project caught my eye. Amazing work on this python repository! ✨ Could you send me more details on Telegram? Also, please review my work and follow me on GitHub @nectariferous. Thanks!
After seeing this work, I read the paper and found that the effect is very good. When reading the code, I found that this line of code seems to cause the indicator to degenerate from pass@1 to pass@k. Is my point of view correct?
MathBlackBox/run_with_earlystopping.py
Line 769 in 390a894
I am not saying that pass@k is not a good indicator. The default evaluation indicator of gsm8k is usually equivalent to pass@1, but https://arxiv.org/pdf/2205.14318 also uses pass@k, and they are far from reaching this score. But if we can clearly mark the relationship between the value of k and the corresponding score, then we can better understand the paper.
Furthermore, it may be difficult to get the ground truth in reality, and pass@1 is actually more in line with reality. Do you know if there is a better way to evaluate pass@1?
If I understand it incorrectly, please kindly correct me.
Hi, the paper was a very interesting read and this technique seems to have a lot of potential. However, looking at the code - if I understand it correctly - I have noticed that a significant portion of it is dependent on the knowledge of the correct answer - the 'ground truth'. If this knowledge is not available to the program until the final result validation, how does the program perform then? Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.