Comments (6)
Interesting - it's not an issue I've really hit since generally I'm trying to get a lot of output (more stuff per prompt = more bang for your buck). It could help explain why I'm getting less useful outputs from some things though, it doesn't have enough space to create the right hierarchies.
The question I guess is whether it will output needlessly large results. It definitely doesn't fill the full amount each time, broadly I've found gpt-4 to be quite good at doing just what it needs to.
in other words, if max token length is strictly always better, why would openai want us to reduce it down, ever?
TBF it's important when it comes to cost & more, particularly as the context size increases. 8k is a lot, if you really just need a short response it's good to be able to cap it.
from developer.
thank you! but quick followup - is it actually always a good thing to request the max token length? will it incentivize the model to hallucinate code it doesnt need? smol is better, no?
in other words, if max token length is strictly always better, why would openai want us to reduce it down, ever?
from developer.
Based on this discussion on the OpenAI community site, max token changes based on the model you're using and if you exceed the max token for a particular model, it will return a 400.
Not sure why they're having the API user set it, but there you go.
from developer.
yea i'm pretty sure choice of max token affects the output. shorter makes it try to end sooner. this is feature not a bug
from developer.
8k is a lot, if you really just need a short response it's good to be able to cap it.
yes but notice that the shortness of the requested tokens actually does somewhat affect the generated output - openai is doing something other than next token prediction, that shortens the answer when its about to go over (this is a weak effect, ive observed this rule been broken too, but ive definitely noticed it)
from developer.
yea i'm pretty sure choice of max token affects the output. shorter makes it try to end sooner. this is feature not a bug
I found this openai community thread reply:
I asked the support and they clarified that GPT-3 will not attempt to create shorter texts with a smaller max_tokens value. The text will indeed just be cut off. So in my case, I guess it makes sense to use a higher value to have more “wiggle room”.
On the other hand I can imagine that OpenAI might do some funky stuff in the background, e.g. prefixes your prompt with "Answer in {max_tokens*0.75} or less words." This would not do exactly that as it's been shown that the model is not THAT capable but it might influence the length.
from developer.
Related Issues (20)
- Please add releases HOT 1
- No Module named 'openai' HOT 5
- args not defined error HOT 2
- stream_handler error: 'content' {} ←[0m ←[0m ←[0m ←[0m ←[0m ←[0m
- NameError: name 'args' is not defined HOT 3
- pip install modal-client error
- OpenAI key error HOT 4
- JavaScript files contain code fences
- Add a new feature - test and debug to create self-healing code
- Timeout issue
- Error: The system cannot find the file specified. HOT 1
- Can anyone fix the Basic Usate in Git Repo mode? HOT 1
- PyPI Package is Broken
- Run demo through pip package, no response, no output, no error reported, program automatically ends HOT 1
- Is there a way to bind with azure openai?
- Errors when running `poetry run python main.py --prompt "poetry prompt"`
- Feature Proposal: New feature proposal HOT 1
- Test
- how to use codellama instead of openai api? HOT 3
- can someone update this with a dockerfile for ubuntu 22.04 so we can run this? currently it doesnt work
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from developer.