Comments (2)
Thanks for the prompt response, loving the tool. Will turn this into a blog post next week!
from h2o-llmstudio.
Hi @tomasonjo
The reason is that you need to align the prompting.
With default setting, we are adding an EOS
token at the end of the prompt. So you would need to call it like that:
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("tomasonjo/movie-generator-small")
model = AutoModelForCausalLM.from_pretrained("tomasonjo/movie-generator-small")
model.half().cuda()
# need to match the input prompt how you are doing it in the LLM Studio Prompt
inputs = tokenizer("\nCreate a Cypher statement to answer the following question:What movies did Tom Hanks star in?<|endoftext|>", return_tensors="pt", add_special_tokens=False).to("cuda")
tokens = model.generate(
**inputs,
max_new_tokens=256,
temperature=0.3,
repetition_penalty=1.2,
num_beams=4
)[0]
tokens = tokens[inputs["input_ids"].shape[1]:]
print(tokenizer.decode(tokens, skip_special_tokens=True))
Which outputs
MATCH (d:Person {name: 'Tom Hanks'})-[:ACTED_IN]->(m:Movie) RETURN {movie: m.title} AS result
An additional newline at the start also usually works well and I added it above, you can play with the prompt and the inference settings a bit.
We have an open issue to generate an automatic model card on HF to exactly describe how a prompt needs to look like based on the settings of the experiment: #5
from h2o-llmstudio.
Related Issues (20)
- [FEATURE] Use Free Form Chat Template to Define Input Styling
- [CHORE] Upgrade dependencies to fix vulnerabilities
- [FEATURE] Add danube2 to default model list
- [FEATURE] Add additional digit of precision for specifying learning rate and other parameters HOT 1
- [BUG] Scheduler should consider gradient accumulation while assigning `epoch_steps`?
- [CODE IMPROVEMENT] Avoid gradient sync during gradient accumulation
- [CODE IMPROVEMENT] Sort data files alphabetically
- ValueError: invalid literal for int() with base 10: ‘Failed to initialize NVML: Unknown Error’ HOT 8
- [BUG] Pipenv missing as a requirement for the `make llmstudio` command HOT 1
- [BUG] Exporting / downloading model larger, than VRAM available (trained with DeepSpeed) fails HOT 8
- [FEATURE] Support for minimum learning rate HOT 1
- [FEATURE] Mixed Precision Dtype
- [BUG] UI freezes when using "Stop streaming" button with a text in the input box
- [FEATURE] Fine-tune CohereForCausalLM Models
- [BUG] Chat window generation parameters not updated
- [FEATURE] Random validation sample for chat interface
- [FEATURE] Add ability to specify dataset problem type during data import HOT 1
- [FEATURE] Option for not saving checkpoint HOT 1
- [FEATURE] Use local LLM deployment as Judge HOT 1
- Compare Zero-Epoch Prediction with Fine-Tuned Prediction as well as Validaiton Score Comparison
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from h2o-llmstudio.