blinkdl/lm-trick-questions

Here we collect trick questions and failed tasks for open source LLMs to improve them.

BlinkDL

A minimalist deep learning library in Javascript using WebGL + asm.js. Runs in your browser.

Currently it is a proof-of-concept (inference only). Note: Convolution is buggy when memories overlap.

The WebGL backend is powered by weblas: https://github.com/waylonflinn/weblas.

Example

https://withablink.coding.me/goPolicyNet/ : a weiqi (baduk, go) policy network in AlphaGo style:

const N = 19;
const NN = N * N;
const nFeaturePlane = 8;
const nFilter = 128;

const x = new BlinkArray();
x.Init('weblas');
x.nChannel = nFeaturePlane;
x.data = new Float32Array(nFeaturePlane * NN);
for (var i = 0; i < NN; i++)
    x.data[5 * NN + i] = 1; // set feature plane for empty board

// pre-act residual network with 6 residual blocks
const bak = new Float32Array(nFilter * NN);
x.Convolution(nFilter, 3);
x.CopyTo(bak);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.Add(bak).CopyTo(bak);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.Add(bak).CopyTo(bak);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.Add(bak).CopyTo(bak);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.Add(bak).CopyTo(bak);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.Add(bak).CopyTo(bak);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.BatchNorm().ReLU().Convolution(nFilter, 3);
x.Add(bak);
x.BatchNorm().ReLU().Convolution(1, 1).Softmax();

Usage

<script src='weblas.js' type='text/javascript'></script>
<script src='BlinkDL.js' type='text/javascript'></script>

Todo

• Convolution (3x3_pad_1 and 1x1), BatchNorm, ReLU, Softmax
• Pooling layer
• FC layer
• Strided convolution
• Transposed convolution
• Webworker and async
• Faster inference with weblas pipeline, WebGPU, WebAssembly
• Memory manager
• Training

lm-trick-questions's Issues

"Repeat this sentence exactly as it is"

Summary

It struggles to return back a given sentence exactly as it is

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
return this sentence exactly as it is
Input:
hello there how are you
Output:
"Hello there, how are you?"

Example 2

Instruction:
return this sentence exactly as it is
Input:
what is 3+4
Output:
6

Example 3

Instruction:
return this sentence exactly as it is
Input:
there are eleven cars in the street, 2 bicycles and a train is also passing nearby
Output:
There are eleven cars in the street, two bicycles and a train passing nearby.

Sentence reversal

Summary

It struggles to return a sentence backwards

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
return this sentence backwards
Input:
hello there how are you
Output:
How are you this hello there

Example 2

Instruction:
return this sentence backwards
Input:
what is 3+4
Output:
return this sentence backwards is 3+4

Example 3

Instruction:
return this sentence backwards
Input:
there are eleven cars in the street, 2 bicycles and a train is also passing nearby
Output:
The sentence "There are eleven cars in the street, 2 bicycles and a train is also passing nearby" can be written as "11 cars, 2 bicycles, and a train are passing nearby."

"Write a sentence using exactly n words"

Summary

It struggles to write a sentence using a specified number of words

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
write a sentence using exactly 3 words
Output:
With great effort, the group succeeded in completing the project.

Example 2

Instruction:
write a sentence using exactly 5 words
Output:
The room was filled with the scent of freshly cut flowers.

Example 3

Instruction:
write a sentence using exactly 11 words
Output:
The sun rose in the east, casting a warm glow over the city.

Integer Multiplication

Summary

It struggles multiplying integers over 2-3 digits large

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
what is 23*83
Output:
23 * 83 = 2089

Example 2

Instruction:
what is 77*39
Output:
77 * 39 = 3993

Example 3

Instruction:
what is 907*32
Output:
907 * 32 = 29504

Simple math word problems

Summary

It struggles struggles with simple math word problems (that include chain of thought)

Training it on these word problems could help RWKV with multistep reasoning as stated in this article by OpenAI:
https://openai.com/research/solving-math-word-problems
They also provide their dataset:
https://github.com/openai/grade-school-math

(note: Issue created by discord user diabolo96)

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
This is a math word problem. You will solve it and provide a simple step by step verification.
Input:
Tim grows 5 trees. Each year he collects 6 lemons from each tree. How many lemons does he get in a decade?
Output:
Let's start by setting up the problem. In a decade, Tim will collect 6 trees * 10 lemons per tree = 60 lemons from each tree. Therefore, in a decade, he will collect 60 * 10 = 600 lemons.
The answer: 600.

Expected Output:
Tim collects a total of 5*6= 30 lemons per tree.
So in a decade, he gets 30*10= 300 lemons.

Word counting

Summary

It struggles counting the number of words in a sentence

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
Count the number of words in the sentence
Input:
hello there how are you
Output:
The sentence "hello there how are you" has 3 words.

Example 2

Instruction:
Count the number of words in the sentence
Input:
Count the number of words in the sentence
Output:
The sentence has 10 words.

Example 3

Instruction:
Count the number of words in the sentence
Input:
what is three plus four?
Output:
The sentence "what is three plus four?" has two words: "what" and "is".

Word replacement

Summary

It fails to return the input sentence with a word replaced

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
return the sentence but replace "how" with "witch"
Input:
hello there how are you
Output:
hello witch how are you?

Example 2

Instruction:
return the sentence but replace "nice" with "airplane"
Input:
the weather is nice today
Output:
The weather is airplane today.

Example 3

Instruction:
return the sentence but replace "street" with "orange"
Input:
there are eleven cars in the street, 2 bicycles and a train is also passing nearby
Output:
There are eleven cars in the street, two bicycles and a train passing nearby.

"Classify each line as either a Statement or a Question."

Summary

It struggles classifying a list of sentences, all given at once, as either statements or questions

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
Classify each of the following lines of text as either a Statement or a Question.
Input:
1 How can I help you?
2 I was wondering about my application.
3 I have a class.
4 Where do I purchase textbooks?
5 How do i add classes?
Output:
1. Statement
2. Question
3. Statement
4. Question
5. Statement

Example 2

Instruction:
Classify each of the following lines of text as either a Statement or a Question.
Input:
1 Bricks are a building material.
2 He asked "how are you doing?"
3 I remember.
4 It's one of the best.
5 What is it?
Output:
1. Statement
2. Question
3. Statement
4. Statement
5. Statement

Example 3

Instruction:
Classify each of the following lines of text as either a Statement or a Question.
Input:
1 I like bananas.
2 I love refrigerators.
3 I have a giant plate of spaghetti?
4 There are eleven cars in the street, 2 bicycles and a train is also passing nearby.
5 Hi?
Output:
Statement: 1, 2, 3, 4
Question: 5

Word Sense Disambiguation

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Instruction:

I will provide you with a word with context and some definitions. You will need to choose the correct definition.

Input:

Word: "idea"
Context: "The idea behind the lottery is to raise money for good causes."
Definitions: [a belief or opinion] [a purpose or reason for doing something] [a suggestion or plan for doing something] [an understanding, thought, or picture in your mind]

Output:

The word "idea" in the context of the lottery refers to a belief or opinion. The definition is related to understanding, thought, or picture in your mind. The correct definition is "an understanding, thought, or picture in your mind."

Expected output: [a purpose or reason for doing something]

Puzzle

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Instruction:

To solve the puzzle.

Input:

A hunter walks one mile south, one mile east, and one mile north and ends up right back where he started. Where is the hunter on the earth?

Output:

The hunter is on the equator.

Expected output:

North Pole, or a circle around the South Pole with a radius of approximately 1 + 1/ 2π miles (1.16 miles) from the pole, you can start from any point within the circle. This means that your walk south will take you to very close to the South Pole, your walk east will take you on a complete circle around the pole, and your walk north will take you back where you started.

"Return back this list of words"

Summary

It struggles to return back a list of words.
(that look like a python array)

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
return back this list of words
Input:
[hello,there,how,are,you]
Output:
[Hello, There, How are you]

Example 2

Instruction:
return back this list of words
Input:
[abc,dsd,aoho,oo]
Output:
['a', 'o', 'd', 's']

Example 3

Instruction:
return back this list of words
Input:
[there,are,eleven,cars,in,the,street,2,bicycles,and,a,train,is,also,passing,nearby]
Output:
There are eleven cars in the street, two bicycles nearby, and a train passing by.

Sentence capitalization

Summary

It fails to return the input sentence capitalized

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
return back this sentence capitalized
Input:
hello there how are you
Output:
Hello there! How are you?

Example 2

Instruction:
return back this sentence capitalized
Input:
The weather is nice today
Output:
The weather is nice today.

Example 3

Instruction:
return back this sentence capitalized
Input:
a blue car
Output:
The sentence "a blue car" is a complete sentence.

Integer Addition

Summary

It struggles adding integers over 2-3 digits large

Model version:

RWKV-4-Raven-7B-v10-Eng99%-Other1%-20230418-ctx8192

Example 1

Instruction:
what is 345+23
Output:
345 + 23 = 358

Example 2

Instruction:
what is 325+754
Output:
859

Example 3

Instruction:
what is 555+888
Output:
555 + 888 = 1353