bigdata-ustc / eduktm Goto Github PK
View Code? Open in Web Editor NEWThe Model Zoo of Knowledge Tracing Models
License: Apache License 2.0
The Model Zoo of Knowledge Tracing Models
License: Apache License 2.0
LBKT的junyi数据预处理,请问可以开源吗谢谢,我按照论文预处理方法还原的过程中总是无法达到,4316340条记录的同时,问题只有701和技能只有39
(A clear and concise description of what the feature is.)
Dear Dr. Tong,
first of all, thank you so much for putting this comprehensive overview of KTMs together. Based on your survey on Knowledge Tracing and the examples and algorithms you collected here I was able to analyse my data using a DKVMN.
While the results of the RNN are very promising I would like to now generate the knowledge state depictions. Do you by any chance are aware of whether the code for the calculations and visualisations can be found somewhere?
Kind regards,
Carl Klukkert
DKT training fails for batch size of 1, but works for larger batch sizes (i.e. 64)
RuntimeErrorTraceback (most recent call last)
<ipython-input-15-0a528344bb33> in <module>
5 # Initialize and train model
6 dkt = DKT(NUM_QUESTIONS, HIDDEN_SIZE, NUM_LAYERS)
----> 7 dkt.train(train_loader, epoch=50)
8
9 # Save weights
/usr/local/lib/python3.6/dist-packages/EduKTM/DKT/DKT.py in train(self, train_data, test_data, epoch, lr)
61 # back propagation
62 optimizer.zero_grad()
---> 63 loss.backward()
64 optimizer.step()
65
/usr/local/lib/python3.6/dist-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
305 create_graph=create_graph,
306 inputs=inputs)
--> 307 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
308
309 def register_hook(self, hook):
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
154 Variable._execution_engine.run_backward(
155 tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 156 allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
157
158
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
Run the example notebook (https://github.com/bigdata-ustc/EduKTM/blob/main/examples/DKT/DKT.ipynb) and set the BATCH_SIZE variable to 1.
Increasing the batch size avoids the error.
Operating System: Ubuntu in docker image: tensorflow/tensorflow:2.6.0-gpu-jupyter.
Also tested in Google Colab notebook.
Python Version: Python 3.6.9
pip freeze:
absl-py==0.13.0
aiocontextvars==0.2.2
altair==4.1.0
ansi2html==1.6.0
anyio==3.3.4
argon2-cffi==20.1.0
asgiref==3.4.1
asn1crypto==0.24.0
astor==0.8.1
astunparse==1.6.3
async-generator==1.10
attrs==21.2.0
Babel==2.9.1
backcall==0.2.0
backports.zoneinfo==0.2.1
base58==2.1.0
beautifulsoup4==4.10.0
bleach==4.0.0
blinker==1.4
Brotli==1.0.9
bs4==0.0.1
cached-property==1.5.2
cachetools==4.2.2
certifi==2021.5.30
cffi==1.14.6
charset-normalizer==2.0.4
clang==5.0
click==7.1.2
contextlib2==21.6.0
contextvars==2.4
cryptography==2.1.4
cycler==0.10.0
dash==2.0.0
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
dataclasses==0.8
decorator==4.4.2
defusedxml==0.7.1
EduData==0.0.18
EduKTM==0.0.9
entrypoints==0.3
et-xmlfile==1.1.0
fastapi==0.70.0
fire==0.4.0
Flask==2.0.2
Flask-Compress==1.10.1
flatbuffers==1.12
gast==0.4.0
gitdb==4.0.8
GitPython==3.1.18
google-auth==1.34.0
google-auth-oauthlib==0.4.5
google-pasta==0.2.0
grpcio==1.39.0
h11==0.12.0
h5py==3.1.0
idna==3.3
immutables==0.16
importlib-metadata==4.6.3
importlib-resources==5.3.0
ipykernel==5.5.6
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.1
joblib==1.1.0
json5==0.9.6
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-core==4.7.1
jupyter-dash==0.4.0
jupyter-http-over-ws==0.0.8
jupyter-server==1.11.1
jupyterlab==3.0.16
jupyterlab-pygments==0.1.2
jupyterlab-server==2.8.2
jupyterlab-widgets==1.0.0
keras==2.6.0
Keras-Preprocessing==1.1.2
keyring==10.6.0
keyrings.alt==3.0
kiwisolver==1.3.1
loguru==0.5.3
longling==1.3.32
lxml==4.6.3
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib==3.3.4
mistune==0.8.4
nbclassic==0.3.3
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.1
networkx==2.5.1
notebook==6.4.3
numpy==1.19.5
oauthlib==3.1.1
openpyxl==3.0.9
opt-einsum==3.3.0
opyrator==0.0.12
packaging==21.0
pandas==1.1.5
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.3.1
plotly==5.3.1
prometheus-client==0.11.0
prompt-toolkit==3.0.19
protobuf==3.17.3
ptyprocess==0.7.0
pyarrow==5.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycrypto==2.6.1
pydantic==1.8.2
pydeck==0.6.2
Pygments==2.9.0
PyGObject==3.26.1
pygraphviz==1.6
pyparsing==2.4.7
pyrsistent==0.18.0
python-apt==1.6.5+ubuntu0.7
python-dateutil==2.8.2
pytz==2021.3
pytz-deprecation-shim==0.1.0.post0
pyxdg==0.25
PyYAML==6.0
pyzmq==22.2.1
qtconsole==5.1.1
QtPy==1.9.0
rarfile==4.0
requests==2.26.0
requests-oauthlib==1.3.0
requests-unixsocket==0.2.0
retrying==1.3.3
rsa==4.7.2
scikit-learn==0.24.2
scipy==1.5.4
seaborn==0.11.2
SecretStorage==2.3.1
Send2Trash==1.8.0
six==1.15.0
sklearn==0.0
smmap==5.0.0
sniffio==1.2.0
soupsieve==2.2.1
starlette==0.16.0
streamlit==1.1.0
tenacity==8.0.1
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.6.0
tensorflow-estimator==2.6.0
termcolor==1.1.0
terminado==0.10.1
testpath==0.5.0
threadpoolctl==3.0.0
toml==0.10.2
toolz==0.11.1
torch==1.10.0
tornado==6.1
tqdm==4.62.3
traitlets==4.3.3
typer==0.4.0
typing-extensions==3.7.4.3
tzdata==2021.4
tzlocal==4.0.1
urllib3==1.26.6
uvicorn==0.15.0
validators==0.18.2
watchdog==2.1.6
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.2.1
Werkzeug==2.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
zipp==3.5.0
Hello, thank you very much for making the LPKT code public, but I overfit when reproducing the assist12 dataset, can you share with me the data processing code of assist12 and ednet?
Hello, Thanks for opening codes for this project.
I want to check the proficiency change of every knowledge during learning process as in the figure 1, 3. of your LPKT paper.
How did you get the proficiency level from 0.0 to 1.0?
I have made the local system the same as the online system and carefully tested the code in my local system (all testing passed). However, I encountered the exception: E AttributeError: '_io.StringIO' object has no attribute 'buffer'
after raising a PR #22 for adding GKT model. To solve the problem, I downgraded flake8 4.0.1 to 3.9.2 in setup.py and got the pycodestyle error report.
I attend to run the program, but I can't download Assist12 and Assistmentchall dataset from their official website. Would you like to share these two dataset.
In LPKT code, how can the knowledge state of students be visualized? It seems that only one question can be answered at a time, so how can we obtain the mastery level of other knowledge concepts?
In DKT of EduKTM, the loss function is BCEWithLogitLoss()
, but there is already a sigmoid()
at the end of forward()
of Net
, So, shouldn’t it be BCELoss()
here?
When I replace BCEWithLogitLoss()
with BCELoss()
, the performance has been greatly improved. ( AUC on 2009_skill_builder_data_corrected from ~0.75 to ~0.80)
This looks like a big mistake, does this mean that there are some problems with the experimental results of many papers?
Hi Dr. Tong,
There's a function named process_raw_pred
in EduKTM/EduKTM/DKT/DKT.py
.
Lines 31 to 37 in c9912f0
According to the codes below (line 56), we can learn thatprocess_raw_pred
is used to process the raw input and the output of the DKT model.
Lines 50 to 58 in c9912f0
I have three questions.
questions = torch.nonzero(raw_question_matrix)[1:, 1] % num_questions
in line 32. [1:, 1]
here we goes from index 1, which means we throw away the first answer whose index is 0. Do you mean that the first value is not predicted and it is meaningless because it depends on no history answer records?pred = raw_pred[: length]
in line 34, here we goes from index 0. Why don't we throw away the first predicted value just like what we did in line 32? e.g. pred = raw_pred[1 : length+1]
truth = torch.nonzero(raw_question_matrix)[1:, 1] // num_questions
in line 36, we use //
so that we can get 0 if non-zeros are in the first half part (correct answers), get 1 if non-zeros are in the second half part(wrong answers).encode_onehot
function in examples/DKT/prepare_dataset.ipynb
, correct answers are in the first half part and wrong answers are in the second half part. Also, 1 stands for a correct answer and 0 stands for a wrong answer.def encode_onehot(sequences, max_step, num_questions):
result = []
for q, a in tqdm.tqdm(sequences, 'convert to one-hot format: '): # e.g. q: [1,2,3] a: [1,0,0]
length = len(q)
# append questions' and answers' length to an integer multiple of max_step
mod = 0 if length % max_step == 0 else (max_step - length % max_step)
onehot = np.zeros(shape=[length + mod, 2 * num_questions])
print(length+mod)
for i, q_id in enumerate(q):
# if a[i]>0(correct answer),index=question id(first half part),else index=question id + question number(second half part)
index = int(q_id if a[i] > 0 else q_id + num_questions)
onehot[i][index] = 1 # correct answers are in the first half part
result = np.append(result, onehot)
return result.reshape(-1, max_step, 2 * num_questions)
So truth = torch.nonzero(raw_question_matrix)[1:, 1] // num_questions
is not consistent with the encoding. To validate my thoughts, I ran the DKT example and print the torch.nonzero(raw_question_matrix)
and truth
, which are used to compare with the encoding result stored in test.txt
.
Here are two screenshots which i got from console and test.txt
,
This shows my thought is right.
Then I added a line truth = torch.tensor([1 if i == 0 else 0 for i in truth])
to test performence.
The average AUC is about 0.73, equal to the moment before adding this line.
Sorry for the long description : ( .Your answers are appreciated!😀
There is only code to preprocess data on ASSISTChallenge in EduKTM-LPKT.
I tried to imitate your code to process the data of ASSIST12 and Ednet, but the experimental results are far from the results in the paper.
Can other code for processing data be open sourced?
https://github.com/bigdata-ustc/EduKTM/blob/main/EduKTM/AKT/AKTNet.py#L273
The mask_fill operation on this line should be replaced with mask_fill_, otherwise it will cause data leakage
origin
scores.masked_fill(mask == 0, -1e32)
solution
scores.masked_fill_(mask == 0, -1e32)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.