Documentation and code are so broken!

[elsatch]

Documentation and code are so broken!

[elsatch]

After reviewing the codebase, it seems like the new-dev creates a new abstraction layer on top of the existing scripts. So I would say that new-dev brach is they way to move forward. This is the existing relationship:

ares.py
│
├── synthetic_generator.py
| └── LLM_as_a_Judge_Adaptation/Generate_Synthetic_Queries_and_Answers.py
│ ├── LLM_as_a_Judge_Adaptation/LLM_Generation_Functions.py
│ └── LLM_as_a_Judge_Adaptation/Filter_Synthetic_Queries.py
│
├── binary_classifier.py
│ └── LLM_as_a_Judge_Adaptation/General_Binary_Classifier.py
│
├── rag_scoring.py
│ └── RAG_Automatic_Evaluation/LLMJudge_RAG_Compared_Scoring.py
│ └── RAG_Automatic_Evaluation/Evaluation_Functions.py
| └── RAG_Automatic_Evaluation/ppi.py
|
└── ues_idp.py
└── RAG_Automatic_Evaluation/Evaluation_Functions.py

[elsatch]

In the training classifier page, the full example code doesn't work:

• The test_set_selection file used is not available in the repo: ratio_0.5_reformatted_full_articles_False_validation_with_negatives.tsv
• Classification_dataset parameter should be a string not a list, otherwise you'll get the following error:
  TypeError: Parameter 'classification_dataset' for classifier_model is expected to be of type str, received list instead.
• There is a missing parameter required to run the script: ValueError: Missing required parameter 'validation_set' for classifier_model. The current code in the library uses validation_set instead of test_set_selection.
• Label_column should not be a list but a string.

[elsatch]

Just to clarify the situation, the documentation at the vercel site is related to the new-dev branch. The legacy documentation in the README.md relates to the scripts only.

I am finding my way on the new documentation, trying to fix typos and routes on the new-dev branch.

[ViceSilva]

Hello,

I've also encountered issues trying to reproduce the results of the paper using the code from this repository's main branch. Do you think using the new developers' branch is better suited for this purpose?

[elsatch]

After the last update, it makes no sense to keep fixing issues in the legacy docs. Let's review the last updates to see how many of these issues remain relevant!