If the questions_batch_size > 1 the code doesn't count for it. In the promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py
|
# Find all matches of the pattern in the text |
|
answer_matches = re.findall(DatasetSpecificProcessing.ANSWER_DELIMITER_PATTERN, generated_text) |
|
|
|
# answer_matches = [self.chat_completion(FINAL_ANSWER_EXTRACTION_PROMPT.format(text=generated_text), "You are an AI assistant. Please follow the users requests.")] |
|
answer_matches = [generated_text] |
|
# |
if there is no match it gets the full text which means one example is checked, however if batch size is bigger than one more than one example should be checked. To fix it you can change line
answer_matches = [generated_text] to
if not answer_matches; answer_matches = [generated_text] Can add that myself too, i fixed it in my code so let me know.
