The LLM ouput is not in the expected format.

[renjjhl]

<I encountered a difficulty, using Azure api access, Azure test cases can access and return the corresponding message, but in the gsm8k dataset demo, run to>:

best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)

-------The LLM ouput is not in the expected format.

[Generated Prompts]:
mutated_prompt_generation=Sorry, I am not able to understand your query. Please try again.
Please run 'az login' to set up an account
mutation_round=1 mutated_sample_prompt=You are given a task description and a prompt instruction and different styles known as meta prompts:
[Task Description]: You are a mathematics expert. You will be given a mathematics problem which you need to solve
[Meta Prompt]: How could I devise an experiment to help solve that problem?
Make a list of ideas for solving this problem, and apply them one by one to the problem to see if any progress can be made.
How could I measure progress on this problem?
How can I simplify the problem so that it is easier to solve?
What are the key assumptions underlying this problem?
Now you need to generate 5 variations of following Instruction adaptively mixing meta prompt while keeping similar semantic meaning.
Make sure to wrap each generated prompt with and
[Prompt Instruction]: Lets think step by step.
[Generated Prompts]:
mutated_prompt_generation=Sorry, I am not able to understand your query. Please try again.
Please run 'az login' to set up an account
mutation_round=2 mutated_sample_prompt=You are given a task description and a prompt instruction and different styles known as meta prompts:

<How do I solve this problem?>

[renjjhl]

Reason found, if don't azure cli login method directly using the corresponding api_key access, please delete promptwizard/glue/common/LLM/llm_mgr py of 32, 33, 34, content is

"
token_provider = get_bearer_token_provider(
AzureCliCredential(), "https://cognitiveservices.azure.com/.default
)
"

And add the api_key field to client:

“api_version=os.environ["OPENAI_API_VERSION"],
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
azure_ad_token_provider=token_provider,
api_key=os.environ["OPENAI_AZURE_APIKEY"]”

[runfish5]

I faced the same error.
In the core_logic.py, I inspected the response (refined_prompts) from the LLM:

refined_prompts = self.chat_completion(critique_refine_prompt, self.prompt_pool.expert_profile)

The error occurs when the LLM uses an unanticipated end-delimiter (</END>/</START> instead of <END>), which the code doesn't support.

Solution:

In promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py,
replace

refined_prompts = re.findall(DatasetSpecificProcessing.TEXT_DELIMITER_PATTERN, refined_prompts)

with:

MAGENTA = '\033[35m'
RESET = '\033[0m'
temp_refined_prompts = refined_prompts
refined_prompts = re.findall(DatasetSpecificProcessing.TEXT_DELIMITER_PATTERN, refined_prompts)
if not refined_prompts:
    refined_prompts = re.findall('(?s)(?<=<START>)(.*?)(?=</START>)', temp_refined_prompts)
    if refined_prompts:
        print(f"{MAGENTA}THERE WERE </START> in there !!!!!{RESET}")

if not refined_prompts:
    refined_prompts = re.findall('(?s)(?<=<START>)(.*?)(?=</END>)', temp_refined_prompts)
    if refined_prompts:
        print(f"{MAGENTA}THERE WERE </END> in there !!!!!{RESET}")

[guo-12]

[Generated Prompts]:
mutated_prompt_generation=Sorry, I am not able to understand your query. Please try again.
Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-zufFKmxrT. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
dataset_subset [{'question': 'Jerry had 7 books and 3 action figures on a shelf in his room. Later he added 2 more action figures to the shelf. How many more books than action figures were on his shelf?', 'answer': '2', 'final_answer': '2'}]
critique_example_set, correct_count
[{'question': 'Jerry had 7 books and 3 action figures on a shelf in his room. Later he added 2 more action figures to the shelf. How many more books than action figures were on his shelf?', 'answer': '2', 'final_answer': '2'}] 0
Loop completed
prompt_score_list [['You are a mathematics expert. You will be given a mathematics problem which you need to solve\nLets think step by step.', 0.0, [{'question': 'Jerry had 7 books and 3 action figures on a shelf in his room. Later he added 2 more action figures to the shelf. How many more books than action figures were on his shelf?', 'answer': '2', 'final_answer': '2'}]]]
Sorted top n prompts: [['You are a mathematics expert. You will be given a mathematics problem which you need to solve\nLets think step by step.', 0.0, [{'question': 'Jerry had 7 books and 3 action figures on a shelf in his room. Later he added 2 more action figures to the shelf. How many more books than action figures were on his shelf?', 'answer': '2', 'final_answer': '2'}]]]
Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-zufFKmxrT. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-zufFK***************************************mxrT. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
Iterations completed: 0%| | 0/3 [00:20<?, ?it/s]
Traceback (most recent call last):
File "/Users/guokai/Desktop/PromptWizard-main /b.py", line 99, in
best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/promptopt/instantiate.py", line 118, in get_best_prompt
self.BEST_PROMPT, self.EXPERT_PROFILE = self.prompt_opt.get_best_prompt(self.prompt_opt_param,use_examples=use_examples,run_without_train_examples=run_without_train_examples,generate_synthetic_examples=generate_synthetic_examples)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py", line 509, in get_best_prompt
refined_prompts = self.refine_prompts(prompt_score_list, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/paramlogger/init.py", line 87, in wrap
args_to_log = run_method_get_io_dict(method_obj, self.DEL_SELF_ARG, *argv, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/paramlogger/utils.py", line 25, in run_method_get_io_dict
output = method_obj(*argv, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py", line 234, in refine_prompts
refined_prompts.append(self.critique_and_refine(prompt, critique_example_set))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/paramlogger/init.py", line 87, in wrap
args_to_log = run_method_get_io_dict(method_obj, self.DEL_SELF_ARG, *argv, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/paramlogger/utils.py", line 25, in run_method_get_io_dict
output = method_obj(*argv, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/guokai/Desktop/PromptWizard-main /promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py", line 161, in critique_and_refine
raise ValueError("The LLM ouput is not in the expected format. Please rerun the code...")
ValueError: The LLM ouput is not in the expected format. Please rerun the code...
(

[yyw794]

use this instead(I face the same problem)

 TEXT_DELIMITER_PATTERN = r"(?s)(?<=<START>)(.*?)(?=(?:<END>|</START>))"
    TEXT_DELIMITER_PATTERN_MUTATION = r"(?s)(?<=<START>)(.*?)(?=(?:<END>|</START>))"