improve docstr of preprocessors (#1227)

* improve docstr of preprocessors

* Update SynapseML version

* RFix test

---------

Co-authored-by: Li Jiang <bnujli@gmail.com>

.github/workflows/python-package.yml

@@ -64,10 +64,12 @@ jobs:
         if: matrix.os == 'ubuntu-latest'
         run: |
           pip install "ray[tune]<2.5.0"
-      - name: If mac, install ray
+      - name: If mac, install ray and xgboost 1
         if: matrix.os == 'macOS-latest'
         run: |
           pip install -e .[ray]
+          # use macOS to test xgboost 1, but macOS also supports xgboost 2
+          pip install "xgboost<2"
       - name: If linux or mac, install prophet on python < 3.9
         if: (matrix.os == 'macOS-latest' || matrix.os == 'ubuntu-latest') && matrix.python-version != '3.9' && matrix.python-version != '3.10'
         run: |

README.md

@@ -14,13 +14,15 @@
     <br>
 </p>
 
-:fire: The automated multi-agent chat framework in [autogen](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen) is in preview from v2.0.0.
+:fire: Heads-up: We're preparing to migrate [AutoGen](https://microsoft.github.io/autogen/) into a dedicated github repository. Alongside this move, we'll also launch a dedicated Discord server and a website for comprehensive documentation.
+
+:fire: The automated multi-agent chat framework in [AutoGen](https://microsoft.github.io/autogen/) is in preview from v2.0.0.
 
 :fire: FLAML is highlighted in OpenAI's [cookbook](https://github.com/openai/openai-cookbook#related-resources-from-around-the-web).
 
-:fire: [autogen](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen) is released with support for ChatGPT and GPT-4, based on [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673).
+:fire: [autogen](https://microsoft.github.io/autogen/) is released with support for ChatGPT and GPT-4, based on [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673).
 
-:fire: FLAML supports AutoML and Hyperparameter Tuning features in [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview) private preview. Sign up for these features at: https://aka.ms/fabric/data-science/sign-up.
+:fire: FLAML supports Code-First AutoML & Tuning – Private Preview in [Microsoft Fabric Data Science](https://learn.microsoft.com/en-us/fabric/data-science/).
 
 
 ## What is FLAML
@@ -32,7 +34,7 @@ and optimizes their performance.
 * For common machine learning tasks like classification and regression, it quickly finds quality models for user-provided data with low computational resources. It is easy to customize or extend. Users can find their desired customizability from a smooth range.
 * It supports fast and economical automatic tuning (e.g., inference hyperparameters for foundation models, configurations in MLOps/LMOps workflows, pipelines, mathematical/statistical models, algorithms, computing experiments, software configurations), capable of handling large search space with heterogeneous evaluation cost and complex constraints/guidance/early stopping.
 
-FLAML is powered by a series of [research studies](/docs/Research) from Microsoft Research and collaborators such as Penn State University, Stevens Institute of Technology, University of Washington, and University of Waterloo.
+FLAML is powered by a series of [research studies](https://microsoft.github.io/FLAML/docs/Research/) from Microsoft Research and collaborators such as Penn State University, Stevens Institute of Technology, University of Washington, and University of Waterloo.
 
 FLAML has a .NET implementation in [ML.NET](http://dot.net/ml), an open-source, cross-platform machine learning framework for .NET.
 
@@ -44,7 +46,7 @@ FLAML requires **Python version >= 3.8**. It can be installed from pip:
 pip install flaml
 ```
 
-Minimal dependencies are installed without extra options. You can install extra options based on the feature you need. For example, use the following to install the dependencies needed by the [`autogen`](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen) package.
+Minimal dependencies are installed without extra options. You can install extra options based on the feature you need. For example, use the following to install the dependencies needed by the [`autogen`](https://microsoft.github.io/autogen/) package.
 ```bash
 pip install "flaml[autogen]"
 ```
@@ -54,7 +56,7 @@ Each of the [`notebook examples`](https://github.com/microsoft/FLAML/tree/main/n
 
 ## Quickstart
 
-* (New) The [autogen](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen) package enables the next-gen GPT-X applications with a generic multi-agent conversation framework.
+* (New) The [autogen](https://microsoft.github.io/autogen/) package enables the next-gen GPT-X applications with a generic multi-agent conversation framework.
 It offers customizable and conversable agents which integrate LLMs, tools and human.
 By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code. For example,
 ```python

flaml/autogen/agentchat/__init__.py

@@ -1,12 +1,12 @@
 from .agent import Agent
-from .responsive_agent import ResponsiveAgent
+from .conversable_agent import ConversableAgent
 from .assistant_agent import AssistantAgent
 from .user_proxy_agent import UserProxyAgent
 from .groupchat import GroupChat, GroupChatManager
 
 __all__ = [
     "Agent",
-    "ResponsiveAgent",
+    "ConversableAgent",
     "AssistantAgent",
     "UserProxyAgent",
     "GroupChat",

flaml/autogen/agentchat/assistant_agent.py

@@ -1,11 +1,11 @@
-from .responsive_agent import ResponsiveAgent
+from .conversable_agent import ConversableAgent
 from typing import Callable, Dict, Optional, Union
 
 
-class AssistantAgent(ResponsiveAgent):
+class AssistantAgent(ConversableAgent):
     """(In preview) Assistant agent, designed to solve a task with LLM.
 
-    AssistantAgent is a subclass of ResponsiveAgent configured with a default system message.
+    AssistantAgent is a subclass of ConversableAgent configured with a default system message.
     The default system message is designed to solve a task with LLM,
     including suggesting python code blocks and debugging.
     `human_input_mode` is default to "NEVER"
@@ -14,13 +14,16 @@ class AssistantAgent(ResponsiveAgent):
     """
 
     DEFAULT_SYSTEM_MESSAGE = """You are a helpful AI assistant.
-    In the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute. You must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.
-    1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time.
-    2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly. Solve the task step by step if you need to.
-    If you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.
-    If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
-    When you find an answer, verify the answer carefully. If a function for planning is provided, call the function to make plans and verify the execution.
-    Reply "TERMINATE" in the end when everything is done.
+Solve tasks using your coding and language skills.
+In the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.
+    1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.
+    2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.
+Solve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.
+When using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.
+If you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.
+If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
+When you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.
+Reply "TERMINATE" in the end when everything is done.
     """
 
     def __init__(
@@ -49,7 +52,7 @@ class AssistantAgent(ResponsiveAgent):
                 default to None (no limit provided, class attribute MAX_CONSECUTIVE_AUTO_REPLY will be used as the limit in this case).
                 The limit only plays a role when human_input_mode is not "ALWAYS".
             **kwargs (dict): Please refer to other kwargs in
-                [ResponsiveAgent](responsive_agent#__init__).
+                [ConversableAgent](conversable_agent#__init__).
         """
         super().__init__(
             name,

flaml/autogen/agentchat/contrib/math_user_proxy_agent.py

@@ -165,7 +165,7 @@ class MathUserProxyAgent(UserProxyAgent):
             default_auto_reply=default_auto_reply,
             **kwargs,
         )
-        self.register_auto_reply(Agent, MathUserProxyAgent._generate_math_reply, 1)
+        self.register_reply([Agent, None], MathUserProxyAgent._generate_math_reply, 1)
         # fixed var
         self._max_invalid_q_per_step = max_invalid_q_per_step
 

flaml/autogen/agentchat/contrib/retrieve_assistant_agent.py

@@ -16,7 +16,7 @@ class RetrieveAssistantAgent(AssistantAgent):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
-        self.register_auto_reply(Agent, RetrieveAssistantAgent._generate_retrieve_assistant_reply)
+        self.register_reply(Agent, RetrieveAssistantAgent._generate_retrieve_assistant_reply)
 
     def _generate_retrieve_assistant_reply(
         self,

flaml/autogen/agentchat/contrib/retrieve_user_proxy_agent.py

@@ -148,7 +148,7 @@ class RetrieveUserProxyAgent(UserProxyAgent):
         self._ipython = get_ipython()
         self._doc_idx = -1  # the index of the current used doc
         self._results = {}  # the results of the current query
-        self.register_auto_reply(Agent, RetrieveUserProxyAgent._generate_retrieve_user_reply)
+        self.register_reply(Agent, RetrieveUserProxyAgent._generate_retrieve_user_reply)
 
     @staticmethod
     def get_max_tokens(model="gpt-3.5-turbo"):

flaml/autogen/agentchat/conversable_agent.py

@@ -21,11 +21,11 @@ except ImportError:
         return x
 
 
-class ResponsiveAgent(Agent):
-    """(Experimental) A class for generic responsive agents which can be configured as assistant or user proxy.
+class ConversableAgent(Agent):
+    """(In preview) A class for generic conversable agents which can be configured as assistant or user proxy.
 
     After receiving each message, the agent will send a reply to the sender unless the msg is a termination msg.
-    For example, AssistantAgent and UserProxyAgent are subclasses of ResponsiveAgent,
+    For example, AssistantAgent and UserProxyAgent are subclasses of this class,
     configured with different default settings.
 
     To modify auto reply, override `generate_reply` method.
@@ -119,12 +119,12 @@ class ResponsiveAgent(Agent):
         self._default_auto_reply = default_auto_reply
         self._reply_func_list = []
         self.reply_at_receive = defaultdict(bool)
-        self.register_auto_reply(Agent, ResponsiveAgent.generate_oai_reply)
-        self.register_auto_reply(Agent, ResponsiveAgent.generate_code_execution_reply)
-        self.register_auto_reply(Agent, ResponsiveAgent.generate_function_call_reply)
-        self.register_auto_reply(Agent, ResponsiveAgent.check_termination_and_human_reply)
+        self.register_reply([Agent, None], ConversableAgent.generate_oai_reply)
+        self.register_reply([Agent, None], ConversableAgent.generate_code_execution_reply)
+        self.register_reply([Agent, None], ConversableAgent.generate_function_call_reply)
+        self.register_reply([Agent, None], ConversableAgent.check_termination_and_human_reply)
 
-    def register_auto_reply(
+    def register_reply(
         self,
         trigger: Union[Type[Agent], str, Agent, Callable[[Agent], bool], List],
         reply_func: Callable,
@@ -145,11 +145,13 @@ class ResponsiveAgent(Agent):
                 - If an agent instance is provided, the reply function will be called when the sender is the agent instance.
                 - If a callable is provided, the reply function will be called when the callable returns True.
                 - If a list is provided, the reply function will be called when any of the triggers in the list is activated.
+                - If None is provided, the reply function will be called only when the sender is None.
+                Note: Be sure to register `None` as a trigger if you would like to trigger an auto-reply function with non-empty messages and `sender=None`.
             reply_func (Callable): the reply function.
                 The function takes a recipient agent, a list of messages, a sender agent and a config as input and returns a reply message.
         ```python
         def reply_func(
-            recipient: ResponsiveAgent,
+            recipient: ConversableAgent,
             messages: Optional[List[Dict]] = None,
             sender: Optional[Agent] = None,
             config: Optional[Any] = None,
@@ -497,7 +499,7 @@ class ResponsiveAgent(Agent):
 
     def initiate_chat(
         self,
-        recipient: "ResponsiveAgent",
+        recipient: "ConversableAgent",
         clear_history: Optional[bool] = True,
         silent: Optional[bool] = False,
         **context,
@@ -520,7 +522,7 @@ class ResponsiveAgent(Agent):
 
     async def a_initiate_chat(
         self,
-        recipient: "ResponsiveAgent",
+        recipient: "ConversableAgent",
         clear_history: Optional[bool] = True,
         silent: Optional[bool] = False,
         **context,
@@ -609,7 +611,7 @@ class ResponsiveAgent(Agent):
         if messages is None:
             messages = self._oai_messages[sender]
         last_n_messages = code_execution_config.pop("last_n_messages", 1)
-        for i in range(last_n_messages):
+        for i in range(min(len(messages), last_n_messages)):
             message = messages[-(i + 1)]
             code_blocks = extract_code(message["content"])
             if len(code_blocks) == 1 and code_blocks[0][0] == UNKNOWN:
@@ -726,6 +728,7 @@ class ResponsiveAgent(Agent):
         """Reply based on the conversation history and the sender.
 
         Either messages or sender must be provided.
+        Register a reply_func with `None` as one trigger for it to be activated when `messages` is non-empty and `sender` is `None`.
         Use registered auto reply functions to generate replies.
         By default, the following functions are checked in order:
         1. check_termination_and_human_reply
@@ -748,17 +751,19 @@ class ResponsiveAgent(Agent):
             str or dict or None: reply. None if no reply is generated.
         """
         assert messages is not None or sender is not None, "Either messages or sender must be provided."
-        if sender is not None:
-            for reply_func_tuple in self._reply_func_list:
-                reply_func = reply_func_tuple["reply_func"]
-                if exclude and reply_func in exclude:
-                    continue
-                if asyncio.coroutines.iscoroutinefunction(reply_func):
-                    continue
-                if self._match_trigger(reply_func_tuple["trigger"], sender):
-                    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
-                    if final:
-                        return reply
+        if messages is None:
+            messages = self._oai_messages[sender]
+
+        for reply_func_tuple in self._reply_func_list:
+            reply_func = reply_func_tuple["reply_func"]
+            if exclude and reply_func in exclude:
+                continue
+            if asyncio.coroutines.iscoroutinefunction(reply_func):
+                continue
+            if self._match_trigger(reply_func_tuple["trigger"], sender):
+                final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
+                if final:
+                    return reply
         return self._default_auto_reply
 
     async def a_generate_reply(
@@ -770,6 +775,7 @@ class ResponsiveAgent(Agent):
         """(async) Reply based on the conversation history and the sender.
 
         Either messages or sender must be provided.
+        Register a reply_func with `None` as one trigger for it to be activated when `messages` is non-empty and `sender` is `None`.
         Use registered auto reply functions to generate replies.
         By default, the following functions are checked in order:
         1. check_termination_and_human_reply
@@ -792,27 +798,29 @@ class ResponsiveAgent(Agent):
             str or dict or None: reply. None if no reply is generated.
         """
         assert messages is not None or sender is not None, "Either messages or sender must be provided."
-        if sender is not None:
-            for reply_func_tuple in self._reply_func_list:
-                reply_func = reply_func_tuple["reply_func"]
-                if exclude and reply_func in exclude:
-                    continue
-                if self._match_trigger(reply_func_tuple["trigger"], sender):
-                    if asyncio.coroutines.iscoroutinefunction(reply_func):
-                        final, reply = await reply_func(
-                            self, messages=messages, sender=sender, config=reply_func_tuple["config"]
-                        )
-                    else:
-                        final, reply = reply_func(
-                            self, messages=messages, sender=sender, config=reply_func_tuple["config"]
-                        )
-                    if final:
-                        return reply
+        if messages is None:
+            messages = self._oai_messages[sender]
+
+        for reply_func_tuple in self._reply_func_list:
+            reply_func = reply_func_tuple["reply_func"]
+            if exclude and reply_func in exclude:
+                continue
+            if self._match_trigger(reply_func_tuple["trigger"], sender):
+                if asyncio.coroutines.iscoroutinefunction(reply_func):
+                    final, reply = await reply_func(
+                        self, messages=messages, sender=sender, config=reply_func_tuple["config"]
+                    )
+                else:
+                    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
+                if final:
+                    return reply
         return self._default_auto_reply
 
     def _match_trigger(self, trigger, sender):
         """Check if the sender matches the trigger."""
-        if isinstance(trigger, str):
+        if trigger is None:
+            return sender is None
+        elif isinstance(trigger, str):
             return trigger == sender.name
         elif isinstance(trigger, type):
             return isinstance(sender, trigger)
@@ -887,10 +895,11 @@ class ResponsiveAgent(Agent):
                 exitcode, logs, image = (
                     1,
                     f"unknown language {lang}",
-                    self._code_execution_config["use_docker"],
+                    None,
                 )
                 # raise NotImplementedError
-            self._code_execution_config["use_docker"] = image
+            if image is not None:
+                self._code_execution_config["use_docker"] = image
             logs_all += "\n" + logs
             if exitcode != 0:
                 return exitcode, logs_all
@@ -953,7 +962,7 @@ class ResponsiveAgent(Agent):
                 content = f"Error: {e}\n You argument should follow json format."
 
             # Try to execute the function
-            if arguments:
+            if arguments is not None:
                 print(
                     colored(f"\n>>>>>>>> EXECUTING FUNCTION {func_name}...", "magenta"),
                     flush=True,

flaml/autogen/agentchat/groupchat.py

@@ -2,7 +2,7 @@ from dataclasses import dataclass
 import sys
 from typing import Dict, List, Optional, Union
 from .agent import Agent
-from .responsive_agent import ResponsiveAgent
+from .conversable_agent import ConversableAgent
 
 
 @dataclass
@@ -12,6 +12,7 @@ class GroupChat:
     agents: List[Agent]
     messages: List[Dict]
     max_round: int = 10
+    admin_name: str = "Admin"  # the name of the admin agent
 
     @property
     def agent_names(self) -> List[str]:
@@ -38,10 +39,18 @@ class GroupChat:
 Read the following conversation.
 Then select the next role from {self.agent_names} to play. Only return the role."""
 
-    def select_speaker(self, last_speaker: Agent, selctor: ResponsiveAgent):
+    def select_speaker(self, last_speaker: Agent, selector: ConversableAgent):
         """Select the next speaker."""
-        selctor.update_system_message(self.select_speaker_msg())
-        final, name = selctor.generate_oai_reply(self.messages)
+        selector.update_system_message(self.select_speaker_msg())
+        final, name = selector.generate_oai_reply(
+            self.messages
+            + [
+                {
+                    "role": "system",
+                    "content": f"Read the above conversation. Then select the next role from {self.agent_names} to play. Only return the role.",
+                }
+            ]
+        )
         if not final:
             # i = self._random.randint(0, len(self._agent_names) - 1)  # randomly pick an id
             return self.next_agent(last_speaker)
@@ -54,8 +63,8 @@ Then select the next role from {self.agent_names} to play. Only return the role.
         return "\n".join([f"{agent.name}: {agent.system_message}" for agent in self.agents])
 
 
-class GroupChatManager(ResponsiveAgent):
-    """(WIP) A chat manager agent that can manage a group chat of multiple agents."""
+class GroupChatManager(ConversableAgent):
+    """(In preview) A chat manager agent that can manage a group chat of multiple agents."""
 
     def __init__(
         self,
@@ -75,7 +84,7 @@ class GroupChatManager(ResponsiveAgent):
             system_message=system_message,
             **kwargs,
         )
-        self.register_auto_reply(Agent, GroupChatManager.run_chat, config=groupchat, reset_config=GroupChat.reset)
+        self.register_reply(Agent, GroupChatManager.run_chat, config=groupchat, reset_config=GroupChat.reset)
         # self._random = random.Random(seed)
 
     def run_chat(
@@ -89,21 +98,36 @@ class GroupChatManager(ResponsiveAgent):
             messages = self._oai_messages[sender]
         message = messages[-1]
         speaker = sender
-        for i in range(config.max_round):
+        groupchat = config
+        for i in range(groupchat.max_round):
             # set the name to speaker's name if the role is not function
             if message["role"] != "function":
                 message["name"] = speaker.name
-            config.messages.append(message)
+            groupchat.messages.append(message)
             # broadcast the message to all agents except the speaker
-            for agent in config.agents:
+            for agent in groupchat.agents:
                 if agent != speaker:
                     self.send(message, agent, request_reply=False, silent=True)
-            if i != config.max_round - 1:
-                # speaker selection msg from an agent
-                speaker = config.select_speaker(speaker, self)
+            if i == groupchat.max_round - 1:
+                # the last round
+                break
+            try:
+                # select the next speaker
+                speaker = groupchat.select_speaker(speaker, self)
+                # let the speaker speak
                 reply = speaker.generate_reply(sender=self)
-                if reply is None:
-                    break
-                speaker.send(reply, self, request_reply=False)
-                message = self.last_message(speaker)
+            except KeyboardInterrupt:
+                # let the admin agent speak if interrupted
+                if groupchat.admin_name in groupchat.agent_names:
+                    # admin agent is one of the participants
+                    speaker = groupchat.agent_by_name(groupchat.admin_name)
+                    reply = speaker.generate_reply(sender=self)
+                else:
+                    # admin agent is not found in the participants
+                    raise
+            if reply is None:
+                break
+            # The speaker sends the message without requesting a reply
+            speaker.send(reply, self, request_reply=False)
+            message = self.last_message(speaker)
         return True, None

flaml/autogen/agentchat/user_proxy_agent.py

@@ -1,14 +1,14 @@
-from .responsive_agent import ResponsiveAgent
+from .conversable_agent import ConversableAgent
 from typing import Callable, Dict, Optional, Union
 
 
-class UserProxyAgent(ResponsiveAgent):
+class UserProxyAgent(ConversableAgent):
     """(In preview) A proxy agent for the user, that can execute code and provide feedback to the other agents.
 
-    UserProxyAgent is a subclass of ResponsiveAgent configured with `human_input_mode` to ALWAYS
+    UserProxyAgent is a subclass of ConversableAgent configured with `human_input_mode` to ALWAYS
     and `llm_config` to False. By default, the agent will prompt for human input every time a message is received.
     Code execution is enabled by default. LLM-based auto reply is disabled by default.
-    To modify auto reply, register a method with (`register_auto_reply`)[responsive_agent#register_auto_reply].
+    To modify auto reply, register a method with (`register_reply`)[conversable_agent#register_reply].
     To modify the way to get human input, override `get_human_input` method.
     To modify the way to execute code blocks, single code block, or function call, override `execute_code_blocks`,
     `run_code`, and `execute_function` methods respectively.

flaml/autogen/code_utils.py

@@ -347,9 +347,9 @@ def execute_code(
         # extract the exit code from the logs
         pattern = re.compile(f"{exit_code_str}(\\d+){exit_code_str}")
         match = pattern.search(logs)
-        exit_code = int(match.group(1))
+        exit_code = 1 if match is None else int(match.group(1))
         # remove the exit code from the logs
-        logs = pattern.sub("", logs)
+        logs = logs if match is None else pattern.sub("", logs)
 
     if original_filename is None:
         os.remove(filepath)

flaml/autogen/oai/completion.py

@@ -48,6 +48,7 @@ class Completion(openai_Completion):
         "gpt-3.5-turbo-0301",  # deprecate in Sep
         "gpt-3.5-turbo-0613",
         "gpt-3.5-turbo-16k",
+        "gpt-3.5-turbo-16k-0613",
         "gpt-35-turbo",
         "gpt-4",
         "gpt-4-32k",
@@ -70,6 +71,7 @@ class Completion(openai_Completion):
         "gpt-3.5-turbo-0301": (0.0015, 0.002),  # deprecate in Sep
         "gpt-3.5-turbo-0613": (0.0015, 0.002),
         "gpt-3.5-turbo-16k": (0.003, 0.004),
+        "gpt-3.5-turbo-16k-0613": (0.003, 0.004),
         "gpt-35-turbo": 0.002,
         "gpt-4": (0.03, 0.06),
         "gpt-4-32k": (0.06, 0.12),
@@ -695,7 +697,7 @@ class Completion(openai_Completion):
                 E.g., `prompt="Complete the following sentence: {prefix}, context={"prefix": "Today I feel"}`.
                 The actual prompt will be:
                 "Complete the following sentence: Today I feel".
-                More examples can be found at [templating](/docs/Use-Cases/Autogen#templating).
+                More examples can be found at [templating](https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#templating).
             use_cache (bool, Optional): Whether to use cached responses.
             config_list (List, Optional): List of configurations for the completion to try.
                 The first one that does not raise an error will be used.
@@ -746,7 +748,11 @@ class Completion(openai_Completion):
                 Also, the "prompt" or "messages" parameter can contain a template (str or Callable) which will be instantiated with the context.
 
         Returns:
-            Responses from OpenAI API.
+            Responses from OpenAI API, with additional fields.
+                - `cost`: the total cost.
+            When `config_list` is provided, the response will contain a few more fields:
+                - `config_id`: the index of the config in the config_list that is used to generate the response.
+                - `pass_filter`: whether the response passes the filter function. None if no filter is provided.
         """
         if ERROR:
             raise ERROR

flaml/automl/automl.py

@@ -246,7 +246,7 @@ class AutoML(BaseEstimator):
                 search is considered to converge.
             force_cancel: boolean, default=False | Whether to forcely cancel Spark jobs if the
                 search time exceeded the time budget.
-            append_log: boolean, default=False | Whetehr to directly append the log
+            append_log: boolean, default=False | Whether to directly append the log
                 records to the input log file if it exists.
             auto_augment: boolean, default=True | Whether to automatically
                 augment rare classes.
@@ -476,12 +476,12 @@ class AutoML(BaseEstimator):
 
     @property
     def feature_transformer(self):
-        """Returns AutoML Transformer"""
+        """Returns feature transformer which is used to preprocess data before applying training or inference."""
         return getattr(self, "_transformer", None)
 
     @property
     def label_transformer(self):
-        """Returns AutoML label transformer"""
+        """Returns label transformer which is used to preprocess labels before scoring, and inverse transform labels after inference."""
         return getattr(self, "_label_transformer", None)
 
     @property
@@ -606,7 +606,7 @@ class AutoML(BaseEstimator):
 
         Args:
             learner_name: A string of the learner's name.
-            learner_class: A subclass of flaml.model.BaseEstimator.
+            learner_class: A subclass of flaml.automl.model.BaseEstimator.
         """
         self._state.learner_classes[learner_name] = learner_class
 
@@ -1381,7 +1381,7 @@ class AutoML(BaseEstimator):
             early_stop: boolean, default=False | Whether to stop early if the
                 search is considered to converge.
             force_cancel: boolean, default=False | Whether to forcely cancel the PySpark job if overtime.
-            append_log: boolean, default=False | Whetehr to directly append the log
+            append_log: boolean, default=False | Whether to directly append the log
                 records to the input log file if it exists.
             auto_augment: boolean, default=True | Whether to automatically
                 augment rare classes.

flaml/automl/model.py

@@ -32,6 +32,7 @@ try:
     from sklearn.ensemble import ExtraTreesRegressor, ExtraTreesClassifier
     from sklearn.linear_model import LogisticRegression
     from sklearn.dummy import DummyClassifier, DummyRegressor
+    from xgboost import __version__ as xgboost_version
 except ImportError:
     pass
 
@@ -212,10 +213,10 @@ class BaseEstimator:
         model = self.estimator_class(**self.params)
         if logger.level == logging.DEBUG:
             # xgboost 1.6 doesn't display all the params in the model str
-            logger.debug(f"flaml.model - {model} fit started with params {self.params}")
+            logger.debug(f"flaml.automl.model - {model} fit started with params {self.params}")
         model.fit(X_train, y_train, **kwargs)
         if logger.level == logging.DEBUG:
-            logger.debug(f"flaml.model - {model} fit finished")
+            logger.debug(f"flaml.automl.model - {model} fit finished")
         train_time = time.time() - current_time
         self._model = model
         return train_time
@@ -455,10 +456,10 @@ class SparkEstimator(BaseEstimator):
         current_time = time.time()
         pipeline_model = self.estimator_class(**self.params, **kwargs)
         if logger.level == logging.DEBUG:
-            logger.debug(f"flaml.model - {pipeline_model} fit started with params {self.params}")
+            logger.debug(f"flaml.automl.model - {pipeline_model} fit started with params {self.params}")
         pipeline_model.fit(df_train)
         if logger.level == logging.DEBUG:
-            logger.debug(f"flaml.model - {pipeline_model} fit finished")
+            logger.debug(f"flaml.automl.model - {pipeline_model} fit finished")
         train_time = time.time() - current_time
         self._model = pipeline_model
         return train_time
@@ -690,12 +691,12 @@ class SparkLGBMEstimator(SparkEstimator):
         current_time = time.time()
         model = self.estimator_class(**self.params, **kwargs)
         if logger.level == logging.DEBUG:
-            logger.debug(f"flaml.model - {model} fit started with params {self.params}")
+            logger.debug(f"flaml.automl.model - {model} fit started with params {self.params}")
         self._model = model.fit(df_train)
         self._model.classes_ = self.model_classes_
         self._model.n_classes_ = self.model_n_classes_
         if logger.level == logging.DEBUG:
-            logger.debug(f"flaml.model - {model} fit finished")
+            logger.debug(f"flaml.automl.model - {model} fit finished")
         train_time = time.time() - current_time
         return train_time
 
@@ -1412,7 +1413,7 @@ class LGBMEstimator(BaseEstimator):
                 callbacks = self.params.pop("callbacks")
                 self._model.set_params(callbacks=callbacks[:-1])
             best_iteration = (
-                self._model.get_booster().best_iteration
+                getattr(self._model.get_booster(), "best_iteration", None)
                 if isinstance(self, XGBoostSklearnEstimator)
                 else self._model.best_iteration_
             )
@@ -1510,8 +1511,6 @@ class XGBoostEstimator(SKLearnEstimator):
         # params["booster"] = params.get("booster", "gbtree")
 
         # use_label_encoder is deprecated in 1.7.
-        from xgboost import __version__ as xgboost_version
-
         if xgboost_version < "1.7.0":
             params["use_label_encoder"] = params.get("use_label_encoder", False)
         if "n_jobs" in config:
@@ -1559,7 +1558,7 @@ class XGBoostEstimator(SKLearnEstimator):
                 obj=obj,
                 callbacks=callbacks,
             )
-            self.params["n_estimators"] = self._model.best_iteration + 1
+            self.params["n_estimators"] = getattr(self._model, "best_iteration", _n_estimators - 1) + 1
         else:
             self._model = xgb.train(self.params, dtrain, _n_estimators, obj=obj)
             self.params["n_estimators"] = _n_estimators
@@ -1620,7 +1619,9 @@ class XGBoostSklearnEstimator(SKLearnEstimator, LGBMEstimator):
         if max_depth == 0:
             params["grow_policy"] = params.get("grow_policy", "lossguide")
             params["tree_method"] = params.get("tree_method", "hist")
-        params["use_label_encoder"] = params.get("use_label_encoder", False)
+        # use_label_encoder is deprecated in 1.7.
+        if xgboost_version < "1.7.0":
+            params["use_label_encoder"] = params.get("use_label_encoder", False)
         return params
 
     def __init__(

flaml/automl/time_series/ts_model.py

@@ -22,7 +22,7 @@ except ImportError:
 import numpy as np
 
 from flaml import tune
-from flaml.model import (
+from flaml.automl.model import (
     suppress_stdout_stderr,
     SKLearnEstimator,
     logger,
@@ -33,7 +33,7 @@ from flaml.model import (
     XGBoostLimitDepthEstimator,
     CatBoostEstimator,
 )
-from flaml.data import TS_TIMESTAMP_COL, TS_VALUE_COL
+from flaml.automl.data import TS_TIMESTAMP_COL, TS_VALUE_COL
 from flaml.automl.time_series.ts_data import (
     TimeSeriesDataset,
     enrich_dataset,

flaml/data.py

@@ -1,9 +0,0 @@
-import warnings
-
-from flaml.automl.data import *
-
-
-warnings.warn(
-    "Importing from `flaml.data` is deprecated. Please use `flaml.automl.data`.",
-    DeprecationWarning,
-)

flaml/default/estimator.py

@@ -105,7 +105,12 @@ def flamlize_estimator(super_class, name: str, task: str, alternatives=None):
                 # if hasattr(self, "_classes"):
                 #     self._classes = self._label_transformer.classes_
                 # else:
-                self.classes_ = self._label_transformer.classes_
+                try:
+                    self.classes_ = self._label_transformer.classes_
+                except AttributeError:
+                    # xgboost 2: AttributeError: can't set attribute
+                    if "xgb" not in estimator_name:
+                        raise
                 if "xgb" not in estimator_name:
                     # rf and et would do inverse transform automatically; xgb doesn't
                     self._label_transformer = None

flaml/model.py

@@ -1,9 +0,0 @@
-import warnings
-
-from flaml.automl.model import *
-
-
-warnings.warn(
-    "Importing from `flaml.model` is deprecated. Please use `flaml.automl.model`.",
-    DeprecationWarning,
-)

flaml/version.py

@@ -1 +1 @@
-__version__ = "2.0.0"
+__version__ = "2.1.1"

notebook/autogen_agentchat_MathChat.ipynb

@@ -15,7 +15,7 @@
    "source": [
     "# Auto Generated Agent Chat: Using MathChat to Solve Math Problems\n",
     "\n",
-    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "MathChat is an experimental convesational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the `AssistantAgent` and `MathUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `MathUserProxyAgent` implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337) or the [blogpost](https://microsoft.github.io/FLAML/blog/2023/06/28/MathChat).\n",
     "\n",

notebook/autogen_agentchat_RetrieveChat.ipynb

@@ -9,7 +9,7 @@
     "# Auto Generated Agent Chat: Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering\n",
     "\n",
     "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
-    "Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "RetrieveChat is a convesational system for retrieve augmented code generation and question answering. In this notebook, we demonstrate how to utilize RetrieveChat to generate code and answer questions based on customized documentations that are not present in the LLM's training dataset. RetrieveChat uses the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `RetrieveAssistantAgent` and  `RetrieveUserProxyAgent` implement a different auto-reply mechanism corresponding to the RetrieveChat prompts.\n",
     "\n",

notebook/autogen_agentchat_chess.ipynb

@@ -16,7 +16,7 @@
     "# Auto Generated Agent Chat: Chess Game Playing While Chitchatting by GPT-4 Agents\n",
     "\n",
     "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
-    "Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "This notebook is modified based on https://github.com/ekzhu/FLAML/blob/evaluation/evaluation/chess/play_chess.ipynb\n",
     "\n",
@@ -35,7 +35,7 @@
    "outputs": [],
    "source": [
     "%%capture --no-stderr\n",
-    "# %pip install flaml[autogen]~=2.0.0\n",
+    "# %pip install flaml[autogen]~=2.1.0\n",
     "%pip install chess -U"
    ]
   },
@@ -79,6 +79,7 @@
     "#         \"model\": {\n",
     "#             \"gpt-3.5-turbo\",\n",
     "#             \"gpt-3.5-turbo-16k\",\n",
+    "#             \"gpt-3.5-turbo-16k-0613\",\n",
     "#             \"gpt-3.5-turbo-0301\",\n",
     "#             \"chatgpt-35-turbo-0301\",\n",
     "#             \"gpt-35-turbo-v0301\",\n",
@@ -157,7 +158,7 @@
     "            llm_config={\"temperature\": 0.0, \"config_list\": config_list_gpt4},\n",
     "            max_consecutive_auto_reply=10,\n",
     "        )\n",
-    "        self.register_auto_reply(autogen.ResponsiveAgent, BoardAgent._generate_board_reply)\n",
+    "        self.register_reply(autogen.ConversableAgent, BoardAgent._generate_board_reply)\n",
     "        self.board = board\n",
     "        self.correct_move_messages = defaultdict(list)\n",
     "\n",
@@ -225,8 +226,8 @@
     "            max_consecutive_auto_reply=max_turns,\n",
     "            **kwargs,\n",
     "        )\n",
-    "        self.register_auto_reply(BoardAgent, ChessPlayerAgent._generate_reply_for_board, config=board_agent.board)\n",
-    "        self.register_auto_reply(ChessPlayerAgent, ChessPlayerAgent._generate_reply_for_player, config=board_agent)\n",
+    "        self.register_reply(BoardAgent, ChessPlayerAgent._generate_reply_for_board, config=board_agent.board)\n",
+    "        self.register_reply(ChessPlayerAgent, ChessPlayerAgent._generate_reply_for_player, config=board_agent)\n",
     "        self.update_max_consecutive_auto_reply(board_agent.max_consecutive_auto_reply(), board_agent)\n",
     "\n",
     "    def _generate_reply_for_board(\n",
@@ -261,7 +262,7 @@
     "            return True, None\n",
     "        # converse with the board until a legal move is made or max allowed retries.\n",
     "        # change silent to False to see that conversation.\n",
-    "        self.initiate_chat(board_agent, clear_history=False, message=message, silent=True)\n",
+    "        self.initiate_chat(board_agent, clear_history=False, message=message, silent=self.human_input_mode == \"NEVER\")\n",
     "        # last message sent by the board agent\n",
     "        last_message = self._oai_messages[board_agent][-1]\n",
     "        if last_message[\"role\"] == \"assistant\":\n",
@@ -1009,7 +1010,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.16"
+   "version": "3.9.17"
   },
   "orig_nbformat": 4
  },

notebook/autogen_agentchat_function_call.ipynb

@@ -17,7 +17,7 @@
    "source": [
     "# Auto Generated Agent Chat: Task Solving with Provided Tools as Functions\n",
     "\n",
-    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to make function calls with the new feature of OpenAI models (in model version 0613). A specified prompt and function configs need to be passed to `AssistantAgent` to initialize the agent. The corresponding functions need to be passed to `UserProxyAgent`, which will be responsible for executing any function calls made by `AssistantAgent`. Besides this requirement of matching descriptions with functions, we recommend checking the system message in the `AssistantAgent` to make sure the instructions align with the function call descriptions.\n",
     "\n",

notebook/autogen_agentchat_groupchat.ipynb

@@ -16,7 +16,7 @@
     "# Auto Generated Agent Chat: Group Chat\n",
     "\n",
     "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
-    "Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "This notebook is modified based on https://github.com/microsoft/FLAML/blob/4ea686af5c3e8ff24d9076a7a626c8b28ab5b1d7/notebook/autogen_multiagent_roleplay_chat.ipynb\n",
     "\n",
@@ -30,12 +30,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 105,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%capture --no-stderr\n",
-    "# %pip install flaml[autogen]~=2.0.0"
+    "# %pip install flaml[autogen]~=2.0.2"
    ]
   },
   {
@@ -50,7 +50,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 106,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -59,7 +59,7 @@
     "config_list_gpt4 = autogen.config_list_from_json(\n",
     "    \"OAI_CONFIG_LIST\",\n",
     "    filter_dict={\n",
-    "        \"model\": [\"gpt-4\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "        \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
     "    },\n",
     ")\n",
     "# config_list_gpt35 = autogen.config_list_from_json(\n",
@@ -122,26 +122,27 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 107,
    "metadata": {},
    "outputs": [],
    "source": [
-    "llm_config = {\"config_list\": config_list_gpt4}\n",
-    "human = autogen.UserProxyAgent(\n",
-    "   name=\"Human\",\n",
+    "llm_config = {\"config_list\": config_list_gpt4, \"seed\": 42}\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "   name=\"User_proxy\",\n",
     "   system_message=\"A human admin.\",\n",
     "   code_execution_config={\"last_n_messages\": 2, \"work_dir\": \"groupchat\"},\n",
+    "   human_input_mode=\"TERMINATE\"\n",
     ")\n",
-    "alice = autogen.AssistantAgent(\n",
-    "    name=\"Alice\",\n",
+    "coder = autogen.AssistantAgent(\n",
+    "    name=\"Coder\",\n",
     "    llm_config=llm_config,\n",
     ")\n",
-    "bob = autogen.AssistantAgent(\n",
-    "    name=\"Bob\",\n",
-    "    system_message=\"Code reviewer. Prevent code execution if unsafe or not well documented. Suggest changes. Otherwise, approve and return the final code to execute.\",\n",
+    "pm = autogen.AssistantAgent(\n",
+    "    name=\"Product_manager\",\n",
+    "    system_message=\"Creative in software product ideas.\",\n",
     "    llm_config=llm_config,\n",
     ")\n",
-    "groupchat = autogen.GroupChat(agents=[human, alice, bob], messages=[], max_round=12)\n",
+    "groupchat = autogen.GroupChat(agents=[user_proxy, coder, pm], messages=[], max_round=12)\n",
     "manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)"
    ]
   },
@@ -155,307 +156,113 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 108,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "\u001b[33mHuman\u001b[0m (to chat_manager):\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
       "\n",
-      "find a latest paper about generative agents\n",
+      "Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "\u001b[33mAlice\u001b[0m (to chat_manager):\n",
+      "\u001b[33mCoder\u001b[0m (to chat_manager):\n",
       "\n",
-      "To accomplish this, we can utilize the \"scholarly\" library in Python, which enables us to search Google Scholar for papers. Here's the Python code to achieve this:\n",
+      "To find the latest paper about GPT-4 on arxiv, I'll provide you with a Python code that fetches the most recent papers from the arxiv API and filters the results to get the most relevant paper related to GPT-4. After fetching the paper, I'll extract the information for potential applications in software. Please execute the following Python code:\n",
       "\n",
       "```python\n",
-      "# filename: googlescholar_search.py\n",
+      "import requests\n",
+      "from bs4 import BeautifulSoup\n",
+      "import re\n",
       "\n",
-      "import scholarly\n",
+      "def fetch_arxiv_papers(query):\n",
+      "    base_url = \"http://export.arxiv.org/api/query?\"\n",
+      "    search_query = \"all:\" + query\n",
+      "    response = requests.get(base_url, params={\"search_query\": search_query, \"sortBy\": \"submittedDate\", \"sortOrder\": \"descending\"})\n",
+      "    return BeautifulSoup(response.content, \"xml\")\n",
       "\n",
-      "def get_latest_paper(query):\n",
-      "    search_query = scholarly.search_pubs(query)\n",
-      "    paper = next(search_query)\n",
-      "    print(\"The latest paper is:\", paper.bib['title'])\n",
-      "    print(\"The abstract of the paper is:\", paper.bib['abstract'])\n",
-      "    print(\"The year of publication is:\", paper.bib['year'])\n",
-      "\n",
-      "get_latest_paper(\"Generative agents\")\n",
-      "```\n",
-      "\n",
-      "To execute this script:\n",
-      "1. Save the code to a file named googlescholar_search.py\n",
-      "2. Run `pip install scholarly` to install the necessary library if you haven't installed it yet.\n",
-      "3. Run `python googlescholar_search.py` to execute the script and get the latest paper on generative agents.\n",
-      "\n",
-      "Please note that Google Scholar doesn't provide a stable API and has rate limit restrictions in place, meaning that if you run this code multiple times in a short period, Google might temporarily block your IP.\n",
-      "\n",
-      "If the rate limit becomes an issue, you might want to consider using a different database or method, such as utilizing APIs from databases like arXiv or Pubmed, or web scraping, in a manner compliant with the website's robots.txt and terms of usage.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mBob\u001b[0m (to chat_manager):\n",
-      "\n",
-      "The provided code seems fine, however, according to Google's Terms of Service API users are restricted from programmatically sending requests to Google Scholar. Even being an unofficial API, it doesn't make using scholarly legal as per the use policy. It's important you consider these limitations when handling this tool and any consequences that may arise on its usage. \n",
-      "\n",
-      "Remember to use APIs responsibly and always in accordance with their terms of service. Without explicit permission from Google, using such a tool can get your IP banned. \n",
-      "\n",
-      "I will suggest to use APIs from databases like arXiv or Pubmed, or webscraping, in a manner that is compliant with the website's robots.txt-file and terms of usage.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mAlice\u001b[0m (to chat_manager):\n",
-      "\n",
-      "I apologize for the oversight. You're correct, direct scraping of Google Scholar violates Google's terms of service. Let's change to use the arXiv API which doesn't have this issue and is more reliable. Here's the python code:\n",
-      "\n",
-      "```python\n",
-      "# filename: arxiv_search.py\n",
-      "import urllib\n",
-      "import feedparser\n",
-      "\n",
-      "def search_arxiv(query: str):\n",
-      "    base_url = 'http://export.arxiv.org/api/query?'\n",
-      "    query = {'search_query' : f'ti:{query}', 'start' : 0, 'max_results' : 1, 'sortBy' : 'submittedDate', 'sortOrder' : 'descending'}\n",
-      "    url = base_url + urllib.parse.urlencode(query)\n",
-      "  \n",
-      "    # connect to arXiv API and get response\n",
-      "    response = urllib.request.urlopen(url).read()\n",
-      "\n",
-      "    # parse the response using feedparser\n",
-      "    feed = feedparser.parse(response)\n",
-      "  \n",
-      "    # get the first (and presumably, the most recent) article in the result\n",
-      "    entry = feed.entries[0]\n",
-      "\n",
-      "    # print details of the most recent article\n",
-      "    print('The latest paper on', query['search_query'], 'that I could find is:\\n')\n",
-      "    print('Title: ', entry.title)\n",
-      "    print('Author: ', entry.author)\n",
-      "    print('Link: ', entry.link)\n",
-      "    print('\\nAbstract: ', entry.summary)\n",
-      "\n",
-      "# search for the latest paper about \"generative agents\"\n",
-      "search_arxiv(\"generative agents\")\n",
-      "```\n",
-      "\n",
-      "To execute this script:\n",
-      "1. Save the code to a file named arxiv_search.py\n",
-      "2. Run `pip install feedparser` to install the necessary library.\n",
-      "3. Run `python arxiv_search.py` to execute the script and get the latest paper on generative agents.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mBob\u001b[0m (to chat_manager):\n",
-      "\n",
-      "The provided code snippet is clear, efficient, and well-documented. It appropriately uses the arXiv API to retrieve the most recent papers about \"generative agents\". The search terms are correctly URI-encoded and passed to the arXiv query API, and proper error handling is in place.\n",
-      "\n",
-      "However, you should ensure that you handle potential exception which may occur when trying to connect to the URL and parse the response. For example, if the internet is disconnected or something is wrong with the server, `urllib.request.urlopen()` will raise a `URLError`. If the returned content is not properly formatted, `feedparser.parse()` may also fail. You should catch and properly handle these exceptions.\n",
-      "\n",
-      "Therefore, I would add these modifications:\n",
-      "\n",
-      "```python\n",
-      "from urllib.error import URLError\n",
-      "\n",
-      "try:\n",
-      "    # connect to arXiv API and get response\n",
-      "    response = urllib.request.urlopen(url).read()\n",
-      "\n",
-      "except URLError as e:\n",
-      "    print(\"There was a problem connecting to the arXiv API:\")\n",
-      "    print(e.reason)\n",
+      "def find_gpt4_paper():\n",
+      "    papers = fetch_arxiv_papers(\"gpt-4\")\n",
+      "    for entry in papers.find_all(\"entry\"):\n",
+      "        title = entry.title.text.strip()\n",
+      "        summary = entry.summary.text.strip()\n",
+      "        if \"gpt-4\" in title.lower() or \"gpt-4\" in summary.lower():\n",
+      "            return {\"title\": title, \"summary\": summary}\n",
       "\n",
+      "gpt4_paper = find_gpt4_paper()\n",
+      "if gpt4_paper:\n",
+      "    print(\"Title:\", gpt4_paper[\"title\"])\n",
+      "    print(\"Summary:\", gpt4_paper[\"summary\"])\n",
       "else:\n",
-      "    try:\n",
-      "        # parse the response using feedparser\n",
-      "        feed = feedparser.parse(response)\n",
-      "      \n",
-      "        # get the first (and presumably, the most recent) article in the result\n",
-      "        entry = feed.entries[0]\n",
-      "\n",
-      "    except Exception as e:\n",
-      "        print(\"There was a problem parsing the result:\")\n",
-      "        print(e)\n",
-      "\n",
-      "    else:\n",
-      "        # print details of the most recent article\n",
-      "        print('The latest paper on', query['search_query'], 'that I could find is:\\n')\n",
-      "        print('Title: ', entry.title)\n",
-      "        print('Author: ', entry.author)\n",
-      "        print('Link: ', entry.link)\n",
-      "        print('\\nAbstract: ', entry.summary)\n",
+      "    print(\"No recent GPT-4 papers found.\")\n",
       "```\n",
       "\n",
-      "The keyword `except` is used to catch and handle exceptions. The modifications suggested include exception handlers for `URLError` (which are raised if there was a problem connecting to the arXiv API) and a generic Exception (which could be any other exception during parsing the response). The `else` keyword allows us to group together the normal operation code, separating it from the error handling code. \n",
-      "\n",
-      "The code is ready to be executed now.\n",
+      "Once we have the paper details, I'll analyze the summary to identify potential applications in software development.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
       ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
-      "\u001b[33mHuman\u001b[0m (to chat_manager):\n",
-      "\n",
-      "exitcode: 1 (execution failed)\n",
-      "Code output: \n",
-      "Traceback (most recent call last):\n",
-      "  File \"\", line 5, in <module>\n",
-      "    response = urllib.request.urlopen(url).read()\n",
-      "NameError: name 'urllib' is not defined\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mAlice\u001b[0m (to chat_manager):\n",
-      "\n",
-      "Apologies for the oversight. It looks like I missed importing the required `urllib.request` module. Please use the following updated code with the necessary import statement:\n",
-      "\n",
-      "```python\n",
-      "# filename: arxiv_search.py\n",
-      "import urllib.request\n",
-      "import urllib.parse\n",
-      "import feedparser\n",
-      "from urllib.error import URLError\n",
-      "\n",
-      "def search_arxiv(query: str):\n",
-      "    base_url = 'http://export.arxiv.org/api/query?'\n",
-      "    query = {'search_query' : f'ti:{query}', 'start' : 0, 'max_results' : 1, 'sortBy' : 'submittedDate', 'sortOrder' : 'descending'}\n",
-      "    url = base_url + urllib.parse.urlencode(query)\n",
-      "  \n",
-      "    try:\n",
-      "        # connect to arXiv API and get response\n",
-      "        response = urllib.request.urlopen(url).read()\n",
-      "\n",
-      "    except URLError as e:\n",
-      "        print(\"There was a problem connecting to the arXiv API:\")\n",
-      "        print(e.reason)\n",
-      "\n",
-      "    else:\n",
-      "        try:\n",
-      "            # parse the response using feedparser\n",
-      "            feed = feedparser.parse(response)\n",
-      "          \n",
-      "            # get the first (and presumably, the most recent) article in the result\n",
-      "            entry = feed.entries[0]\n",
-      "\n",
-      "        except Exception as e:\n",
-      "            print(\"There was a problem parsing the result:\")\n",
-      "            print(e)\n",
-      "\n",
-      "        else:\n",
-      "            # print details of the most recent article\n",
-      "            print('The latest paper on', query['search_query'], 'that I could find is:\\n')\n",
-      "            print('Title: ', entry.title)\n",
-      "            print('Author: ', entry.author)\n",
-      "            print('Link: ', entry.link)\n",
-      "            print('\\nAbstract: ', entry.summary)\n",
-      "\n",
-      "# search for the latest paper about \"generative agents\"\n",
-      "search_arxiv(\"generative agents\")\n",
-      "```\n",
-      "\n",
-      "To execute this script:\n",
-      "1. Save the code to a file named arxiv_search.py\n",
-      "2. Run `pip install feedparser` to install the necessary library.\n",
-      "3. Run `python arxiv_search.py` to execute the script and get the latest paper on generative agents.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mBob\u001b[0m (to chat_manager):\n",
-      "\n",
-      "The amendment is correctly appended. You have properly imported the required `urllib.request` module. The code is now ready for execution. It correctly searches for the latest paper about \"generative agents\" using the arXiv API, and handles potential exceptions that may arise during the connection to the API or while parsing the response. Please proceed to execute this code.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[31m\n",
-      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
-      "\u001b[33mHuman\u001b[0m (to chat_manager):\n",
-      "\n",
-      "exitcode: 1 (execution failed)\n",
-      "Code output: \n",
-      "Traceback (most recent call last):\n",
-      "  File \"arxiv_search.py\", line 4, in <module>\n",
-      "    import feedparser\n",
-      "ModuleNotFoundError: No module named 'feedparser'\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mAlice\u001b[0m (to chat_manager):\n",
-      "\n",
-      "My apologies for the inconvenience. It appears that the 'feedparser' module is not installed. Please install the 'feedparser' module by running the following command:\n",
-      "\n",
-      "```sh\n",
-      "pip install feedparser\n",
-      "```\n",
-      "\n",
-      "Once you have installed the 'feedparser' module, please execute the arxiv_search.py script again:\n",
-      "\n",
-      "```sh\n",
-      "python arxiv_search.py\n",
-      "```\n",
-      "\n",
-      "This should execute the script and fetch the latest paper on generative agents.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[33mBob\u001b[0m (to chat_manager):\n",
-      "\n",
-      "That's correct. Make sure to install the 'feedparser' module using the provided command, and then you should be able to execute the updated arxiv_search.py script successfully. The script will search for the latest paper about \"generative agents\" using the arXiv API, and return information about the most recent article it finds.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\u001b[31m\n",
-      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is sh)...\u001b[0m\n",
-      "\u001b[31m\n",
-      ">>>>>>>> EXECUTING CODE BLOCK 1 (inferred language is sh)...\u001b[0m\n",
-      "\u001b[33mHuman\u001b[0m (to chat_manager):\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
       "\n",
       "exitcode: 0 (execution succeeded)\n",
       "Code output: \n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Collecting feedparser\n",
-      "  Downloading feedparser-6.0.10-py3-none-any.whl (81 kB)\n",
-      "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.1/81.1 KB 17.2 MB/s eta 0:00:00\n",
-      "Collecting sgmllib3k\n",
-      "  Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB)\n",
-      "  Preparing metadata (setup.py): started\n",
-      "  Preparing metadata (setup.py): finished with status 'done'\n",
-      "Building wheels for collected packages: sgmllib3k\n",
-      "  Building wheel for sgmllib3k (setup.py): started\n",
-      "  Building wheel for sgmllib3k (setup.py): finished with status 'done'\n",
-      "  Created wheel for sgmllib3k: filename=sgmllib3k-1.0.0-py3-none-any.whl size=6046 sha256=867dc31954f27685ad79808f2ca2b5d8235496de750c61f110c200ba664a50e4\n",
-      "  Stored in directory: /home/vscode/.cache/pip/wheels/65/7a/a7/78c287f64e401255dff4c13fdbc672fed5efbfd21c530114e1\n",
-      "Successfully built sgmllib3k\n",
-      "Installing collected packages: sgmllib3k, feedparser\n",
-      "Successfully installed feedparser-6.0.10 sgmllib3k-1.0.0\n",
+      "Title: FIMO: A Challenge Formal Dataset for Automated Theorem Proving\n",
+      "Summary: We present FIMO, an innovative dataset comprising formal mathematical problem\n",
+      "statements sourced from the International Mathematical Olympiad (IMO)\n",
+      "Shortlisted Problems. Designed to facilitate advanced automated theorem proving\n",
+      "at the IMO level, FIMO is currently tailored for the Lean formal language. It\n",
+      "comprises 149 formal problem statements, accompanied by both informal problem\n",
+      "descriptions and their corresponding LaTeX-based informal proofs. Through\n",
+      "initial experiments involving GPT-4, our findings underscore the existing\n",
+      "limitations in current methodologies, indicating a substantial journey ahead\n",
+      "before achieving satisfactory IMO-level automated theorem proving outcomes.\n",
       "\n",
-      "The latest paper on ti:generative agents that I could find is:\n",
       "\n",
-      "Title:  LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image\n",
-      "  Generation\n",
-      "Author:  Tat-Seng Chua\n",
-      "Link:  http://arxiv.org/abs/2308.05095v1\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mProduct_manager\u001b[0m (to chat_manager):\n",
       "\n",
-      "Abstract:  In the text-to-image generation field, recent remarkable progress in Stable\n",
-      "Diffusion makes it possible to generate rich kinds of novel photorealistic\n",
-      "images. However, current models still face misalignment issues (e.g.,\n",
-      "problematic spatial relation understanding and numeration failure) in complex\n",
-      "natural scenes, which impedes the high-faithfulness text-to-image generation.\n",
-      "Although recent efforts have been made to improve controllability by giving\n",
-      "fine-grained guidance (e.g., sketch and scribbles), this issue has not been\n",
-      "fundamentally tackled since users have to provide such guidance information\n",
-      "manually. In this work, we strive to synthesize high-fidelity images that are\n",
-      "semantically aligned with a given textual prompt without any guidance. Toward\n",
-      "this end, we propose a coarse-to-fine paradigm to achieve layout planning and\n",
-      "image generation. Concretely, we first generate the coarse-grained layout\n",
-      "conditioned on a given textual prompt via in-context learning based on Large\n",
-      "Language Models. Afterward, we propose a fine-grained object-interaction\n",
-      "diffusion method to synthesize high-faithfulness images conditioned on the\n",
-      "prompt and the automatically generated layout. Extensive experiments\n",
-      "demonstrate that our proposed method outperforms the state-of-the-art models in\n",
-      "terms of layout and image generation. Our code and settings are available at\n",
-      "\\url{https://layoutllm-t2i.github.io}.\n",
+      "Based on the paper titled \"FIMO: A Challenge Formal Dataset for Automated Theorem Proving\" and its summary, the potential applications of GPT-4 in software development can be related to the field of automated theorem proving.\n",
       "\n",
+      "1. **Automated theorem proving**: GPT-4 can be utilized in the development of automated theorem proving software that attempts to prove complex mathematical problems taken from International Mathematical Olympiad (IMO) or other challenging sources. By fine-tuning GPT-4 with a dataset like FIMO consisting of formal mathematical problems, the model can potentially better understand the problem statements and generate appropriate proofs.\n",
+      "\n",
+      "2. **Mathematical problem-solving assistants**: Software tools can be developed using GPT-4 to guide users in solving complex mathematical problems. The AI model can be integrated into educational platforms, online math tutoring services, or even standalone tools to help make solving problems easier and faster for students and professionals alike.\n",
+      "\n",
+      "3. **Formal language translation**: GPT-4 can potentially be integrated into software for translating between formal languages, assisting in the understanding and comparison of various formal systems. This would be especially useful in research communities employing different formal languages and wanting to share ideas and results.\n",
+      "\n",
+      "4. **Mathematical proof checking**: GPT-4 can be employed in proof-checking software to identify and correct inconsistencies. By improving the correctness of proofs, this application would ultimately help users save time and contribute to the overall quality of mathematical research.\n",
+      "\n",
+      "Please note that this paper highlights the current limitations of GPT-4 in the context of IMO-level theorem proving. Nevertheless, these potential applications suggest directions for further research and software development as the model and related techniques continue to improve.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mUser_proxy\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCoder\u001b[0m (to chat_manager):\n",
+      "\n",
+      "TERMINATE\n",
       "\n",
       "--------------------------------------------------------------------------------\n"
      ]
     }
    ],
    "source": [
-    "human.initiate_chat(manager, message=\"find a latest paper about generative agents\")"
+    "user_proxy.initiate_chat(manager, message=\"Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\")\n",
+    "# type exit to terminate the chat"
    ]
   }
  ],

notebook/autogen_agentchat_groupchat_research.ipynb

@@ -0,0 +1,566 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Auto Generated Agent Chat: Performs Research with Multi-Agent Group Chat\n",
+    "\n",
+    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
+    "```bash\n",
+    "pip install flaml[autogen]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "# %pip install flaml[autogen]~=2.0.3"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API Endpoint\n",
+    "\n",
+    "The [`config_list_from_json`](https://microsoft.github.io/FLAML/docs/reference/autogen/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from flaml import autogen\n",
+    "\n",
+    "config_list_gpt4 = autogen.config_list_from_json(\n",
+    "    \"OAI_CONFIG_LIST\",\n",
+    "    filter_dict={\n",
+    "        \"model\": [\"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). Only the gpt-4-32k models are kept in the list based on the filter condition.\n",
+    "\n",
+    "The config list looks like the following:\n",
+    "```python\n",
+    "config_list = [\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your OpenAI API key here>',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "    {\n",
+    "        'model': 'gpt-4-32k-0314',\n",
+    "        'api_key': '<your Azure OpenAI API key here>',\n",
+    "        'api_base': '<your Azure OpenAI API base here>',\n",
+    "        'api_type': 'azure',\n",
+    "        'api_version': '2023-06-01-preview',\n",
+    "    },\n",
+    "]\n",
+    "```\n",
+    "\n",
+    "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
+    "\n",
+    "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct Agents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gpt4_config = {\n",
+    "    \"seed\": 42,  # change the seed for different trials\n",
+    "    \"temperature\": 0,\n",
+    "    \"config_list\": config_list_gpt4,\n",
+    "    \"request_timeout\": 120,\n",
+    "}\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "   name=\"Admin\",\n",
+    "   system_message=\"A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.\",\n",
+    "   code_execution_config=False,\n",
+    ")\n",
+    "engineer = autogen.AssistantAgent(\n",
+    "    name=\"Engineer\",\n",
+    "    llm_config=gpt4_config,\n",
+    "    system_message='''Engineer. You follow an approved plan. You write python/shell code to solve tasks. Wrap the code in a code block that specifies the script type. The user can't modify your code. So do not suggest incomplete code which requires others to modify. Don't use a code block if it's not intended to be executed by the executor.\n",
+    "Don't include multiple code blocks in one response. Do not ask others to copy and paste the result. Check the execution result returned by the executor.\n",
+    "If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\n",
+    "''',\n",
+    ")\n",
+    "scientist = autogen.AssistantAgent(\n",
+    "    name=\"Scientist\",\n",
+    "    llm_config=gpt4_config,\n",
+    "    system_message=\"\"\"Scientist. You follow an approved plan. You are able to categorize papers after seeing their abstracts printed. You don't write code.\"\"\"\n",
+    ")\n",
+    "planner = autogen.AssistantAgent(\n",
+    "    name=\"Planner\",\n",
+    "    system_message='''Planner. Suggest a plan. Revise the plan based on feedback from admin and critic, until admin approval.\n",
+    "The plan may involve an engineer who can write code and a scientist who doesn't write code.\n",
+    "Explain the plan first. Be clear which step is performed by an engineer, and which step is performed by a scientist.\n",
+    "''',\n",
+    "    llm_config=gpt4_config,\n",
+    ")\n",
+    "executor = autogen.UserProxyAgent(\n",
+    "    name=\"Executor\",\n",
+    "    system_message=\"Executor. Execute the code written by the engineer and report the result.\",\n",
+    "    human_input_mode=\"NEVER\",\n",
+    "    code_execution_config={\"last_n_messages\": 3, \"work_dir\": \"paper\"},\n",
+    ")\n",
+    "critic = autogen.AssistantAgent(\n",
+    "    name=\"Critic\",\n",
+    "    system_message=\"Critic. Double check plan, claims, code from other agents and provide feedback. Check whether the plan includes adding verifiable info such as source URL.\",\n",
+    "    llm_config=gpt4_config,\n",
+    ")\n",
+    "groupchat = autogen.GroupChat(agents=[user_proxy, engineer, scientist, planner, executor, critic], messages=[], max_round=50)\n",
+    "manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=gpt4_config)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Start Chat"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mAdmin\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mPlanner\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Plan:\n",
+      "\n",
+      "1. Engineer: Write a script to scrape the arXiv website for papers related to LLM (Language Model) applications published in the last week. The script should extract the title, authors, abstract, and link to the paper.\n",
+      "\n",
+      "2. Scientist: Review the scraped data to identify the different domains in which LLM is applied. This could be based on keywords in the title or abstract, or the scientist's knowledge of the field.\n",
+      "\n",
+      "3. Engineer: Modify the script to categorize the papers based on the domains identified by the scientist. The script should output a markdown table with columns for the domain, title, authors, abstract, and link.\n",
+      "\n",
+      "4. Scientist: Review the markdown table to ensure the papers are correctly categorized and the information is accurate.\n",
+      "\n",
+      "5. Engineer: Make any necessary revisions to the script based on the scientist's feedback.\n",
+      "\n",
+      "6. Scientist: Give final approval of the markdown table.\n",
+      "\n",
+      "7. Engineer: Submit the final markdown table.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCritic\u001b[0m (to chat_manager):\n",
+      "\n",
+      "The plan seems solid and well-structured. However, it lacks the inclusion of verifiable information such as source URLs. Here's a revised version:\n",
+      "\n",
+      "1. Engineer: Write a script to scrape the arXiv website for papers related to LLM (Language Model) applications published in the last week. The script should extract the title, authors, abstract, and link to the paper.\n",
+      "\n",
+      "2. Scientist: Review the scraped data to identify the different domains in which LLM is applied. This could be based on keywords in the title or abstract, or the scientist's knowledge of the field.\n",
+      "\n",
+      "3. Engineer: Modify the script to categorize the papers based on the domains identified by the scientist. The script should output a markdown table with columns for the domain, title, authors, abstract, and link (source URL).\n",
+      "\n",
+      "4. Scientist: Review the markdown table to ensure the papers are correctly categorized and the information is accurate. Check the source URLs to verify the information.\n",
+      "\n",
+      "5. Engineer: Make any necessary revisions to the script based on the scientist's feedback.\n",
+      "\n",
+      "6. Scientist: Give final approval of the markdown table, ensuring all entries have a valid source URL.\n",
+      "\n",
+      "7. Engineer: Submit the final markdown table.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mAdmin\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mEngineer\u001b[0m (to chat_manager):\n",
+      "\n",
+      "I'm sorry for the confusion, but as an AI model developed by OpenAI, I don't have the ability to directly scrape websites or access real-time data from external databases or websites, including arXiv. However, I can help you write a Python script that uses the arXiv API to fetch the data you need.\n",
+      "\n",
+      "Here's a Python script that uses the `arxiv` package to search for papers related to \"LLM applications\" from the last week. This script will print out the title, authors, summary, and URL of each paper.\n",
+      "\n",
+      "```python\n",
+      "import arxiv\n",
+      "import datetime\n",
+      "\n",
+      "# Calculate the date one week ago\n",
+      "one_week_ago = (datetime.datetime.now() - datetime.timedelta(days=7)).strftime('%Y%m%d%H%M%S')\n",
+      "\n",
+      "# Search for papers on LLM applications\n",
+      "search = arxiv.Search(\n",
+      "  query=\"LLM applications\",\n",
+      "  max_results=100,\n",
+      "  sort_by=arxiv.SortCriterion.SubmittedDate,\n",
+      "  sort_order=arxiv.SortOrder.Descending\n",
+      ")\n",
+      "\n",
+      "for result in search.get():\n",
+      "  # Check if the paper was submitted in the last week\n",
+      "  if result.submitted.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "    print(f\"Title: {result.title}\")\n",
+      "    print(f\"Authors: {', '.join(author.name for author in result.authors)}\")\n",
+      "    print(f\"Summary: {result.summary}\")\n",
+      "    print(f\"URL: {result.entry_id}\")\n",
+      "    print(\"\\n\")\n",
+      "```\n",
+      "\n",
+      "You can install the `arxiv` package using pip:\n",
+      "\n",
+      "```shell\n",
+      "pip install arxiv\n",
+      "```\n",
+      "\n",
+      "Please note that you'll need to replace `\"LLM applications\"` with the actual search term you're interested in. The arXiv API doesn't support searching by the submission date, so the script fetches the 100 most recent results and filters them manually. You may need to increase `max_results` if there are more than 100 papers submitted per week on your topic.\n",
+      "\n",
+      "To create a markdown table, you'll need to format the output like this:\n",
+      "\n",
+      "```markdown\n",
+      "| Title | Authors | Summary | URL |\n",
+      "|-------|---------|---------|-----|\n",
+      "| ...   | ...     | ...     | ... |\n",
+      "```\n",
+      "\n",
+      "You can do this in Python using f-strings:\n",
+      "\n",
+      "```python\n",
+      "print(\"| Title | Authors | Summary | URL |\")\n",
+      "print(\"|-------|---------|---------|-----|\")\n",
+      "for result in search.get():\n",
+      "  # Check if the paper was submitted in the last week\n",
+      "  if result.submitted.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "    print(f\"| {result.title} | {', '.join(author.name for author in result.authors)} | {result.summary} | {result.entry_id} |\")\n",
+      "```\n",
+      "\n",
+      "This will print a markdown table to the console. If you want to write the table to a file, you can open a file and use the `write` method instead of `print`.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mExecutor\u001b[0m (to chat_manager):\n",
+      "\n",
+      "exitcode: 1 (execution failed)\n",
+      "Code output: \n",
+      ":15: DeprecationWarning: The 'get' method is deprecated, use 'results' instead\n",
+      "  for result in search.get():\n",
+      "Traceback (most recent call last):\n",
+      "  File \"\", line 17, in <module>\n",
+      "    if result.submitted.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "AttributeError: 'Result' object has no attribute 'submitted'\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mEngineer\u001b[0m (to chat_manager):\n",
+      "\n",
+      "I apologize for the confusion. It seems there was a mistake in the code. The `arxiv` package's `Result` object doesn't have a `submitted` attribute. Instead, we should use the `published` attribute to get the date when the paper was published.\n",
+      "\n",
+      "Here's the corrected code:\n",
+      "\n",
+      "```python\n",
+      "import arxiv\n",
+      "import datetime\n",
+      "\n",
+      "# Calculate the date one week ago\n",
+      "one_week_ago = (datetime.datetime.now() - datetime.timedelta(days=7)).strftime('%Y%m%d%H%M%S')\n",
+      "\n",
+      "# Search for papers on LLM applications\n",
+      "search = arxiv.Search(\n",
+      "  query=\"LLM applications\",\n",
+      "  max_results=100,\n",
+      "  sort_by=arxiv.SortCriterion.Relevance,\n",
+      "  sort_order=arxiv.SortOrder.Descending\n",
+      ")\n",
+      "\n",
+      "print(\"| Title | Authors | Summary | URL |\")\n",
+      "print(\"|-------|---------|---------|-----|\")\n",
+      "for result in search.results():\n",
+      "  # Check if the paper was published in the last week\n",
+      "  if result.published.strftime('%Y%m%d%H%M%S') > one_week_ago:\n",
+      "    print(f\"| {result.title} | {', '.join(author.name for author in result.authors)} | {result.summary} | {result.entry_id} |\")\n",
+      "```\n",
+      "\n",
+      "This script will print a markdown table of papers related to \"LLM applications\" that were published in the last week. The table includes the title, authors, summary, and URL of each paper.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mExecutor\u001b[0m (to chat_manager):\n",
+      "\n",
+      "exitcode: 0 (execution succeeded)\n",
+      "Code output: \n",
+      "| Title | Authors | Summary | URL |\n",
+      "|-------|---------|---------|-----|\n",
+      "| Large Language Models as Data Preprocessors | Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada | Large Language Models (LLMs), typified by OpenAI's GPT series and Meta's\n",
+      "LLaMA variants, have marked a significant advancement in artificial\n",
+      "intelligence. Trained on vast amounts of text data, LLMs are capable of\n",
+      "understanding and generating human-like text across a diverse range of topics.\n",
+      "This study expands on the applications of LLMs, exploring their potential in\n",
+      "data preprocessing, a critical stage in data mining and analytics applications.\n",
+      "We delve into the applicability of state-of-the-art LLMs such as GPT-3.5,\n",
+      "GPT-4, and Vicuna-13B for error detection, data imputation, schema matching,\n",
+      "and entity matching tasks. Alongside showcasing the inherent capabilities of\n",
+      "LLMs, we highlight their limitations, particularly in terms of computational\n",
+      "expense and inefficiency. We propose an LLM-based framework for data\n",
+      "preprocessing, which integrates cutting-edge prompt engineering techniques,\n",
+      "coupled with traditional methods like contextualization and feature selection,\n",
+      "to improve the performance and efficiency of these models. The effectiveness of\n",
+      "LLMs in data preprocessing is evaluated through an experimental study spanning\n",
+      "12 datasets. GPT-4 emerged as a standout, achieving 100\\% accuracy or F1 score\n",
+      "on 4 datasets, suggesting LLMs' immense potential in these tasks. Despite\n",
+      "certain limitations, our study underscores the promise of LLMs in this domain\n",
+      "and anticipates future developments to overcome current hurdles. | http://arxiv.org/abs/2308.16361v1 |\n",
+      "| Large language models in medicine: the potentials and pitfalls | Jesutofunmi A. Omiye, Haiwen Gui, Shawheen J. Rezaei, James Zou, Roxana Daneshjou | Large language models (LLMs) have been applied to tasks in healthcare,\n",
+      "ranging from medical exam questions to responding to patient questions. With\n",
+      "increasing institutional partnerships between companies producing LLMs and\n",
+      "healthcare systems, real world clinical application is coming closer to\n",
+      "reality. As these models gain traction, it is essential for healthcare\n",
+      "practitioners to understand what LLMs are, their development, their current and\n",
+      "potential applications, and the associated pitfalls when utilized in medicine.\n",
+      "This review and accompanying tutorial aim to give an overview of these topics\n",
+      "to aid healthcare practitioners in understanding the rapidly changing landscape\n",
+      "of LLMs as applied to medicine. | http://arxiv.org/abs/2309.00087v1 |\n",
+      "| Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following | Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, Pheng-Ann Heng | We introduce Point-Bind, a 3D multi-modality model aligning point clouds with\n",
+      "2D image, language, audio, and video. Guided by ImageBind, we construct a joint\n",
+      "embedding space between 3D and multi-modalities, enabling many promising\n",
+      "applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D\n",
+      "open-world understanding. On top of this, we further present Point-LLM, the\n",
+      "first 3D large language model (LLM) following 3D multi-modal instructions. By\n",
+      "parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of\n",
+      "Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction\n",
+      "data, but exhibits superior 3D and multi-modal question-answering capacity. We\n",
+      "hope our work may cast a light on the community for extending 3D point clouds\n",
+      "to multi-modality applications. Code is available at\n",
+      "https://github.com/ZiyuGuo99/Point-Bind_Point-LLM. | http://arxiv.org/abs/2309.00615v1 |\n",
+      "| Where Would I Go Next? Large Language Models as Human Mobility Predictors | Xinglei Wang, Meng Fang, Zichao Zeng, Tao Cheng | Accurate human mobility prediction underpins many important applications\n",
+      "across a variety of domains, including epidemic modelling, transport planning,\n",
+      "and emergency responses. Due to the sparsity of mobility data and the\n",
+      "stochastic nature of people's daily activities, achieving precise predictions\n",
+      "of people's locations remains a challenge. While recently developed large\n",
+      "language models (LLMs) have demonstrated superior performance across numerous\n",
+      "language-related tasks, their applicability to human mobility studies remains\n",
+      "unexplored. Addressing this gap, this article delves into the potential of LLMs\n",
+      "for human mobility prediction tasks. We introduce a novel method, LLM-Mob,\n",
+      "which leverages the language understanding and reasoning capabilities of LLMs\n",
+      "for analysing human mobility data. We present concepts of historical stays and\n",
+      "context stays to capture both long-term and short-term dependencies in human\n",
+      "movement and enable time-aware prediction by using time information of the\n",
+      "prediction target. Additionally, we design context-inclusive prompts that\n",
+      "enable LLMs to generate more accurate predictions. Comprehensive evaluations of\n",
+      "our method reveal that LLM-Mob excels in providing accurate and interpretable\n",
+      "predictions, highlighting the untapped potential of LLMs in advancing human\n",
+      "mobility prediction techniques. We posit that our research marks a significant\n",
+      "paradigm shift in human mobility modelling, transitioning from building complex\n",
+      "domain-specific models to harnessing general-purpose LLMs that yield accurate\n",
+      "predictions through language instructions. The code for this work is available\n",
+      "at https://github.com/xlwang233/LLM-Mob. | http://arxiv.org/abs/2308.15197v1 |\n",
+      "| Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model | Kazuki Hori, Kanata Suzuki, Tetsuya Ogata | The application of the Large Language Model (LLM) to robot action planning\n",
+      "has been actively studied. The instructions given to the LLM by natural\n",
+      "language may include ambiguity and lack of information depending on the task\n",
+      "context. It is possible to adjust the output of LLM by making the instruction\n",
+      "input more detailed; however, the design cost is high. In this paper, we\n",
+      "propose the interactive robot action planning method that allows the LLM to\n",
+      "analyze and gather missing information by asking questions to humans. The\n",
+      "method can minimize the design cost of generating precise robot instructions.\n",
+      "We demonstrated the effectiveness of our method through concrete examples in\n",
+      "cooking tasks. However, our experiments also revealed challenges in robot\n",
+      "action planning with LLM, such as asking unimportant questions and assuming\n",
+      "crucial information without asking. Shedding light on these issues provides\n",
+      "valuable insights for future research on utilizing LLM for robotics. | http://arxiv.org/abs/2308.15684v1 |\n",
+      "| AskIt: Unified Programming Interface for Programming with Large Language Models | Katsumi Okuda, Saman Amarasinghe | In the evolving landscape of software development, Large Language Models\n",
+      "(LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating\n",
+      "adeptness across numerous tasks, from text summarization to code generation.\n",
+      "While these abilities open up novel avenues in software design and crafting,\n",
+      "their incorporation presents substantial challenges. Developers grapple with\n",
+      "decisions surrounding the direct embedding of LLMs within applications versus\n",
+      "employing them for code generation. Moreover, effective prompt design becomes a\n",
+      "critical concern, given the necessity of data extraction from natural language\n",
+      "outputs. To address these intricacies, this paper introduces AskIt, a\n",
+      "domain-specific language (DSL) specifically designed for LLMs. AskIt simplifies\n",
+      "LLM integration, offering type-guided output control, template-based function\n",
+      "definitions, and a unified interface that diminishes the distinction between\n",
+      "LLM-based code generation and application integration. Furthermore, through\n",
+      "Programming by Example (PBE), AskIt harnesses the power of few-shot learning at\n",
+      "the programming language level. Our evaluations underscore AskIt's potency.\n",
+      "Across 50 tasks, AskIt generated concise prompts for the given tasks, achieving\n",
+      "a 16.14% reduction in prompt length relative to benchmarks. Additionally, by\n",
+      "enabling the transition from direct LLM application usage to function\n",
+      "generation, AskIt achieved significant speedups, as observed in our GSM8K\n",
+      "benchmark experiments. Through these advancements, AskIt streamlines the\n",
+      "integration of LLMs in software development, offering a more efficient,\n",
+      "versatile approach for leveraging emergent abilities. The implementations of\n",
+      "AskIt in TypeScript and Python are available at\n",
+      "https://github.com/katsumiok/ts-askit and https://github.com/katsumiok/pyaskit,\n",
+      "respectively. | http://arxiv.org/abs/2308.15645v1 |\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mScientist\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Here are the papers related to LLM applications published in the last week:\n",
+      "\n",
+      "| Domain | Title | Authors | Summary | URL |\n",
+      "|-------|---------|---------|-----|-----|\n",
+      "| Data Preprocessing | [Large Language Models as Data Preprocessors](http://arxiv.org/abs/2308.16361v1) | Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada | This study expands on the applications of LLMs, exploring their potential in data preprocessing, a critical stage in data mining and analytics applications. | [Link](http://arxiv.org/abs/2308.16361v1) |\n",
+      "| Medicine | [Large language models in medicine: the potentials and pitfalls](http://arxiv.org/abs/2309.00087v1) | Jesutofunmi A. Omiye, Haiwen Gui, Shawheen J. Rezaei, James Zou, Roxana Daneshjou | This review and accompanying tutorial aim to give an overview of these topics to aid healthcare practitioners in understanding the rapidly changing landscape of LLMs as applied to medicine. | [Link](http://arxiv.org/abs/2309.00087v1) |\n",
+      "| 3D Understanding, Generation, and Instruction Following | [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](http://arxiv.org/abs/2309.00615v1) | Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, Pheng-Ann Heng | We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. | [Link](http://arxiv.org/abs/2309.00615v1) |\n",
+      "| Human Mobility Prediction | [Where Would I Go Next? Large Language Models as Human Mobility Predictors](http://arxiv.org/abs/2308.15197v1) | Xinglei Wang, Meng Fang, Zichao Zeng, Tao Cheng | This article delves into the potential of LLMs for human mobility prediction tasks. | [Link](http://arxiv.org/abs/2308.15197v1) |\n",
+      "| Robotics | [Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model](http://arxiv.org/abs/2308.15684v1) | Kazuki Hori, Kanata Suzuki, Tetsuya Ogata | In this paper, we propose the interactive robot action planning method that allows the LLM to analyze and gather missing information by asking questions to humans. | [Link](http://arxiv.org/abs/2308.15684v1) |\n",
+      "| Software Development | [AskIt: Unified Programming Interface for Programming with Large Language Models](http://arxiv.org/abs/2308.15645v1) | Katsumi Okuda, Saman Amarasinghe | This paper introduces AskIt, a domain-specific language (DSL) specifically designed for LLMs. | [Link](http://arxiv.org/abs/2308.15645v1) |\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mCritic\u001b[0m (to chat_manager):\n",
+      "\n",
+      "The scientist has done a good job categorizing the papers into different domains and providing a summary for each. The markdown table is correctly formatted and includes the source URL for each paper, which allows for verification of the information. The domains identified are diverse, indicating a broad range of applications for Large Language Models (LLMs). This review and categorization should provide a useful resource for anyone interested in the recent applications of LLMs.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "user_proxy.initiate_chat(\n",
+    "    manager,\n",
+    "    message=\"\"\"\n",
+    "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create Group Chat without Critic for Comparison"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mAdmin\u001b[0m (to chat_manager):\n",
+      "\n",
+      "\n",
+      "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mPlanner\u001b[0m (to chat_manager):\n",
+      "\n",
+      "Plan:\n",
+      "\n",
+      "1. Engineer: Write a script to scrape the arXiv website for papers related to LLM (Language Model) applications published in the last week. The script should extract the title, authors, abstract, and link to the paper.\n",
+      "\n",
+      "2. Scientist: Review the scraped data to identify the different domains in which LLM is applied. This could be based on keywords in the title or abstract, or the scientist's knowledge of the field.\n",
+      "\n",
+      "3. Engineer: Modify the script to categorize the papers based on the domains identified by the scientist. The script should output a markdown table with columns for the domain, title, authors, abstract, and link.\n",
+      "\n",
+      "4. Scientist: Review the markdown table to ensure the papers are correctly categorized and the information is accurate.\n",
+      "\n",
+      "5. Engineer: Make any necessary revisions to the script based on the scientist's feedback.\n",
+      "\n",
+      "6. Scientist: Give final approval of the markdown table.\n",
+      "\n",
+      "7. Engineer: Submit the final markdown table.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "groupchat_nocritic = autogen.GroupChat(agents=[user_proxy, engineer, scientist, planner, executor], messages=[], max_round=50)\n",
+    "for agent in groupchat.agents:\n",
+    "    agent.reset()\n",
+    "manager_nocritic = autogen.GroupChatManager(groupchat=groupchat_nocritic, llm_config=gpt4_config)\n",
+    "user_proxy.initiate_chat(\n",
+    "    manager_nocritic,\n",
+    "    message=\"\"\"\n",
+    "find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "flaml",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

notebook/autogen_agentchat_planning.ipynb

@@ -20,7 +20,7 @@
     "# Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent\n",
     "\n",
     "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
-    "Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "In this notebook, we demonstrate how to use multiple agents to work together and accomplish a task which requires finding info from the web and coding. `AssistantAgent` is an LLM-based agent that can write and debug Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We further create a planning agent for the assistant agent to consult. The planning agent is a variation of the LLM-based `AssistantAgent` with a different system message.\n",
     "\n",
@@ -45,7 +45,7 @@
    },
    "outputs": [],
    "source": [
-    "# %pip install flaml[autogen]~=2.0.0 docker"
+    "# %pip install flaml[autogen]~=2.0.2 docker"
    ]
   },
   {
@@ -75,7 +75,7 @@
     "config_list = autogen.config_list_from_json(\n",
     "    \"OAI_CONFIG_LIST\",\n",
     "    filter_dict={\n",
-    "        \"model\": [\"gpt-4\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "        \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
     "    },\n",
     ")"
    ]
@@ -221,7 +221,9 @@
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "To suggest a fix to an open good first issue of FLAML, we first need to fetch the list of open issues labeled as \"good first issue\" from the FLAML GitHub repository. We can do this using the GitHub API. Here is a Python script that fetches and prints the list of such issues.\n",
+      "To suggest a fix to an open good first issue of FLAML, we first need to fetch the list of open issues labeled as \"good first issue\" from the FLAML GitHub repository. We can do this using the GitHub API.\n",
+      "\n",
+      "Here is a Python script that uses the requests library to fetch the list of open issues labeled as \"good first issue\" from the FLAML GitHub repository.\n",
       "\n",
       "```python\n",
       "# filename: fetch_issues.py\n",
@@ -231,51 +233,49 @@
       "\n",
       "def fetch_issues():\n",
       "    url = \"https://api.github.com/repos/microsoft/FLAML/issues\"\n",
-      "    response = requests.get(url, params={\"state\": \"open\", \"labels\": \"good first issue\"})\n",
+      "    params = {\n",
+      "        \"state\": \"open\",\n",
+      "        \"labels\": \"good first issue\"\n",
+      "    }\n",
+      "    response = requests.get(url, params=params)\n",
       "    issues = response.json()\n",
-      "\n",
       "    for issue in issues:\n",
-      "        print(f\"Issue #{issue['number']}: {issue['title']}\")\n",
+      "        print(f\"Issue ID: {issue['id']}, Title: {issue['title']}, URL: {issue['html_url']}\")\n",
       "\n",
       "fetch_issues()\n",
       "```\n",
       "\n",
-      "Please save this script as `fetch_issues.py` and run it. It will print the list of open issues labeled as \"good first issue\" in the FLAML repository.\n",
+      "Please run this script to fetch the list of open issues. After that, we can select one issue and suggest a fix for it.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
       ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
       "\u001b[31m\n",
-      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "exitcode: 0 (execution succeeded)\n",
       "Code output: \n",
-      "Issue #1120: use_label_encoder warning with xgboost\n",
-      "Issue #1034: Use appropriate wait time for retry based on the error message. \n",
-      "Issue #1029: Issues with Adding Custom APIs in Auto Generation\n",
-      "Issue #981: Running flaml[tune] using \"-O\" flag for python interpreter (optimization - disables assertions) crashes\n",
-      "Issue #903: Conditional parameter flow2 crash\n",
-      "Issue #884: indentation space\n",
-      "Issue #882: Check if openml version is required\n",
-      "Issue #834: Adjust the indent\n",
-      "Issue #821: pip install flaml FAIL\n",
-      "Issue #807: Isolate the ensemble part and expose it to users\n",
-      "Issue #805: how to pass categorical features names or indices to learner\n",
-      "Issue #785: Flaml/LightGBM - Shouldn't I found better/faster or equal results from FLAML than direct LightGBM?\n",
-      "Issue #764: Add an announcement of the discord channel\n",
-      "Issue #748: Documentation about small budget\n",
-      "Issue #737: Make zero-shot automl more discoverable\n",
-      "Issue #509: New HCrystalBall release\n",
-      "Issue #429: samples about conversion to ONNX\n",
-      "Issue #413: support anomaly detection\n",
-      "Issue #304: CatBoost Fails with Keyword 'groups'\n",
+      "Issue ID: 1809297895, Title: Moving function execution out of UserProxyAgent to be an openai util, URL: https://github.com/microsoft/FLAML/issues/1135\n",
+      "Issue ID: 1799114476, Title: use_label_encoder warning with xgboost, URL: https://github.com/microsoft/FLAML/issues/1120\n",
+      "Issue ID: 1705274482, Title: Use appropriate wait time for retry based on the error message. , URL: https://github.com/microsoft/FLAML/issues/1034\n",
+      "Issue ID: 1702580697, Title: Issues with Adding Custom APIs in Auto Generation, URL: https://github.com/microsoft/FLAML/issues/1029\n",
+      "Issue ID: 1658981020, Title: Running flaml[tune] using \"-O\" flag for python interpreter (optimization - disables assertions) crashes, URL: https://github.com/microsoft/FLAML/issues/981\n",
+      "Issue ID: 1560969891, Title: Conditional parameter flow2 crash, URL: https://github.com/microsoft/FLAML/issues/903\n",
+      "Issue ID: 1538549388, Title: indentation space, URL: https://github.com/microsoft/FLAML/issues/884\n",
+      "Issue ID: 1531028010, Title: Check if openml version is required, URL: https://github.com/microsoft/FLAML/issues/882\n",
+      "Issue ID: 1470354491, Title: Adjust the indent, URL: https://github.com/microsoft/FLAML/issues/834\n",
+      "Issue ID: 1456950742, Title: pip install flaml FAIL, URL: https://github.com/microsoft/FLAML/issues/821\n",
+      "Issue ID: 1441047067, Title: Isolate the ensemble part and expose it to users, URL: https://github.com/microsoft/FLAML/issues/807\n",
+      "Issue ID: 1440171793, Title: how to pass categorical features names or indices to learner, URL: https://github.com/microsoft/FLAML/issues/805\n",
+      "Issue ID: 1429945686, Title: Flaml/LightGBM - Shouldn't I found better/faster or equal results from FLAML than direct LightGBM?, URL: https://github.com/microsoft/FLAML/issues/785\n",
+      "Issue ID: 1408240042, Title: Add an announcement of the discord channel, URL: https://github.com/microsoft/FLAML/issues/764\n",
+      "Issue ID: 1396515109, Title: Documentation about small budget, URL: https://github.com/microsoft/FLAML/issues/748\n",
+      "Issue ID: 1378268096, Title: Make zero-shot automl more discoverable, URL: https://github.com/microsoft/FLAML/issues/737\n",
+      "Issue ID: 1189515901, Title: New HCrystalBall release, URL: https://github.com/microsoft/FLAML/issues/509\n",
+      "Issue ID: 1114253143, Title: samples about conversion to ONNX, URL: https://github.com/microsoft/FLAML/issues/429\n",
+      "Issue ID: 1107488969, Title: support anomaly detection, URL: https://github.com/microsoft/FLAML/issues/413\n",
+      "Issue ID: 1061332179, Title: CatBoost Fails with Keyword 'groups', URL: https://github.com/microsoft/FLAML/issues/304\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
@@ -284,7 +284,7 @@
       "\u001b[32m***** Suggested function Call: ask_planner *****\u001b[0m\n",
       "Arguments: \n",
       "{\n",
-      "  \"message\": \"Here are the open good first issues of FLAML: \\n\\nIssue #1120: use_label_encoder warning with xgboost\\nIssue #1034: Use appropriate wait time for retry based on the error message.\\nIssue #1029: Issues with Adding Custom APIs in Auto Generation\\nIssue #981: Running flaml[tune] using \\\"-O\\\" flag for python interpreter (optimization - disables assertions) crashes\\nIssue #903: Conditional parameter flow2 crash\\nIssue #884: indentation space\\nIssue #882: Check if openml version is required\\nIssue #834: Adjust the indent\\nIssue #821: pip install flaml FAIL\\nIssue #807: Isolate the ensemble part and expose it to users\\nIssue #805: how to pass categorical features names or indices to learner\\nIssue #785: Flaml/LightGBM - Shouldn't I found better/faster or equal results from FLAML than direct LightGBM?\\nIssue #764: Add an announcement of the discord channel\\nIssue #748: Documentation about small budget\\nIssue #737: Make zero-shot automl more discoverable\\nIssue #509: New HCrystalBall release\\nIssue #429: samples about conversion to ONNX\\nIssue #413: support anomaly detection\\nIssue #304: CatBoost Fails with Keyword 'groups'\\n\\nPlease suggest a fix for one of these issues.\"\n",
+      "\"message\": \"We have fetched a list of open issues labeled as 'good first issue' from the FLAML GitHub repository. Now, we need to select one issue and suggest a fix for it. Could you please provide a plan for this?\"\n",
       "}\n",
       "\u001b[32m************************************************\u001b[0m\n",
       "\n",
@@ -295,124 +295,92 @@
       ">>>>>>>> EXECUTING FUNCTION ask_planner...\u001b[0m\n",
       "\u001b[33mplanner_user\u001b[0m (to planner):\n",
       "\n",
-      "Here are the open good first issues of FLAML: \n",
-      "\n",
-      "Issue #1120: use_label_encoder warning with xgboost\n",
-      "Issue #1034: Use appropriate wait time for retry based on the error message.\n",
-      "Issue #1029: Issues with Adding Custom APIs in Auto Generation\n",
-      "Issue #981: Running flaml[tune] using \"-O\" flag for python interpreter (optimization - disables assertions) crashes\n",
-      "Issue #903: Conditional parameter flow2 crash\n",
-      "Issue #884: indentation space\n",
-      "Issue #882: Check if openml version is required\n",
-      "Issue #834: Adjust the indent\n",
-      "Issue #821: pip install flaml FAIL\n",
-      "Issue #807: Isolate the ensemble part and expose it to users\n",
-      "Issue #805: how to pass categorical features names or indices to learner\n",
-      "Issue #785: Flaml/LightGBM - Shouldn't I found better/faster or equal results from FLAML than direct LightGBM?\n",
-      "Issue #764: Add an announcement of the discord channel\n",
-      "Issue #748: Documentation about small budget\n",
-      "Issue #737: Make zero-shot automl more discoverable\n",
-      "Issue #509: New HCrystalBall release\n",
-      "Issue #429: samples about conversion to ONNX\n",
-      "Issue #413: support anomaly detection\n",
-      "Issue #304: CatBoost Fails with Keyword 'groups'\n",
-      "\n",
-      "Please suggest a fix for one of these issues.\n",
+      "We have fetched a list of open issues labeled as 'good first issue' from the FLAML GitHub repository. Now, we need to select one issue and suggest a fix for it. Could you please provide a plan for this?\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33mplanner\u001b[0m (to planner_user):\n",
       "\n",
-      "Without certain details about the issues, it's a bit challenging to suggest explicit fixes. However, I'll give you a roadmap to tackle some of these:\n",
+      "Sure, here's a plan for selecting one issue from the list and suggesting a fix for it:\n",
       "\n",
-      "1. **Issue #1120: use_label_encoder warning with xgboost**\n",
+      "1. Import the fetched list of open issues labeled as 'good first issue' from the FLAML GitHub repository into your AI assistant. \n",
+      "2. Examine the list for common issue attributes like 'title', 'description', 'labels', 'issue number', 'created at', and 'updated at'. \n",
+      "3. To select a suitable issue for fixing, apply a selection criteria based on your preferences, such as prioritizing by the 'created at' attribute in descending order to pick the most recent issue, or filtering by a specific label in addition to 'good first issue'. Write code to filter and sort the issues accordingly.\n",
+      "4. Inspect the execution result. If the selection criteria are not applied correctly, modify the code to fix any errors.\n",
+      "5. Once the issue is selected, read the issue's title, description, and any linked resources or documents to understand the problem to be solved.\n",
+      "6. Break down the issue into smaller tasks that can be addressed by writing code, and create a step-by-step plan.\n",
       "\n",
-      "Reasoning: It appears like the `use_label_encoder` warning stems from deprecated usage in the XGBoost module that FLAML utilizes. \n",
+      "For instance, the following could be smaller tasks to address the selected issue:\n",
+      "  a. Understand the issue's background and requirements.\n",
+      "  b. Write clear and concise instructions to reproduce the issue.\n",
+      "  c. Analyze existing code or tests related to the issue.\n",
+      "  d. Devise a solution to fix the issue.\n",
+      "  e. Implement the solution in separate code pieces.\n",
+      "  f. Verify that the solution addresses the issue.\n",
+      "  g. Write unit tests to ensure the solution is robust and handles edge cases.\n",
       "\n",
-      "Plan: Update the call to the XGBoost module in FLAML to either stop using the label encoder or specify explicitly that the label encoder should be used, based on how FLAML is currently set up.\n",
+      "7. Inspect the execution result. If the issue is misunderstood or the tasks' breakdown is incorrect, revise the understanding of the issue and modify the tasks accordingly.\n",
+      "8. With the defined tasks and step-by-step plan, work on each task, and test the implemented code to ensure the issue is solved.\n",
+      "9. If any issues arise during the task execution, analyze the errors and adjust the plan or code accordingly.\n",
+      "10. Once the issue is fixed, prepare a pull request on GitHub, mentioning the issue number and giving a brief description of the solution in the merge request.\n",
       "\n",
-      "2. **Issue #903: Conditional parameter flow2 crash**\n",
-      "\n",
-      "Reasoning: This is likely a bug in the code where a condition is not properly handled.\n",
-      "\n",
-      "Plan: Identify the conditions causing the crash, and update the code to handle the condition correctly.\n",
-      "\n",
-      "3. **Issue #882: Check if openml version is required**\n",
-      "\n",
-      "Reasoning: The issue suggests that FLAML may be compatible with multiple versions of the `openml` library, but this isn't specified anywhere.\n",
-      "\n",
-      "Plan: Identify the minimum required version of the `openml` library that FLAML can operate with. If there's no minimum version, then there's no issue. If there is, update the documentation and the package requirements to show this.\n",
-      "\n",
-      "4. **Issue #821: pip install flaml FAIL**\n",
-      "\n",
-      "Reasoning: Users are unable to install FLAML using pip. This could be due to unmet dependancies, or issues with the package itself.\n",
-      "\n",
-      "Plan: Investigate the specific errors when trying to install via pip. If dependencies are missing or incompatible, update the package requirements. If there are issues with the package, fix those issues.\n",
-      "\n",
-      "5. **Issue #805: how to pass categorical features names or indices to learner**\n",
-      "\n",
-      "Reasoning: It seems users are facing difficulty in passing categorical features. This could be due to unclear or incorrect documentation, or missing functionality.\n",
-      "\n",
-      "Plan: Inspect the current documentation and functionality. Update the documentation to clarify how to pass these features, and if the functionality is missing, implement the functionality.\n",
-      "\n",
-      "Remember to thoroughly inspect the execution result for each issue to ensure your fix is effective. If the plan does not appear to fix the issue, adjust your plan based on the error message or unexpected behavior.\n",
+      "Remember that this is meant to be a general plan, and the specific tasks may vary depending on the selected issue. Adjust the plan as needed, based on the selected issue's requirements and your problem-solving approach.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "\u001b[32m***** Response from calling function \"ask_planner\" *****\u001b[0m\n",
-      "Without certain details about the issues, it's a bit challenging to suggest explicit fixes. However, I'll give you a roadmap to tackle some of these:\n",
+      "Sure, here's a plan for selecting one issue from the list and suggesting a fix for it:\n",
       "\n",
-      "1. **Issue #1120: use_label_encoder warning with xgboost**\n",
+      "1. Import the fetched list of open issues labeled as 'good first issue' from the FLAML GitHub repository into your AI assistant. \n",
+      "2. Examine the list for common issue attributes like 'title', 'description', 'labels', 'issue number', 'created at', and 'updated at'. \n",
+      "3. To select a suitable issue for fixing, apply a selection criteria based on your preferences, such as prioritizing by the 'created at' attribute in descending order to pick the most recent issue, or filtering by a specific label in addition to 'good first issue'. Write code to filter and sort the issues accordingly.\n",
+      "4. Inspect the execution result. If the selection criteria are not applied correctly, modify the code to fix any errors.\n",
+      "5. Once the issue is selected, read the issue's title, description, and any linked resources or documents to understand the problem to be solved.\n",
+      "6. Break down the issue into smaller tasks that can be addressed by writing code, and create a step-by-step plan.\n",
       "\n",
-      "Reasoning: It appears like the `use_label_encoder` warning stems from deprecated usage in the XGBoost module that FLAML utilizes. \n",
+      "For instance, the following could be smaller tasks to address the selected issue:\n",
+      "  a. Understand the issue's background and requirements.\n",
+      "  b. Write clear and concise instructions to reproduce the issue.\n",
+      "  c. Analyze existing code or tests related to the issue.\n",
+      "  d. Devise a solution to fix the issue.\n",
+      "  e. Implement the solution in separate code pieces.\n",
+      "  f. Verify that the solution addresses the issue.\n",
+      "  g. Write unit tests to ensure the solution is robust and handles edge cases.\n",
       "\n",
-      "Plan: Update the call to the XGBoost module in FLAML to either stop using the label encoder or specify explicitly that the label encoder should be used, based on how FLAML is currently set up.\n",
+      "7. Inspect the execution result. If the issue is misunderstood or the tasks' breakdown is incorrect, revise the understanding of the issue and modify the tasks accordingly.\n",
+      "8. With the defined tasks and step-by-step plan, work on each task, and test the implemented code to ensure the issue is solved.\n",
+      "9. If any issues arise during the task execution, analyze the errors and adjust the plan or code accordingly.\n",
+      "10. Once the issue is fixed, prepare a pull request on GitHub, mentioning the issue number and giving a brief description of the solution in the merge request.\n",
       "\n",
-      "2. **Issue #903: Conditional parameter flow2 crash**\n",
-      "\n",
-      "Reasoning: This is likely a bug in the code where a condition is not properly handled.\n",
-      "\n",
-      "Plan: Identify the conditions causing the crash, and update the code to handle the condition correctly.\n",
-      "\n",
-      "3. **Issue #882: Check if openml version is required**\n",
-      "\n",
-      "Reasoning: The issue suggests that FLAML may be compatible with multiple versions of the `openml` library, but this isn't specified anywhere.\n",
-      "\n",
-      "Plan: Identify the minimum required version of the `openml` library that FLAML can operate with. If there's no minimum version, then there's no issue. If there is, update the documentation and the package requirements to show this.\n",
-      "\n",
-      "4. **Issue #821: pip install flaml FAIL**\n",
-      "\n",
-      "Reasoning: Users are unable to install FLAML using pip. This could be due to unmet dependancies, or issues with the package itself.\n",
-      "\n",
-      "Plan: Investigate the specific errors when trying to install via pip. If dependencies are missing or incompatible, update the package requirements. If there are issues with the package, fix those issues.\n",
-      "\n",
-      "5. **Issue #805: how to pass categorical features names or indices to learner**\n",
-      "\n",
-      "Reasoning: It seems users are facing difficulty in passing categorical features. This could be due to unclear or incorrect documentation, or missing functionality.\n",
-      "\n",
-      "Plan: Inspect the current documentation and functionality. Update the documentation to clarify how to pass these features, and if the functionality is missing, implement the functionality.\n",
-      "\n",
-      "Remember to thoroughly inspect the execution result for each issue to ensure your fix is effective. If the plan does not appear to fix the issue, adjust your plan based on the error message or unexpected behavior.\n",
+      "Remember that this is meant to be a general plan, and the specific tasks may vary depending on the selected issue. Adjust the plan as needed, based on the selected issue's requirements and your problem-solving approach.\n",
       "\u001b[32m********************************************************\u001b[0m\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "Based on the planner's suggestions, let's tackle the first issue: \"Issue #1120: use_label_encoder warning with xgboost\". \n",
+      "Based on the plan, let's select the most recent issue from the list. The most recent issue is \"Moving function execution out of UserProxyAgent to be an openai util\" with the URL: https://github.com/microsoft/FLAML/issues/1135\n",
       "\n",
-      "The warning is likely due to a deprecated usage in the XGBoost module that FLAML utilizes. To fix this, we need to update the call to the XGBoost module in FLAML to either stop using the label encoder or specify explicitly that the label encoder should be used, based on how FLAML is currently set up.\n",
+      "To suggest a fix for this issue, we need to understand the problem first. Please visit the issue URL and read the issue's description and any comments to understand the problem.\n",
       "\n",
-      "Here is a general approach to fix this issue:\n",
+      "Once you understand the problem, here are the steps you can follow to suggest a fix:\n",
       "\n",
-      "1. Clone the FLAML repository to your local machine.\n",
-      "2. Navigate to the file where XGBoost is being called.\n",
-      "3. Look for the `use_label_encoder` parameter in the XGBoost function call.\n",
-      "4. If `use_label_encoder` is set to True, you can try setting it to False to see if the warning disappears.\n",
-      "5. If the warning persists, it might be necessary to refactor the code to avoid using label encoding in XGBoost, or to handle label encoding before data is passed to XGBoost.\n",
-      "6. After making changes, run the tests to make sure your changes do not break anything.\n",
-      "7. If everything is fine, commit your changes and create a pull request.\n",
+      "1. Understand the issue's background and requirements: Read the issue description and any linked resources or documents to understand what needs to be done.\n",
       "\n",
-      "Please note that without the actual codebase and the specific details of the issue, this is a general approach and might need to be adjusted based on the actual code and setup of FLAML. \n",
+      "2. Write clear and concise instructions to reproduce the issue: Based on the issue description, write down the steps to reproduce the issue. This will help you understand the problem better and will be useful when you test your fix.\n",
+      "\n",
+      "3. Analyze existing code or tests related to the issue: Look at the code in the FLAML repository that is related to the issue. Try to understand how it works and how it might be causing the issue.\n",
+      "\n",
+      "4. Devise a solution to fix the issue: Based on your understanding of the problem and the code, come up with a solution to fix the issue. This might involve writing new code or modifying existing code.\n",
+      "\n",
+      "5. Implement the solution in separate code pieces: Write the code for your solution. Make sure to break it down into small, manageable pieces. This will make it easier to understand and test.\n",
+      "\n",
+      "6. Verify that the solution addresses the issue: Test your code to make sure it solves the issue. You can do this by following the instructions you wrote to reproduce the issue.\n",
+      "\n",
+      "7. Write unit tests to ensure the solution is robust and handles edge cases: In addition to testing your solution manually, write unit tests to automatically test your code. This will help ensure that your solution works in all cases and that future changes to the code won't break your solution.\n",
+      "\n",
+      "8. Prepare a pull request on GitHub: Once you're confident that your solution works, prepare a pull request on GitHub. In the pull request description, mention the issue number and give a brief description of your solution.\n",
+      "\n",
+      "Please note that this is a general plan and the specific steps may vary depending on the issue. Adjust the plan as needed based on the issue's requirements and your problem-solving approach. \n",
       "\n",
       "TERMINATE\n",
       "\n",

notebook/autogen_agentchat_stream.ipynb

@@ -20,7 +20,7 @@
     "# Interactive LLM Agent Dealing with Data Stream\n",
     "\n",
     "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
-    "Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "In this notebook, we demonstrate how to use customized agents to continuously acquires news from the web and ask for investment suggestions.\n",
     "\n",
@@ -45,7 +45,7 @@
    },
    "outputs": [],
    "source": [
-    "# %pip install flaml[autogen]~=2.0.0"
+    "# %pip install flaml[autogen]~=2.1.0"
    ]
   },
   {
@@ -258,7 +258,7 @@
     "            )\n",
     "        return False, None\n",
     "\n",
-    "user_proxy.register_auto_reply(autogen.AssistantAgent, add_data_reply, 1, config={\"news_stream\": data})"
+    "user_proxy.register_reply(autogen.AssistantAgent, add_data_reply, 1, config={\"news_stream\": data})"
    ]
   },
   {

notebook/autogen_agentchat_two_users.ipynb

@@ -19,7 +19,7 @@
    "source": [
     "# Auto Generated Agent Chat: Collaborative Task Solving with Multiple Agents and Human Users\n",
     "\n",
-    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "In this notebook, we demonstrate an application involving multiple agents and human users to work together and accomplish a task. `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We create multiple `UserProxyAgent` instances which can represent different human users.\n",
     "\n",
@@ -44,7 +44,7 @@
    },
    "outputs": [],
    "source": [
-    "# %pip install flaml[autogen]~=2.0.0"
+    "# %pip install flaml[autogen]~=2.0.2"
    ]
   },
   {
@@ -138,9 +138,10 @@
     "    )\n",
     "\n",
     "    expert.initiate_chat(assistant_for_expert, message=message)\n",
-    "    expert.human_input_mode, expert.max_consecutive_auto_reply = \"NEVER\", 0\n",
+    "    expert.stop_reply_at_receive(assistant_for_expert)\n",
+    "    # expert.human_input_mode, expert.max_consecutive_auto_reply = \"NEVER\", 0\n",
     "    # final message sent from the expert\n",
-    "    expert.send(\"summarize the solution\", assistant_for_expert)\n",
+    "    expert.send(\"summarize the solution and explain the answer in an easy-to-understand way\", assistant_for_expert)\n",
     "    # return the last message the expert received\n",
     "    return expert.last_message()[\"content\"]\n"
    ]
@@ -163,6 +164,7 @@
    "source": [
     "assistant_for_student = autogen.AssistantAgent(\n",
     "    name=\"assistant_for_student\",\n",
+    "    system_message=\"You are a helpful assistant. Reply TERMINATE when the task is done.\",\n",
     "    llm_config={\n",
     "        \"request_timeout\": 600,\n",
     "        \"seed\": 42,\n",
@@ -218,7 +220,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "student (to assistant_for_student):\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
       "\n",
       "Find $a + b + c$, given that $x+y \\neq -1$ and \n",
       "\\begin{align}\n",
@@ -229,317 +231,242 @@
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
       "\n",
-      "This is a system of linear equations. We can solve it using Python's sympy library, which provides a function to solve systems of equations. Here is the Python code to solve it:\n",
+      "To find the values of $a$, $b$, and $c$, we need to solve the system of equations. However, the system of equations you provided seems to be incomplete or incorrect. The equations are not properly formatted, and it's unclear how $x$ and $y$ are related to $a$, $b$, and $c$. \n",
       "\n",
-      "```python\n",
-      "from sympy import symbols, Eq, solve\n",
-      "\n",
-      "# define the symbols\n",
-      "a, b, c, x, y = symbols('a b c x y')\n",
-      "\n",
-      "# define the equations\n",
-      "eq1 = Eq(a*x + b*y + c, x + 7)\n",
-      "eq2 = Eq(a + b*x + c*y, 2*x + 6*y)\n",
-      "eq3 = Eq(a*y + b + c*x, 4*x + y)\n",
-      "\n",
-      "# solve the system of equations\n",
-      "solution = solve((eq1, eq2, eq3), (a, b, c))\n",
-      "\n",
-      "# calculate a + b + c\n",
-      "result = solution[a] + solution[b] + solution[c]\n",
-      "\n",
-      "print(result)\n",
-      "```\n",
-      "\n",
-      "Please run this code to find the values of a, b, and c, and their sum.\n",
+      "Could you please provide the correct system of equations?\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> USING AUTO REPLY FOR THE USER...\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "student (to assistant_for_student):\n",
-      "\n",
-      "exitcode: 0 (execution succeeded)\n",
-      "Code output: \n",
-      "(2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
-      "\n",
-      "The output is the expression for $a + b + c$ in terms of $x$ and $y$. It's a bit complex, but it's the correct solution given the system of equations. \n",
-      "\n",
-      "If you have specific values for $x$ and $y$ (with the condition $x + y \\neq -1$), you can substitute them into this expression to get a numerical value for $a + b + c$.\n",
-      "\n",
-      "If you don't have specific values for $x$ and $y$, this is the most simplified form of $a + b + c$ that we can get from the given system of equations.\n",
-      "\n",
-      "TERMINATE\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> USING AUTO REPLY FOR THE USER...\n",
-      "student (to assistant_for_student):\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
       "\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
       "\n",
-      "TERMINATE\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "student (to assistant_for_student):\n",
-      "\n",
-      "I'm not sure if the answer is correct\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
-      "\n",
-      "***** Suggested function Call: ask_expert *****\n",
+      "\u001b[32m***** Suggested function Call: ask_expert *****\u001b[0m\n",
       "Arguments: \n",
       "{\n",
-      "  \"message\": \"The user has a system of equations and wants to find the sum of the variables a, b, and c. The system of equations is as follows:\\n\\nax + by + c = x + 7,\\na + bx + cy = 2x + 6y,\\nay + b + cx = 4x + y.\\n\\nI used sympy to solve the system of equations and got the following expression for a + b + c:\\n\\n(2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\\n\\nThe user is not sure if the answer is correct. Could you please verify this?\"\n",
+      "  \"message\": \"The user provided a system of equations and asked to find $a + b + c$, given that $x+y \\\\neq -1$ and \\n\\\\begin{align}\\n\\tax + by + c & = x + 7,\\\\\\n\\ta + bx + cy & = 2x + 6y,\\\\\\n\\tay + b + cx & = 4x + y.\\n\\\\end{align}. However, the system of equations seems to be incomplete or incorrectly formatted. How should I proceed?\"\n",
       "}\n",
-      "***********************************************\n",
+      "\u001b[32m***********************************************\u001b[0m\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[35m\n",
+      ">>>>>>>> EXECUTING FUNCTION ask_expert...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
       "\n",
-      ">>>>>>>> USING AUTO REPLY FOR THE USER...\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "The user has a system of equations and wants to find the sum of the variables a, b, and c. The system of equations is as follows:\n",
-      "\n",
-      "ax + by + c = x + 7,\n",
-      "a + bx + cy = 2x + 6y,\n",
-      "ay + b + cx = 4x + y.\n",
-      "\n",
-      "I used sympy to solve the system of equations and got the following expression for a + b + c:\n",
-      "\n",
-      "(2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
-      "\n",
-      "The user is not sure if the answer is correct. Could you please verify this?\n",
+      "The user provided a system of equations and asked to find $a + b + c$, given that $x+y \\neq -1$ and \n",
+      "\\begin{align}\n",
+      "\tax + by + c & = x + 7,\\\n",
+      "\ta + bx + cy & = 2x + 6y,\\\n",
+      "\tay + b + cx & = 4x + y.\n",
+      "\\end{align}. However, the system of equations seems to be incomplete or incorrectly formatted. How should I proceed?\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
       "\n",
-      "Sure, I can help you verify the solution. Let's first define the given system of equations and the expression for a + b + c using sympy, and then we'll check if the expression is correct by substituting it back into the original equations. Here's the code to do that:\n",
+      "To proceed, we can first try to solve the given system of equations and see if we can find a unique solution for $a$, $b$, and $c$. If we can't find a unique solution, we can then determine if there's any additional information needed or if the system is indeed incomplete or incorrectly formatted.\n",
       "\n",
-      "```python\n",
-      "from sympy import symbols, Eq, solve\n",
+      "Let's solve the given system of equations using Python's SymPy library.\n",
       "\n",
-      "x, y, a, b, c = symbols('x y a b c')\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
       "\n",
-      "eq1 = Eq(a*x + b*y + c, x + 7)\n",
-      "eq2 = Eq(a + b*x + c*y, 2*x + 6*y)\n",
-      "eq3 = Eq(a*y + b + c*x, 4*x + y)\n",
+      "make sure you simplify the answer\n",
       "\n",
-      "abc_expression = (2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
       "\n",
-      "solutions = solve((eq1, eq2, eq3), (a, b, c))\n",
+      "Sure, I will simplify the answer. Let's solve the given system of equations using Python's SymPy library and simplify the result.\n",
       "\n",
-      "abc_sum = sum(solutions.values())\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
       "\n",
-      "print(\"abc_expression:\", abc_expression)\n",
-      "print(\"abc_sum:\", abc_sum)\n",
-      "print(\"abc_expression equals abc_sum:\", abc_expression.equals(abc_sum))\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "First, let's install the SymPy library if you haven't already. Execute the following command to install it:\n",
+      "\n",
+      "```sh\n",
+      "pip install sympy\n",
       "```\n",
       "\n",
-      "Please execute the above code, and we'll see if the expression for a + b + c is correct.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> NO HUMAN INPUT RECEIVED. USING AUTO REPLY FOR THE USER...\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "exitcode: 0 (execution succeeded)\n",
-      "Code output: \n",
-      "abc_expression: (2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
-      "abc_sum: (2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
-      "abc_expression equals abc_sum: True\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
-      "\n",
-      "Great! The output shows that the expression for a + b + c is indeed correct, as the abc_expression equals the abc_sum. You can trust the solution provided by sympy.\n",
-      "\n",
-      "TERMINATE\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "can you simplify it?\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
-      "\n",
-      "Sure, we can simplify the expression using sympy's simplify function. Here's the code to do that:\n",
-      "\n",
-      "```python\n",
-      "from sympy import simplify\n",
-      "\n",
-      "simplified_expression = simplify(abc_expression)\n",
-      "\n",
-      "print(\"Simplified expression:\", simplified_expression)\n",
-      "```\n",
-      "\n",
-      "Please execute the above code to get the simplified expression.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> NO HUMAN INPUT RECEIVED. USING AUTO REPLY FOR THE USER...\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "exitcode: 1 (execution failed)\n",
-      "Code output: \n",
-      "Traceback (most recent call last):\n",
-      "  File \"/workspaces/FLAML/notebook/expert/tmp_code_fc1e7ed6cfddd1970c30e3cc25694093.py\", line 3, in <module>\n",
-      "    simplified_expression = simplify(abc_expression)\n",
-      "NameError: name 'abc_expression' is not defined\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
-      "\n",
-      "Apologies for the error. I forgot to include the definition of the abc_expression in the code. Here's the corrected code:\n",
-      "\n",
-      "```python\n",
-      "from sympy import symbols, simplify\n",
-      "\n",
-      "x, y = symbols('x y')\n",
-      "\n",
-      "abc_expression = (2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1) + (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1) + (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
-      "\n",
-      "simplified_expression = simplify(abc_expression)\n",
-      "\n",
-      "print(\"Simplified expression:\", simplified_expression)\n",
-      "```\n",
-      "\n",
-      "Please execute the above code to get the simplified expression.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> NO HUMAN INPUT RECEIVED. USING AUTO REPLY FOR THE USER...\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "exitcode: 0 (execution succeeded)\n",
-      "Code output: \n",
-      "Simplified expression: 7\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
-      "\n",
-      "Great! The simplified expression for a + b + c is 7. This means that regardless of the values of x and y, the sum of the variables a, b, and c from the given system of equations will always be 7.\n",
-      "\n",
-      "TERMINATE\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> NO HUMAN INPUT RECEIVED. USING AUTO REPLY FOR THE USER...\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
-      "\n",
-      "TERMINATE\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "expert (to assistant_for_expert):\n",
-      "\n",
-      "summarize the solution\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_expert (to expert):\n",
-      "\n",
-      "The user had a system of equations and an expression for the sum of the variables a, b, and c. The user wanted to verify if the expression was correct. The AI assistant used sympy to define the system of equations and the expression, and then checked if the expression was correct by comparing it with the sum of the solutions for a, b, and c. The result was True, indicating that the expression was correct.\n",
-      "\n",
-      "The user then asked to simplify the expression. The AI assistant used sympy's simplify function to simplify the expression, and the result was 7. This means that regardless of the values of x and y, the sum of the variables a, b, and c from the given system of equations will always be 7.\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "student (to assistant_for_student):\n",
-      "\n",
-      "***** Response from calling function \"ask_expert\" *****\n",
-      "The user had a system of equations and an expression for the sum of the variables a, b, and c. The user wanted to verify if the expression was correct. The AI assistant used sympy to define the system of equations and the expression, and then checked if the expression was correct by comparing it with the sum of the solutions for a, b, and c. The result was True, indicating that the expression was correct.\n",
-      "\n",
-      "The user then asked to simplify the expression. The AI assistant used sympy's simplify function to simplify the expression, and the result was 7. This means that regardless of the values of x and y, the sum of the variables a, b, and c from the given system of equations will always be 7.\n",
-      "*******************************************************\n",
-      "\n",
-      "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
-      "\n",
-      "I apologize for the confusion earlier. After consulting with an expert, it appears that the sum of a, b, and c from the given system of equations is actually a constant value, not dependent on x and y. \n",
-      "\n",
-      "The correct answer is 7. \n",
-      "\n",
-      "Here is the corrected Python code:\n",
+      "Now, let's solve the given system of equations using Python's SymPy library and simplify the result.\n",
       "\n",
       "```python\n",
       "from sympy import symbols, Eq, solve, simplify\n",
       "\n",
-      "# define the symbols\n",
       "a, b, c, x, y = symbols('a b c x y')\n",
       "\n",
-      "# define the equations\n",
-      "eq1 = Eq(a*x + b*y + c, x + 7)\n",
-      "eq2 = Eq(a + b*x + c*y, 2*x + 6*y)\n",
-      "eq3 = Eq(a*y + b + c*x, 4*x + y)\n",
+      "eq1 = Eq(a * x + b * y + c, x + 7)\n",
+      "eq2 = Eq(a + b * x + c * y, 2 * x + 6 * y)\n",
+      "eq3 = Eq(a * y + b + c * x, 4 * x + y)\n",
       "\n",
-      "# solve the system of equations\n",
-      "solution = solve((eq1, eq2, eq3), (a, b, c))\n",
+      "solutions = solve((eq1, eq2, eq3), (a, b, c))\n",
       "\n",
-      "# calculate a + b + c\n",
-      "result = solution[a] + solution[b] + solution[c]\n",
+      "simplified_solutions = {key: simplify(value) for key, value in solutions.items()}\n",
+      "a_val, b_val, c_val = simplified_solutions[a], simplified_solutions[b], simplified_solutions[c]\n",
+      "sum_abc = simplify(a_val + b_val + c_val)\n",
       "\n",
-      "# simplify the result\n",
-      "simplified_result = simplify(result)\n",
-      "\n",
-      "print(simplified_result)\n",
+      "print(f\"a: {a_val}\")\n",
+      "print(f\"b: {b_val}\")\n",
+      "print(f\"c: {c_val}\")\n",
+      "print(f\"a + b + c: {sum_abc}\")\n",
       "```\n",
       "\n",
-      "Please run this code to find the simplified sum of a, b, and c.\n",
+      "Please execute the above Python code to solve the system of equations and simplify the result.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> USING AUTO REPLY FOR THE USER...\n",
-      "student (to assistant_for_student):\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is sh)...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 1 (inferred language is python)...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
       "\n",
       "exitcode: 0 (execution succeeded)\n",
       "Code output: \n",
-      "7\n",
+      "Defaulting to user installation because normal site-packages is not writeable\n",
+      "Requirement already satisfied: sympy in /home/vscode/.local/lib/python3.9/site-packages (1.12)\n",
+      "Requirement already satisfied: mpmath>=0.19 in /home/vscode/.local/lib/python3.9/site-packages (from sympy) (1.3.0)\n",
+      "\n",
+      "a: (x**2 - 3*x*y + 2*x + y**2 - y)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "b: (2*x**2 - 3*x + y)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "c: (4*x**2 - 4*x*y - 6*x + 6*y**2 - 7*y + 7)/(x**2 - x*y - x + y**2 - y + 1)\n",
+      "a + b + c: 7\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
       "\n",
-      "Great! The output confirms that the sum of $a$, $b$, and $c$ from the given system of equations is indeed 7, regardless of the values of $x$ and $y$ (with the condition $x + y \\neq -1$). \n",
+      "The given system of equations has been solved and simplified. The result is:\n",
       "\n",
-      "I hope this answers your question satisfactorily. \n",
+      "$$\n",
+      "a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}.\n",
+      "$$\n",
+      "\n",
+      "The sum $a + b + c$ is equal to $7$. Since we were able to find a unique solution for $a$, $b$, and $c$, the system of equations is complete and correctly formatted.\n",
       "\n",
       "TERMINATE\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "\n",
-      ">>>>>>>> USING AUTO REPLY FOR THE USER...\n",
-      "student (to assistant_for_student):\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
       "\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
-      "assistant_for_student (to student):\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
       "\n",
       "TERMINATE\n",
       "\n",
-      "--------------------------------------------------------------------------------\n"
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n",
+      "\u001b[33mexpert\u001b[0m (to assistant_for_expert):\n",
+      "\n",
+      "summarize the solution and explain the answer in an easy-to-understand way\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_expert\u001b[0m (to expert):\n",
+      "\n",
+      "Sure, let me summarize the solution and explain the answer in an easy-to-understand way.\n",
+      "\n",
+      "We were given a system of three equations with three unknowns, $a$, $b$, and $c$. The equations were:\n",
+      "\n",
+      "1. $ax + by + c = x + 7$\n",
+      "2. $a + bx + cy = 2x + 6y$\n",
+      "3. $ay + b + cx = 4x + y$\n",
+      "\n",
+      "We were asked to find the sum $a + b + c$ given that $x + y \\neq -1$.\n",
+      "\n",
+      "We used Python's SymPy library to solve the system of equations and found the following solutions for $a$, $b$, and $c$:\n",
+      "\n",
+      "$$\n",
+      "a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}.\n",
+      "$$\n",
+      "\n",
+      "These solutions are expressed in terms of $x$ and $y$. However, we were able to find the sum $a + b + c$ without knowing the values of $x$ and $y$. The sum $a + b + c$ simplifies to $7$.\n",
+      "\n",
+      "In conclusion, the sum $a + b + c$ for the given system of equations is equal to $7$.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
+      "\n",
+      "\u001b[32m***** Response from calling function \"ask_expert\" *****\u001b[0m\n",
+      "Sure, let me summarize the solution and explain the answer in an easy-to-understand way.\n",
+      "\n",
+      "We were given a system of three equations with three unknowns, $a$, $b$, and $c$. The equations were:\n",
+      "\n",
+      "1. $ax + by + c = x + 7$\n",
+      "2. $a + bx + cy = 2x + 6y$\n",
+      "3. $ay + b + cx = 4x + y$\n",
+      "\n",
+      "We were asked to find the sum $a + b + c$ given that $x + y \\neq -1$.\n",
+      "\n",
+      "We used Python's SymPy library to solve the system of equations and found the following solutions for $a$, $b$, and $c$:\n",
+      "\n",
+      "$$\n",
+      "a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}, \\\\\n",
+      "c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}.\n",
+      "$$\n",
+      "\n",
+      "These solutions are expressed in terms of $x$ and $y$. However, we were able to find the sum $a + b + c$ without knowing the values of $x$ and $y$. The sum $a + b + c$ simplifies to $7$.\n",
+      "\n",
+      "In conclusion, the sum $a + b + c$ for the given system of equations is equal to $7$.\n",
+      "\u001b[32m*******************************************************\u001b[0m\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
+      "\n",
+      "The solution to the system of equations you provided is:\n",
+      "\n",
+      "1. $a = \\frac{x^2 - 3xy + 2x + y^2 - y}{x^2 - xy - x + y^2 - y + 1}$\n",
+      "2. $b = \\frac{2x^2 - 3x + y}{x^2 - xy - x + y^2 - y + 1}$\n",
+      "3. $c = \\frac{4x^2 - 4xy - 6x + 6y^2 - 7y + 7}{x^2 - xy - x + y^2 - y + 1}$\n",
+      "\n",
+      "These solutions are expressed in terms of $x$ and $y$. However, we can find the sum $a + b + c$ without knowing the values of $x$ and $y$. The sum $a + b + c$ simplifies to $7$.\n",
+      "\n",
+      "So, the sum $a + b + c$ for the given system of equations is equal to $7$.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[33mstudent\u001b[0m (to assistant_for_student):\n",
+      "\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[33massistant_for_student\u001b[0m (to student):\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n",
+      "\u001b[31m\n",
+      ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n"
      ]
     }
    ],

notebook/autogen_agentchat_web_info.ipynb

@@ -20,7 +20,7 @@
     "# Auto Generated Agent Chat: Solving Tasks Requiring Web Info\n",
     "\n",
     "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
-    "Please find documentation about this feature [here](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen#agents).\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
     "\n",
     "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to perform tasks which require acquiring info from the web:\n",
     "* discuss a paper based on its URL.\n",
@@ -49,7 +49,7 @@
    },
    "outputs": [],
    "source": [
-    "# %pip install flaml[autogen]~=2.0.0 docker"
+    "# %pip install flaml[autogen]~=2.0.2 docker"
    ]
   },
   {
@@ -183,30 +183,34 @@
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "To determine who should read this paper, let's first extract the abstract and analyze its content. We can use Python with the `requests` and `BeautifulSoup` libraries to do this. Please execute the following code:\n",
+      "To determine who should read the paper, we need to first understand the content and context of the paper. We can do this by fetching the abstract of the paper from the provided URL and analyzing it. \n",
+      "\n",
+      "Here is a Python script that uses the BeautifulSoup library to scrape the abstract of the paper from the webpage. \n",
       "\n",
       "```python\n",
+      "# Python script to scrape the abstract of the paper\n",
+      "\n",
       "import requests\n",
       "from bs4 import BeautifulSoup\n",
       "\n",
+      "def get_abstract(url):\n",
+      "    response = requests.get(url)\n",
+      "    soup = BeautifulSoup(response.text, 'html.parser')\n",
+      "    abstract = soup.find('blockquote', attrs={'class': 'abstract mathjax'}).text.strip()\n",
+      "    return abstract\n",
+      "\n",
       "url = \"https://arxiv.org/abs/2306.01337\"\n",
-      "response = requests.get(url)\n",
-      "soup = BeautifulSoup(response.text, \"html.parser\")\n",
-      "abstract = soup.find(\"blockquote\", {\"class\": \"abstract\"}).text.strip()\n",
+      "abstract = get_abstract(url)\n",
       "print(abstract)\n",
       "```\n",
       "\n",
-      "After executing the code, I will analyze the abstract and suggest who should read the paper.\n",
+      "Please run this script and provide the output. Based on the abstract, I can suggest who might be interested in reading this paper.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
-      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "exitcode: 0 (execution succeeded)\n",
@@ -227,16 +231,23 @@
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "Based on the abstract, the paper is about using Large Language Models (LLMs), specifically GPT-4, to solve complex mathematical problems. The authors propose a conversational problem-solving framework called MathChat and evaluate it on difficult high school competition problems from the MATH dataset.\n",
+      "Based on the abstract, the paper is about using Large Language Models (LLMs), specifically GPT-4, to solve complex mathematical problems. The paper introduces a new conversational problem-solving framework called MathChat and evaluates its performance on difficult high school competition problems from the MATH dataset.\n",
       "\n",
-      "The target audience for this paper includes:\n",
+      "Given this, the paper would be of interest to the following groups:\n",
       "\n",
-      "1. Researchers and practitioners in the field of artificial intelligence, specifically those working with large language models like GPT-4.\n",
-      "2. Mathematicians and educators interested in the application of AI for solving complex mathematical problems.\n",
-      "3. Developers working on AI-based conversational systems or chatbots for problem-solving in various domains.\n",
-      "4. Students and researchers in natural language processing, machine learning, and AI who are interested in the intersection of these fields with mathematics.\n",
+      "1. **Researchers in Artificial Intelligence and Natural Language Processing**: The paper discusses the use of a large language model (GPT-4) for problem-solving, which is a key research area in AI and NLP.\n",
       "\n",
-      "If you belong to any of these categories or have a general interest in AI and its applications, you might find this paper valuable. TERMINATE\n",
+      "2. **Mathematicians and Math Educators**: The paper focuses on solving complex mathematical problems, so those with a background in mathematics might find the techniques and results interesting.\n",
+      "\n",
+      "3. **Data Scientists and Machine Learning Engineers**: These professionals often use models like GPT-4 in their work and might be interested in new applications and techniques.\n",
+      "\n",
+      "4. **Students studying AI, NLP, or Mathematics**: The paper could provide valuable insights for these students into how AI can be used in problem-solving.\n",
+      "\n",
+      "5. **Developers working on AI-based chatbots or conversational agents**: The paper introduces a new conversational problem-solving framework, which could be of interest to these developers.\n",
+      "\n",
+      "Please note that while the paper is likely to be of interest to these groups, the specific relevance will depend on the individual's specific interests and research needs.\n",
+      "\n",
+      "TERMINATE\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
@@ -249,7 +260,7 @@
     "user_proxy.initiate_chat(\n",
     "    assistant,\n",
     "    message=\"\"\"\n",
-    "Who should read this paper: https://arxiv.org/abs/2306.01337\n",
+    "Who should read this paper: https://arxiv.org/abs/2308.08155\n",
     "\"\"\",\n",
     ")"
    ]
@@ -278,127 +289,119 @@
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "To get the YTD gain of the 10 largest technology companies, we can use a Python script to fetch the data from a financial API like Yahoo Finance. Please execute the following Python code:\n",
+      "To get the YTD gain of the 10 largest technology companies, we need to do the following:\n",
+      "\n",
+      "1. Identify the 10 largest technology companies. We can use the list of the largest technology companies by market capitalization. This list can change frequently, so we need to get the latest data. We can use web scraping to get this data from a reliable source.\n",
+      "\n",
+      "2. Get the YTD gain for each of these companies. We can use a financial data API to get this data. Yahoo Finance is a popular source for this kind of data.\n",
+      "\n",
+      "Here is a Python script that uses the BeautifulSoup library for web scraping and the yfinance library to get data from Yahoo Finance. This script will print the 10 largest technology companies and their YTD gains.\n",
       "\n",
       "```python\n",
-      "import datetime\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
       "import requests\n",
-      "import pandas as pd\n",
       "from bs4 import BeautifulSoup\n",
+      "import yfinance as yf\n",
+      "import pandas as pd\n",
       "\n",
-      "def get_top_tech_companies():\n",
-      "    url = \"https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/technology/\"\n",
-      "    response = requests.get(url)\n",
-      "    soup = BeautifulSoup(response.text, \"html.parser\")\n",
-      "    table = soup.find(\"table\", {\"class\": \"tv-data-table\"})\n",
-      "    rows = table.find_all(\"tr\")[1:11]\n",
-      "    symbols = [row.find(\"a\").text for row in rows]\n",
-      "    return symbols\n",
+      "# Get the list of largest tech companies by market cap\n",
+      "url = \"https://companiesmarketcap.com/technology/largest-technology-companies-by-market-cap/\"\n",
+      "response = requests.get(url)\n",
+      "soup = BeautifulSoup(response.text, 'html.parser')\n",
       "\n",
-      "def get_ytd_gain(symbol):\n",
-      "    today = datetime.date.today()\n",
-      "    start_date = datetime.date(today.year, 1, 1)\n",
-      "    url = f\"https://query1.finance.yahoo.com/v7/finance/download/{symbol}?period1={int(start_date.timestamp())}&period2={int(today.timestamp())}&interval=1d&events=history&includeAdjustedClose=true\"\n",
-      "    response = requests.get(url)\n",
-      "    data = pd.read_csv(pd.StringIO(response.text))\n",
-      "    ytd_gain = (data[\"Close\"].iloc[-1] - data[\"Close\"].iloc[0]) / data[\"Close\"].iloc[0] * 100\n",
-      "    return ytd_gain\n",
+      "# Find the table with the data\n",
+      "table = soup.find('table')\n",
       "\n",
-      "top_tech_companies = get_top_tech_companies()\n",
-      "ytd_gains = {symbol: get_ytd_gain(symbol) for symbol in top_tech_companies}\n",
+      "# Get the ticker symbols of the 10 largest companies\n",
+      "tickers = []\n",
+      "for row in table.find_all('tr')[1:11]:  # Skip the header row\n",
+      "    ticker = row.find_all('td')[1].text\n",
+      "    tickers.append(ticker)\n",
       "\n",
-      "for symbol, gain in ytd_gains.items():\n",
-      "    print(f\"{symbol}: {gain:.2f}%\")\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
+      "    ticker_data = yf.Ticker(ticker)\n",
+      "    hist = ticker_data.history(period=\"ytd\")\n",
+      "    start_price = hist.iloc[0]['Close']\n",
+      "    end_price = hist.iloc[-1]['Close']\n",
+      "    ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "    ytd_gains[ticker] = ytd_gain\n",
+      "\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
       "```\n",
       "\n",
-      "This code will fetch the 10 largest technology companies from TradingView and then calculate their YTD gains using Yahoo Finance data.\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the BeautifulSoup, requests, yfinance, and pandas libraries. If you don't have these libraries installed, you can install them with the command `pip install beautifulsoup4 requests yfinance pandas`.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
       ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "exitcode: 1 (execution failed)\n",
       "Code output: \n",
       "Traceback (most recent call last):\n",
-      "  File \"/workspaces/FLAML/notebook/web/tmp_code_2e1e4c6201cb82d42fbeb0754e008d0f.py\", line 24, in <module>\n",
-      "    top_tech_companies = get_top_tech_companies()\n",
-      "  File \"/workspaces/FLAML/notebook/web/tmp_code_2e1e4c6201cb82d42fbeb0754e008d0f.py\", line 11, in get_top_tech_companies\n",
-      "    rows = table.find_all(\"tr\")[1:11]\n",
+      "  File \"ytd_gain.py\", line 18, in <module>\n",
+      "    for row in table.find_all('tr')[1:11]:  # Skip the header row\n",
       "AttributeError: 'NoneType' object has no attribute 'find_all'\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "I apologize for the error. It seems that the website structure has changed, and the script is unable to find the table containing the top technology companies. Let's try an alternative approach using the Yahoo Finance library. Please install the library first by executing:\n",
+      "I apologize for the error. It seems that the script failed to find the table in the webpage. This could be due to changes in the webpage structure or because the page uses JavaScript to load the table, which BeautifulSoup can't handle.\n",
       "\n",
-      "```sh\n",
-      "pip install yfinance\n",
-      "```\n",
+      "Let's try a different approach. We'll use the yfinance library to get the data for the 10 largest technology companies. These are currently: Apple (AAPL), Microsoft (MSFT), Alphabet (GOOGL), Amazon (AMZN), Facebook (FB), Tesla (TSLA), Nvidia (NVDA), Paypal (PYPL), Adobe (ADBE), and Netflix (NFLX).\n",
       "\n",
-      "After installing the library, please execute the following Python code:\n",
+      "Here's the updated Python script:\n",
       "\n",
       "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
       "import yfinance as yf\n",
-      "import datetime\n",
       "\n",
-      "top_tech_companies = [\n",
-      "    \"AAPL\", \"MSFT\", \"GOOGL\", \"GOOG\", \"AMZN\", \"FB\", \"TSLA\", \"NVDA\", \"ASML\", \"ADBE\"\n",
-      "]\n",
+      "# List of the 10 largest tech companies by market cap\n",
+      "tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'FB', 'TSLA', 'NVDA', 'PYPL', 'ADBE', 'NFLX']\n",
       "\n",
-      "def get_ytd_gain(symbol):\n",
-      "    today = datetime.date.today()\n",
-      "    start_date = datetime.date(today.year, 1, 1)\n",
-      "    stock = yf.Ticker(symbol)\n",
-      "    data = stock.history(start=start_date, end=today)\n",
-      "    ytd_gain = (data[\"Close\"].iloc[-1] - data[\"Close\"].iloc[0]) / data[\"Close\"].iloc[0] * 100\n",
-      "    return ytd_gain\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
+      "    ticker_data = yf.Ticker(ticker)\n",
+      "    hist = ticker_data.history(period=\"ytd\")\n",
+      "    start_price = hist.iloc[0]['Close']\n",
+      "    end_price = hist.iloc[-1]['Close']\n",
+      "    ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "    ytd_gains[ticker] = ytd_gain\n",
       "\n",
-      "ytd_gains = {symbol: get_ytd_gain(symbol) for symbol in top_tech_companies}\n",
-      "\n",
-      "for symbol, gain in ytd_gains.items():\n",
-      "    print(f\"{symbol}: {gain:.2f}%\")\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
       "```\n",
       "\n",
-      "This code uses a predefined list of the 10 largest technology companies and calculates their YTD gains using the Yahoo Finance library.\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the yfinance library. If you don't have this library installed, you can install it with the command `pip install yfinance`.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
       ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "exitcode: 1 (execution failed)\n",
       "Code output: \n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: yfinance in /home/vscode/.local/lib/python3.9/site-packages (0.2.26)\n",
-      "Requirement already satisfied: numpy>=1.16.5 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (1.25.1)\n",
-      "Requirement already satisfied: pandas>=1.3.0 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (1.3.3)\n",
-      "Requirement already satisfied: beautifulsoup4>=4.11.1 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (4.12.2)\n",
-      "Requirement already satisfied: html5lib>=1.1 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (1.1)\n",
-      "Requirement already satisfied: pytz>=2022.5 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (2023.3)\n",
-      "Requirement already satisfied: appdirs>=1.4.4 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (1.4.4)\n",
-      "Requirement already satisfied: lxml>=4.9.1 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (4.9.3)\n",
-      "Requirement already satisfied: requests>=2.31 in /usr/local/lib/python3.9/site-packages (from yfinance) (2.31.0)\n",
-      "Requirement already satisfied: frozendict>=2.3.4 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (2.3.8)\n",
-      "Requirement already satisfied: multitasking>=0.0.7 in /home/vscode/.local/lib/python3.9/site-packages (from yfinance) (0.0.11)\n",
-      "Requirement already satisfied: soupsieve>1.2 in /home/vscode/.local/lib/python3.9/site-packages (from beautifulsoup4>=4.11.1->yfinance) (2.4.1)\n",
-      "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.9/site-packages (from html5lib>=1.1->yfinance) (1.16.0)\n",
-      "Requirement already satisfied: webencodings in /home/vscode/.local/lib/python3.9/site-packages (from html5lib>=1.1->yfinance) (0.5.1)\n",
-      "Requirement already satisfied: python-dateutil>=2.7.3 in /home/vscode/.local/lib/python3.9/site-packages (from pandas>=1.3.0->yfinance) (2.8.2)\n",
-      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.31->yfinance) (3.4)\n",
-      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests>=2.31->yfinance) (2023.5.7)\n",
-      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests>=2.31->yfinance) (2.0.3)\n",
-      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/site-packages (from requests>=2.31->yfinance) (3.2.0)\n",
-      "\n",
-      "FB: No timezone found, symbol may be delisted\n",
+      "FB: No data found, symbol may be delisted\n",
       "Traceback (most recent call last):\n",
-      "  File \"/workspaces/FLAML/notebook/web/tmp_code_52c69e134e808cf8ba196ac60faede00.py\", line 16, in <module>\n",
-      "    ytd_gains = {symbol: get_ytd_gain(symbol) for symbol in top_tech_companies}\n",
-      "  File \"/workspaces/FLAML/notebook/web/tmp_code_52c69e134e808cf8ba196ac60faede00.py\", line 16, in <dictcomp>\n",
-      "    ytd_gains = {symbol: get_ytd_gain(symbol) for symbol in top_tech_companies}\n",
-      "  File \"/workspaces/FLAML/notebook/web/tmp_code_52c69e134e808cf8ba196ac60faede00.py\", line 13, in get_ytd_gain\n",
-      "    ytd_gain = (data[\"Close\"].iloc[-1] - data[\"Close\"].iloc[0]) / data[\"Close\"].iloc[0] * 100\n",
+      "  File \"ytd_gain.py\", line 13, in <module>\n",
+      "    start_price = hist.iloc[0]['Close']\n",
       "  File \"/home/vscode/.local/lib/python3.9/site-packages/pandas/core/indexing.py\", line 931, in __getitem__\n",
       "    return self._getitem_axis(maybe_callable, axis=axis)\n",
       "  File \"/home/vscode/.local/lib/python3.9/site-packages/pandas/core/indexing.py\", line 1566, in _getitem_axis\n",
@@ -411,155 +414,165 @@
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "I apologize for the error. It seems that there is an issue with the data for the \"FB\" symbol. Let's add a try-except block to handle this issue. Please execute the following Python code:\n",
+      "I apologize for the error. It seems that the script failed to get the data for Facebook (FB). This could be due to a temporary issue with the Yahoo Finance API.\n",
+      "\n",
+      "Let's modify the script to handle this situation. We'll add a try-except block to skip any companies for which we can't get data.\n",
+      "\n",
+      "Here's the updated Python script:\n",
       "\n",
       "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
       "import yfinance as yf\n",
-      "import datetime\n",
       "\n",
-      "top_tech_companies = [\n",
-      "    \"AAPL\", \"MSFT\", \"GOOGL\", \"GOOG\", \"AMZN\", \"FB\", \"TSLA\", \"NVDA\", \"ASML\", \"ADBE\"\n",
-      "]\n",
+      "# List of the 10 largest tech companies by market cap\n",
+      "tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'FB', 'TSLA', 'NVDA', 'PYPL', 'ADBE', 'NFLX']\n",
       "\n",
-      "def get_ytd_gain(symbol):\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
       "    try:\n",
-      "        today = datetime.date.today()\n",
-      "        start_date = datetime.date(today.year, 1, 1)\n",
-      "        stock = yf.Ticker(symbol)\n",
-      "        data = stock.history(start=start_date, end=today)\n",
-      "        ytd_gain = (data[\"Close\"].iloc[-1] - data[\"Close\"].iloc[0]) / data[\"Close\"].iloc[0] * 100\n",
-      "        return ytd_gain\n",
-      "    except Exception as e:\n",
-      "        print(f\"Error fetching data for {symbol}: {e}\")\n",
-      "        return None\n",
+      "        ticker_data = yf.Ticker(ticker)\n",
+      "        hist = ticker_data.history(period=\"ytd\")\n",
+      "        start_price = hist.iloc[0]['Close']\n",
+      "        end_price = hist.iloc[-1]['Close']\n",
+      "        ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "        ytd_gains[ticker] = ytd_gain\n",
+      "    except IndexError:\n",
+      "        print(f\"Could not get data for {ticker}\")\n",
       "\n",
-      "ytd_gains = {symbol: get_ytd_gain(symbol) for symbol in top_tech_companies}\n",
-      "\n",
-      "for symbol, gain in ytd_gains.items():\n",
-      "    if gain is not None:\n",
-      "        print(f\"{symbol}: {gain:.2f}%\")\n",
-      "    else:\n",
-      "        print(f\"{symbol}: Data not available\")\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
       "```\n",
       "\n",
-      "This code adds error handling to the `get_ytd_gain` function, so if there is an issue with a specific symbol, it will print an error message and continue with the remaining symbols.\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the yfinance library. If you don't have this library installed, you can install it with the command `pip install yfinance`.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
       ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "exitcode: 0 (execution succeeded)\n",
       "Code output: \n",
-      "Error fetching data for FB: single positional indexer is out-of-bounds\n",
-      "AAPL: 57.03%\n",
-      "MSFT: 41.90%\n",
-      "GOOGL: 48.77%\n",
-      "GOOG: 48.28%\n",
-      "AMZN: 54.06%\n",
-      "FB: Data not available\n",
-      "TSLA: 146.48%\n",
-      "NVDA: 226.67%\n",
-      "ASML: 31.38%\n",
-      "ADBE: 56.97%\n",
+      "Could not get data for FB\n",
+      "AAPL: 46.41%\n",
+      "MSFT: 37.63%\n",
+      "GOOGL: 51.13%\n",
+      "AMZN: 56.22%\n",
+      "TSLA: 131.46%\n",
+      "NVDA: 237.68%\n",
+      "PYPL: -16.49%\n",
+      "ADBE: 59.79%\n",
+      "NFLX: 43.41%\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "I'm glad the code executed successfully. Here are the YTD gains for the 10 largest technology companies, except for Facebook (FB), for which data is not available:\n",
+      "The script has successfully fetched the Year-to-Date (YTD) gain for the largest technology companies, except for Facebook (FB). The YTD gains are as follows:\n",
       "\n",
-      "- AAPL: 57.03%\n",
-      "- MSFT: 41.90%\n",
-      "- GOOGL: 48.77%\n",
-      "- GOOG: 48.28%\n",
-      "- AMZN: 54.06%\n",
-      "- TSLA: 146.48%\n",
-      "- NVDA: 226.67%\n",
-      "- ASML: 31.38%\n",
-      "- ADBE: 56.97%\n",
+      "- Apple (AAPL): 46.41%\n",
+      "- Microsoft (MSFT): 37.63%\n",
+      "- Alphabet (GOOGL): 51.13%\n",
+      "- Amazon (AMZN): 56.22%\n",
+      "- Tesla (TSLA): 131.46%\n",
+      "- Nvidia (NVDA): 237.68%\n",
+      "- Paypal (PYPL): -16.49%\n",
+      "- Adobe (ADBE): 59.79%\n",
+      "- Netflix (NFLX): 43.41%\n",
       "\n",
-      "Please note that these values are subject to change as the market fluctuates. If you need updated values, you can re-run the code.\n",
+      "Unfortunately, the script could not fetch the data for Facebook (FB). This could be due to a temporary issue with the Yahoo Finance API or the ticker symbol might have changed.\n",
+      "\n",
+      "Please note that these values are subject to change as the stock market fluctuates. \n",
       "\n",
       "TERMINATE\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
-      "FB->META\n",
+      "FB -> META\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "I apologize for the confusion. Facebook has changed its stock symbol from \"FB\" to \"META\" due to the company's rebranding. Let's update the code with the new symbol and execute it again:\n",
+      "Thank you for the update. Facebook Inc. has indeed rebranded to Meta Platforms Inc. and its ticker symbol has changed from FB to META on the NASDAQ stock exchange.\n",
+      "\n",
+      "Let's update the script to reflect this change:\n",
       "\n",
       "```python\n",
+      "# filename: ytd_gain.py\n",
+      "\n",
       "import yfinance as yf\n",
-      "import datetime\n",
       "\n",
-      "top_tech_companies = [\n",
-      "    \"AAPL\", \"MSFT\", \"GOOGL\", \"GOOG\", \"AMZN\", \"META\", \"TSLA\", \"NVDA\", \"ASML\", \"ADBE\"\n",
-      "]\n",
+      "# List of the 10 largest tech companies by market cap\n",
+      "# Updated Facebook (FB) to Meta Platforms (META)\n",
+      "tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'META', 'TSLA', 'NVDA', 'PYPL', 'ADBE', 'NFLX']\n",
       "\n",
-      "def get_ytd_gain(symbol):\n",
+      "# Get the YTD gain for each company\n",
+      "ytd_gains = {}\n",
+      "for ticker in tickers:\n",
       "    try:\n",
-      "        today = datetime.date.today()\n",
-      "        start_date = datetime.date(today.year, 1, 1)\n",
-      "        stock = yf.Ticker(symbol)\n",
-      "        data = stock.history(start=start_date, end=today)\n",
-      "        ytd_gain = (data[\"Close\"].iloc[-1] - data[\"Close\"].iloc[0]) / data[\"Close\"].iloc[0] * 100\n",
-      "        return ytd_gain\n",
-      "    except Exception as e:\n",
-      "        print(f\"Error fetching data for {symbol}: {e}\")\n",
-      "        return None\n",
+      "        ticker_data = yf.Ticker(ticker)\n",
+      "        hist = ticker_data.history(period=\"ytd\")\n",
+      "        start_price = hist.iloc[0]['Close']\n",
+      "        end_price = hist.iloc[-1]['Close']\n",
+      "        ytd_gain = (end_price - start_price) / start_price * 100\n",
+      "        ytd_gains[ticker] = ytd_gain\n",
+      "    except IndexError:\n",
+      "        print(f\"Could not get data for {ticker}\")\n",
       "\n",
-      "ytd_gains = {symbol: get_ytd_gain(symbol) for symbol in top_tech_companies}\n",
-      "\n",
-      "for symbol, gain in ytd_gains.items():\n",
-      "    if gain is not None:\n",
-      "        print(f\"{symbol}: {gain:.2f}%\")\n",
-      "    else:\n",
-      "        print(f\"{symbol}: Data not available\")\n",
+      "# Print the results\n",
+      "for ticker, ytd_gain in ytd_gains.items():\n",
+      "    print(f\"{ticker}: {ytd_gain:.2f}%\")\n",
       "```\n",
       "\n",
-      "This code replaces \"FB\" with \"META\" in the list of top technology companies and calculates the YTD gains for the updated list.\n",
+      "To run this script, save it to a file named `ytd_gain.py`, then run it with the command `python ytd_gain.py`.\n",
+      "\n",
+      "Please note that this script requires the yfinance library. If you don't have this library installed, you can install it with the command `pip install yfinance`.\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[31m\n",
       ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
+      "\u001b[31m\n",
+      ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
       "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
       "\n",
       "exitcode: 0 (execution succeeded)\n",
       "Code output: \n",
-      "AAPL: 57.03%\n",
-      "MSFT: 41.90%\n",
-      "GOOGL: 48.77%\n",
-      "GOOG: 48.28%\n",
-      "AMZN: 54.06%\n",
-      "META: 160.93%\n",
-      "TSLA: 146.48%\n",
-      "NVDA: 226.67%\n",
-      "ASML: 31.38%\n",
-      "ADBE: 56.97%\n",
+      "AAPL: 46.34%\n",
+      "MSFT: 37.55%\n",
+      "GOOGL: 51.08%\n",
+      "AMZN: 56.17%\n",
+      "META: 133.67%\n",
+      "TSLA: 131.02%\n",
+      "NVDA: 237.38%\n",
+      "PYPL: -16.55%\n",
+      "ADBE: 59.71%\n",
+      "NFLX: 43.44%\n",
       "\n",
       "\n",
       "--------------------------------------------------------------------------------\n",
       "\u001b[33massistant\u001b[0m (to user_proxy):\n",
       "\n",
-      "Great! The code executed successfully, and here are the updated YTD gains for the 10 largest technology companies, including Meta Platforms (formerly Facebook):\n",
+      "The script has successfully fetched the Year-to-Date (YTD) gain for the largest technology companies, including Meta Platforms (formerly Facebook). The YTD gains are as follows:\n",
       "\n",
-      "- AAPL: 57.03%\n",
-      "- MSFT: 41.90%\n",
-      "- GOOGL: 48.77%\n",
-      "- GOOG: 48.28%\n",
-      "- AMZN: 54.06%\n",
-      "- META: 160.93%\n",
-      "- TSLA: 146.48%\n",
-      "- NVDA: 226.67%\n",
-      "- ASML: 31.38%\n",
-      "- ADBE: 56.97%\n",
+      "- Apple (AAPL): 46.34%\n",
+      "- Microsoft (MSFT): 37.55%\n",
+      "- Alphabet (GOOGL): 51.08%\n",
+      "- Amazon (AMZN): 56.17%\n",
+      "- Meta Platforms (META): 133.67%\n",
+      "- Tesla (TSLA): 131.02%\n",
+      "- Nvidia (NVDA): 237.38%\n",
+      "- Paypal (PYPL): -16.55%\n",
+      "- Adobe (ADBE): 59.71%\n",
+      "- Netflix (NFLX): 43.44%\n",
       "\n",
-      "Please note that these values are subject to change as the market fluctuates. If you need updated values, you can re-run the code.\n",
+      "Please note that these values are subject to change as the stock market fluctuates. \n",
       "\n",
       "TERMINATE\n",
       "\n",

notebook/automl_classification.ipynb

@@ -80,7 +80,7 @@
    ],
    "source": [
     "from minio.error import ServerError\n",
-    "from flaml.data import load_openml_dataset\n",
+    "from flaml.automl.data import load_openml_dataset\n",
     "\n",
     "try:\n",
     "    X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=1169, data_dir='./')\n",
@@ -1252,7 +1252,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
     "    get_output_from_log(filename=settings['log_file_name'], time_budget=240)\n",
     "for config in config_history:\n",
@@ -1540,7 +1540,7 @@
    "outputs": [],
    "source": [
     "''' SKLearnEstimator is the super class for a sklearn learner '''\n",
-    "from flaml.model import SKLearnEstimator\n",
+    "from flaml.automl.model import SKLearnEstimator\n",
     "from flaml import tune\n",
     "from flaml.automl.task.task import CLASSIFICATION\n",
     "\n",

notebook/automl_flight_delays.ipynb

@@ -37,383 +37,20 @@
         "\n",
         "In this notebook, we use one real data example (binary classification) to showcase how to use FLAML library.\n",
         "\n",
-        "FLAML requires `Python>=3.7`. To run this notebook example, please install the following packages."
+        "FLAML requires `Python>=3.8`. To run this notebook example, please install the following packages."
       ]
     },
     {
       "cell_type": "code",
-      "execution_count": 39,
+      "execution_count": null,
       "metadata": {
         "jupyter": {
           "outputs_hidden": true
         }
       },
-      "outputs": [
-        {
-          "data": {
-            "application/vnd.livy.statement-meta+json": {
-              "execution_finish_time": "2023-04-09T03:11:05.782522Z",
-              "execution_start_time": "2023-04-09T03:11:05.7822033Z",
-              "livy_statement_state": "available",
-              "parent_msg_id": "18b2ee64-09c4-4ceb-8975-e4ed43d7c41a",
-              "queued_time": "2023-04-09T03:10:33.571519Z",
-              "session_id": "7",
-              "session_start_time": null,
-              "spark_jobs": null,
-              "spark_pool": null,
-              "state": "finished",
-              "statement_id": -1
-            },
-            "text/plain": [
-              "StatementMeta(, 7, -1, Finished, Available)"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {},
-          "execution_count": 39,
-          "metadata": {},
-          "output_type": "execute_result"
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Collecting flaml[synapse]==1.1.3\n",
-            "  Using cached FLAML-1.1.3-py3-none-any.whl (224 kB)\n",
-            "Collecting xgboost==1.6.1\n",
-            "  Using cached xgboost-1.6.1-py3-none-manylinux2014_x86_64.whl (192.9 MB)\n",
-            "Collecting pandas==1.5.1\n",
-            "  Using cached pandas-1.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)\n",
-            "Collecting numpy==1.23.4\n",
-            "  Using cached numpy-1.23.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)\n",
-            "Collecting openml\n",
-            "  Using cached openml-0.13.1-py3-none-any.whl\n",
-            "Collecting scipy>=1.4.1\n",
-            "  Using cached scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)\n",
-            "Collecting scikit-learn>=0.24\n",
-            "  Using cached scikit_learn-1.2.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.8 MB)\n",
-            "Collecting lightgbm>=2.3.1\n",
-            "  Using cached lightgbm-3.3.5-py3-none-manylinux1_x86_64.whl (2.0 MB)\n",
-            "Collecting pyspark>=3.0.0\n",
-            "  Using cached pyspark-3.3.2-py2.py3-none-any.whl\n",
-            "Collecting optuna==2.8.0\n",
-            "  Using cached optuna-2.8.0-py3-none-any.whl (301 kB)\n",
-            "Collecting joblibspark>=0.5.0\n",
-            "  Using cached joblibspark-0.5.1-py3-none-any.whl (15 kB)\n",
-            "Collecting python-dateutil>=2.8.1\n",
-            "  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
-            "Collecting pytz>=2020.1\n",
-            "  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)\n",
-            "Collecting cliff\n",
-            "  Using cached cliff-4.2.0-py3-none-any.whl (81 kB)\n",
-            "Collecting packaging>=20.0\n",
-            "  Using cached packaging-23.0-py3-none-any.whl (42 kB)\n",
-            "Collecting cmaes>=0.8.2\n",
-            "  Using cached cmaes-0.9.1-py3-none-any.whl (21 kB)\n",
-            "Collecting sqlalchemy>=1.1.0\n",
-            "  Using cached SQLAlchemy-2.0.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)\n",
-            "Collecting tqdm\n",
-            "  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)\n",
-            "Collecting alembic\n",
-            "  Using cached alembic-1.10.3-py3-none-any.whl (212 kB)\n",
-            "Collecting colorlog\n",
-            "  Using cached colorlog-6.7.0-py2.py3-none-any.whl (11 kB)\n",
-            "Collecting xmltodict\n",
-            "  Using cached xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
-            "Collecting requests\n",
-            "  Using cached requests-2.28.2-py3-none-any.whl (62 kB)\n",
-            "Collecting minio\n",
-            "  Using cached minio-7.1.14-py3-none-any.whl (77 kB)\n",
-            "Collecting liac-arff>=2.4.0\n",
-            "  Using cached liac_arff-2.5.0-py3-none-any.whl\n",
-            "Collecting pyarrow\n",
-            "  Using cached pyarrow-11.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35.0 MB)\n",
-            "Collecting joblib>=0.14\n",
-            "  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)\n",
-            "Collecting wheel\n",
-            "  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)\n",
-            "Collecting py4j==0.10.9.5\n",
-            "  Using cached py4j-0.10.9.5-py2.py3-none-any.whl (199 kB)\n",
-            "Collecting six>=1.5\n",
-            "  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
-            "Collecting threadpoolctl>=2.0.0\n",
-            "  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
-            "Collecting urllib3\n",
-            "  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)\n",
-            "Collecting certifi\n",
-            "  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)\n",
-            "Collecting idna<4,>=2.5\n",
-            "  Using cached idna-3.4-py3-none-any.whl (61 kB)\n",
-            "Collecting charset-normalizer<4,>=2\n",
-            "  Using cached charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB)\n",
-            "Collecting typing-extensions>=4.2.0\n",
-            "  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)\n",
-            "Collecting greenlet!=0.4.17\n",
-            "  Using cached greenlet-2.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (618 kB)\n",
-            "Collecting importlib-metadata\n",
-            "  Using cached importlib_metadata-6.2.0-py3-none-any.whl (21 kB)\n",
-            "Collecting importlib-resources\n",
-            "  Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)\n",
-            "Collecting Mako\n",
-            "  Using cached Mako-1.2.4-py3-none-any.whl (78 kB)\n",
-            "Collecting autopage>=0.4.0\n",
-            "  Using cached autopage-0.5.1-py3-none-any.whl (29 kB)\n",
-            "Collecting cmd2>=1.0.0\n",
-            "  Using cached cmd2-2.4.3-py3-none-any.whl (147 kB)\n",
-            "Collecting stevedore>=2.0.1\n",
-            "  Using cached stevedore-5.0.0-py3-none-any.whl (49 kB)\n",
-            "Collecting PrettyTable>=0.7.2\n",
-            "  Using cached prettytable-3.6.0-py3-none-any.whl (27 kB)\n",
-            "Collecting PyYAML>=3.12\n",
-            "  Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)\n",
-            "Collecting attrs>=16.3.0\n",
-            "  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)\n",
-            "Collecting pyperclip>=1.6\n",
-            "  Using cached pyperclip-1.8.2-py3-none-any.whl\n",
-            "Collecting wcwidth>=0.1.7\n",
-            "  Using cached wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)\n",
-            "Collecting zipp>=0.5\n",
-            "  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)\n",
-            "Collecting pbr!=2.1.0,>=2.0.0\n",
-            "  Using cached pbr-5.11.1-py2.py3-none-any.whl (112 kB)\n",
-            "Collecting MarkupSafe>=0.9.2\n",
-            "  Using cached MarkupSafe-2.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n",
-            "Installing collected packages: wcwidth, pytz, pyperclip, py4j, zipp, xmltodict, wheel, urllib3, typing-extensions, tqdm, threadpoolctl, six, PyYAML, pyspark, PrettyTable, pbr, packaging, numpy, MarkupSafe, liac-arff, joblib, idna, greenlet, colorlog, charset-normalizer, certifi, autopage, attrs, stevedore, sqlalchemy, scipy, requests, python-dateutil, pyarrow, minio, Mako, joblibspark, importlib-resources, importlib-metadata, cmd2, cmaes, xgboost, scikit-learn, pandas, cliff, alembic, optuna, openml, lightgbm, flaml\n",
-            "  Attempting uninstall: wcwidth\n",
-            "    Found existing installation: wcwidth 0.2.6\n",
-            "    Uninstalling wcwidth-0.2.6:\n",
-            "      Successfully uninstalled wcwidth-0.2.6\n",
-            "  Attempting uninstall: pytz\n",
-            "    Found existing installation: pytz 2023.3\n",
-            "    Uninstalling pytz-2023.3:\n",
-            "      Successfully uninstalled pytz-2023.3\n",
-            "  Attempting uninstall: pyperclip\n",
-            "    Found existing installation: pyperclip 1.8.2\n",
-            "    Uninstalling pyperclip-1.8.2:\n",
-            "      Successfully uninstalled pyperclip-1.8.2\n",
-            "  Attempting uninstall: py4j\n",
-            "    Found existing installation: py4j 0.10.9.5\n",
-            "    Uninstalling py4j-0.10.9.5:\n",
-            "      Successfully uninstalled py4j-0.10.9.5\n",
-            "  Attempting uninstall: zipp\n",
-            "    Found existing installation: zipp 3.15.0\n",
-            "    Uninstalling zipp-3.15.0:\n",
-            "      Successfully uninstalled zipp-3.15.0\n",
-            "  Attempting uninstall: xmltodict\n",
-            "    Found existing installation: xmltodict 0.13.0\n",
-            "    Uninstalling xmltodict-0.13.0:\n",
-            "      Successfully uninstalled xmltodict-0.13.0\n",
-            "  Attempting uninstall: wheel\n",
-            "    Found existing installation: wheel 0.40.0\n",
-            "    Uninstalling wheel-0.40.0:\n",
-            "      Successfully uninstalled wheel-0.40.0\n",
-            "  Attempting uninstall: urllib3\n",
-            "    Found existing installation: urllib3 1.26.15\n",
-            "    Uninstalling urllib3-1.26.15:\n",
-            "      Successfully uninstalled urllib3-1.26.15\n",
-            "  Attempting uninstall: typing-extensions\n",
-            "    Found existing installation: typing_extensions 4.5.0\n",
-            "    Uninstalling typing_extensions-4.5.0:\n",
-            "      Successfully uninstalled typing_extensions-4.5.0\n",
-            "  Attempting uninstall: tqdm\n",
-            "    Found existing installation: tqdm 4.65.0\n",
-            "    Uninstalling tqdm-4.65.0:\n",
-            "      Successfully uninstalled tqdm-4.65.0\n",
-            "  Attempting uninstall: threadpoolctl\n",
-            "    Found existing installation: threadpoolctl 3.1.0\n",
-            "    Uninstalling threadpoolctl-3.1.0:\n",
-            "      Successfully uninstalled threadpoolctl-3.1.0\n",
-            "  Attempting uninstall: six\n",
-            "    Found existing installation: six 1.16.0\n",
-            "    Uninstalling six-1.16.0:\n",
-            "      Successfully uninstalled six-1.16.0\n",
-            "  Attempting uninstall: PyYAML\n",
-            "    Found existing installation: PyYAML 6.0\n",
-            "    Uninstalling PyYAML-6.0:\n",
-            "      Successfully uninstalled PyYAML-6.0\n",
-            "  Attempting uninstall: pyspark\n",
-            "    Found existing installation: pyspark 3.3.2\n",
-            "    Uninstalling pyspark-3.3.2:\n",
-            "      Successfully uninstalled pyspark-3.3.2\n",
-            "  Attempting uninstall: PrettyTable\n",
-            "    Found existing installation: prettytable 3.6.0\n",
-            "    Uninstalling prettytable-3.6.0:\n",
-            "      Successfully uninstalled prettytable-3.6.0\n",
-            "  Attempting uninstall: pbr\n",
-            "    Found existing installation: pbr 5.11.1\n",
-            "    Uninstalling pbr-5.11.1:\n",
-            "      Successfully uninstalled pbr-5.11.1\n",
-            "  Attempting uninstall: packaging\n",
-            "    Found existing installation: packaging 23.0\n",
-            "    Uninstalling packaging-23.0:\n",
-            "      Successfully uninstalled packaging-23.0\n",
-            "  Attempting uninstall: numpy\n",
-            "    Found existing installation: numpy 1.23.4\n",
-            "    Uninstalling numpy-1.23.4:\n",
-            "      Successfully uninstalled numpy-1.23.4\n",
-            "  Attempting uninstall: MarkupSafe\n",
-            "    Found existing installation: MarkupSafe 2.1.2\n",
-            "    Uninstalling MarkupSafe-2.1.2:\n",
-            "      Successfully uninstalled MarkupSafe-2.1.2\n",
-            "  Attempting uninstall: liac-arff\n",
-            "    Found existing installation: liac-arff 2.5.0\n",
-            "    Uninstalling liac-arff-2.5.0:\n",
-            "      Successfully uninstalled liac-arff-2.5.0\n",
-            "  Attempting uninstall: joblib\n",
-            "    Found existing installation: joblib 1.2.0\n",
-            "    Uninstalling joblib-1.2.0:\n",
-            "      Successfully uninstalled joblib-1.2.0\n",
-            "  Attempting uninstall: idna\n",
-            "    Found existing installation: idna 3.4\n",
-            "    Uninstalling idna-3.4:\n",
-            "      Successfully uninstalled idna-3.4\n",
-            "  Attempting uninstall: greenlet\n",
-            "    Found existing installation: greenlet 2.0.2\n",
-            "    Uninstalling greenlet-2.0.2:\n",
-            "      Successfully uninstalled greenlet-2.0.2\n",
-            "  Attempting uninstall: colorlog\n",
-            "    Found existing installation: colorlog 6.7.0\n",
-            "    Uninstalling colorlog-6.7.0:\n",
-            "      Successfully uninstalled colorlog-6.7.0\n",
-            "  Attempting uninstall: charset-normalizer\n",
-            "    Found existing installation: charset-normalizer 3.1.0\n",
-            "    Uninstalling charset-normalizer-3.1.0:\n",
-            "      Successfully uninstalled charset-normalizer-3.1.0\n",
-            "  Attempting uninstall: certifi\n",
-            "    Found existing installation: certifi 2022.12.7\n",
-            "    Uninstalling certifi-2022.12.7:\n",
-            "      Successfully uninstalled certifi-2022.12.7\n",
-            "  Attempting uninstall: autopage\n",
-            "    Found existing installation: autopage 0.5.1\n",
-            "    Uninstalling autopage-0.5.1:\n",
-            "      Successfully uninstalled autopage-0.5.1\n",
-            "  Attempting uninstall: attrs\n",
-            "    Found existing installation: attrs 22.2.0\n",
-            "    Uninstalling attrs-22.2.0:\n",
-            "      Successfully uninstalled attrs-22.2.0\n",
-            "  Attempting uninstall: stevedore\n",
-            "    Found existing installation: stevedore 5.0.0\n",
-            "    Uninstalling stevedore-5.0.0:\n",
-            "      Successfully uninstalled stevedore-5.0.0\n",
-            "  Attempting uninstall: sqlalchemy\n",
-            "    Found existing installation: SQLAlchemy 2.0.9\n",
-            "    Uninstalling SQLAlchemy-2.0.9:\n",
-            "      Successfully uninstalled SQLAlchemy-2.0.9\n",
-            "  Attempting uninstall: scipy\n",
-            "    Found existing installation: scipy 1.10.1\n",
-            "    Uninstalling scipy-1.10.1:\n",
-            "      Successfully uninstalled scipy-1.10.1\n",
-            "  Attempting uninstall: requests\n",
-            "    Found existing installation: requests 2.28.2\n",
-            "    Uninstalling requests-2.28.2:\n",
-            "      Successfully uninstalled requests-2.28.2\n",
-            "  Attempting uninstall: python-dateutil\n",
-            "    Found existing installation: python-dateutil 2.8.2\n",
-            "    Uninstalling python-dateutil-2.8.2:\n",
-            "      Successfully uninstalled python-dateutil-2.8.2\n",
-            "  Attempting uninstall: pyarrow\n",
-            "    Found existing installation: pyarrow 11.0.0\n",
-            "    Uninstalling pyarrow-11.0.0:\n",
-            "      Successfully uninstalled pyarrow-11.0.0\n",
-            "  Attempting uninstall: minio\n",
-            "    Found existing installation: minio 7.1.14\n",
-            "    Uninstalling minio-7.1.14:\n",
-            "      Successfully uninstalled minio-7.1.14\n",
-            "  Attempting uninstall: Mako\n",
-            "    Found existing installation: Mako 1.2.4\n",
-            "    Uninstalling Mako-1.2.4:\n",
-            "      Successfully uninstalled Mako-1.2.4\n",
-            "  Attempting uninstall: joblibspark\n",
-            "    Found existing installation: joblibspark 0.5.1\n",
-            "    Uninstalling joblibspark-0.5.1:\n",
-            "      Successfully uninstalled joblibspark-0.5.1\n",
-            "  Attempting uninstall: importlib-resources\n",
-            "    Found existing installation: importlib-resources 5.12.0\n",
-            "    Uninstalling importlib-resources-5.12.0:\n",
-            "      Successfully uninstalled importlib-resources-5.12.0\n",
-            "  Attempting uninstall: importlib-metadata\n",
-            "    Found existing installation: importlib-metadata 6.2.0\n",
-            "    Uninstalling importlib-metadata-6.2.0:\n",
-            "      Successfully uninstalled importlib-metadata-6.2.0\n",
-            "  Attempting uninstall: cmd2\n",
-            "    Found existing installation: cmd2 2.4.3\n",
-            "    Uninstalling cmd2-2.4.3:\n",
-            "      Successfully uninstalled cmd2-2.4.3\n",
-            "  Attempting uninstall: cmaes\n",
-            "    Found existing installation: cmaes 0.9.1\n",
-            "    Uninstalling cmaes-0.9.1:\n",
-            "      Successfully uninstalled cmaes-0.9.1\n",
-            "  Attempting uninstall: xgboost\n",
-            "    Found existing installation: xgboost 1.6.1\n",
-            "    Uninstalling xgboost-1.6.1:\n",
-            "      Successfully uninstalled xgboost-1.6.1\n",
-            "  Attempting uninstall: scikit-learn\n",
-            "    Found existing installation: scikit-learn 1.2.2\n",
-            "    Uninstalling scikit-learn-1.2.2:\n",
-            "      Successfully uninstalled scikit-learn-1.2.2\n",
-            "  Attempting uninstall: pandas\n",
-            "    Found existing installation: pandas 1.5.1\n",
-            "    Uninstalling pandas-1.5.1:\n",
-            "      Successfully uninstalled pandas-1.5.1\n",
-            "  Attempting uninstall: cliff\n",
-            "    Found existing installation: cliff 4.2.0\n",
-            "    Uninstalling cliff-4.2.0:\n",
-            "      Successfully uninstalled cliff-4.2.0\n",
-            "  Attempting uninstall: alembic\n",
-            "    Found existing installation: alembic 1.10.3\n",
-            "    Uninstalling alembic-1.10.3:\n",
-            "      Successfully uninstalled alembic-1.10.3\n",
-            "  Attempting uninstall: optuna\n",
-            "    Found existing installation: optuna 2.8.0\n",
-            "    Uninstalling optuna-2.8.0:\n",
-            "      Successfully uninstalled optuna-2.8.0\n",
-            "  Attempting uninstall: openml\n",
-            "    Found existing installation: openml 0.13.1\n",
-            "    Uninstalling openml-0.13.1:\n",
-            "      Successfully uninstalled openml-0.13.1\n",
-            "  Attempting uninstall: lightgbm\n",
-            "    Found existing installation: lightgbm 3.3.5\n",
-            "    Uninstalling lightgbm-3.3.5:\n",
-            "      Successfully uninstalled lightgbm-3.3.5\n",
-            "  Attempting uninstall: flaml\n",
-            "    Found existing installation: FLAML 1.1.3\n",
-            "    Uninstalling FLAML-1.1.3:\n",
-            "      Successfully uninstalled FLAML-1.1.3\n",
-            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
-            "virtualenv 20.14.0 requires platformdirs<3,>=2, but you have platformdirs 3.2.0 which is incompatible.\n",
-            "tensorflow 2.4.1 requires six~=1.15.0, but you have six 1.16.0 which is incompatible.\n",
-            "tensorflow 2.4.1 requires typing-extensions~=3.7.4, but you have typing-extensions 4.5.0 which is incompatible.\n",
-            "pmdarima 1.8.2 requires numpy~=1.19.0, but you have numpy 1.23.4 which is incompatible.\n",
-            "koalas 1.8.0 requires numpy<1.20.0,>=1.14, but you have numpy 1.23.4 which is incompatible.\n",
-            "gevent 21.1.2 requires greenlet<2.0,>=0.4.17; platform_python_implementation == \"CPython\", but you have greenlet 2.0.2 which is incompatible.\n",
-            "azureml-dataset-runtime 1.34.0 requires pyarrow<4.0.0,>=0.17.0, but you have pyarrow 11.0.0 which is incompatible.\n",
-            "azureml-core 1.34.0 requires urllib3<=1.26.6,>=1.23, but you have urllib3 1.26.15 which is incompatible.\u001b[0m\u001b[31m\n",
-            "\u001b[0mSuccessfully installed Mako-1.2.4 MarkupSafe-2.1.2 PrettyTable-3.6.0 PyYAML-6.0 alembic-1.10.3 attrs-22.2.0 autopage-0.5.1 certifi-2022.12.7 charset-normalizer-3.1.0 cliff-4.2.0 cmaes-0.9.1 cmd2-2.4.3 colorlog-6.7.0 flaml-1.1.3 greenlet-2.0.2 idna-3.4 importlib-metadata-6.2.0 importlib-resources-5.12.0 joblib-1.2.0 joblibspark-0.5.1 liac-arff-2.5.0 lightgbm-3.3.5 minio-7.1.14 numpy-1.23.4 openml-0.13.1 optuna-2.8.0 packaging-23.0 pandas-1.5.1 pbr-5.11.1 py4j-0.10.9.5 pyarrow-11.0.0 pyperclip-1.8.2 pyspark-3.3.2 python-dateutil-2.8.2 pytz-2023.3 requests-2.28.2 scikit-learn-1.2.2 scipy-1.10.1 six-1.16.0 sqlalchemy-2.0.9 stevedore-5.0.0 threadpoolctl-3.1.0 tqdm-4.65.0 typing-extensions-4.5.0 urllib3-1.26.15 wcwidth-0.2.6 wheel-0.40.0 xgboost-1.6.1 xmltodict-0.13.0 zipp-3.15.0\n",
-            "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.\n",
-            "You should consider upgrading via the '/nfs4/pyenv-bfada21f-d1ed-44b9-a41d-4ff480d237e7/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
-            "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
-          ]
-        },
-        {
-          "data": {},
-          "execution_count": 39,
-          "metadata": {},
-          "output_type": "execute_result"
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Warning: PySpark kernel has been restarted to use updated packages.\n",
-            "\n"
-          ]
-        }
-      ],
+      "outputs": [],
       "source": [
-        "%pip install flaml[synapse]==1.1.3 xgboost==1.6.1 pandas==1.5.1 numpy==1.23.4 openml --force-reinstall"
+        "%pip install flaml[automl,synapse] xgboost==1.6.1 pandas==1.5.1 numpy==1.23.4 openml --force-reinstall"
       ]
     },
     {
@@ -480,7 +117,7 @@
         }
       ],
       "source": [
-        "from flaml.data import load_openml_dataset\n",
+        "from flaml.automl.data import load_openml_dataset\n",
         "X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=1169, data_dir='./')"
       ]
     },
@@ -1389,7 +1026,7 @@
         }
       ],
       "source": [
-        "from flaml.data import get_output_from_log\n",
+        "from flaml.automl.data import get_output_from_log\n",
         "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
         "    get_output_from_log(filename=settings['log_file_name'], time_budget=240)\n",
         "for config in config_history:\n",
@@ -1861,7 +1498,7 @@
         }
       ],
       "source": [
-        "!pip install rgf-python   "
+        "%pip install rgf-python   "
       ]
     },
     {
@@ -1898,9 +1535,9 @@
       ],
       "source": [
         "''' SKLearnEstimator is the super class for a sklearn learner '''\n",
-        "from flaml.model import SKLearnEstimator\n",
+        "from flaml.automl.model import SKLearnEstimator\n",
         "from flaml import tune\n",
-        "from flaml.data import CLASSIFICATION\n",
+        "from flaml.automl.data import CLASSIFICATION\n",
         "\n",
         "\n",
         "class MyRegularizedGreedyForest(SKLearnEstimator):\n",

notebook/automl_lightgbm.ipynb

@@ -28,7 +28,7 @@
     "\n",
     "In this notebook, we demonstrate how to use FLAML library to tune hyperparameters of LightGBM with a regression example.\n",
     "\n",
-    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the `automl` option (this option is introduced from version 2, for version 1 it is installed by default):\n",
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the `automl` option (this option is introduced from version 2, for version 1 it is installed by default):\n",
     "```bash\n",
     "pip install flaml[automl]\n",
     "```"
@@ -87,7 +87,7 @@
     }
    ],
    "source": [
-    "from flaml.data import load_openml_dataset\n",
+    "from flaml.automl.data import load_openml_dataset\n",
     "X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=537, data_dir='./')"
    ]
   },
@@ -509,7 +509,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
     "    get_output_from_log(filename=settings['log_file_name'], time_budget=60)\n",
     "\n",
@@ -852,7 +852,7 @@
     "        coef[0] * hess + coef[1] * hess_rmse + coef[2] * hess_mae\n",
     "\n",
     "\n",
-    "from flaml.model import LGBMEstimator\n",
+    "from flaml.automl.model import LGBMEstimator\n",
     "\n",
     "''' create a customized LightGBM learner class with your objective function '''\n",
     "class MyLGBM(LGBMEstimator):\n",

notebook/automl_nlp.ipynb

@@ -21,7 +21,7 @@
     "\n",
     "In this notebook, we demonstrate how to use the FLAML library to fine tune an NLP language model with hyperparameter search. We will use [flaml.tune](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function) with the built in GPU in colab for the tuning. However, if you have a machine with more than 1 GPU, you can also use FLAML's [parallel tuning](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML#parallel-tuning) with the ray tune option. \n",
     "\n",
-    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the `[automl,hf,blendsearch]` option:\n",
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the `[automl,hf,blendsearch]` option:\n",
     "```bash\n",
     "pip install flaml[automl,hf,blendsearch]; \n",
     "```"
@@ -2107,7 +2107,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
     "    get_output_from_log(filename=automl_settings['log_file_name'], time_budget=3000)\n",
     "for config in config_history:\n",
@@ -3460,7 +3460,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "import matplotlib.pyplot as plt\n",
     "import numpy as np\n",
     "\n",
@@ -4098,7 +4098,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
     "    get_output_from_log(filename=automl_settings['log_file_name'], time_budget=3000)\n",
     "for config in config_history:\n",
@@ -5136,7 +5136,7 @@
    ],
    "source": [
     "\n",
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
     "    get_output_from_log(filename=automl_settings['log_file_name'], time_budget=3000)\n",
     "for config in config_history:\n",

notebook/automl_time_series_forecast.ipynb

@@ -22,7 +22,7 @@
     "\n",
     "In this notebook, we demonstrate how to use FLAML library for time series forecasting tasks: univariate time series forecasting (only time), multivariate time series forecasting (with exogneous variables) and forecasting discrete values.\n",
     "\n",
-    "FLAML requires Python>=3.7. To run this notebook example, please install flaml with the [automl,ts_forecast] option:\n"
+    "FLAML requires Python>=3.8. To run this notebook example, please install flaml with the [automl,ts_forecast] option:\n"
    ]
   },
   {
@@ -1518,7 +1518,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, train_loss_history = \\\n",
     "    get_output_from_log(filename=settings['log_file_name'], time_budget=180)\n",
     "\n",

notebook/automl_xgboost.ipynb

@@ -28,7 +28,7 @@
     "\n",
     "In this notebook, we demonstrate how to use FLAML library to tune hyperparameters of XGBoost with a regression example.\n",
     "\n",
-    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the `automl` option (this option is introduced from version 2, for version 1 it is installed by default):\n",
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the `automl` option (this option is introduced from version 2, for version 1 it is installed by default):\n",
     "```bash\n",
     "pip install flaml[automl]\n",
     "```"
@@ -44,6 +44,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "slideshow": {
@@ -87,11 +88,12 @@
     }
    ],
    "source": [
-    "from flaml.data import load_openml_dataset\n",
+    "from flaml.automl.data import load_openml_dataset\n",
     "X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=537, data_dir='./')"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "slideshow": {
@@ -509,6 +511,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "slideshow": {
@@ -761,7 +764,7 @@
     }
    ],
    "source": [
-    "from flaml.data import get_output_from_log\n",
+    "from flaml.automl.data import get_output_from_log\n",
     "time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
     "    get_output_from_log(filename=settings['log_file_name'], time_budget=60)\n",
     "\n",
@@ -804,6 +807,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -832,6 +836,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -922,6 +927,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -1839,7 +1845,7 @@
     "    return grad, hess\n",
     "\n",
     "# create customized XGBoost learners class with your objective function\n",
-    "from flaml.model import XGBoostEstimator\n",
+    "from flaml.automl.model import XGBoostEstimator\n",
     "\n",
     "\n",
     "class MyXGB1(XGBoostEstimator):\n",

notebook/integrate_azureml.ipynb

@@ -28,7 +28,7 @@
     "\n",
     "In this notebook, we use one real data example (binary classification) to showcase how to use FLAML library together with AzureML.\n",
     "\n",
-    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [automl,azureml] option:\n",
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [automl,azureml] option:\n",
     "```bash\n",
     "pip install flaml[automl,azureml]\n",
     "```"
@@ -88,7 +88,7 @@
    },
    "outputs": [],
    "source": [
-    "from flaml.data import load_openml_dataset\n",
+    "from flaml.automl.data import load_openml_dataset\n",
     "X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=1169, data_dir='./')"
    ]
   },

notebook/integrate_sklearn.ipynb

@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -12,6 +13,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -40,7 +42,7 @@
     "\n",
     "In this notebook, we use one real data example (binary classification) to showcase how to use FLAML library.\n",
     "\n",
-    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the `[automl]` option (this option is introduced from version 2, for version 1 it is installed by default):\n",
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the `[automl]` option (this option is introduced from version 2, for version 1 it is installed by default):\n",
     "```bash\n",
     "pip install flaml[automl]\n",
     "```"
@@ -56,6 +58,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -76,6 +79,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -83,6 +87,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -109,7 +114,7 @@
     }
    ],
    "source": [
-    "from flaml.data import load_openml_dataset\n",
+    "from flaml.automl.data import load_openml_dataset\n",
     "X_train, X_test, y_train, y_test = load_openml_dataset(\n",
     "    dataset_id=1169, data_dir='./', random_state=1234, dataset_format='array')"
    ]
@@ -135,6 +140,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -232,6 +238,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -449,7 +456,7 @@
     {
      "data": {
       "text/plain": [
-       "<flaml.model.XGBoostSklearnEstimator at 0x7f03a5eada00>"
+       "<flaml.automl.model.XGBoostSklearnEstimator at 0x7f03a5eada00>"
       ]
      },
      "execution_count": 10,
@@ -462,6 +469,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [

notebook/zeroshot_lightgbm.ipynb

@@ -28,7 +28,7 @@
     "\n",
     "In this notebook, we demonstrate a basic use case of zero-shot AutoML with FLAML.\n",
     "\n",
-    "FLAML requires `Python>=3.7`. To run this notebook example, please install the [autozero] option:"
+    "FLAML requires `Python>=3.8`. To run this notebook example, please install the [autozero] option:"
    ]
   },
   {
@@ -130,7 +130,7 @@
     }
    ],
    "source": [
-    "from flaml.data import load_openml_dataset\n",
+    "from flaml.automl.data import load_openml_dataset\n",
     "X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=537, data_dir='./')"
    ]
   },

test/autogen/agentchat/test_assistant_agent.py

@@ -69,6 +69,7 @@ def test_gpt35(human_input_mode="NEVER", max_consecutive_auto_reply=5):
             "model": {
                 "gpt-3.5-turbo",
                 "gpt-3.5-turbo-16k",
+                "gpt-3.5-turbo-16k-0613",
                 "gpt-3.5-turbo-0301",
                 "chatgpt-35-turbo-0301",
                 "gpt-35-turbo-v0301",

test/autogen/agentchat/test_async.py

@@ -98,7 +98,7 @@ async def test_stream():
                 )
             return False, None
 
-    user_proxy.register_auto_reply(autogen.AssistantAgent, add_data_reply, 1, config={"news_stream": data})
+    user_proxy.register_reply(autogen.AssistantAgent, add_data_reply, 1, config={"news_stream": data})
 
     await user_proxy.a_initiate_chat(
         assistant,

test/autogen/agentchat/test_conversable_agent.py

@@ -1,48 +1,48 @@
 import pytest
-from flaml.autogen.agentchat import ResponsiveAgent
+from flaml.autogen.agentchat import ConversableAgent
 
 
 def test_trigger():
-    agent = ResponsiveAgent("a0", max_consecutive_auto_reply=0, llm_config=False, human_input_mode="NEVER")
-    agent1 = ResponsiveAgent("a1", max_consecutive_auto_reply=0, human_input_mode="NEVER")
-    agent.register_auto_reply(agent1, lambda recipient, messages, sender, config: (True, "hello"))
+    agent = ConversableAgent("a0", max_consecutive_auto_reply=0, llm_config=False, human_input_mode="NEVER")
+    agent1 = ConversableAgent("a1", max_consecutive_auto_reply=0, human_input_mode="NEVER")
+    agent.register_reply(agent1, lambda recipient, messages, sender, config: (True, "hello"))
     agent1.initiate_chat(agent, message="hi")
     assert agent1.last_message(agent)["content"] == "hello"
-    agent.register_auto_reply("a1", lambda recipient, messages, sender, config: (True, "hello a1"))
+    agent.register_reply("a1", lambda recipient, messages, sender, config: (True, "hello a1"))
     agent1.initiate_chat(agent, message="hi")
     assert agent1.last_message(agent)["content"] == "hello a1"
-    agent.register_auto_reply(
-        ResponsiveAgent, lambda recipient, messages, sender, config: (True, "hello responsive agent")
+    agent.register_reply(
+        ConversableAgent, lambda recipient, messages, sender, config: (True, "hello conversable agent")
     )
     agent1.initiate_chat(agent, message="hi")
-    assert agent1.last_message(agent)["content"] == "hello responsive agent"
-    agent.register_auto_reply(
+    assert agent1.last_message(agent)["content"] == "hello conversable agent"
+    agent.register_reply(
         lambda sender: sender.name.startswith("a"), lambda recipient, messages, sender, config: (True, "hello a")
     )
     agent1.initiate_chat(agent, message="hi")
     assert agent1.last_message(agent)["content"] == "hello a"
-    agent.register_auto_reply(
+    agent.register_reply(
         lambda sender: sender.name.startswith("b"), lambda recipient, messages, sender, config: (True, "hello b")
     )
     agent1.initiate_chat(agent, message="hi")
     assert agent1.last_message(agent)["content"] == "hello a"
-    agent.register_auto_reply(
+    agent.register_reply(
         ["agent2", agent1], lambda recipient, messages, sender, config: (True, "hello agent2 or agent1")
     )
     agent1.initiate_chat(agent, message="hi")
     assert agent1.last_message(agent)["content"] == "hello agent2 or agent1"
-    agent.register_auto_reply(
+    agent.register_reply(
         ["agent2", "agent3"], lambda recipient, messages, sender, config: (True, "hello agent2 or agent3")
     )
     agent1.initiate_chat(agent, message="hi")
     assert agent1.last_message(agent)["content"] == "hello agent2 or agent1"
-    pytest.raises(ValueError, agent.register_auto_reply, 1, lambda recipient, messages, sender, config: (True, "hi"))
+    pytest.raises(ValueError, agent.register_reply, 1, lambda recipient, messages, sender, config: (True, "hi"))
     pytest.raises(ValueError, agent._match_trigger, 1, agent1)
 
 
 def test_context():
-    agent = ResponsiveAgent("a0", max_consecutive_auto_reply=0, llm_config=False, human_input_mode="NEVER")
-    agent1 = ResponsiveAgent("a1", max_consecutive_auto_reply=0, human_input_mode="NEVER")
+    agent = ConversableAgent("a0", max_consecutive_auto_reply=0, llm_config=False, human_input_mode="NEVER")
+    agent1 = ConversableAgent("a1", max_consecutive_auto_reply=0, human_input_mode="NEVER")
     agent1.send(
         {
             "content": "hello {name}",
@@ -77,8 +77,8 @@ def test_context():
 
 
 def test_max_consecutive_auto_reply():
-    agent = ResponsiveAgent("a0", max_consecutive_auto_reply=2, llm_config=False, human_input_mode="NEVER")
-    agent1 = ResponsiveAgent("a1", max_consecutive_auto_reply=0, human_input_mode="NEVER")
+    agent = ConversableAgent("a0", max_consecutive_auto_reply=2, llm_config=False, human_input_mode="NEVER")
+    agent1 = ConversableAgent("a1", max_consecutive_auto_reply=0, human_input_mode="NEVER")
     assert agent.max_consecutive_auto_reply() == agent.max_consecutive_auto_reply(agent1) == 2
     agent.update_max_consecutive_auto_reply(1)
     assert agent.max_consecutive_auto_reply() == agent.max_consecutive_auto_reply(agent1) == 1
@@ -105,9 +105,9 @@ def test_max_consecutive_auto_reply():
     assert agent1.reply_at_receive[agent] is False and agent.reply_at_receive[agent1] is True
 
 
-def test_responsive_agent():
-    dummy_agent_1 = ResponsiveAgent(name="dummy_agent_1", human_input_mode="ALWAYS")
-    dummy_agent_2 = ResponsiveAgent(name="dummy_agent_2", human_input_mode="TERMINATE")
+def test_conversable_agent():
+    dummy_agent_1 = ConversableAgent(name="dummy_agent_1", human_input_mode="ALWAYS")
+    dummy_agent_2 = ConversableAgent(name="dummy_agent_2", human_input_mode="TERMINATE")
 
     # monkeypatch.setattr(sys, "stdin", StringIO("exit"))
     dummy_agent_1.receive("hello", dummy_agent_2)  # receive a str
@@ -154,8 +154,29 @@ def test_responsive_agent():
     assert dummy_agent_1.system_message == "new system message"
 
 
+def test_generate_reply():
+    def add_num(num_to_be_added):
+        given_num = 10
+        return num_to_be_added + given_num
+
+    dummy_agent_2 = ConversableAgent(name="user_proxy", human_input_mode="TERMINATE", function_map={"add_num": add_num})
+    messsages = [{"function_call": {"name": "add_num", "arguments": '{ "num_to_be_added": 5 }'}, "role": "assistant"}]
+
+    # when sender is None, messages is provided
+    assert (
+        dummy_agent_2.generate_reply(messages=messsages, sender=None)["content"] == "15"
+    ), "generate_reply not working when sender is None"
+
+    # when sender is provided, messages is None
+    dummy_agent_1 = ConversableAgent(name="dummy_agent_1", human_input_mode="ALWAYS")
+    dummy_agent_2._oai_messages[dummy_agent_1] = messsages
+    assert (
+        dummy_agent_2.generate_reply(messages=None, sender=dummy_agent_1)["content"] == "15"
+    ), "generate_reply not working when messages is None"
+
+
 if __name__ == "__main__":
     test_trigger()
     # test_context()
     # test_max_consecutive_auto_reply()
-    # test_responsive_agent(pytest.monkeypatch)
+    # test_conversable_agent(pytest.monkeypatch)

test/autogen/agentchat/test_groupchat.py

@@ -2,14 +2,14 @@ from flaml import autogen
 
 
 def test_chat_manager():
-    agent1 = autogen.ResponsiveAgent(
+    agent1 = autogen.ConversableAgent(
         "alice",
         max_consecutive_auto_reply=2,
         human_input_mode="NEVER",
         llm_config=False,
         default_auto_reply="This is alice sepaking.",
     )
-    agent2 = autogen.ResponsiveAgent(
+    agent2 = autogen.ConversableAgent(
         "bob",
         max_consecutive_auto_reply=2,
         human_input_mode="NEVER",
@@ -33,14 +33,14 @@ def test_chat_manager():
 
 def test_plugin():
     # Give another Agent class ability to manage group chat
-    agent1 = autogen.ResponsiveAgent(
+    agent1 = autogen.ConversableAgent(
         "alice",
         max_consecutive_auto_reply=2,
         human_input_mode="NEVER",
         llm_config=False,
         default_auto_reply="This is alice sepaking.",
     )
-    agent2 = autogen.ResponsiveAgent(
+    agent2 = autogen.ConversableAgent(
         "bob",
         max_consecutive_auto_reply=2,
         human_input_mode="NEVER",
@@ -48,8 +48,8 @@ def test_plugin():
         default_auto_reply="This is bob speaking.",
     )
     groupchat = autogen.GroupChat(agents=[agent1, agent2], messages=[], max_round=2)
-    group_chat_manager = autogen.ResponsiveAgent(name="deputy_manager", llm_config=False)
-    group_chat_manager.register_auto_reply(
+    group_chat_manager = autogen.ConversableAgent(name="deputy_manager", llm_config=False)
+    group_chat_manager.register_reply(
         autogen.Agent,
         reply_func=autogen.GroupChatManager.run_chat,
         config=groupchat,

test/autogen/oai/test_completion.py

@@ -134,6 +134,7 @@ def test_nocontext():
                 "model": {
                     "gpt-3.5-turbo",
                     "gpt-3.5-turbo-16k",
+                    "gpt-3.5-turbo-16k-0613",
                     "gpt-3.5-turbo-0301",
                     "chatgpt-35-turbo-0301",
                     "gpt-35-turbo-v0301",
@@ -169,6 +170,7 @@ def test_humaneval(num_samples=1):
             "model": {
                 "gpt-3.5-turbo",
                 "gpt-3.5-turbo-16k",
+                "gpt-3.5-turbo-16k-0613",
                 "gpt-3.5-turbo-0301",
                 "chatgpt-35-turbo-0301",
                 "gpt-35-turbo-v0301",

test/autogen/test_code.py

@@ -36,6 +36,7 @@ here = os.path.abspath(os.path.dirname(__file__))
 #     #         "model": {
 #     #             "gpt-3.5-turbo",
 #     #             "gpt-3.5-turbo-16k",
+#     #             "gpt-3.5-turbo-16k-0613",
 #     #             "gpt-3.5-turbo-0301",
 #     #             "chatgpt-35-turbo-0301",
 #     #             "gpt-35-turbo-v0301",

test/autogen/test_function_call.py

@@ -118,6 +118,14 @@ def test_execute_function():
     assert user.execute_function(func_call=func_call)[1]["content"] == "15"
     assert user.execute_function(func_call=func_call)[1]["content"] == "20"
 
+    # 3. test calling a function with no arguments
+    def get_number():
+        return 42
+
+    user = UserProxyAgent("user", function_map={"get_number": get_number})
+    func_call = {"name": "get_number", "arguments": "{}"}
+    assert user.execute_function(func_call)[1]["content"] == "42"
+
 
 if __name__ == "__main__":
     test_json_extraction()

test/automl/test_classification.py

@@ -277,12 +277,15 @@ class TestClassification(unittest.TestCase):
         import subprocess
         import sys
 
+        current_xgboost_version = xgb.__version__
         subprocess.check_call([sys.executable, "-m", "pip", "install", "xgboost==1.3.3", "--user"])
         automl = AutoML()
         automl.fit(X_train=X_train, y_train=y_train, **automl_settings)
         print(automl.feature_names_in_)
         print(automl.feature_importances_)
-        subprocess.check_call([sys.executable, "-m", "pip", "install", "-U", "xgboost", "--user"])
+        subprocess.check_call(
+            [sys.executable, "-m", "pip", "install", "-U", f"xgboost=={current_xgboost_version}", "--user"]
+        )
 
     def test_ray_classification(self):
         X, y = load_breast_cancer(return_X_y=True)

test/automl/test_split.py

@@ -91,7 +91,7 @@ def test_groups():
 def test_stratified_groupkfold():
     from sklearn.model_selection import StratifiedGroupKFold
     from minio.error import ServerError
-    from flaml.data import load_openml_dataset
+    from flaml.automl.data import load_openml_dataset
 
     try:
         X_train, _, y_train, _ = load_openml_dataset(dataset_id=1169, data_dir="test/")

test/spark/test_0sparkml.py

@@ -17,13 +17,14 @@ else:
         from pyspark.ml.feature import VectorAssembler
         from flaml.automl.spark.utils import to_pandas_on_spark
 
+        postfix_version = "-spark3.3," if pyspark.__version__ > "3.2" else ","
         spark = (
             pyspark.sql.SparkSession.builder.appName("MyApp")
             .master("local[2]")
             .config(
                 "spark.jars.packages",
                 (
-                    "com.microsoft.azure:synapseml_2.12:0.10.2,"
+                    f"com.microsoft.azure:synapseml_2.12:0.11.3{postfix_version}"
                     "org.apache.hadoop:hadoop-azure:3.3.5,"
                     "com.microsoft.azure:azure-storage:8.6.6,"
                     f"org.mlflow:mlflow-spark:{mlflow.__version__}"
@@ -172,15 +173,16 @@ def test_spark_input_df():
     try:
         model = automl.model.estimator
         predictions = model.transform(test_data)
+        predictions.show()
 
-        from synapse.ml.train import ComputeModelStatistics
+        # from synapse.ml.train import ComputeModelStatistics
 
-        metrics = ComputeModelStatistics(
-            evaluationMetric="classification",
-            labelCol="Bankrupt?",
-            scoredLabelsCol="prediction",
-        ).transform(predictions)
-        metrics.show()
+        # metrics = ComputeModelStatistics(
+        #     evaluationMetric="classification",
+        #     labelCol="Bankrupt?",
+        #     scoredLabelsCol="prediction",
+        # ).transform(predictions)
+        # metrics.show()
     except AttributeError:
         print("No fitted model because of too short training time.")
 

website/blog/2023-04-21-LLM-tuning-math/index.mdx

@@ -69,6 +69,6 @@ The need for model selection, parameter tuning and cost saving is not specific t
 ## For Further Reading
 
 * [Research paper about the tuning technique](https://arxiv.org/abs/2303.04673)
-* [Documentation about `flaml.autogen`](/docs/Use-Cases/Autogen)
+* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/)
 
 *Do you have any experience to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.*

website/blog/2023-05-07-1M-milestone/index.mdx

@@ -19,7 +19,7 @@ We'd also like to take the opportunity to reflect on FLAML's past achievements a
 
 ### Bring AutoML to One's Fingertips
 FLAML offers an off-the-shelf AutoML solution that enables users to quickly discover high-quality models or configurations for common ML/AI tasks. By automatically selecting models and hyperparameters for training or inference, FLAML saves users time and effort. FLAML has significantly reduced development time for developers and data scientists alike, while also providing a convenient way to integrate new algorithms into the pipeline, enabling easy extensions and large-scale parallel tuning. These features make FLAML a valuable tool in R&D efforts for many enterprise users.
-FLAML is capable of handling a variety of common ML tasks, such as [classification](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Classification), [regression](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Regression), [time series forecasting](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Time%20series%20forecast), [NLP tasks](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Rank), and [generative tasks](https://microsoft.github.io/FLAML/docs/Use-Cases/Autogen), providing a comprehensive solution for various applications.
+FLAML is capable of handling a variety of common ML tasks, such as [classification](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Classification), [regression](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Regression), [time series forecasting](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Time%20series%20forecast), [NLP tasks](https://microsoft.github.io/FLAML/docs/Examples/AutoML-Rank), and [generative tasks](https://microsoft.github.io/autogen/), providing a comprehensive solution for various applications.
 
 ### Speed and Efficiency: The FLAML Advantage
 What sets FLAML apart from other AutoML libraries is its exceptional efficiency, thanks to the economical and efficient hyperparameter optimization and model selection methods developed in our [research](https://microsoft.github.io/FLAML/docs/Research). FLAML is also capable of handling large search spaces with heterogeneous evaluation costs, complex constraints, guidance, and early stopping. The [zero-shot AutoML](https://microsoft.github.io/FLAML/docs/Use-Cases/Zero-Shot-AutoML) option further reduces the cost of AutoML, making FLAML an even more attractive solution for a wide range of applications with low resources.
@@ -37,7 +37,7 @@ We invite contributions from anyone interested in this topic and look forward to
 
 ## For Further Reading
 
-* [Documentation about `flaml.autogen`](/docs/Use-Cases/Autogen)
+* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/)
 * [Code Example: Tune chatGPT for Math Problem Solving with FLAML](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_chatgpt_gpt4.ipynb)
 
 *Do you have any experience to share about LLM applications? Do you like to see more support or research of LLMOps? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.*

website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx

@@ -144,7 +144,7 @@ An example notebook to run this experiment can be found at: https://github.com/m
 
 ## Discussion
 
-Our solution is quite simple to [implement](/docs/reference/autogen/code_utils#implement) using a generic interface offered in [`flaml.autogen`](/docs/Use-Cases/Autogen#logic-error), yet the result is quite encouraging.
+Our solution is quite simple to implement using a generic interface offered in [`flaml.autogen`](https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#logic-error), yet the result is quite encouraging.
 
 While the specific way of generating assertions is application-specific, the main ideas are general in LLM operations:
 * Generate multiple responses to select - especially useful when selecting a good response is relatively easier than generating a good response at one shot.
@@ -164,5 +164,5 @@ There are many directions of extensions in research and development:
 
 ## For Further Reading
 
-* [Documentation](/docs/Use-Cases/Autogen) about `flaml.autogen` and [Research paper](https://arxiv.org/abs/2303.04673).
+* [Documentation](https://microsoft.github.io/autogen/) about `flaml.autogen` and [Research paper](https://arxiv.org/abs/2303.04673).
 * [Blog post](/blog/2023/04/21/LLM-tuning-math) about a related study for math.

website/blog/2023-06-28-MathChat/index.mdx

@@ -89,6 +89,6 @@ Further work can be done to enhance this framework or math problem-solving in ge
 ## For Further Reading
 
 * [Research paper of MathChat](https://arxiv.org/abs/2306.01337)
-* [Documentation about `flaml.autogen`](/docs/Use-Cases/Autogen)
+* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/)
 
 *Are you working on applications that involve math problem-solving? Would you appreciate additional research or support on the application of LLM-based agents for math problem-solving? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.*

website/blog/2023-07-14-Local-LLMs/index.mdx

@@ -143,5 +143,5 @@ print(response)
 
 ## For Further Reading
 
-* [Documentation](/docs/Use-Cases/Autogen) about `flaml.autogen`
+* [Documentation](https://microsoft.github.io/autogen/) about `flaml.autogen`
 * [Documentation](https://github.com/lm-sys/FastChat) about FastChat.

website/docs/Examples/AutoGen-AgentChat.md

@@ -1,15 +1,3 @@
 # AutoGen - Automated Multi Agent Chat
 
-`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance via multi-agent conversation.
-Please find documentation about this feature [here](/docs/Use-Cases/Autogen#agents).
-
-Links to notebook examples:
-* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)
-* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_human_feedback.ipynb)
-* [Solve Tasks Requiring Web Info](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_web_info.ipynb)
-* [Use Provided Tools as Functions](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_function_call.ipynb)
-* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_planning.ipynb)
-* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_two_users.ipynb)
-* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_chess.ipynb)
-* [Automated Task Solving by Group Chat](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb)
-* [Automated Continual Learning from New Data](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_stream.ipynb)
+Please refer to https://microsoft.github.io/autogen/docs/Examples/AutoGen-AgentChat.

website/docs/Examples/AutoGen-OpenAI.md

@@ -1,7 +1,7 @@
 # AutoGen - Tune GPT Models
 
 `flaml.autogen` offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The research study finds that tuning hyperparameters can significantly improve the utility of them.
-Please find documentation about this feature [here](/docs/Use-Cases/Autogen#enhanced-inference).
+Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/#enhanced-inference).
 
 Links to notebook examples:
 * [Optimize for Code Generation](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_openai_completion.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_openai_completion.ipynb)

website/docs/Examples/Default-Flamlized.md

@@ -45,7 +45,7 @@ LGBMRegressor(colsample_bytree=0.7019911744574896,
 ### Suggest hyperparameters without training
 
 ```
-from flaml.data import load_openml_dataset
+from flaml.automl.data import load_openml_dataset
 from flaml.default import LGBMRegressor
 from flaml.ml import sklearn_metric_loss_score
 

website/docs/Getting-Started.md

@@ -20,7 +20,7 @@ Install FLAML from pip: `pip install flaml`. Find more options in [Installation]
 
 There are several ways of using flaml:
 
-#### (New) [Autogen](/docs/Use-Cases/Autogen)
+#### (New) [AutoGen](https://microsoft.github.io/autogen/)
 
 Autogen enables the next-gen GPT-X applications with a generic multi-agent conversation framework.
 It offers customizable and conversable agents which integrate LLMs, tools and human.
@@ -118,7 +118,7 @@ Then, you can use it just like you use the original `LGMBClassifier`. Your other
 
 ### Where to Go Next?
 
-* Understand the use cases for [Autogen](/docs/Use-Cases/Autogen), [Task-oriented AutoML](/docs/Use-Cases/Task-Oriented-Automl), [Tune user-defined function](/docs/Use-Cases/Tune-User-Defined-Function) and [Zero-shot AutoML](/docs/Use-Cases/Zero-Shot-AutoML).
+* Understand the use cases for [AutoGen](https://microsoft.github.io/autogen/), [Task-oriented AutoML](/docs/Use-Cases/Task-Oriented-Automl), [Tune user-defined function](/docs/Use-Cases/Tune-User-Defined-Function) and [Zero-shot AutoML](/docs/Use-Cases/Zero-Shot-AutoML).
 * Find code examples under "Examples": from [AutoGen - AgentChat](/docs/Examples/AutoGen-AgentChat) to [Tune - PyTorch](/docs/Examples/Tune-PyTorch).
 * Learn about [research](/docs/Research) around FLAML and check [blogposts](/blog).
 * Chat on [Discord](https://discord.gg/Cppx2vSPVP).

website/docs/Use-Cases/Autogen.md

@@ -1,552 +1,3 @@
-# AutoGen: Enabling Next-Gen GPT-X Applications
+# AutoGen for Large Language Models
 
-`flaml.autogen` simplifies the orchestration, automation and optimization of a complex GPT-X workflow. It maximizes the performance of GPT-X models and augments their weakness. It enables building next-gen GPT-X applications based on multi-agent conversations with minimal effort.
-
-## Features
-
-* A unified multi-agent conversation framework as a high-level abstraction of using foundation models. It offers customizable and conversable agents which integrate LLM, tool and human.
-By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code.
-* A drop-in replacement of `openai.Completion` or `openai.ChatCompletion` as an enhanced inference API. It allows easy performance tuning, utilities like API unification & caching, and advanced usage patterns, such as error handling, multi-config inference, context programming etc.
-
-The package is under active development with more features upcoming.
-
-## Agents
-
-[`flaml.autogen.agentchat`](/docs/reference/autogen/agentchat/agent) offers a multi-agent conversation framework, featuring capable, customizable and conversable agents which integrate LLM, tool and human via automated agent chat.
-
-### Basic Concept
-
-We have designed a generic `ResponsiveAgent` class for Agents that are capable of conversing with each other through the exchange of messages to jointly finish a task. An agent can communicate with other agents and perform actions. Different agents can differ in what actions they perform after receiving messages. Two representative subclasses are `AssistantAgent` and `UserProxyAgent`.
-
-- `AssistantAgent`. Designed to act as an assistant by responding to user requests. It could write Python code (in a Python coding block) for a user to execute when a message (typically a description of a task that needs to be solved) is received. Under the hood, the Python code is written by LLM (e.g., GPT-4). It can also receive the execution results and suggest code with bug fix. Its behavior can be altered by passing a new system message. The LLM [inference](#enhanced-inference) configuration can be configured via `llm_config`.
-- `UserProxyAgent`. Serves as a proxy for the human user. Upon receiving a message, the UserProxyAgent will either solicit the human user's input or prepare an automatically generated reply. The chosen action depends on the settings of the `human_input_mode` and `max_consecutive_auto_reply` when the `UserProxyAgent` instance is constructed, and whether a human user input is available.
-By default, the automatically generated reply is crafted based on automatic code execution. The `UserProxyAgent` triggers code execution automatically when it detects an executable code block in the received message and no human user input is provided. Code execution can be disabled by setting `code_execution_config` to False. LLM-based response is disabled by default. It can be enabled by setting `llm_config` to a dict corresponding to the [inference](#enhanced-inference) configuration.
-When `llm_config` is set to a dict, `UserProxyAgent` can generate replies using an LLM when code execution is not performed.
-
-The auto-reply capability of `ResponsiveAgent` allows for more autonomous multi-agent communication while retaining the possibility of human intervention.
-One can also easily extend it by registering auto_reply functions with the `register_auto_reply()` method.
-
-### Basic Example
-
-Example usage of the agents to solve a task with code:
-```python
-from flaml.autogen import AssistantAgent, UserProxyAgent
-
-# create an AssistantAgent instance named "assistant"
-assistant = AssistantAgent(name="assistant")
-
-# create a UserProxyAgent instance named "user_proxy"
-user_proxy = UserProxyAgent(
-    name="user_proxy",
-    human_input_mode="NEVER",  # in this mode, the agent will never solicit human input but always auto reply
-)
-
-# the assistant receives a message from the user, which contains the task description
-user.initiate_chat(
-    assistant,
-    message="""What date is today? Which big tech stock has the largest year-to-date gain this year? How much is the gain?""",
-)
-```
-In the example above, we create an AssistantAgent named "assistant" to serve as the assistant and a UserProxyAgent named "user_proxy" to serve as a proxy for the human user.
-1. The assistant receives a message from the user_proxy, which contains the task description.
-2. The assistant then tries to write Python code to solve the task and sends the response to the user_proxy.
-3. Once the user_proxy receives a response from the assistant, it tries to reply by either soliciting human input or preparing an automatically generated reply. In this specific example, since `human_input_mode` is set to `"NEVER"`, the user_proxy will not solicit human input but send an automatically generated reply (auto reply). More specifically, the user_proxy executes the code and uses the result as the auto-reply.
-4. The assistant then generates a further response for the user_proxy. The user_proxy can then decide whether to terminate the conversation. If not, steps 3 and 4 are repeated.
-
-Please find a visual illustration of how UserProxyAgent and AssistantAgent collaboratively solve the above task below:
-![Agent Chat Example](images/agent_example.png)
-
-### Human Input Mode
-
-The `human_input_mode` parameter of `UserProxyAgent` controls the behavior of the agent when it receives a message. It can be set to `"NEVER"`, `"ALWAYS"`, or `"TERMINATE"`.
-- Under the mode `human_input_mode="NEVER"`, the multi-turn conversation between the assistant and the user_proxy stops when the number of auto-reply reaches the upper limit specified by `max_consecutive_auto_reply` or the received message is a termination message according to `is_termination_msg`.
-- When `human_input_mode` is set to `"ALWAYS"`, the user proxy agent solicits human input every time a message is received; and the conversation stops when the human input is "exit", or when the received message is a termination message and no human input is provided.
-- When `human_input_mode` is set to `"TERMINATE"`, the user proxy agent solicits human input only when a termination message is received or the number of auto replies reaches `max_consecutive_auto_reply`.
-
-### Function Calling
-To leverage [function calling capability of OpenAI's Chat Completions API](https://openai.com/blog/function-calling-and-other-api-updates?ref=upstract.com), one can pass in a list of callable functions or class methods to `UserProxyAgent`, which corresponds to the description of functions passed to OpenAI's API.
-
-Example usage of the agents to solve a task with function calling feature:
-```python
-from flaml.autogen import AssistantAgent, UserProxyAgent
-
-# put the descriptions of functions in config to be passed to OpenAI's API
-llm_config = {
-    "model": "gpt-4-0613",
-    "functions": [
-        {
-            "name": "python",
-            "description": "run cell in ipython and return the execution result.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "cell": {
-                        "type": "string",
-                        "description": "Valid Python cell to execute.",
-                    }
-                },
-                "required": ["cell"],
-            },
-        },
-        {
-            "name": "sh",
-            "description": "run a shell script and return the execution result.",
-            "parameters": {
-                "type": "object",
-                "properties": {
-                    "script": {
-                        "type": "string",
-                        "description": "Valid shell script to execute.",
-                    }
-                },
-                "required": ["script"],
-            },
-        },
-    ],
-}
-
-# create an AssistantAgent instance named "assistant"
-chatbot = AssistantAgent("assistant", **llm_config)
-
-# create a UserProxyAgent instance named "user_proxy"
-user_proxy = UserProxyAgent(
-    "user_proxy",
-    human_input_mode="NEVER",
-)
-
-# define functions according to the function desription
-from IPython import get_ipython
-
-def exec_python(cell):
-    ipython = get_ipython()
-    result = ipython.run_cell(cell)
-    log = str(result.result)
-    if result.error_before_exec is not None:
-        log += f"\n{result.error_before_exec}"
-    if result.error_in_exec is not None:
-        log += f"\n{result.error_in_exec}"
-    return log
-
-def exec_sh(script):
-    return user_proxy.execute_code_blocks([("sh", script)])
-
-# register the functions
-user_proxy.register_function(
-    function_map={
-        "python": exec_python,
-        "sh": exec_sh,
-    }
-)
-
-# start the conversation
-user_proxy.initiate_chat(
-    chatbot,
-    message="Draw two agents chatting with each other with an example dialog.",
-)
-```
-
-### Notebook Examples
-
-*Interested in trying it yourself? Please check the following notebook examples:*
-* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)
-* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_human_feedback.ipynb)
-* [Solve Tasks Requiring Web Info](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_web_info.ipynb)
-* [Use Provided Tools as Functions](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_function_call.ipynb)
-* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_planning.ipynb)
-* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_two_users.ipynb)
-* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_chess.ipynb)
-* [Automated Task Solving by Group Chat](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb)
-* [Automated Continual Learning from New Data](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_stream.ipynb)
-
-## Enhanced Inference
-
-One can use [`flaml.autogen.Completion.create`](/docs/reference/autogen/oai/completion#create) to perform inference.
-There are a number of benefits of using `autogen` to perform inference: performance tuning, API unification, caching, error handling, multi-config inference, result filtering, templating and so on.
-
-### Tune Inference Parameters
-
-*Links to notebook examples:*
-* [Optimize for Code Generation](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_openai_completion.ipynb)
-* [Optimize for Math](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_chatgpt_gpt4.ipynb)
-
-#### Choices to optimize
-
-The cost of using foundation models for text generation is typically measured in terms of the number of tokens in the input and output combined. From the perspective of an application builder using foundation models, the use case is to maximize the utility of the generated text under an inference budget constraint (e.g., measured by the average dollar cost needed to solve a coding problem). This can be achieved by optimizing the hyperparameters of the inference,
-which can significantly affect both the utility and the cost of the generated text.
-
-The tunable hyperparameters include:
-1. model - this is a required input, specifying the model ID to use.
-1. prompt/messages - the input prompt/messages to the model, which provides the context for the text generation task.
-1. max_tokens - the maximum number of tokens (words or word pieces) to generate in the output.
-1. temperature - a value between 0 and 1 that controls the randomness of the generated text. A higher temperature will result in more random and diverse text, while a lower temperature will result in more predictable text.
-1. top_p - a value between 0 and 1 that controls the sampling probability mass for each token generation. A lower top_p value will make it more likely to generate text based on the most likely tokens, while a higher value will allow the model to explore a wider range of possible tokens.
-1. n - the number of responses to generate for a given prompt. Generating multiple responses can provide more diverse and potentially more useful output, but it also increases the cost of the request.
-1. stop - a list of strings that, when encountered in the generated text, will cause the generation to stop. This can be used to control the length or the validity of the output.
-1. presence_penalty, frequency_penalty - values that control the relative importance of the presence and frequency of certain words or phrases in the generated text.
-1. best_of - the number of responses to generate server-side when selecting the "best" (the one with the highest log probability per token) response for a given prompt.
-
-The cost and utility of text generation are intertwined with the joint effect of these hyperparameters.
-There are also complex interactions among subsets of the hyperparameters. For example,
-the temperature and top_p are not recommended to be altered from their default values together because they both control the randomness of the generated text, and changing both at the same time can result in conflicting effects; n and best_of are rarely tuned together because if the application can process multiple outputs, filtering on the server side causes unnecessary information loss; both n and max_tokens will affect the total number of tokens generated, which in turn will affect the cost of the request.
-These interactions and trade-offs make it difficult to manually determine the optimal hyperparameter settings for a given text generation task.
-
-*Do the choices matter? Check this [blogpost](/blog/2023/04/21/LLM-tuning-math) to find example tuning results about gpt-3.5-turbo and gpt-4.*
-
-
-With `flaml.autogen`, the tuning can be performed with the following information:
-1. Validation data.
-1. Evaluation function.
-1. Metric to optimize.
-1. Search space.
-1. Budgets: inference and optimization respectively.
-
-#### Validation data
-
-Collect a diverse set of instances. They can be stored in an iterable of dicts. For example, each instance dict can contain "problem" as a key and the description str of a math problem as the value; and "solution" as a key and the solution str as the value.
-
-#### Evaluation function
-
-The evaluation function should take a list of responses, and other keyword arguments corresponding to the keys in each validation data instance as input, and output a dict of metrics. For example,
-
-```python
-def eval_math_responses(responses: List[str], solution: str, **args) -> Dict:
-    # select a response from the list of responses
-    answer = voted_answer(responses)
-    # check whether the answer is correct
-    return {"success": is_equivalent(answer, solution)}
-```
-
-[`flaml.autogen.code_utils`](/docs/reference/autogen/code_utils) and [`flaml.autogen.math_utils`](/docs/reference/autogen/math_utils) offer some example evaluation functions for code generation and math problem solving.
-
-#### Metric to optimize
-
-The metric to optimize is usually an aggregated metric over all the tuning data instances. For example, users can specify "success" as the metric and "max" as the optimization mode. By default, the aggregation function is taking the average. Users can provide a customized aggregation function if needed.
-
-#### Search space
-
-Users can specify the (optional) search range for each hyperparameter.
-
-1. model. Either a constant str, or multiple choices specified by `flaml.tune.choice`.
-1. prompt/messages. Prompt is either a str or a list of strs, of the prompt templates. messages is a list of dicts or a list of lists, of the message templates.
-Each prompt/message template will be formatted with each data instance. For example, the prompt template can be:
-"{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\boxed{{}}."
-And `{problem}` will be replaced by the "problem" field of each data instance.
-1. max_tokens, n, best_of. They can be constants, or specified by `flaml.tune.randint`, `flaml.tune.qrandint`, `flaml.tune.lograndint` or `flaml.qlograndint`. By default, max_tokens is searched in [50, 1000); n is searched in [1, 100); and best_of is fixed to 1.
-1. stop. It can be a str or a list of strs, or a list of lists of strs or None. Default is None.
-1. temperature or top_p. One of them can be specified as a constant or by `flaml.tune.uniform` or `flaml.tune.loguniform` etc.
-Please don't provide both. By default, each configuration will choose either a temperature or a top_p in [0, 1] uniformly.
-1. presence_penalty, frequency_penalty. They can be constants or specified by `flaml.tune.uniform` etc. Not tuned by default.
-
-#### Budgets
-
-One can specify an inference budget and an optimization budget.
-The inference budget refers to the average inference cost per data instance.
-The optimization budget refers to the total budget allowed in the tuning process. Both are measured by dollars and follow the price per 1000 tokens.
-
-#### Perform tuning
-
-Now, you can use [`flaml.autogen.Completion.tune`](/docs/reference/autogen/oai/completion#tune) for tuning. For example,
-
-```python
-from flaml import autogen
-
-config, analysis = autogen.Completion.tune(
-    data=tune_data,
-    metric="success",
-    mode="max",
-    eval_func=eval_func,
-    inference_budget=0.05,
-    optimization_budget=3,
-    num_samples=-1,
-)
-```
-
-`num_samples` is the number of configurations to sample. -1 means unlimited (until optimization budget is exhausted).
-The returned `config` contains the optimized configuration and `analysis` contains an [ExperimentAnalysis](/docs/reference/tune/analysis#experimentanalysis-objects) object for all the tried configurations and results.
-
-The tuend config can be used to perform inference.
-
-### API unification
-
-`flaml.autogen.Completion.create` is compatible with both `openai.Completion.create` and `openai.ChatCompletion.create`, and both OpenAI API and Azure OpenAI API. So models such as "text-davinci-003", "gpt-3.5-turbo" and "gpt-4" can share a common API.
-When chat models are used and `prompt` is given as the input to `flaml.autogen.Completion.create`, the prompt will be automatically converted into `messages` to fit the chat completion API requirement. One advantage is that one can experiment with both chat and non-chat models for the same prompt in a unified API.
-
-For local LLMs, one can spin up an endpoint using a package like [simple_ai_server](https://github.com/lhenault/simpleAI) and [FastChat](https://github.com/lm-sys/FastChat), and then use the same API to send a request. See [here](/blog/2023/07/14/Local-LLMs) for examples on how to make inference with local LLMs.
-
-When only working with the chat-based models, `flaml.autogen.ChatCompletion` can be used. It also does automatic conversion from prompt to messages, if prompt is provided instead of messages.
-
-### Caching
-
-API call results are cached locally and reused when the same request is issued. This is useful when repeating or continuing experiments for reproducibility and cost saving. It still allows controlled randomness by setting the "seed", using [`set_cache`](/docs/reference/autogen/oai/completion#set_cache) or specifying in `create()`.
-
-### Error handling
-
-#### Runtime error
-
-It is easy to hit error when calling OpenAI APIs, due to connection, rate limit, or timeout. Some of the errors are transient. `flaml.autogen.Completion.create` deals with the transient errors and retries automatically. Initial request timeout, retry timeout and retry time interval can be configured via `request_timeout`, `retry_timeout` and `flaml.autogen.Completion.retry_time`.
-
-Moreover, one can pass a list of configurations of different models/endpoints to mitigate the rate limits. For example,
-
-```python
-response = autogen.Completion.create(
-    config_list=[
-        {
-            "model": "gpt-4",
-            "api_key": os.environ.get("AZURE_OPENAI_API_KEY"),
-            "api_type": "azure",
-            "api_base": os.environ.get("AZURE_OPENAI_API_BASE"),
-            "api_version": "2023-06-01-preview",
-        },
-        {
-            "model": "gpt-3.5-turbo",
-            "api_key": os.environ.get("OPENAI_API_KEY"),
-            "api_type": "open_ai",
-            "api_base": "https://api.openai.com/v1",
-            "api_version": None,
-        },
-        {
-            "model": "llama-7B",
-            "api_base": "http://127.0.0.1:8080",
-            "api_type": "open_ai",
-            "api_version": None,
-        }
-    ],
-    prompt="Hi",
-)
-```
-
-It will try querying Azure OpenAI gpt-4, OpenAI gpt-3.5-turbo, and a locally hosted llama-7B one by one, ignoring AuthenticationError, RateLimitError and Timeout,
-until a valid result is returned. This can speed up the development process where the rate limit is a bottleneck. An error will be raised if the last choice fails. So make sure the last choice in the list has the best availability.
-
-#### Logic error
-
-Another type of error is that the returned response does not satisfy a requirement. For example, if the response is required to be a valid json string, one would like to filter the responses that are not. This can be achieved by providing a list of configurations and a filter function. For example,
-
-```python
-def valid_json_filter(context, config, response):
-    for text in autogen.Completion.extract_text(response):
-        try:
-            json.loads(text)
-            return True
-        except ValueError:
-            pass
-    return False
-
-response = autogen.Completion.create(
-    config_list=[{"model": "text-ada-001"}, {"model": "gpt-3.5-turbo"}, {"model": "text-davinci-003"}],
-    prompt="How to construct a json request to Bing API to search for 'latest AI news'? Return the JSON request.",
-    filter_func=valid_json_filter,
-)
-```
-
-The example above will try to use text-ada-001, gpt-3.5-turbo, and text-davinci-003 iteratively, until a valid json string is returned or the last config is used. One can also repeat the same model in the list for multiple times to try one model multiple times for increasing the robustness of the final response.
-
-*Advanced use case: Check this [blogpost](/blog/2023/05/18/GPT-adaptive-humaneval) to find how to improve GPT-4's coding performance from 68% to 90% while reducing the inference cost.*
-
-### Templating
-
-If the provided prompt or message is a template, it will be automatically materialized with a given context. For example,
-
-```python
-response = autogen.Completion.create(
-    context={"problem": "How many positive integers, not exceeding 100, are multiples of 2 or 3 but not 4?"},
-    prompt="{problem} Solve the problem carefully.",
-    allow_format_str_template=True,
-    **config
-)
-```
-
-A template is either a format str, like the example above, or a function which produces a str from several input fields, like the example below.
-
-```python
-def content(turn, context):
-    return "\n".join(
-        [
-            context[f"user_message_{turn}"],
-            context[f"external_info_{turn}"]
-        ]
-    )
-
-messages = [
-    {
-        "role": "system",
-        "content": "You are a teaching assistant of math.",
-    },
-    {
-        "role": "user",
-        "content": partial(content, turn=0),
-    },
-]
-context = {
-    "user_message_0": "Could you explain the solution to Problem 1?",
-    "external_info_0": "Problem 1: ...",
-}
-
-response = autogen.ChatCompletion.create(context, messages=messages, **config)
-messages.append(
-    {
-        "role": "assistant",
-        "content": autogen.ChatCompletion.extract_text(response)[0]
-    }
-)
-messages.append(
-    {
-        "role": "user",
-        "content": partial(content, turn=1),
-    },
-)
-context.append(
-    {
-        "user_message_1": "Why can't we apply Theorem 1 to Equation (2)?",
-        "external_info_1": "Theorem 1: ...",
-    }
-)
-response = autogen.ChatCompletion.create(context, messages=messages, **config)
-```
-
-### Logging (Experimental)
-
-When debugging or diagnosing an LLM-based system, it is often convenient to log the API calls and analyze them. `flaml.autogen.Completion` and `flaml.autogen.ChatCompletion` offer an easy way to collect the API call histories. For example, to log the chat histories, simply run:
-```python
-flaml.autogen.ChatCompletion.start_logging()
-```
-The API calls made after this will be automatically logged. They can be retrieved at any time by:
-```python
-flaml.autogen.ChatCompletion.logged_history
-```
-To stop logging, use
-```python
-flaml.autogen.ChatCompletion.stop_logging()
-```
-If one would like to append the history to an existing dict, pass the dict like:
-```python
-flaml.autogen.ChatCompletion.start_logging(history_dict=existing_history_dict)
-```
-By default, the counter of API calls will be reset at `start_logging()`. If no reset is desired, set `reset_counter=False`.
-
-There are two types of logging formats: compact logging and individual API call logging. The default format is compact.
-Set `compact=False` in `start_logging()` to switch.
-
-* Example of a history dict with compact logging.
-```python
-{
-    """
-    [
-        {
-            'role': 'system',
-            'content': system_message,
-        },
-        {
-            'role': 'user',
-            'content': user_message_1,
-        },
-        {
-            'role': 'assistant',
-            'content': assistant_message_1,
-        },
-        {
-            'role': 'user',
-            'content': user_message_2,
-        },
-        {
-            'role': 'assistant',
-            'content': assistant_message_2,
-        },
-    ]""": {
-        "created_at": [0, 1],
-        "cost": [0.1, 0.2],
-    }
-}
-```
-
-* Example of a history dict with individual API call logging.
-```python
-{
-    0: {
-        "request": {
-            "messages": [
-                {
-                    "role": "system",
-                    "content": system_message,
-                },
-                {
-                    "role": "user",
-                    "content": user_message_1,
-                }
-            ],
-            ... # other parameters in the request
-        },
-        "response": {
-            "choices": [
-                "messages": {
-                    "role": "assistant",
-                    "content": assistant_message_1,
-                },
-            ],
-            ... # other fields in the response
-        }
-    },
-    1: {
-        "request": {
-            "messages": [
-                {
-                    "role": "system",
-                    "content": system_message,
-                },
-                {
-                    "role": "user",
-                    "content": user_message_1,
-                },
-                {
-                    "role": "assistant",
-                    "content": assistant_message_1,
-                },
-                {
-                    "role": "user",
-                    "content": user_message_2,
-                },
-            ],
-            ... # other parameters in the request
-        },
-        "response": {
-            "choices": [
-                "messages": {
-                    "role": "assistant",
-                    "content": assistant_message_2,
-                },
-            ],
-            ... # other fields in the response
-        }
-    },
-}
-```
-It can be seen that the individual API call history contains redundant information of the conversation. For a long conversation the degree of redundancy is high.
-The compact history is more efficient and the individual API call history contains more details.
-
-### Other Utilities
-
-- a [`cost`](/docs/reference/autogen/oai/completion#cost) function to calculate the cost of an API call.
-- a [`test`](/docs/reference/autogen/oai/completion#test) function to conveniently evaluate the configuration over test data.
-- an [`extract_text_or_function_call`](/docs/reference/autogen/oai/completion#extract_text_or_function_call) function to extract the text or function call from a completion or chat response.
-
-
-## Utilities for Applications
-
-### Code
-
-[`flaml.autogen.code_utils`](/docs/reference/autogen/code_utils) offers code-related utilities, such as:
-- a [`improve_code`](/docs/reference/autogen/code_utils#improve_code) function to improve code for a given objective.
-- a [`generate_assertions`](/docs/reference/autogen/code_utils#generate_assertions) function to generate assertion statements from function signature and docstr.
-- a [`implement`](/docs/reference/autogen/code_utils#implement) function to implement a function from a definition.
-- a [`eval_function_completions`](/docs/reference/autogen/code_utils#eval_function_completions) function to evaluate the success of a function completion task, or select a response from a list of responses using generated assertions.
-
-### Math
-
-[`flaml.autogen.math_utils`](/docs/reference/autogen/math_utils) offers utilities for math problems, such as:
-- a [eval_math_responses](/docs/reference/autogen/math_utils#eval_math_responses) function to select a response using voting, and check if the final answer is correct if the canonical solution is provided.
-
-## For Further Reading
-
-*Interested in the research that leads to this package? Please check the following papers.*
-* [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673). Chi Wang, Susan Xueqing Liu, Ahmed H. Awadallah. ArXiv preprint arXiv:2303.04673 (2023).
-* [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337). Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2306.01337 (2023).
+Please refer to https://microsoft.github.io/autogen/.

website/docs/Use-Cases/Task-Oriented-AutoML.md

@@ -135,7 +135,7 @@ The estimator list can contain one or more estimator names, each corresponding t
 #### Guidelines on tuning a custom estimator
 
 To tune a custom estimator that is not built-in, you need to:
-1. Build a custom estimator by inheritting [`flaml.model.BaseEstimator`](/docs/reference/automl/model#baseestimator-objects) or a derived class.
+1. Build a custom estimator by inheritting [`flaml.automl.model.BaseEstimator`](/docs/reference/automl/model#baseestimator-objects) or a derived class.
 For example, if you have a estimator class with scikit-learn style `fit()` and `predict()` functions, you only need to set `self.estimator_class` to be that class in your constructor.
 
 ```python
@@ -177,7 +177,7 @@ class MyRegularizedGreedyForest(SKLearnEstimator):
         return space
 ```
 
-In the constructor, we set `self.estimator_class` as `RGFClassifier` or `RGFRegressor` according to the task type. If the estimator you want to tune does not have a scikit-learn style `fit()` and `predict()` API, you can override the `fit()` and `predict()` function of `flaml.model.BaseEstimator`, like [XGBoostEstimator](/docs/reference/automl/model#xgboostestimator-objects). Importantly, we also add the `task="binary"` parameter in the signature of `__init__` so that it doesn't get grouped together with the `**config` kwargs that determines the parameters with which the underlying estimator (`self.estimator_class`) is constructed. If your estimator doesn't use one of the parameters that it is passed, for example some regressors in `scikit-learn` don't use the `n_jobs` parameter, it is enough to add `n_jobs=None` to the signature so that it is ignored by the `**config` dict.
+In the constructor, we set `self.estimator_class` as `RGFClassifier` or `RGFRegressor` according to the task type. If the estimator you want to tune does not have a scikit-learn style `fit()` and `predict()` API, you can override the `fit()` and `predict()` function of `flaml.automl.model.BaseEstimator`, like [XGBoostEstimator](/docs/reference/automl/model#xgboostestimator-objects). Importantly, we also add the `task="binary"` parameter in the signature of `__init__` so that it doesn't get grouped together with the `**config` kwargs that determines the parameters with which the underlying estimator (`self.estimator_class`) is constructed. If your estimator doesn't use one of the parameters that it is passed, for example some regressors in `scikit-learn` don't use the `n_jobs` parameter, it is enough to add `n_jobs=None` to the signature so that it is ignored by the `**config` dict.
 
 2. Give the custom estimator a name and add it in AutoML. E.g.,
 
@@ -524,10 +524,10 @@ The best model can be obtained by the `model` property of an `AutoML` instance.
 ```python
 automl.fit(X_train, y_train, task="regression")
 print(automl.model)
-# <flaml.model.LGBMEstimator object at 0x7f9b502c4550>
+# <flaml.automl.model.LGBMEstimator object at 0x7f9b502c4550>
 ```
 
-[`flaml.model.LGBMEstimator`](/docs/reference/automl/model#lgbmestimator-objects) is a wrapper class for LightGBM models. To access the underlying model, use the `estimator` property of the `flaml.model.LGBMEstimator` instance.
+[`flaml.automl.model.LGBMEstimator`](/docs/reference/automl/model#lgbmestimator-objects) is a wrapper class for LightGBM models. To access the underlying model, use the `estimator` property of the `flaml.automl.model.LGBMEstimator` instance.
 
 ```python
 print(automl.model.estimator)