Teams of humans and AI agents will be the model for the organization of the future.

by Tom Evslin

There are many tasks which Artificial Intelligence (AI) can’t perform well without help and supervision; it has an unfortunate tendency to hallucinate (make up an answer). For that matter, there are many tasks humans can’t perform well without help or supervision and humans are expensive and in short supply, at least in the work force. Two techniques have been developed in the last year of experimentation with AI and Large Language Models (LLMs) like ChatGPT: multi-agent collaboration and human-in-the-loop processes. Both help mitigate hallucination and produce content faster and more cheaply than humans can alone.

Multi-Agent Collaboration

An agent is nothing more than a packaged use of an LLM. The simplest agents accept a human prompt, ask AI to respond, and return the result. More complex agents retrieve information the LLM has not been trained on from a database or from a web search and send that information along with the request to the LLM. Agents can ask the LLM to write code in order to generate a response to the human prompt. For example, if the human wants a graph of the Dow Jones Average (DJA) for the next month compared to the S&P, several agents might be involved. The LLM will not have been trained on data as recent as last month, and so one agent retrieves the daily closing prices of both the DJA and the S&P from the web using the general knowledge of the LLM to formulate the web requests. A second agent writes computer code necessary to produce a graph of the two indices on the same set of axes. A third agent executes the code and the resulting graph is displayed to the user.

Using multiple agents is also a good way to reduce hallucination. Agent One (the writer) generates a draft answer to a question; Agent Two (the fact-checker) does web searches to validate assertions in the draft; Agent Three is a reflection agent which critiques the draft both using the fact-checking results and looking for bad style or bad usage. The answer then goes back to Agent One for revision and back though the gauntlet of agents Two and Three until all are satisfied. The final answer is sent to the human who asked the question. Interestingly (for technical reasons I won’t explain here) even if all agents use the same LLM, they can disagree constructively. However, using different LLMs for the different agents makes the whole process even more robust and reliable.

Human-in-the-Loop

Suppose the writer, fact-checker, and reflection agents above never reach agreement. Drafts could go round and round forever chewing up expensive LLM cycles. One solution is an arbitrator agent. Another solution is to add a human into the loop. For example, after the first draft has been written, fact-checked, and critiqued, a human is given an opportunity to redraft herself, to modify or accept the critique, and to arbitrate the fact-checking. Depending on what the human decides, the article could go another round and then come back to the human again. The process ends when the human accepts a draft, perhaps after making some more modifications. In more complex processes, humans can be involved in any step the application designer feels is helpful.

A Demonstration

I programmed a demonstration application to write a news story using either the minutes or a transcript of a meeting as a source. With the shrinkage of local news outlets, there are never enough reporters to attend all the public meetings which affect our lives and should influence our votes. AI and LLMs alone can’t create a well-written news story accurately enough. But human editors working with AI can turn out stories which are well-written and accurate… and cheap enough for a local newsroom to afford.

First the human editor specifies how many words the article should be and whether the source is on the internet or the editor’s computer using the screen below.

In the next screen (not shown) the editor gives a URL or selects a local file as the source. An input agent obtains the source document and extracts text from it. The writer agent uses the transcript or minutes to draft an article. The reflection agent does double duty: fact-checking against the source document and checking style. The human editor can then edit either the article or the critique or accept the article as written using the dialog pictured below. If there is a human or agent critique, the cycle repeats with a rewrite.

The source document I used for this article was the draft minutes of a Stowe Select Board meeting. The reflection agent criticized the first draft for not starting out with the most important facts (“burying the lede” in news jargon), for not clearly explaining the disagreement among the Board Members, and from being sloppy with some facts. The second draft was better in these respects and was accepted by the reflection agent. However, I (playing editor) felt it was still not clear in one respect and wrote my own critique. The third draft passed muster. The whole dialog is down below but it’s long.

You can play with this demo app at https://meeting-reporter.streamlit.app/. I don’t charge anything for using it, but you do need to provide your own OpenAI API key to get past the first screen because the LLM being used is ChatGPT4. OpenAI will only charge you a few cents per use, but you do need an account and a paid API Key. If you don’t know what an API key is, you are like most of the world and you don’t want or need one, at least not yet.

Note to nerds: if you want to see or play with the source code, it is open source and available at https://github.com/tevslin/meeting-reporter. It uses Langgraph and Streamlit libraries with a tkinter version available and is hosted in the Streamlit cloud. The LLM is ChatGPT4-Turbo.

The author, an author, entrepreneur, former Vermont state cabinet officer, lives in Stowe. He founded NG Advantage, a natural gas truck delivery company. This commentary is republished with permission from his blog, Fractals of Change.