LLMs Will Be A Top Consumer of APIs, But We’re in the Early Days
Some initial thoughts from recent DeepLearning.AI courses on interacting with Large Language Models
2024 update: In this article I talk about using generative AI to consume APIs. If you’d like to learn to build a Python API and consume it with generative AI, I’m writing a book 📕 with O’Reilly about that 🚀. It is called Hands-On APIs for AI and Data Science, and the early release is out now with the first five chapters. Take a look!
The overlap between data science and APIs is an interesting one, but has tended to be a one-way glass: data scientists use APIs all the time, but API practitioners haven’t had much exposure to data science.
But the machine learning models data scientists developed are proving their value, and application developers have taken notice. Applications are calling machine-learning-powered APIs for chat agents, customer recommendations, and other capabilities. The biggest drivers, no surprise, are the generative AI applications powered by Large Language Models (LLMs) such as the OpenAI GPT models, LLaMA, and others.
When AI Calls You Back
We’ve just about come full circle now, as the industry works to give LLM-powered applications access to their APIs for a variety of purposes:
- To solve the “knowledge cutoff” problem, when a model is clueless about anything that happened since it was trained. (Which leads to entertaining social media fails.)
- To allow the models to look up information from current sources or securely access enterprise sources.
- To enable the models to DO something (like place a customer order) instead of just chatting about it.
Recent Training Courses
To understand where we stand today on this topic, I recently (Summer 2023) took a couple of very useful introductory courses about LLMs that were produced by DeepLearning.ai:
- ChatGPT Prompt Engineering for Developers — This was the course I needed to get me past the ChatGPT headlines and begin understanding the central role of the Prompt/Completion loop.
- Generative AI with LLMs on AWS — Very thorough coverage of the work in the trenches to put LLMs to real use in enterprise settings. Heavy focus on addressing practical issues such as cost, performance, and bias. Most relevant to this article was the discussion of using LLMs in real-world applications that need to access external sources.
A Few Quick Takeaways from The Courses (with a special focus on APIs)
Start Preparing Your APIs for LLM Consumers
Although this area of the field is brand new (more on that below), major effort is clearly being focused on giving the models access to external resources.
API providers may compete for market share by positioning their APIs for access by LLM-powered apps. This could be a profitable avenue for API providers, assuming they have a business plan for their APIs.
The default underlying method for this will be calling APIs that are described by an OpenAPI specification, which is good news since those are already the industry defaults. Additional proactive steps, such as creating vendor-specific plugins will also be useful for broader adoption.
For API-interested readers, here are a couple of areas of research to keep an eye on (and my initial understanding of them):
- ReAct — A method for combining reasoning and actions, to allow LLMs to access external data.
- LangChain — A framework for creating applications that are data-aware using LLMs.
- Agents — A method of teeing up actions for LLMs to take.
- Plugins — Vendor-specific methods to add your APIs to the marketplace or ecosystem of the major models. For example ChatGPT plugins [March 2024 note: Plugins were replaced by Actions].
The Prompt is Everything…For Now
These courses together hammered home the fact that currently to interact with LLMs, the only door is the Prompt and Completion. In a ChatGPT context, you can think of the prompt as the questions that you ask and the completion as the answers that you get back.
However, the capabilities of these LLMs are unlocked by passing very carefully structured prompts that explain the context of what you’re looking for, specify the format it should be returned in, and provide examples of what you mean.
What surprised me in the second course is that so many of the advanced capabilities that are being added to the LLMs such as accessing external APIs and performing step-by-step reasoning are bolted onto the LLM and somewhat artificially fed to the model in that same prompt text. It has a somewhat Frankenstein feel to it, in the way that all of this complicated work is being done and fed to the black box in a limited text string.
Communicating to the model only through the prompt seems like a bandaid solution; surely we’ll be interacting with the models in a more traditional structured fashion in the near future.
How Comfortable Are You Analyzing Research Papers?
In IT, we are used to a cascade of acronyms, so a few more (LORA, PAL, ReAct, RLHF) are no sweat. What’s new to me is how many of these acronyms come from very recent research papers. In the second course, almost all of the techniques described were initial implementations of research that had been published within the last year or two.
Of course, citing research papers gives an intellectual flavor to the field (and is a big reason for the “science” in data science). However, I’m not sure how prepared IT and business professionals are to sift through the world of academic papers and base their core systems on them.
One particular source of concern is that very little of this research is reproducible, increasing the risk that many of these techniques will hit a dead end before solving the problem they’re targeting. This will likely lead to frustration from enterprise IT, and a loss of trust.
You Really Need To Learn Python
In the software development world, there is never a shortage of new programming languages to learn. But it’s clear from every training that you attend and every demo that is provided — Python is the default language that AI tooling and techniques are developed and demonstrated in. Even an introductory course like the Prompt Engineering course above is implemented as a Jupyter notebook using Python code. If you’ve been waiting to get your hands dirty in Python, wait no more.
AI Usage Statement: I used online grammar software to edit the content and improve the style of this article.
