++++Learn how to offload complex logic and mathematical reasoning from the LLM to a programmatic runtime like Python, ensuring higher reliability and precision.
Program-Aided Language Models (PAL) 🐍
This content is adapted from Prompting Guide: PAL. It has been curated and organized for educational purposes on this portfolio. No copyright infringement is intended.
Introduction
While Chain-of-Thought (CoT) prompting allows LLMs to perform reasoning, they often struggle with precise mathematical calculations or complex logic. Program-Aided Language Models (PAL), proposed by Gao et al. (2022) (opens in a new tab), solves this by using the LLM to generate code as intermediate reasoning steps.
The final execution is then offloaded to a programmatic runtime, such as a Python interpreter, rather than relying on the model's "mental math."
Image Source: Gao et al. (2022)
PAL vs. Chain-of-Thought
In CoT, the model generates free-form text that leads to a solution. In PAL, the model generates a Python script. This shift ensures:
- Precision: Algorithmic calculations are 100% accurate.
- Redundancy: The reasoning logic is codified and can be easily audited or reused.
- Complexity: Tasks involving date arithmetic or complex symbolic logic become trivial.
Example: Date Understanding
One of PAL's strongest use cases is interpreting and calculating dates. Here's how it works using LangChain and OpenAI:
The Prompt
We provide the model with a few exemplars where reasoning is mapped to Python's datetime logic:
# Q: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
# If 2015 is coming in 36 hours, then today is 36 hours before.
today = datetime(2015, 1, 1) - relativedelta(hours=36)
# One week from today,
one_week_from_today = today + relativedelta(weeks=1)
# The answer formatted with %m/%d/%Y is
one_week_from_today.strftime('%m/%d/%Y')The Output
When asked: "Today is 27 February 2023. I was born exactly 25 years ago. What is the date I was born in MM/DD/YYYY?", PAL generates:
# If today is 27 February 2023 and I was born 25 years ago, then I was born 25 years before.
today = datetime(2023, 2, 27)
# I was born 25 years before,
born = today - relativedelta(years=25)
# The answer formatted with %m/%d/%Y is
born.strftime('%m/%d/%Y')Executing this code yields the precise answer: 02/27/1998.
Tools Matter: PAL is a precursor to modern "Code Interpreter" features found in models like GPT-4. By giving the model a playground to execute code, we turn a "fuzzy" predictor into a reliable logic engine.
[!TIP] PAL is particularly powerful when combined with Prompt Chaining. Use the LLM to structure the logic, and use Python to execute the math.