Engineering

Self-Consistency ⚖️

Improve LLM performance on complex arithmetic and commonsense reasoning by sampling multiple, diverse reasoning paths and selecting the most consistent answer.

Driptanil DattaSoftware Developer

Mar 20266 min read

Introduction

Perhaps one of the more advanced techniques for prompt engineering is Self-Consistency. Proposed by Wang et al. (2022) (opens in a new tab), self-consistency aims "to replace the naive greedy decoding used in chain-of-thought prompting."

The core idea is to sample multiple, diverse reasoning paths through few-shot CoT and use those generations to select the most consistent answer. This significantly boosts performance on tasks involving arithmetic and commonsense reasoning.

The "Sister's Age" Problem

Let's look at a task where standard reasoning often fails due to simple arithmetic lapses.

Improving with Self-Consistency

To solve this using self-consistency, we first provide several few-shot CoT exemplars (like the ones from Table 17 in Wang et al. 2022) to prime the model for reasoning. We then generate multiple outputs for the same question.

The Majority Vote

While some reasoning paths might still lead to incorrect results (like Output 3), by taking a majority vote of the final answers (67 appearing twice vs. 35 once), the model can arrive at the objectively correct answer.

This works because there are usually many ways to reason correctly, but only a few ways to reason incorrectly in a way that arrives at the same wrong answer.

Why it Works

• Replacement of Greedy Decoding: Standard "greedy" decoding picks the most likely next token at every step, which can lock the model into an early reasoning error.
• Diversity as a Filter: By sampling diverse paths, the model "filters" out random hallucinations or logic slips through statistical consensus.
• Complementary to CoT: Self-consistency doesn't replace Chain-of-Thought; it enhances it by providing a robust decision-making layer on top of the reasoning tokens.