Architecture

Overview

Snowflake calls its shape a new architecture for data warehousing: multi-cluster, shared data, in the cloud. It is built from the two traditional architectures it replaces — taking the advantage of each and dropping the drawback — so the quickest way to understand it is to understand what came before.

Three layers, and the whole page is about them: cloud services, compute (virtual warehouses) and storage.

The two traditional architectures

Shared disk

An architecture used in distributed computing in which nodes share the same disk device but each node has its own private memory. All the data sits in one disk, and different nodes reach the same data from that single disk.

The advantage is consistency: any update to the data on that disk is immediately visible to every node reading it. The challenge is contention — while one node is accessing the disk, other nodes can be left queuing.

Shared nothing

An architecture in which each node is independent and the nodes are interconnected by a network. Every node is a processor, main memory and a disk of its own.

The advantage is independence: every node reads its own disk without waiting for anyone. The drawback is that the data has to be maintained in several places at once — an update landing on the first disk takes time to reach the second and the third, because the storage is decentralised.

Read the last two rows together and the trade is obvious: one buys consistency with contention, the other buys concurrency with staleness. Both are single-cluster.

What Snowflake did with them

It took the benefit of each and built multi-cluster, shared data: centralised, scale-out storage with multiple independent compute clusters.

There is one central place the data lives, as in shared disk — so an update is visible everywhere and nothing has to be synchronised between copies. And the compute clusters reading it are independent of one another, as in shared nothing — so no processor waits in a queue because another one is reading. The contention that shared disk paid for consistency is gone, and the drift that shared nothing paid for concurrency is gone with it.

shared disk        shared nothing        multi-cluster, shared data

 [n] [n] [n]        [n] [n] [n]            [c] [c] [c]   independent
  \   |   /          |   |   |              \   |   /     compute
   \  |  /          (d) (d) (d)              \  |  /
    [ disk ]         3 copies                [  one store  ]

 one store,         no contention,         one store AND
 contention         copies drift           no contention

The three layers

The separation is the point — it is what makes instant, automatic scalability and elasticity possible, because compute can grow or shrink without the data moving.

Cloud services: the brain

It coordinates activities across Snowflake and runs on compute instances Snowflake provisions itself — so there is no infrastructure here for you to manage. Every query passes through it before it reaches storage, and it holds the metadata for every object in the account.

Management

Centralised management of all the storage, and of all the compute that works with that storage — every virtual warehouse you build is coordinated from here.

Availability

Built-in continuous availability, so the data is reachable 24x7 across zones. Snowflake transparently synchronises data across availability zones that have geographic separation and separate power grids. Updates and patches are fully online, and the whole layer is fully managed by Snowflake.

Optimisation

A SQL optimiser: code you write goes through this layer first, is optimised against the standard, and only then runs against the storage layer. It also gathers statistics automatically, so queries return faster without anyone maintaining them.

Transactions

Every create, read, update and delete against a database goes through here. It is fully ACID — atomic, consistent, isolated, durable — and it manages concurrency when many users are running queries at once.

Security

• Authentication — without proper credentials, this layer is what stops a login reaching the account
• Access control for users and roles — every user and role you create is managed here
• Encryption and key management

Metadata

It holds the metadata for every object — and, as the storage section below shows, for every micro-partition. That store is what turns a query into a partition it can skip.

Storage: hybrid columnar

• Hybrid columnar layout
• Automatic micro-partitioning
• Natural data clustering and optimisation
• Native semi-structured data support and optimisation
• All storage within Snowflake is billable, and billed compressed

Micro-partitions

Contiguous units of storage that hold table data — the physical data files that make up Snowflake's logical tables.

The one-line version: immutability is why time travel exists. Because a write never overwrites a partition, the old version is still there to be read — which is what lets you retrieve data you have lost, as long as retention still covers it.

Worked example: six rows, two partitions

A table of six rows is written to two micro-partitions — the first three rows in one, the second three in the other:

table                    micro-partition files        partition metadata

ID  Name                 P1  ID   1     2     3       ID:   1 - 3
1   John        ─────▶        Name John  Scott Mary    Name: J - S
2   Scott
3   Mary                  P2  ID   4     5     6       ID:   4 - 6
4   Jane                      Name Jane  Jack  Claire  Name: C - J
5   Jack
6   Claire

Now run a query with WHERE id = 3. Snowflake reads the partition metadata, sees that ID 3 falls inside P1's range and outside P2's, goes straight to P1 and never scans P2. That is the whole mechanism behind "self-tuning": pruning, driven by metadata the services layer already holds.

What the metadata records

Snowflake collects and maintains this automatically, at two levels.

The MIN/MAX range per column is the field pruning actually reads. Everything else is what makes a row count or a table size return instantly, without touching the data.

Compute: virtual warehouses

A virtual warehouse is a named wrapper around a cluster of servers with CPU, memory and SSD in the cloud. Any job that needs compute runs on one, and while it runs it consumes credits — this is the layer you are charged for.

T-shirt sizing

Sizes run from Extra Small up to 6X-Large. Extra Small is one server per cluster, and each size up doubles — more servers in the cluster, more CPU, more memory. A complex query that will not run on a small warehouse runs on a larger one, faster.

The doubling continues to 5X and 6X-Large. The same query for the same hour costs 1 credit on X-Small and 128 on 4X-Large — the walkthrough this page comes from treats a credit as roughly a dollar to make the point land, but the real rate is a commercial number, not an architectural one.

How the billing clock works

Credits are charged on the number of virtual warehouses you use, their size, and how long you run them. Usage is billed per second, with a one-minute minimum: the first execution is charged a full 60 seconds, and after that it is per-second for as long as queries are running.

Standard and multi-cluster warehouses

Scaling up and scaling out

Two different problems with two different answers, and confusing them is how a bill grows without a query getting faster.

Workload segmentation

Because compute is separate from storage, one set of data can be read by several warehouses at once — each sized for its own job. Warehouses should reflect units of workload management: ETL, BI and dashboards, data science.

                      one central store: Prod

  Reporting          Interactive        Continuous       ETL &
  (segmented)        dashboard          loading          maintenance
  2X-Large           X-Large x5         4 TB/day         Large
                     (auto-scale)       Medium, <5min SLA

  each is its own warehouse, on the same data,
  so one team's work never slows another's

That last line is the reason to do it. The data arrives from a single location, and the processes on top of it are sized and scaled independently — a heavy overnight ETL cannot make a dashboard slow.

Why the architecture shows up in a migration

Four of the mechanisms on this page are the reason the platform's findings about Snowflake read the way they do.

Where this page comes from

The architecture, the three layers, the micro-partition numbers, the sizing table and the workload example are Snowflake's own material, taken from an introductory walkthrough of them. The credit figures are the walkthrough's illustration of the doubling rule, not a price list.

The last section is not from that source — it is this platform's reading of which architectural facts change a migration.

See also

• Product Overview — what the product is, and the six features this architecture delivers.
• Editions — which tier gates multi-cluster warehouses and 90-day time travel.
• Snowflake — the capability matrix and DDL profile.