How to Load S3 Data Into Databricks With Unity Catalog

Short answer: Create an external location in Unity Catalog, grant a storage credential to an S3 bucket path, then read the files with spark.read or COPY INTO. You need IAM permissions on the AWS side and CREATE EXTERNAL LOCATION on the Databricks side. Budget about 30 minutes the first time.

Most of the pain here is not Databricks. It is the IAM role trust policy, which fails silently if you get the external ID wrong.

What is Unity Catalog’s external location model?

Unity Catalog separates storage credentials (an IAM role Databricks can assume) from external locations (a bucket path scoped to users and groups). That split is what lets a data engineer read s3://acme-raw/sales/ without owning the AWS access key. Databricks manages the assume-role handshake.

If you are on the free edition, note that external S3 access requires a paid workspace or a trial with attachments enabled. Free edition limits outbound storage setup.

How do you set up the AWS side?

1. In AWS IAM, create a role that trusts Databricks’ account principal, and paste the external ID Databricks shows you. This string is tied to your metastore — reusing a role across workspaces with a mismatched ID is the most common failure.
2. Attach a least-privilege policy scoped to the bucket prefix:

{"Effect":"Allow","Action":["s3:GetObject","s3:GetObjectVersion","s3:ListBucket","s3:GetBucketLocation"],"Resource":["arn:aws:s3:::acme-raw","arn:aws:s3:::acme-raw/sales/*"]}

3. Copy the role ARN. Note the bucket region while you are there; a region mismatch adds latency and, for some actions, hard errors.

How do you create the credential and external location?

In the Databricks SQL editor or a notebook cell:

CREATE STORAGE CREDENTIAL acme_s3
  WITH AWS_IAM_ROLE = 'arn:aws:iam::111122223333:role/databricks-s3';

CREATE EXTERNAL LOCATION acme_sales
  URL 's3://acme-raw/sales/'
  WITH (STORAGE CREDENTIAL acme_s3);

Then grant access — nobody, including you, can read the path until a grant exists:

GRANT READ FILES ON EXTERNAL LOCATION acme_sales TO `data-engineers`;

Run LIST 's3://acme-raw/sales/' to confirm. If it returns zero rows, the grant is missing or the IAM policy lacks ListBucket.

Should you use COPY INTO or a direct read?

Depends on volume and whether files land incrementally.

Approach	Good for	Cost profile
spark.read	one-off exploration, small to mid files	full scan each run
COPY INTO	incremental ingestion, daily file drops	retries only new files
Auto Loader	streaming ingestion, thousands of files	tracks state in checkpoint

For a nightly drop of 200 CSV files, COPY INTO is the right default. It tracks which files it already loaded, so a rerun does not duplicate rows. The tradeoff: it needs a target Delta table defined first, and schema drift requires mergeSchema options.

COPY INTO main.sales.orders
FROM 's3://acme-raw/sales/orders/'
FILEFORMAT = CSV
FORMAT_OPTIONS ('header' = 'true', 'inferSchema' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');

What breaks most often?

• External ID mismatch. Error reads like a permissions problem but the trust policy is the culprit.
• Missing s3:ListBucket. Individual object reads work while listing fails, which makes the bug confusing.
• Region mismatch. Multi-region buckets cost more and occasionally time out.
• Partition columns in the path untyped. Add PARTITIONED BY or your queries scan the whole prefix.

Also worth saying: Databricks bills serverless SQL compute by the second, but a badly partitioned table turns a 5-second query into a 40-second one. Costs follow query shape, not file size.

Who should bother with Unity Catalog?

Teams with more than a couple of engineers, or anyone who needs column-level audit trails. A solo analyst reading one Parquet file does not need a storage credential abstraction; a direct spark.read.parquet with an instance profile is faster to set up and easier to reason about.

The honest downside: governance setup is front-loaded. You spend an afternoon on IAM before you see a single row, and if your org rotates roles quarterly, you repeat part of it.

Quick Summary

• Prerequisites: an IAM role with a Databricks external ID, a bucket policy, and CREATE EXTERNAL LOCATION.
• Two objects to create: storage credential, then external location.
• Grant is mandatory: GRANT READ FILES before any query works.
• Incremental loads: COPY INTO avoids duplicate rows on rerun.
• Not for: free-edition workspaces or one-off small-file reads.

Last updated October 2026.