Andrew Mercer
on this page

What it is

TerraDS is a public dataset of Terraform programs mined from GitHub, published by researchers at the University of St. Gallen and armasuisse (Christoph Bühler, David Spielmann, Roland Meier, Guido Salvaneschi) and presented at MSR 2025 (the Mining Software Repositories conference, Data and Tool Showcase track). Before it, there was no large, reusable corpus of real-world HCL for studying how people actually write Terraform.

It isn't a tool you run in a pipeline. It's data: a SQLite database of metadata plus an archive of the HCL itself, which you query and analyse.

Version 1 on Zenodo (DOI 10.5281/zenodo.14217386, November 2024):

Count
Repositories (permissively licensed) 62,406
Terraform modules (directories with .tf files) 279,344
Resources 1,773,991
… managed (resource) 1,484,185
… data sources (data) 289,806

The MSR paper reports slightly different totals (67,360 repositories in the abstract), and Zenodo notes a newer version exists. Check Zenodo for the latest version and quote numbers from the version you use.

Licence: the dataset is CC BY 4.0. Each included repository has its own permissive licence (MIT, Apache-2.0, BSD and similar), recorded in the database; respect those if you redistribute code from it.

What's in the download

File Size (v1) Contents
TerraDS.sqlite ~434 MB Metadata for repositories, modules and resources
TerraDS.tar.gz ~351 MB The HCL source of every repository (non-Terraform files stripped)
HCL Dataset Tools.zip ~66 kB Snapshot of the tools that built the dataset
mkdir -p ~/data/terrads && cd ~/data/terrads
curl -L -o TerraDS.sqlite "https://zenodo.org/records/14217386/files/TerraDS.sqlite?download=1"
curl -L -o TerraDS.tar.gz "https://zenodo.org/records/14217386/files/TerraDS.tar.gz?download=1"
md5sum TerraDS.sqlite TerraDS.tar.gz
# expected (v1): 92e2fffd8208b2a290204555ec500977  TerraDS.sqlite
#                664716c1dd690b14add572cd5a41f192  TerraDS.tar.gz

Treat it like any downloaded archive: extract into its own directory, and don't run anything you find inside it. It's third-party code.

Database schema

From the tools' source (src/Database/Models and the Go analyzer):

Repositories                     Modules                          Resources
────────────                     ───────                          ─────────
Id              INTEGER PK  ◄──  RepositoryId    INTEGER FK  ◄──  ModuleId      INTEGER FK
Name            TEXT             Id              INTEGER PK       Id            INTEGER PK
FullName        TEXT  owner/repo Path            TEXT  relative   ResourceType  TEXT  'Managed' | 'Data'
CreatedAt       TEXT             Providers       TEXT  JSON array Name          TEXT  label
LatestCommitAt  TEXT             ModuleCalls     TEXT  JSON array Type          TEXT  e.g. azurerm_subnet
LatestCommitSha TEXT                             of {name, source} Provider     TEXT  provider name
Description     TEXT             DiagnosticMessages TEXT  parse diagnostics
License         TEXT
SizeInKb        INTEGER
ForkCount       INTEGER
StarCount       INTEGER
HtmlUrl / GitUrl / SshUrl / CloneUrl  TEXT
Homepage        TEXT
Archived        INTEGER (bool)
Topics          TEXT  JSON array
  • A module is any directory containing .tf files, so a repo with modules/vnet, modules/aks and envs/prod has three rows.
  • Modules were analysed with HashiCorp's terraform-config-inspect, which reads configuration statically (no init, no variable values).
  • ModuleCalls holds the module blocks: their local name and source string.

Always confirm against the file you downloaded:

sqlite3 TerraDS.sqlite '.tables'
sqlite3 TerraDS.sqlite '.schema Modules'

Querying it

sqlite3 -header -column TerraDS.sqlite

Which providers are used most?

SELECT Provider, COUNT(*) AS resources
FROM Resources
GROUP BY Provider
ORDER BY resources DESC
LIMIT 15;

Most common Azure resource types

SELECT Type, COUNT(*) AS n
FROM Resources
WHERE Type LIKE 'azurerm\_%' ESCAPE '\' AND ResourceType = 'Managed'
GROUP BY Type
ORDER BY n DESC
LIMIT 30;

Where do people source modules from?

-- Classify module call sources: registry, git, local path, other
SELECT
  CASE
    WHEN json_extract(c.value, '$.source') LIKE './%'
      OR json_extract(c.value, '$.source') LIKE '../%'  THEN 'local'
    WHEN json_extract(c.value, '$.source') LIKE 'git::%'
      OR json_extract(c.value, '$.source') LIKE 'github.com/%' THEN 'git'
    WHEN json_extract(c.value, '$.source') LIKE '%/%/%'  THEN 'registry'
    ELSE 'other'
  END AS kind,
  COUNT(*) AS calls
FROM Modules m, json_each(m.ModuleCalls) c
GROUP BY kind
ORDER BY calls DESC;

Most-used public registry modules

SELECT json_extract(c.value, '$.source') AS source, COUNT(*) AS calls
FROM Modules m, json_each(m.ModuleCalls) c
WHERE json_extract(c.value, '$.source') NOT LIKE '.%'
  AND json_extract(c.value, '$.source') NOT LIKE 'git%'
GROUP BY source
ORDER BY calls DESC
LIMIT 25;

Unpinned Git module sources

-- Git sources with no ?ref= are pinned to whatever the default branch is
SELECT r.FullName, m.Path, json_extract(c.value, '$.source') AS source
FROM Modules m
JOIN Repositories r ON r.Id = m.RepositoryId,
     json_each(m.ModuleCalls) c
WHERE json_extract(c.value, '$.source') LIKE 'git%'
  AND json_extract(c.value, '$.source') NOT LIKE '%ref=%'
LIMIT 50;
SELECT DISTINCT r.FullName, r.StarCount, r.License, r.HtmlUrl
FROM Resources res
JOIN Modules m      ON m.Id = res.ModuleId
JOIN Repositories r ON r.Id = m.RepositoryId
WHERE res.Type = 'azurerm_kubernetes_cluster'
ORDER BY r.StarCount DESC
LIMIT 20;

Label naming styles

-- How often do labels use hyphens (legal, but awkward) vs snake_case?
SELECT
  SUM(Name LIKE '%-%')            AS hyphenated,
  SUM(Name NOT LIKE '%-%')        AS other,
  ROUND(100.0 * SUM(Name LIKE '%-%') / COUNT(*), 2) AS pct_hyphenated
FROM Resources;

Modules that failed to parse

SELECT COUNT(*) FROM Modules WHERE DiagnosticMessages IS NOT NULL AND DiagnosticMessages <> '';

For heavier analysis, load the database into pandas, DuckDB (INSTALL sqlite; ATTACH 'TerraDS.sqlite') or Polars.

Working with the source archive

The analyzer wrote one .tar.gz per repository, named by the repository's Id, and the outer archive bundles them:

mkdir -p ~/data/terrads/src && cd ~/data/terrads/src
tar -tzf ../TerraDS.tar.gz | head      # check the layout first
tar -xzf ../TerraDS.tar.gz             # → repositories/<Id>.tar.gz ... (adjust paths below if yours differs)
ls repositories | head

# Extract one repository by Id (look the Id up in SQLite first)
mkdir -p repo-1234 && tar -xzf repositories/1234.tar.gz -C repo-1234

Modules.Path is relative to the root of that repository archive, so repo-1234/<Path> is the module directory.

Extract only what you need. 62k nested archives expand to a lot of small files.

Practical uses

For a practitioner rather than a researcher, TerraDS is a big test corpus and a sample of community habits.

Test your own tooling against real-world HCL. Run a parser or linter over thousands of modules to find the edge cases your own repos don't have: odd heredocs, legacy 0.11 syntax, JSON configuration, unusual escapes. If you write HCL tooling in Rust with hcl-rs, or Python with python-hcl2, this is the fastest way to harden it.

# Example: run your naming policy over a sample of Azure modules
sqlite3 -csv TerraDS.sqlite "
  SELECT DISTINCT m.RepositoryId, m.Path
  FROM Resources res JOIN Modules m ON m.Id = res.ModuleId
  WHERE res.Type = 'azurerm_resource_group'
  ORDER BY random() LIMIT 200;" > sample.csv

while IFS=, read -r id path; do
  path=${path//\"/}
  mkdir -p "work/$id" && tar -xzf "repositories/$id.tar.gz" -C "work/$id" 2>/dev/null
  conftest test --parser hcl2 --policy ~/policy "work/$id/$path"/*.tf || true
done < sample.csv

Benchmark policies and scanners. How many real modules would your naming rules or Checkov baseline flag? A rule that fires on 60% of public code is probably too noisy, or catching a genuinely common mistake.

Learn common interfaces before writing a module. Look at how the most-starred repos structure azurerm_kubernetes_cluster or azurerm_linux_web_app modules: which arguments they expose, how they handle optional blocks.

Evidence for practices docs. Queries like "share of Git module sources without ?ref=" turn "pin your modules" from advice into a statistic for internal guidelines or a blog post.

Evaluating AI-generated Terraform. Several 2025–2026 papers build LLM training and evaluation sets on top of TerraDS, filtering it through terraform validate, terraform plan and OPA policies. That says something about the raw corpus: plenty of public Terraform doesn't validate or violates basic security policy. Keep that in mind when trusting generated code (ecosystem: AI).

Reproducing or extending it

The tools (prg-grp/hcl-dataset-tools, also in the Zenodo download) rebuild the dataset in four steps, orchestrated with just:

Step Tool Language Does
1 RepositorySearcher C# (.NET) Queries the GitHub search API (language:hcl, day by day to stay under the 1,000-result cap) and stores repository metadata
2 Analyzer Go Shallow-clones permissively licensed repos, finds Terraform directories, parses them with terraform-config-inspect, stores modules and resources
3 Cleaner Python Deletes non-Terraform files, empty directories, repos with no modules/resources
4 Compressor Go Packs everything into the archive
git clone https://github.com/prg-grp/hcl-dataset-tools && cd hcl-dataset-tools
cp .env.dist .env            # set GITHUB_TOKEN and DB_DIR
just fetch-metadata
just download-repositories
just cleanup-repositories
just archive-repositories

Requirements: .NET SDK, Go, Python with pandas, just, a GitHub token, and a lot of disk and time. RepositorySearcher's queries file is plain CSV, so you could narrow it (only repos with azurerm in the topic, only recent repos) to build a smaller, fresher, Azure-focused corpus.

Limitations

  • A snapshot. Version 1 reflects GitHub in 2024. Newer syntax (ephemeral resources, write-only arguments, actions, list resources) is rare or absent.
  • GitHub only, permissive licences only. No GitLab, no private code, no GPL repos. Enterprise Terraform likely looks different.
  • Static metadata. No variable values, tfvars, plans or state. count/for_each instances aren't expanded; a resource block counts once.
  • Quality varies. Tutorials, abandoned experiments and forks sit beside production modules. Filter by stars, recency or archived status for anything quality-sensitive.
  • Terragrunt and OpenTofu-specific files (terragrunt.hcl, .tofu) aren't analysed as Terraform modules.

References

Citation:

Bühler, C., Spielmann, D., Meier, R., & Salvaneschi, G. TerraDS: A Dataset for Terraform HCL Programs (v1). Zenodo. https://doi.org/10.5281/zenodo.14217386