What it is¶
TerraDS is a public dataset of Terraform programs mined from GitHub, published by researchers at the University of St. Gallen and armasuisse (Christoph Bühler, David Spielmann, Roland Meier, Guido Salvaneschi) and presented at MSR 2025 (the Mining Software Repositories conference, Data and Tool Showcase track). Before it, there was no large, reusable corpus of real-world HCL for studying how people actually write Terraform.
It isn't a tool you run in a pipeline. It's data: a SQLite database of metadata plus an archive of the HCL itself, which you query and analyse.
Version 1 on Zenodo (DOI 10.5281/zenodo.14217386, November 2024):
| Count | |
|---|---|
| Repositories (permissively licensed) | 62,406 |
Terraform modules (directories with .tf files) |
279,344 |
| Resources | 1,773,991 |
… managed (resource) |
1,484,185 |
… data sources (data) |
289,806 |
The MSR paper reports slightly different totals (67,360 repositories in the abstract), and Zenodo notes a newer version exists. Check Zenodo for the latest version and quote numbers from the version you use.
Licence: the dataset is CC BY 4.0. Each included repository has its own permissive licence (MIT, Apache-2.0, BSD and similar), recorded in the database; respect those if you redistribute code from it.
What's in the download¶
| File | Size (v1) | Contents |
|---|---|---|
TerraDS.sqlite |
~434 MB | Metadata for repositories, modules and resources |
TerraDS.tar.gz |
~351 MB | The HCL source of every repository (non-Terraform files stripped) |
HCL Dataset Tools.zip |
~66 kB | Snapshot of the tools that built the dataset |
mkdir -p ~/data/terrads && cd ~/data/terrads
curl -L -o TerraDS.sqlite "https://zenodo.org/records/14217386/files/TerraDS.sqlite?download=1"
curl -L -o TerraDS.tar.gz "https://zenodo.org/records/14217386/files/TerraDS.tar.gz?download=1"
md5sum TerraDS.sqlite TerraDS.tar.gz
# expected (v1): 92e2fffd8208b2a290204555ec500977 TerraDS.sqlite
# 664716c1dd690b14add572cd5a41f192 TerraDS.tar.gz
Treat it like any downloaded archive: extract into its own directory, and don't run anything you find inside it. It's third-party code.
Database schema¶
From the tools' source (src/Database/Models and the Go analyzer):
Repositories Modules Resources
──────────── ─────── ─────────
Id INTEGER PK ◄── RepositoryId INTEGER FK ◄── ModuleId INTEGER FK
Name TEXT Id INTEGER PK Id INTEGER PK
FullName TEXT owner/repo Path TEXT relative ResourceType TEXT 'Managed' | 'Data'
CreatedAt TEXT Providers TEXT JSON array Name TEXT label
LatestCommitAt TEXT ModuleCalls TEXT JSON array Type TEXT e.g. azurerm_subnet
LatestCommitSha TEXT of {name, source} Provider TEXT provider name
Description TEXT DiagnosticMessages TEXT parse diagnostics
License TEXT
SizeInKb INTEGER
ForkCount INTEGER
StarCount INTEGER
HtmlUrl / GitUrl / SshUrl / CloneUrl TEXT
Homepage TEXT
Archived INTEGER (bool)
Topics TEXT JSON array
- A module is any directory containing
.tffiles, so a repo withmodules/vnet,modules/aksandenvs/prodhas three rows. - Modules were analysed with HashiCorp's
terraform-config-inspect, which reads configuration statically (noinit, no variable values). ModuleCallsholds themoduleblocks: their local name andsourcestring.
Always confirm against the file you downloaded:
sqlite3 TerraDS.sqlite '.tables'
sqlite3 TerraDS.sqlite '.schema Modules'
Querying it¶
sqlite3 -header -column TerraDS.sqlite
Which providers are used most?¶
SELECT Provider, COUNT(*) AS resources
FROM Resources
GROUP BY Provider
ORDER BY resources DESC
LIMIT 15;
Most common Azure resource types¶
SELECT Type, COUNT(*) AS n
FROM Resources
WHERE Type LIKE 'azurerm\_%' ESCAPE '\' AND ResourceType = 'Managed'
GROUP BY Type
ORDER BY n DESC
LIMIT 30;
Where do people source modules from?¶
-- Classify module call sources: registry, git, local path, other
SELECT
CASE
WHEN json_extract(c.value, '$.source') LIKE './%'
OR json_extract(c.value, '$.source') LIKE '../%' THEN 'local'
WHEN json_extract(c.value, '$.source') LIKE 'git::%'
OR json_extract(c.value, '$.source') LIKE 'github.com/%' THEN 'git'
WHEN json_extract(c.value, '$.source') LIKE '%/%/%' THEN 'registry'
ELSE 'other'
END AS kind,
COUNT(*) AS calls
FROM Modules m, json_each(m.ModuleCalls) c
GROUP BY kind
ORDER BY calls DESC;
Most-used public registry modules¶
SELECT json_extract(c.value, '$.source') AS source, COUNT(*) AS calls
FROM Modules m, json_each(m.ModuleCalls) c
WHERE json_extract(c.value, '$.source') NOT LIKE '.%'
AND json_extract(c.value, '$.source') NOT LIKE 'git%'
GROUP BY source
ORDER BY calls DESC
LIMIT 25;
Unpinned Git module sources¶
-- Git sources with no ?ref= are pinned to whatever the default branch is
SELECT r.FullName, m.Path, json_extract(c.value, '$.source') AS source
FROM Modules m
JOIN Repositories r ON r.Id = m.RepositoryId,
json_each(m.ModuleCalls) c
WHERE json_extract(c.value, '$.source') LIKE 'git%'
AND json_extract(c.value, '$.source') NOT LIKE '%ref=%'
LIMIT 50;
Popular repositories using a given resource¶
SELECT DISTINCT r.FullName, r.StarCount, r.License, r.HtmlUrl
FROM Resources res
JOIN Modules m ON m.Id = res.ModuleId
JOIN Repositories r ON r.Id = m.RepositoryId
WHERE res.Type = 'azurerm_kubernetes_cluster'
ORDER BY r.StarCount DESC
LIMIT 20;
Label naming styles¶
-- How often do labels use hyphens (legal, but awkward) vs snake_case?
SELECT
SUM(Name LIKE '%-%') AS hyphenated,
SUM(Name NOT LIKE '%-%') AS other,
ROUND(100.0 * SUM(Name LIKE '%-%') / COUNT(*), 2) AS pct_hyphenated
FROM Resources;
Modules that failed to parse¶
SELECT COUNT(*) FROM Modules WHERE DiagnosticMessages IS NOT NULL AND DiagnosticMessages <> '';
For heavier analysis, load the database into pandas, DuckDB (INSTALL sqlite; ATTACH 'TerraDS.sqlite') or Polars.
Working with the source archive¶
The analyzer wrote one .tar.gz per repository, named by the repository's Id, and the outer archive bundles them:
mkdir -p ~/data/terrads/src && cd ~/data/terrads/src
tar -tzf ../TerraDS.tar.gz | head # check the layout first
tar -xzf ../TerraDS.tar.gz # → repositories/<Id>.tar.gz ... (adjust paths below if yours differs)
ls repositories | head
# Extract one repository by Id (look the Id up in SQLite first)
mkdir -p repo-1234 && tar -xzf repositories/1234.tar.gz -C repo-1234
Modules.Path is relative to the root of that repository archive, so repo-1234/<Path> is the module directory.
Extract only what you need. 62k nested archives expand to a lot of small files.
Practical uses¶
For a practitioner rather than a researcher, TerraDS is a big test corpus and a sample of community habits.
Test your own tooling against real-world HCL. Run a parser or linter over thousands of modules to find the edge cases your own repos don't have: odd heredocs, legacy 0.11 syntax, JSON configuration, unusual escapes. If you write HCL tooling in Rust with hcl-rs, or Python with python-hcl2, this is the fastest way to harden it.
# Example: run your naming policy over a sample of Azure modules
sqlite3 -csv TerraDS.sqlite "
SELECT DISTINCT m.RepositoryId, m.Path
FROM Resources res JOIN Modules m ON m.Id = res.ModuleId
WHERE res.Type = 'azurerm_resource_group'
ORDER BY random() LIMIT 200;" > sample.csv
while IFS=, read -r id path; do
path=${path//\"/}
mkdir -p "work/$id" && tar -xzf "repositories/$id.tar.gz" -C "work/$id" 2>/dev/null
conftest test --parser hcl2 --policy ~/policy "work/$id/$path"/*.tf || true
done < sample.csv
Benchmark policies and scanners. How many real modules would your naming rules or Checkov baseline flag? A rule that fires on 60% of public code is probably too noisy, or catching a genuinely common mistake.
Learn common interfaces before writing a module. Look at how the most-starred repos structure azurerm_kubernetes_cluster or azurerm_linux_web_app modules: which arguments they expose, how they handle optional blocks.
Evidence for practices docs. Queries like "share of Git module sources without ?ref=" turn "pin your modules" from advice into a statistic for internal guidelines or a blog post.
Evaluating AI-generated Terraform. Several 2025–2026 papers build LLM training and evaluation sets on top of TerraDS, filtering it through terraform validate, terraform plan and OPA policies. That says something about the raw corpus: plenty of public Terraform doesn't validate or violates basic security policy. Keep that in mind when trusting generated code (ecosystem: AI).
Reproducing or extending it¶
The tools (prg-grp/hcl-dataset-tools, also in the Zenodo download) rebuild the dataset in four steps, orchestrated with just:
| Step | Tool | Language | Does |
|---|---|---|---|
| 1 | RepositorySearcher | C# (.NET) | Queries the GitHub search API (language:hcl, day by day to stay under the 1,000-result cap) and stores repository metadata |
| 2 | Analyzer | Go | Shallow-clones permissively licensed repos, finds Terraform directories, parses them with terraform-config-inspect, stores modules and resources |
| 3 | Cleaner | Python | Deletes non-Terraform files, empty directories, repos with no modules/resources |
| 4 | Compressor | Go | Packs everything into the archive |
git clone https://github.com/prg-grp/hcl-dataset-tools && cd hcl-dataset-tools
cp .env.dist .env # set GITHUB_TOKEN and DB_DIR
just fetch-metadata
just download-repositories
just cleanup-repositories
just archive-repositories
Requirements: .NET SDK, Go, Python with pandas, just, a GitHub token, and a lot of disk and time. RepositorySearcher's queries file is plain CSV, so you could narrow it (only repos with azurerm in the topic, only recent repos) to build a smaller, fresher, Azure-focused corpus.
Limitations¶
- A snapshot. Version 1 reflects GitHub in 2024. Newer syntax (ephemeral resources, write-only arguments, actions, list resources) is rare or absent.
- GitHub only, permissive licences only. No GitLab, no private code, no GPL repos. Enterprise Terraform likely looks different.
- Static metadata. No variable values, tfvars, plans or state.
count/for_eachinstances aren't expanded; a resource block counts once. - Quality varies. Tutorials, abandoned experiments and forks sit beside production modules. Filter by stars, recency or archived status for anything quality-sensitive.
- Terragrunt and OpenTofu-specific files (
terragrunt.hcl,.tofu) aren't analysed as Terraform modules.
References¶
- Zenodo record (dataset + tools): https://zenodo.org/records/14217386
- Paper (MSR 2025): https://roland-meier.ch/files/2025_MSR_terra-ds-hcl-dataset.pdf
- MSR 2025 talk listing: https://2025.msrconf.org/details/msr-2025-data-and-tool-showcase-track/12/TerraDS-A-Dataset-for-Terraform-HCL-Programs
- Tools: https://github.com/prg-grp/hcl-dataset-tools
Citation:
Bühler, C., Spielmann, D., Meier, R., & Salvaneschi, G. TerraDS: A Dataset for Terraform HCL Programs (v1). Zenodo. https://doi.org/10.5281/zenodo.14217386