This dataset consists of two parts:
Siva files with Git repositories.
Index file in CSV format.
Besides, there is a number of auxiliary datasets:
configs.tar.xz - raw git config files for each siva.
heads.csv.xz - mapping from HEAD UUID to repository name.
Since the second version of PGA, we additionally provide the derived dataset of UASTs, extracted from the files in the latest revision of each repository.
pga - explore the dataset, or download its contents easily.
pga-create - reproduce PGA dataset generation.
borges-indexer - exports a CSV file with metadata from repositories fetched with Borges.
list_heads - lists files in each HEAD contained in siva.
To see the full list of repositories in the dataset or download it, you will need to install pga. Simply install Go and then run
go get github.com/src-d/datasets/PublicGitArchive/pga.
Then to list all of the repositories in the dataset, simply run:
pga list siva
If you'd rather get a detailed dump of the dataset (not including the file contents) you can choose either
pga list siva -f json or
pga list siva -f csv.
To download the full dataset, execute:
pga get siva
Or if you want to download only those repositories containing at least a line of Java code:
pga get siva -l java
pga command has
-j/--workers argument which specifies the number of downloading threads to run, it defaults to 10.
For more information, check the pga documentation, or simply run
Refer to pga-create documentation for more details about how PGA is generated.
We understand that some GitHub projects may become private or deleted with time. Previous dataset snapshots will continue to include such dead code. If you are the author and want to remove your project from all present and future public snapshots, please send a request to