A crawler for Zig dependencies

generated a sqlite database of git repositories with zig dependencies. Also includes a small web application to view the database contents.

Repositories are indexed from these sites if they have the topics zig or zig-package assigned:

• https://github.com
• https://gitlab.com
• https://codeberg.org

Dependencies are resolved between those sites even if those topics aren't set based on build.zig.zon urls.

Hosted on gitlab pages: https://dasimmet.gitlab.io/zig-git-crawler/

# build the exe
zig build
# setup an API token for github:
export GITHUB_API_AUTH_TOKEN=********************
# crawl through major all git hosting sites, create the db
zig build run
# alternatively, download the db from the ci crawl process:
zig build download-db
# query the db
sqlite3 zig-out/www/zig-git.db "SELECT * FROM view_manifest_tree"
# build the webapp:
zig build web
# run the dev webserver:
zig build dev
# open the browser to view it: http://localhost:8080/www

Database Schema:

TODO

Web UI

• Needs quite a bit more explaining whats going on.
• Search for repositories in the main list
• Provide numerical indicators for dependencies and dependants
• Better and searchable selectors for repo and ref in the details view.
• The dependency/dependant queries have a depth field, however they don't take path dependencies into account since the sql query was easier to implement.

Crawler

• Crawler is hitting Github/Codeberg API limits quickly and needs a few runs to populate.
• Github's Api is limited to 1000 search results. For now, I keep a list of search urls based on the repo creation date to work around it. Could be solved by calculating the URLs based on whats already in the DB.
• Crawler does not update the repository entries of dependencies, only those repos with zig and zig-package tags. New commits and manifests are fetched though.
• Between runs, the crawler relies on keeping the sqlite db file on the runner cache. Maybe it could be stored and downloaded to/from gitlab's package registry instead.