⏱️ Lectura: 11 min
The Fedora 45 pipeline isn’t a single command: it’s four separate systems that turn a commit into the ISO you download from getfedora.org. Every time a packager runs git push on a package, it kicks off a chain that runs through Koji, Bodhi, and Pungi, three tools that have supported every release of the project for almost two decades.
📑 En este artículo
Developer supakeen documented that entire journey in a technical post published this week: from the initial commit in dist-git to the final compose that produces ISOs, cloud images, containers, and OSTree trees. This article summarizes the Fedora 45 pipeline with practical examples for anyone packaging software for the project, or anyone who simply wants to understand what runs behind every dnf upgrade.
TL;DR
- Fedora packages live in Git at src.fedoraproject.org, with large binaries kept outside the repository.
- fedpkg build sends the exact commit to Koji, the build system that has been active in Fedora since version 7 (2007).
- Koji assembles every build in a clean Mock chroot, with no state inherited from previous builds.
- Bodhi moves updates from pending to testing to stable: with +3 karma or enough days in testing, it moves automatically.
- Critical path packages need 14 days in testing instead of 7, and more karma to reach stable.
- Pungi orchestrates the final compose and produces ISOs, cloud images, containers, and OSTree trees.
- Package hosting is migrating from Pagure (src.fedoraproject.org) to Forgejo (forge.fedoraproject.org).
Introduction
The Fedora 45 pipeline connects four independent systems: dist-git stores the source code, Koji compiles the packages, Bodhi filters what reaches production, and Pungi assembles the final release. None of these components is new. Koji has run since Fedora 7, released in 2007, and Bodhi has accompanied it for more than a decade. What changes between versions are implementation details, like the way Pungi generates the boot.iso in Fedora 45.
Understanding this flow is useful for something concrete: knowing why a package takes days to reach stable, why a build never inherits the state of previous builds, and what to look at when something breaks between the commit and the final release.
What Happened
Fedora 45 doesn’t have a final release date yet, and the pipeline that produces it is still being refined through several ongoing ‘change proposals’. The most relevant one touches the generation of the boot.iso, the minimal image that boots the Anaconda installer and represents a large share of the compose work.
In parallel, Fedora is moving forward with migrating package hosting from Pagure, at src.fedoraproject.org, to Forgejo, at forge.fedoraproject.org. Both platforms serve the same role within the Fedora 45 pipeline: hosting the spec, the downstream patches, and the sources file that points to the upstream tarball in the lookaside cache. What changes is the interface and the engine behind it.
Context and History
It all starts in dist-git. Every Fedora package lives in its own Git repository with an RPM spec file, the downstream patches, and a sources file that points to the original tarball. Large binaries don’t go into Git: they stay in a separate lookaside cache, while the spec and patches do have full version history.
Packagers interact with dist-git through fedpkg, a CLI that wraps common operations: cloning repos, uploading tarballs, submitting builds, and creating updates. What matters about fedpkg build is what it actually does: it builds a URL pointing to a specific commit in the repo and passes it to Koji, the build system.
git clone https://src.fedoraproject.org/rpms/htop.git
cd htop
git checkout rawhide
# edit htop.spec: version, release, %changelog
fedpkg buildThat fedpkg build builds the package from that exact commit, never from the packager’s local disk. The result is reproducible: two people running the same command against the same commit get the same build.
dist-git branches map to releases: rawhide is continuous development, f45 is the stabilized branch for Fedora 45. Each branch lives independently and follows its own gating rules.
Technical Details of the Fedora 45 Pipeline
Koji has been Fedora’s build system since version 7 of the project. It follows a hub-and-spoke architecture: the hub is a passive XML-RPC server sitting in front of a PostgreSQL database, and the builder daemons poll the hub for pending work. Each builder creates a clean Mock chroot for every build, compiles inside it, and uploads the result. No build inherits state from previous builds or from packages manually installed on the builder the week before.
💡 Tip: if a build fails on Koji but compiles on your local machine, suspect the environment first. Mock starts from scratch every time, so any dependency you installed by hand isn’t available there.
Koji’s organization is based on tags: named collections of builds. A build target maps a build request to two tags, one that defines the buildroot (which packages are available during compilation) and a destination tag (where the finished build lands). Tags support multiple inheritance, so a Fedora 45 tag can be stacked on top of a base tag without duplicating the entire package tree.
Koji doesn’t just produce RPMs. Through its plugin and content generator system, it also orchestrates image builds: Kiwi via the kiwiBuild task type, Image Builder artifacts via imageBuilderBuild, and OSTree composes via runroot tasks.
A fresh build in Koji doesn’t reach users of a stable branch on its own. That’s what Bodhi is for, Fedora’s update gating system. A packager submits an update with one or more builds, and that update moves through a sequence of states: pending, testing, stable. Users and automated tests give karma (+1 / -1). With +3 karma or enough days in testing, the update moves to stable automatically. With +1 karma, the maintainer can push it manually (+2 for critical packages).
Bodhi handles all of this by manipulating Koji tags: when an update moves from testing to stable, it moves the build from the f45-updates-testing tag to the f45-updates tag, and then invokes Pungi to compose the updates repository, the actual yum/dnf repo that dnf upgrade pulls from.
Critical path packages (the ones the system needs to boot and function) follow stricter rules: 14 days in testing instead of 7, and more karma required. Bodhi also integrates with Greenwave and ResultsDB for CI gating, so a failing automated test can block an update before it reaches stable.
For Rawhide the process is different: packages that aren’t critical path go to stable immediately, unless a gating policy exists (because the package is in the critical path or has its own policy), in which case it can still be held back until it passes the checks.
The last link is Pungi, the compose orchestrator. Pungi doesn’t do the heavy lifting itself: it coordinates the tools that do, making sure everything builds from the same consistent set of packages. The name is a reference to the pungi, the snake charmer’s instrument, because it “charms” Anaconda, the Fedora installer. The joke has survived since 2006.
To understand how the three pieces fit together, it helps to compare them side by side:
| Component | What it does | Analogy | When it comes into play |
|---|---|---|---|
| Koji | Compiles RPMs and images in clean Mock chroots | The factory | On every fedpkg build |
| Bodhi | Gates which builds reach stable based on karma and CI | Quality control | On every update, between testing and stable |
| Pungi | Orchestrates the final compose: ISOs, cloud images, containers, OSTree | The assembly line | On every nightly or release compose |
The following diagram summarizes the full journey, from the packager’s git push to the final artifacts that get published:
flowchart TD
A["Packager: git push to dist-git"] --> B["fedpkg build"]
B --> C["Koji: build in clean Mock chroot"]
C --> D["Bodhi: testing and karma"]
D --> E["Stable f45-updates tag"]
E --> F["Pungi: compose"]
F --> G[("ISO, cloud image, container, OSTree")]
How to Try It
You don’t need to be a Fedora packager to inspect the Fedora 45 pipeline live. Koji’s web interface, at koji.fedoraproject.org, works the same from Windows, macOS, or Linux: all you need is a browser to see the history of builds, tags, and ongoing tasks.
For queries from the terminal, the Koji client installs differently depending on the operating system:
# Linux (Fedora / RHEL / derivatives)
sudo dnf install koji fedpkg
# macOS and Windows (via WSL, or Python directly on any OS)
pip install kojiWith the client installed, two commands are enough to see the real status of a package:
koji list-builds --package=htop --state=COMPLETE
koji buildinfo htop-3.3.0-1.fc45The first command lists the completed builds for a package; the second shows the detail of a specific build: which tag contains it, who triggered it, and how long it took. To see the status of an update in Bodhi, the equivalent flow is visiting bodhi.fedoraproject.org and searching for the package: there you can see the accumulated karma, days in testing, and CI results via Greenwave.
Impact and Analysis
The design of the Fedora 45 pipeline prioritizes reproducibility over speed. Every build starts from a clean chroot, every update goes through a minimum number of days in testing, and every compose is assembled from a consistent snapshot of packages. That rigidity comes at a cost: a critical package can take more than two weeks to reach an end user after the packager makes the initial commit.
In exchange, the project gains something few distros document as well: any build is traceable back to an exact dist-git commit, and any decision to promote something to stable is logged in Bodhi with its karma and CI results. For teams packaging internal software using Koji or Bodhi as a reference (several organizations run their own instance), this level of traceability is replicable without depending on Fedora itself.
📌 Note: Koji’s multiple-inheritance tag model is what makes it possible to keep separate buildroots for Rawhide, F44, and F45 without tripling the configuration: each tag inherits from the base tag and only declares its differences.
For teams in Latin America running Fedora on servers or as a base for cloud images, understanding this pipeline helps read a package’s changelog more clearly: every version that reaches production went through Koji, accumulated karma in Bodhi, and was included in a Pungi compose with a verifiable date and hash.
What’s Next
supakeen’s post remains a living document while Fedora 45 has no final release date: several ‘change proposals’ could still change how the boot.iso is generated, which is a large portion of Pungi’s work. The dist-git migration from Pagure to Forgejo isn’t finished either, so the Fedora 45 pipeline could look different in the next cycle, even though the three core components (Koji, Bodhi, Pungi) will stay the same.
The original author plans to update the document every cycle or every few Fedora cycles, so it serves as a historical reference as well as a snapshot of the current state.
📖 Summary on Telegram: View summary
Try it yourself: install the client with sudo dnf install koji fedpkg and run koji list-builds –package=<your-favorite-package> to see the Fedora 45 pipeline with real data, today.
Frequently Asked Questions
What is dist-git in Fedora?
It’s the set of Git repositories, hosted at src.fedoraproject.org (Pagure) and migrating to forge.fedoraproject.org (Forgejo), where each package stores its spec, its downstream patches, and the sources file that points to the upstream tarball.
Why does Koji compile in a clean chroot every time?
To guarantee reproducibility. If the builder had packages installed from previous builds, two people could get different results from the same commit. Mock creates that environment from scratch for every build.
How do I know if a package is ‘critical path’?
These are the packages needed for the system to boot and function. Bodhi requires 14 days in testing instead of 7 for them, and more karma before allowing a manual push to stable.
What’s the difference between Koji and Pungi?
Koji compiles individual packages and images. Pungi doesn’t compile anything itself: it orchestrates Koji, Anaconda, and other tools to assemble the complete release (ISOs, cloud images, containers, OSTree) from a consistent set of packages.
Why is Pungi called that?
It’s a reference to the pungi, the snake charmer’s instrument, because the tool “charms” Anaconda, the Fedora installer. The joke has circulated in the project since 2006.
Can I replicate this pipeline outside of Fedora?
Yes. Koji, Bodhi, and Pungi are independent open source projects, and several organizations deploy them to manage their own internal RPM repositories with the same tags and gating model.
References
- The Fedora 45 Sausage Factory: supakeen’s original walkthrough of the complete Fedora 45 pipeline.
- Koji: web interface for Fedora’s build system, active since Fedora 7.
- Bodhi: update gating system with karma, testing, and stable.
- Fedora Docs: fedpkg: official guide to the CLI packagers use to interact with dist-git and Koji.
📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
Imagen destacada: Foto de Pankaj Patel en Unsplash
0 Comments