cleanup-unreferenced-packages

Cleanup Unreferenced Packages

Prerequisites

• The gh CLI must be authenticated with access to dotnet/dotnet (the VMR) and dotnet/source-build-assets repos.
• The working directory must be the root of the source-build-assets repository.
• The local repo must be synced with upstream (dotnet/source-build-assets) main, not a personal fork's main. Before starting, ensure the working tree reflects upstream:

  git remote add upstream https://github.com/dotnet/source-build-assets.git 2>/dev/null || true
  git fetch upstream
  git checkout upstream/main
  

  All package existence checks and deletions must be performed against upstream main, not a fork's potentially stale main branch.

Workflow

Step 1: Obtain the SBA Usage Report

Download sbaPackageUsage.json from the latest successful VMR unified build (dotnet-unified-build, definition 1330) on the main branch. This is an Azure DevOps pipeline.

# Find the latest successful build on main
az pipelines runs list \
  --organization https://dev.azure.com/dnceng \
  --project internal \
  --pipeline-ids 1330 \
  --branch main \
  --status completed \
  --result succeeded \
  --top 1 \
  --query "[0].id" -o tsv

Next, find the artifact containing the usage report. The artifact name follows the pattern SB_<distro>_Online_MsftSdk_x64_BuildLogs_Attempt1 where <distro> is the current CentOS Stream version (e.g., CentOSStream10). List the build's artifacts to find the matching one:

# List artifacts to find the CentOS Stream source-build leg
az pipelines runs artifact list \
  --organization https://dev.azure.com/dnceng \
  --project internal \
  --run-id <build_id> \
  --query "[?contains(name, 'SB_CentOSStream')].name" -o tsv

Download the report from the identified artifact:

az pipelines runs artifact download \
  --organization https://dev.azure.com/dnceng \
  --project internal \
  --run-id <build_id> \
  --artifact-name "<artifact_name>" \
  --path /tmp/vmr-report

If the az CLI is not available, the artifact can also be downloaded via the AzDO REST API using curl with a PAT, or the user can provide a local path to the report directly.

Parse the JSON file and extract the UnreferencedSbaPackages array. Each entry is a path indicating the package type and identity:

• /__w/1/s/src/source-build-assets/src/referencePackages/src/<name>/<version>
• /__w/1/s/src/source-build-assets/src/textOnlyPackages/src/<name>/<version>
• /__w/1/s/src/source-build-assets/src/externalPackages/...
• /__w/1/s/src/source-build-assets/src/targetPacks/...

Normalize these to <type>/<name>/<version> tuples (e.g., referencePackages/system.buffers/4.5.1).

Step 2: Load the Cleanup Config

Read .github/skills/cleanup-unreferenced-packages/cleanup-packages-config.json from the repo.

{
  "knownFalsePositives": ["textOnlyPackages/microsoft.build.notargets/3.7.0", ...]
}

These packages are protected — they must not be deleted.

Step 3: Determine Packages Not Yet Flowed to VMR

SBA tip-of-main may contain package additions or deletions that haven't flowed into the VMR yet. Additions would appear unreferenced because the VMR hasn't seen them. Deletions would appear unreferenced because the VMR still has the old state — this is expected and those should still be candidates for deletion (they're already gone from SBA).

How to detect:

1. Fetch the VMR's src/source-manifest.json to find which SBA commit is incorporated:

   gh api repos/dotnet/dotnet/contents/src/source-manifest.json --jq '.content' | base64 -d | jq -r '.repositories[] | select(.path == "src/source-build-assets") | .commitSha'
   

2. Compare that commit to the current HEAD:

   git log --oneline <vmr-sba-sha>..HEAD -- src/
   

3. Identify package directories that were added in that range (additions that haven't flowed are the ones we need to protect):

   git diff --name-status --diff-filter=A <vmr-sba-sha>..HEAD -- src/referencePackages/src/ src/textOnlyPackages/src/ src/externalPackages/ src/targetPacks/
   

4. Extract the package identifiers (<name>/<version>) from added paths. These are not-yet-flowed packages — add them to the protected set.

Note: Packages that were deleted in SBA since the VMR commit may still appear in the unreferenced report (because the VMR still has them). This is fine — leave them in the deletion candidates since they've already been intentionally removed from SBA.

Step 4: Classify Candidates by Age

Apply a tiered grace-period strategy based on when each package was added to the repository.

Compute the initial deletion candidates:

initialProtected = knownFalsePositives + notYetFlowedPackages (combined set)
candidatesForDeletion = unreferencedPackages - initialProtected

Determine the age of each candidate by finding the commit that first introduced its directory:

# Get the date when a package was first added
git log --diff-filter=A --follow --format=%aI -- "src/<type>/src/<name>/<version>/" | tail -1

If no commit is found (e.g., the package predates the repo's git history), treat the package as having an age exceeding the mature threshold.

Apply the following classification:

Step 5: Identify Authors for Middle-Window Packages

For packages in the middle window (added between 14 and 60 days ago), identify the original author so they can be @-mentioned in the PR.

Identify the original author:

# Get the commit that added the package
git log --diff-filter=A --follow --format="%H %an <%ae>" -- "src/<type>/src/<name>/<version>/" | tail -1

Then find the associated PR to get the GitHub username:

# Find PR associated with the commit
gh pr list --repo dotnet/source-build-assets --search "<commit_sha>" --state merged --json number,author --jq '.[0].author.login'

Record the author and addition date for each middle-window package — these will be included in the PR body (see Step 7).

Step 5a: Resolve Dependencies of Middle-Window Packages

Middle-window packages may have transitive dependencies that are also in the deletion candidate list. Those dependencies should also be treated as middle-window (and their authors notified in the PR).

For each middle-window package, resolve its dependencies:

1. Build the package to produce a .nupkg:

   ./build.sh -sb --projects /full/path/to/<package>.csproj
   

2. Extract dependencies from the nupkg:

   unzip -p artifacts/packages/Release/Shipping/<PackageId>.<Version>.nupkg "*.nuspec" | grep -A1 "<dependency "
   

3. Any dependency that appears in the mature deletion list should be reclassified as middle-window (attributed to the same author). These will appear in the Action Required table in the PR alongside the package that depends on them. Repeat transitively.

Display the final classification:

=== Cleanup Report ===

Protected (will NOT be deleted):
  - textOnlyPackages/microsoft.build.notargets/3.7.0 (known false positive)
  - referencePackages/some.new.package/1.0.0 (recent: added 5 days ago)

Deleting (<N> packages):
  Mature:
    - <type>/<name>/<version> (age: 95 days)
  Middle window (authors will be notified in PR):
    - <type>/<name>/<version> (age: 25 days, author: @username)
  Dependencies of middle-window packages:
    - <type>/<name>/<version> (dependency of <parent-package>, <parent-package-2>)

Proceed to Step 6 with both mature and middle-window packages.

Step 6: Delete Packages

For each package in the confirmed deletion list:

1. Determine the package type from its path:

   • src/referencePackages/src/<name>/<version>/ → reference package
   • src/textOnlyPackages/src/<name>/<version>/ → text-only package
   • src/targetPacks/src/<name>/<version>/ → targeting pack
   • src/externalPackages/ → external package (includes submodule, project, patches)

2. Delete the package directory:

   rm -rf src/referencePackages/src/<name>/<version>
   # or
   rm -rf src/textOnlyPackages/src/<name>/<version>
   # or
   rm -rf src/targetPacks/src/<name>/<version>
   

3. For external packages, also:

   • Remove the submodule: git rm src/externalPackages/src/<component>
   • Remove the project file: rm src/externalPackages/projects/<component>.proj
   • Remove patches: rm -rf src/externalPackages/patches/<component>/

4. Check if deleting a versioned package leaves an empty parent directory (e.g., the <name>/ dir has no more versions) and clean it up if so.

Step 7: Commit and Create PR

git add -A
git commit -m "Remove unreferenced packages

Removed <N> packages that are no longer referenced by the VMR build.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>"

Create a PR against the upstream repo (dotnet/source-build-assets), not a personal fork.

If there are middle-window packages, include an "Action Required" section in the PR body that @-mentions the original authors. This gives them visibility and a chance to revert specific deletions before the PR merges.

gh pr create \
  --repo dotnet/source-build-assets \
  --title "[main] Remove unreferenced packages" \
  --body "Removed the following packages which were detected as unreferenced:

<list each deleted package as type/name/version, one per line>

## Action Required

The following packages were added recently but are not yet referenced in the VMR.
If uptake is still in progress, ask Copilot to revert these package deletions from
this PR.

| Package | Added | Author |
|---------|-------|--------|
| \`<type>/<name>/<version>\` | <N> days ago | @<author> |

Dependencies of the above packages (also being removed):
- \`<type>/<name>/<version>\` (dependency of \`<parent-package>\`, \`<parent-package-2>\`)

---
Generated by the [cleanup-unreferenced-packages](.github/skills/cleanup-unreferenced-packages/SKILL.md) skill."

Omit the "Action Required" section entirely if there are no middle-window packages. Omit the "Dependencies" list if there are no reclassified transitive dependencies.

After the PR is created, trigger a full VMR source build to validate the removals:

gh pr comment <pr_number> --repo dotnet/source-build-assets --body "/azp run source-build-assets-unified-build"

Important Notes

• The knownFalsePositives in the config exist because microsoft.build.notargets and microsoft.build.traversal are MSBuild SDKs referenced in global.json — the usage reporter doesn't detect SDK-style usage.
• When in doubt, err on the side of keeping a package. It's much easier to delete later than to re-add and coordinate flows.
• Middle-window author notifications happen directly in the PR body via @-mentions. If a package's uptake is still in progress, the author can ask Copilot to revert those specific deletions from the PR.

Config File Schema

Location: .github/skills/cleanup-unreferenced-packages/cleanup-packages-config.json