diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml new file mode 100644 index 0000000000..848710369f --- /dev/null +++ b/.github/workflows/coverage.yml @@ -0,0 +1,104 @@ +name: Coverage + +on: + pull_request_target: + paths: + - 'django/**/*.py' + - 'tests/**/*.py' + branches: + - main + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + +jobs: + diff-coverage: + if: github.repository == 'django/django' + name: Diff Coverage (Windows) + runs-on: windows-latest + steps: + - name: Checkout + uses: actions/checkout@v5 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: '3.14' + cache: 'pip' + cache-dependency-path: 'tests/requirements/py3.txt' + + - name: Install dependencies + run: | + python -m pip install --upgrade pip wheel + python -m pip install -r tests/requirements/py3.txt -e . + python -m pip install 'coverage[toml]' diff-cover + + - name: Run tests with coverage + env: + PYTHONPATH: ${{ github.workspace }}/tests + COVERAGE_PROCESS_START: ${{ github.workspace }}/tests/.coveragerc + RUNTESTS_DIR: ${{ github.workspace }}/tests + run: | + python -Wall tests/runtests.py -v2 + + - name: Generate coverage report + if: success() + env: + COVERAGE_RCFILE: ${{ github.workspace }}/tests/.coveragerc + RUNTESTS_DIR: ${{ github.workspace }}/tests + run: | + python -m coverage combine + python -m coverage report --show-missing + python -m coverage xml -o tests/coverage.xml + + - name: Run diff-cover + if: success() + run: | + if (Test-Path 'tests/coverage.xml') { + diff-cover tests/coverage.xml --compare-branch=origin/main --fail-under=0 > tests/diff-cover-report.md + } else { + Set-Content -Path tests/diff-cover-report.md -Value 'No coverage.xml found; skipping diff-cover.' + } + + - name: Post/update PR comment + if: success() + uses: actions/github-script@v8 + with: + script: | + const fs = require('fs'); + const reportPath = 'tests/diff-cover-report.md'; + let body = 'No coverage data available.'; + if (fs.existsSync(reportPath)) { + body = fs.readFileSync(reportPath, 'utf8'); + } + const commentBody = '### 📊 Coverage Report for Changed Files\n\n```\n' + body + '\n```\n\n**Note:** Missing lines are warnings only. Some lines may not be covered by SQLite tests as they are database-specific.\n\nFor more information about code coverage on pull requests, see the [contributing documentation](https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#code-coverage-on-pull-requests).'; + + const { data: comments } = await github.rest.issues.listComments({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + }); + for (const c of comments) { + if ((c.body || '').includes('📊 Coverage Report for Changed Files')) { + await github.rest.issues.deleteComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: c.id, + }); + } + } + + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + body: commentBody, + }); diff --git a/docs/internals/contributing/writing-code/submitting-patches.txt b/docs/internals/contributing/writing-code/submitting-patches.txt index 841a2109dc..428acffba1 100644 --- a/docs/internals/contributing/writing-code/submitting-patches.txt +++ b/docs/internals/contributing/writing-code/submitting-patches.txt @@ -438,6 +438,9 @@ All code changes * If the change is backwards incompatible in any way, is there a note in the release notes (``docs/releases/A.B.txt``)? * Is Django's test suite passing? +* If there is a :ref:`code coverage report ` + comment on the pull request, have you reviewed the missing coverage in + context (considering database/platform-specific limitations)? * If the change affects the Django admin or rendered HTML output, has :ref:`accessibility testing ` been done? diff --git a/docs/internals/contributing/writing-code/unit-tests.txt b/docs/internals/contributing/writing-code/unit-tests.txt index cba4ba7397..f304970a29 100644 --- a/docs/internals/contributing/writing-code/unit-tests.txt +++ b/docs/internals/contributing/writing-code/unit-tests.txt @@ -394,6 +394,49 @@ settings file defines ``coverage_html`` as the output directory for the report and also excludes several directories not relevant to the results (test code or external code included in Django). +.. _code-coverage-on-pull-requests: + +Code coverage on pull requests +------------------------------ + +Django's continuous integration (CI) system automatically runs code coverage +analysis on pull requests and posts a comment with a diff coverage report. This +helps reviewers see which lines in the changed code are covered by tests. + +**What the coverage report shows:** + +The coverage report posted on pull requests uses `diff-cover`_ to analyze only +the lines that were changed or added in the PR. It shows: + +* Lines that are covered by tests (✓) +* Lines that are not covered by tests (✗) +* Lines that cannot be covered (e.g., comments, blank lines) + +.. _diff-cover: https://github.com/Bachmann1234/diff_cover + +**Important limitations:** + +When reviewing coverage reports on pull requests, keep these limitations in +mind: + +* **Database-specific code:** The CI coverage job runs tests using SQLite on + Windows. Code paths specific to other databases (PostgreSQL, MySQL, Oracle) + will appear as "not covered" even if database-specific tests exist. This is + expected and acceptable. + +* **Platform-specific code:** Similarly, code that only runs on certain + operating systems (Linux, macOS) will appear as not covered when run on + Windows. + +* **Coverage doesn't equal quality:** A line being "covered" only means it was + executed during tests. It doesn't guarantee the line is well-tested or that + all edge cases are handled. During review, assess test quality beyond just + coverage numbers. + + +Missing coverage should be considered a warning rather than a blocker and +should be evaluated in context. + .. _contrib-apps: Contrib apps diff --git a/zizmor.yml b/zizmor.yml index 76e53f73cc..e0401fa716 100644 --- a/zizmor.yml +++ b/zizmor.yml @@ -1,6 +1,7 @@ rules: dangerous-triggers: ignore: + - coverage.yml - labels.yml - new_contributor_pr.yml unpinned-uses: