Automating GitHub repository LOC counts into structured CSV files for remote teams

Manual Spreadsheets vs. Automated Pipelines: A Tale of Two Quarterly Reviews

Last March, I watched our engineering manager spend eleven hours across three days manually tallying lines of code across forty-seven GitHub repositories. She exported git logs, pasted them into Google Sheets, deduplicated merges by hand, and still ended up with numbers that didn't reconcile between her spreadsheet and the finance team's audit. By the time she presented to leadership, two repositories had already received new commits that invalidated her counts.

Six months later, that same quarterly review took four minutes.

The difference wasn't a better spreadsheet. It was a GitHub Actions workflow that automated repository LOC counts into structured CSV files, pushed them to a shared drive, and let every remote team lead pull the numbers on their own schedule. This article walks through exactly how we built that pipeline, where the early versions failed, and what the final configuration looks like for a distributed engineering organization.

For remote teams operating across four or more time zones, manual code metrics collection is not just inefficient. It's structurally unreliable. By the time a human finishes counting, the codebase has already moved on.

Why Ad-Hoc LOC Counting Breaks Down for Remote Teams

The Visibility Problem

Our team spans San Francisco, Berlin, Bangalore, and São Paulo. When a VP asks "how much did each squad ship this quarter," the question sounds simple. The answer never is.

Consider the wrong approach: pinging each team lead on Slack and asking them to run git log --stat on their repositories. Team A sends back a screenshot. Team B sends a raw text file. Team C forgets to exclude generated code. Team D includes vendor dependencies. You now have four data sets that cannot be meaningfully compared.

Now consider the right approach: a single automated job that runs the same LOC counting logic against every repository, outputs identically structured CSV files, and deposits them in a location everyone can access. Team leads don't run anything. They consume.

Where Manual Collection Fails Specifically

The failure points are predictable. First, consistency—humans apply filtering rules inconsistently. One developer excludes package-lock.json; another doesn't. Second, timing—if Team A runs their count on Monday and Team B runs theirs on Thursday, you're comparing snapshots from different moments. Third, attribution—manual counting often loses the connection between commits and authors, which matters when you're evaluating individual contributions for performance reviews.

The automated pipeline eliminates all three problems by centralizing the logic, timestamping every run, and preserving author-level granularity in the CSV structure.

Building the GitHub Actions Workflow: Failed Attempts vs. Working Solution

Attempt One: Shell Scripts in CI (Too Fragile)

Our first version was a bash script that cloned each repository, ran cloc against it, and appended results to a CSV. It worked for three repositories. At twelve repositories, the job started timing out. At twenty, the CSV formatting broke because cloc's output didn't align cleanly with our column structure when certain file types were absent.

The lesson: shell scripts are fine for prototypes, but they don't scale across dozens of repositories with varying language compositions.

Attempt Two: Python Script with GitHub API (Better, but Incomplete)

The second version used PyGithub to pull commit statistics via the API and a Python script to aggregate LOC deltas. This was more robust. But GitHub's API rate limits became a bottleneck when we queried forty-plus repositories in a single run, and the API's commit statistics endpoint doesn't always reflect accurate line counts for squashed merges.

The Working Solution: Scheduled Action with Local Clone and Cloc

What finally worked was a hybrid approach. A scheduled GitHub Action runs nightly. It checks out each target repository, runs cloc with a consistent configuration file, parses the JSON output, and writes a structured CSV row per repository. The CSV is committed to a metrics repository and also uploaded as a workflow artifact.

Here's the core of the workflow:

name: LOC Metrics Collection
on:
  schedule:
    - cron: '0 6 * * *'
  workflow_dispatch:

jobs:
  collect-loc:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout metrics repo
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install cloc
        run: sudo apt-get install -y cloc

      - name: Collect LOC data
        run: python scripts/collect_loc.py
        env:
          GH_TOKEN: ${{ secrets.GH_TOKEN }}

      - name: Commit CSV
        run: |
          git config user.name "metrics-bot"
          git config user.email "metrics-bot@company.com"
          git add data/loc_metrics.csv
          git commit -m "Update LOC metrics $(date -u +'%Y-%m-%d')"
          git push

The Python script iterates over a repository list, clones each one shallowly, runs cloc --json, extracts the relevant fields, and appends a row to the CSV. Each row includes: repository name, date, total lines, code lines, comment lines, blank lines, primary language, and a breakdown by language.

Structuring the CSV: Generic Output vs. Remote-Team-Ready Schema

The Wrong Schema

Our first CSV had three columns: repo, date, total_lines. It was technically correct and practically useless. Remote team leads couldn't tell which language dominated a repository. Finance couldn't separate generated code from hand-written code. Engineering directors couldn't see comment-to-code ratios, which we use as a rough proxy for documentation health.

The Working Schema

The final CSV structure has twelve columns. Here's a sample row from our production output:

repo_name,collection_date,branch,total_lines,code_lines,comment_lines,blank_lines,primary_language,primary_language_pct,secondary_language,author_count,last_30d_commits
payments-service,2024-10-15,main,48217,38104,6203,3910,TypeScript,79.1,SQL,12,147

That single row tells a remote team lead more than a screenshot ever could. The primary_language_pct field lets them compare codebase composition across teams without opening the repository. The author_count and last_30d_commits fields give immediate context for whether the LOC number represents active development or a legacy codebase in maintenance mode.

Why Column Order Matters for Distributed Consumption

This sounds trivial. It isn't. When team leads in Bangalore open the CSV at 9 AM local time and team leads in San Francisco open it at 5 PM their time, they need to parse the same file identically. We learned this the hard way: an early version reordered columns based on which languages were present in a given run. Two team leads ended up comparing different columns, thinking they were looking at the same metric. Now the column order is fixed in the script, and missing values are explicitly written as N/A rather than left blank.

Distribution: Email Attachments vs. Self-Service Access

The Wrong Distribution Model

Initially, the Action emailed the CSV to a mailing list of engineering managers every morning. This created three problems. First, inboxes got cluttered—nobody needs a daily LOC update. Second, the email attachment was often blocked by corporate mail filters for the Bangalore team. Third, historical data lived in scattered inboxes, making trend analysis impossible.

The Working Distribution Model

The CSV now lives in a dedicated metrics repository with a clean directory structure:

data/
  loc_metrics/
    2024/
      10/
        loc_metrics_2024-10-15.csv
        loc_metrics_2024-10-16.csv
      09/
        loc_metrics_2024-09-01.csv

Every remote team lead has read access to this repository. They can pull historical data, run their own analyses, and generate reports on their own schedule. We also maintain a rolling current_quarter.csv that appends daily snapshots, so quarterly reviews don't require manual aggregation.

The metrics repository itself serves as the single source of truth. When the VP asks for quarterly numbers, the answer is a link to a file, not a freshly generated spreadsheet built under time pressure.

Lessons from Six Months of Automated LOC Collection

What the Numbers Actually Told Us

After running this pipeline for six months across forty-seven repositories, patterns emerged that manual counting had never revealed. Our payments-service repository grew by 14,203 lines in Q3, but 11,800 of those were generated protobuf files. The actual hand-written code growth was 2,403 lines. Without the language breakdown in the CSV, we would have reported the larger number and fundamentally mischaracterized the team's output.

Conversely, our infrastructure-as-code repository showed negative LOC growth—net negative 1,847 lines over the quarter. That's not a regression. It's a signal that the team successfully consolidated redundant Terraform modules. The CSV made this visible; the old manual process would have flagged it as a problem requiring investigation.

What We Got Wrong and Fixed

Three corrections worth noting for anyone implementing a similar pipeline:

First, we initially counted all branches. This inflated numbers because feature branches duplicated mainline code. The fix: count only the default branch unless explicitly flagged otherwise.

Second, we didn't exclude vendored code in early runs. A single committed node_modules directory added 340,000 phantom lines to one repository's count. The fix: a configuration file that skips vendor/, node_modules/, and dist/ directories consistently across all repositories.

Third, we ran the job at midnight UTC, which happened to coincide with peak CI usage for our Bangalore team's deployment window. The fix: moving the schedule to 6 AM UTC, which falls in a low-activity window across all four time zones.

The Broader Principle

Code metrics automation for remote teams isn't really about counting lines of code. It's about creating a shared, trustworthy data layer that every team can reason over without coordinating a meeting. The CSV is the interface. The GitHub Action is the engine. The discipline of consistent schema and reliable scheduling is what makes the whole system credible.

When a distributed engineering organization can trust its own numbers, conversations about productivity stop being arguments about methodology and start being discussions about the work itself. That shift—from debating the data to acting on it—is the actual return on investment. The eleven hours our engineering manager recovered each quarter were a welcome bonus, but they were never the point.