Skip to content
BranchesTags

Repository files navigation

puppetsync

Description

Run Puppet Bolt Plans to manage your GitHub repos' code like infrastructure-as-code!

  • Enforce a "baseline" across multiple GitHub repos, using Puppet and Bolt tasks
  • Automate the CM/workflow for each change: git commit messages, GitHub forks & PRs
  • Separate plans to submit, approve, and merge GitHub PRs, so different roles can run them (if required by CM policy)

Puppetsync Plans Overview

Setup

Requirements

  • Puppet Bolt 3.0+, installed from an OS package (don't use the RubyGem)
  • The git command must be available
    • SSH + ssh-agent must be set up to push changes
  • Some specific environment variables are required to handle API authentication (e.g., GitHub)
  • Runtime dependencies (installed by ./Rakefile install)
    • Puppet modules (defined in bolt project's bolt-project.yaml):
    • Ruby Gems (defined in gem.deps.rb): octokit, etc

Quickstart

Initial setup

  1. Before running any plans, from the top level of this repository:

    command -v rvm && rvm use system    # Make sure you're using packaged `bolt`
./Rakefile install                  # Install Puppet module + Ruby Gem deps;
bolt plan show                      # Verify `puppetsync::` plans are visible

  2. Set the environment variable GITHUB_API_TOKEN.

  3. At this point, you are ready to run a plan. The plan to run will depend on your role:

    Plan Role Purpose
    puppetsync Maintainer Applies baseline to each repo, submits changes as PRs
    puppetsync::approve_github_prs Approver Approves all PRs from a specific puppetsync session
    puppetsync::merge_github_prs Maintainer Merges all approved PRs from a specific puppetsync

Preparing to run puppetsync

⚠️ You only need to change these files when preparing a new puppetsync!

  1. Find/add a repolist file for the repos you want to affect under data/sync/repolists/
  2. Copy and customize a Puppetsync config file under data/sync/configs/ to fit your workflow
  3. Set the environment variable for GitHub API authentication
  4. (Optional) Develop Puppet code/Hiera data/Tasks to provide new features

Note: If you are just approving or merging PRs, you only need the config file from the puppetsync session (the plans find the session's open PRs by its git.feature_branch).

Running puppetsync

# (PROTIP: don't actually expose API tokens on the CLI when running commands)

GITHUB_API_TOKEN=$GITHUB_API_TOKEN \
    bolt plan run puppetsync config={CONFIG_NAME} repolist={REPOLIST_NAME}

See the puppetsync reference for details.

Running puppetsync::approve_github_prs

GITHUB_API_TOKEN=$GITHUB_API_TOKEN \
    bolt plan run puppetsync::approve_github_prs

See the puppetsync::approve_github_prs reference for details.

Running puppetsync::merge_github_prs

GITHUB_API_TOKEN=$GITHUB_API_TOKEN \
    bolt plan run puppetsync::merge_github_prs

See the puppetsync::merge_github_prs reference for details.

Running puppetsync::set_github_pr_state

GITHUB_API_TOKEN=$GITHUB_API_TOKEN \
    bolt plan run puppetsync::set_github_pr_state state=draft   # or state=ready

See the puppetsync::set_github_pr_state reference for details.

Usage

Syncing repos

After setup, sync all repos by running:

    /opt/puppetlabs/bin/bolt plan run puppetsync \
      config=CONFIG_NAME repolist=REPOLIST_NAME

If the config and repolist's latest.yaml files are symlinked to the target config and repolist, you don't have to specify config= or repolist= (for brevity, following examples will assume this is the case):

    /opt/puppetlabs/bin/bolt plan run puppetsync

To see what's going on under the hood (potentially less irritating when apply() appears to hang for a long time when updating a lot of repos):

    /opt/puppetlabs/bin/bolt plan run puppetsync --log-level info

    # Alternatively (warning: LOTS of info):
    /opt/puppetlabs/bin/bolt plan run puppetsync --log-level debug

Inspecting pipeline stages

To list all pipeline stages in a plan (and inspect which stages will be skipped), run:

    bolt plan run puppetsync options='{"list_pipeline_stages": true}' \
      config=CONFIG_NAME repolist=REPOLIST_NAME

These steps can be specified/commented out in the Puppetsync config file, under the corresponding plan.

Example:

bolt plan run puppetsync \
  options='{"list_pipeline_stages": true}' \
  github_token=x

#   Starting: plan puppetsync
#   ===== SKIPPING PIPELINE STAGE DUE TO CONFIGURATION: install_gems
#   - checkout_git_feature_branch_in_each_repo
#   - apply_puppet_role
#   - git_commit_changes
#   - ensure_github_fork
#   - ensure_git_remote
#   - git_push_to_remote
#   - ensure_github_pr
#   Finished: plan puppetsync in 0.04 sec
#   Plan completed successfully with no result

At the time the command above was run, the corresponding Puppetsync config file contained the following puppetsync.plans.sync.stages:

puppetsync:
  plans:
    sync:
      stages:
        # - install_gems
        - checkout_git_feature_branch_in_each_repo
        - apply_puppet_role
        - git_commit_changes
        - ensure_github_fork
        - ensure_git_remote
        - git_push_to_remote
        - ensure_github_pr

Reference

Environment variables

These environment variables are necessary to fork GitHub repositories and submit Pull Requests:

Env variable Purpose
GITHUB_API_TOKEN GitHub API token

(Recommended) To stop Bolt from collecting analytics, set this environment variable:

Env variable Purpose
BOLT_DISABLE_ANALYTICS=true Prevent bolt's analytics from phoning home to tell Puppet about everything you do

Puppetsync config

The workflow of a specific Puppetsync session (sync -> apply -> merge of related PRs) is controlled by a single configuration data structure, defined in Hiera using the key puppetsync::plan_config.

Typically, the puppetsync::plan_config data structure is defined in its own Hiera YAML file, located at data/sync/configs/{CONFIG_NAME}.yaml. The Hiera file's name is is the CONFIG_NAME in a Puppetsync plan's config=CONFIG_NAME

Example:

---
puppetsync::plan_config:
  permitted_project_types:
    - pupmod
    - pupmod_skeleton
  plans:
    # clone_git_repos: false     # set to `false` when applying manual updates on a second run
    # clear_before_clone: false  # set to `false` when applying manual updates on a second run
    sync:
      stages:
        - install_gems
        - checkout_git_feature_branch_in_each_repo
        - apply_puppet_role
        - git_commit_changes
        - ensure_github_fork
        - ensure_git_remote
        - git_push_to_remote
        - ensure_github_pr

    approve_github_pr:
      clone_git_repos: false
      stages:
        - install_gems
        - approve_github_pr_for_each_repo

    merge_github_pr:
      clone_git_repos: false
      stages:
        - install_gems
        - merge_github_pr_for_each_repo

git:
  feature_branch: SIMP-1234
  commit_message: |
    Update to new CI pipeline

    This patch updates the CI pipeline in %COMPONENT% to the current
    standardized baseline.

github:
  pr_user: op-ct  # This should be the account that *submitted* the PRs (Used
                  # by idempotency checks when approving/merging PRs)
  approval_message: ':+1: lgtm'
  draft: true     # (optional) create the session's PRs as drafts. Applies at
                  # creation time only; flip existing PRs with the
                  # puppetsync::set_github_pr_state plan

Puppetsync repolist

Data about each repo/branch to target. The data is defined in a Hiera YAML file, located at data/sync/repolists/{REPOLIST_NAME}.yaml. Each repolist is named after its file.

puppetsync::repos_config:

  https://github.com/simp/pupmod-simp-acpid:
    branch: master

  https://github.com/simp/pupmod-simp-aide:
    branch: master

  https://github.com/simp/pupmod-simp-at:
    branch: master

  https://github.com/simp/pupmod-simp-auditd:
    branch: master

  # ... and so on

Dynamic repolists (GitHub org inventory)

Instead of hand-maintaining the repo list, a repolist file can define puppetsync::repos_source to build it from the GitHub API at run time (see data/sync/repolists/github-org.yaml for a ready-to-use example):

puppetsync::repos_source:
  org: simp
  include: ['pupmod-*', 'puppet-*', 'rubygem-*']  # name globs
  include_forks:                                  # forks the org maintains
    - rubygem-simp-rspec-puppet-facts
    - pupmod-voxpupuli-selinux
  • Archived and empty repos are excluded; forks are excluded unless they match include_forks
  • Repos with the puppetsync-ignore GitHub topic are excluded — set the topic to opt a repo out without touching puppetsync
  • Each repo's branch comes from the API's default branch (so e.g. pupmod-voxpupuli-selinux syncs simp-master automatically)
  • An informational snapshot of the generated list is written to data/sync/repolists/generated-{CONFIG_NAME}.yaml (gitignored). The approve/merge plans don't need it — they enumerate the session's open PRs directly by feature branch

Plans

Each plan:

  • Reads its config data from the Puppetsync config file
  • Reads its repolist data from the Puppetsync repolist file
  • Has its own specific configuration under the keys (plans.sync, plans.approve_github_pr, and plans.merge_github_pr)
  • Executes its workflow as a series of pipeline stages for each repo in the repolist file (in Hiera at data/sync/configs/{CONFIG_FILE}.yaml).

puppetsync

The main plan (puppetsync) clones and updates each repo in the Puppetsync repolist. It (idempotently) ensures a GitHub PR exists for each change.

Workflow:

  1. Clone :git repositories defined in the Puppetsync repolist
    • (disable with clone_git_repos: false)

It will then execute the following pipeline stages for each repo (in parallel):

  1. Check out a new git feature branch
  2. Apply Puppet manifests to enforce a common repository asset baseline
  3. Commit changes to git with a templated commit message (git.commit_message)
  4. Ensure the user has forked repository on GitHub
  5. Push changes up to the user's forked repository
  6. Submit a Pull Request to merge the changes back the original repository and branch

If an individual repo encounters failures during a stage, it will be held back while the other repos proceed with their workflows.

Repos that need no changes are safe to sync: they are reported as unchanged and skip the GitHub fork/push/PR stages.

All failures are summarized after the full plan finishes executing.

puppetsync::approve_github_prs

Idempotently approves every open PR from user github.pr_user on branch git.feature_branch, enumerated directly from the GitHub API (no repolist needed).

puppetsync::merge_github_prs

Idempotently merges every approved PR from user github.pr_user on branch git.feature_branch, enumerated directly from the GitHub API (no repolist needed).

puppetsync::set_github_pr_state

Idempotently marks every open PR from user github.pr_user on branch git.feature_branch as draft or ready (required state parameter), enumerated directly from the GitHub API (no repolist needed). Pairs with the config key github.draft, which controls the state PRs are created with during a sync.

Manually installing dependencies

Use bolt to download the project's dependencies from bolt-project.yaml and gems.deps.rb:

   /opt/puppetlabs/bolt/bin/gem install --user-install -g gem.deps.rb
   /opt/puppetlabs/bin/bolt module install

The Rakefile can be used as a shortcut:

  ./Rakefile install

Run ./Rakefile -T & ./Rakefile -D to see other tasks & descriptions

Troubleshooting

Error: Ignoring <x> because its extensions are not built.

Cause: Running bolt plan run puppetsync from a Ruby interpreter other than the bolt package.

Fix: Make sure you're not using RVM. If necessary, invoke the packaged bolt executable directly:

command -v rvm && rvm use system    # make sure you're using the packaged `bolt`
./Rakefile install                  # Install Puppet module and Ruby Gem deps
bolt plan show --filter puppetsync  # Validate bolt is working

## ^^^ If that still didn't work:
# /opt/puppetlabs/bolt/bin/bolt show --filter puppetsync

Characteristic error messages:

Ignoring bcrypt_pbkdf-1.0.1 because its extensions are not built. Try: gem pristine bcrypt_pbkdf --version 1.0.1
Ignoring byebug-11.1.3 because its extensions are not built. Try: gem pristine byebug --version 11.1.3
Ignoring byebug-11.1.1 because its extensions are not built. Try: gem pristine byebug --version 11.1.1
Ignoring byebug-11.0.1 because its extensions are not built. Try: gem pristine byebug --version 11.0.1
Ignoring ed25519-1.2.4 because its extensions are not built. Try: gem pristine ed25519 --version 1.2.4
Ignoring executable-hooks-1.6.0 because its extensions are not built. Try: gem pristine executable-hooks --version 1.6.0
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
...

Limitations

  • Requires git to be configured with SSH, with keys loaded into a running agent
  • Probably only works from an *nix host

About

Bolt project to standardize assets across many GitHub repos and PR the changes (Should probably be called "boltsync")

Topics

baselineboltgithacktoberfestpuppet

Resources

Readme
Activity
Custom properties

Stars

1 star

Watchers

6 watching

Forks

5 forks
Report repository

Releases

Packages

Used by

Contributors

Languages