diff --git a/MAINTAINING.md b/MAINTAINING.md index d221fc099..d178e0e29 100644 --- a/MAINTAINING.md +++ b/MAINTAINING.md @@ -153,13 +153,36 @@ and publishes. Do this when the new major becomes the stable release. -1. **Repoint the `latest` symlink:** +1. **Port content drift from the old version.** The Phase 1 copy is frozen at the + moment it was made: edits merged to the old collection afterward are absent from + the new one, and nothing surfaces them — they merge without conflict and the new + collection's pages are simply older. Find the copy point (the commit that added + the new collection) and list every old-collection change since: + + ```console + git log $(git log --diff-filter=A --format=%H -1 -- docs/_openvox_9x/index.md)..master \ + -- docs/_openvox_8x _data/nav/openvox_8x.yml + ``` + + Port each relevant change onto the new collection — a path-rewritten apply + usually works as-is: + + ```console + git show -- docs/_openvox_8x | sed 's|_openvox_8x|_openvox_9x|g' | git apply + ``` + + Review ported content for version-specific prose, same as the Phase 1 sweep. + Re-run this check immediately before merging the promotion PR; if a docs freeze + is ever warranted, it only needs to cover the window between that final check + and the merge. + +2. **Repoint the `latest` symlink:** ```console ln -sfn _openvox_9x docs/_openvox_latest ``` -2. **Point the `latest` collection's navigation at the new version.** The +3. **Point the `latest` collection's navigation at the new version.** The `//latest/` pages belong to the `openvox_latest` collection, so their nav has to move from 8.x to 9.x: - In `_config.yml`, change the `openvox_latest` defaults scope from @@ -177,17 +200,17 @@ Do this when the new major becomes the stable release. base: /openvox/latest/ ``` -3. **In `_data/products.yml`:** set the OpenVox `latest:` to `9x` (a targeted +4. **In `_data/products.yml`:** set the OpenVox `latest:` to `9x` (a targeted per-product edit — don't sweep every product's `latest:`), and **freeze 8.x** by pinning its `ref:` to its final 8.x tag (so the frozen collection stays reproducible). -4. **No-redirect check:** the site has no redirect mechanism. Once `latest` points at +5. **No-redirect check:** the site has no redirect mechanism. Once `latest` points at 9.x, any page **removed or renamed** in 9.x will 404 at `/openvox/latest/` for `latest` bookmarks (the content still lives at `/openvox/8.x/`). Diff the 8.x vs 9.x page sets and decide how to handle removed pages before promoting. -5. Rebuild and verify: `/openvox/latest/` now serves the 9.x content, `/openvox/8.x/` +6. Rebuild and verify: `/openvox/latest/` now serves the 9.x content, `/openvox/8.x/` stays frozen, and the version selector marks 9.x as `latest`. ### Rollback diff --git a/_config.yml b/_config.yml index 38826d6c6..1509f8ed8 100644 --- a/_config.yml +++ b/_config.yml @@ -59,6 +59,9 @@ collections: openvox_8x: output: true permalink: '/openvox/8.x/:path:output_ext' + openvox_9x: + output: true + permalink: '/openvox/9.x/:path:output_ext' openvox-server_latest: output: true @@ -118,6 +121,11 @@ defaults: type: openvox_8x values: nav: openvox_8x + - scope: + path: '' + type: openvox_9x + values: + nav: openvox_9x - scope: path: '' diff --git a/_data/nav/openvox_9x.yml b/_data/nav/openvox_9x.yml new file mode 100644 index 000000000..544df48fa --- /dev/null +++ b/_data/nav/openvox_9x.yml @@ -0,0 +1,431 @@ +--- +- text: OpenVox 9 Platform + items: + - text: OpenVox release notes + link: "release_notes.html" + - text: OpenVox known issues + link: "known_issues.html" + - text: System requirements + link: "system_requirements.html" + - text: About openvox-agent + link: "about_agent.html" + - text: Component versions in recent releases + link: "component_versions.html" + - text: Supported platforms + link: "supported_platforms.html" +- text: Getting started + items: + - text: Getting started with OpenVox + link: "getting_started.html" + - text: Try OpenVox locally with crafty (experimental) + link: "getting_started_local.html" + - text: Manage a service with a module + link: "manage_a_service_with_a_module.html" +- text: Installing and upgrading + items: + - text: Overview of OpenVox's architecture + link: "architecture.html" + - text: Before you begin + link: "install_pre.html" + - text: OpenVox repositories and packages + link: "openvox_platform.html" + - text: 'Install agent: Linux' + link: "install_linux.html" + - text: 'Install agent: Windows' + link: "install_windows.html" + - text: 'Install agent: macOS' + link: "install_osx.html" + - text: What gets installed and where (agent) + link: "install_what_and_where.html" + - text: Upgrading OpenVox 9 + link: "upgrade_minor.html" +- text: Configuration + items: + - text: About OpenVox's settings + link: "config_about_settings.html" + - text: Short list of important settings + link: "config_important_settings.html" + - text: 'puppet.conf: The main config file' + link: "config_file_main.html" + - text: 'environment.conf: Per-environment settings' + link: "config_file_environment.html" + - text: 'auth.conf (LEGACY): HTTPS authorization' + link: "config_file_auth.html" + - text: 'fileserver.conf: Custom fileserver mount points' + link: "config_file_fileserver.html" + - text: 'puppetdb.conf: OpenVoxDB server locations' + link: "config_file_puppetdb.html" + - text: 'hiera.yaml: Data lookup configuration' + link: "hiera_config_yaml_5.html" + - text: 'autosign.conf: Basic certificate autosigning' + link: "config_file_autosign.html" + - text: 'csr_attributes.yaml: Certificate extensions' + link: "config_file_csr_attributes.html" + - text: 'custom_trusted_oid_mapping.yaml: Short names for cert extension OIDs' + link: "config_file_oid_map.html" + - text: 'device.conf: Network hardware access' + link: "config_file_device.html" + - text: 'routes.yaml: Advanced plugin routing' + link: "config_file_routes.html" + - text: Checking values of settings + link: "config_print.html" + - text: Editing settings on the command line + link: "config_set.html" + - text: Complete list of settings (configuration reference) + link: "configuration.html" +- text: Important directories and files + items: + - text: Code and data directory (codedir) + link: "dirs_codedir.html" + - text: Config directory (confdir) + link: "dirs_confdir.html" + - text: The main manifest + link: "dirs_manifest.html" + - text: The modulepath + link: "dirs_modulepath.html" + - text: The SSLdir + link: "dirs_ssldir.html" + - text: Cache directory (vardir) + link: "dirs_vardir.html" +- text: Environments + items: + - text: About environments + link: "environments_about.html" + - text: Creating environments + link: "environments_creating.html" + - text: Environment isolation + link: "environment_isolation.html" +- text: Modules + items: + - text: Fundamentals + link: "modules_fundamentals.html" + - text: Beginner's guide to writing modules + link: "bgtm.html" + - text: Scaffolding modules with Jig + link: "/ecosystem/latest/devkit/jig.html" + - text: Plugins in modules + link: "plugins_in_modules.html" + - text: Installing modules + link: "modules_installing.html" + - text: Publishing modules on the Puppet Forge + link: "modules_publishing.html" + - text: Module metadata and metadata.json + link: "modules_metadata.html" + - text: Documenting modules + link: "modules_documentation.html" + - text: OpenVox Strings + link: "openvox_strings.html" + - text: OpenVox Strings style guide + link: "openvox_strings_style.html" + - text: Module cheat sheet + link: "cheatsheet_module.html" +- text: OpenVox's services and tools + items: + - text: OpenVox's commands + link: "services_commands.html" + - text: Running OpenVox commands on Windows + link: "services_commands_windows.html" + - text: OpenVox agent on *nix + link: "services_agent_unix.html" + - text: OpenVox agent on Windows + link: "services_agent_windows.html" + - text: Puppet apply + link: "services_apply.html" +- text: The Puppet language + items: + - text: Visual index + link: "lang_visual_index.html" + - text: Basics + link: "lang_summary.html" + - text: Puppet language style guide + link: "style_guide.html" + - text: Handling file paths on Windows + link: "lang_windows_file_paths.html" + - text: Variables + link: "lang_variables.html" + - text: Resources + link: "lang_resources.html" + - text: Resources (advanced) + link: "lang_resources_advanced.html" + - text: Relationships and ordering + link: "lang_relationships.html" + - text: Classes + link: "lang_classes.html" + - text: Defined resource types + link: "lang_defined_types.html" + - text: Type aliases + link: "lang_type_aliases.html" + - text: Expressions and operators + link: "lang_expressions.html" + - text: Conditional statements and expressions + link: "lang_conditional.html" + - text: Function calls + link: "lang_functions.html" + - text: Docs for built-in functions + link: "function.html" + - text: Node definitions + link: "lang_node_definitions.html" + - text: Comments + link: "lang_comments.html" + - text: Facts and built-in variables + link: "lang_facts_and_builtin_vars.html" + - text: Reserved words and acceptable names + link: "lang_reserved.html" + - text: Writing functions (Puppet language) + link: "lang_write_functions_in_puppet.html" + - text: Custom types + link: "custom_types.html" + - text: Provider development + link: "provider_development.html" + - text: About values and data types + link: "lang_data.html" + - text: Strings + link: "lang_data_string.html" + - text: Numbers + link: "lang_data_number.html" + - text: Booleans + link: "lang_data_boolean.html" + - text: Arrays + link: "lang_data_array.html" + - text: Hashes + link: "lang_data_hash.html" + - text: Regular expressions + link: "lang_data_regexp.html" + - text: URI + link: "lang_data_uri.html" + - text: Binary + link: "lang_data_binary.html" + - text: Timestamp and Timespan + link: "lang_data_time.html" + - text: SemVer and SemVerRange + link: "lang_data_semver.html" + - text: Error + link: "lang_data_error.html" + - text: Sensitive + link: "lang_data_sensitive.html" + - text: Undef + link: "lang_data_undef.html" + - text: Resource references + link: "lang_data_resource_reference.html" + - text: Resource types + link: "lang_data_resource_type.html" + - text: Default + link: "lang_data_default.html" + - text: Data type syntax + link: "lang_data_type.html" + - text: Abstract data types + link: "lang_data_abstract.html" + - text: Typecasting + link: "lang_typecasting.html" + - text: Using templates + link: "lang_template.html" + - text: Embedded Puppet (EPP) template syntax + link: "lang_template_epp.html" + - text: Embedded Ruby (ERB) template syntax + link: "lang_template_erb.html" + - text: Iteration and loops + link: "lang_iteration.html" + - text: Lambdas (code blocks) + link: "lang_lambdas.html" + - text: Resource defaults + link: "lang_defaults.html" + - text: Resource collectors + link: "lang_collectors.html" + - text: Virtual resources + link: "lang_virtual.html" + - text: Exported resources + link: "lang_exported.html" + - text: Tags + link: "lang_tags.html" + - text: Run stages + link: "lang_run_stages.html" + - text: Containment of resources + link: "lang_containment.html" + - text: Scope + link: "lang_scope.html" + - text: Namespaces and autoloading + link: "lang_namespaces.html" +- text: Writing custom functions + items: + - text: Introduction to writing functions + link: "functions_basics.html" + - text: Writing functions (Puppet language) + link: "lang_write_functions_in_puppet.html" + - text: Overview and examples + link: "functions_ruby_overview.html" + - text: Defining signatures + link: "functions_ruby_signatures.html" + - text: Special features in implementation methods + link: "functions_ruby_implementation.html" + - text: Documenting Ruby functions + link: "functions_ruby_documenting.html" + - text: Writing functions (legacy Ruby API --- avoid) + link: "functions_legacy.html" +- text: Designing system configs (roles and profiles) + items: + - text: The roles and profiles method + link: "the_roles_and_profiles_method.html" + - text: Roles and profiles example + link: "roles_and_profiles_example.html" + - text: Designing advanced profiles + link: "designing_advanced_profiles.html" + - text: Designing convenient roles + link: "designing_convenient_roles.html" +- text: Hiera + items: + - text: About Hiera + link: "hiera_intro.html" + - text: Interactive Hiera Demo + link: "hiera_demo.html" + - text: Getting started with Hiera + link: "hiera_quick.html" + - text: Configuring Hiera + link: "hiera_config_yaml_5.html" + - text: Creating and editing data + link: "hiera_merging.html" + - text: Looking up data with Hiera + link: "hiera_automatic.html" + - text: Writing new data backends + link: "hiera_custom_backends.html" + - text: Upgrading to Hiera 5 + link: "hiera_migrate.html" +- text: Resource types + items: + - text: Overview + link: "types/overview.html" + - text: All resource types (single-page reference) + link: "type.html" + - text: Core types cheat sheet + link: "cheatsheet_core_types.html" + - text: Optional resource types for Windows + link: "resources_windows_optional.html" + - text: Augeas tips and examples + link: "resources_augeas.html" + - text: exec + link: "types/exec.html" + - text: Exec tips and examples for Windows + link: "resources_exec_windows.html" + - text: file + link: "types/file.html" + - text: File tips and examples for Windows + link: "resources_file_windows.html" + - text: filebucket + link: "types/filebucket.html" + - text: group + link: "types/group.html" + - text: User and group tips and examples for Windows + link: "resources_user_group_windows.html" + - text: notify + link: "types/notify.html" + - text: package + link: "types/package.html" + - text: Package tips and examples for Windows + link: "resources_package_windows.html" + - text: resources + link: "types/resources.html" + - text: schedule + link: "types/schedule.html" + - text: Scheduled task tips and examples + link: "resources_scheduled_task_windows.html" + - text: service + link: "types/service.html" + - text: Service tips and examples + link: "resources_service.html" + - text: stage + link: "types/stage.html" + - text: tidy + link: "types/tidy.html" + - text: user + link: "types/user.html" + - text: User and group tips and examples for Windows + link: "resources_user_group_windows.html" +- text: 'Reports: Tracking OpenVox''s activity' + items: + - text: About reporting + link: "reporting_about.html" + - text: Built-in report processors + link: "report.html" + - text: Writing custom report processors + link: "reporting_write_processors.html" + - text: Puppet::Transaction::Report object format + link: "format_report.html" +- text: Extensions for assigning classes to nodes + items: + - text: External node classifiers + link: "nodes_external.html" +- text: Misc. references + items: + - text: Settings (configuration reference) + link: "configuration.html" + - text: Functions + link: "function.html" + - text: Metaparameters + link: "metaparameter.html" + - text: Built-in report processors + link: "report.html" + - text: Indirections and termini + link: "indirection.html" + - text: Ruby API for developing extensions + link: "yard/" +- text: Man pages + items: + - text: Index + link: "man/overview.html" + - text: puppet agent + link: "man/agent.html" + - text: puppet apply + link: "man/apply.html" + - text: puppet module + link: "man/module.html" + - text: puppet resource + link: "man/resource.html" + - text: puppet lookup + link: "man/lookup.html" + - text: puppet config + link: "man/config.html" + - text: puppet describe + link: "man/describe.html" + - text: puppet device + link: "man/device.html" + - text: puppet doc + link: "man/doc.html" + - text: puppet help + link: "man/help.html" + - text: puppet node + link: "man/node.html" + - text: puppet parser + link: "man/parser.html" + - text: puppet plugin + link: "man/plugin.html" + - text: puppet catalog + link: "man/catalog.html" + - text: puppet facts + link: "man/facts.html" + - text: puppet filebucket + link: "man/filebucket.html" + - text: puppet report + link: "man/report.html" +- text: HTTP API + items: + - text: OpenVox Server HTTP API + link: "/openvox-server/latest/http_api_index.html" +- text: SSL and certificates + items: + - text: Using an external CA + link: "config_ssl_external_ca.html" + - text: Configuring autosigning + link: "ssl_autosign.html" + - text: CSR attributes and certificate extensions + link: "ssl_attributes_extensions.html" + - text: Regenerating all certificates in a deployment + link: "ssl_regenerate_certificates.html" + - text: Adding a custom CA for HTTPS-inspecting proxies + link: "ssl_custom_ca_proxy.html" +- text: Details about OpenVox's internals + items: + - text: Agent/server HTTPS communications + link: "subsystem_agent_server_comm.html" + - text: Catalog compilation + link: "subsystem_catalog_compilation.html" + - text: Msgpack serialization support + link: "subsystem_msgpack.html" diff --git a/_data/nav_map.yml b/_data/nav_map.yml index 1315751c2..011273e4f 100644 --- a/_data/nav_map.yml +++ b/_data/nav_map.yml @@ -10,6 +10,10 @@ collections: openvox_latest|openvox_8x base: /openvox/latest/ +- nav_key: openvox_9x + collections: openvox_9x + base: /openvox/9.x/ + - nav_key: openvox-server_8x collections: openvox-server_latest|openvox-server_8x base: /openvox-server/latest/ diff --git a/_data/navigation.yml b/_data/navigation.yml index 37b57f493..6576fa864 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -3,7 +3,7 @@ collections: [openfact_latest, openfact_5x] - title: OpenVox url: /openvox/latest/ - collections: [openvox_latest, openvox_8x] + collections: [openvox_latest, openvox_8x, openvox_9x] - title: OpenVox Server url: /openvox-server/latest/ collections: [openvox-server_latest, openvox-server_8x] diff --git a/_data/products.yml b/_data/products.yml index 164e32ce8..5a8873f59 100644 --- a/_data/products.yml +++ b/_data/products.yml @@ -36,6 +36,11 @@ openvox: latest: 8x references: references:openvox versions: + - id: 9x + label: "9.x" + collection: _openvox_9x + base: /openvox/9.x/ + ref: "9.0.0-beta2" - id: 8x label: "8.x" collection: _openvox_8x diff --git a/docs/_openvox_9x/.gitignore b/docs/_openvox_9x/.gitignore new file mode 100644 index 000000000..ec6b67fec --- /dev/null +++ b/docs/_openvox_9x/.gitignore @@ -0,0 +1,15 @@ +# Generated reference pages — copied here by CI (bundle exec rake references:openvox) +configuration.md +function.md +indirection.md +metaparameter.md +report.md +type.md +type.json +type_strings.md +type_strings.json +raw_strings_data_output.json +strings.json +man/ +types/ +types_strings/ diff --git a/docs/_openvox_9x/READMEtemplate.txt b/docs/_openvox_9x/READMEtemplate.txt new file mode 100644 index 000000000..3ab89f1cb --- /dev/null +++ b/docs/_openvox_9x/READMEtemplate.txt @@ -0,0 +1,79 @@ +# modulename + +#### Table of Contents + +1. [Description](#description) +1. [Setup - The basics of getting started with [modulename]](#setup) + * [What [modulename] affects](#what-[modulename]-affects) + * [Setup requirements](#setup-requirements) + * [Beginning with [modulename]](#beginning-with-[modulename]) +1. [Usage - Configuration options and additional functionality](#usage) +1. [Limitations - OS compatibility, etc.](#limitations) +1. [Development - Guide for contributing to the module](#development) + +## Description + +Briefly tell users why they might want to use your module. Explain what your module does and what kind of problems users can solve with it. + +This should be a fairly short description helps the user decide if your module is what they want. + +## Setup + +### What [modulename] affects **OPTIONAL** + +If it's obvious what your module touches, you can skip this section. For example, folks can probably figure out that your mysql_instance module affects their MySQL instances. + +If there's more that they should know about, though, this is the place to mention: + +* Files, packages, services, or operations that the module will alter, impact, or execute. +* Dependencies that your module automatically installs. +* Warnings or other important notices. + +### Setup Requirements **OPTIONAL** + +If your module requires anything extra before setting up (pluginsync enabled, another module, etc.), mention it here. + +If your most recent release breaks compatibility or requires particular steps for upgrading, you might want to include an additional "Upgrading" section here. + +### Beginning with [modulename] + +The very basic steps needed for a user to get the module up and running. This can include setup steps, if necessary, or it can be an example of the most basic use of the module. + +## Usage + +Include usage examples for common use cases in the **Usage** section. Show your users how to use your module to solve problems, and be sure to include code examples. Include three to five examples of the most important or common tasks a user can accomplish with your module. Show users how to accomplish more complex tasks that involve different types, classes, and functions working in tandem. + +## Reference + +This section is deprecated. Instead, add reference information to your code as Puppet Strings comments, and then use Strings to generate a REFERENCE.md in your module. For details on how to add code comments and generate documentation with Strings, see the Puppet Strings [documentation](https://docs.openvoxproject.org/openvox/latest/openvox_strings.html) and [style guide](https://docs.openvoxproject.org/openvox/latest/openvox_strings_style.html) + +If you aren't ready to use Strings yet, manually create a REFERENCE.md in the root of your module directory and list out each of your module's classes, defined types, facts, functions, Puppet tasks, task plans, and resource types and providers, along with the parameters for each. + +For each element (class, defined type, function, and so on), list: + + * The data type, if applicable. + * A description of what the element does. + * Valid values, if the data type doesn't make it obvious. + * Default value, if any. + +For example: + +``` +### `pet::cat` + +#### Parameters + +##### `meow` + +Enables vocalization in your cat. Valid options: 'string'. + +Default: 'medium-loud'. +``` + +## Limitations + +In the Limitations section, list any incompatibilities, known issues, or other warnings. + +## Development + +In the Development section, tell other users the ground rules for contributing to your project and how they should submit their work. diff --git a/docs/_openvox_9x/_environment_conf_settings.md b/docs/_openvox_9x/_environment_conf_settings.md new file mode 100644 index 000000000..d0149b93a --- /dev/null +++ b/docs/_openvox_9x/_environment_conf_settings.md @@ -0,0 +1,8 @@ + +In this version of Puppet, the environment.conf file is only allowed to override five settings: + +- `modulepath` +- `manifest` +- `config_version` +- `environment_timeout` +- `static_catalogs` diff --git a/docs/_openvox_9x/_hiera.yaml_v5.md b/docs/_openvox_9x/_hiera.yaml_v5.md new file mode 100644 index 000000000..ea61e3f4c --- /dev/null +++ b/docs/_openvox_9x/_hiera.yaml_v5.md @@ -0,0 +1,32 @@ + +```yaml +--- +version: 5 +defaults: # Used for any hierarchy level that omits these keys. + datadir: data # This path is relative to hiera.yaml's directory. + data_hash: yaml_data # Use the built-in YAML backend. + +hierarchy: + - name: "Per-node data" # Human-readable name. + path: "nodes/%{trusted.certname}.yaml" # File path, relative to datadir. + # ^^^ IMPORTANT: include the file extension! + + - name: "Per-datacenter business group data" # Uses custom facts. + path: "location/%{facts.whereami}/%{facts.group}.yaml" + + - name: "Global business group data" + path: "groups/%{facts.group}.yaml" + + - name: "Per-datacenter secret data (encrypted)" + lookup_key: eyaml_lookup_key # Uses non-default backend. + path: "secrets/%{facts.whereami}.eyaml" + options: + pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem + pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem + + - name: "Per-OS defaults" + path: "os/%{facts.os.family}.yaml" + + - name: "Common data" + path: "common.yaml" +``` diff --git a/docs/_openvox_9x/_hiera_context_object.md b/docs/_openvox_9x/_hiera_context_object.md new file mode 100644 index 000000000..0d65c429d --- /dev/null +++ b/docs/_openvox_9x/_hiera_context_object.md @@ -0,0 +1,143 @@ + +## The `Puppet::LookupContext` object + +To support caching and other needs, Hiera provides backends a special `Puppet::LookupContext` object, which has several methods you can call for various effects. + +* In [Ruby functions](./functions_ruby_overview.html), this is a normal Ruby object of class `Puppet::LookupContext`, and you can call methods with standard Ruby syntax (like `context.not_found`). +* In [Puppet language functions](./lang_write_functions_in_puppet.html), the context object appears as a special data type (Object) that has methods attached. Right now, there isn't anything else in the Puppet language that acts like this. + + You can call its methods using Puppet's [chained function call syntax](./lang_functions.html#chained-function-calls) with the method name instead of a normal function --- for example, `$context.not_found`. For methods that take a block, use Puppet's lambda syntax (parameters outside block) instead of Ruby's block syntax (parameters inside block). + +The following methods are available: + +* [`not_found()`][method_not], for bailing out of a lookup. +* [`interpolate(value)`][method_interpolate], for handing Hiera interpolation tokens in values. +* [`environment_name()`][method_env], to find out which environment this is. +* [`module_name()`][method_module], to find out which module this is. +* [`cache(key, value)`][method_cache], for caching information between function runs. +* [`cache_all(hash)`][method_cache_all], for caching several things at once. +* [`cached_value(key)`][method_cached], for retrieving cached values. +* [`cache_has_key(key)`][method_haskey], for checking the cache. +* [`cached_entries()`][method_allcached], for dumping the whole cache. +* [`cached_file_data(path) {|content| ...}`][method_cached_file], for high-performance reading of data files. +* [`explain() || { 'message' }`][method_explain], for helpful debug messages. + + +### `not_found()` + +[method_not]: #not_found + +Tells Hiera to move on to the next data source. Call this method when your function can't find a value for a given lookup. **This method does not return.** + +For `data_hash` backends, use this when the requested data source doesn't exist. (If it exists and is empty, return an empty hash.) Missing data sources aren't an issue when using `path`/`glob` settings, but are important for backends that locate their own data sources. + +For `lookup_key` and `data_dig` backends, use this when a requested key isn't present in the data source or the data source doesn't exist. Don't return `undef`/`nil` for missing keys, since that's a legal value that can be set in data. + +### `interpolate(value)` + +[method_interpolate]: #interpolatevalue + +Returns the provided value, but with any Hiera interpolation tokens (like `%{variable}` or `%{lookup('key')}`) replaced by their value. This lets you opt-in to allowing Hiera-style interpolation in your backend's data sources. Works recursively on arrays and hashes; hashes can interpolate into both keys and values. + +In `data_hash` backends, interpolation is automatically supported and you don't need to call this method. + +In `lookup_key` and `data_dig` backends, you **must** call this method if you want to support interpolation; if you don't, Hiera assumes you have your own thing going on. + +### `environment_name()` + +[method_env]: #environment_name + +Returns the name of the environment whose hiera.yaml called the function. Returns `undef` (in Puppet) or `nil` (in Ruby) if the function was called by the global or module layer. + +### `module_name()` + +[method_module]: #module_name + +Returns the name of the module whose hiera.yaml called the function. Returns `undef` (in Puppet) or `nil` (in Ruby) if the function was called by the global or environment layer. + +### `cache(key, value)` + +[method_cache]: #cachekey-value + +Caches a value, in a per-data-source private cache; also returns the cached value. + +On future lookups in this data source, you can retrieve values with `cached_value(key)`. Cached values are immutable, but you can replace the value for an existing key. Cache keys can be anything valid as a key for a Ruby hash. (Notably, this means you can use `nil` as a key.) + +For example, on its first invocation for a given YAML file, the built-in `eyaml_lookup_key` backend reads the whole file and caches it, and then decrypts only the specific value that was requested. +On subsequent lookups into that file, it gets the encrypted value from the cache instead of reading the file from disk again. +It also caches decrypted values, so that it won't have to decrypt again if the same key is looked up repeatedly. + +The cache is also useful for storing session keys or connection objects for backends that access a network service. + +#### Cache lifetime and scope + +Each `Puppet::LookupContext` cache only lasts for the duration of the current catalog compilation; a node can't access values cached for a previous node. + +Hiera creates a separate cache for each *combination of inputs for a function call,* including inputs like `name` that are configured in hiera.yaml but not passed to the function. So not only does each hierarchy level have its own cache, but hierarchy levels that use multiple paths have a separate cache for each path. + +If any inputs to a function change (for example, a path interpolates a local variable whose value changes between lookups), Hiera uses a fresh cache. + + +### `cache_all(hash)` + +[method_cache_all]: #cache_allhash + +Caches all the key/value pairs from a given hash; returns `undef` (in Puppet) or `nil` (in Ruby). + +### `cached_value(key)` + +[method_cached]: #cached_valuekey + +Returns a previously cached value from the per-data-source private cache. Returns `nil` or `undef` if no value with this name has been cached. See [`cache(key, value)`][method_cache] above for more info about how the cache works. + +### `cache_has_key(key)` + +[method_haskey]: #cache_has_keykey + +Checks whether the cache has a value for a given key yet. Returns `true` or `false`. + +### `cached_entries()` + +[method_allcached]: #cached_entries + +Returns everything in the per-data-source cache, as an iterable object. Note that this iterable object isn't a hash; if you want a hash, you can use `Hash($context.all_cached())` (in the Puppet language) or `Hash[context.all_cached()]` (in Ruby). + +### `cached_file_data(path) {|content| ...}` + +[method_cached_file]: #cached_file_datapath-content- + +> **Note:** The header above uses Ruby's block syntax. To call this method in the Puppet language, you would use `cached_file_data(path) |content| { ... }`. + +For best performance, use this method to read files in Hiera backends. + +Returns the content of the specified file, as a string. If an optional block is provided, it passes the content to the block and returns the block's return value. For example, the built-in JSON backend uses a block to parse JSON and return a hash: + +```ruby + context.cached_file_data(path) do |content| + begin + JSON.parse(content) + rescue JSON::ParserError => ex + # Filename not included in message, so we add it here. + raise Puppet::DataBinding::LookupError, "Unable to parse (#{path}): #{ex.message}" + end + end +``` + +On repeated access to a given file, Hiera checks whether the file has changed on disk. If it hasn't, Hiera uses cached data instead of reading and parsing the file again. + +This method **does not** use the same per-data-source caches as `cache(key, value)` and friends. It uses a separate cache that lasts across multiple catalog compilations, and is tied to [Puppet Server's environment cache](/openvox-server/latest/admin-api/v1/environment-cache.html). + +Since the cache can outlive a given node's catalog compilation, do not do any node-specific pre-processing (like calling `context.interpolate`) in this method's block. + +### `explain() { 'message' }` + +[method_explain]: #explain--message- + +> **Note:** The header above uses Ruby's block syntax. To call this method in the Puppet language, you would use `explain() || { 'message' }`. In both cases, the provided block must take zero arguments. + +Adds a message, which appears in debug messages or when using `puppet lookup --explain`. The block provided to this function must return a string. + +This is meant for complex lookups where a function tries several different things before arriving at the value. Note that the built-in backends don't use the `explain` method, and they still have relatively verbose explanations; this is for when you need to go above and beyond that. + +Feel free to not worry about performance when constructing your message; Hiera never executes the explain block unless debugging is enabled. + diff --git a/docs/_openvox_9x/_hiera_options_hash.md b/docs/_openvox_9x/_hiera_options_hash.md new file mode 100644 index 000000000..2e89b6cae --- /dev/null +++ b/docs/_openvox_9x/_hiera_options_hash.md @@ -0,0 +1,39 @@ + +### The options hash + +Hierarchy levels are configured in [hiera.yaml](./hiera_config_yaml_5.html). When calling a backend function, Hiera passes a modified version of that configuration as a hash. + +The options hash contains the following keys: + +* `path` --- The absolute path to a file on disk. Only present if the user set one of the `path`, `paths`, `glob`, or `globs` settings. Hiera ensures the file exists before passing it to the function. + + > **Note:** If your backend uses data files, use the context object's [`cached_file_data` method][method_cached_file] to read them. +* `uri` --- A URI that your function can use to locate a data source. Only present if the user set `uri` or `uris`. Hiera doesn't verify the URI before passing it to the function. +* Every key from the hierarchy level's `options` setting. In your documentation, make sure to list any options your backend requires or accepts. Note that the `path` and `uri` keys are reserved. + +For example: this hierarchy level in hiera.yaml... + +```yaml + - name: "Secret data: per-node, per-datacenter, common" + lookup_key: eyaml_lookup_key # eyaml backend + datadir: data + paths: + - "secrets/nodes/%{trusted.certname}.eyaml" + - "secrets/location/%{facts.whereami}.eyaml" + - "common.eyaml" + options: + pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem + pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem +``` + +...would result in several different options hashes (depending on the current node's facts, whether the files exist, etc.), but they would all resemble the following: + +```ruby +{ + 'path' => '/etc/puppetlabs/code/environments/production/data/secrets/nodes/web01.example.com.eyaml', + 'pkcs7_private_key' => '/etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem', + 'pkcs7_public_key' => '/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem' +} +``` + +In your function's signature, you can validate the options hash by using [the Struct data type](./lang_data_abstract.html#struct) to restrict its contents. In particular, note that you can disable all of the `path` and `glob` settings for your backend by disallowing the `path` key in the options hash. diff --git a/docs/_openvox_9x/_naming_functions.md b/docs/_openvox_9x/_naming_functions.md new file mode 100644 index 000000000..3bfa83f7b --- /dev/null +++ b/docs/_openvox_9x/_naming_functions.md @@ -0,0 +1,41 @@ + +Function names generally resemble these examples: + +* `num2bool` (a function that could come from anywhere) +* `postgresql::acls_to_resource_hash` (a function in the `postgresql` module) +* `environment::hash_from_api_call` (a function in an environment) + +Function names are almost the same as +[class names](./lang_reserved.html#classes-and-defined-resource-types). +They consist of one or more segments; each segment must start with a lowercase letter, and can include: + +* Lowercase letters. +* Numbers. +* Underscores. + +If a name has multiple segments, they are separated by the double-colon (`::`) namespace separator. + +In other words, each segment should match the following regular expression: + + \A[a-z][a-z0-9_]*\Z + +...and the full name should match the following regular expression: + + \A([a-z][a-z0-9_]*)(::[a-z][a-z0-9_]*)*\Z + +Function names can be either _global_ or _namespaced._ + +* Global names have only one segment (like `str2bool`), and can be used in any module or environment. + + Global names are shorter, but they're not guaranteed to be unique --- two modules might use the same function name, in which case OpenVox won't necessarily load the one you want. +* Namespaced names have multiple segments (like `stdlib::str2bool`), and are guaranteed to be unique. The first segment is dictated by the function's location: + * In an environment, it must be the literal word `environment` (like `environment::str2bool`). + * In a module, it must be the module's name (like `stdlib::str2bool`, for a function stored in the `stdlib` module). + + Functions usually only have two name segments, although it's legal to use more. + +Some illegal function names: + +* `6_pack` (must start with a letter) +* `_hash_from_api_call` (must start with a letter) +* `Find-Resource` (can only contain lowercase letters, numbers, and underscores) diff --git a/docs/_openvox_9x/_naming_variables.md b/docs/_openvox_9x/_naming_variables.md new file mode 100644 index 000000000..ab2364aa6 --- /dev/null +++ b/docs/_openvox_9x/_naming_variables.md @@ -0,0 +1,33 @@ + +[qualified_var]: ./lang_variables.html#accessing-out-of-scope-variables + +Variable names begin with a `$` (dollar sign) and are case-sensitive. + +Most variable names **must start** with a lowercase letter or an underscore. The exception is regex capture variables, which are named with only numbers. + +Variable names can include: + +* Uppercase and lowercase letters +* Numbers +* Underscores (`_`) + +If the first character is an underscore, that variable should only be accessed from its own local scope; using qualified variable names where any namespace segment begins with `_` is deprecated. + +Note that [some variable names are reserved.](./lang_reserved.html#reserved-variable-names) + +#### Qualified Variable Names + +[Qualified variable][qualified_var] names are prefixed with the name of their scope and the `::` (double colon) namespace separator. (For example, the `$vhostdir` variable from the `apache::params` class would be `$apache::params::vhostdir`.) + +Optionally, the name of the very first namespace can be empty, representing the top namespace. In previous versions of the Puppet language, this was often used to work around bugs, but it's not necessary in this version. The main use is to indicate to readers that you're accessing a top-scope variable, e.g. `$::is_virtual`. + +#### Regular Expressions For Variable Names + +Short variable names should match the following regular expression: + + \A\$[a-z0-9_][a-zA-Z0-9_]*\Z + +Qualified variable names should match the following regular expression: + + \A\$([a-z][a-z0-9_]*)?(::[a-z][a-z0-9_]*)*::[a-z0-9_][a-zA-Z0-9_]*\Z + diff --git a/docs/_openvox_9x/_nodename_certname.md b/docs/_openvox_9x/_nodename_certname.md new file mode 100644 index 000000000..d85bcfd16 --- /dev/null +++ b/docs/_openvox_9x/_nodename_certname.md @@ -0,0 +1,10 @@ + +[node_name_fact]: ./configuration.html#node_name_fact +[node_name_value]: ./configuration.html#node_name_value + +> #### Note on Non-Certname Node Names +> +> Although it's possible to set something other than the [certname][] as the node name (using either the [`node_name_fact`][node_name_fact] or [`node_name_value`][node_name_value] setting), we don't generally recommend it. +> It allows you to re-use one node certificate for many nodes, but it reduces security, makes it harder to reliably identify nodes, and can interfere with other features. +> +> Setting a non-certname node name is **not officially supported** in Puppet Enterprise. diff --git a/docs/_openvox_9x/_puppet_types_to_ruby_types.md b/docs/_openvox_9x/_puppet_types_to_ruby_types.md new file mode 100644 index 000000000..694d7267c --- /dev/null +++ b/docs/_openvox_9x/_puppet_types_to_ruby_types.md @@ -0,0 +1,26 @@ + +[boolean]: ./lang_data_boolean.html +[undef]: ./lang_data_undef.html +[string]: ./lang_data_string.html +[number]: ./lang_data_number.html +[array]: ./lang_data_array.html +[hash]: ./lang_data_hash.html +[default]: ./lang_data_default.html +[regexp]: ./lang_data_regexp.html +[resource reference]: ./lang_data_resource_reference.html +[lambda]: ./lang_lambdas.html +[data type]: ./lang_data_type.html + +Puppet type | Ruby class +------------------------|------------------------------------------------------------------------------- +[Boolean][] | `Boolean` +[Undef][] | `NilClass` (value `nil`) +[String][] | `String` +[Number][] | subtype of `Numeric` +[Array][] | `Array` +[Hash][] | `Hash` +[Default][] | `Symbol` (value `:default`) +[Regexp][] | `Regexp` +[Resource reference][] | `Puppet::Pops::Types::PResourceType`, or `Puppet::Pops::Types::PHostClassType` +[Lambda][] (code block) | `Puppet::Pops::Evaluator::Closure` +[Data type][] (`Type`) | A type class under `Puppet::Pops::Types`, e.g. `Puppet::Pops::Types::PIntegerType` diff --git a/docs/_openvox_9x/_registered_oids.md b/docs/_openvox_9x/_registered_oids.md new file mode 100644 index 000000000..cfce593d7 --- /dev/null +++ b/docs/_openvox_9x/_registered_oids.md @@ -0,0 +1,35 @@ + +The "ppRegCertExt" OID range contains the following OIDs: + +Numeric ID | Short Name | Descriptive Name +------------------------|--------------------|-------------------------- +1.3.6.1.4.1.34380.1.1.1 | `pp_uuid` | Puppet Node UUID +1.3.6.1.4.1.34380.1.1.2 | `pp_instance_id` | Puppet Node Instance ID +1.3.6.1.4.1.34380.1.1.3 | `pp_image_name` | Puppet Node Image Name +1.3.6.1.4.1.34380.1.1.4 | `pp_preshared_key` | Puppet Node Preshared Key +1.3.6.1.4.1.34380.1.1.5 | `pp_cost_center` | Puppet Node Cost Center Name +1.3.6.1.4.1.34380.1.1.6 | `pp_product` | Puppet Node Product Name +1.3.6.1.4.1.34380.1.1.7 | `pp_project` | Puppet Node Project Name +1.3.6.1.4.1.34380.1.1.8 | `pp_application` | Puppet Node Application Name +1.3.6.1.4.1.34380.1.1.9 | `pp_service` | Puppet Node Service Name +1.3.6.1.4.1.34380.1.1.10 | `pp_employee` | Puppet Node Employee Name +1.3.6.1.4.1.34380.1.1.11 | `pp_created_by` | Puppet Node `created_by` Tag +1.3.6.1.4.1.34380.1.1.12 | `pp_environment` | Puppet Node Environment Name +1.3.6.1.4.1.34380.1.1.13 | `pp_role` | Puppet Node Role Name +1.3.6.1.4.1.34380.1.1.14 | `pp_software_version` | Puppet Node Software Version +1.3.6.1.4.1.34380.1.1.15 | `pp_department` | Puppet Node Department Name +1.3.6.1.4.1.34380.1.1.16 | `pp_cluster` | Puppet Node Cluster Name +1.3.6.1.4.1.34380.1.1.17 | `pp_provisioner` | Puppet Node Provisioner Name +1.3.6.1.4.1.34380.1.1.18 | `pp_region` | Puppet Node Region Name +1.3.6.1.4.1.34380.1.1.19 | `pp_datacenter` | Puppet Node Datacenter Name +1.3.6.1.4.1.34380.1.1.20 | `pp_zone` | Puppet Node Zone Name +1.3.6.1.4.1.34380.1.1.21 | `pp_network` | Puppet Node Network Name +1.3.6.1.4.1.34380.1.1.22 | `pp_securitypolicy` | Puppet Node Security Policy Name +1.3.6.1.4.1.34380.1.1.23 | `pp_cloudplatform` | Puppet Node Cloud Platform Name +1.3.6.1.4.1.34380.1.1.24 | `pp_apptier` | Puppet Node Application Tier +1.3.6.1.4.1.34380.1.1.25 | `pp_hostname` | Puppet Node Hostname + +The "ppAuthCertExt" OID range contains the following OIDs: + +1.3.6.1.4.1.34380.1.3.1 | `pp_authorization` | Certificate Extension Authorization +1.3.6.1.4.1.34380.1.3.13 | `pp_auth_role` | Puppet Node Role Name for Authorization diff --git a/docs/_openvox_9x/about_agent.md b/docs/_openvox_9x/about_agent.md new file mode 100644 index 000000000..7c89f8137 --- /dev/null +++ b/docs/_openvox_9x/about_agent.md @@ -0,0 +1,52 @@ +--- +layout: default +title: "About openvox-agent" +--- + +[release_notes]: ./release_notes.html +[services_agent]: ./services_agent_unix.html +[services_apply]: ./services_apply.html +[openvox_server]: /openvox-server/latest/ +[component_versions]: ./component_versions.html + +`openvox-agent` is the core OpenVox package for managed nodes and standalone use. +It bundles the OpenVox runtime and the dependencies needed to run it, so it is the +package you install on Linux, macOS, and Windows agent nodes. + +## What `openvox-agent` provides + +The package includes: + +- the `puppet` and `facter` commands +- the OpenVox agent service +- the runtime needed for `puppet apply` +- bundled dependencies such as Ruby and OpenSSL + +After installation, you can run the [OpenVox agent service][services_agent] or use +[standalone apply workflows][services_apply] on the same host. + +## Relationship to other packages + +OpenVox is usually installed as a small set of packages: + +- `openvox-agent` for agent nodes and standalone use +- `openvox-server` for catalog compilation and CA services on server nodes +- `openvoxdb` for exported resources, reports, and inventory data +- `openvoxdb-termini` on server nodes that need to talk to OpenVoxDB + +`openvox-server` depends on `openvox-agent`, so server upgrades often pull in a +newer `openvox-agent` package on the server at the same time. + +## Versions and release notes + +The `openvox-agent` package version is a package release number, not a separate +product line. It can change when bundled dependencies or packaging details change, +even if the core OpenVox language and command behavior stay the same. + +For package-specific changes, see the [OpenVox release notes][release_notes] and the +latest [OpenVox Server documentation][openvox_server]. + +For the bundled versions of OpenFact, Ruby, OpenSSL, and curl in each +`openvox-agent` release, see +[Component versions in recent OpenVox releases][component_versions], which also +covers the server and database components. diff --git a/docs/_openvox_9x/architecture.markdown b/docs/_openvox_9x/architecture.markdown new file mode 100644 index 000000000..d4a68a05c --- /dev/null +++ b/docs/_openvox_9x/architecture.markdown @@ -0,0 +1,147 @@ +--- +layout: default +title: "Overview of OpenVox's architecture" +--- + +[agent_unix]: services_agent_unix.html +[agent_win]: services_agent_windows.html +[https_walkthrough]: subsystem_agent_server_comm.html +[server_http]: /openvox-server/latest/http_api_index.html +[auth.conf]: config_file_auth.html +[catalog_compilation]: subsystem_catalog_compilation.html +[report handlers]: report.html +[lang_basics]: lang_summary.html +[apply]: services_apply.html +[OpenVoxDB]: /openvoxdb/latest/ +[resource]: lang_resources.html +[OpenVox Server]: /openvox-server/latest/ + +You can configure systems with OpenVox either in a client-server architecture, using the +**OpenVox agent** and **OpenVox server** applications, or in a stand-alone architecture, using +the **OpenVox apply** application. + +## Catalogs + +A catalog is a document that describes the desired system state for one specific computer. It +lists all of the resources that need to be managed, as well as any dependencies between those +resources. + +OpenVox configures systems in two stages: + +1. Compile a catalog. +2. Apply the catalog. + +To compile a catalog, OpenVox uses several sources of information. For more info, see the pages +on [basics of the OpenVox language][lang_basics] and [catalog compilation][catalog_compilation]. + +## The agent-server architecture + +When set up as an agent-server architecture, an OpenVox server controls the configuration +information, and each managed agent node requests its own configuration catalog from the server. + +In this architecture, managed nodes run the **OpenVox agent** application, usually as a +background or scheduled service. One or more servers run the **OpenVox server** application, +[OpenVox Server][]. + +Periodically, each OpenVox agent sends facts to the OpenVox server, and requests a catalog. The +server compiles and returns that node's catalog, using several sources of information it has +access to. + +Once it receives a catalog, OpenVox agent applies it to the node by checking each [resource][] +the catalog describes. If it finds any resources that are not in their desired state, it makes +the changes necessary to correct them. Or, in no-op mode, it reports on what changes would have +been done. + +After applying the catalog, the agent sends a report to the OpenVox server. + +For more information, see: + +* [OpenVox Agent on \*nix Systems][agent_unix] +* [OpenVox Agent on Windows Systems][agent_win] +* [OpenVox Server][] + +### Communications and security + +OpenVox agent nodes and OpenVox servers communicate by HTTPS with client SSL verification. + +The OpenVox Server provides an HTTP interface, with [various endpoints][server_http] available. +When requesting or submitting anything to the server, the agent makes an HTTPS request to one +of those endpoints. + +Client-verified HTTPS means each server or agent must have an identifying SSL certificate. They +each examine their counterpart's certificate to decide whether to allow an exchange of +information. + +OpenVox includes a built-in certificate authority for managing certificates. Agents can +automatically request certificates through the server's HTTP API. You can use the **puppet ssl** +command on an agent to inspect local certificates. The **puppetserver ca** command on the server +manages new and existing certificates. Agents can then download the signed certificates. + +For more information, see: + +* [A walkthrough of OpenVox's HTTPS communications][https_walkthrough] +* [The OpenVox server's HTTP API][server_http] +* [The OpenVox server's auth.conf file][auth.conf] +* [Background reference on SSL and HTTPS.](/docs/background/ssl/) + +## The stand-alone architecture + +Alternatively, OpenVox can run in a stand-alone architecture, where each managed node has its +own complete copy of your configuration info and compiles its own catalog. + +In this architecture, managed nodes run the **OpenVox apply** application, usually as a +scheduled task or cron job. You can also run it on demand for initial configuration of a server +or for smaller configuration tasks. + +Like the OpenVox server application, OpenVox apply needs access to several sources of +configuration data, which it uses to compile a catalog for the node it is managing. + +After OpenVox apply compiles the catalog, it immediately applies it by checking each [resource][] +the catalog describes. If it finds any resources that are not in their desired state, it makes +the changes necessary to correct them. Or, in no-op mode, it reports on what changes would have +been needed. + +After applying the catalog, OpenVox apply stores a report on disk. You can configure it to send +reports to a central service. + +For more information, see the documentation for [the OpenVox apply application][apply]. + +## Differences between agent-server and stand-alone + +In general, OpenVox apply can do the same things as the combination of OpenVox agent and OpenVox +server, but there are several trade-offs around security and the ease of certain tasks. + +If you don't have a preference, you should select the agent-server architecture. If you have +questions, considering these trade-offs helps you make your decision. + +* **Principle of least privilege.** In agent-server OpenVox, each agent only gets its own + configuration, and is unable to see how other nodes are configured. With OpenVox apply, it's + impractical to do this, so every node has access to complete knowledge about how your site is + configured. Depending on how you're configuring your systems, this can potentially raise the + risks of horizontal privilege escalation. +* **Ease of centralized reporting and inventory.** Agents send reports to the OpenVox server by + default, and the server can be configured with any number of [report handlers][] to pass these + on to other services. You can also connect the server to [OpenVoxDB][], a powerful tool for + querying inventory and activity data. OpenVox apply nodes handle their own information, so if + you're using [OpenVoxDB][] or sending reports to another service, _each_ node needs to be + configured and authorized to connect to it. +* **Ease of updating configurations.** Only OpenVox servers have the Puppet modules, main + manifests, and other data necessary for compiling catalogs. This means that when you need to + update your systems' configurations, you only need to update content on one (or a few) OpenVox + servers. In a decentralized `puppet apply` deployment, you'll need to sync new configuration + code and data to every node. +* **CPU and memory usage on managed machines.** Since OpenVox agent doesn't compile its own + catalogs, it uses fewer resources on the machines it manages, leaving them with more capacity + for their designated tasks. +* **Need for a dedicated OpenVox server.** The OpenVox server takes on the performance load of + compiling all catalogs, and it should usually be a dedicated machine with a fast processor, + lots of RAM, and a fast disk. Not everybody wants to (or is able to) allocate that, and OpenVox + apply can get around the need for it. +* **Need for good network connectivity.** Agents need to be able to reach the OpenVox server at + a reliable hostname in order to configure themselves. If a system lives in a degraded or + isolated network environment, you might want it to be more self-sufficient. +* **Security overhead.** Agents and servers use HTTPS to secure their communications and + authenticate each other, and every system involved needs an SSL certificate. OpenVox includes a + built-in CA to easily manage certificates, but it's even easier to not manage them at all. (Of + course, you'll still need to manage security somehow, since you're probably using Rsync or + something to update OpenVox content on every node.) diff --git a/docs/_openvox_9x/bgtm.md b/docs/_openvox_9x/bgtm.md new file mode 100644 index 000000000..0975c6b3d --- /dev/null +++ b/docs/_openvox_9x/bgtm.md @@ -0,0 +1,341 @@ +--- +layout: default +title: Beginner's guide to writing modules +--- + +[structure]: ./images/bgtmclassstructure.png + +Learn how to create fantastic modules by following module best practices [standards and architecture](./style_guide.html). + +Contributors to this guide have spent years creating Puppet modules, falling into every pitfall, trap, and mistake you could hope to make. This guide is intended to help you avoid our mistakes through an approachable introduction to module best practices. + +Before you begin, you should be familiar with Puppet such that you have a basic understanding of the Puppet [language](./lang_summary.html), you know what constitutes a [class](./lang_classes.html), and you understand the basic module [structure](./modules_fundamentals.html). + +## Giving your module purpose + +Before you begin writing your module, you must define what it will do. Defining the range of your module's work helps you create concise modules that are easy to work with. + +Your module should have one area of responsibility. For example, a good module addresses installing MySQL but **does not address** installing another program/service that requires MySQL. + +To help plan your module appropriately, consider: + +* What task do you need your module to accomplish? +* What work is your module addressing? +* What higher function should your module have within your Puppet environment? + +> **Tip**: If you describe the function of your module and you find yourself using the word 'and', it's time to split the module at the 'and'. + +It is standard practice for Puppet users to have 200 or more modules in an environment. Simple is better. Each module in your environment should contain related resources that enable it to accomplish a task. Create multiple modules for more complex needs. The practice of having many small, focused modules promotes code reuse and turns modules into building blocks. + +As an example, let's take a look at the [`puppetlabs-puppetdb`](https://forge.puppet.com/puppetlabs/puppetdb) module. This module deals solely with the the setup, configuration, and management of PuppetDB. However, PuppetDB stores its data in a PostgreSQL database. +Rather than having the module manage PostgreSQL, the author included the [`puppetlabs-postgresql`](https://forge.puppet.com/puppetlabs/postgresql) module as a dependency, leveraging the postgresql module's classes and resources to build out the right configuration for PuppetDB. +Similarly, the `puppetdb-module` needs to manipulate the `puppet.conf` file in order to operate PuppetDB. Instead of having the `puppetdb-module` handle `puppet.conf` changes internally, the author used the [`puppetlabs-inifile`](https://forge.puppet.com/puppetlabs/inifile) module to enable `puppetlabs-puppetdb` to make only the required edits to `puppet.conf`. + +## Structuring your module + +The ideal module manages a single piece of software from installation through setup, configuration, and service management. + +This section covers: + +* [How to design your module's classes](#class-design). +* [How to develop useful parameters](#parameters). +* [How best to order your classes (rather than resources)](#ordering). +* [How to leverage and utilize dependencies](#dependencies). + +To demonstrate a real-world best practices standard module, we will walk through the structure of VoxPupuli's [puppet-chrony](https://github.com/voxpupuli/puppet-chrony) module. + +### Class design + +A good module is comprised of small, self-contained classes that each do only one thing. Classes within a module are similar to functions in programming, using parameters to perform related steps that create a coherent whole. + +In general, the best practice naming convention is that the file must be named the same as the class or definition that is contained within (with the sole exception of the [main class](#module)), and classes must be named after their function. + +In terms of class structure we recommend the following (more detail below): + +![module class structure][structure] + +#### `module` + +The main class of any module must share the name of the module and be located in the `init.pp` file. The name and location of the main module class is extremely important, as it guides the [autoloader](./lang_namespaces.html#autoloader-behavior) behavior. +The main class of a module is its interface point and ought to be the only parameterized class if possible. Limiting the parameterized classes to just the main class allows you to control usage of the entire module with the inclusion of a single class. This class should provide sensible defaults so that a user can get going with `include module`. + +For instance, the main `chrony` class in the `chrony` module looks like this: + +```puppet +class chrony ( + Array[Stdlib::IP::Address] $bindaddress = [], + Array[String] $bindcmdaddress = ['127.0.0.1', '::1'], + Optional[String] $initstepslew = undef, + Array[String] $cmdacl = [], + NotUndef $commandkey = 0, + Stdlib::Unixpath $config = '/etc/chrony/chrony.conf', + Stdlib::Filemode $config_mode = '0644', + Boolean $config_keys_manage = true, + Array[String[1]] $keys = [], + Stdlib::Unixpath $driftfile = '/var/lib/chrony/drift', + ... +``` + +Note that each static default is declared inline, right on the parameter, so the value is visible in the class signature and in generated reference documentation. + +#### `module::install` + +The install class must be located in the `install.pp` file. It should contain all of the resources related to getting the software that the module manages onto the node. + +The install class must be named `module::install`, as in the `chrony` module: + +```puppet +class chrony::install { + assert_private() + + package { 'chrony': + ensure => $chrony::package_ensure, + name => $chrony::package_name, + source => $chrony::package_source, + provider => $chrony::package_provider, + } +} +``` + +The `install`, `config`, and `service` classes are private: they are declared only by the main `chrony` class, never directly by users. Each one calls `assert_private()`, which fails compilation with a clear message if the class is declared from outside its own module. See [public and private classes](./style_guide.html#public-and-private) for more on this distinction. + +#### `module::config` + +The resources related to configuring the installed software should be placed in a config class. The config class must be named `module::config` and must be located in the `config.pp` file. + +For example, see the `module::config` class in the `chrony` module: + +```puppet +class chrony::config { + assert_private() + + file { $chrony::config: + ensure => file, + owner => 0, + group => 0, + mode => $chrony::config_mode, + content => epp($chrony::config_template, + { + servers => chrony::server_array_to_hash($chrony::servers, ['iburst']), + pools => chrony::server_array_to_hash($chrony::pools, ['iburst']), + peers => chrony::server_array_to_hash($chrony::peers), + } + ), + } +... +``` + +#### `module::service` + +The remaining service resources, and anything else related to the running state of the software, should be contained in the service class. The service class must be named `module::service` and must be located in the `service.pp` file. + +For example: + +```puppet +class chrony::service { + assert_private() + + if $chrony::service_manage { + service { $chrony::service_name: + ensure => $chrony::service_ensure, + enable => $chrony::service_enable, + } + } + + if $chrony::wait_manage { + service { $chrony::wait_name: + ensure => $chrony::wait_ensure, + enable => $chrony::wait_enable, + } + } +} +``` + +### Parameters + +Parameters form the public API of your module. + +They are the most important interface you expose, and you should take care to balance to the number and variety of parameters so that users can customize their interactions with the module. Below, we walk through best practices for naming and developing parameters. + +#### Naming parameters + +Naming consistency is imperative for community comprehension and assists in troubleshooting and collaborating on module development. + +Best practices recommend the pattern of `thing_property` for naming parameters. + +For example, in the `chrony` module the service resource uses `service_name`, `service_ensure`, and `service_enable`: + +```puppet +class chrony::service { + + if $chrony::service_manage { + service { $chrony::service_name: + ensure => $chrony::service_ensure, + enable => $chrony::service_enable, + } + } + +} +``` + +If you have a parameter that toggles an entire function on and off, the naming convention can be amended to `thing_manage`. This applies, in particular, to Boolean toggles, such as when the module manages the service altogether. The `thing_manage` convention allows you to wrap all of the resources in an `if $service_manage {}` test, as shown in the `chrony` example above. + +Consistent naming across modules helps with the readability and usability of your code. + +#### Number of parameters + +To maximize the usability of your module, make it flexible by adding parameters. Parameters enable users to customize their use of your module. + +You must not hardcode data in your modules, and having more parameters is the best alternative. Hardcoding data in your module makes it inflexible, and means your module requires manifest changes to be used in even slightly different circumstances. + +Avoid adding parameters that allow you to override templates. When your parameters allow template overrides, users can override your template with a custom template that contains additional hardcoded parameters. +Hardcoded parameters in templates inhibits flexibility over time. It is far better to create more parameters and then modify the original template, or have a parameter which accepts an arbitrary chunk of text added to the template, than it is to override the template with a customized one. + +For an example of a module that capitalizes on offering many parameters, please see [puppetlabs-apache](https://forge.puppet.com/puppetlabs/apache). + +#### Parameter defaults + +Where you put a parameter's default value depends on whether that value is the same on every supported operating system: + +* **Static defaults that are identical across every supported OS** belong inline in the parameter declaration in `init.pp`. +* **OS-specific defaults** belong in module Hiera data with a per-OS hierarchy, resolved through automatic parameter lookup. + +Keeping static defaults inline puts the value right where the parameter is declared, which is what module reviewers expect and what keeps the default easy to find. When defaults live only in Hiera, someone reading `init.pp` can't tell whether a parameter has a default elsewhere or must be supplied. + +Inline defaults also render in generated reference documentation with any toolchain. Defaults placed only in `data/common.yaml` are invisible to upstream [puppet-strings](https://github.com/puppetlabs/puppet-strings/issues/250), though OpenVox's [openvox-strings](https://github.com/voxpupuli/openvox-strings/pull/27) can now read them. + +Give each default a single home. Duplicating a value between `init.pp` and module Hiera data means two sources of truth that can drift apart. + +Reserve `Optional[T] = undef` for parameters where `undef` is a genuine runtime value, such as a parameter that toggles an optional feature off. Avoid declaring a parameter `Optional` simply to defer its default to Hiera when the parameter will always receive a value, since that misleads users into thinking `undef` is a supported state. + +If a parameter is genuinely required and has no sensible default, give it no default at all so that catalog compilation fails clearly when the value is missing, rather than using `Optional[T] = undef`. + +For more on the mechanics of parameter defaults and Hiera data in modules, see the [Puppet language style guide](./style_guide.html#parameter-defaults). + +### Ordering + +Best practice is to base all order-related dependencies (such as `require` and `before`) on classes rather than resources. Class-based ordering allows you to shield the implementation details of each class from the other classes. + +For example: + +```puppet + file { 'configuration': + ensure => present, + require => Class['module::install'], + } +``` + +Rather than making a `require` to several packages, the above ordering allows you to refactor and improve `module::install` without adjusting the manifests of other classes to match the changes. + +#### Containment and anchoring + +To allow other modules to form ordering relationships with your module, ensure that your main classes explicitly _contain_ any subordinate classes they declare. + +Classes do not _automatically_ contain the classes they declare. This is because classes can be declared in several places via `include` and similar functions. To contain classes, use [the `contain` function](./function.html#contain). For more information and context about containment, see [the containment docs](./lang_containment.html). + +For example, the `chrony` module uses containment in the main `chrony` class: + +```puppet +contain chrony::install +contain chrony::config +contain chrony::service + +Class['chrony::install'] +-> Class['chrony::config'] +~> Class['chrony::service'] +``` + +### Dependencies + +If your module's functionality depends on another module, then you must list these dependencies and include them directly. + +This means you must `include x` in the main class to ensure the dependency is included in the catalog. You must also add the dependency to the module's [metadata.json](./style_guide.html#module-metadata) and `.fixtures.yml`. (`.fixtures.yml` is a file used exclusively by RSpec to pull in dependencies required to successfully run unit tests.) + +## Testing your module + +Ensure that the module works in a variety of conditions, and that the options and parameters of your module work together to an appropriate end result. + +We recommend several testing frameworks available to help you write unit and acceptance tests. Some of these tools are already included in the Puppet Development Kit (PDK). + +> **Note:** The open source version of PDK is no longer supported by Puppet. It may still be functional, but no further updates or bug fixes will be provided. + +### rspec-puppet + +The `rspec-puppet` gem provides a unit-testing framework for Puppet. It extends RSpec to allow the testing framework to understand Puppet catalogs, the artifact it specializes in testing. You can write tests, as in the below example, to test that aspects of your module work as intended. + +```ruby +it { should contain_file('configuration') } +``` + +RSpec lets you provide facts, like `os['family']`, in order to test the module in various scenarios. + +A typical use of RSpec is to iterate over a list of operating systems, asserting that the package and service should exist in the catalog for every operating system your module supports. + +To learn more, see the [rspec-puppet documentation](https://puppetlabs.github.io/rspec-puppet/). + +### puppetlabs-spec-helper + +The [puppetlabs-spec-helper](https://github.com/puppetlabs/puppetlabs_spec_helper) gem automates some of the tasks required to test modules. + +> **Note:** This gem is no longer needed for any Vox Pupuli modules. + +This is especially useful in conjunction with `rspec-puppet`, as `puppetlabs-spec-helper` provides default Rake tasks that allow you to standardize testing across modules. It also provides some code to connect `rspec-puppet` with modules. Add it to the Gemfile of the project, and then add the following line to the Rakefile: + +```ruby +require 'puppetlabs_spec_helper/rake_tasks' +``` + +### Beaker-rspec + +[Beaker-rspec](https://github.com/voxpupuli/beaker-rspec) is an acceptance/integration testing framework. + +It provisions one or more virtual machines on various hypervisors (such as [Vagrant](https://www.vagrantup.com/)) and then checks the result of applying your module in a realistic environment. + +#### serverspec + +[Serverspec](https://serverspec.org/) provides additional testing constructs (such as `be_running` and `be_installed`) for beaker-rspec. It allows you to abstract away details of the underlying distribution when testing. It lets you write tests like: + +```ruby +describe service('httpd') do + it { should be_running } +end +``` + +It then knows how to translate `be_running` into shell commands for different distributions. + +## Versioning your module + +Modules, like any other piece of software, must be versioned and released when changes are made. Use semantic versioning, which sets out specific rules for when to increment major and minor versions. + +After you've decided on the new version number, adjust the version number in the `metadata.json` file. + +This allows you to create a list of dependencies in the `metadata.json` file of your modules with specific versions of dependent modules, which ensures your module isn't used with an old dependency that won't work. Versioning also enables workflow management by allowing you to easily use different versions of modules in different environments. + +## Documenting your module + +We recommend that you document your module with a README explaining how your module works and a Reference section detailing information about your module's classes, defined types, functions, and resource types and providers. + +For guidance, see our modules documentation [guide](./modules_documentation.html) and the [documentation](./style_guide.html#documentation) section of the OpenVox Language Style Guide. + +## Releasing your module + +We encourage you to publish your modules on the [Puppet Forge](https://forge.puppet.com). + +Sharing your modules allows other users to write improvements to the modules you make available and contribute them back to you, effectively giving you free improvements to your modules. + +Additionally, publishing your modules to the Forge helps foster community among Puppet users, and allows other Puppet community members to download and use your module. +If the Puppet community routinely releases and iterates on modules on the Forge, the quality of available modules increases dramatically and gives you access to more modules to download and modify for your own purposes. Details on how to publish modules to the Forge can be found in the [module publishing guide](./modules_publishing.html). + +## Community Resources + +For beginning module authors, a variety of community resources are available. + +[Module basics](./modules_fundamentals.html) + +[OpenVox Language Style Guide](./style_guide.html) + +[The Forge](https://forge.puppet.com) + +[Vox Pupuli community channels](https://voxpupuli.org/connect/) + +[puppetmodule.info](https://www.puppetmodule.info) — open source Puppet module documentation server, generating fresh docs for Puppet modules and popular Git repositories diff --git a/docs/_openvox_9x/cheatsheet_core_types.md b/docs/_openvox_9x/cheatsheet_core_types.md new file mode 100644 index 000000000..f425341cf --- /dev/null +++ b/docs/_openvox_9x/cheatsheet_core_types.md @@ -0,0 +1,209 @@ +--- +layout: default +title: Core types cheat sheet +--- + +[exec]: ./type.html#exec +[user]: ./type.html#user +[group]: ./type.html#group + +A quick reference guide for the core Puppet types. + +For detailed information about these types, see the related topic Type reference. + +Related topics: + +* [Type reference](./type.html) + +## The Trifecta + +Package/file/service: Learn it, live it, love it. Even if this is the only Puppet you know, you can still get a whole lot done. + +```puppet +package { 'openssh-server': + ensure => installed, +} + +file { '/etc/ssh/sshd_config': + source => 'puppet:///modules/sshd/sshd_config', + owner => 'root', + group => 'root', + mode => '0640', + notify => Service['sshd'], # sshd restarts whenever you edit this file. + require => Package['openssh-server'], +} + +service { 'sshd': + ensure => running, + enable => true, +} +``` + + + +### file + +Manages files, directories, and symlinks. + +#### Important Attributes + +* [`ensure`](./type.html#file-attribute-ensure) -- Whether the file should exist, and what it should be. Allowed values: + * `file` (a normal file) + * `directory` (a directory) + * `link` (a symlink) + * `present` (anything) + * `absent` +* [`path`](./type.html#file-attribute-path) -- The full path to the file on disk; **defaults to title.** +* [`owner`](./type.html#file-attribute-owner) -- By name or UID. +* [`group`](./type.html#file-attribute-group) -- By name or GID. +* [`mode`](./type.html#file-attribute-mode) -- Must be specified exactly. Does the right thing for directories. + +#### For Normal Files + +* [`source`](./type.html#file-attribute-source) -- Where to download contents for the file. Usually a `puppet:///` URL. +* [`content`](./type.html#file-attribute-content) -- The file's desired contents, as a string. Most useful when paired with [templates](./lang_template.html), but you can also use the output of the [file function](./function.html#file). + +#### For Directories + +* [`source`](./type.html#file-attribute-source) -- Where to download contents for the directory, when `recurse => true`. +* [`recurse`](./type.html#file-attribute-recurse) -- Whether to recursively manage files in the directory. +* [`purge`](./type.html#file-attribute-purge) -- Whether unmanaged files in the directory should be deleted, when `recurse => true`. + +#### For Symlinks + +* [`target`](./type.html#file-attribute-target) -- The symlink target. (Required when `ensure => link`.) + +#### Other Notable Attributes + +* [`backup`](./type.html#file-attribute-backup) +* [`checksum`](./type.html#file-attribute-checksum) +* [`force`](./type.html#file-attribute-force) +* [`ignore`](./type.html#file-attribute-ignore) +* [`links`](./type.html#file-attribute-links) +* [`recurselimit`](./type.html#file-attribute-recurselimit) +* [`replace`](./type.html#file-attribute-replace) + +### package + +Manages software packages. + +#### Important Attributes + +* [`name`](./type.html#package-attribute-name) -- The name of the package, as known to your packaging system; **defaults to title.** +* [`ensure`](./type.html#package-attribute-ensure) -- Whether the package should be installed, and what version to use. Allowed values: + * `present` + * `latest` (implies `present`) + * any version string (implies `present`) + * `absent` + * `purged` (Potentially dangerous. Ensures absent, then zaps configuration files and dependencies, including those that other packages depend on. Provider-dependent.) +* [`source`](./type.html#package-attribute-source) -- Where to obtain the package, if your system's packaging tools don't use a repository. +* [`provider`](./type.html#package-attribute-provider) -- Which packaging system to use (e.g. Yum vs. Rubygems), if a system has more than one available. + +### service + +Manages services running on the node. Like with packages, some platforms have better tools than others, so read up. + +You can make services restart whenever a file changes, with the `subscribe` or `notify` metaparameters. For more info, read the related topic about relationships + +Related topics: + +* [Relationships](./lang_relationships.html) + +#### Important Attributes + +* [`name`](./type.html#service-attribute-name) -- The name of the service to run; **defaults to title.** +* [`ensure`](./type.html#service-attribute-ensure) -- The desired status of the service. Allowed values: + * `running` (or `true`) + * `stopped` (or `false`) +* [`enable`](./type.html#service-attribute-enable) -- Whether the service should start on boot. Doesn't work on all systems. +* [`hasrestart`](./type.html#service-attribute-hasrestart) -- Whether to use the init script's restart command instead of stop+start. Defaults to false. +* [`hasstatus`](./type.html#service-attribute-hasstatus) -- Whether to use the init script's status command. Defaults to true. + +#### Other Notable Attributes + +If a service has a bad init script, you can work around it and manage almost anything using the [`status`](./type.html#service-attribute-status), [`start`](./type.html#service-attribute-start), [`stop`](./type.html#service-attribute-stop), [`restart`](./type.html#service-attribute-restart), [`pattern`](./type.html#service-attribute-pattern), and [`binary`](./type.html#service-attribute-binary) attributes. + + +## Hello World + +### notify + +Logs an arbitrary message, at the `notice` log level. This appears in the POSIX syslog or Windows Event Log on the OpenVox agent node and is also logged in reports. + +```puppet +notify { 'This message is getting logged on the agent node.': } +``` + +#### Important Attributes + +* [`message`](./type.html#notify-attribute-message) -- **Defaults to title.** + +## Grab bag + +### [exec][] + +Executes an arbitrary command on the agent node. When using execs, you must either make sure the command can be safely run multiple times, or specify that it should only run under certain conditions. + +#### Important Attributes + +* [`command`](./type.html#exec-attribute-command) -- The command to run; **defaults to title.** If this isn't a fully-qualified path, use the `path` attribute. +* [`path`](./type.html#exec-attribute-path) -- Where to look for executables, as a colon-separated list or an array. +* [`returns`](./type.html#exec-attribute-returns) -- Which exit codes indicate success. Defaults to `0`. +* [`environment`](./type.html#exec-attribute-environment) -- An array of environment variables to set (for example, `['MYVAR=somevalue', 'OTHERVAR=othervalue']`). + +#### Attributes to Limit When a Command Should Run + +* [`creates`](./type.html#exec-attribute-creates) -- A file to look for before running the command. The command only runs if the file doesn’t exist. +* [`refreshonly`](./type.html#exec-attribute-refreshonly) -- If `true`, the command only run if a resource it subscribes to (or a resource which notifies it) has changed. +* [`onlyif`](./type.html#exec-attribute-onlyif) -- A command or array of commands; if any have a non-zero return value, the command won't run. +* [`unless`](./type.html#exec-attribute-unless) -- The opposite of onlyif. + +#### Other Notable Attributes + +[`cwd`](./type.html#exec-attribute-cwd), [`group`](./type.html#exec-attribute-group), [`logoutput`](./type.html#exec-attribute-logoutput), , [`timeout`](./type.html#exec-attribute-timeout), [`tries`](./type.html#exec-attribute-tries), [`try_sleep`](./type.html#exec-attribute-try_sleep), [`user`](./type.html#exec-attribute-user). + +### [user][] + +Manages user accounts; mostly used for system users. + +```puppet +user { 'jane': + ensure => present, + uid => '507', + gid => 'admin', + shell => '/bin/zsh', + home => '/home/jane', + managehome => true, +} +``` + +#### Important Attributes + +* [`name`](./type.html#user-attribute-name) -- The name of the user; **defaults to title.** +* [`ensure`](./type.html#user-attribute-ensure) -- Whether the user should exist. Allowed values: + * `present` + * `absent` + * `role` +* [`uid`](./type.html#user-attribute-uid) -- The user ID. Must be specified numerically; chosen automatically if omitted. Read-only on Windows. +* [`gid`](./type.html#user-attribute-gid) -- The user’s primary group. Can be specified numerically or by name. (Not used on Windows; use `groups` instead.) +* [`groups`](./type.html#user-attribute-groups) -- An array of other groups to which the user belongs. (Don't include the group specified as the `gid`.) +* [`home`](./type.html#user-attribute-home) -- The user's home directory. +* [`managehome`](./type.html#user-attribute-managehome) -- Whether to manage the home directory when managing the user; if you don't set this to true, you'll need to create the user's home directory manually. +* [`shell`](./type.html#user-attribute-shell) -- The user's login shell. + +#### Other Notable Attributes + +[`comment`](./type.html#user-attribute-comment), [`expiry`](./type.html#user-attribute-expiry), [`membership`](./type.html#user-attribute-membership), [`password`](./type.html#user-attribute-password), [`password_max_age`](./type.html#user-attribute-password_max_age), [`password_min_age`](./type.html#user-attribute-password_min_age), [`purge_ssh_keys`](./type.html#user-attribute-purge_ssh_keys), [`salt`](./type.html#user-attribute-salt). + +### [group][] + +Manages groups. + +#### Important Attributes + +* [`name`](./type.html#group-attribute-name) -- The name of the group; **defaults to title.** +* [`ensure`](./type.html#group-attribute-ensure) -- Whether the group should exist. Allowed values: + * `present` + * `absent` +* [`gid`](./type.html#group-attribute-gid) -- The group ID; must be specified numerically, and is chosen automatically if omitted. Read-only on Windows. +* [`members`](./type.html#group-attribute-members) -- Users and groups that should be members of the group. Only applicable to certain operating systems; see the full type reference for details. diff --git a/docs/_openvox_9x/cheatsheet_module.md b/docs/_openvox_9x/cheatsheet_module.md new file mode 100644 index 000000000..ebce4c3a4 --- /dev/null +++ b/docs/_openvox_9x/cheatsheet_module.md @@ -0,0 +1,135 @@ +--- +layout: default +title: Module cheat sheet +--- + + +[installing]: ./modules_installing.html +[fundamentals]: ./modules_fundamentals.html +[plugins]: ./plugins_in_modules.html +[publishing]: ./modules_publishing.html +[documentation]: ./modules_documentation.html + +## Module cheat sheet + +A quick reference to Puppet module terms and concepts. + +For detailed explanations of Puppet modules, see the related topics about modules. + +Related topics: + +* [Module fundamentals][fundamentals]: How to use and write Puppet modules. +* [Installing modules][installing]: How to install pre-built modules from the Puppet Forge. +* [Publishing modules][publishing]: How to publish your modules to the Puppet Forge. +* [Using plug-ins][plugins]: How to arrange plug-ins (like custom facts and custom resource types) in modules and sync them to agent nodes. +* [Documenting modules][documentation]: A module README template and information on providing directions for your module. + +**Example Module: `/etc/puppetlabs/puppet/modules/apache`** + +### manifests + +This directory holds the module's Puppet code. + +* Each `.pp` file should contain one and only one class or defined type. +* Filenames and class/defined type names are related; see the examples below. +* Within a module, the special `$module_name` variable always contains the module's name. + +#### apache/manifests/init.pp + +```puppet +class apache { + ... +} +``` + +`init.pp` is special; it should contain a class (or defined type) with the same name as the module. + +#### apache/manifests/vhost.pp + +```puppet +define apache::vhost( + $port, + $docroot, +) { + ... +} +``` + +Other classes (and defined types) should be named +`modulename::filename` (without the .pp extension). + + +#### apache/manifests/config/ssl.pp + +```puppet +class apache::config::ssl { + ... +} +``` + +Subdirectories add intermediate namespaces. + +### files + +Nodes can download any files in this directory from Puppet's built-in file server. + +* Use the source attribute to download file contents from the server. +* Use puppet:/// URIs to specify which file to fetch. +* Files in this directory are served at `puppet:///modules/modulename/ +filename`. + + +#### apache/files/httpd.conf + +To fetch this file: + +```puppet +file { '/etc/apache2/httpd.conf': + ensure => file, + source => 'puppet:///modules/apache/httpd.conf', +} +``` + +#### apache/files/extra/ssl + +Puppet's file server can navigate any subdirectories: + +```puppet +file { '/etc/apache2/httpd-ssl.conf': + ensure => file, + source => 'puppet:///modules/apache/extra/ssl', +} +``` + +### lib + +This directory holds Ruby plugins, which can add features to Puppet and Facter. + +Capache/lib/puppet/type/apache_setting.rb** +A custom type. + +**apache/lib/puppet/parser/functions/htpasswd.rb** +A custom function. + +apache/lib/facter/apache_confdir.rb** +A custom fact. + +### templates + +This directory holds ERB templates. + +* Use the template function to create a string by rendering a template. +* Use the content attribute to fill file contents with a string. +* Template files are referenced as modulename/filename.erb. + +#### apache/templates/vhost.erb + +To use this template: + +```puppet +file {'/etc/apache2/sites-enabled/wordpress.conf': + ensure => file, + content => template('apache/vhost.erb'), +} +``` + diff --git a/docs/_openvox_9x/complete_resource_example.markdown b/docs/_openvox_9x/complete_resource_example.markdown new file mode 100644 index 000000000..71f5383ad --- /dev/null +++ b/docs/_openvox_9x/complete_resource_example.markdown @@ -0,0 +1,120 @@ +--- +layout: default +title: Complete Resource Example +--- + +## Resource Creation + +Nearly every resource needs to be able to be created and destroyed, and resources have to have names, so we'll start with those two features. +Puppet's property support has a helper method called `ensurable` that handles modeling creation and destruction; it creates an `ensure` property and adds `absent` and `present` values for it, which in turn require three methods on the provider, `create`, `destroy`, and `exists?`. +Here's the first start to the resource. We're going to create one called 'file' --- this is an example of how to create a resource for something Puppet already has. + + +```ruby +Puppet::Type.newtype(:file) do + @doc = "Manage a file (the simple version)." + + ensurable + + newparam(:name) do + desc "The full path to the file." + end +end +``` + + +Here we have provided the resource type name (`file`), a documentation string, a parameter for the name of the file, and used the ensurable method to say that the file is both createable and destroyable. + +To see how we would use this on the provider side, let's look at a simple provider: + +```ruby +Puppet::Type.type(:file).provide(:posix) do + desc "Normal Unix-like POSIX support for file management." + + def create + File.open(@resource[:name], "w") { |f| f.puts "" } # Create an empty file + end + + def destroy + File.unlink(@resource[:name]) + end + + def exists? + File.exists?(@resource[:name]) + end +end +``` + +Note that the providers use a `desc` instead of `@doc =` to specify their documentation. + +In addition to the docs and the provider name, we provide the three methods that the `ensure` property requires. You can see that in this case we're just using Ruby's built-in File abilities to create an empty file, remove the file, or test whether the file exists. + +Let's enhance our resource somewhat by adding the ability to manage the file mode. Here's the code we need to add to the resource: + +```ruby +newproperty(:mode) do + desc "Manage the file's mode." + + defaultto "640" +end +``` + +Notice that we're specifying a default value, and that it is a string instead of an integer (file modes are in octal, and most of us are used to specifying integers in decimal). You can pass a lambda to `defaultto` instead of a value, if you don't have a simple value. + +Add this code to the provider to understand modes: + +```ruby +def create + File.open(@resource[:name], "w") { |f| f.puts "" } # Create an empty file + # Make sure the mode is correct + should_mode = @resource.should(:mode) + unless self.mode == should_mode + self.mode = should_mode + end +end + +# Return the mode as an octal string, not as an integer. +def mode + if File.exists?(@resource[:name]) + "%o" % (File.stat(@resource[:name]).mode & 007777) + else + :absent + end +end + +# Set the file mode, converting from a string to an integer. +def mode=(value) + File.chmod(Integer("0" + value), @resource[:name]) +end +``` + +Note that the getter method returns the value, it doesn't attempt to modify the resource itself. Also, when the setter gets passed the value it is supposed to set; it doesn't attempt to figure out the appropriate value to use. This should always be true of how providers are implemented. + +Also notice that the `ensure` property, when created by the `ensurable` method, behaves differently because it uses methods for creation and destruction of the file, whereas normal properties use getter and setter methods. +When a resource is being created, Puppet expects the `create` method (or, actually, any changes done within ensure) to make any other necessary changes. +This is because most often resources are created already configured correctly, so it doesn't make sense for Puppet to test it manually (for example, useradd support is set up to add all specified properties when useradd is run, so usermod doesn't need to be run afterward). + +You can see how the `absent` and `present` values are defined by looking in the property.rb file; here's the most important snippet: + +```ruby +newvalue(:present) do + if @resource.provider and @resource.provider.respond_to?(:create) + @resource.provider.create + else + @resource.create + end + nil # return nil so the event is autogenerated +end + +newvalue(:absent) do + if @resource.provider and @resource.provider.respond_to?(:destroy) + @resource.provider.destroy + else + @resource.destroy + end + nil # return nil so the event is autogenerated +end +``` + +There are a lot of other options in creating properties, parameters, and providers, but this should provide a decent starting point. + diff --git a/docs/_openvox_9x/component_versions.md b/docs/_openvox_9x/component_versions.md new file mode 100644 index 000000000..12b1934ab --- /dev/null +++ b/docs/_openvox_9x/component_versions.md @@ -0,0 +1,128 @@ +--- +layout: default +title: "Component versions in recent OpenVox releases" +--- + +[about_agent]: ./about_agent.html +[openfact]: /openfact/latest/ +[openbolt]: /openbolt/latest/ +[openbolt_apply]: /openbolt/latest/applying_manifest_blocks.html +[server_install_pre]: /openvox-server/latest/install_pre.html +[openvoxdb_postgres]: /openvoxdb/latest/configure_postgres.html +[sbom_tools]: https://github.com/OpenVoxProject/openvox-sbom-tools + +{% assign nav_key = page.nav %} + +This page lists the versions of each stack component shipped in recent OpenVox +releases, so you can answer "what's actually in this release?" in one place. + +## Why there are several tables + +Unlike a single bundled product, OpenVox ships its components on **independent +version lines**. `openvox-agent`, `openvox-server`, `openvoxdb`, and `openbolt` are +released separately and do not share a version number (for example, the newest +agent, server, and database releases all carry different versions, and OpenBolt is +on its own 5.x line). There is no single "OpenVox platform version" that pins all +of them at once, so each component is shown in its own table, keyed by that +component's release. + +The bundled-component columns are **generated** from the per-release SBOMs published +by [openvox-sbom-tools][sbom_tools] for every component, so they don't drift. Columns +that aren't bundled anywhere (Java and PostgreSQL) are supported-version requirements +maintained by hand; see the note under each table. + + + +## Agent and runtime components + +These ship inside the `openvox-agent` package (see [About openvox-agent][about_agent]). +The OpenFact column is the **bundled** OpenFact version and links to the +[OpenFact documentation][openfact], which is the authoritative source for OpenFact +changes; this page is only a pointer. + +| OpenVox release | OpenFact | Ruby | OpenSSL | curl | +| --- | --- | --- | --- | --- | +{% for r in site.data.agent_release_contents[nav_key] %}| {{ r.release }} | [{{ r.openfact }}][openfact] | {{ r.ruby }} | {{ r.openssl }} | {{ r.curl }} | +{% endfor %} + +## Server components + +These ship with the `openvox-server` package. JRuby is the bundled version, read +from the server's per-release SBOM. + +| OpenVox Server release | JRuby | Java | +| --- | --- | --- | +{% for r in site.data.server_release_contents[nav_key] %}| {{ r.release }} | {{ r.jruby }} | 17, 21 | +{% endfor %} + +> **Java is not bundled.** OpenVox Server requires a supported JDK to be installed +> separately. The Java column shows the currently supported major versions, not a +> per-release pin; see [Before you install OpenVox Server][server_install_pre]. + +## Data components + +OpenVoxDB ships in the `openvoxdb` package on its own release line. The +`openvoxdb-termini` package (the terminus plugins that let OpenVox Server and +agents talk to OpenVoxDB) is released in lockstep at the **same version** as +`openvoxdb`, so it is not listed separately. + +Jetty is the bundled HTTP server, read from the OpenVoxDB SBOM. + +| OpenVoxDB release | Jetty | Java | PostgreSQL | +| --- | --- | --- | --- | +{% for r in site.data.openvoxdb_release_contents[nav_key] %}| {{ r.release }} | {{ r.jetty }} | 11, 17 | 11+ (14+ recommended) | +{% endfor %} + +> **Java and PostgreSQL are not bundled.** OpenVoxDB runs on a JVM and connects to a +> PostgreSQL server you install separately (the `puppet-openvoxdb` module can install +> PostgreSQL for you). The Java column shows the currently supported major versions, +> and the PostgreSQL column the supported minimum (PostgreSQL 11; version 14 or newer +> recommended) — neither is a per-release pin. See +> [Configuring PostgreSQL][openvoxdb_postgres]. + +## OpenBolt + +[OpenBolt][openbolt] is the orchestration tool. It is not part of the +agent/server/data stack above and ships on its own **5.x** release line, bundling +its own runtime. See the [OpenBolt documentation][openbolt] for OpenBolt's own +release notes. + +OpenBolt is the only OpenVox package that **bundles r10k**. Although r10k is +typically run on a server to deploy environments from a control repo, it is not +shipped in `openvox-server` (or `openvox-agent`); on a server you install it +separately, for example with the `puppet/r10k` module or a `gem install`. + +| OpenBolt release | OpenVox | Ruby | OpenSSL | r10k | +| --- | --- | --- | --- | --- | +{% for r in site.data.openbolt_release_contents %}| {{ r.release }} | {{ r.openvox }} | {{ r.ruby }} | {{ r.openssl }} | {{ r.r10k }} | +{% endfor %} + +> **The OpenVox column is the bundled version, not something you install.** OpenBolt +> bundles OpenVox for [`bolt apply`][openbolt_apply]; its gemspec declares a range +> (`~> 8.0`) and the version shown here is the exact one resolved into the package at +> build time. For `bolt apply`, OpenBolt compiles the catalog with this bundled +> OpenVox and installs the `openvox-agent` package on targets via `apply_prep`, so +> you don't install OpenVox separately to use it. + + + +## Regenerating this page + +The agent/runtime, server, OpenVoxDB, and OpenBolt columns are generated from +upstream release metadata. Regenerate the data with: + +```bash +bundle exec rake references:component_versions +``` + +The agent, server, and OpenVoxDB tables are per-OpenVox-series: each task writes a +file named for the collection's nav_key, so the page renders its own series via +`site.data.[page.nav]`. With the 8.x defaults this writes +`_data/agent_release_contents/openvox_8x.yml`, +`_data/server_release_contents/openvox_8x.yml`, and +`_data/openvoxdb_release_contents/openvox_8x.yml`. OpenBolt is independent of the +OpenVox major and is shared across series in `_data/openbolt_release_contents.yml`. + +When a 9.x collection is added, run the per-series tasks again with `SERIES=9.` +(and an appropriate `MIN_RELEASE`); they write `…/openvox_9x.yml` files, and the +copied 9.x page reads them automatically through its own `page.nav`. diff --git a/docs/_openvox_9x/config_about_settings.markdown b/docs/_openvox_9x/config_about_settings.markdown new file mode 100644 index 000000000..345a24b76 --- /dev/null +++ b/docs/_openvox_9x/config_about_settings.markdown @@ -0,0 +1,168 @@ +--- +layout: default +title: "Configuration: How Puppet is configured" +--- + +[short list]: ./config_important_settings.html +[conf_ref]: ./configuration.html +[puppetserver_config]: /openvox-server/latest/configuration.html + +Puppet's commands and services can be extensively configured, and its settings can be specified in a variety of places. + +See also: + +- [Short list of important settings][short list] +- [The configuration reference][conf_ref] + +Settings can be set in the main config file. Puppet's main config file is called `puppet.conf`. + +## Main settings and extra config files + +When we mention "settings" in the context of Puppet, we usually mean the main settings. These are the settings that are listed in the configuration reference. They are valid in `puppet.conf` and available for use on the command line. These settings configure nearly all of Puppet's core features. + +However, there are also about nine extra configuration files --- things like `auth.conf` and `puppetdb.conf`. These files exist for several reasons: + +- The main settings only support a few types of values. Some things just can't be configured without complex data structures, so they needed separate files. (Authorization rules and custom CSR attributes are in this category.) +- Puppet currently doesn't allow extensions to add new settings to `puppet.conf`. This means some settings that _should_ be main settings (like the PuppetDB server) can't be. + +### Puppet Server configuration + +Puppet Server honors almost all settings in `puppet.conf` and should pick them up automatically. However, for some tasks, such as configuring the webserver or an external Certificate Authority, there are Puppet Server-specific configuration files and settings. + +See [Puppet Server: Configuration][puppetserver_config] + +## Settings are loaded on startup + +When any Puppet command or service starts up, it gets values for all of its settings. Any of these settings could change the way that command or service behaves. + +A command or service _only_ reads its settings _once;_ if something needs to be reconfigured, it needs to be restarted or run again. + +## Settings on the command line + +Settings from the command line have top priority, and **always override settings from the config file.** When a Puppet command or service is started, you can specify any setting as a command line option. + +Settings require two hyphens and the name of the setting on the command line: + +`$ sudo puppet agent --test --noop --certname temporary-name.example.com` + +## Basic settings + +For most settings, you specify the option and follow it with a value. An equals sign between the two (`=`) is optional, and you can optionally put values in quotes. + +All three of these are equivalent to setting `certname = temporary-name.example.com` in `puppet.conf`: + +`--certname=temporary-name.example.com` + +`--certname temporary-name.example.com` + +`--certname "temporary-name.example.com"` + +## Boolean settings + +Settings whose only valid values are `true` and `false`, use a shorter format. Specifying the option alone sets the setting to `true`, or prefixing the option with `no-` sets it to false. + +This means: + +`--noop` is equivalent to setting `noop = true` in `puppet.conf`. + +`--no-noop` is equivalent to setting `noop = false` in `puppet.conf`. + +## Default values + +If a setting isn't specified on the command line or in `puppet.conf`, it falls back to a default value. Default values for all settings are listed in the configuration reference. + +Some default values are based on other settings --- when this is the case, the default is shown using the other setting as a variable (similar to `$ssldir/certs`). + +## Configuring locale settings + +Puppet 5.1 added support for locale-specific strings in output, and it detects your locale from your system configuration. This provides localized strings, report messages, and log messages for the locale's language when available. + +Upon startup, Puppet looks for a set of environment variables on \*nix systems, or the code page setting on Windows. When Puppet finds one that is set, it uses that locale whether it is run from the command line or as a service. + +For help setting your operating system's locale or adding new locales, consult its documentation. This section covers setting the locale for Puppet services. + +### Checking your locale settings on \*nix and macOS + +To check your current locale settings, run the `locale` command. This outputs the settings used by your current shell. + +```console +$ locale +LANG="en_US.UTF-8" +LC_COLLATE="en_US.UTF-8" +LC_CTYPE="en_US.UTF-8" +LC_MESSAGES="en_US.UTF-8" +LC_MONETARY="en_US.UTF-8" +LC_NUMERIC="en_US.UTF-8" +LC_TIME="en_US.UTF-8" +LC_ALL= +``` + +To see which locales are supported by your system, run `locale -a`, which outputs a list of available locales. Note that Puppet might not have localized strings for every available locale. + +To check the current status of environment variables that might conflict with or override your locale settings, use the `set` command. For example, this command lists the set environment variables and searches for those containing `LANG` or `LC_`: + +```console +sudo set | egrep 'LANG|LC_' +``` + +### Checking your locale settings on Windows + +To check your current locale setting, run the `Get-WinSystemLocale` command from PowerShell. + +```powershell +PS C:\> Get-WinSystemLocale +LCID Name DisplayName +---- ---- ----------- +1033 en-US English (United States) +``` + +To check your system's current code page setting, run the `chcp` command. + +### Setting your locale on *nix with an environment variable + +You can use environment variables to set your locale for processes started on the command line. For most Linux distributions, set the `LANG` variable to your preferred locale, and the `LANGUAGE` variable to an empty string. On SLES, also set the `LC_ALL` variable to an empty string. + +For example, to set the locale to Japanese for a terminal session on SLES: + +```console +export LANG=ja_JP.UTF-8 +export LANGUAGE='' +export LC_ALL='' +``` + +To set the locale for the OpenVox agent service, you can add these `export` statements to: + +- `/etc/sysconfig/puppet` on RHEL and its derivatives +- `/etc/default/puppet` on Debian, Ubuntu, and their derivatives + +After updating the file, restart the Puppet service to apply the change. + +### Setting your locale for the OpenVox agent service on macOS + +To set the locale for the OpenVox agent service on macOS, update the `LANG` setting in the `/Library/LaunchDaemons/com.puppetlabs.puppet.plist` file. + +```xml + + LANG + ja_JP.UTF-8 + +``` + +After updating the file, restart the Puppet service to apply the change. + +### Setting your locale on Windows + +On Windows, Puppet uses the `LANG` environment variable if it is set. If not, it uses the configured region, as set in the Administrator tab of the Region control panel. + +On Windows 10, you can use PowerShell to set the system locale: + +```powershell +Set-WinSystemLocale en-US +``` + +### Disabling internationalized strings + +Puppet 5.3.2 added the optional Boolean `disable_i18n` setting, which you can configure in `puppet.conf`. If set to `true`, Puppet disables localized strings in log messages, reports, and parts of the command-line interface. +This can improve performance when using Puppet modules, especially if [environment caching](./configuration.html#environment_timeout) is disabled, and even if you don't need localized strings or the modules aren't localized. This setting is `false` by default in open source Puppet. + +If you're experiencing performance issues, configure this setting in the `[server]` section of the OpenVox Server's `puppet.conf` file. To force unlocalized messages, which are in English by default, configure this section in a node's `[main]` or `[user]` sections of `puppet.conf`. diff --git a/docs/_openvox_9x/config_file_auth.markdown b/docs/_openvox_9x/config_file_auth.markdown new file mode 100644 index 000000000..969d5e3c9 --- /dev/null +++ b/docs/_openvox_9x/config_file_auth.markdown @@ -0,0 +1,248 @@ +--- +layout: default +title: "Config files: auth.conf (LEGACY)" +--- + +[api]: /openvox-server/latest/http_api_index.html +[default_file]: https://github.com/puppetlabs/puppet/blob/4.3.0/conf/auth.conf +[environment]: ./environments_about.html +[server_auth_conf]: /openvox-server/latest/config_file_auth.html +[puppetserver.conf]: /openvox-server/latest/config_file_puppetserver.html +[confdir]: ./dirs_confdir.html + + +Access to Puppet's HTTPS API is configured in `auth.conf`. + +> ## **Important:** This is a deprecated config file +> +> Puppet Server has a [new HOCON-formatted `auth.conf` file][server_auth_conf], which is a full replacement for the old `auth.conf` format described on this page. The old `auth.conf` file will be removed in a future release of Puppet. +> +> Until then, Puppet Server uses a combination of the new `auth.conf` file and this legacy `auth.conf` file: +> +> * For most `/puppet/v3` endpoints, it defaults to the **legacy `auth.conf`.** +> * You can completely switch to the new `auth.conf` by setting [puppetserver.conf][] > `jruby-puppet` > `use-legacy-auth-conf: false`. +> * For `certificate_status` and `puppet-admin-api`, it uses the **new `auth.conf`.** +> * However, it will use the old `client-whitelist` settings instead if they're present. +> * For most `/puppet-ca/v1` endpoints and any new `/puppet/v3` endpoints added during the Puppet Server 2.x series, it only uses the **new `auth.conf`.** +> +> The default location of the new `auth.conf` is `/etc/puppetlabs/puppetserver/conf.d/auth.conf`. See [the Puppet Server `auth.conf` docs][server_auth_conf] for details. + +## Puppet's HTTPS API + +The OpenVox agent service requests configurations over HTTPS, and the OpenVox Server application provides several HTTPS endpoints to support this. (For example, requesting a catalog uses a different endpoint than submitting a report.) There are also a few endpoints that aren't used by OpenVox agent. + +Because some endpoints should have restricted access (for example, a node shouldn't request another node's configuration catalog), the OpenVox Server has a list of access rules for all of its HTTPS services. You can edit these rules in `auth.conf`. + +## Location + +The `auth.conf` file is located at `$confdir/auth.conf` by default. + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +## Example + +```text +path /puppet/v3/environments +method find +allow * + +# allow nodes to retrieve their own catalog +path ~ ^/puppet/v3/catalog/([^/]+)$ +method find +allow $1 + +# allow nodes to retrieve their own node definition +path ~ ^/puppet/v3/node/([^/]+)$ +method find +allow $1 + +# allow all nodes to store their own reports +path ~ ^/puppet/v3/report/([^/]+)$ +method save +allow $1 + +# control access to the custom user_files mount point +path ~ ^/puppet/v3/file_(metadata|content)s?/user_files/ +auth yes +allow *.example.com +allow_ip 192.168.100.0/24 + +# allow all nodes to access all file services. +path /puppet/v3/file +allow * + +path /puppet/v3/status +method find +allow * + +# allow all nodes to access the certificates services +path /puppet-ca/v1/certificate_revocation_list/ca +method find +allow * + +### Unauthenticated ACLs, for clients without valid certificates; authenticated +### clients can also access these paths, though they rarely need to. + +# allow access to the CA certificate; unauthenticated nodes need this +# in order to validate the OpenVox Server's certificate +path /puppet-ca/v1/certificate/ca +auth any +method find +allow * + +# allow nodes to retrieve the certificate they requested earlier +path /puppet-ca/v1/certificate/ +auth any +method find +allow * + +# allow nodes to request a new certificate +path /puppet-ca/v1/certificate_request +auth any +method find, save +allow * + +# deny everything else; this ACL is not strictly necessary, but +# illustrates the default policy. +path / +auth any +``` + +## Access control behavior + +Whenever OpenVox Server receives a valid HTTPS request, it checks it against its full list of authorization rules, in order. As soon as it finds a rule that matches the request, it will use that rule's `allow` and `allow_ip` permissions to decide whether to allow the request. If the request isn't allowed, Puppet will deny it, and will not check any further authorization rules. + +In other words, authorization rules work like simple firewall rules. If you want to specifically allow a request that could be caught and rejected by some more general rule, you need to put the more specific rule earlier in the auth.conf file. + +### Default auth rules + +OpenVox Server uses two sets of auth rules: the rules from auth.conf, which it checks first, and a set of hardcoded default rules, which it only checks if a request doesn't match any rules in auth.conf. + +If you are modifying auth.conf at all, **you should never rely on the hardcoded default rules.** Start with [a default auth.conf that explicitly includes copies of all of the default rules][default_file]. + +There are two reasons for this: + +* Visibility. It's easier to see where your custom rules should go if you can see the whole picture. +* Poor behavior in the default rules code. If an ACL in auth.conf has the same `path` value as a default rule, Puppet will magically exclude the default rule even if the additional directives in the ACL mean they match completely disjunct sets of nodes. + +## File format + +The auth.conf file is an ordered list of access control lists (ACLs). ACLs are separated by one or more empty lines. + +The file can also include comments, which are lines starting with `#`. Comments do not count as empty lines for separating ACLs. + +## ACL syntax + + +```text +path ~ ^/puppet/v3/report/([^/]+)$ +method save +allow $1 +``` + +An ACL is a series of adjacent lines, with one directive per line. It describes some set of requests, and says who is allowed to make those requests. + +The following directives describe which requests should match the ACL: + +* `path`: Which URLs the ACL applies to. **Required.** Must be the first directive in the ACL. +* `environment`: Which environments the ACL applies to. Optional; defaults to all environments. +* `method`: Which HTTP methods the ACL applies to. Optional; defaults to all methods. +* `auth`: Whether the ACL applies to client-verified or non-client-verified HTTPS requests. Optional; defaults to `yes` (verified). + +The following directives control who is allowed to make requests that match the ACL: + +* `allow`: Which certificate names or hostnames can make matching requests. Optional; defaults to allowing no one. +* `allow_ip`: Which IP addresses can make matching requests. Optional; defaults to allowing no one. + +An ACL can include multiple `allow` and `allow_ip` directives. + +There are also `deny` and `deny_ip` directives, but their behavior is complicated and unintuitive. Avoid them. + +### `path` + +Which URLs the ACL applies to. **Required.** Must be the first directive in the ACL. + +**Allowed values:** This directive must describe some set of URLs in the `puppet` or `puppet-ca` APIs. You can specify a group of URLs as a prefix, or as a regular expression. + +#### URL prefix + +```text +path /puppet/v3/report +``` + +If the value of `path` is just an absolute path, OpenVox Server interprets it as a prefix. The ACL will match any URL that _begins_ with that string. + +#### Regular expression + +```text +path ~ ^/puppet/v3/report/([^/]+)$ +``` + +If the value of `path` is a tilde (`~`), a space, and then a regular expression, the ACL will match any URL that matches the regular expression. Regexps in paths should NOT be delimited with slashes. + +> **Note:** You should almost always include at least a start anchor (`^`) in your regular expressions, to prevent them from matching URLs you didn't intend. + +If a regular expression path includes capturing parentheses, you can reference the captures in `allow` directives with numbered variables like `$1`. + +### `environment` + +Which environments the ACL applies to. + +**Allowed values:** A valid [environment][] or a comma-separated list of environments. Optional; defaults to all environments if omitted. + +Most of Puppet's endpoints require an environment to be provided as a URL parameter. See the [HTTPS API docs][api] for details. + +### `method` + +Which HTTP methods the ACL applies to. + +**Allowed values:** `find`, `search`, `save`, `destroy`, or a comma-separated list of those values. Optional; defaults to all methods if omitted. + +This directive is kind of obfuscated, and you have to map these indirector methods to the actual HTTP methods you want to control. + +Indirector | HTTP +-----------|------ +find | GET and POST +search | GET and POST, for endpoints whose names end in "s" or "_search" +save | PUT +destroy | DELETE + +### `auth` + +Whether the ACL applies to client-verified or non-client-verified HTTPS requests. + +**Allowed values:** `yes`, `any`, `no` (with `on` and `off` as synonyms). Must be a single value. Optional; defaults to `yes` (verified) if omitted. + +OpenVox agent makes client-verified requests to fetch configuration data and submit reports, but makes unverified requests to ask for a certificate. + +If you set `auth any`, it allows nodes to access an endpoint without a valid certificate. (Setting it to `no` is not very useful, since it will _reject_ requests that have valid certificates.) + +### `allow` + +Which certificate names or hostnames can make requests that match the ACL. For client-verified requests, Puppet will check `allow` directives against the common name (CN) from the client's SSL certificate. For unverified requests, Puppet will use reverse DNS to figure out the client's hostname, and compare that to the `allow` directives. + +**Allowed values:** One of the following (or a comma-separated list of them): + +* A certificate name (for client-verified requests) +* A hostname (for unverified requests only) +* A glob of certificate names or hostnames, with an asterisk (`*`) in place of the leftmost segment of the name (e.g. `*.delivery.example.com`). +* A regular expression, delimited with slashes (`/`), matching some number of certificate names or hostnames (e.g. `/^[\w-]+.example.com$/`). +* The string `*`, which will allow _all_ requests. + +Optional; if you don't specify any `allow` or `allow_ip` directives, Puppet will reject all requests matching the ACL. + +If an ACL's `path` was a regular expression with capturing parentheses, its `allow` directives can reference the captured text with numbered variables like `$1`. This is useful for things like requesting catalogs, where the name of the node is included in the URL and nodes should only be able to access their own catalogs. + +### `allow_ip` + +Which IP addresses can make matching requests. + +**Allowed values:** One of the following: + +* A single IP address. +* A glob representing a group of IP addresses (e.g. `192.168.100.*`). +* CIDR notation representing a group of IP addresses (e.g. `192.168.100.0/24`). + +An `allow_ip` directive will apply to both client-verified and unverified requests. + +Optional; if you don't specify any `allow` or `allow_ip` directives, Puppet will reject all requests matching the ACL. diff --git a/docs/_openvox_9x/config_file_autosign.markdown b/docs/_openvox_9x/config_file_autosign.markdown new file mode 100644 index 000000000..ff325d0cf --- /dev/null +++ b/docs/_openvox_9x/config_file_autosign.markdown @@ -0,0 +1,43 @@ +--- +layout: default +title: "Config files: autosign.conf" +--- + +[autosigning]: ./ssl_autosign.html +[autosign setting]: ./configuration.html#autosign +[confdir]: ./dirs_confdir.html + +The `autosign.conf` file can allow certain certificate requests to be automatically signed. It is only valid on the CA OpenVox Server server; an OpenVox Server not serving as a CA does not use `autosign.conf`. + +## More about autosigning + +> **Warning:** Because any host can provide any certname when requesting a certificate, basic autosigning is essentially **insecure**. Use it only when you fully trust any computer capable of connecting to the OpenVox Server. + +Puppet also provides a policy-based autosigning interface using custom policy executables, which can be more flexible and secure than the `autosign.conf` whitelist but more complex to configure. + +For more information, see [the documentation about certificate autosigning][autosigning]. + +## Location + +Puppet looks for `autosign.conf` at `$confdir/autosign.conf` by default. To change this path, configure the [`autosign` setting][autosign setting] in the `[server]` section of `puppet.conf`. + +The default `confdir` path depends on your operating system. [See the confdir documentation for more information.][confdir] + +> **Note:** The `autosign.conf` file must not be executable by the OpenVox Server's user account. If the `autosign` setting points to an executable file, Puppet instead treats it like a custom policy executable even if it contains a valid `autosign.conf` whitelist. + +## Format + +The `autosign.conf` file is a line-separated list of certnames or domain name globs. Each line represents a node name or group of node names for which the CA OpenVox Server will automatically sign certificate requests. + +```text +rebuilt.example.com +*.scratch.example.com +*.local +``` + +Domain name globs do not function as normal globs: an asterisk can only represent one or more subdomains at the front of a certname that resembles a fully qualified domain name (FQDN). If your certnames don't look like FQDNs, the `autosign.conf` whitelist might not be effective. + +> **Note:** The `autosign.conf` file can safely be an empty file or not-existent, even if the `autosign` setting is enabled. An empty or non-existent `autosign.conf` file is an empty whitelist, meaning that Puppet does not autosign any requests. +> If you create `autosign.conf` as a non-executable file and add certnames to it, Puppet then automatically uses the file to whitelist incoming requests without needing to modify `puppet.conf`. +> +> To _explicitly_ disable autosigning, set `autosign = false` in the `[server]` section of the CA OpenVox Server's `puppet.conf`, which disables CA autosigning even if `autosign.conf` or a custom policy executable exists. diff --git a/docs/_openvox_9x/config_file_csr_attributes.markdown b/docs/_openvox_9x/config_file_csr_attributes.markdown new file mode 100644 index 000000000..8ca563c7e --- /dev/null +++ b/docs/_openvox_9x/config_file_csr_attributes.markdown @@ -0,0 +1,72 @@ +--- +layout: default +title: "Config files: csr_attributes.yaml" +--- + +[csr_attributes]: configuration.html#csr_attributes + +The `csr_attributes.yaml` file defines custom data for new certificate signing requests (CSRs). It can set: + +* CSR attributes (transient data used for pre-validating requests) +* Certificate extension requests (permanent data to be embedded in a signed certificate) + +This file is only consulted when a new CSR is created (e.g. when an agent node is first attempting to join a Puppet deployment). It cannot modify existing certificates. + +> **Note:** For details on how to use this file, see the [documentation for CSR attributes and certificate extensions](./ssl_attributes_extensions.html). + +## Location + +The `csr_attributes.yaml` file is located at `$confdir/csr_attributes.yaml` by default. Its location is configurable with the [`csr_attributes` setting][csr_attributes]. + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +[confdir]: ./dirs_confdir.html + +## Example + +```yaml +--- +custom_attributes: + 1.2.840.113549.1.9.7: 342thbjkt82094y0uthhor289jnqthpc2290 +extension_requests: + pp_uuid: ED803750-E3C7-44F5-BB08-41A04433FE2E + pp_image_name: my_ami_image + pp_preshared_key: 342thbjkt82094y0uthhor289jnqthpc2290 +``` + +## Format + +The `csr_attributes` file must be a YAML hash containing one or both of the following keys: + +* `custom_attributes` +* `extension_requests` + +The value of each key must also be a hash, where: + +* Each key is a valid [object identifier (OID)](http://en.wikipedia.org/wiki/Object_identifier). Note that [Puppet-specific OIDs][puppet_oids] can optionally be referenced by short name instead of by numeric ID. (In the example above, `pp_uuid` is a short name for a Puppet-specific OID.) +* Each value is an object that can be cast to a string (that is, numbers are allowed but arrays are not). + +### Allowed OIDs for custom attributes + +Custom attributes can use any public or site-specific OID, **with the exception of the OIDs used for core X.509 functionality.** This means you can't re-use existing OIDs for things like subject alternative names. + +One useful OID is the "challengePassword" attribute --- `1.2.840.113549.1.9.7`. This is a rarely-used corner of X.509 which can easily be repurposed to hold a pre-shared key. The benefit of using this instead of an arbitrary OID is that it will appear by name when using OpenSSL to dump the CSR to text; OIDs that `openssl req` can't recognize will be displayed as numerical strings. + +Also note that the Puppet-specific OIDs listed below can also be used in CSR attributes. + +### Allowed OIDs for extension requests + +Extension request OIDs **must** be under the "ppRegCertExt" (`1.3.6.1.4.1.34380.1.1`) or "ppPrivCertExt" (`1.3.6.1.4.1.34380.1.2`) OID arcs. + +Puppet provides several registered OIDs (under "ppRegCertExt") for the most common kinds of extension information, as well as a private OID range ("ppPrivCertExt") for site-specific extension information. The benefits of using the registered OIDs are: + +* They can be referenced in `csr_attributes.yaml` using their short names instead of their numeric IDs. +* When using Puppet tools to print certificate info, they will appear using their descriptive names instead of their numeric IDs. + +The private range is available for any information you want to embed into a certificate that isn't already in wide use elsewhere. It is completely unregulated, and its contents are expected to be different in every Puppet deployment. + +[puppet_oids]: #puppet-specific-registered-ids + +#### Puppet-specific registered IDs + +{% include_relative _registered_oids.md %} diff --git a/docs/_openvox_9x/config_file_device.markdown b/docs/_openvox_9x/config_file_device.markdown new file mode 100644 index 000000000..2a40aad54 --- /dev/null +++ b/docs/_openvox_9x/config_file_device.markdown @@ -0,0 +1,57 @@ +--- +layout: default +title: "Config files: device.conf" +--- + +[puppet-device]: ./man/device.html +[deviceconfig]: ./configuration.html#deviceconfig +[confdir]: ./dirs_confdir.html + +The `puppet-device` subcommand retrieves catalogs from the OpenVox Server and applies them to remote devices. +Devices to be managed by the `puppet-device` subcommand are configured in `device.conf`. + +[See the puppet-device documentation for details.][puppet-device] + +## Location + +The `device.conf` file is located at `$confdir/device.conf` by default, +and its location is configurable with the [`deviceconfig`][deviceconfig] setting. + +The location of `confdir` depends on your operating system. +[See the confdir documentation for details.][confdir] + +## Format + +The `device.conf` file is an INI-like file, with one section per device: + +```ini +[device001.example.com] +type cisco +url ssh://admin:password@device001.example.com +debug +``` + +The section name specifies the `certname` of the device. + +The values for the `type` and `url` properties are specific to each type of device. + +The the optional `debug` property specifies transport-level debugging, +and is limited to telnet and ssh transports. + + +For Cisco devices, the `url` is in the following format: + +```text +scheme://user:password@hostname/query +``` + +with: + +* scheme: either `ssh` or `telnet` + +* user: optional connection username, depending on the device configuration + +* password: connection password + +* query: optional `?enable=` parameter whose value is the enable password + diff --git a/docs/_openvox_9x/config_file_environment.markdown b/docs/_openvox_9x/config_file_environment.markdown new file mode 100644 index 000000000..5202fc8ec --- /dev/null +++ b/docs/_openvox_9x/config_file_environment.markdown @@ -0,0 +1,96 @@ +--- +layout: default +title: "Config files: environment.conf" +--- + +[environment]: ./environments_about.html +[modulepath]: configuration.html#modulepath +[puppet.conf]: ./config_file_main.html +[basemodulepath]: configuration.html#basemodulepath +[main manifest]: ./dirs_manifest.html +[configuring_timeout]: ./configuration.html#environment_timeout + +Any [environment][] can contain an `environment.conf` file. This file can override several settings whenever the OpenVox Server is serving nodes assigned to that environment. + +## Location + +Each environment.conf file should be stored in an [environment][]. It should be at the top level of its home environment, next to the `manifests` and `modules` directories. + +For example, if your environments are in the default directory (`$codedir/environments`), the `test` environment's config file should be located at `$codedir/environments/test/environment.conf`. + +## Example + +```ini +# /etc/puppetlabs/code/environments/test/environment.conf + +# Puppet Enterprise requires $basemodulepath; see note below under "modulepath". +modulepath = site:dist:modules:$basemodulepath + +# Use our custom script to get a git commit for the current state of the code: +config_version = get_environment_commit.sh +``` + +## Format + +The environment.conf file uses the same INI-like format as [puppet.conf][], with one exception: it cannot contain config sections like `[main]`. All settings in environment.conf must be outside any config section. + +### Relative paths in values + +Most of the allowed settings accept **file paths** or **lists of paths** as their values. + +If any of these paths are **relative paths** --- that is, they start _without_ a leading slash or drive letter --- they will be resolved relative to that environment's main directory. + +For example, if you set `config_version = get_environment_commit.sh` in the `test` environment, Puppet will use the file at `/etc/puppetlabs/code/environments/test/get_environment_commit.sh`. + +### Interpolation in values + +The settings in environment.conf can use the values of other settings as variables (e.g., `$codedir`). Additionally, the `config_version` setting can use the special `$environment` variable, which gets replaced with the name of the active environment. + +The most useful variables to interpolate into environment.conf settings are: + +* `$basemodulepath` --- useful for including the default module directories in the `modulepath` setting. Puppet Enterprise users should usually include this in the value of `modulepath`, since PE uses modules in the `basemodulepath` to configure orchestration and other features. +* `$environment` --- useful as a command line argument to your `config_version` script. _You can interpolate this variable only in the `config_version` setting._ +* `$codedir` --- useful for locating files. + +## Allowed Settings + +{% include_relative _environment_conf_settings.md %} + +### `modulepath` + +The list of directories Puppet will load modules from. See [the reference page on the modulepath][modulepath] for more details about how Puppet uses it. + +If this setting isn't set, the modulepath for the environment will be: + +```text +:$basemodulepath +``` + +That is, Puppet will add the environment's `modules` directory to the value of the [`basemodulepath` setting][basemodulepath] from [puppet.conf][], with the environment's modules getting priority. If the `modules` directory is empty or absent, Puppet will only use modules from directories in the `basemodulepath`. A directory environment will never use the global `modulepath` from [puppet.conf][]. + +### `manifest` + +The [main manifest][] the OpenVox Server will use when compiling catalogs for this environment. This can be one file or a directory of manifests to be evaluated in alphabetical order. Puppet manages this path as a directory if one exists or if the path ends with a slash (`/`) or dot (`.`). + +If this setting isn't set, Puppet will use the environment's `manifests` directory as the main manifest, even if it is empty or absent. A directory environment will never use the global `manifest` from [puppet.conf][]. + +### `config_version` + +A script Puppet can run to determine the configuration version. + +Puppet automatically adds a **config version** to every catalog it compiles, as well as to messages in reports. The version is an arbitrary piece of data that can be used to identify catalogs and events. + +You can specify an executable script that will determine an environment's config version by setting `config_version` in its environment.conf file. Puppet will run this script when compiling a catalog for a node in the environment, and use its output as the config version. + +**Note:** If you're using a system binary like `git rev-parse`, make sure to specify the absolute path to it! If `config_version` is set to a relative path, Puppet will look for the binary _in the environment,_ not in the system's `PATH`. + +If this setting isn't set, the config version will be the **time** at which the catalog was compiled (as the number of seconds since January 1, 1970). A directory environment will never use the global `config_version` from [puppet.conf][]. + +### `environment_timeout` + +How long the OpenVox Server should cache the data it loads from an environment. If present, this will override the value of `environment_timeout` from [puppet.conf][]. + +* Unless you have a specific reason, we recommend only setting `environment_timeout` globally, in puppet.conf. +* We also don't recommend using any value other than `0` or `unlimited`. + +For more information about configuring the environment timeout, [see the timeout section of the Configuring Environments page.][configuring_timeout] diff --git a/docs/_openvox_9x/config_file_fileserver.markdown b/docs/_openvox_9x/config_file_fileserver.markdown new file mode 100644 index 000000000..ee00ef65d --- /dev/null +++ b/docs/_openvox_9x/config_file_fileserver.markdown @@ -0,0 +1,59 @@ +--- +layout: default +title: "Config files: fileserver.conf" +--- + +[file]: ./type.html#file +[module_files]: ./modules_fundamentals.html#files-in-modules +[fileserverconfig]: ./configuration.html#fileserverconfig +[auth_conf]: /openvox-server/latest/config_file_auth.html +[custom_mount]: ./file_serving.html +[mount_auth_examples]: ./file_serving.html#controlling-access-to-a-custom-mount-point-in-authconf + +The `fileserver.conf` file configures custom static mount points for Puppet's file server. If custom mount points are present, [`file` resources][file] can access them with their `source` attributes. + +## When to use `fileserver.conf` + +This file is only necessary if you are [creating custom mount points.][custom_mount] + +Puppet automatically serves files from the `files` directory of every module, and most users find this sufficient. ([More info on serving files from modules][module_files].) However, custom mount points are useful for things that shouldn't be stored in version control with your modules, like very large files and sensitive credentials. + +## Location + +The `fileserver.conf` file is located at `$confdir/fileserver.conf` by default. Its location is configurable with the [`fileserverconfig` setting][fileserverconfig]. + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +[confdir]: ./dirs_confdir.html + +## Example + +```ini +# Files in the /path/to/files directory will be served +# at puppet:///extra_files/. +[extra_files] + path /etc/puppetlabs/puppet/extra_files + allow * +``` + +This `fileserver.conf` file would create a new mount point named `extra_files`. + +> **Caution:** You should always restrict write access to mounted directories. The file server will follow any symlinks in a file server mount, including links to files that agent nodes should not access (like SSL keys). +> +> When following symlinks, the file server can access any files readable by Puppet Server's user account. + +## Format + +`fileserver.conf` uses a one-off format that resembles an INI file without the equals (`=`) signs. It is a series of mount-point stanzas, where each stanza consists of: + +* A `[mount_point_name]` surrounded by square brackets. This will become the name used in `puppet:///` URLs for files in this mount point. +* A `path ` directive, where `` is an absolute path on disk. This is where the mount point's files are stored. +* An `allow *` directive. + +### Deprecated security directives + +Before [`auth.conf`][auth_conf] existed, `fileserver.conf` could use `allow` and `deny` directives to control which nodes can access various files. This feature is now deprecated, and will be removed in a future release of Puppet. + +Instead, you can use `auth.conf` to control access to mount points. [The page on setting up mount points has details and examples.][mount_auth_examples] + +The only security directive that should be present in `fileserver.conf` is an `allow *` directive for every mount point. diff --git a/docs/_openvox_9x/config_file_main.markdown b/docs/_openvox_9x/config_file_main.markdown new file mode 100644 index 000000000..087c738bc --- /dev/null +++ b/docs/_openvox_9x/config_file_main.markdown @@ -0,0 +1,170 @@ +--- +layout: default +title: "Config files: The main config file (puppet.conf)" +--- + +[conf_ref]: ./configuration.html +[about]: ./config_about_settings.html +[short]: ./config_important_settings.html +[config]: ./configuration.html#config +[subcommands]: ./man/overview.html +[reports]: ./configuration.html#reports +[ssldir]: ./configuration.html#ssldir +[environmentpath]: ./configuration.html#environmentpath +[puppetserver_diff]: /openvox-server/latest/puppet_conf_setting_diffs.html + +The `puppet.conf` file is Puppet's main config file. It configures all of the Puppet commands and services, including OpenVox agent, OpenVox Server, Puppet apply, and Puppet cert. Nearly all of the settings listed in the [configuration reference][conf_ref] can be set in puppet.conf. + +It resembles a standard INI file, with a few syntax extensions. Settings can go into application-specific sections, or into a `[main]` section that affects all applications. + +(Useful background info: [about settings][about], [short list of settings][short], [full list of settings][conf_ref].) + + +## Location + +The puppet.conf file is always located at `$confdir/puppet.conf`. + +Although its location is configurable with the [`config` setting][config], it can only be set on the command line (e.g. `puppet agent -t --config ./temporary_config.conf`). + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +[confdir]: ./dirs_confdir.html + + +## Examples + +### Example agent config + +```ini +[main] +certname = agent01.example.com +server = puppet +environment = production +runinterval = 1h +``` + +### Example master config + +```ini +[main] +certname = puppetmaster01.example.com +server = puppet +environment = production +runinterval = 1h +strict_variables = true + +[server] +dns_alt_names = puppetmaster01,puppetmaster01.example.com,puppet,puppet.example.com +reports = puppetdb +storeconfigs_backend = puppetdb +storeconfigs = true +environment_timeout = unlimited +``` + +## Format + +The puppet.conf file consists of one or more **config sections,** each of which can contain any number of **settings.** + +The file can also include **comment lines** at any point. + +### Config sections + +```ini +[main] + certname = puppetmaster01.example.com +``` + +A config section is a group of settings. It consists of: + +* Its **name**, enclosed in square brackets. The `[name]` of the config section must be on its own line, with no leading space. +* Any number of **setting lines**, which can be indented for readability. +* Any number of empty lines or comment lines. + +As soon as a new config section `[name]` appears in the file, the former config section is closed and the new one begins. A given config section should only occur once in the file. + +Puppet uses four **config sections**: + +* `main` is the global section used by all commands and services. It can be overridden by the other sections. +* `master` is used by the OpenVox Server service and the Puppet cert command. +* `agent` is used by the OpenVox agent service. +* `user` is used by the Puppet apply command, as well as many of the less common [Puppet subcommands][subcommands]. + +Puppet prefers to use settings from one of the three application-specific sections (`master`, `agent`, or `user`). If it doesn't find a setting in the application section, it will use the value from `main`. (If `main` doesn't set one, it will fall back to the default value.) + +### Puppet Server ignores some config settings + +If you're using Puppet Server, you should note that it honors almost all settings in `puppet.conf` and should pick them up automatically. However, [some Puppet Server settings differ from a Ruby OpenVox Server’s `puppet.conf` settings][puppetserver_diff]. + +### Comment lines + +```ini +# This is a comment. +``` + +Comment lines start with a hash sign (`#`). They can be indented with any amount of leading space. + +Partial-line comments (e.g. `report = true # this enables reporting`) are not allowed, and will be treated as part of the value of the setting. To be treated as a comment, the hash sign must be the first non-space character on the line. + +### Setting lines + +```ini +certname = puppetmaster01.example.com +``` + +A setting line consists of: + +* Any amount of leading space (optional). +* The name of a **setting.** +* An **equals sign** (`=`), which can optionally be surrounded by any number of spaces. +* A **value** for the setting. + +### Special types of values for settings + +Generally, the value of a setting will be a single word. However, there are a few special types of values: + +#### Lists of words + +Some settings (like [`reports`][reports]) can accept multiple values, which should be specified as a comma-separated list (with optional spaces after commas). Example: `report = http,puppetdb` + +#### Paths + +Some settings (like [`environmentpath`][environmentpath]) take a list of directories. The directories should be separated by the system path separator character, which is colon (`:`) on \*nix platforms and semicolon (`;`) on Windows. + +```ini +# *nix version: +environmentpath = $codedir/special_environments:$codedir/environments +# Windows version: +environmentpath = $codedir/environments;C:\ProgramData\PuppetLabs\code\environment +``` + +Path lists are ordered; Puppet will always check the first directory first, then move on to the others if it doesn't find what it needs. + +#### Files or directories + +Settings that take a single file or directory (like [`ssldir`][ssldir]) can accept an optional hash of permissions. When starting up, Puppet will enforce those permissions on the file or directory. + +You generally shouldn't do this, as the defaults are good for most users. However, if you need to, you can specify permissions by putting a hash like this after the path: + +```ini +ssldir = $vardir/ssl {owner = service, mode = 0771} +``` + +The allowed keys in the hash are `owner`, `group`, and `mode`. There are only two valid values for the `owner` and `group` keys: + +* `root` --- the root or Administrator user or group should own the file. +* `service` --- the user or group that the Puppet service is running as should own the file. (The service's user and group are specified by the `user` and `group` settings. On an OpenVox Server running open source Puppet, these default to `puppet`; on Puppet Enterprise they default to `pe-puppet`.) + +### Interpolating variables in settings + +The values of settings are available as variables within puppet.conf, and you can insert them into the values of other settings. To reference a setting as a variable, prefix its name with a dollar sign (`$`): + +```ini +ssldir = $vardir/ssl +``` + +Not all settings are equally useful; there's no real point in interpolating `$ssldir` into `basemodulepath`, for example. We recommend that you use only the following variables: + +* `$codedir` +* `$confdir` +* `$vardir` + diff --git a/docs/_openvox_9x/config_file_oid_map.md b/docs/_openvox_9x/config_file_oid_map.md new file mode 100644 index 000000000..aae79820d --- /dev/null +++ b/docs/_openvox_9x/config_file_oid_map.md @@ -0,0 +1,65 @@ +--- +layout: default +title: "Config files: custom_trusted_oid_mapping.yaml" +--- + +[extensions]: ./ssl_attributes_extensions.html +[mapping_setting]: ./configuration.html#trusted_oid_mapping_file +[csr_attributes]: ./config_file_csr_attributes.html +[trusted]: ./lang_facts_and_builtin_vars.html#trusted-facts +[registered]: ./ssl_attributes_extensions.html#puppet-specific-registered-ids + +The `custom_trusted_oid_mapping.yaml` file lets you set your own short names for [certificate extension][extensions] object identifiers (OIDs), which can make [the `$trusted` variable][trusted] more useful. + +It is only valid on an OpenVox Server server; in Puppet apply, the compiler doesn't add certificate extensions to `$trusted`. + +## More about certificate extensions + +When a node requests a certificate, it can ask the CA to include some additional, permanent metadata in that cert. (OpenVox agent uses [the `csr_attributes.yaml` file][csr_attributes] to decide what extensions to request.) + +If the CA signs a certificate with extensions included, those extensions are available as [trusted facts][trusted] in the top-scope `$trusted` variable. Your manifests or node classifier can then use those trusted facts to decide which nodes can receive which configurations. + +By default, the [Puppet-specific registered OIDs][registered] appear as keys with convenient short names in the `$trusted[extensions]` hash, and any other OIDs appear as raw numerical IDs. You can use the `custom_trusted_oid_mapping.yaml` file to map other OIDs to short names, which will replace the numerical OIDs in `$trusted[extensions]`. + +For more info, see: + +* [CSR Attributes and Certificate Extensions][extensions] +* [The `csr_attributes.yaml` File][csr_attributes] +* [Trusted Facts][trusted] + +### Limitations of OID mapping + +Mapping OIDs in this file _only_ affects the keys in the `$trusted[extensions]` hash. It does not affect: + +* What an agent can request in its `csr_attributes.yaml` file --- anything but Puppet-specific registered extensions must still be numerical OIDs. + +## Location + +The OID mapping file is located at `$confdir/custom_trusted_oid_mapping.yaml` by default. Its location is configurable with [the `trusted_oid_mapping_file` setting][mapping_setting]. + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +[confdir]: ./dirs_confdir.html + + +## Example + +```yaml +--- +oid_mapping: + 1.3.6.1.4.1.34380.1.2.1.1: + shortname: 'myshortname' + longname: 'My Long Name' + 1.3.6.1.4.1.34380.1.2.1.2: + shortname: 'myothershortname' + longname: 'My Other Long Name' +``` + +## Format + +The `custom_trusted_oid_mapping.yaml` must be a YAML hash containing a single key called `oid_mapping`. + +The value of the `oid_mapping` key must be a hash whose keys are numerical OIDs. The value for each OID must be a hash with two keys: + +* `shortname` for the one-word name that will be used in the `$trusted[extensions]` hash. +* `longname` for a more descriptive name (not currently used for anything). diff --git a/docs/_openvox_9x/config_file_puppetdb.markdown b/docs/_openvox_9x/config_file_puppetdb.markdown new file mode 100644 index 000000000..e82842a8e --- /dev/null +++ b/docs/_openvox_9x/config_file_puppetdb.markdown @@ -0,0 +1,43 @@ +--- +layout: default +title: "Config files: puppetdb.conf" +--- + +[puppetdb_connection]: /openvoxdb/latest/puppetdb_connection.html + + +The `puppetdb.conf` file configures how Puppet should connect to one or more [PuppetDB](/openvoxdb/latest/) servers. It is only used if you are using PuppetDB and have [connected your OpenVox Server to it](/openvoxdb/latest/connect_puppet_server.html). + +## PuppetDB documentation + +If you're using PuppetDB 3.0 or higher, [`puppetdb.conf` is documented in the PuppetDB docs.][puppetdb_connection] + +## PuppetDB 2.3 and earlier + +The following description does **not** apply to PuppetDB 3.0 and later. If you're using a newer version, see [the PuppetDB docs][puppetdb_connection] instead, as the file format has changed. + +### Location + +The `puppetdb.conf` file is always located at `$confdir/puppetdb.conf`. Its location is **not** configurable. + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +[confdir]: ./dirs_confdir.html + +### Example + +```ini +[main] +server = puppetdb.example.com +port = 8081 +``` + +### Format + +The `puppetdb.conf` file uses the same INI-like format as `puppet.conf`, but only uses a `[main]` section and only has two settings: + +* The `server` setting must be set to the hostname of the PuppetDB server. +* The `port` setting must be set to the port of the PuppetDB server. + +See the [PuppetDB manual](/openvoxdb/latest/) for more information. + diff --git a/docs/_openvox_9x/config_file_routes.markdown b/docs/_openvox_9x/config_file_routes.markdown new file mode 100644 index 000000000..116a4d9cc --- /dev/null +++ b/docs/_openvox_9x/config_file_routes.markdown @@ -0,0 +1,39 @@ +--- +layout: default +title: "Config files: routes.yaml" +--- + +[route_file]: ./configuration.html#route_file + +The `routes.yaml` file overrides configuration settings involving indirector termini, and allows termini to be set in greater detail than `puppet.conf` allows. + +## Context + +The `routes.yaml` file works around config limitations in Puppet's internals and makes it possible to use certain extensions to Puppet, most notably PuppetDB. Most users shouldn't freely edit the file, and should only make changes that are explicitly recommended by the setup instructions for the extension they are trying to install. + +## Location + +The `routes.yaml` file is located at `$confdir/routes.yaml` by default. Its location is configurable with the [`route_file` setting][route_file]. + +The location of the `confdir` depends on your OS. [See the confdir documentation for details.][confdir] + +[confdir]: ./dirs_confdir.html + +## Example + +```yaml +--- +master: + facts: + terminus: puppetdb + cache: yaml +``` + +## Format + +The `routes.yaml` file should be a YAML hash. + +* Each top level key should be the name of a run mode (`master`, `agent`, or `user`), and its value should be another hash. + * Each key of these second-level hashes should be the name of an indirection, and its value should be another hash. + * The only keys allowed in these third-level hashes are `terminus` and `cache`. The value of each of these keys should be the name of a valid terminus for the indirection named above. + diff --git a/docs/_openvox_9x/config_important_settings.markdown b/docs/_openvox_9x/config_important_settings.markdown new file mode 100644 index 000000000..c4c67b598 --- /dev/null +++ b/docs/_openvox_9x/config_important_settings.markdown @@ -0,0 +1,158 @@ +--- +layout: default +title: "Configuration: Short list of important settings" +--- + +[cli_settings]: ./config_about_settings.html#settings-on-the-command-line +[config_reference]: ./configuration.html +[environments]: ./environments_about.html +[multi_master]: /openvox-server/latest/scaling_puppet_server.html +[enc]: ./nodes_external.html +[meta_noop]: ./metaparameter.html#noop +[meta_schedule]: ./metaparameter.html#schedule +[lang_tags]: ./lang_tags.html +[modulepath_dir]: ./dirs_modulepath.html +[report_reference]: ./report.html +[write_reports]: ./reporting_write_processors.html +[puppetdb_install]: /openvoxdb/latest/connect_puppet_server.html +[static_compiler]: ./static_catalogs.html +[ssl_autosign]: ./ssl_autosign.html + +[reports]: ./configuration.html#reports +[server]: ./configuration.html#server +[ca_server]: ./configuration.html#ca_server +[report_server]: ./configuration.html#report_server +[certname]: ./configuration.html#certname +[environment]: ./configuration.html#environment +[noop]: ./configuration.html#noop +[priority]: ./configuration.html#priority +[report]: ./configuration.html#report +[tags]: ./configuration.html#tags +[trace]: ./configuration.html#trace +[profile]: ./configuration.html#profile +[graph]: ./configuration.html#graph +[show_diff]: ./configuration.html#show_diff +[usecacheonfailure]: ./configuration.html#usecacheonfailure +[ignoreschedules]: ./configuration.html#ignoreschedules +[prerun_command]: ./configuration.html#prerun_command +[postrun_command]: ./configuration.html#postrun_command +[runinterval]: ./configuration.html#runinterval +[waitforcert]: ./configuration.html#waitforcert +[splay]: ./configuration.html#splay +[splaylimit]: ./configuration.html#splaylimit +[daemonize]: ./configuration.html#daemonize +[onetime]: ./configuration.html#onetime +[dns_alt_names]: ./configuration.html#dns_alt_names +[basemodulepath]: ./configuration.html#basemodulepath +[node_terminus]: ./configuration.html#node_terminus +[external_nodes]: ./configuration.html#external_nodes +[storeconfigs]: ./configuration.html#storeconfigs +[storeconfigs_backend]: ./configuration.html#storeconfigs_backend +[catalog_terminus]: ./configuration.html#catalog_terminus +[ca_ttl]: ./configuration.html#ca_ttl +[autosign]: ./configuration.html#autosign +[environmentpath]: ./configuration.html#environmentpath +[environment_timeout]: ./configuration.html#environment_timeout +[configuring_timeout]: ./configuration.html#environment_timeout +[puppetserver_config_files]: /openvox-server/latest/configuration.html +[settings_diffs]: /openvox-server/latest/puppet_conf_setting_diffs.html +[puppet_admin]: /openvox-server/latest/config_file_puppetserver.html +[jruby_puppet]: /openvox-server/latest/tuning_guide.html#puppet-server-and-jruby +[jvm_heap_config]: /openvox-server/latest/install_from_packages.html +[puppetserver_ca]: /openvox-server/latest/puppet_conf_setting_diffs.html +[service_bootstrap]: /openvox-server/latest/configuration.html#service-bootstrapping +[sourceaddress]: ./configuration.html#sourceaddress + +Puppet has about 200 settings, all of which are listed in the [configuration reference][config_reference]. Most users can ignore about 170 of those. + +This page lists the most important ones. (We assume here that you're okay with default values for things like the port Puppet uses for network traffic.) The link for each setting will go to the long description in the configuration reference. + +> **Why so many settings?** There are a lot of settings that are rarely useful but still make sense, but there are also at least a hundred that shouldn't be configurable at all. +> +> This is basically a historical accident. Due to the way Puppet's code is arranged, the settings system was always the easiest way to publish global constants that are dynamically initialized on startup. This means a lot of things have crept in there regardless of whether they needed to be configurable. + +## Settings for agents (all nodes) + + +Roughly in order of importance. Most of these can go in either `[main]` or `[agent]`, or be [specified on the command line][cli_settings]. + +### Basics + +* [`server`][server] --- The OpenVox Server server to request configurations from. Defaults to `puppet`; change it if that's not your server's name. + * [`ca_server`][ca_server] and [`report_server`][report_server] --- If you're using multiple masters, you'll need to centralize the CA; one of the ways to do this is by configuring `ca_server` on all agents. [See the multiple masters guide][multi_master] for more details. The `report_server` setting works about the same way, although whether you need to use it depends on how you're processing reports. +* [`certname`][certname] --- The node's certificate name, and the unique identifier it uses when requesting catalogs; defaults to the fully qualified domain name. + * For best compatibility, you should limit the value of `certname` to only use letters, numbers, periods, underscores, and dashes. (That is, it should match `/\A[a-z0-9._-]+\Z/`.) + * The special value `ca` is reserved, and can't be used as the certname for a normal node. +* [`environment`][environment] --- The [environment][environments] to request when contacting the OpenVox Server. It's only a request, though; the master's [ENC][] can override this if it chooses. Defaults to `production`. +* [`sourceaddress`][sourceaddress] --- The address on a multihomed host to use for the agent's communication with the master server. + +{% include_relative _nodename_certname.md %} + +### Run behavior + +These settings affect the way Puppet applies catalogs. + +* [`noop`][noop] --- If enabled, the agent won't do any work; instead, it will look for changes that _should_ be made, then report to the master about what it would have done. This can be overridden per-resource with the [`noop` metaparameter][meta_noop]. +* [`priority`][priority] --- Allows you to "nice" OpenVox agent so it won't starve other applications of CPU resources while it's applying a catalog. +* [`report`][report] --- Whether to send reports. Defaults to true; usually shouldn't be disabled, but you might have a reason. +* [`tags`][tags] --- Lets you limit the Puppet run to only include resources with certain [tags][lang_tags]. +* [`trace`][trace], [`profile`][profile], [`graph`][graph], and [`show_diff`][show_diff] --- Tools for debugging or learning more about an agent run. Extra-useful when combined with the `--test` and `--debug` CLI options. +* [`usecacheonfailure`][usecacheonfailure] --- Whether to fall back to the last known good catalog if the master fails to return a good catalog. The default behavior is good, but you might have a reason to disable it. +* [`ignoreschedules`][ignoreschedules] --- If you use [schedules][meta_schedule], this can be useful when doing an initial Puppet run to set up new nodes. +* [`prerun_command`][prerun_command] and [`postrun_command`][postrun_command] --- Commands to run on either side of a Puppet run. + +### Service behavior + +These settings affect the way OpenVox agent acts when running as a long-lived service. + +* [`runinterval`][runinterval] --- How often to do a Puppet run, when running as a service. +* [`waitforcert`][waitforcert] --- Whether to keep trying back if the agent can't initially get a certificate. The default behavior is good, but you might have a reason to disable it. + +### Useful when running agent from cron + +* [`splay`][splay] and [`splaylimit`][splaylimit] --- Together, these allow you to spread out agent runs. When running the agent as a daemon, the services will usually have been started far enough out of sync to make this a non-issue, but it's useful with cron agents. + For example, if your agent cron job happens on the hour, you could set `splay = true` and `splaylimit = 60m` to keep the master from getting briefly hammered and then left idle for the next 50 minutes. +* [`daemonize`][daemonize] --- Whether to daemonize. Set this to false when running the agent from cron. +* [`onetime`][onetime] --- Whether to exit after finishing the current Puppet run. Set this to true when running the agent from cron. + +## Settings for OpenVox Server servers + + +Many of these settings are also important for standalone Puppet apply nodes, since they act as their own OpenVox Server. + +These settings should usually go in `[server]`. However, if you're using Puppet apply in production, put them in `[main]` instead. + +### Basics + +* [`dns_alt_names`][dns_alt_names] --- A list of hostnames the server is allowed to use when acting as an OpenVox Server. The hostname your agents use in their `server` setting **must** be included in either this setting or the master's `certname` setting. Note that this setting is only used when initially generating the OpenVox Server's certificate --- if you need to change the DNS names, you must: + 1. Turn off the Puppet server service. + 2. Run `sudo puppetserver ca clean --certname `. + 3. Run `sudo puppetserver ca generate --certname --subject-alt-names ,,...`. + 4. Re-start the Puppet server service. +* [`environment_timeout`][environment_timeout] --- For better performance, you can set this to `unlimited` and make refreshing the OpenVox Server a part of your standard code deployment process. See [the timeout section of the Configuring Environments page][configuring_timeout] for more details. +* [`environmentpath`][environmentpath] --- Controls where Puppet finds directory environments. See [the page on directory environments][environments] for details. +* [`basemodulepath`][basemodulepath] --- A list of directories containing Puppet modules that can be used in all environments. [See the modulepath page][modulepath_dir] for details. +* [`reports`][reports] --- Which report handlers to use. For a list of available report handlers, see [the report reference][report_reference]. You can also [write your own report handlers][write_reports]. Note that the report handlers might require settings of their own. + +### Puppet Server related settings + +Puppet Server has [its own configuration files][puppetserver_config_files]; consequently, there are [several settings in `puppet.conf` that Puppet Server ignores][settings_diffs]. + +* [`puppet-admin`][puppet_admin] --- Settings to control which authorized clients can use the admin interface. +* [`jruby-puppet`][jruby_puppet] --- Provides details on tuning JRuby for better performance. +* [`JAVA_ARGS`][jvm_heap_config] --- Instructions on tuning the Puppet Server memory allocation. + +### Extensions + +These features configure add-ons and optional features. + +* [`node_terminus`][node_terminus] and [`external_nodes`][external_nodes] --- The ENC settings. If you're using an [ENC][], set these to `exec` and the path to your ENC script, respectively. +* [`storeconfigs`][storeconfigs] and [`storeconfigs_backend`][storeconfigs_backend] --- Used for setting up PuppetDB. See [the PuppetDB docs for details.][puppetdb_install] +* [`catalog_terminus`][catalog_terminus] --- This can enable the optional static compiler. If you have lots of `file` resources in your manifests, the static compiler lets you sacrifice some extra CPU work on your OpenVox Server to gain faster configuration and reduced HTTPS traffic on your agents. [See the static catalogs page][static_compiler] for details. + +### CA settings + +* **Acting as a CA** --- OpenVox Server provides the certificate authority, run as a JRuby service rather than a `ca` setting. There should be only one CA in a deployment, so in a [multi-server][multi_master] setup you disable the CA service on all but one node via [service bootstrapping][service_bootstrap] (the `ca.cfg` file). See also the [OpenVox Server `ca` settings][puppetserver_ca]. + +* [`ca_ttl`][ca_ttl] --- How long newly signed certificates should be valid for. +* [`autosign`][autosign] --- Whether (and how) to autosign certificates. See [the autosigning page][ssl_autosign] for details. diff --git a/docs/_openvox_9x/config_print.markdown b/docs/_openvox_9x/config_print.markdown new file mode 100644 index 000000000..ad52cafb9 --- /dev/null +++ b/docs/_openvox_9x/config_print.markdown @@ -0,0 +1,133 @@ +--- +layout: default +title: "Configuration: Checking values of settings" +--- + + +[config_sections]: ./config_file_main.html#config-sections +[setting_sources]: ./config_about_settings.html +[confdir_sys]: ./dirs_confdir.html#location +[environment]: ./environments_about.html +[confdir]: ./dirs_confdir.html +[vardir]: ./dirs_vardir.html +[modulepath]: ./dirs_modulepath.html + +Puppet settings are highly dynamic, and their values can come from [several different places][setting_sources]. + +To see the actual settings values that a Puppet service will use, it's often best to ask Puppet itself. The `puppet config print` command lets you do this. + +## General Usage + +The `puppet config print` command loads and evaluates settings, and can imitate any of Puppet's other commands and services when doing so. The `--section` and `--environment` options let you control how settings are loaded; for details, see the sections below on imitating different services. + +> **Note:** To ensure that you're seeing the values Puppet will use when running as a service, be sure to use `sudo` or run the command as `root` or `Administrator`. If you run `puppet config print` as some other user, Puppet might not use the [system config file.][confdir_sys] + +### To see the value of one setting + +```console +sudo puppet config print [--section ] [--environment ] +``` + +This will show _just the value_ of ``. + +--- + +### To see the value of multiple settings + +```console +sudo puppet config print [...] [--section ] [--environment ] +``` + +This will show `name = value` pairs for all requested settings. + +--- + +### To see the value of all settings + +```console +sudo puppet config print [--section ] [--environment ] +``` + +This will show `name = value` pairs for all settings. + + +### Config Sections + +The `--section` option specifies which [section of puppet.conf][config_sections] to use when finding settings. It is optional, and defaults to `main`. Valid sections are: + +* `main` **(default)** --- used by all commands and services +* `master` --- used by the OpenVox Server service +* `agent` --- used by the OpenVox agent service +* `user` --- used by the Puppet apply command and most other commands + +As usual, the other sections will override the `main` section if they contain a setting; if they don't, the value from `main` will be used, or a default value if the setting isn't present there. + +### Environments + +The `--environment` option specifies which [environment][] to use when finding settings. It is optional and defaults to the value of the `environment` setting in the `user` section (usually `production`, since it's rare to specify an environment in `user`). + +Note that you can only specify environments that already exist. + +This option is generally only useful when looking up settings used by the OpenVox Server service, since it's rare to use environment config sections for Puppet apply and OpenVox agent. + + +## Imitating OpenVox Server + +To see the settings the OpenVox Server service would use: + +* Specify `--section server`. +* Use the `--environment` option to specify the environment you want settings for, or let it default to `production`. +* Remember to use `sudo`. +* If your OpenVox Server uses a non-standard [confdir][] or [vardir][], specify them on the command line with the `--confdir` and `--vardir` options; otherwise you might not see the correct values for settings. + +### Examples + +To see the effective [modulepath][] used in the `dev` environment: + +```console +$ sudo puppet config print modulepath --section server --environment dev +/etc/puppetlabs/code/environments/dev/modules:/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules +``` + +To see whether PuppetDB is configured for exported resources: + +```console +$ sudo puppet config print storeconfigs storeconfigs_backend --section server +storeconfigs = true +storeconfigs_backend = puppetdb +``` + +## Imitating Puppet Agent + +To see the settings the OpenVox agent service would use: + +* Specify `--section agent`. +* Remember to use `sudo`. +* If you are seeing something unexpected, check your OpenVox agent init script or cron job to make sure it is using the standard [confdir][] and [vardir][], is running as root, and isn't overriding other settings with command line options. If it's doing anything unusual, you might have to set more options for the config print command. + +### Example + +To see whether the agent is configured to use manifest ordering when applying the catalog: + +```console +$ sudo puppet config print ordering --section agent +manifest +``` + +## Imitating Puppet Apply + +To see the settings the Puppet apply command would use: + +* Specify `--section user`. +* Remember to use `sudo`. +* If you are seeing something unexpected, check the cron job or script that is responsible for configuring the machine with Puppet apply. Make sure it is using the standard [confdir][] and [vardir][], is running as root, and isn't overriding other settings with command line options. If it's doing anything unusual, you might have to set more options for the config print command. + +### Example + +To see whether Puppet apply is configured to use reports: + +```console +$ sudo puppet config print report reports --section user +report = true +reports = store,http +``` diff --git a/docs/_openvox_9x/config_set.markdown b/docs/_openvox_9x/config_set.markdown new file mode 100644 index 000000000..02117b6bb --- /dev/null +++ b/docs/_openvox_9x/config_set.markdown @@ -0,0 +1,89 @@ +--- +layout: default +title: "Configuration: Editing settings on the command line" +--- + +[config_sections]: ./config_file_main.html#config-sections +[puppet.conf]: ./config_file_main.html +[confdir_sys]: ./dirs_confdir.html#location + +Puppet loads most of its settings from [the puppet.conf config file.][puppet.conf] You can edit this file directly, or you can change individual settings with the `puppet config set` command. + +> ## When to Use This +> +> We recommend using `puppet config set` for: +> +> * Fast one-off config changes +> * Scriptable config changes in provisioning tools +> +> If you find yourself changing many settings at once, you might prefer to edit the puppet.conf file or manage it with a template. + +## Usage + + +To assign a new value to a setting, run: + +```console +sudo puppet config set --section +``` + +This will declaratively set the value of `` to `` (in the specified config section). It will work the same way regardless of whether the setting already had a value. + +### Config sections + +The `--section` option specifies which [section of puppet.conf][config_sections] to modify. It is optional, and defaults to `main`. Valid sections are: + +* `main` **(default)** --- used by all commands and services +* `master` --- used by the OpenVox Server service +* `agent` --- used by the OpenVox agent service +* `user` --- used by the Puppet apply command and most other commands + +If modifying the [system config file][confdir_sys], be sure to use `sudo` or run the command as `root` or `Administrator`. + +## Example + + +**Before:** + +```ini +# /etc/puppetlabs/puppet/puppet.conf +[main] +certname = agent01.example.com +server = master.example.com +vardir = /var/opt/lib/pe-puppet + +[agent] +report = true +graph = true +pluginsync = true + +[server] +dns_alt_names = master,master.example.com,puppet,puppet.example.com +``` + +**Commands:** + +```console +sudo puppet config set reports puppetdb --section server +sudo puppet config set ordering manifest +``` + +**After:** + +```ini +# /etc/puppetlabs/puppet/puppet.conf +[main] +certname = agent01.example.com +server = master.example.com +vardir = /var/opt/lib/pe-puppet +ordering = manifest + +[agent] +report = true +graph = true +pluginsync = true + +[server] +dns_alt_names = master,master.example.com,puppet,puppet.example.com +reports = puppetdb +``` diff --git a/docs/_openvox_9x/config_ssl_external_ca.markdown b/docs/_openvox_9x/config_ssl_external_ca.markdown new file mode 100644 index 000000000..db40f0597 --- /dev/null +++ b/docs/_openvox_9x/config_ssl_external_ca.markdown @@ -0,0 +1,112 @@ +--- +layout: default +title: "SSL configuration: External CA support" +--- + +[conf]: ./config_file_main.html + +In lieu of its built-in certificate authority (CA) and public key infrastructure (PKI) tools, Puppet can use an existing external CA for all of its secure socket layer (SSL) communications. + +This page describes the supported and tested configurations for external CAs in this version of Puppet. If you have an external CA use case that isn't covered here, please contact Puppet so we can learn more about it. + +## Supported external CA configurations + +This version of Puppet supports _some_ external CA configurations, but not every possible arrangement. We fully support the following setups: + +1. [Single self-signed CA which directly issues SSL certificates.](#option-1-single-ca) +2. [Puppet Server functioning as an intermediate CA of a root self-signed CA.](#option-2-puppet-server-functioning-as-an-intermediate-ca) + +These are fully supported by Puppet, which means: + +* Issues that arise in one of these three arrangements are considered **bugs,** and we'll fix them ASAP. +* Issues that arise in any _other_ external CA setup are considered **feature requests,** and we'll consider whether to expand our support. + +## General notes and requirements + +### PEM encoding of credentials is mandatory + +Puppet always expects its SSL credentials to be in `.pem` format. + +### Normal Puppet certificate requirements still apply + +Any Puppet Server certificate must contain the DNS name at which agent nodes will attempt to contact that server, either as the subject common name (CN) or as a Subject Alternative Name (DNS). + +## Option 1: Single CA + +When Puppet uses its internal CA, it defaults to a single CA configuration. A single externally issued CA can also be used in a similar manner. + +```text + +------------------------+ + | | + | Root self-signed CA | + | | + +------+----------+------+ + | | + +----------+ +------------+ + | | + v v + +-----------------+ +----------------+ + | | | | + | Server SSL Cert | | Agent SSL Cert | + | | | | + +-----------------+ +----------------+ +``` + +This configuration is all-or-nothing rather than mix-and-match. When using an external CA, the built-in Puppet CA service **must** be disabled and cannot be used to issue SSL certificates. + +Additionally, Puppet cannot automatically distribute certificates in this configurations --- you must have your own complete system for issuing and distributing certificates. + +### Puppet Server + +Configure Puppet Server in three steps: + +* Disable the internal CA service. +* Ensure that the certname will never change. +* Put certificates/keys in place on disk. + +1. Edit Puppet Server's `/etc/puppetlabs/puppetserver/services.d/ca.cfg` file to disable the internal CA. Comment out the line following "To enable the CA service..." and uncomment the line following "To disable the CA service...", as follows: + + ```text + # To enable the CA service, leave the following line uncommented + # puppetlabs.services.ca.certificate-authority-service/certificate-authority-service + # To disable the CA service, comment out the above line and uncomment the line below + puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service + ``` + +2. Set a static value for the `certname` setting in [`puppet.conf`][conf]: + + ```ini + [server] + certname = puppetserver.example.com + ``` + + Setting a static value keeps Puppet from getting confused if the machine's hostname ever changes. The value must be whatever certname you'll use to issue the server's certificate. It must not be blank. +3. Put the credentials from your external CA on disk in the correct locations. These locations must match what's configured in your [webserver.conf file](/openvox-server/latest/config_file_webserver.html). If you haven't changed those settings, you can run the following commands to find the default locations: + + Credential | File location + -----------------------------------|------------------------------------------- + Server SSL certificate | `puppet config print hostcert --section server` + Server SSL certificate private key | `puppet config print hostprivkey --section server` + Root CA certificate | `puppet config print localcacert --section server` + Root certificate revocation list | `puppet config print hostcrl --section server` + +If you've put the credentials in the correct locations, you shouldn't need to change any additional settings. + +### OpenVox agent + +You don't need to change any settings. + +Put the external credentials into the correct filesystem locations. You can run the following commands to find the appropriate locations: + +Credential | File location +----------------------------------|----------------------------------------- +Agent SSL certificate | `puppet config print hostcert --section agent` +Agent SSL certificate private key | `puppet config print hostprivkey --section agent` +Root CA certificate | `puppet config print localcacert --section agent` +Root certificate revocation list | `puppet config print hostcrl --section agent` + +## Option 2: Puppet Server functioning as an intermediate CA + +Puppet Server can operate as an intermediate CA to an external root CA. Please +refer to [Using Puppet Server as an intermediate certificate authority](/openvox-server/latest/intermediate_ca.html) +for a complete walk through of setting it up. diff --git a/docs/_openvox_9x/custom_types.markdown b/docs/_openvox_9x/custom_types.markdown new file mode 100644 index 000000000..44a00c01f --- /dev/null +++ b/docs/_openvox_9x/custom_types.markdown @@ -0,0 +1,512 @@ +--- +layout: default +title: Custom Types +--- + +[package_type]: type.html#package +[module]: modules_fundamentals.html +[custom_functions]: lang_write_functions_in_puppet.html +[custom_facts]: /openfact/latest/custom_facts.html +[symbol]: http://www.ruby-doc.org/core/Symbol.html +[ruby_block]: http://www.robertsosinski.com/2008/12/21/understanding-ruby-blocks-procs-and-lambdas/ +[markdown]: http://commonmark.org/ +[metaparameters]: lang_resources.html#metaparameters +[inpage_whitespace]: #type-documentation +[namevar]: lang_resources.html#namenamevar + +# Custom Types + +This page describes how to create your own custom resource types to add new resource types to Puppet. It covers the nature of the type/provider split, how to develop the type file, and how types and providers interact; for more complete details on developing providers, see [the Provider Development page](./provider_development.html). + +Puppet types and providers must always be written in Ruby. If you're new to Ruby, what is going on should still be somewhat evident from the examples below, but some experience with Ruby is definitely recommended. + +The internals of how types are created have changed over Puppet's lifetime, and this document will focus on best practices, skipping over all the things you can but probably shouldn't do. + +> **Note:** Often the best way to learn types and providers is to read the existing type and providers in Puppet's core codebase. +> One warning: Don't start with the `file` type; start with `user` or `package` instead. New extension writers often expect that `file` would be a nice easy one to get started with, and it's actually an incredibly complicated morass of special cases that most types just don't have to deal with. +> +> User or package, not file. + +## Types and Providers + +When making a new Puppet type, you will create two things: + +* The "type" itself, which is a model of the resource type. It defines what parameters are available, handles input validation, and determines what features a provider can (or should) provide. +* One or more providers for that type, which implements the type by translating its capabilities into specific operations on a system. (For example, the [package][package_type] has `yum` and `apt` providers which implement package resources on Red Hat-like and Debian-like systems, respectively.) + +## Deploying and Using Types and Providers + +To use new types and providers, two conditions must be met: + +1. The type and providers must be present in a [module][] on the OpenVox Server server. Like other types of plugin (such as [custom functions][custom_functions] and [custom facts][custom_facts]), they should go in the module's `lib` directory: + * Type files should be located at `lib/puppet/type/.rb`. + * Provider files should be located at `lib/puppet/provider//.rb`. +2. In an agent/server deployment, agent nodes automatically sync plugins (including custom types and providers) from the server's modules at the start of each run, so no configuration is needed. (The old `pluginsync` setting that controlled this was removed in OpenVox 9.) + +When running without a server using puppet apply, plugin sync is not involved, but the module containing the type and providers must be present on each node. + +See [the Plugins In Modules page](./plugins_in_modules.html) for more details on distributing custom types and facts via modules. + + +## Types + +When defining the resource type, focus on what the resource can do, +not how it does it. + +### Creating a Type + +Types are created by calling the `newtype` method on the `Puppet::Type` class: + +```ruby + # lib/puppet/type/database.rb + Puppet::Type.newtype(:database) do + @doc = "Create a new database." + # ... the code ... + end +``` + +* The name of the type is the only required argument to `newtype`. The name must be a [Ruby symbol][symbol], and the name of the file containing the type must match the type's name. +* The `newtype` method also requires a [block of code][ruby_block], specified with either curly braces (`{ ... }`) or the `do ... end` syntax. This code block will implement the type, and contains all of the properties and parameters. The block will not be passed any arguments. + +> #### Options +> +> When creating a type, you can also specify options after the name. There is currently only one option available. +> +> Options must be specified as a hash, although Ruby method arguments allow you to leave the curly braces off of hashes. +> +> * `:self_refresh => true` --- Cause resources of this type to **refresh** (as if they had received an event via a notify/subscribe relationship) whenever a change is made to the resource. Most notably used in the core `mount` type. + + +### Type Documentation + +You can and should write a string describing the resource type and assign it to the `@doc` instance variable. This string can be extracted by the `puppet doc --reference type` command (which outputs a complete type reference which will include your new type) and the `puppet describe` command (which outputs information about specific types). + +The string should be in [Markdown][] format (avoiding dialect-specific features that aren't universally supported). When the Puppet tools extract the string, they will strip the greatest common amount of leading whitespace from the front of each line, excluding the first line. For example: + +```ruby +Puppet::Type.newtype(:database) do + @doc = %q{Creates a new database. Depending + on the provider, this might create relational + databases or NoSQL document stores. + + Example: + + database { 'mydatabase': + ensure => present, + owner => root, + } + } +end +``` + +In this example, any whitespace would be trimmed from the first line (in this case, it's zero spaces), then the greatest common amount would be trimmed from remaining lines. +Three lines have four leading spaces, two lines have six, and two lines have eight, so four leading spaces would be trimmed from each line. This leaves the example code block indented by four spaces, and thus doesn't break the Markdown formatting. + +### Properties and Parameters + +The bulk of a type consists of **properties** and **parameters.** + +Both properties and parameters will become the resource attributes available when declaring a resource of the new type. The difference between the two is subtle but important: + +* Properties should map more or less directly to something measurable on the target system. For example, the UID and GID of a user account would be properties, since their current state can be queried or changed. In practical terms, setting a value for a property causes a method to be called on the provider. +* Parameters change how Puppet manages a resource, but do not necessarily map directly to something measurable. For example, the `user` type's `managehome` attribute is a parameter --- its value affects what Puppet does, but the question of whether Puppet is managing a home directory isn't an innate property of the user account. + +Additionally, there are a few special attributes called [metaparameters][], which are supported by all resource types. These don't need to be handled when creating new types; they're implemented elsewhere. + +A normal type will define **multiple properties** and must define **at least one parameter.** + + +### Properties + +Here's where we define how the resource really works. In most +cases, it's the properties that interact with your resource's +providers. If you define a property named owner, then when you are +retrieving the state of your resource, then the owner property will +call the owner method on the provider. In turn, when you are +setting the state (because the resource is out of sync), then the +owner property will call the owner= method to set the state on +disk. + +There's one common exception to this: The ensure property is +special because it's used to create and destroy resources. You can +set this property up on your resource type just by calling the +ensurable method in your type definition: + +```ruby + Puppet::Type.newtype(:database) do + ensurable + ... + end +``` + +This property uses three methods on the provider: create, destroy, +and exists?. The last method, somewhat obviously, is a boolean to +determine if the resource current exists. If a resource's ensure +property is out of sync, then no other properties will be checked +or modified. + +You can modify how ensure behaves, such as by adding other valid +values and determining what methods get called as a result; see +existing types like package for examples. + +The rest of the properties are defined a lot like you define the +types, with the newproperty method, which should be called on the +type: + +```ruby + Puppet::Type.newtype(:database) do + ensurable + newproperty(:owner) do + desc "The owner of the database." + ... + end + end +``` + +Note the call to desc; this sets the documentation string for this +property, and for Puppet types that get distributed with Puppet, it +is extracted as part of the Type reference. + +When Puppet was first developed, there would normally be a lot of +code in this property definition. Now, however, you normally only +define valid values or set up validation and munging. If you +specify valid values, then Puppet will only accept those values, +and it will automatically handle accepting either strings or +symbols. In most cases, you only define allowed values for ensure, +but it works for other properties, too: + +```ruby + newproperty(:enable) do + newvalue(:true) + newvalue(:false) + end +``` + +You can attach code to the value definitions (this code would be +called instead of the property= method), but it's normally +unnecessary. + +For most properties, though, it is sufficient to set up +validation: + +```ruby + newproperty(:owner) do + validate do |value| + unless value =~ /^\w+/ + raise ArgumentError, "%s is not a valid user name" % value + end + end + end +``` + +Note that the order in which you define your properties can be +important: Puppet keeps track of the definition order, and it +always checks and fixes properties in the order they are defined. + +#### Customizing Behaviour + +By default, if a property is assigned multiple values in an array: + +* It is considered in sync if _any_ of those values matches the current +value. +* If none of those values match, the _first one_ will be used when syncing the property. + +If, instead, the property should only be in sync if _all_ +values match the current value (e.g., a list of times in a cron +job), you can declare this: + +```ruby + newproperty(:minute, :array_matching => :all) do # :array_matching defaults to :first + ... + end +``` + +You can also customize how information about your property gets +logged. You can create an `is_to_s` method to change how the +current values are described, `should_to_s` to change how the +desired values are logged, and `change_to_s` to change the overall +log message for changes. See current types for examples. + +#### Handling Property Values + +Handling values set on properties is currently somewhat confusing, +and will hopefully be fixed in the future. When a resource is +created with a list of desired values, those values are stored in +each property in its @should instance variable. You can retrieve +those values directly by calling should on your resource (although +note that when `:array_matching` is set to `:first` you get the first +value in the array, otherwise you get the whole array): + +```ruby + myval = should(:color) +``` + +When you're not sure (or don't care) whether you're dealing with a +property or parameter, it's best to use value: + +```ruby + myvalue = value(:color) +``` + +### Parameters + +Parameters are defined essentially exactly the same as properties; +the only difference between them is that parameters never result in +methods being called on providers. + +To define a new parameter, call the `newparam` method. This method takes the name of the parameter (as a symbol) as its argument, as well as a block of code. You can and should provide documentation for each parameter by calling the `desc` method inside its block. Leading whitespace is trimmed from multiline strings [as described above][inpage_whitespace]. + +```ruby + newparam(:name) do + desc "The name of the database." + end +``` + +#### Namevar + +Every type must have at least **one mandatory parameter:** the [**namevar.**][namevar] This parameter will uniquely identify each resource of the type on the target system --- for example, the path of a file on disk, the name of a user account, or the name of a package. + +If the user doesn't specify a value for the namevar when declaring a resource, its value will default to the **title** of the resource. + +There are three ways to designate a namevar. Every type must have **exactly one** parameter that meets **exactly one** of these criteria: + +**Option 1:** Create a parameter whose name is `:name`. Since most types just use `:name` as the namevar, it gets special treatment and will automatically become the namevar. + +```ruby + newparam(:name) do + desc "The name of the database." + end +``` + +**Option 2:** Provide the `:namevar => true` option as an additional argument to the `newparam` call. This allows you to use a namevar with a different, more descriptive name (such as the `file` type's `path` parameter). + +```ruby + newparam(:path, :namevar => true) do + ... + end +``` + +**Option 3:** Call the `isnamevar` method (which takes no arguments) inside the parameter's code block. This allows you to use a namevar with a different, more descriptive name. There is no practical difference between this and option 2. + +```ruby + newparam(:path) do + isnamevar + ... + end +``` + +> ##### Errors When Namevar is Absent +> +> If you try to create a type that lacks a namevar, you'll see one of two errors when declaring resources of that type, depending on the Puppet version. +> +> **Puppet 2.7:** +> +> ```console +> $ puppet apply -e "testing { h: }" +> Error: undefined method `merge' for []:Array +> ``` +> +> **Puppet 3:** +> +> ```console +> $ puppet apply -e "testing { h: }" +> Error: No set of title patterns matched the title "h". +> ``` +> +> The fact that these are not particularly helpful is tracked as [issue 5220](http://projects.puppetlabs.com/issues/5220). + +#### Specifying Allowed Values + +If your parameter has a fixed list of valid values, you can declare +them all at once: + +```ruby + newparam(:color) do + newvalues(:red, :green, :blue, :purple) + end +``` + +You can specify regexes in addition to literal values; matches +against regexes always happen after equality comparisons against +literal values, and those matches are not converted to symbols. For +instance, given the following definition: + +```ruby + newparam(:color) do + desc "Your color, and stuff." + + newvalues(:blue, :red, /.+/) + end +``` + +If you provide blue as the value, then your parameter will get set +to :blue, but if you provide green, then it will get set to +"green". + +#### Validation and Munging + +If your parameter does not have a defined list of values, or you +need to convert the values in some way, you can use the validate +and munge hooks: + +```ruby + newparam(:color) do + desc "Your color, and stuff." + + newvalues(:blue, :red, /.+/) + + validate do |value| + if value == "green" + raise ArgumentError, + "Everyone knows green databases don't have enough RAM" + else + super + end + end + + munge do |value| + case value + when :mauve, :violet # are these colors really any different? + :purple + else + super + end + end + end +``` + +The default validate method looks for values defined using +newvalues and if there are any values defined it accepts only those +values (this is exactly how allowed values are validated). The +default munge method converts any values that are specifically +allowed into symbols. If you override either of these methods, note +that you lose this value handling and symbol conversion, which +you'll have to call super for. + +Values are always validated before they're munged. + +Lastly, validation and munging _only\*_ happen when a value is +assigned. They have no role to play at all during use of a given +value, only during assignment. + +#### Boolean Parameters + +Boolean parameters are common. To avoid repetition, some utilities are available: + +```ruby + require 'puppet/parameter/boolean' + # ... + newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean) +``` + +There are two parts here. The `:parent => Puppet::Parameter::Boolean` part +configures the parameter to accept lots of names for true and false, to make +things easy for your users. The `:boolean => true` creates a boolean method +on the type class to return the value of the parameter. In this example, the +method would be named `force?`. + +### Automatic Relationships + +Your type can specify automatic relationships it can have with +resources. You can use autorequire, autobefore, autonotify, and +autosubscribe, which all require a resource type as an argument, +and your code should return a list of resource names that your +resource could be related to. + +```ruby + autorequire(:user) do + self[:user] + end +``` + +Note that this won't throw an error if resources with those names +do not exist; the purpose of this hook is to make sure that if any +required resources are being managed, they get applied before the +requiring resource. + + +### Agent-Side Pre-Run Resource Validation (Puppet 3.7 and Later) + +A resource can have prerequisites on the target, without which it cannot be synced. In some cases, if the absence of these prerequisites would be catastrophic, you might want to abort the whole catalog run if you detect a missing prerequisite. + +In this situation, you can define a method in your type named `pre_run_check`. This method can do any check you want. It should take no arguments, and should raise a `Puppet::Error` if the catalog run should be aborted. + +This method is **only available in Puppet 3.7 and later.** (In earlier versions of Puppet, adding a `pre_run_check` method will have no effect.) + +If a `pre_run_check` method is present in the type, OpenVox agent and Puppet apply will run the check for every resource of the type before attempting to apply the catalog. It will collect any errors raised, and present all of them before aborting the catalog run. + +As a trivial example, here's a pre-run check that will fail randomly, about one time out of six: + +```ruby + Puppet::Type.newtype(:thing) do + newparam :name, :namevar => true + + def pre_run_check + if(rand(6) == 0) + raise Puppet::Error, "Puppet roulette failed, no catalog for you!" + end + end + end +``` + + +## Providers + +Look at the [Provider Development](./provider_development.html) +page for intimate detail; this document will only +cover how the resource types and providers need to interact. Because +the properties call getter and setter methods on the providers, +except in the case of ensure, the providers must define getters and +setters for each property. + +### Provider Features + +Puppet allows you to +declare what features providers can have. The type declares the +features and what's required to make them work, and then the +providers can either be tested for whether they suffice or they can +declare that they have the features. Additionally, individual +properties and parameters in the type can declare that they require +one or more specific features, and Puppet will throw an error if +those parameters are used with providers missing those features: + +```ruby + newtype(:coloring) do + feature :paint, "The ability to paint.", :methods => [:paint] + feature :draw, "The ability to draw." + + newparam(:color, :required_features => %w{paint}) do + ... + end + end +``` + +The first argument to the feature method is the name of the +feature, the second argument is its description, and after that is +a hash of options that help Puppet determine whether the feature is +available. The only option currently supported is specifying one or +more methods that must be defined on the provider. If no methods +are specified, then the provider needs to specifically declare that +it has that feature: + +```ruby + Puppet::Type.type(:coloring).provide(:drawer) do + has_feature :draw + end +``` + +The provider can specify multiple available features at once with +has\_features. + +When you define features on your type, Puppet automatically defines +a bunch of class methods on the provider: + +* feature?: Passed a feature name, will return true if the + feature is available or false otherwise. +* features: Returns a list of all supported features on the + provider. +* satisfies?: Passed a list of feature, will return true if they + are all available, false otherwise. + +Additionally, each feature gets a separate boolean method, so the +above example would result in a paint? method on the provider. diff --git a/docs/_openvox_9x/designing_advanced_profiles.md b/docs/_openvox_9x/designing_advanced_profiles.md new file mode 100644 index 000000000..23371d95d --- /dev/null +++ b/docs/_openvox_9x/designing_advanced_profiles.md @@ -0,0 +1,1032 @@ +--- +layout: default +title: "Designing advanced profiles" +--- + +In this advanced example, we iteratively refactor our basic roles and profiles example to handle real-world concerns. The final result is — with only minor +differences — the Jenkins profile we use in production here at Puppet. + +Along the way, we'll explain our choices and point out some of the common trade-offs you'll encounter as you design your own profiles. + +Here's the basic Jenkins profile we're starting with: + +```puppet +# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp +class profile::jenkins::master ( + String $jenkins_port = '9091', + String $java_dist = 'jdk', + String $java_version = 'latest', +) { + + class { 'jenkins': + configure_firewall => true, + install_java => false, + port => $jenkins_port, + config_hash => { + 'HTTP_PORT' => { 'value' => $jenkins_port }, + 'JENKINS_PORT' => { 'value' => $jenkins_port }, + }, + } + + class { 'java': + distribution => $java_dist, + version => $java_version, + before => Class['jenkins'], + } +} +``` + +## First refactor: Split out Java + +We want to manage Jenkins masters *and* Jenkins agent nodes. We won't cover agent profiles in detail, but the first issue we encountered is that they also need Java. + +We could copy and paste the Java class declaration; it's small, so keeping multiple copies up-to-date might not be too burdensome. But instead, we decided to +break Java out into a separate profile. This way we can manage it once, then include the Java profile in both the agent and master profiles. + +> **Note:** This is a common trade-off. Keeping a chunk of code in only one place (often called the DRY — "don't repeat yourself" — principle) makes it more +> maintainable and less vulnerable to rot. But it has a cost: your individual profile classes become less readable, and you must view more files to see what a +> profile actually does. To reduce that readability cost, try to break code out in units that make inherent sense. In this case, the Java profile's job is simple +> enough to guess by its name — your colleagues don't have to read its code to know that it manages Java 8. Comments can also help. + +First, decide how configurable Java should be on Jenkins machines. After looking at our past usage, we realized that we only use two options: either we install +Oracle's Java 8 distribution, or we default to OpenJDK 7, which the Jenkins module manages. This means we can: + +- Make our new Java profile really simple: hardcode Java 8 and take no configuration. +- Replace the two Java parameters from `profile::jenkins::master` with one Boolean parameter (whether to let Jenkins handle Java). + +> **Note:** This is rule 4 in action. We reduce our profile's configuration surface by combining multiple questions into one. + +Here's the new parameter list: + +```puppet +class profile::jenkins::master ( + String $jenkins_port = '9091', + Boolean $install_jenkins_java = true, +) { # ... +``` + +And here's how we choose which Java to use: + +```puppet + class { 'jenkins': + configure_firewall => true, + install_java => $install_jenkins_java, # <--- here + port => $jenkins_port, + config_hash => { + 'HTTP_PORT' => { 'value' => $jenkins_port }, + 'JENKINS_PORT' => { 'value' => $jenkins_port }, + }, + } + + # When not using the jenkins module's java version, install java8. + unless $install_jenkins_java { include profile::jenkins::usage::java8 } +``` + +And our new Java profile: + +```puppet +# Sets up java8 for Jenkins on Debian +# +class profile::jenkins::usage::java8 { + motd::register { 'Java usage profile (profile::jenkins::usage::java8)': } + + # OpenJDK 7 is already managed by the Jenkins module. + # ::jenkins::install_java or ::jenkins::agent::install_java should be false to use this profile + # this can be set through the class parameter $install_jenkins_java + case $::osfamily { + 'debian': { + class { 'java': + distribution => 'oracle-jdk8', + version => '8u92', + } + + package { 'tzdata-java': + ensure => latest, + } + } + default: { + notify { "profile::jenkins::usage::java8 cannot set up JDK on ${::osfamily}": } + } + } +} +``` + +### Diff of first refactor + +```diff +@@ -1,13 +1,12 @@ + # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp + class profile::jenkins::master ( +- String $jenkins_port = '9091', +- String $java_dist = 'jdk', +- String $java_version = 'latest', ++ String $jenkins_port = '9091', ++ Boolean $install_jenkins_java = true, + ) { + + class { 'jenkins': + configure_firewall => true, +- install_java => false, ++ install_java => $install_jenkins_java, + port => $jenkins_port, + config_hash => { + 'HTTP_PORT' => { 'value' => $jenkins_port }, +@@ -15,9 +14,6 @@ class profile::jenkins::master ( + }, + } + +- class { 'java': +- distribution => $java_dist, +- version => $java_version, +- before => Class['jenkins'], +- } ++ # When not using the jenkins module's java version, install java8. ++ unless $install_jenkins_java { include profile::jenkins::usage::java8 } + } +``` + +## Second refactor: Manage the heap + +At Puppet, we manage the Java heap size for the Jenkins app. Production servers didn't have enough memory for heavy use. + +The Jenkins module has a `jenkins::sysconfig` defined type for managing system properties, so we'll use it: + +```puppet + # Manage the heap size on the master, in MB. + if($::memorysize_mb =~ Number and $::memorysize_mb > 8192) + { + # anything over 8GB we should keep max 4GB for OS and others + $heap = sprintf('%.0f', $::memorysize_mb - 4096) + } else { + # This is calculated as 50% of the total memory. + $heap = sprintf('%.0f', $::memorysize_mb * 0.5) + } + # Set java params, like heap min and max sizes. See + # https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties + jenkins::sysconfig { 'JAVA_ARGS': + value => join([ + "-Xms${heap}m", + "-Xmx${heap}m", + '-Djava.awt.headless=true', + '-XX:+UseConcMarkSweepGC', + '-XX:+CMSClassUnloadingEnabled', + "-Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"", + ], ' '), + } +``` + +> **Note:** Rule 4 again — we couldn't hardcode this, because we have some smaller Jenkins masters that can't spare the extra memory. But since our production +> masters are always on more powerful machines, we can calculate the heap based on the machine's memory size, which we can access as a fact. This lets us avoid +> extra configuration. + +### Diff of second refactor + +```diff +@@ -16,4 +16,20 @@ class profile::jenkins::master ( + + # When not using the jenkins module's java version, install java8. + unless $install_jenkins_java { include profile::jenkins::usage::java8 } ++ ++ # Manage the heap size on the master, in MB. ++ if($::memorysize_mb =~ Number and $::memorysize_mb > 8192) ++ { ++ # anything over 8GB we should keep max 4GB for OS and others ++ $heap = sprintf('%.0f', $::memorysize_mb - 4096) ++ } else { ++ # This is calculated as 50% of the total memory. ++ $heap = sprintf('%.0f', $::memorysize_mb * 0.5) ++ } ++ # Set java params, like heap min and max sizes. See ++ # https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties ++ jenkins::sysconfig { 'JAVA_ARGS': ++ value => join([ ++ "-Xms${heap}m", ++ "-Xmx${heap}m", ++ '-Djava.awt.headless=true', ++ '-XX:+UseConcMarkSweepGC', ++ '-XX:+CMSClassUnloadingEnabled', ++ "-Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"", ++ ], ' '), ++ } ++ + } +``` + +## Third refactor: Pin the version + +We dislike surprise upgrades, so we pin Jenkins to a specific version. We do this with a direct package URL instead of by adding Jenkins to our internal package +repositories. Your organization might choose to do it differently. + +First, we add a parameter to control upgrades. Now we can set a new value in `.../data/groups/ci/dev.yaml` while leaving `.../data/groups/ci.yaml` alone — our +dev machines will get the new Jenkins version first, and we can ensure everything works as expected before upgrading our prod machines. + +```puppet +class profile::jenkins::master ( + Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb', + # ... +) { # ... +``` + +Then, we set the necessary parameters in the Jenkins class: + +```puppet + class { 'jenkins': + lts => true, # <-- here + repo => true, # <-- here + direct_download => $direct_download, # <-- here + version => 'latest', # <-- here + service_enable => true, + service_ensure => running, + configure_firewall => true, + install_java => $install_jenkins_java, + port => $jenkins_port, + config_hash => { + 'HTTP_PORT' => { 'value' => $jenkins_port }, + 'JENKINS_PORT' => { 'value' => $jenkins_port }, + }, + } +``` + +This was a good time to explicitly manage the Jenkins *service,* so we did that as well. + +### Diff of third refactor + +```diff +@@ -1,10 +1,17 @@ + # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp + class profile::jenkins::master ( +- String $jenkins_port = '9091', +- Boolean $install_jenkins_java = true, ++ String $jenkins_port = '9091', ++ Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb', ++ Boolean $install_jenkins_java = true, + ) { + + class { 'jenkins': ++ lts => true, ++ repo => true, ++ direct_download => $direct_download, ++ version => 'latest', ++ service_enable => true, ++ service_ensure => running, + configure_firewall => true, + install_java => $install_jenkins_java, + port => $jenkins_port, +``` + +## Fourth refactor: Manually manage the user account + +We manage a lot of user accounts in our infrastructure, so we handle them in a unified way. The `profile::server` class pulls in `virtual::users`, which has a +lot of virtual resources we can selectively realize depending on who needs to log into a given machine. + +> **Note:** This has a cost — it's action at a distance, and you need to read more files to see which users are enabled for a given profile. But we decided the +> benefit was worth it: since all user accounts are written in one or two files, it's easy to see all the users that might exist, and ensure that they're managed +> consistently. + +We're accepting difficulty in one place (where we can comfortably handle it) to banish difficulty in another place (where we worry it would get out of hand). +Making this choice required that we know our colleagues and their comfort zones, and that we know the limitations of our existing code base and supporting +services. + +So, for this example, we'll change the Jenkins profile to work the same way; we'll manage the `jenkins` user alongside the rest of our user accounts. While we're +doing that, we'll also manage a few directories that can be problematic depending on how Jenkins is packaged. + +Some values we need are used by Jenkins agents as well as masters, so we're going to store them in a params class, which is a class that sets shared variables and +manages no resources. This is a heavyweight solution, so you should wait until it provides real value before using it. In our case, we had a lot of OS-specific +agent profiles (not shown in these examples), and they made a params class worthwhile. + +> **Note:** Just as before, "don't repeat yourself" is in tension with "keep it readable." Find the balance that works for you. + +```puppet + # We rely on virtual resources that are ultimately declared by profile::server. + include profile::server + + # Some default values that vary by OS: + include profile::jenkins::params + $jenkins_owner = $profile::jenkins::params::jenkins_owner + $jenkins_group = $profile::jenkins::params::jenkins_group + $master_config_dir = $profile::jenkins::params::master_config_dir + + file { '/var/run/jenkins': ensure => 'directory' } + + # Because our account::user class manages the '${master_config_dir}' directory + # as the 'jenkins' user's homedir (as it should), we need to manage + # `${master_config_dir}/plugins` here to prevent the upstream + # rtyler-jenkins module from trying to manage the homedir as the config + # dir. For more info, see the upstream module's `manifests/plugin.pp` + # manifest. + file { "${master_config_dir}/plugins": + ensure => directory, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0755', + require => [Group[$jenkins_group], User[$jenkins_owner]], + } + + Account::User <| tag == 'jenkins' |> + + class { 'jenkins': + lts => true, + repo => true, + direct_download => $direct_download, + version => 'latest', + service_enable => true, + service_ensure => running, + configure_firewall => true, + install_java => $install_jenkins_java, + manage_user => false, # <-- here + manage_group => false, # <-- here + manage_datadirs => false, # <-- here + port => $jenkins_port, + config_hash => { + 'HTTP_PORT' => { 'value' => $jenkins_port }, + 'JENKINS_PORT' => { 'value' => $jenkins_port }, + }, + } +``` + +Three things to notice in the code above: + +- We manage users with a homegrown `account::user` defined type, which declares a `user` resource plus a few other things. +- We use an `Account::User` resource collector to realize the Jenkins user. This relies on `profile::server` being declared. +- We set the Jenkins class's `manage_user`, `manage_group`, and `manage_datadirs` parameters to false. +- We're now explicitly managing the `plugins` directory and the `run` directory. + +### Diff of fourth refactor + +```diff +@@ -5,6 +5,33 @@ class profile::jenkins::master ( + Boolean $install_jenkins_java = true, + ) { + ++ # We rely on virtual resources that are ultimately declared by profile::server. ++ include profile::server ++ ++ # Some default values that vary by OS: ++ include profile::jenkins::params ++ $jenkins_owner = $profile::jenkins::params::jenkins_owner ++ $jenkins_group = $profile::jenkins::params::jenkins_group ++ $master_config_dir = $profile::jenkins::params::master_config_dir ++ ++ file { '/var/run/jenkins': ensure => 'directory' } ++ ++ # Because our account::user class manages the '${master_config_dir}' directory ++ # as the 'jenkins' user's homedir (as it should), we need to manage ++ # `${master_config_dir}/plugins` here to prevent the upstream ++ # rtyler-jenkins module from trying to manage the homedir as the config ++ # dir. For more info, see the upstream module's `manifests/plugin.pp` ++ # manifest. ++ file { "${master_config_dir}/plugins": ++ ensure => directory, ++ owner => $jenkins_owner, ++ group => $jenkins_group, ++ mode => '0755', ++ require => [Group[$jenkins_group], User[$jenkins_owner]], ++ } ++ ++ Account::User <| tag == 'jenkins' |> ++ + class { 'jenkins': + lts => true, + repo => true, +@@ -14,6 +41,9 @@ class profile::jenkins::master ( + service_ensure => running, + configure_firewall => true, + install_java => $install_jenkins_java, ++ manage_user => false, ++ manage_group => false, ++ manage_datadirs => false, + port => $jenkins_port, + config_hash => { + 'HTTP_PORT' => { 'value' => $jenkins_port }, +``` + +## Fifth refactor: Manage more dependencies + +Jenkins always needs Git installed (since we use Git for source control at Puppet), and it needs SSH keys to access private Git repos and run commands on +Jenkins agent nodes. We also have a standard list of Jenkins plugins we use, so we manage those too. + +Managing Git is pretty easy: + +```puppet + package { 'git': + ensure => present, + } +``` + +SSH keys are less easy, because they are sensitive content. We can't check them into version control with the rest of our Puppet code, so we put them in a custom mount point on one specific Puppet server. + +Since this server is different from our normal Puppet servers, we made a rule about accessing it: you must look up the hostname from data instead of hardcoding +it. This lets us change it in only one place if the secure server ever moves. + +```puppet + $secure_server = lookup('puppetlabs::ssl::secure_server') + + file { "${master_config_dir}/.ssh": + ensure => directory, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0700', + } + + file { "${master_config_dir}/.ssh/id_rsa": + ensure => file, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0600', + source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins", + } + + file { "${master_config_dir}/.ssh/id_rsa.pub": + ensure => file, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0640', + source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub", + } +``` + +Plugins are also a bit tricky, because we have a few Jenkins masters where we want to manually configure plugins. So we'll put the base list in a separate +profile, and use a parameter to control whether we use it. + +```puppet +class profile::jenkins::master ( + Boolean $manage_plugins = false, + # ... +) { + # ... + if $manage_plugins { + include profile::jenkins::master::plugins + } +``` + +In the plugins profile, we can use the `jenkins::plugin` resource type provided by the Jenkins module. + +```puppet +# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master/plugins.pp +class profile::jenkins::master::plugins { + jenkins::plugin { 'audit2db': } + jenkins::plugin { 'credentials': } + jenkins::plugin { 'jquery': } + jenkins::plugin { 'job-import-plugin': } + jenkins::plugin { 'ldap': } + jenkins::plugin { 'mailer': } + jenkins::plugin { 'metadata': } + # ... and so on. +} +``` + +### Diff of fifth refactor + +```diff +@@ -1,6 +1,7 @@ + # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp + class profile::jenkins::master ( + String $jenkins_port = '9091', ++ Boolean $manage_plugins = false, + Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb', + Boolean $install_jenkins_java = true, + ) { +@@ -14,6 +15,20 @@ class profile::jenkins::master ( + $jenkins_group = $profile::jenkins::params::jenkins_group + $master_config_dir = $profile::jenkins::params::master_config_dir + ++ if $manage_plugins { ++ # About 40 jenkins::plugin resources: ++ include profile::jenkins::master::plugins ++ } ++ ++ # Sensitive info (like SSH keys) isn't checked into version control like the ++ # rest of our modules; instead, it's served from a custom mount point on a ++ # designated server. ++ $secure_server = lookup('puppetlabs::ssl::secure_server') ++ ++ package { 'git': ++ ensure => present, ++ } ++ + file { '/var/run/jenkins': ensure => 'directory' } + + # Because our account::user class manages the '${master_config_dir}' directory +@@ -69,4 +84,29 @@ class profile::jenkins::master ( + value => join([ + "-Xms${heap}m", + "-Xmx${heap}m", + '-Djava.awt.headless=true', + '-XX:+UseConcMarkSweepGC', + '-XX:+CMSClassUnloadingEnabled', + "-Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"", + ], ' '), + } + ++ # Deploy the SSH keys that Jenkins needs to manage its agent machines and ++ # access Git repos. ++ file { "${master_config_dir}/.ssh": ++ ensure => directory, ++ owner => $jenkins_owner, ++ group => $jenkins_group, ++ mode => '0700', ++ } ++ ++ file { "${master_config_dir}/.ssh/id_rsa": ++ ensure => file, ++ owner => $jenkins_owner, ++ group => $jenkins_group, ++ mode => '0600', ++ source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins", ++ } ++ ++ file { "${master_config_dir}/.ssh/id_rsa.pub": ++ ensure => file, ++ owner => $jenkins_owner, ++ group => $jenkins_group, ++ mode => '0640', ++ source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub", ++ } ++ + } +``` + +## Sixth refactor: Manage logging and backups + +Backing up is usually a good idea. + +We can use our homegrown `backup` module, which provides a `backup::job` resource type (`profile::server` takes care of its prerequisites). But we should make +backups optional, so people don't accidentally post junk to our backup server if they're setting up an ephemeral Jenkins instance to test something. + +```puppet +class profile::jenkins::master ( + Boolean $backups_enabled = false, + # ... +) { + # ... + if $backups_enabled { + backup::job { "jenkins-data-${::hostname}": + files => $master_config_dir, + } + } +} +``` + +Also, our teams gave us some conflicting requests for Jenkins logs: + +- Some people want it to use syslog, like most other services. +- Others want a distinct log file so syslog doesn't get spammed, and they want the file to rotate more quickly than it does by default. + +That implies a new parameter. We'll make one called `$jenkins_logs_to_syslog` and default it to `undef`. If you set it to a standard syslog facility (like +`daemon.info`), Jenkins will log there instead of its own file. + +We'll use `jenkins::sysconfig` and our homegrown `logrotate::job` to do the work: + +```puppet +class profile::jenkins::master ( + Optional[String[1]] $jenkins_logs_to_syslog = undef, + # ... +) { + # ... + if $jenkins_logs_to_syslog { + jenkins::sysconfig { 'JENKINS_LOG': + value => $jenkins_logs_to_syslog, + } + } + # ... + logrotate::job { 'jenkins': + log => '/var/log/jenkins/jenkins.log', + options => [ + 'daily', + 'copytruncate', + 'missingok', + 'rotate 7', + 'compress', + 'delaycompress', + 'notifempty' + ], + } +} +``` + +### Diff of sixth refactor + +```diff +@@ -1,8 +1,10 @@ + # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp + class profile::jenkins::master ( + String $jenkins_port = '9091', ++ Boolean $backups_enabled = false, + Boolean $manage_plugins = false, + Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb', ++ Optional[String[1]] $jenkins_logs_to_syslog = undef, + Boolean $install_jenkins_java = true, + ) { + +@@ -84,6 +86,15 @@ class profile::jenkins::master ( + value => join([ + "-Xms${heap}m", + "-Xmx${heap}m", + '-Djava.awt.headless=true', + '-XX:+UseConcMarkSweepGC', + '-XX:+CMSClassUnloadingEnabled', + "-Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"", + ], ' '), + } + ++ # Forward jenkins master logs to syslog. ++ # When set to facility.level the jenkins_log will use that value instead of a ++ # separate log file eg. daemon.info ++ if $jenkins_logs_to_syslog { ++ jenkins::sysconfig { 'JENKINS_LOG': ++ value => $jenkins_logs_to_syslog, ++ } ++ } ++ + # Deploy the SSH keys that Jenkins needs to manage its agent machines and + # access Git repos. + file { "${master_config_dir}/.ssh": +@@ -109,4 +120,29 @@ class profile::jenkins::master ( + source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub", + } + ++ # Back up Jenkins' data. ++ if $backups_enabled { ++ backup::job { "jenkins-data-${::hostname}": ++ files => $master_config_dir, ++ } ++ } ++ ++ # (QENG-1829) Logrotate rules: ++ # Jenkins' default logrotate config retains too much data: by default, it ++ # rotates jenkins.log weekly and retains the last 52 weeks of logs. ++ # Considering we almost never look at the logs, let's rotate them daily ++ # and discard after 7 days to reduce disk usage. ++ logrotate::job { 'jenkins': ++ log => '/var/log/jenkins/jenkins.log', ++ options => [ ++ 'daily', ++ 'copytruncate', ++ 'missingok', ++ 'rotate 7', ++ 'compress', ++ 'delaycompress', ++ 'notifempty' ++ ], ++ } ++ + } +``` + +## Seventh refactor: Use a reverse proxy for HTTPS + +We want the Jenkins web interface to use HTTPS, which we'll accomplish with an Nginx reverse proxy. We'll also standardize the ports: the Jenkins app will always +bind to its default port, and the proxy will always serve over 443 for HTTPS and 80 for HTTP. + +If we want to keep vanilla HTTP available, we'll provide an `$ssl` parameter. If set to `false` (the default), you can access Jenkins via both HTTP and HTTPS. +We'll also add a `$site_alias` parameter, so the proxy can listen on a hostname other than the node's main FQDN. + +```puppet +class profile::jenkins::master ( + Boolean $ssl = false, + Optional[String[1]] $site_alias = undef, + # IMPORTANT: notice that $jenkins_port is removed. + # ... +``` + +We'll set `configure_firewall => false` in the Jenkins class: + +```puppet + class { 'jenkins': + lts => true, + repo => true, + direct_download => $direct_download, + version => 'latest', + service_enable => true, + service_ensure => running, + configure_firewall => false, # <-- here + install_java => $install_jenkins_java, + manage_user => false, + manage_group => false, + manage_datadirs => false, + # IMPORTANT: notice that port and config_hash are removed. + } +``` + +We need to deploy SSL certificates where Nginx can reach them. Since we serve a lot of things over HTTPS, we already had a profile for that: + +```puppet + # Deploy the SSL certificate/chain/key for sites on this domain. + include profile::ssl::delivery_wildcard +``` + +This is also a good time to add some info for the message of the day, handled by puppetlabs/motd: + +```puppet + motd::register { 'Jenkins CI master (profile::jenkins::master)': } + + if $site_alias { + motd::register { 'jenkins-site-alias': + content => @("END"), + profile::jenkins::master::proxy + + Jenkins site alias: ${site_alias} + |-END + order => 25, + } + } +``` + +The bulk of the work will be handled by a new profile called `profile::jenkins::master::proxy`. We're omitting the code for brevity; in summary, what it does is: + +- Include `profile::nginx`. +- Use resource types from the jfryman/nginx to set up a vhost, and to force a redirect to HTTPS if we haven't enabled vanilla HTTP. +- Set up logstash forwarding for access and error logs. +- Include `profile::fw::https` to manage firewall rules, if necessary. + +Then, we declare that profile in our main profile: + +```puppet + class { 'profile::jenkins::master::proxy': + site_alias => $site_alias, + require_ssl => $ssl, + } +``` + +> **Important:** We are now breaking rule 1, the most important rule of the roles and profiles method. Why? +> +> Because `profile::jenkins::master::proxy` is a "private" profile that belongs solely to `profile::jenkins::master`. It will never be declared by any role or any other profile. +> +> This is the only exception to rule 1: if you're separating out code *for the sole purpose of readability* — that is, if you could paste the private profile's +> contents into the main profile for the exact same effect — you can use a resource-like declaration on the private profile. This lets you consolidate your data +> lookups and make the private profile's inputs more visible, while keeping the main profile a little cleaner. If you do this, you must make sure to document +> that the private profile is private. +> +> If there is any chance that this code will be reused by another profile, obey rule 1. + +### Diff of seventh refactor + +```diff +@@ -1,8 +1,9 @@ + # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp + class profile::jenkins::master ( +- String $jenkins_port = '9091', + Boolean $backups_enabled = false, + Boolean $manage_plugins = false, ++ Boolean $ssl = false, ++ Optional[String[1]] $site_alias = undef, + Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb', + Optional[String[1]] $jenkins_logs_to_syslog = undef, + Boolean $install_jenkins_java = true, +@@ -11,6 +12,9 @@ class profile::jenkins::master ( + # We rely on virtual resources that are ultimately declared by profile::server. + include profile::server + ++ # Deploy the SSL certificate/chain/key for sites on this domain. ++ include profile::ssl::delivery_wildcard ++ + # Some default values that vary by OS: + include profile::jenkins::params + $jenkins_owner = $profile::jenkins::params::jenkins_owner +@@ -22,6 +26,31 @@ class profile::jenkins::master ( + include profile::jenkins::master::plugins + } + ++ motd::register { 'Jenkins CI master (profile::jenkins::master)': } ++ ++ # This adds the site_alias to the message of the day for convenience when ++ # logging into a server via FQDN. Because of the way motd::register works, we ++ # need a sort of funny formatting to put it at the end (order => 25) and to ++ # list a class so there isn't a random "--" at the end of the message. ++ if $site_alias { ++ motd::register { 'jenkins-site-alias': ++ content => @("END"), ++ profile::jenkins::master::proxy ++ ++ Jenkins site alias: ${site_alias} ++ |-END ++ order => 25, ++ } ++ } ++ ++ # This is a "private" profile that sets up an Nginx proxy -- it's only ever ++ # declared in this class, and it would work identically pasted inline. ++ # But since it's long, this class reads more cleanly with it separated out. ++ class { 'profile::jenkins::master::proxy': ++ site_alias => $site_alias, ++ require_ssl => $ssl, ++ } ++ + # Sensitive info (like SSH keys) isn't checked into version control like the + # rest of our modules; instead, it's served from a custom mount point on a + # designated server. +@@ -56,16 +85,11 @@ class profile::jenkins::master ( + version => 'latest', + service_enable => true, + service_ensure => running, +- configure_firewall => true, ++ configure_firewall => false, + install_java => $install_jenkins_java, + manage_user => false, + manage_group => false, + manage_datadirs => false, +- port => $jenkins_port, +- config_hash => { +- 'HTTP_PORT' => { 'value' => $jenkins_port }, +- 'JENKINS_PORT' => { 'value' => $jenkins_port }, +- }, + } + + # When not using the jenkins module's java version, install java8. +``` + +## The final profile code + +After all of this refactoring (and a few more minor adjustments), here's the final code for `profile::jenkins::master`. + +```puppet +# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp +# Class: profile::jenkins::master +# +# Install a Jenkins master that meets Puppet's internal needs. +# +class profile::jenkins::master ( + Boolean $backups_enabled = false, + Boolean $manage_plugins = false, + Boolean $ssl = false, + Optional[String[1]] $site_alias = undef, + Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb', + Optional[String[1]] $jenkins_logs_to_syslog = undef, + Boolean $install_jenkins_java = true, +) { + + # We rely on virtual resources that are ultimately declared by profile::server. + include profile::server + + # Deploy the SSL certificate/chain/key for sites on this domain. + include profile::ssl::delivery_wildcard + + # Some default values that vary by OS: + include profile::jenkins::params + $jenkins_owner = $profile::jenkins::params::jenkins_owner + $jenkins_group = $profile::jenkins::params::jenkins_group + $master_config_dir = $profile::jenkins::params::master_config_dir + + if $manage_plugins { + # About 40 jenkins::plugin resources: + include profile::jenkins::master::plugins + } + + motd::register { 'Jenkins CI master (profile::jenkins::master)': } + + # This adds the site_alias to the message of the day for convenience when + # logging into a server via FQDN. Because of the way motd::register works, we + # need a sort of funny formatting to put it at the end (order => 25) and to + # list a class so there isn't a random "--" at the end of the message. + if $site_alias { + motd::register { 'jenkins-site-alias': + content => @("END"), + profile::jenkins::master::proxy + + Jenkins site alias: ${site_alias} + |-END + order => 25, + } + } + + # This is a "private" profile that sets up an Nginx proxy -- it's only ever + # declared in this class, and it would work identically pasted inline. + # But since it's long, this class reads more cleanly with it separated out. + class { 'profile::jenkins::master::proxy': + site_alias => $site_alias, + require_ssl => $ssl, + } + + # Sensitive info (like SSH keys) isn't checked into version control like the + # rest of our modules; instead, it's served from a custom mount point on a + # designated server. + $secure_server = lookup('puppetlabs::ssl::secure_server') + + # Dependencies: + # - Pull in apt if we're on Debian. + # - Pull in the 'git' package, used by Jenkins for Git polling. + # - Manage the 'run' directory (fix for busted Jenkins packaging). + if $::osfamily == 'Debian' { include apt } + + package { 'git': + ensure => present, + } + + file { '/var/run/jenkins': ensure => 'directory' } + + # Because our account::user class manages the '${master_config_dir}' directory + # as the 'jenkins' user's homedir (as it should), we need to manage + # `${master_config_dir}/plugins` here to prevent the upstream + # rtyler-jenkins module from trying to manage the homedir as the config + # dir. For more info, see the upstream module's `manifests/plugin.pp` + # manifest. + file { "${master_config_dir}/plugins": + ensure => directory, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0755', + require => [Group[$jenkins_group], User[$jenkins_owner]], + } + + Account::User <| tag == 'jenkins' |> + + class { 'jenkins': + lts => true, + repo => true, + direct_download => $direct_download, + version => 'latest', + service_enable => true, + service_ensure => running, + configure_firewall => false, + install_java => $install_jenkins_java, + manage_user => false, + manage_group => false, + manage_datadirs => false, + } + + # When not using the jenkins module's java version, install java8. + unless $install_jenkins_java { include profile::jenkins::usage::java8 } + + # Manage the heap size on the master, in MB. + if($::memorysize_mb =~ Number and $::memorysize_mb > 8192) + { + # anything over 8GB we should keep max 4GB for OS and others + $heap = sprintf('%.0f', $::memorysize_mb - 4096) + } else { + # This is calculated as 50% of the total memory. + $heap = sprintf('%.0f', $::memorysize_mb * 0.5) + } + # Set java params, like heap min and max sizes. See + # https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties + jenkins::sysconfig { 'JAVA_ARGS': + value => join([ + "-Xms${heap}m", + "-Xmx${heap}m", + '-Djava.awt.headless=true', + '-XX:+UseConcMarkSweepGC', + '-XX:+CMSClassUnloadingEnabled', + "-Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"", + ], ' '), + } + + # Forward jenkins master logs to syslog. + # When set to facility.level the jenkins_log will use that value instead of a + # separate log file eg. daemon.info + if $jenkins_logs_to_syslog { + jenkins::sysconfig { 'JENKINS_LOG': + value => $jenkins_logs_to_syslog, + } + } + + # Deploy the SSH keys that Jenkins needs to manage its agent machines and + # access Git repos. + file { "${master_config_dir}/.ssh": + ensure => directory, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0700', + } + + file { "${master_config_dir}/.ssh/id_rsa": + ensure => file, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0600', + source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins", + } + + file { "${master_config_dir}/.ssh/id_rsa.pub": + ensure => file, + owner => $jenkins_owner, + group => $jenkins_group, + mode => '0640', + source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub", + } + + # Back up Jenkins' data. + if $backups_enabled { + backup::job { "jenkins-data-${::hostname}": + files => $master_config_dir, + } + } + + # (QENG-1829) Logrotate rules: + # Jenkins' default logrotate config retains too much data: by default, it + # rotates jenkins.log weekly and retains the last 52 weeks of logs. + # Considering we almost never look at the logs, let's rotate them daily + # and discard after 7 days to reduce disk usage. + logrotate::job { 'jenkins': + log => '/var/log/jenkins/jenkins.log', + options => [ + 'daily', + 'copytruncate', + 'missingok', + 'rotate 7', + 'compress', + 'delaycompress', + 'notifempty' + ], + } + +} +``` diff --git a/docs/_openvox_9x/designing_convenient_roles.md b/docs/_openvox_9x/designing_convenient_roles.md new file mode 100644 index 000000000..367bccd06 --- /dev/null +++ b/docs/_openvox_9x/designing_convenient_roles.md @@ -0,0 +1,190 @@ +--- +layout: default +title: "Designing convenient roles" +--- + +There are several approaches to building roles, and you must decide which ones are most convenient for you and your team. + +High-quality roles strike a balance between readability and maintainability. For most people, the benefit of seeing the entire role in a single file outweighs +the maintenance cost of repetition. Later, if you find the repetition burdensome, you can change your approach to reduce it. This might involve combining +several similar roles into a more complex role, creating sub-roles that other roles can include, or pushing more complexity into your profiles. + +So, begin with granular roles and deviate from them only in small, carefully considered steps. + +Here's the basic Jenkins role we're starting with: + +```puppet +class role::jenkins::master { + include profile::base + include profile::server + include profile::jenkins::master +} +``` + +## First approach: Granular roles + +The simplest approach is to make one role per type of node, period. For example, the Puppet Release Engineering (RE) team manages some additional resources on their Jenkins masters. + +With granular roles, we'd have at least two Jenkins master roles. A basic one: + +```puppet +class role::jenkins::master { + include profile::base + include profile::server + include profile::jenkins::master +} +``` + +...and an RE-specific one: + +```puppet +class role::jenkins::master::release { + include profile::base + include profile::server + include profile::jenkins::master + include profile::jenkins::master::release +} +``` + +The benefits of this setup are: + +- Readability — By looking at a single class, you can immediately see which profiles make up each type of node. +- Simplicity — Each role is just a linear list of profiles. + +Some drawbacks are: + +- Role bloat — If you have a lot of only-slightly-different nodes, you'll quickly have a large number of roles. +- Repetition — The two roles above are almost identical, with one difference. If they're two separate roles, it's harder to see how they're related to each other, and updating them can be more annoying. + +## Second approach: Conditional logic + +Alternatively, you can use conditional logic to handle differences between closely-related kinds of nodes. + +```puppet +class role::jenkins::master::release { + include profile::base + include profile::server + include profile::jenkins::master + + if $facts['group'] == 'release' { + include profile::jenkins::master::release + } +} +``` + +The benefits of this approach are: + +- You have fewer roles, and they're easy to maintain. + +The drawbacks are: + +- Reduced readability...maybe. Conditional logic isn't usually hard to read, especially in a simple case like this, but you might feel tempted to add a bunch of + new custom facts to accommodate complex roles. This can make roles much harder to read, because a reader must also know what those facts mean. + + In short, be careful of turning your node classification system inside-out. You might have a better time if you separate the roles and assign them with your + node classifier. + +## Third approach: Nested roles + +Another way of reducing repetition is to let roles include other roles. + +```puppet +class role::jenkins::master { + # Parent role: + include role::server + # Unique classes: + include profile::jenkins::master +} + +class role::jenkins::master::release { + # Parent role: + include role::jenkins::master + # Unique classes: + include profile::jenkins::master::release +} +``` + +In this example, we reduce boilerplate by having `role::jenkins::master` include `role::server`. When `role::jenkins::master::release` includes +`role::jenkins::master`, it automatically gets `role::server` as well. With this approach, any given role only needs to: + +- Include the "parent" role that it most resembles. +- Include the small handful of classes that differentiate it from its parent. + +The benefits of this approach are: + +- You have fewer roles, and they're easy to maintain. +- Increased visibility in your node classifier. + +The drawbacks are: + +- Reduced readability: you have to open more files to see the real content of a role. This isn't much of a problem if you only go one level deep, but it can get wild around three or four. + +## Fourth approach: Multiple roles per node + +In general, we recommend that you assign only one role to a node. In an infrastructure where nodes usually provide one primary service, that's the best way to work. + +However, if your nodes tend to provide more than one primary service, it can make sense to assign multiple roles. + +For example, say you have a large application that is usually composed of an application server, a database server, and a web server. To enable lighter-weight +testing during development, you've decided to provide an "all-in-one" node type to your developers. You could do this by creating a new +`role::our_application::monolithic` class, which includes all of the profiles that compose the three normal roles, but you might find it simpler to use your node +classifier to assign all three roles (`role::our_application::app`, `role::our_application::db`, and `role::our_application::web`) to those all-in-one machines. + +The benefit of this approach is: + +- You have fewer roles, and they're easy to maintain. + +The drawbacks are: + +- There's no actual "role" that describes your multi-purpose nodes; instead, the source of truth for what's on them is spread out between your roles and your + node classifier, and you must cross-reference to understand their configurations. This reduces readability. +- The normal and all-in-one versions of a complex application are likely to have other subtle differences you need to account for, which might mean making your + "normal" roles more complex. It's possible that making a separate role for this kind of node would *reduce* your overall complexity, even though it increases + the number of roles and adds repetition. + +## Fifth approach: Super profiles + +Since profiles can already include other profiles, you can decide to enforce an additional rule at your business: all profiles must include any other profiles +needed to manage a complete node that provides that service. + +For example, our `profile::jenkins::master` class could include both `profile::server` and `profile::base`, and you could manage a Jenkins master server by +directly assigning `profile::jenkins::master` in your node classifier. In other words, a "main" profile would do all the work that a role usually does, and the +roles layer would no longer be necessary. + +The benefits of this approach are: + +- The chain of dependencies for a complex service can be more clear this way. +- Depending on how you conceptualize code, this can be easier in a lot of ways! + +The drawbacks are: + +- Loss of flexibility. This reduces the number of ways in which your roles can be combined, and reduces your ability to use alternate implementations of dependencies for nodes with different requirements. +- Reduced readability, on a much grander scale. Like with nested roles, you lose the advantage of a clean, straightforward list of what a node consists of. + Unlike nested roles, you also lose the clear division between "top-level" complete system configurations (roles) and "mid-level" groupings of technologies + (profiles). Not every profile makes sense as an entire system, so you'll need some way to keep track of which profiles are the top-level ones. + + Some people really find continuous hierarchies easier to reason about than sharply divided layers. If everyone in your organization is on the same page about + this, a "profiles and profiles" approach might make sense. But we strongly caution you against it unless you're very sure; for most people, a true roles and + profiles approach works better. Try the well-traveled path first. + +## Sixth approach: Building roles in the node classifier + +Instead of building roles with the Puppet language and then assigning them to nodes with your node classifier, you might find your classifier flexible enough to build roles directly. + +For example, you might create a "Jenkins masters" group in your node classifier and assign it the `profile::base`, `profile::server`, and +`profile::jenkins::master` classes, doing much the same job as our basic `role::jenkins::master` class. + +> **Important:** If you're doing this, make sure you don't set parameters for profiles in the classifier. Continue to use Hiera or Puppet lookup to configure profiles. +> +> This is because profiles are allowed to include other profiles, which interacts badly with the resource-like behavior that node classifiers use to set class parameters. + +The benefits of this approach are: + +- Your node classifier becomes much more powerful, and can be a central point of collaboration for managing nodes. +- Increased readability: a node's page in the classifier displays the full content of its role, without having to cross-reference with manifests in your `role` + module. + +The drawbacks are: + +- Loss of flexibility. The Puppet language's conditional logic is often more flexible and convenient than most node classifiers. +- Your roles are no longer in the same code repository as your profiles, and it's more difficult to make them follow the same code promotion processes. diff --git a/docs/_openvox_9x/dirs_codedir.markdown b/docs/_openvox_9x/dirs_codedir.markdown new file mode 100644 index 000000000..d567afded --- /dev/null +++ b/docs/_openvox_9x/dirs_codedir.markdown @@ -0,0 +1,45 @@ +--- +layout: default +title: "Code and data directory (codedir)" +--- + +[codedir]: ./configuration.html#codedir +[puppetserver_conf]: /openvox-server/latest/config_file_puppetserver.html + +Puppet's `codedir` is the main directory for Puppet code and data. It contains environments (which contain your manifests and modules), a global modules directory for all environments, and your Hiera data. + +## Location + +Puppet's codedir can be found at one of the following locations: + +* \*nix Systems: `/etc/puppetlabs/code` +* Windows: `%PROGRAMDATA%\PuppetLabs\code` (usually `C:\ProgramData\PuppetLabs\code`) +* non-root users: `~/.puppetlabs/etc/code` + +When Puppet is running as either root, a Windows user with administrator privileges, or the `puppet` user, it will use a system-wide codedir. When running as a non-root user, it will use a codedir in that user's home directory. + +The system codedir is what you usually want to use, since you will usually run Puppet's commands and services as root or `puppet`. (Note that admin commands like `puppet module` must be run with `sudo` to use the same codedir as OpenVox agent or OpenVox Server.) + +### Configuration + +The location of the codedir can be configured in puppet.conf with [the `codedir` setting][codedir], but note that Puppet Server doesn't use that setting; it has its own `jruby-puppet.master-code-dir` setting [in puppetserver.conf][puppetserver_conf]. If you're using a non-default codedir, _you must change both settings._ + +## Interpolation of `$codedir` + +Since the value of the codedir is discovered before other settings, you can safely reference it (with the `$codedir` variable) in the value of any other setting in puppet.conf: + + [server] + environmentpath = $codedir/override_environments:$codedir/environments + +If you need to set nonstandard values for some settings, this allows you to avoid absolute paths and keep your Puppet-related files together. + +### Contents + +Puppet's codedir contains environments, modules, and Hiera data. Its contents are used by OpenVox Server and Puppet apply, but not by OpenVox agent. + +Almost everything in the codedir has its own page of documentation. + +### Code and Data Directories + +* [`environments`](./environments_creating.html) --- contains alternate versions of the `modules` and `manifests` directories, to allow code changes to be tested on smaller sets of nodes before entering production. +* [`modules`](./dirs_modulepath.html) --- the main directory for Puppet's modules. diff --git a/docs/_openvox_9x/dirs_confdir.markdown b/docs/_openvox_9x/dirs_confdir.markdown new file mode 100644 index 000000000..29798bdd8 --- /dev/null +++ b/docs/_openvox_9x/dirs_confdir.markdown @@ -0,0 +1,62 @@ +--- +layout: default +title: "Directories: Config directory (confdir)" +--- + +[puppetserver_conf]: /openvox-server/latest/config_file_puppetserver.html + +Puppet's `confdir` is the main directory for Puppet's configuration. It contains config files and SSL data. + +## Location + +Puppet's confdir can be found at one of the following locations: + +* \*nix Systems: `/etc/puppetlabs/puppet` +* Windows: `%PROGRAMDATA%\PuppetLabs\puppet\etc` (usually `C:\ProgramData\PuppetLabs\puppet\etc`) +* non-root users: `~/.puppetlabs/etc/puppet` + +When Puppet is running as either root, a Windows user with administrator privileges, or the `puppet` user, it will use a system-wide confdir. When running as a non-root user, it will use a confdir in that user's home directory. + +The system confdir is what you usually want to use, since you will usually run Puppet's commands and +services as root or `puppet`. (Note that admin commands like `puppetserver ca` must be run with `sudo` +to use the same confdir as OpenVox agent or OpenVox Server.) + +### Configuration + +Puppet's confdir can be specified on the command line with the `--confdir` option, but it can't be set +via puppet.conf. (This is because it needs the `confdir` to even find the config file.) If `--confdir` +isn't specified when a Puppet application is started, it will always use the default confdir location. + +Puppet Server uses the `jruby-puppet.master-conf-dir` setting [in puppetserver.conf][puppetserver_conf] +to configure its confdir. Note that if you're using a non-default confdir, you must also specify +`--confdir` whenever you run commands like `puppet module` to ensure they use the same directories as +Puppet Server. + +## Interpolation of `$confdir` + +Since the value of the confdir is discovered before other settings, you can safely reference it (with the `$confdir` variable) in the value of any other setting in puppet.conf. + +If you need to set nonstandard values for some settings, this allows you to avoid absolute paths and keep your Puppet-related files together. + +## Contents + +Puppet's confdir contains several config files and the SSL data. Their locations can be changed with settings, but most users should use the default layout. + +Almost everything in the confdir has its own page of documentation. + +Items labeled "master only" below can also be present on standalone Puppet apply nodes, since they act as both masters and agents. + +### SSL Data + +* [`ssl`](./dirs_ssldir.html) --- contains each node's certificate infrastructure. (All nodes.) + +### Config files + +* [`puppet.conf`](./config_file_main.html) --- Puppet's main config file. (Any node.) +* [`auth.conf`](./config_file_auth.html) --- access control rules for the OpenVox Server's network services. (Server only.) +* [`autosign.conf`](./config_file_autosign.html) --- a list of pre-approved certificate requests. (CA master only.) +* [`csr_attributes.yaml`](./config_file_csr_attributes.html) --- optional data to be inserted into new certificate requests. (Any node.) +* [`device.conf`](./config_file_device.html) --- configuration for network devices managed by the `puppet device` command. (Any node acting as an intermediary to configure network devices.) +* [`fileserver.conf`](./config_file_fileserver.html) --- configuration for additional fileserver mount points. (Server only.) +* [`hiera.yaml`](./hiera_config_yaml_5.html) --- global configuration for the Hiera data lookup system. Note that environments and modules can have their own hiera.yaml files. (Master, or standalone nodes running Puppet apply.) +* [`routes.yaml`](./config_file_routes.html) --- advanced configuration of indirector behavior. (Server only.) diff --git a/docs/_openvox_9x/dirs_manifest.markdown b/docs/_openvox_9x/dirs_manifest.markdown new file mode 100644 index 000000000..ba247c70d --- /dev/null +++ b/docs/_openvox_9x/dirs_manifest.markdown @@ -0,0 +1,49 @@ +--- +layout: default +title: "Directories: The main manifest" +--- + +[environment]: ./environments_about.html +[catalog_compilation]: ./subsystem_catalog_compilation.html +[print_settings]: ./config_print.html +[default_manifest]: ./configuration.html#default_manifest +[disable_per_environment_manifest]: ./configuration.html#disable_per_environment_manifest +[environment.conf]: ./config_file_environment.html +[puppet.conf]: ./config_file_main.html +[configuring environments]: ./environments_creating.html +[creating environments]: ./environments_creating.html + +Puppet always starts compiling with either a single manifest file or a directory of manifests that get treated like a single file. This main starting point is called the **main manifest** or **site manifest.** + +For more information on how the site manifest is used in catalog compilation, see [the reference page on catalog compilation][catalog_compilation]. + +## Location + +### With Puppet apply + +The `puppet apply` command requires a manifest as an argument on the command line. (For example: `puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp`.) It can be a single file or a directory of files. + +The `puppet apply` command does not automatically use an environment's manifest. Instead, it always uses the manifest you pass to it. + +### With OpenVox Server + +OpenVox Server always uses the main manifest set by the current node's [environment][]. The main manifest can be a single file or a directory of `.pp` files. + +By default, the main manifest for a given environment is `//manifests`. (For example: `/etc/puppetlabs/code/environments/production/manifests`.) You can configure the manifest per-environment, and you can also configure the default for all environments. + +* An environment can use the `manifest` setting in [environment.conf][] to choose its main manifest. This can be an absolute path or a path relative to the environment's main directory. If absent, it defaults to the value of [the `default_manifest` setting][default_manifest] from [puppet.conf][]. +* [The `default_manifest` setting][default_manifest] defaults to `./manifests`. Like the `manifest` setting, the value of `default_manifest` can be an absolute path or a path relative to the environment's main directory. +* You can also force all environments to use the `default_manifest` (ignoring their own `manifest` settings) by setting [`disable_per_environment_manifest = true`][disable_per_environment_manifest] in puppet.conf. + +For more details, see: + +* [Configuring Environments][] +* [Creating Environments][] + +To check the manifest your OpenVox Server will use for a given environment, [run `puppet config print manifest --section server --environment `][print_settings]. + +## Directory behavior (vs. single file) + +If the main manifest is a directory, Puppet parses every `.pp` file in the directory in alphabetical order and evaluate the combined manifest. It descends into all subdirectories of the manifest directory and loads files in depth-first order. (For example, if the manifest directory contains a directory named `01` and a file named `02.pp`, it will parse all the files in `01` before `02`.) + +Puppet acts as though the whole directory were just one big manifest; for example, a variable assigned in the file `01_all_nodes.pp` would be accessible in `node_web01.pp`. diff --git a/docs/_openvox_9x/dirs_modulepath.markdown b/docs/_openvox_9x/dirs_modulepath.markdown new file mode 100644 index 000000000..fb4a5d5b2 --- /dev/null +++ b/docs/_openvox_9x/dirs_modulepath.markdown @@ -0,0 +1,107 @@ +--- +layout: default +title: "Directories: The modulepath (default config)" +--- + +[module_fundamentals]: ./modules_fundamentals.html +[environments]: ./environments_about.html +[basemodulepath_setting]: ./configuration.html#basemodulepath +[config_print]: ./config_print.html +[environment.conf]: ./config_file_environment.html + +The OpenVox Server service and the `puppet apply` command both load most of their content from modules. (See the page on [module structure and behavior][module_fundamentals] for more details.) + +Puppet automatically loads modules from one or more directories. The list of directories Puppet will find modules in is called the **modulepath.** + +The modulepath is set by the current node's [environment][environments]. + +## Format + +`/etc/puppetlabs/code/environments/production/modules:/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules` + +The modulepath is a list of directories separated by the system _path-separator character._ On \*nix systems, this is the colon (`:`, as seen above), and on Windows it is the semi-colon (`;`). + +It is an ordered list, with earlier directories having priority over later ones. See ["Loading Content from Modules"][inpage_loading] below. + +## Contents + +Every directory in the modulepath should only contain valid Puppet modules. + +The names of those modules must only contain letters, numbers, and underscores. Dashes and periods are **no longer valid** and cause errors when attempting to use the module. + +For details about module contents and structure, see [the documentation on modules][module_fundamentals]. + +## Location + +The modulepath is set by the current node's [environment][environments]. By default, it is usually something like: + +`:$codedir/modules:/opt/puppetlabs/puppet/modules` + +You can view the effective modulepath for any environment by specifying the environment when [requesting the setting value][config_print]: + +```bash +sudo puppet config print modulepath --section server --environment test +/etc/puppetlabs/code/environments/test/modules:/etc/puppetlabs/code/modules:/usr/share/puppet/modules +``` + +## Configuration + +Each environment can set its full modulepath in [environment.conf][] with the `modulepath` setting. The default value is that environment's `modules` directory followed by the **base modulepath.** + +When running `puppet apply` on the command line, you also have the option of directly setting the modulepath with the `--modulepath` flag. + +### The `modulepath` setting + +The `modulepath` setting can only be set in [environment.conf][]. It configures the entire modulepath for that environment. + +The default value of `modulepath` is `./modules:$basemodulepath`. + +Note that the modulepath can include relative paths, such as `./modules` or `./site`. Puppet looks for these paths inside the environment's directory. + +If you want an environment to have access to the global module directories, it should include `$basemodulepath`. + +### The base modulepath + +The **base modulepath** is a list of _global_ module directories for use with all [environments][]. It can be configured with [the `basemodulepath` setting][basemodulepath_setting], but its default value is probably suitable for you unless you're doing something unusual. + +The default value of the `basemodulepath` setting is `$codedir/modules:/opt/puppetlabs/puppet/modules`. (On Windows, it will just use `$codedir\modules`.) + +### Using `--modulepath` + +When running `puppet apply`, you can supply a full modulepath as a command line option. This overrides the modulepath from the current environment. + +## Loading content from modules + +[inpage_loading]: #loading-content-from-modules + +Puppet uses modules from every directory in the modulepath. + +### Empty and absent directories + +Directories in the modulepath can be empty, and might even be absent. In both cases, this is not an error; it just means Puppet does not load modules from those directories. + +If no modules are present across the entire modulepath, or if modules are present but none of them contains a `lib` directory, then OpenVox agent will log an error when attempting to sync plugins from the OpenVox Server. This error is benign and will not prevent the rest of the Puppet run. + +### Duplicate or conflicting modules and content + +If the modulepath contains multiple modules with the same name, Puppet uses the version from the directory that comes _earliest_ in the modulepath. This allows directories earlier in the modulepath to override later directories. + +For most content, this earliest-module-wins behavior is on an all-or-nothing, **per-module** basis --- **all** of the manifests, files, and templates in the winning version will be available for use, and *_none_ of that content from any subsequent versions will be available. This behavior covers: + +- Puppet code (from `manifests`) +- Files (from `files`) +- Templates (from `templates`) +- External facts (from `facts.d`) +- Ruby plugins synced to agent nodes (from `lib`) + +> **However,** Puppet occasionally shows problematic behavior with **Ruby plugins loaded directly from modules.** This includes: +> +> - Plugins used by the OpenVox Server (custom resource types, custom functions) +> - Plugins used by `puppet apply` +> - Plugins that happen to be present in OpenVox agent's modulepath (which should generally be empty, but might not be when running OpenVox agent on a node that is also an OpenVox Server server) +> +> With these plugins, the earlier module still wins, but the plugins are handled on a **per-file** basis instead of per-module. This means that if a duplicate module in a later directory has **additional** plugin files that don't exist in the winning module, those extra files will be loaded, and Puppet will use a mixture of files from the winning and duplicate modules. +> +> The upshot is, if you refactor a module's Ruby plugins and then maintain two versions of that module in your modulepath, it can sometimes result in weirdness. +> +> This is essentially the same Ruby loading problem that environments have. It's not intentional, but it's not likely to get fixed soon, since it's a byproduct of the way Ruby works and Puppet only has a limited amount of control over it. diff --git a/docs/_openvox_9x/dirs_ssldir.markdown b/docs/_openvox_9x/dirs_ssldir.markdown new file mode 100644 index 000000000..ae1e0a811 --- /dev/null +++ b/docs/_openvox_9x/dirs_ssldir.markdown @@ -0,0 +1,93 @@ +--- +layout: default +title: "Directories: SSLdir" +--- + + +[ssldir]: ./configuration.html#ssldir +[cadir]: ./configuration.html#cadir +[cacrl]: ./configuration.html#cacrl +[cacert]: ./configuration.html#cacert +[cakey]: ./configuration.html#cakey +[capub]: ./configuration.html#capub +[cert_inventory]: ./configuration.html#cert_inventory +[csrdir]: ./configuration.html#csrdir +[serial]: ./configuration.html#serial +[signeddir]: ./configuration.html#signeddir +[requestdir]: ./configuration.html#requestdir +[hostcsr]: ./configuration.html#hostcsr +[certdir]: ./configuration.html#certdir +[hostcert]: ./configuration.html#hostcert +[localcacert]: ./configuration.html#localcacert +[hostcrl]: ./configuration.html#hostcrl +[privatedir]: ./configuration.html#privatedir +[passfile]: ./configuration.html#passfile +[privatekeydir]: ./configuration.html#privatekeydir +[hostprivkey]: ./configuration.html#hostprivkey +[publickeydir]: ./configuration.html#publickeydir +[hostpubkey]: ./configuration.html#hostpubkey +[vardir]: ./dirs_vardir.html +[confdir]: ./dirs_confdir.html +[certname]: ./configuration.html#certname + + +Puppet stores its certificate infrastructure in the `ssldir` directory. This directory has a similar structure on all Puppet nodes, whether they are agent nodes, OpenVox Server servers, or the certificate authority (CA) master. + +## Location of the `ssldir` directory + +By default, the `ssldir` directory is located at `$confdir/ssl`. For more information about the `confdir` folder, see [confdir][confdir]. + +Its location can be configured with the [`ssldir` setting][ssldir]. To see what the location is on one of your nodes, run `puppet config print ssldir`. + +> **Note:** Some third-party Puppet packages for Linux put the ssldir in the [vardir][] instead of the [confdir][]. The right place for it in the filesystem hierarchy is debatable; the contents are automatically generated and will tend to grow, but are also important, relatively difficult to replace, and can be considered configuration. +> +> If a distro changes the `ssldir` directory location, it will do so by setting `ssldir` in the `$confdir/puppet.conf` file, usually in the `[main]` section. You can find out its location by running `puppet config print ssldir`. + +## What the `ssldir` directory contains + +The `ssldir` directory contains Puppet certificates, private keys, certificate signing requests (CSRs), and other cryptographic documents. + +The `ssldir` directory on Agent nodes and OpenVox Servers contain a private key (`private_keys/.pem`), a public key (`public_keys/`), a signed certificate (`certs/.pem`), a copy of the CA certificate (`certs/ca.pem`), and a copy of the certificate revocation list (CRL) (`crl.pem`). +They usually also retain a copy of their CSR after submitting it (`certificate_requests/.pem`). If these files don't exist, they are either generated locally or requested from the CA OpenVox Server. + +Since agent and master credentials are identified by [certname][], an OpenVox agent process and OpenVox Server process running on the same server can use the same credentials. + +The `ssldir` directory for the Puppet CA, which runs on the CA OpenVox Server server, contains similar credentials: private and public keys, certificate, master copy of the CRL. +It also maintains a list of all signed certificates in the deployment, a copy of each signed certificate, and an incrementing serial number for new certificates. All of the CA's data is stored in the `ca` subdirectory, to keep it separated from any general Puppet credentials on the same server. + +## The `ssldir` directory structure + +All of the files and directories in the `ssldir` directory have corresponding Puppet settings, which can be used to individually change their locations. However, this is generally not recommended. + +The permissions mode of the `ssldir` directory should be 0771, and it and every file it contains should be owned by the user that Puppet runs as: root or Administrator on OpenVox agent nodes, and defaulting to `puppet` or `pe-puppet` on an OpenVox Server server. Ownership and permissions in the `ssldir` directory should be managed automatically. + +The `ssldir` has the following structure: + +* `ca` _(directory)_ --- Contains all files used by Puppet's built-in certificate authority (CA). This directory must exist only on the CA OpenVox Server server. Mode: 0755. Setting: [`cadir`][cadir]. + * `ca_crl.pem` --- The master copy of the certificate revocation list (CRL) managed by the CA. Mode: 0644. Setting: [`cacrl`][cacrl]. + * `ca_crt.pem` --- The CA's self-signed certificate. This cannot be used as an OpenVox Server or OpenVox agent certificate; it can only be used to sign certificates. Mode: 0644. Setting: [`cacert`][cacert]. + * `ca_key.pem` --- The CA's private key. Tied for most security-critical file in the entire Puppet certificate infrastructure. Mode: 0640. Setting: [`cakey`][cakey]. + * `ca_pub.pem` --- The CA's public key. Mode: 0644. Setting: [`capub`][capub]. + * `inventory.txt` --- A list of all certificates the CA has signed, along with their serial numbers and validity periods. Mode: 0644. Setting: [`cert_inventory`][cert_inventory]. + * `private` _(directory)_ --- Contains only one file. Mode: 0750. Setting: `caprivatedir`. + * `ca.pass` --- The (randomly generated) password to the CA's private key. Tied for most security-critical file in the entire Puppet certificate infrastructure. Mode: 0640. Setting: `capass`. + * `requests` _(directory)_ --- Contains certificate signing requests (CSRs) that were received but have not yet been signed. The CA deletes CSRs from this directory after signing them. Mode: 0755. Setting: [`csrdir`][csrdir]. + * `.pem` --- Individual CSR files. + * `serial` --- A file containing the serial number for the next certificate the CA will sign. This is incremented with each new certificate signed. Mode: 0644. Setting: [`serial`][serial]. + * `signed` _(directory)_ --- Contains copies of all certificates the CA has signed. Mode: 0755. Setting: [`signeddir`][signeddir]. + * `.pem` --- Individual signed certificate files. +* `certificate_requests` _(directory)_ --- Contains any CSRs generated by this node in preparation for submission to the CA. CSRs persist in this directory even after they have been submitted and signed. Mode: 0755. Setting: [`requestdir`][requestdir]. + * `.pem` --- This node's CSR. Mode: 0644. Setting: [`hostcsr`][hostcsr]. +* `certs` _(directory)_ --- Contains any signed certificates present on this node. This includes the node's own certificate, as well as a copy of the CA certificate (for use when validating certificates presented by other nodes). Mode: 0755. Setting: [`certdir`][certdir]. + * `.pem` --- This node's certificate. Mode: 0644. Setting: [`hostcert`][hostcert]. + * `ca.pem` --- A local copy of the CA certificate. Mode: 0644. Setting: [`localcacert`][localcacert]. +* `crl.pem` --- A copy of the certificate revocation list (CRL) retrieved from the CA, for use by OpenVox agent or OpenVox Server. Mode: 0644. Setting: [`hostcrl`][hostcrl]. +* `private` _(directory)_ --- Usually does not contain any files. Mode: 0750. Setting: [`privatedir`][privatedir]. + * `password` --- The password to a node's private key. Usually not present. The conditions in which this file would exist are not defined. Mode: 0640. Setting: [`passfile`][passfile]. +* `private_keys` _(directory)_ --- Contains any private keys present on this node. This should generally + only include the node's own private key, although on the CA it might also contain any private keys + created by the `puppetserver ca generate` command. It will never contain the private key for the CA + certificate. Mode: 0750. Setting: [`privatekeydir`][privatekeydir]. + * `.pem` --- This node's private key. Mode: 0600. Setting: [`hostprivkey`][hostprivkey]. +* `public_keys` _(directory)_ --- Contains any public keys generated by this node in preparation for generating a CSR. Mode: 0755. Setting: [`publickeydir`][publickeydir]. + * `.pem` --- This node's public key. Mode: 0644. Setting: [`hostpubkey`][hostpubkey]. diff --git a/docs/_openvox_9x/dirs_vardir.markdown b/docs/_openvox_9x/dirs_vardir.markdown new file mode 100644 index 000000000..5269812a8 --- /dev/null +++ b/docs/_openvox_9x/dirs_vardir.markdown @@ -0,0 +1,95 @@ +--- +layout: default +title: "Directories: The cache directory (vardir)" +--- + +[config_ref]: ./configuration.html +[puppetserver_conf]: /openvox-server/latest/config_file_puppetserver.html + +Puppet's cache directory, sometimes called `vardir`, contains dynamic or growing data that Puppet creates in the course of its normal operations. Some of this data can be mined for interesting analysis, or to integrate other tools with Puppet. Other parts are just infrastructure and can be ignored. + +## Location of the `vardir` directory + +Puppet Server's cache directory defaults to `/opt/puppetlabs/server/data/puppetserver`. + +The cache directory for OpenVox agent and Puppet apply can be found at one of the following locations: + +* \*nix Systems: `/var/opt/puppetlabs/puppet/cache` +* non-root users: `~/.puppetlabs/opt/puppet/cache` +* Windows: `%PROGRAMDATA%\PuppetLabs\puppet\cache` (usually `C:\Program Data\PuppetLabs\puppet\cache`) + +When Puppet is running as either root, a Windows user with administrator privileges, or the `puppet` user, it will use a system-wide cache directory. When running as a non-root user, it will use a cache directory in that user's home directory. + +The system cache directory is what you usually want to use, since you will usually run Puppet's commands +and services as root or `puppet`. (Note that admin commands like `puppetserver ca` must be run with +`sudo` to use the same directories as OpenVox agent or OpenVox Server.) + +### Configuring the location of the cache directory + +You can specify Puppet's cache directory on the command line by using the `--vardir` option, but you can't set it in `puppet.conf`. If `--vardir` isn't specified when a Puppet application is started, it will always use the default cache directory location. + +Puppet Server uses the `jruby-puppet.master-var-dir` setting [in `puppetserver.conf`][puppetserver_conf] to configure its cache directory. + +## Interpolation of `$vardir` + +Because the value of the `vardir` is discovered before other settings, you can reference it with the `$vardir` variable in the value of any other setting in `puppet.conf` or on the command line. + +For example: + + [main] + ssldir = $vardir/ssl + +If you need to set nonstandard values for some settings, this allows you to avoid absolute paths and keep your Puppet-related files together. + +## Contents of the cache directory + +The `vardir` directory contains several subdirectories. Most of these subdirectories contain a variable amount of automatically generated data. Some of them contain notable individual files. Some directories are used only by agent or master processes. + +The `vardir` directory has the following default structure. Most of the files and directories can have their locations changed with settings in `puppet.conf`. The link for each item goes to its description in the [configuration reference][config_ref]. + +* [`bucket` (`bucketdir`)][bucketdir] +* [`client_data` (`client_datadir`)][client_datadir] +* [`clientbucket` (`clientbucketdir`)][clientbucketdir] +* [`client_yaml` (`clientyamldir`)][clientyamldir] +* [`devices` (`devicedir`)][devicedir] +* [`lib/facter` (`factpath`)][factpath] +* [`facts` (`factpath`)][factpath] +* [`facts.d` (`pluginfactdest`)][pluginfactdest] +* [`lib` (`libdir`)][libdir] (also [plugindest][]) --- Puppet uses this as a cache for plugins (custom facts, types and providers, functions) synced from an OpenVox Server. Do not directly change it. If you delete it, the plugins will be restored on the next Puppet run. +* [`puppet-module` (`module_working_dir`)][module_working_dir] + * `skeleton` (`module_skeleton_dir`) +* [`reports` (`reportdir`)][reportdir] --- When the `store` report is enabled, an OpenVox Server will store all reports received from agents as YAML files in this directory. These can be easily mined for analysis by an out-of-band process. +* [`server_data` (`serverdatadir`)][serverdatadir] +* [`state` (`statedir`)][statedir] + * [`agent_catalog_run.lock` (`agent_catalog_run_lockfile`)][agent_catalog_run_lockfile] + * [`agent_disabled.lock` (`agent_disabled_lockfile`)][agent_disabled_lockfile] + * [`classes.txt` (`classfile`)][classfile] --- This file is a favorite for external integration. It lists all of the classes assigned to this agent node. + * [`graphs` (`graphdir`)][graphdir] --- Agent nodes write a set of `.dot` graph files to this directory when graphing is enabled. These graphs can be used to diagnose problems with catalog application, as well as to visualize the configuration catalog. + * [`last_run_summary.yaml` (`lastrunfile`)][lastrunfile] + * [`last_run_report.yaml` (`lastrunreport`)][lastrunreport] + * [`resources.txt` (`resourcefile`)][resourcefile] + * [`state.yaml` (`statefile`)][statefile] +* [`yaml` (`yamldir`)][yamldir] + +[bucketdir]: ./configuration.html#bucketdir +[client_datadir]: ./configuration.html#client_datadir +[clientbucketdir]: ./configuration.html#clientbucketdir +[clientyamldir]: ./configuration.html#clientyamldir +[devicedir]: ./configuration.html#devicedir +[factpath]: ./configuration.html#factpath +[pluginfactdest]: ./configuration.html#pluginfactdest +[libdir]: ./configuration.html#libdir +[plugindest]: ./configuration.html#plugindest +[module_working_dir]: ./configuration.html#module_working_dir +[reportdir]: ./configuration.html#reportdir +[serverdatadir]: ./configuration.html#server_datadir +[statedir]: ./configuration.html#statedir +[agent_catalog_run_lockfile]: ./configuration.html#agent_catalog_run_lockfile +[agent_disabled_lockfile]: ./configuration.html#agent_disabled_lockfile +[classfile]: ./configuration.html#classfile +[graphdir]: ./configuration.html#graphdir +[lastrunfile]: ./configuration.html#lastrunfile +[lastrunreport]: ./configuration.html#lastrunreport +[resourcefile]: ./configuration.html#resourcefile +[statefile]: ./configuration.html#statefile +[yamldir]: ./configuration.html#yamldir diff --git a/docs/_openvox_9x/environment_isolation.md b/docs/_openvox_9x/environment_isolation.md new file mode 100644 index 000000000..c713794be --- /dev/null +++ b/docs/_openvox_9x/environment_isolation.md @@ -0,0 +1,93 @@ +--- +layout: default +title: "Environment Isolation" +description: "Generating metadata to isolate resources in environments in OpenVox" +--- + +## Environment isolation + +Environment isolation prevents resource types from leaking between your various environments. + +If you use multiple environments with OpenVox, you might encounter issues with multiple versions of the same +resource type leaking between your various environments on the server. This doesn't happen with OpenVox's +built-in resource types, but it can happen with any other resource types. + +This problem occurs because Ruby resource type bindings are global in the Ruby runtime. The first loaded +version of a Ruby resource type takes priority, and subsequent requests to compile in other environments get +that first-loaded version. Environment isolation solves this issue by generating and using metadata that +describes the resource type implementation, instead of using the Ruby resource type implementation, when +compiling catalogs. + +> **Note:** Other environment isolation problems, such as external helper logic issues or varying versions of +> required gems, are not solved by the generated metadata approach. This fixes only resource type leaking. +> Resource type leaking is a problem that affects only servers, not agents. + +## Enable environment isolation + +To use environment isolation, generate metadata files that OpenVox can use instead of the default Ruby +resource type implementations. + +1. On the command line, run `puppet generate types --environment ` for each of your environments. + For example, to generate metadata for your production environment, run: + + ```bash + puppet generate types --environment production + ``` + +2. Whenever you deploy a new version of OpenVox, overwrite previously generated metadata by running: + + ```bash + puppet generate types --environment --force + ``` + +## Enable environment isolation with r10k + +To use environment isolation with r10k, generate types for each environment every time r10k deploys new code. + +1. Use one of the following methods: + * Modify your existing r10k hook to run the `generate types` command after code deployment. + * Create a script that first runs r10k for an environment, then runs `generate types` as a post-run + command. +2. If you have enabled environment-level purging in r10k, whitelist the `resource_types` folder so that + r10k doesn't purge it. + +## Troubleshooting environment isolation + +If the `generate types` command cannot generate certain types, if the generated type has missing or +inaccurate information, or if generation itself has errors or fails, you will get a catalog compilation error +of "type not found" or "attribute not found." + +To fix these errors: + +1. Ensure that your OpenVox resource types are correctly implemented. Refactor any problem resource types. +2. Regenerate the metadata by removing the environment's `.resource_types` directory and running + `generate types` again. +3. If you continue to get catalog compilation errors, disable environment isolation to help isolate the error. + +To disable environment isolation: + +1. Remove the `generate types` command from any r10k hooks. +2. Remove the `.resource_types` directory. + +## The `generate types` command + +When you run `puppet generate types`, it scans the entire environment for resource type implementations, +excluding core OpenVox resource types. + +The command accepts the following options: + +Option | Description +---------------------------|------------ +`--environment ` | The environment for which to generate metadata. Defaults to `production`. +`--force` | Overwrite all previously generated metadata. + +For each resource type implementation it finds, the command generates a corresponding metadata file in the +`/.resource_types` directory. It also syncs the directory so that: + +* Types removed from modules are removed from `resource_types`. +* Types added to modules are added to `resource_types`. +* Types that have not changed (based on timestamp) are kept as-is. +* Types that have changed (based on timestamp) are overwritten with freshly generated metadata. + +The generated metadata files have a `.pp` extension and are read-only. Do not delete them, modify them, or +use expressions from them in manifests. diff --git a/docs/_openvox_9x/environments_about.markdown b/docs/_openvox_9x/environments_about.markdown new file mode 100644 index 000000000..34ef4c844 --- /dev/null +++ b/docs/_openvox_9x/environments_about.markdown @@ -0,0 +1,82 @@ +--- +layout: default +title: "About environments" +--- + +[modulepath]: ./dirs_modulepath.html +[main manifest]: ./dirs_manifest.html + +## Environments + +Environments are isolated groups of OpenVox agent nodes. + +An OpenVox server serves each environment with its own main manifest and module path. This lets you use different versions +of the same modules for different groups of nodes, which is useful for testing changes to your Puppet code before +implementing them on production machines. + +Related topics: [main manifests][main manifest] and [module paths][modulepath]. + +## Access environment name in Puppet manifests + +If you want to share code across environments, you can use the `$environment` variable in your Puppet manifests. + +To get the name of the current environment, use the `$environment` variable, which is set by the OpenVox server. + +## Environments scenarios + +The main uses for environments fall into three categories: permanent test environments, temporary test environments, +and divided infrastructure. + +### Permanent test environments + +In a permanent test environment, there is a stable group of test nodes where all changes must succeed before they can +be merged into the production code. The test nodes are a smaller version of the whole production infrastructure. They +are either short-lived cloud instances or longer-lived virtual machines (VMs) in a private cloud. These nodes stay in +the test environment for their whole lifespan. + +### Temporary test environments + +In a temporary test environment, you can test a single change or group of changes by checking the changes out of +version control into the `$codedir/environments` directory, where it will be detected as a new environment. A temporary +test environment can either have a descriptive name or use the commit ID from the version that it is based on. Temporary +environments are good for testing individual changes, especially if you need to iterate quickly while developing them. +Once you're done with a temporary environment, you can delete it. The nodes in a temporary environment are short-lived +cloud instances or VMs, which are destroyed when the environment ends. + +### Divided infrastructure + +If parts of your infrastructure are managed by different teams that don't need to coordinate their code, you can split +them into environments. + +## Environments limitations + +Environments have limitations, including leakage and conflicts with exported resources. + +### Plugins can leak between environments + +Environment leakage occurs when different versions of Ruby files, such as resource types, exist in multiple +environments. When these files are loaded on the server, the first version loaded is treated as global. Subsequent +requests in other environments get that first loaded version. Environment leakage does not affect the agent, as agents +are only in one environment at any given time. + +### Exported resources can conflict or cross over + +Nodes in one environment can collect resources that were exported from another environment, which causes problems --- +either a compilation error due to identically titled resources, or creation and management of unintended resources. +The solution is to run separate OpenVox servers for each environment if you use exported resources. + +## Troubleshooting environment leaks + +Use one of the following methods to avoid environment leaks: + +* For resource types, you can avoid environment leaks with the `puppet generate types` command as described in the + environment isolation documentation. This command generates resource type metadata files to ensure that each + environment uses the right version of each type. + +* The leakage issue occurs only with the `Puppet::Parser::Functions` API. To fix this, rewrite functions with the + modern functions API, which is not affected by environment leakage. You can include helper code in the function + definition, but if helper code is more complex, it should be packaged as a gem and installed for all environments. + +* Report processors and indirector termini are still affected by this problem, so put them in your global Ruby + directories rather than in your environments. If they are in your environments, you must ensure they all have the + same content. diff --git a/docs/_openvox_9x/environments_creating.markdown b/docs/_openvox_9x/environments_creating.markdown new file mode 100644 index 000000000..d1788cf86 --- /dev/null +++ b/docs/_openvox_9x/environments_creating.markdown @@ -0,0 +1,204 @@ +--- +layout: default +title: "Creating Environments" +--- + +[environment.conf]: ./config_file_environment.html +[modulepath]: ./dirs_modulepath.html +[manifest_dir]: ./dirs_manifest.html +[hiera.yaml]: ./hiera_config_yaml_5.html +[default_manifest]: ./configuration.html#default_manifest +[puppet.conf]: ./config_file_main.html +[writingenc]: ./nodes_external.html + +## Environment structure + +An environment is a branch that gets turned into a directory on your OpenVox server. They follow several conventions. + +When you create an environment, give it the following structure: + +* It contains a `modules` directory, which becomes part of the environment's default module path. +* It contains a `manifests` directory, which will be the environment's default main manifest. +* It can optionally contain a `hiera.yaml` file. +* It can optionally contain an `environment.conf` file, which can locally override configuration settings, + including `modulepath` and `manifest`. + +> **Note:** Environment names can contain lowercase letters, numbers, and underscores. They must match the +> following regular expression: `\A[a-z0-9_]+\Z` + +Related topics: [environment.conf][environment.conf] + +## Environment resources + +An environment specifies resources that the OpenVox server will use when compiling catalogs for agent nodes. +The `modulepath`, the main manifest, Hiera data, and the config version script can all be specified in +`environment.conf`. + +### The `modulepath` + +* The `modulepath` is the list of directories OpenVox will load modules from. +* By default, OpenVox will load modules first from the environment's `modules` directory, and second from the + server's `puppet.conf` file's `basemodulepath` setting, which can be multiple directories. +* If the `modules` directory is empty or absent, OpenVox will only use modules from directories in the + `basemodulepath`. + +Related topics: [The modulepath (default config)][modulepath] + +### The main manifest + +* The main manifest is OpenVox's starting point for compiling a catalog. +* Unless you specify otherwise in `environment.conf`, an environment will use OpenVox's global + `default_manifest` setting to determine its main manifest. +* The value of this setting can be an absolute path to a manifest that all environments will share, or a + relative path to a file or directory inside each environment. +* The default value of `default_manifest` is `./manifests` — the environment's own manifests directory. +* If the file or directory specified by `default_manifest` is empty or absent, OpenVox will not fall back to + any other manifest. Instead, it behaves as if it is using a blank main manifest. + +Related topics: [main manifest][manifest_dir], [environment.conf][environment.conf], +[default_manifest setting][default_manifest], [puppet.conf][puppet.conf]. + +### Hiera data + +Each environment can use its own Hiera hierarchy and provide its own data. + +Related topics: [Hiera: Config file syntax][hiera.yaml]. + +### The config version script + +OpenVox automatically adds a config version to every catalog it compiles, as well as to messages in reports. +The version is an arbitrary piece of data that can be used to identify catalogs and events. By default, the +config version will be the time at which the catalog was compiled (as the number of seconds since January 1, +1970). + +### The environment.conf file + +An environment can contain an `environment.conf` file, which can override values for certain settings: + +* `modulepath` +* `manifest` +* `config_version` +* `environment_timeout` + +Related topics: [environment.conf][environment.conf] + +## Create an environment + +Environments are turned on by default. Create an environment by adding a new directory of configuration data. + +1. Inside your code directory, create a directory called `environments`. +2. Inside the `environments` directory, create a directory with the name of your new environment: + `$codedir/environments/` +3. Create a `modules` directory and a `manifests` directory inside the environment directory. These two + directories will contain your Puppet code. + +### Configure a modulepath + +1. Set `modulepath` in the environment's `environment.conf` file. If you set a value for this setting, the + global `modulepath` setting from `puppet.conf` will not be used by the environment. +2. Check the `modulepath` by specifying the environment when requesting the setting value: + + ```bash + sudo puppet config print modulepath --section server --environment test + ``` + +### Configure a main manifest + +1. Set `manifest` in the environment's `environment.conf` file. As with the global `default_manifest` + setting, you can specify a relative path (resolved within the environment's directory) or an absolute path. +2. To lock all environments to a single global manifest, use the `disable_per_environment_manifest` setting, + which prevents any environment from setting its own main manifest. + +### Configure a config version script + +1. Specify a path to the script in the `config_version` setting in `environment.conf`. OpenVox runs this + script when compiling a catalog for a node in the environment, and uses its output as the config version. + +> **Note:** If you're using a system binary like `git rev-parse`, specify the absolute path to it. If +> `config_version` is set to a relative path, OpenVox will look for the binary in the environment, not in +> the system's PATH. + +## Assign nodes to environments via an ENC + +You can assign agent nodes to environments by using an external node classifier (ENC). By default, all nodes +are assigned to a default environment named `production`. + +1. Ensure that the `environment` key is set in the YAML output that the ENC returns. If the `environment` key + isn't set, the OpenVox server will use the environment requested by the agent. + +> **Note:** The value from the ENC is authoritative if it exists. If the ENC doesn't specify an environment, +> the node's config value is used. + +Related topics: [Writing ENCs][writingenc] + +## Assign nodes to environments via the agent's config file + +You can assign agent nodes to environments by editing the agent's `puppet.conf` file. By default, all nodes +are assigned to a default environment named `production`. + +1. Open the agent's `puppet.conf` file in an editor. +2. Find the `environment` setting in either the `agent` or `main` section. +3. Set the value of the `environment` setting to the name of the desired environment. + +When that node requests a catalog from the OpenVox server, it will request that environment. If you are +using an ENC and it specifies an environment for that node, the ENC value will override the config file. + +> **Note:** Nodes can't be assigned to unconfigured environments. If a node is assigned to an environment +> that doesn't exist, the OpenVox server will fail to compile its catalog. The one exception is if the +> default `production` environment doesn't exist — in that case, the agent will successfully retrieve an +> empty catalog. + +## Global settings for configuring environments + +The settings in the server's `puppet.conf` file configure how OpenVox finds and uses environments. + +### `environmentpath` + +* `environmentpath` is the list of directories where OpenVox will look for environments. The default value + is `$codedir/environments`. +* If you have more than one directory, separate them by colons and put them in order of precedence: + `$codedir/temp_environments:$codedir/environments` +* If environments with the same name exist in both paths, OpenVox uses the first one it encounters. +* Put the `environmentpath` setting in the `main` section of `puppet.conf`. + +### `basemodulepath` + +* `basemodulepath` lists directories of global modules that all environments can access by default. +* The default includes `$codedir/modules` for user-accessible modules. +* Add additional directories of global modules by setting your own value for `basemodulepath`. + +Related topics: [modulepath][modulepath]. + +### `default_manifest` + +* `default_manifest` specifies the main manifest for any environment that doesn't set a `manifest` value in + `environment.conf`. +* The default value is `./manifests` — the environment's own manifests directory. +* The value can be an absolute path to one manifest shared by all environments, or a relative path to a file + or directory inside each environment's directory. + +Related topics: [default_manifest setting][default_manifest]. + +### `disable_per_environment_manifest` + +* When set to `true`, OpenVox uses the same global manifest for every environment. +* If an environment specifies a different manifest in `environment.conf`, OpenVox will not compile catalogs + for nodes in that environment. +* If this setting is `true`, the `default_manifest` value must be an absolute path. + +### `environment_timeout` + +* `environment_timeout` sets how often the OpenVox server refreshes information about environments. It can + be overridden per-environment. +* This setting defaults to `0` (caching disabled), which lowers performance but makes it easy for new users + to deploy updated Puppet code. +* Once your code deployment process is mature, change this setting to `unlimited`. + +To configure `environment_timeout`: + +1. Set `environment_timeout = unlimited` in `puppet.conf`. +2. Change your code deployment process to refresh the OpenVox server whenever you deploy updated code. + +> **Note:** Only use the value `0` or `unlimited`. Most OpenVox servers use a pool of Ruby interpreters, +> which all have their own cache timers. When these timers are out of sync, agents can be served inconsistent +> catalogs. To avoid that inconsistency, refresh the server when deploying. diff --git a/docs/_openvox_9x/file_serving.markdown b/docs/_openvox_9x/file_serving.markdown new file mode 100644 index 000000000..b8dc35264 --- /dev/null +++ b/docs/_openvox_9x/file_serving.markdown @@ -0,0 +1,120 @@ +--- +layout: default +title: "Adding file server mount points" +--- + +[module_files]: ./modules_fundamentals.html#files-in-modules +[fileserver.conf]: ./config_file_fileserver.html +[auth.conf]: /openvox-server/latest/config_file_auth.html +[disable_legacy]: /openvox-server/latest/config_file_puppetserver.html +[external facts]: /openfact/latest/custom_facts.html#external-facts + +Puppet Server includes a file server for transferring static file content to agents; this is what's used whenever a `file` resource has a `source => puppet:///...` attribute specified. + +Generally, [files are stored in modules][module_files]. But if you need to serve larger files that shouldn't be in source control or shouldn't be distributed with a module, you can make a custom file server mount point and let Puppet serve those files from another directory. + +## Summary + +To create a new mount point, you must: + +- Choose a directory on disk for the mount point. Make sure Puppet Server can access it, and put files in it as needed. +- Edit `fileserver.conf` on your Puppet Server node, so Puppet knows which directory to associate with the new mount point. +- Edit `auth.conf` if you want to restrict which nodes can access this mount point. + +Once the mount point is working, you can reference its files at `puppet:////`. + +## What's a mount point, in a Puppet URI? + +Puppet URIs are constructed like this: + +`puppet:////` + +- `` is optional, which is why you usually see `puppet:///` URIs with three slashes. There's little reason to specify a server, since the default is almost always what you want. (It's the value of the `server` setting in OpenVox agent, and a special mock server with a `modules` mount point in Puppet apply.) +- `` is a unique identifier for some collection of files. There are basically three kinds: + - Custom mount points correspond to an arbitrary directory. The rest of this page is about these. + - The special `modules` mount point serves files from the `files` directory of every module. It behaves as if someone had copied the `files` directory from every module into one big directory, renaming each of them with the name of their module. (So the files in `apache/files/...` are available at `puppet:///modules/apache/...`) + - The special `plugins` mount point serves files from the `lib` directory of every module. It behaves as if someone had copied the _contents_ of every `lib` directory into one big directory, with no additional namespacing. OpenVox agent uses this mount point when syncing plugins before a run, but there's no reason to use it in a `file` resource. + - The special `pluginfacts` mount point serves files from the `facts.d` directory of every module, to support [external facts][]. It behaves like the `plugins` mount point, but with a different source directory. + - The special `locales` mount point serves files fromt he `locales` directory of every module, to support automatic downloading of module translations to agents. It also behaves like the `plugins` mount point, and also has a different source directory. +- `` is the remainder of the path to the file, starting from the directory (or imaginary directory) that corresponds to the mount point. + +## Creating a new mount point in `fileserver.conf` + +`fileserver.conf` uses an INI-like syntax. [The `fileserver.conf` page][fileserver.conf] has a complete description, but all you need to know is: + +```ini +[] + path + allow * + +[installer_files] + path /etc/puppetlabs/puppet/installer_files + allow * +``` + +In the example above, a file at `/etc/puppetlabs/puppet/installer_files/oracle.pkg` would be available in manifests as `puppet:///installer_files/oracle.pkg`. + +Make sure that the `puppet` user can access that directory and its contents. + +Always include the `allow *` line, since the default behavior is to deny all access. If you need to control access to a custom mount point, do so in [`auth.conf`][auth.conf]. Putting authorization rules in `fileserver.conf` is deprecated. + +> **Caution:** You should always restrict write access to mounted directories. The file server will follow any symlinks in a file server mount, including links to files that agent nodes should not access (like SSL keys). +> +> When following symlinks, the file server can access any files readable by Puppet Server's user account. + +## Controlling access to a custom mount point in `auth.conf` + +By default, any node with a valid certificate can access the files in your new mount point --- if it can fetch a catalog, it can fetch files; if it can't, it can't. This is the same behavior as the special `modules` and `plugins` mount points. + +If necessary, you can restrict access to a custom mount point in [`auth.conf`][auth.conf]. + +### New-style `auth.conf` + +If you've [disabled the legacy `auth.conf` file by setting `jruby-puppet.use-legacy-auth-conf: false`][disable_legacy], you'll be adding a rule to [Puppet Server's HOCON-format `auth.conf` file][auth.conf], located at `/etc/puppetlabs/puppetserver/conf.d/auth.conf`. + +Your new auth rule must meet the following requirements: + +- It matches requests to all four of these prefixes: + - `/puppet/v3/file_metadata/` + - `/puppet/v3/file_metadatas/` + - `/puppet/v3/file_content/` + - `/puppet/v3/file_contents/` +- Its `sort-order` must be lower than 500, so that it overrides the default rule for the file server. + +For example: + +```text +{ + # Allow limited access to files in /etc/puppetlabs/puppet/installer_files: + match-request: { + path: "^/puppet/v3/file_(content|metadata)s?/installer_files" + type: regex + } + allow: "*.dev.example.com" + sort-order: 400 + name: "dev.example.com large installer files" +}, +``` + +### Legacy `auth.conf` + +If you haven't disabled the legacy `auth.conf` file, you'll be adding a stanza to `/etc/puppetlabs/puppet/auth.conf`. + +Your new auth rule must meet the following requirements: + +- It matches requests to all four of these prefixes: + - `/puppet/v3/file_metadata/` + - `/puppet/v3/file_metadatas/` + - `/puppet/v3/file_content/` + - `/puppet/v3/file_contents/` +- It is located earlier in the `auth.conf` file than the default `/puppet/v3/file` rule. + +For example: + +```text +# Allow limited access to files in /etc/puppetlabs/puppet/installer_files: +path ~ ^/file_(metadata|content)s?/installer_files/ +auth yes +allow *.dev.example.com +allow_ip 192.168.100.0/24 +``` diff --git a/docs/_openvox_9x/format_report.markdown b/docs/_openvox_9x/format_report.markdown new file mode 100644 index 000000000..3438f8f60 --- /dev/null +++ b/docs/_openvox_9x/format_report.markdown @@ -0,0 +1,322 @@ +--- +layout: default +title: "Formats: Reports" +--- + +[about reporting]: reporting_about.html +[report processors]: report.html +[custom report processor]: reporting_write_processors.html +[reportdir]: /openvox/latest/configuration.html#reportdir +[exclude_unchanged_resources]: /openvox/latest/configuration.html#exclude_unchanged_resources + +Every time OpenVox applies a catalog, it builds a `Puppet::Transaction::Report` object describing what +happened during the run: the status of every resource, the events Puppet applied, the log messages it +produced, and metrics about the run as a whole. That object is what [report processors][report processors] +receive, and it is what OpenVox serializes when it sends a report to a server or writes one to disk. + +This page describes **report format 12**, which OpenVox 8 produces. Use it when you write a +[custom report processor][custom report processor] or consume reports from another tool. For an +introduction to how reporting works, see [About reporting][about reporting]. + +## How reports are serialized + +A report is a nested structure of five object types: + +- A **Report** at the top level. +- **Metric** objects, one per metric category, under `metrics`. +- **ResourceStatus** objects, one per resource, under `resource_statuses`. +- **Event** objects, under each resource status's `events`. +- **Log** objects under `logs`. + +When an agent sends a report to OpenVox Server, it serializes this structure as JSON. The built-in +`store` processor instead writes it to [`reportdir`][reportdir] as YAML containing Ruby object tags, so +prefer the JSON form when you exchange reports between tools. + +The canonical machine-readable definition is the +[`report.json` schema](https://github.com/OpenVoxProject/openvox/blob/main/api/schemas/report.json) in the +OpenVox source. + +## Report format versions + +The `report_format` field identifies the structure of the report, not the version of OpenVox that produced +it. It changes only when the objects that make up a report change. + +Every OpenVox release produces format 12. OpenVox inherited that format from the Puppet 8 codebase it +forked from, and the format has not changed since Puppet 7.0.0, so a tool that reads format 12 works +across every OpenVox version. + +Earlier formats come from Puppet releases that predate OpenVox. You encounter them only in reports from +older Puppet agents: + +| Format | First shipped in | +| ------ | --------------------------------------- | +| 7 | Puppet 5.0.0 | +| 8 | Puppet 5.4.0 | +| 9 | Puppet 5.5.0 | +| 10 | Puppet 5.5.3 | +| 11 | Puppet 6.19.0 | +| 12 | Puppet 7.0.0, and every OpenVox release | + +Two of those changes still matter when you consume older reports. Format 11 added `server_used` and kept +`master_used` as an alias, so reports in that format carry both fields. Format 12 removed `master_used` +from the output, but OpenVox still reads it from incoming reports when `server_used` is absent, so older +reports remain loadable. + +## Report + +These fields are present in every report, although some values can be `null` when they do not apply to the +run: + +| Field | Type | Description | +| ----------------------- | ---------------- | ----------- | +| `host` | String | The node the report is for. | +| `time` | String | When the run started, in ISO 8601 format with a nine-digit second fraction. | +| `configuration_version` | Integer or String | The configuration version of the catalog. An integer of seconds since the epoch unless you set your own versioning scheme. | +| `transaction_uuid` | String | A UUID identifying the transaction. The agent sends the same UUID when it requests the catalog, which lets you connect a catalog to its report. | +| `report_format` | Integer | The report format version. `12` in OpenVox 8. | +| `puppet_version` | String | The version of OpenVox that produced the report. | +| `status` | String | The outcome of the run: `failed`, `changed`, or `unchanged`. | +| `transaction_completed` | Boolean | Whether the transaction finished evaluating without an unhandled exception. | +| `noop` | Boolean | Whether the run was started in no-op mode. | +| `noop_pending` | Boolean | Whether the run declined to apply changes because of no-op mode. | +| `environment` | String | The environment the run used, such as `production`. | +| `logs` | Array | The Log objects generated during the run. | +| `metrics` | Object | A map of metric category name to Metric object. | +| `resource_statuses` | Object | A map of resource name, in `Type[title]` form, to ResourceStatus object. | +| `corrective_change` | Boolean | Whether any event in the run was a corrective change. | + +These fields appear only when OpenVox has a value for them, so consumers must treat them as optional: + +| Field | Type | Description | +| ----------------------- | ---------------- | ----------- | +| `server_used` | String | The `servername:port` of the server that compiled the catalog. If failover occurred, this is the first server successfully contacted. Absent for `puppet apply` runs. | +| `catalog_uuid` | String | A UUID identifying a specific catalog, which lets you connect one catalog to multiple reports. | +| `code_id` | String | The identifier of the code the compiler used. | +| `job_id` | String | The identifier of the job this transaction belongs to. | +| `cached_catalog_status` | String | Whether a cached catalog was used, and why: `not_used`, `explicitly_requested`, or `on_failure`. | + +## Metric + +Each entry in `metrics` is a Metric object with three fields: + +| Field | Type | Description | +| -------- | ------ | ----------- | +| `name` | String | The category name: `resources`, `time`, `events`, or `changes`. | +| `label` | String | The category name in title form, such as `Resources`. | +| `values` | Array | The measurements in this category. | + +Every entry in `values` is itself a three-element array of `[name, label, value]`, for example +`["changed", "Changed", 2]`. + +The four categories measure: + +- `resources` — counts of resources in each state: `total`, `skipped`, `failed`, `failed_to_restart`, + `restarted`, `changed`, `out_of_sync`, `scheduled`, and `corrective_change`. All of these are present + even when the count is zero. +- `time` — how long things took, in seconds. Contains one entry per resource type that OpenVox evaluated, + named after the type in lowercase, plus `config_retrieval`, `transaction_evaluation`, + `catalog_application`, and `total`. +- `events` — counts of events by status. `total`, `success`, and `failure` are always present; `noop` and + `audit` appear only when the run produced events with those statuses. +- `changes` — a single `total` entry counting the changes in the transaction. + +## ResourceStatus + +Each value in `resource_statuses` describes what happened to one resource: + +| Field | Type | Description | +| ------------------- | ---------------- | ----------- | +| `resource_type` | String | The capitalized type name, such as `File`. | +| `title` | String | The resource title. | +| `resource` | String | The resource name in `Type[title]` form. Deprecated: this always matches the key this status is stored under. | +| `provider_used` | String or null | The provider the resource used. | +| `file` | String or null | The manifest that declared the resource. | +| `line` | Integer or null | The line in that manifest. | +| `evaluation_time` | Number or null | How long the resource took to evaluate, in seconds. | +| `change_count` | Integer | How many properties changed. | +| `out_of_sync_count` | Integer | How many properties were out of sync. | +| `tags` | Array of String | The tags on the resource. | +| `time` | String | When the resource was evaluated, in ISO 8601 format with a nine-digit second fraction. | +| `events` | Array | The Event objects for this resource. | +| `skipped` | Boolean | Whether OpenVox skipped the resource. | +| `failed_to_restart` | Boolean | Whether OpenVox failed to restart the resource after another resource notified it. | +| `containment_path` | Array of String | The containers, such as classes and defined types, that contain the resource, ordered from outermost to innermost. | +| `corrective_change` | Boolean | Whether a change or no-op event on this resource corrected unexpected drift. | +| `out_of_sync` | Boolean | Deprecated: true when `out_of_sync_count` is greater than zero. | +| `changed` | Boolean | Deprecated: true when `change_count` is greater than zero. | +| `failed` | Boolean | Deprecated: whether OpenVox hit an error while evaluating the resource. | + +The [`exclude_unchanged_resources`][exclude_unchanged_resources] setting, which defaults to `true`, omits +resources that did not change from the serialized report. Resources that changed, failed, or were skipped +are kept, so do not assume every resource in the catalog appears here. The `resources` metrics still count +the whole catalog, which means `resources.total` is normally larger than the number of entries in +`resource_statuses`. +{: .tip } + +## Event + +Each event records one property that OpenVox audited or attempted to change: + +| Field | Type | Description | +| ------------------- | ----------------------------- | ----------- | +| `property` | String or null | The property the event is about. | +| `previous_value` | String, Array, Object, or null | The value of the property before the change. | +| `desired_value` | String, Array, Object, or null | The value the manifest specified. | +| `historical_value` | String, Array, Object, or null | The audited value from an earlier run, when known. | +| `message` | String | The log message this event generated. | +| `name` | String | The name of the event, such as `file_created`. | +| `status` | String | `success` if the property was out of sync and was corrected, `failure` if correcting it errored, `noop` if it was left alone because of no-op mode, or `audit` if the property was in sync and being audited. | +| `time` | String | When the property was evaluated, in ISO 8601 format with a nine-digit second fraction. | +| `audited` | Boolean | Whether the property is being audited. | +| `redacted` | Boolean or null | Whether OpenVox redacted the event, which it does for `Sensitive` values. | +| `corrective_change` | Boolean | Whether the event corrected unexpected drift between runs. | + +OpenVox converts event values to strings before serializing them, so rich data types arrive as strings +rather than as their original Puppet types. + +## Log + +Each entry in `logs` is a log message from the run: + +| Field | Type | Description | +| --------- | --------------- | ----------- | +| `level` | String | The severity: `debug`, `info`, `notice`, `warning`, `err`, `alert`, `emerg`, or `crit`. | +| `message` | String | The message itself. | +| `source` | String | Where the message came from: a resource, a property of a resource, or the string `Puppet`. | +| `tags` | Array of String | The tags on the source. | +| `time` | String | When the message was sent, in ISO 8601 format with a nine-digit second fraction. | +| `file` | String or null | The manifest that triggered the message. | +| `line` | Integer or null | The line in that manifest. | + +## Example + +This report comes from a `puppet apply` run that created a file and evaluated a `notify` resource. Log +entries and resource statuses are trimmed for length, and an agent run would also include `server_used`: + +```json +{ + "host": "web01.example.com", + "time": "2026-07-20T06:16:41.086156000-04:00", + "configuration_version": 1784542601, + "transaction_uuid": "cf3624e9-744e-44d6-8c5c-057b812eb581", + "report_format": 12, + "puppet_version": "8.28.1", + "status": "changed", + "transaction_completed": true, + "noop": false, + "noop_pending": false, + "environment": "production", + "logs": [ + { + "level": "notice", + "message": "defined content as '{sha256}5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03'", + "source": "/Stage[main]/Main/File[/etc/motd]/ensure", + "tags": ["notice", "file", "class"], + "time": "2026-07-20T06:16:41.087980000-04:00", + "file": "/etc/puppetlabs/code/environments/production/manifests/site.pp", + "line": 1 + }, + { + "level": "notice", + "message": "Applied catalog in 0.00 seconds", + "source": "Puppet", + "tags": ["notice"], + "time": "2026-07-20T06:16:41.090383000-04:00", + "file": null, + "line": null + } + ], + "metrics": { + "resources": { + "name": "resources", + "label": "Resources", + "values": [ + ["total", "Total", 9], + ["skipped", "Skipped", 0], + ["failed", "Failed", 0], + ["failed_to_restart", "Failed to restart", 0], + ["restarted", "Restarted", 0], + ["changed", "Changed", 2], + ["out_of_sync", "Out of sync", 2], + ["scheduled", "Scheduled", 0], + ["corrective_change", "Corrective change", 0] + ] + }, + "time": { + "name": "time", + "label": "Time", + "values": [ + ["file", "File", 0.000751], + ["notify", "Notify", 0.000333], + ["config_retrieval", "Config retrieval", 0.071568], + ["transaction_evaluation", "Transaction evaluation", 0.002611], + ["catalog_application", "Catalog application", 0.004089], + ["total", "Total", 0.004249] + ] + }, + "changes": { + "name": "changes", + "label": "Changes", + "values": [ + ["total", "Total", 2] + ] + }, + "events": { + "name": "events", + "label": "Events", + "values": [ + ["total", "Total", 2], + ["failure", "Failure", 0], + ["success", "Success", 2] + ] + } + }, + "resource_statuses": { + "File[/etc/motd]": { + "title": "/etc/motd", + "file": "/etc/puppetlabs/code/environments/production/manifests/site.pp", + "line": 1, + "resource": "File[/etc/motd]", + "resource_type": "File", + "provider_used": "posix", + "containment_path": ["Stage[main]", "Main", "File[/etc/motd]"], + "evaluation_time": 0.000751, + "tags": ["file", "class"], + "time": "2026-07-20T06:16:41.087274000-04:00", + "failed": false, + "failed_to_restart": false, + "changed": true, + "out_of_sync": true, + "skipped": false, + "change_count": 1, + "out_of_sync_count": 1, + "events": [ + { + "audited": false, + "property": "ensure", + "previous_value": "absent", + "desired_value": "file", + "historical_value": null, + "message": "defined content as '{sha256}5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03'", + "name": "file_created", + "status": "success", + "time": "2026-07-20T06:16:41.087353000-04:00", + "redacted": null, + "corrective_change": false + } + ], + "corrective_change": false + } + }, + "corrective_change": false, + "catalog_uuid": "70898098-dd70-46af-a8a5-328fe921a5e9", + "cached_catalog_status": "not_used" +} +``` + +To produce a report you can inspect yourself, run `puppet apply` with reporting enabled and read the YAML +that the `store` processor writes to [`reportdir`][reportdir]: + +```console +puppet apply --report --reports=store manifest.pp +``` diff --git a/docs/_openvox_9x/functions_basics.md b/docs/_openvox_9x/functions_basics.md new file mode 100644 index 000000000..b8b1524a6 --- /dev/null +++ b/docs/_openvox_9x/functions_basics.md @@ -0,0 +1,57 @@ +--- +layout: default +title: "Writing custom functions: Introduction" +--- + +[catalog]: ./lang_summary.html#compilation-and-catalogs +[manifest]: ./lang_summary.html +[call]: ./lang_functions.html +[forge]: https://forge.puppetlabs.com/ +[stdlib]: http://forge.puppetlabs.com/puppetlabs/stdlib +[built-in]: ./function.html +[module]: ./modules_fundamentals.html +[environment]: ./environments_about.html + +[func_puppet]: ./lang_write_functions_in_puppet.html +[func_legacy]: ./functions_legacy.html +[func_modern]: ./functions_ruby_overview.html + +[lambda]: ./lang_lambdas.html + +## What are functions? + +Functions are plugins used during [catalog compilation][catalog]. When a Puppet [manifest][] calls a function, that function runs and returns a value. + +* Related documentation: [The Puppet language: Calling functions][call] + +Most functions only produce values, but functions can also: + +* Cause side effects that modify the [catalog][]. (For example, the `include` function adds classes to the catalog.) +* Evaluate a provided block of Puppet code, using their other arguments to determine how that code runs. + +Most functions take one or more arguments, which determine their return value and the behavior of any side effects. + +Puppet includes many built-in functions, and more are available in modules on the [Puppet Forge][forge], particularly in the [puppetlabs-stdlib][stdlib] module. + +You can also write your own custom functions. + +## Custom functions + +If you need to manipulate data or talk to third-party services during [catalog compilation][catalog], and if the [built-in functions][built-in] (or functions from Forge modules) aren't sufficient, you can write new functions for Puppet. + +Custom functions work just like Puppet's built-in functions: you can call them during catalog compilation to produce a value (and sometimes cause side effects). You can use your custom functions locally, and you can also share them with other users. + +To make a custom function available to Puppet, you must put it it a [module][] or in an [environment][], in the specific locations where Puppet expects to find functions. + +Puppet offers three interfaces for writing custom functions: + +Interface | Description +----------|------------ +[The Puppet language][func_puppet] | The easiest way to write functions, which you can use without knowing any Ruby. However, it's less powerful than the Ruby API: pure Puppet functions can only have one signature per function, and can't take a [lambda][] (block of Puppet code). +[The modern Ruby functions API][func_modern] (`Puppet::Functions`) | The most powerful and flexible way to write functions. It requires some knowledge of Ruby. +[The legacy Ruby functions API][func_legacy] (`Puppet::Parser::Functions`) | **Avoid.** This deprecated API has major problems and remains only for backward compatibility with very old modules. Use the Puppet language or the modern Ruby API instead. + +## Guidelines for writing custom functions + +* Whenever possible, avoid causing side effects. "Side effects" are any effect other than producing a value; in the context of Puppet, it usually means modifying the catalog by adding classes or resources to it. +* In Ruby functions, it's sometimes possible to change the values of existing variables. Never do this. Everything else in Puppet relies on this never happening. diff --git a/docs/_openvox_9x/functions_legacy.md b/docs/_openvox_9x/functions_legacy.md new file mode 100644 index 000000000..27a296d62 --- /dev/null +++ b/docs/_openvox_9x/functions_legacy.md @@ -0,0 +1,231 @@ +--- +layout: default +title: "The legacy Ruby functions API" +--- + +[environments]: ./environments_about.html +[func_modern]: ./functions_ruby_overview.html +[func_puppet]: ./lang_write_functions_in_puppet.html + +Puppet includes two Ruby APIs for writing custom functions. This page is about the legacy API, which uses the `Puppet::Parser::Functions` namespace. + +> **Important:** This API has severe problems --- most notably, functions that use it will leak between [environments][]. You should never use this API unless you absolutely must support Puppet 3. +> +> * If you want to write functions in Ruby without this API's failures and limitations, use [the modern Ruby functions API.][func_modern] +> * If you want an easier way to write functions, try [writing them in the Puppet language.][func_puppet] + + +## Where to put your functions + +Functions are implemented in individual .rb files (whose filenames must match the names of their respective functions), and should be distributed in modules. Put custom functions in the lib/puppet/parser/functions subdirectory of your module. + +## First Function --- small steps + +New functions are defined by executing the `newfunction` method +inside the `Puppet::Parser::Functions` module. You pass the name of +the function as a symbol to `newfunction`, and the code to be run as +a block. So a trivial function to write a string to a file in /tmp +might look like this: + +```ruby + module Puppet::Parser::Functions + newfunction(:write_line_to_file) do |args| + filename = args[0] + str = args[1] + File.open(filename, 'a') {|fd| fd.puts str } + end + end +``` + +To use this function, it's as simple as using it in your manifest: + +```ruby + write_line_to_file('/tmp/some_file', "Hello world!") +``` + +(Note that this is not a useful function by any stretch of the imagination.) + +The arguments to the function are passed into the block via the +`args` argument to the block. This is simply an array of all of the +arguments given in the manifest when the function is called. +There's no real parameter validation, so you'll need to do that +yourself. + +> **Note:** Accepting an `args` argument in the `newfunction() do` block is mandatory, even if you won't be doing anything with the arguments. + +This simple `write_line_to_file` function is an example of a +*statement* function. It performs an action, and does not return a +value. The other type of function +is an *rvalue* function, which you must use in a context which +requires a value, such as an `if` statement, a `case` statement, or a +variable or attribute assignment. You could implement a `rand` +function like this: + +```ruby + module Puppet::Parser::Functions + newfunction(:rand, :type => :rvalue) do |args| + rand(args.empty? ? 0 : args[0]) + end + end +``` + +This function works identically to the Ruby built-in rand function. +Randomising things isn't quite as useful as you might think, +though. The first use for a `rand` function that springs to mind is +probably to vary the minute of a cron job. For instance, to stop +all your machines from running a job at the same time, you might do +something like: + +```ruby + cron { run_some_job_at_a_random_time: + command => "/usr/local/sbin/some_job", + minute => rand(60) + } +``` + +But the problem here is quite simple: every time the Puppet client +runs, the rand function gets re-evaluated, and your cron job moves +around. The moral: just because a function *seems* like a good +idea, don't be so quick to assume that it'll be the answer to all +your problems. + +## Using Facts and Variables + +Which raises the question: what *should* you do if you want to splay +your cron jobs on different machines? +The trick is to tie the minute value to something that's invariant +in time, but different across machines. Perhaps the MD5 +hash of the hostname, modulo 60, or maybe the IP address of the +host converted to an integer, modulo 60. Neither +guarantees uniqueness, but you can't really expect that with a range +of no more than 60 anyway. + +But given that functions are run on the OpenVox server, how do you get at +the hostname or IP address of the agent node? +The answer is that facts returned by facter can be used in our +functions. + +### Example 1 + +```ruby + require 'ipaddr' + + module Puppet::Parser::Functions + newfunction(:minute_from_address, :type => :rvalue) do |args| + IPAddr.new(lookupvar('ipaddress')).to_i % 60 + end + end +``` + +### Example 2 + +```ruby + require 'md5' + + module Puppet::Parser::Functions + newfunction(:hour_from_fqdn, :type => :rvalue) do |args| + MD5.new(lookupvar('fqdn')).to_s.hex % 24 + end + end +``` + +### Example 3 + +```ruby + module Puppet::Parser::Functions + newfunction(:has_fact, :type => :rvalue) do |arg| + lookupvar(arg[0]) != nil + end + end +``` + +Basically, to get a fact's or variable's value, you just call +`lookupvar('FACT NAME')`. + +## Calling Functions from Functions + +Functions can be accessed from other functions by +prepending `function_` to the name of the function you are trying to call. This will cause Puppet to automatically locate and load the function; you shouldn't need to call any special methods to make a function available. + +Also keep in mind that when calling a puppet function from the puppet DSL, arguments are all passed in as an anonymous array. This is not the case when calling the function from within Ruby. To work around this, you must create the anonymous array yourself by putting the arguments (even if there is only one argument) inside square brackets like this: + +```ruby + [ arg1, arg1, arg3 ] +``` + +### Example + +```ruby + module Puppet::Parser::Functions + newfunction(:myfunc2, :type => :rvalue) do |args| + function_myfunc1( [ arg1, arg2, ... ] ) + end + end +``` + +## Handling Errors + +To throw a parse/compile error in your function, in a similar +manner to the `fail()` function: + +```ruby + raise Puppet::ParseError, "my error" +``` + +## Referencing Custom Functions In Templates + +To call a custom function within a [Puppet Template](./lang_template.html), you can do: + +```erb +<%= scope.function_namegoeshere(["one","two"]) %> +``` + +Replace "namegoeshere" with the function name, and even if there is only one argument, still +include the array brackets. + +## Notes on Backward Compatibility + +### Accessing Files With Older Versions of Puppet + +In Puppet 2.6.0 and later, functions can access files with the expectation that +it will just work. In versions prior to 2.6.0, functions that accessed files +had to explicitly warn the parser to recompile the configuration if the files +they relied on changed. + +If you find yourself needing to write custom functions for older versions of Puppet, the relevant instructions are preserved below. + +#### Accessing Files in Puppet 0.23.2 through 0.24.9 + +Until Puppet 0.25.0, safe file access was achieved by adding `self.interp.newfile($filename)` to the function. E.g., to accept a file name and return the last line of that file: + +```ruby + module Puppet::Parser::Functions + newfunction(:file_last_line, :type => :rvalue) do |args| + self.interp.newfile(args[0]) + lines = IO.readlines(args[0]) + lines[lines.length - 1] + end + end +``` + +#### Accessing Files in Puppet 0.25.x + +In release 0.25.0, the necessary code changed to: + +```ruby + parser = Puppet::Parser::Parser.new(environment) + parser.watch_file($filename) +``` + +This new code was used identically to the older code: + +```ruby + module Puppet::Parser::Functions + newfunction(:file_last_line, :type => :rvalue) do |args| + parser = Puppet::Parser::Parser.new(environment) + parser.watch_file($filename) + lines = IO.readlines(args[0]) + lines[lines.length - 1] + end + end +``` diff --git a/docs/_openvox_9x/functions_ruby_documenting.md b/docs/_openvox_9x/functions_ruby_documenting.md new file mode 100644 index 000000000..d3cdb644e --- /dev/null +++ b/docs/_openvox_9x/functions_ruby_documenting.md @@ -0,0 +1,67 @@ +--- +layout: default +title: "Writing functions in Ruby: Documenting Ruby functions" +--- + +[puppet strings]: https://github.com/voxpupuli/openvox-strings +[overview]: ./functions_ruby_overview.html +[signatures]: ./functions_ruby_signatures.html + +[Puppet Strings][], a free documentation tool for OpenVox, can extract code details and specially-formatted comments to build +documentation pages for functions. This page describes the proper formatting to make your comments work well with Strings. + +> **Note:** This is one of several pages describing the Ruby functions API. Before reading it, make sure you understand the [overview of this API][overview] and how to [define function signatures][signatures]. + +## Examples + +Full content for this page is coming soon. In the meantime, the following examples show how to format comments in two situations: +a function with two explicit signatures, and a function with an automatic signature. + +```ruby +# Subtracts two things. +Puppet::Functions.create_function(:subtract) do + # Subtracts two integers. + # @param x The first integer. + # @param y The second integer. + # @return [Integer] Returns x - y. + # @example Subtracting two integers. + # subtract(5, 1) => 4 + dispatch :subtract_ints do + param 'Integer', :x + param 'Integer', :y + end + + # Subtracts two arrays. + # @param x The first array. + # @param y The second array. + # @return [Array] Returns x - y. + # @example Subtracting two arrays. + # subtract([3, 2, 1], [1]) => [3, 2] + dispatch :subtract_arrays do + param 'Array', :x + param 'Array', :y + end + + def subtract_ints(x, y) + x - y + end + + def subtract_arrays(x, y) + x - y + end +end +``` + +```ruby +# Says goodbye (in the master's output). +Puppet::Functions.create_function(:goodbye) do + # @param [String] name The name of the person to say goodbye to. + # @return [Undef] + # @example Saying goodbye is hard to do. + # goodbye('world') => 'goodbye world' + def goodbye(name) + puts "goodbye #{name}!" + nil + end +end +``` diff --git a/docs/_openvox_9x/functions_ruby_implementation.md b/docs/_openvox_9x/functions_ruby_implementation.md new file mode 100644 index 000000000..573abef24 --- /dev/null +++ b/docs/_openvox_9x/functions_ruby_implementation.md @@ -0,0 +1,115 @@ +--- +layout: default +title: "Writing functions in Ruby: Using special features in implementation methods" +--- + +[overview]: ./functions_ruby_overview.html +[signatures]: ./functions_ruby_signatures.html +[documenting]: ./functions_ruby_documenting.html +[parser scope]: https://www.rubydoc.info/gems/puppet/Puppet/Parser/Scope +[variables]: ./lang_variables.html +[facts]: ./lang_facts_and_builtin_vars.html +[trusted data]: ./lang_facts_and_builtin_vars.html#trusted-facts +[server data]: ./lang_facts_and_builtin_vars.html#server_facts-variable +[lambdas]: ./lang_lambdas.html +[proc]: https://ruby-doc.org/core/Proc.html + +For the most part, implementation methods are normal Ruby. However, there are some special features available for accessing +OpenVox variables, working with provided blocks of Puppet code, and calling other functions. + +> **Note:** This is one of several pages describing the Ruby functions API. Before reading it, make sure you understand the [overview of this API][overview] and how to [define function signatures][signatures]. + +## Accessing OpenVox variables + +Most functions should only use the arguments they are passed. However, you also have the option of accessing +globally-reachable [OpenVox variables][variables]. The main use case for this is accessing [facts][], [trusted data][], or [server data][]. + +> **Note:** Functions cannot access **local** variables in the scope from which they were called. They can only access global variables or fully-qualified class variables. + +To access variables, use the special `closure_scope` method, which takes no arguments and returns [a `Puppet::Parser::Scope` object][parser scope]. + +The only method you should call on the scope object is `#[](varname)`, which returns the value of the specified variable. Make sure to exclude the `$` from the variable name. + +Example: + +```ruby +Puppet::Functions.create_function(:'mymodule::fqdn_rand') do + dispatch :fqdn do + # no arguments + end + + def fqdn() + scope = closure_scope + fqdn = scope['facts']['networking']['fqdn'] + # ... + end +end +``` + +## Working with lambdas (code blocks) + +If their signatures allow it (see [Defining function signatures][signatures]), functions can accept [lambdas][] (blocks of Puppet code). Once a function _has_ a lambda, it will generally need to execute it. + +To do this, use Ruby's normal block calling conventions. + +### Checking for a block with `block_given?` + +If your signature specified an optional code block, your implementation method can check for its presence with the `block_given?` method. This is `true` if a block was provided, `false` if not. + +### Executing a block with `yield()` + +When you know a block was provided, you can execute it any number of times with the `yield()` method. + +The arguments to `yield` will be passed as arguments to the lambda; since your signature probably specified the number and +type of arguments the lambda should expect, you should be able to call it with confidence. + +The return value of the `yield` call will be the return value of the provided lambda. + +### Capturing a block as a Proc + +If you need to introspect a provided lambda, or pass it on to some other method, an implementation method can capture it as a +[Proc][] by specifying an extra argument with an ampersand (`&`) flag. This works the same way as capturing a Ruby block as a Proc. + +Once you've captured the block, you can execute it with `#call` instead of `yield`. You can also use any other Proc instance methods to examine it. + +```ruby +def implementation(arg1, arg2, *splat_arg, &block) + # Now the `block` variable has the provided lambda, as a Proc. + block.call(arg1, arg2, splat_arg) +end +``` + +## Calling other functions + +If you want to call another Puppet function (like `include`) from inside a function, use the special `call_function(name, *args, &block)` method. + +```ruby +# Flatten an array of arrays of strings, then pass it to include: +def include_nested(array_of_arrays) + call_function('include', *array_of_arrays.flatten) +end +``` + +* The first argument must be the name of the function to call, as a string. +* The next arguments can be any data type that the called function accepts. They will be passed as arguments to the called function. +* The last argument may be a Ruby Proc, or a Puppet [lambda][lambdas] previously captured as a Proc (see above). You can also provide a block of Ruby code using the normal block syntax. + +```ruby +def my_function1(a, b, &block) + # passing given Proc + call_function('my_other_function', a, b, &block) +end + +def my_function2(a, b) + # using a Ruby block + call_function('my_other_function', a, b) { |x| ... } +end +``` + +## Next pages + +To make this API reference easier to use, we've split some of its larger topics into separate pages. Please read the following pages to learn the remainder of the Ruby functions API: + +* [Documenting Ruby functions][documenting]. Puppet Strings, a free documentation tool for OpenVox, can extract + documentation from functions and display it to your module's users. This page describes how to format your code + comments to work well with Strings. diff --git a/docs/_openvox_9x/functions_ruby_overview.md b/docs/_openvox_9x/functions_ruby_overview.md new file mode 100644 index 000000000..439dd6fe8 --- /dev/null +++ b/docs/_openvox_9x/functions_ruby_overview.md @@ -0,0 +1,124 @@ +--- +layout: default +title: "Writing functions in Ruby: Overview and examples" +--- + +[signatures]: ./functions_ruby_signatures.html +[implementation]: ./functions_ruby_implementation.html +[documenting]: ./functions_ruby_documenting.html +[func_puppet]: ./lang_write_functions_in_puppet.html +[func_legacy]: ./functions_legacy.html +[module]: ./modules_fundamentals.html +[environment]: ./environments_about.html +[symbol]: https://ruby-doc.org/core/Symbol.html +[data types]: ./lang_data_type.html + +OpenVox includes two Ruby APIs for writing custom functions. This page is about the modern API, which uses the `Puppet::Functions` namespace. + +* If you want an easier way to write functions, try [writing them in the Puppet language.][func_puppet] +* If you absolutely must support Puppet 3, you can use [the legacy Ruby functions API.][func_legacy] + +## Basic syntax + +```ruby +# /etc/puppetlabs/code/environments/production/modules/mymodule/lib/puppet/functions/mymodule/upcase.rb +Puppet::Functions.create_function(:'mymodule::upcase') do + dispatch :up do + param 'String', :some_string + end + + def up(some_string) + some_string.upcase + end +end +``` + +To write a new function in Ruby, use the `Puppet::Functions.create_function` method. You don't need to `require` any Puppet +libraries to make it available; OpenVox handles that automatically when it loads the function file. + +The `create_function` method requires: + +* A function name. +* A block of code (which takes no arguments). This block should contain: + * One or more signatures, to configure the function's arguments. To build signatures, use the `dispatch` method and the parameter methods. [Signatures are fully described in a separate page.][signatures] + * An implementation method for each signature. The return value of the implementation method will be the return value of the function. + +In summary, with the pieces labled: + +```ruby +Puppet::Functions.create_function(:) do + dispatch : do + param '', : + ... + end + + def (, ...) + + end +end +``` + +## Location + +A Ruby function must be placed in its own file, in the `lib/puppet/functions` directory of either a [module][] or an [environment][]. + +The filename must match the name of the function, and have the `.rb` extension. For namespaced functions, each segment prior +to the final one must be a subdirectory of `functions`, and the final segment must be the filename. + +Examples: + +Function name | File location +----------------------|-------------- +`upcase` | `/mymodule/lib/puppet/functions/upcase.rb` +`upcase` | `/etc/puppetlabs/code/environments/production/lib/puppet/functions/upcase.rb` +`mymodule::upcase` | `/mymodule/lib/puppet/functions/mymodule/upcase.rb` +`environment::upcase` | `/etc/puppetlabs/code/environments/production/lib/puppet/functions/environment/upcase.rb` + +## Function names + + +{% include_relative _naming_functions.md %} + + +### Pass names to `create_function` as symbols + +When you call the `Puppet::Functions.create_function` method, you should pass the function's name to it as a Ruby [symbol][]. (Although it can accept a string, we recommend always using a symbol.) + +To turn a function name into a symbol: + +* If the name is global, prefix it with a colon (like `:str2bool`). +* If it's namespaced: quote the name, then prefix the full quoted string with a colon (like `:'stdlib::str2bool'`). + +## Behavior of Ruby functions + +Ruby functions can have multiple signatures. When a function is called, OpenVox checks each signature in order, comparing +the allowed arguments to the arguments that were actually passed. Arguments are checked using OpenVox's +[data type system][data types], the same way class parameters are checked. + +As soon as OpenVox finds a signature that can accept the provided arguments, it calls the associated implementation method, +passing the arguments to that method. When the method finishes running and returns a value, OpenVox uses that as the +function's return value. + +If none of the function's signatures match the provided arguments, OpenVox fails compilation and logs an error message describing the mismatch between the provided and expected arguments. + +### Conversion of Puppet and Ruby data types + +When function arguments are passed to a Ruby method, they're converted to Ruby objects. Similarly, the method's return value is converted to a Puppet data type when the Puppet manifest regains control. + +OpenVox converts data types between the Puppet language and Ruby as follows: + + +{% include_relative _puppet_types_to_ruby_types.md %} + + +## Next pages + +To make this API reference easier to use, we've split some of its larger topics into separate pages. Please read the following pages to learn the remainder of the Ruby functions API: + +* [Defining function signatures][signatures]. This page describes the `dispatch` method and the parameter methods. +* [Using special features in implementation methods][implementation]. For the most part, implementation methods are basic + Ruby. However, there are some special features available for accessing OpenVox variables, working with provided blocks + of Puppet code, and calling other functions. +* [Documenting Ruby functions][documenting]. Puppet Strings, a free documentation tool for OpenVox, can extract + documentation from functions and display it to your module's users. This page describes how to format your code + comments to work well with Strings. diff --git a/docs/_openvox_9x/functions_ruby_signatures.md b/docs/_openvox_9x/functions_ruby_signatures.md new file mode 100644 index 000000000..88205a119 --- /dev/null +++ b/docs/_openvox_9x/functions_ruby_signatures.md @@ -0,0 +1,217 @@ +--- +layout: default +title: "Writing functions in Ruby: Defining function signatures" +--- + +[overview]: ./functions_ruby_overview.html +[symbol]: https://ruby-doc.org/core/Symbol.html +[ruby_string]: https://ruby-doc.org/core/String.html +[data type]: ./lang_data_type.html +[lambda]: ./lang_lambdas.html +[call]: ./lang_functions.html +[callable]: ./lang_data_abstract.html#callable +[variant]: ./lang_data_abstract.html#variant +[abstract data types]: ./lang_data_abstract.html +[implementation]: ./functions_ruby_implementation.html +[documenting]: ./functions_ruby_documenting.html + +Functions can specify how many arguments they expect, and can specify a data type for each argument. The rule set for a function's arguments is called a **signature.** + +Since functions support more advanced argument checking than Ruby does, the functions API uses a lightweight domain-specific language (DSL) to specify signatures. + +> **Note:** This is one of several pages describing the Ruby functions API. Before reading it, make sure you understand the [overview of this API.][overview] + +## Number of signatures + +A function written in Ruby can have more than one signature. + +Using multiple signatures is an easy way to have a function behave differently when passed different types or quantities of +arguments --- instead of writing complex logic to decide what to do, you can write separate implementations and let OpenVox +figure out which one to use. + +If a function has multiple signatures, OpenVox checks them in the order they're written and uses the first one to match the provided arguments. + +## Using automatic signatures + +If your function only needs one signature, and you're willing to skip the API's data type checking, you can use an automatic signature. To do so: + +* Do not write a `dispatch` block. +* Define one implementation method whose name _matches the final namespace segment_ of the function's name. + +```ruby +Puppet::Functions.create_function(:'stdlib::camelcase') do + def camelcase(str) + str.split('_').map{|e| e.capitalize}.join + end +end +``` + +In this case, since the last segment of `stdlib::camelcase` is `camelcase`, we must define a method named `camelcase`. + +### Drawbacks of automatic signatures + +Although functions with automatic signatures are simpler to write, they give worse error messages when called incorrectly. +Users will get a useful error if they call the function with a wrong number of arguments, but if they give the wrong _type_ +of argument, they'll get something unhelpful. (For example, if you pass the function above a number instead of a string, +it reports `Error: Evaluation Error: Error while evaluating a Function Call, undefined method 'split' for 5:Fixnum` +at the call site.) + +If your function might be used by anyone other than yourself, you should support your users by writing a signature with `dispatch`. + +## Writing signatures with `dispatch` + +To write a signature, use the `dispatch` method. + +```ruby + # A signature that takes a single string argument + dispatch :camelcase do + param 'String', :input_string + return_type 'String' # optional + end +``` + +`dispatch` takes: + +* The name of an implementation method, provided as a Ruby [symbol][]. + * The corresponding method must be defined somewhere in the `create_function` block, usually after all the signatures. +* A block of code, which should only contain calls to the parameter and return methods (described below). + +## Parameter methods + +In the code block of a `dispatch` statement, you can specify arguments with special parameter methods. All of these methods take two arguments: + +* The allowed data type for the argument, as a [string][ruby_string]. + * Types are specified using OpenVox's [data type syntax][data type]. +* A user-facing name for the argument, as a [symbol][]. + * This name is only used in documentation and error messages; it doesn't have to match the argument names in the implementation method. + +The order in which you call these methods is important: the function's first argument should go first, the second one second, etc. + +The following parameter methods are available: + +Method name | Description +----------------------------------------------|------------ +`param` or `required_param` | A mandatory argument. You can use any number of these. **Position:** All mandatory arguments must come first. +`optional_param` | An argument that can be omitted. You can use any number of these. When there are multiple optional arguments, users can only pass latter ones if they also provide values for the prior ones. This also applies to repeated arguments. **Position:** Must come _after_ any required arguments. +`repeated_param` or `optional_repeated_param` | A repeatable argument, which can receive zero or more values. A signature can only use one repeatable argument. **Position:** Must come _after_ any non-repeating arguments. +`required_repeated_param` | A repeatable argument, which must receive one or more values. A signature can only use one repeatable argument. **Position:** Must come _after_ any non-repeating arguments. +`block_param` or `required_block_param` | A mandatory [lambda][] (block of Puppet code). A signature can only use one block. **Position:** Must come _after_ all other arguments. +`optional_block_param` | An optional [lambda][] (block of Puppet code). A signature can only use one block. **Position:** Must come _after_ all other arguments. + +### More about repeatable arguments + +When specifying a repeatable argument, note that: + +* In your implementation method, the repeatable argument appears as an array, which contains all the provided values that weren't assigned to earlier, non-repeatable arguments. +* The specified data type is matched against _each value_ for the repeatable argument, not the repeatable argument as a + whole. For example, if you want to accept any number of numbers, you should specify + `repeated_param 'Numeric', :values_to_average`, not `repeated_param 'Array[Numeric]', :values_to_average`. + +### More about blocks of code + +Functions can receive blocks of Puppet code, as described in [the docs on calling functions.][call] + +The data type for a block argument should always be [`Callable`][callable], or a [`Variant`][variant] that only contains `Callable`s. + +The `Callable` type can optionally specify the type and quantity of parameters that the lambda should accept; for example, +`Callable[String, String]` matches any lambda that can be called with a pair of strings. For more details, +[see the docs on the `Callable` type.][callable] + +For details on how to execute a provided block in your implementation method, see [Using special features in implementation methods.][implementation] + +### Matching arguments with implementation methods + +The implementation method that corresponds to a signature must be able to accept any combination of arguments that the signature might allow. + +Most notably, this means: + +* If the signature has optional arguments, the corresponding method arguments need default values. Otherwise, the function will fail if the arguments are omitted. + + For example: + + ```ruby + dispatch :epp do + required_param 'String', :template_file + optional_param 'Hash', :parameters_hash + end + + def epp(template_file, parameters_hash = {}) + # Note that parameters_hash defaults to an empty hash. + end + ``` + +* If the signature has a repeatable argument, the method must use a splat parameter (like `*args`) as its final argument. + + For example: + + ```ruby + dispatch :average do + required_repeated_param 'Numeric', :values_to_average + end + + def average(*values) + # Inside the method, the `values` variable will be an array of numbers. + end + ``` + +## The `return_type` method + +After specifying a signature's arguments, you can use the `return_type` method to specify the data type of its return value. +This method takes one argument: a [Puppet data type][data type], specified as a string. + +```ruby +dispatch :camelcase do + param 'String', :input_string + return_type 'String' +end +``` + +The return type serves two purposes: documentation, and insurance. + +* Puppet Strings can include information about the return value of a function in generated docs. +* If something goes wrong and your function returns the wrong type (like `nil` when a string is expected), it will fail + early with an informative error instead of allowing compilation to continue with an incorrect value. + +## Specifying local type aliases + +If you are using complicated [abstract data types][] to validate arguments, and if you need to use these types in multiple signatures, they can sometimes become difficult to work with. + +In these cases, you can specify short aliases for your complex types and use the short names in your signatures. +Centralizing the complex part like this can make your function more maintainable by reducing copy-pasted code. + +To specify aliases, use the `local_types` method. + +* You must call `local_types` only once, _before_ any signatures. +* `local_types` takes a block, which should only contain calls to the `type` method. +* The `type` method takes a single [string][ruby_string] argument, of the form `' = '`. + * The name should be a capitalized, CamelCase word, similar to a Ruby class name or the existing [Puppet data types][data type]. + * The type should be a valid [Puppet data type][data type]. + +Example: + +```ruby +local_types do + type 'PartColor = Enum[blue, red, green, mauve, teal, white, pine]' + type 'Part = Enum[cubicle_wall, chair, wall, desk, carpet]' + type 'PartToColorMap = Hash[Part, PartColor]' +end + +dispatch :define_colors do + param 'PartToColorMap', :part_color_map +end + +def define_colors(part_color_map) + # etc +end +``` + +## Next pages + +To make this API reference easier to use, we've split some of its larger topics into separate pages. Please read the following pages to learn the remainder of the Ruby functions API: + +* [Using special features in implementation methods][implementation]. For the most part, implementation methods are basic + Ruby. However, there are some special features available for accessing OpenVox variables, working with provided blocks + of Puppet code, and calling other functions. +* [Documenting Ruby functions][documenting]. Puppet Strings, a free documentation tool for OpenVox, can extract + documentation from functions and display it to your module's users. This page describes how to format your code + comments to work well with Strings. diff --git a/docs/_openvox_9x/getting_started.markdown b/docs/_openvox_9x/getting_started.markdown new file mode 100644 index 000000000..2f799973e --- /dev/null +++ b/docs/_openvox_9x/getting_started.markdown @@ -0,0 +1,198 @@ +--- +layout: default +title: "Getting started with OpenVox" +--- + +# Getting started with OpenVox + +This guide walks you through the steps to go from a fresh installation to a working +infrastructure managed with Puppet code. By the end you will have an OpenVox Server, +enrolled agents, and a control repository deployed with r10k. + +--- + +**Want to try the full workflow locally first?** See +[Try OpenVox locally with crafty](./getting_started_local.html) for a Docker Compose +environment that mirrors these steps without needing dedicated servers. + +--- + +## Step 1: Install OpenVox Server + +Install OpenVox Server on the node that will compile and serve **catalogs** to your +agents. A catalog is the compiled set of resources and configuration that Puppet applies +to a node. + +Follow the [OpenVox Server installation guide](/openvox-server/latest/install_from_packages.html) +for full instructions. When complete, the `puppetserver` service should be running and +reachable on port 8140. + +--- + +## Step 2: Install and enroll agents + +Install `openvox-agent` on each node you want to manage, then connect it to the server. + +1. Review the [pre-install tasks](./install_pre.html) for system requirements and + network prerequisites. +2. Install the agent: + - [Linux](./install_linux.html) + - [Windows](./install_windows.html) + - [macOS](./install_osx.html) +3. Run a test check-in on the agent. This submits a certificate signing request (CSR) + to the server — OpenVox uses mutual TLS so both sides must trust each other before + the server will issue a catalog: + + ```bash + sudo /opt/puppetlabs/bin/puppet agent --test + ``` + +4. Sign the certificate on the server to approve the agent: + + ```bash + sudo /opt/puppetlabs/bin/puppetserver ca list + sudo /opt/puppetlabs/bin/puppetserver ca sign --certname + ``` + +--- + +## Step 3: Set up a control repository + +A control repository is a Git repository that holds all your Puppet environments. +Each branch becomes an [environment](./environments_about.html) on the server. +r10k reads this repository and deploys branches to `/etc/puppetlabs/code/environments/`. + +### Create the repository + +Start by cloning the [puppetlabs/control-repo](https://github.com/puppetlabs/control-repo) +template, which provides a well-structured starting point: + +```bash +git clone https://github.com/puppetlabs/control-repo.git +cd control-repo +``` + +Create a new empty repository on your Git host (GitHub, GitLab, Gitea, or any host your +server can reach), then point the clone at it. Note that OpenVox uses `production` as +the default environment — make sure your repository's default branch is named +`production`, not `main`: + +```bash +git remote remove origin +git remote add origin +git push -u origin production +``` + +The template's key files are: + +- **`Puppetfile`** — lists modules r10k installs into the environment. Add modules from + the Puppet Forge here as your infrastructure grows. +- **`environment.conf`** — configures the module path to include the `site-modules/` + directory alongside Forge modules. +- **`site-modules/`** — where your own roles, profiles, and custom modules live. +- **`manifests/site.pp`** — the main manifest, which is the entry point for node + classification. +- **`data/`** — Hiera data files, pre-configured with a basic hierarchy. + +### Install r10k + +[r10k](https://github.com/puppetlabs/r10k) is a code deployment tool that reads your +control repository and installs each branch as a Puppet environment, along with any +modules listed in its Puppetfile. + +On the OpenVox Server, install r10k using the Ruby runtime that ships with OpenVox: + +```bash +sudo /opt/puppetlabs/puppet/bin/gem install r10k +``` + +> **Tip:** Once your infrastructure is established, consider managing r10k with the +> [`puppet/r10k`](https://forge.puppet.com/modules/puppet/r10k) Forge module. It +> installs r10k and manages `r10k.yaml` as Puppet resources, so changes to your r10k +> configuration go through the same code review and deployment workflow as everything else. + +### Configure r10k + +Create the r10k configuration directory and a minimal `r10k.yaml`: + +```bash +sudo mkdir -p /etc/puppetlabs/r10k +``` + +**`/etc/puppetlabs/r10k/r10k.yaml`:** + +```yaml +cachedir: '/var/cache/r10k' +sources: + control: + remote: 'https://github.com/your-org/control-repo' + basedir: '/etc/puppetlabs/code/environments' +``` + +Replace the `remote` value with the URL of your control repository. If the server +needs an SSH key to clone from your Git host, configure that key for the root user +before running r10k. + +### Deploy your first environment + +Deploy all branches from the control repository: + +```bash +sudo /opt/puppetlabs/puppet/bin/r10k deploy environment -v +``` + +When complete, `/etc/puppetlabs/code/environments/production/` will contain the +files from your repository's `production` branch. + +--- + +## Step 4: Write and apply Puppet code + +With a deployed environment you are ready to write Puppet code and apply it to nodes. +To confirm the full loop is working, add a `notify` resource to +`manifests/site.pp` in your control repository: + +```puppet +node default { + notify { 'Hello from OpenVox!': + message => 'Your first Puppet catalog change is working.', + } +} +``` + +Commit and push the change to the `production` branch of your control repository, +redeploy with r10k, then run the agent: + +```bash +sudo /opt/puppetlabs/puppet/bin/r10k deploy environment production -v +sudo /opt/puppetlabs/bin/puppet agent --test +``` + +You should see a `Notice: Your first Puppet catalog change is working.` line in the +output confirming the agent applied the updated catalog. + +From here, a few places to go deeper: + +- **[Hello world! Quick start guide](./quick_start_helloworld.html)** — write your first + class and apply it to a node. +- **[Roles and profiles](./the_roles_and_profiles_method.html)** — the recommended pattern + for structuring code in larger deployments. +- **[Installing modules](./modules_installing.html)** — add community modules from the + Puppet Forge to your Puppetfile and deploy them with r10k. +- **[Introduction to Hiera](./hiera_intro.html)** — separate your data from your code + using Hiera. + +--- + +## What's next? + +- **Add more environments** — create a new branch in your control repository and run + `r10k deploy environment -v` to deploy it. Use environments for testing changes + before promoting to production. +- **Automate r10k deploys** — use the [`r10k::webhook` class](https://github.com/voxpupuli/puppet-r10k#webhook) + from the `puppet/r10k` module to set up a webhook that triggers `r10k deploy environment` on every push. +- **Expand your node inventory** — install agents on additional nodes and assign + them classes in `manifests/site.pp` or through [node definitions](./lang_node_definitions.html). +- **Classify nodes at scale** — use an [External Node Classifier](./nodes_external.html) + to assign classes from an external source instead of maintaining node definitions + in `manifests/site.pp`. diff --git a/docs/_openvox_9x/getting_started_local.markdown b/docs/_openvox_9x/getting_started_local.markdown new file mode 100644 index 000000000..141af0224 --- /dev/null +++ b/docs/_openvox_9x/getting_started_local.markdown @@ -0,0 +1,135 @@ +--- +layout: default +title: "Try OpenVox locally with crafty (experimental)" +--- + +# Try OpenVox locally with crafty (experimental) + +Before setting up production infrastructure you can run the full OpenVox stack — +server, OpenVoxDB, an agent, and r10k — locally using Docker Compose via +[voxpupuli/crafty](https://github.com/voxpupuli/crafty). This mirrors the steps in the +[Getting started guide](./getting_started.html) and is a fast way to get familiar with +the workflow before committing to a real installation. + +## Prerequisites + +- Docker and Docker Compose installed and running. +- A fork of [puppetlabs/control-repo](https://github.com/puppetlabs/control-repo) on + your Git host. Open the repository on GitHub and click **Fork** to create a copy + under your own account. You will point the server at your fork so you can push + changes and see them applied in Step 4. + +Clone crafty and change into the OSS example directory: + +```bash +git clone https://github.com/voxpupuli/crafty +cd crafty/openvox/oss +``` + +--- + +## Step 1: Start the OpenVox Server + +The server container runs r10k automatically on startup, so configure your control +repository before bringing it up. Open `compose.yaml`, find the commented-out +`R10K_REMOTE` line, uncomment it, and update it to point at your fork: + +```yaml +R10K_REMOTE: https://github.com//control-repo.git +``` + +Then start the stack. The server takes a minute to become healthy as it bootstraps +its CA and runs r10k — start it now and continue reading while it initialises: + +```bash +docker compose --profile openvox up -d +``` + +Check readiness at any point with: + +```bash +docker compose ps +``` + +--- + +## Step 2: Install and enroll agents + +Once all containers report healthy, run the agent container. crafty enables +[autosigning](./ssl_autosign.html), so the certificate is approved automatically — +no manual signing step is needed: + +```bash +docker compose --profile test run --remove-orphans testing agent -t +``` + +The agent connects to the server, has its certificate signed, and applies the catalog +compiled from your control repository. A successful run ends with output like: + +```text +Notice: Catalog compiled by puppet +Notice: Applied catalog in 0.01 seconds +``` + +--- + +## Step 3: Verify your control repository + +The server runs r10k during startup and deploys each branch of your control repository +as a Puppet environment. Verify the `production` environment was deployed: + +```bash +docker exec oss-openvoxserver-1 ls /etc/puppetlabs/code/environments/ +``` + +The container is named `oss-openvoxserver-1` by default; adjust if yours differs. +You should see a `production/` directory containing the files from your fork's +`production` branch. + +--- + +## Step 4: Write and apply Puppet code + +The agent run in Step 2 already compiled and applied a catalog from the `production` +environment. To iterate on your Puppet code: + +1. Push a change to the `production` branch of your fork. For example, add a `notify` + resource to `manifests/site.pp`: + + ```puppet + node default { + notify { 'Hello from OpenVox!': + message => 'Your first Puppet catalog change is working.', + } + } + ``` + +2. Trigger r10k to redeploy: + + ```bash + docker exec oss-openvoxserver-1 r10k deploy environment production -v + ``` + +3. Run the agent again to apply the updated catalog: + + ```bash + docker compose --profile test run --remove-orphans testing agent -t + ``` + +--- + +## Tear down + +```bash +./clean.sh +``` + +This removes all containers and volumes, giving you a clean slate for the next run. + +--- + +## Next steps + +Once you are comfortable with the workflow, follow the +[Getting started guide](./getting_started.html) to set up a production installation +with real servers and agents. diff --git a/docs/_openvox_9x/hiera_automatic.md b/docs/_openvox_9x/hiera_automatic.md new file mode 100644 index 000000000..a85d09e19 --- /dev/null +++ b/docs/_openvox_9x/hiera_automatic.md @@ -0,0 +1,246 @@ +--- +layout: default +title: "Looking up data with Hiera" +--- + +[data_type]: ./lang_data_type.html +[editing_data]: ./hiera_merging.html +[merging]: ./hiera_merging.html#merge-behaviors + +## Class parameters + +OpenVox looks up the values for class parameters in Hiera, using the fully-qualified name of the parameter (`myclass::parameter_one`) as a lookup key. + +Most classes need configuration, and you can specify them as parameters to a class. This will look up the needed data if not directly given when the class is included in a catalog. +There are several ways OpenVox sets values for class parameters, in this order: + +1. If you're doing a resource-like declaration, OpenVox uses parameters that are explicitly set (if explicitly setting `undef`, a looked-up value or default will be used). + +2. OpenVox uses Hiera, using `::` as the lookup key. For example, it looks up `ntp::servers` for the `ntp` class's `$servers` parameter. + +3. If a parameter still has no value, OpenVox uses the default value from the parameter's default value expression in the class's definition. + +4. If any parameters have no value and no default, OpenVox fails compilation with an error. + +For example, you can set servers for the `NTP` class like this: + +```yaml +# /etc/puppetlabs/code/production/data/nodes/web01.example.com.yaml +--- +ntp::servers: + - time.example.com + - 0.pool.ntp.org +``` + +> Note: The best way to manage this is to use the roles and profiles method, which allows you to store a smaller amount of more meaningful data in Hiera. + +## Puppet lookup + +The `lookup` function uses Hiera to retrieve a value for a given key. + +By default, the lookup function returns the first value found and fails compilation if no values are available. You can also configure the lookup function to merge multiple values into one. + +When looking up a key, Hiera searches up to four hierarchy layers of data, in the following order: + +1. Global hierarchy. +2. The current environment's hierarchy. +3. The indicated module's hierarchy, if the key is of the form `::`. +4. If not found and the module's hierarchy has a `default_hierarchy` entry in its `hiera.yaml` — the lookup is repeated if steps 1–3 did not produce a value. + +> Note: Hiera checks the global layer before the environment layer. If no global `hiera.yaml` file has been configured, Hiera defaults are used. +> If you do not want it to use the defaults, you can create an empty `hiera.yaml` file in `/etc/puppetlabs/puppet/hiera.yaml`. + +### Arguments + +You must provide the name of a key to look up, and can optionally provide other arguments. You can combine these arguments in the following ways: + +- `lookup( , [], [], [] )` +- `lookup( [], )` +- `lookup( as above ) |$key| { }` — lambda returns a default value + +Arguments in `[square brackets]` are optional. + +> Note: Giving a hash of options containing `default_value` at the same time as giving a lambda means that the lambda will win. +> A `default_values_hash` wins over the lambda if it has a value for the looked-up key. + +### Merge behaviors + +Hiera uses a hierarchy of data sources, and a given key can have values in multiple sources. +By default (unless you use one of the merge strategies) it is priority/"first found wins", in which case the search ends as soon as a value is found. + +> Note: Data sources can use the `lookup_options` metadata key to request a specific merge behavior for a key. The lookup function will use that requested behavior unless you specify one. + +### Examples + +Look up a key and return the first value found: + +```puppet +lookup('ntp::service_name') +``` + +A unique merge lookup of class names, then adding all of those classes to the catalog: + +```puppet +lookup('classes', Array[String], 'unique').include +``` + +A deep hash merge lookup of user data, letting higher priority sources remove values by prefixing them with `--`: + +```puppet +lookup( { 'name' => 'users', + 'merge' => { + 'strategy' => 'deep', + 'knockout_prefix' => '--', + }, +}) +``` + +## Arguments accepted by lookup + +You must provide the key's name. The other arguments are optional. + +- `` (String or Array) — The name of the key to look up. This can also be an array of keys. + If Hiera doesn't find anything for the first key, it tries with the subsequent ones, only resorting to a default value if none of them succeed. +- `` (data Type) — A data type that must match the retrieved value; if not, the lookup (and catalog compilation) will fail. Defaults to `Data` which accepts any normal value. +- `` (String or Hash; see [Merge behaviors][merging]) — Whether and how to combine multiple values. If present, this overrides any merge behavior specified in the data sources. + Defaults to no value; Hiera will use merge behavior from the data sources if present, and will otherwise do a first-found lookup. +- `` (any normal value) — If present, lookup returns this when it can't find a normal value. Default values are never merged with found values. + Like a normal value, the default must match the value type. + Defaults to no value; if Hiera can't find a normal value, the lookup (and compilation) will fail. +- `` (Hash) — Alternate way to set the arguments above, plus some less common additional options. + If you pass an options hash, you can't combine it with any regular arguments (except ``). An options hash can have the following keys: + - `'name'` — Same as `` (argument 1). You can pass this as an argument or in the hash, but not both. + - `'value_type'` — Same as ``. + - `'merge'` — Same as ``. + - `'default_value'` — Same as ``. + - `'default_values_hash'` (Hash) — A hash of lookup keys and default values. If Hiera can't find a normal value, it will check this hash for the requested key before giving up. + You can combine this with `default_value` or a lambda, which will be used if the key isn't present in this hash. Defaults to an empty hash. + - `'override'` (Hash) — A hash of lookup keys and override values. OpenVox will check for the requested key in the overrides hash first. + If found, it returns that value as the final value, ignoring merge behavior. Defaults to an empty hash. + - `lookup` can take a lambda, which must accept a single parameter. This is yet another way to set a default value for the lookup; + if no results are found, OpenVox will pass the requested key to the lambda and use its result as the default value. + +Related topics: [Data type][data_type]. + +## Using puppet lookup + +The `puppet lookup` command is the command line interface (CLI) for the lookup function. + +The `puppet lookup` command lets you do Hiera lookups from the command line. It needs to be run on a node that has a copy of your Hiera data. +You can log into an OpenVox server node and run `puppet lookup` with sudo. + +The most common version of this command is: + +```sh +puppet lookup --node --environment --explain +``` + +The `puppet lookup` command searches your Hiera data and returns a value for the requested lookup key, so you can test and explore your data. +It replaces the `hiera` command. Hiera relies on a node's facts to locate the relevant data sources. +By default, `puppet lookup` uses facts from the node you run the command on, but you can get data for any other node with the `--node NAME` option. +If possible, the lookup command will use the requested node's most recent stored facts from PuppetDB. +If PuppetDB is not configured or you want to provide other fact values, pass facts from a JSON or YAML file with the `--facts FILE` option. + +### Usage + +```sh +puppet lookup [--help] [--type ] [--merge first|unique|hash|deep] + [--knock-out-prefix ] [--sort-merged-arrays] + [--merge-hash-arrays] [--explain] [--environment ] + [--default ] [--node ] [--facts ] + [--compile] [--render-as s|json|yaml|binary|msgpack] keys +``` + +### Command examples + +To look up `key_name` using the local node's facts: + +```sh +puppet lookup key_name +``` + +To look up `key_name` with agent.local's facts: + +```sh +puppet lookup --node agent.local key_name +``` + +To get the first value found for `key_name_one` and `key_name_two` with agent.local's facts while merging values and knocking out the prefix `foo`: + +```sh +puppet lookup --node agent.local --merge deep --knock-out-prefix foo key_name_one key_name_two +``` + +To look up `key_name` with agent.local's facts, and return a default value of `bar` if nothing was found: + +```sh +puppet lookup --node agent.local --default bar key_name +``` + +To see an explanation of how the value for `key_name` would be found, using agent.local's facts: + +```sh +puppet lookup --node agent.local --explain key_name +``` + +## Puppet lookup command options + +The `puppet lookup` command has the following options: + +- `--help` — Print a usage message. +- `--explain` — Explain the details of how the lookup was performed and where the final value came from, or the reason no value was found. + Useful when debugging Hiera data. If `--explain` isn't specified, lookup exits with 0 if a value was found and 1 if not. With `--explain`, lookup always exits with 0 unless there is a major error. +- `--node ` — Specify which node to look up data for; defaults to the node where the command is run. + If the node where you're running this command is configured to talk to PuppetDB, the command will use the requested node's most recent facts. Otherwise, override facts with the `--facts` option. +- `--facts ` — Specify a JSON or YAML file that contains key-value mappings to override the facts for this lookup. Any facts not specified in this file maintain their original value. +- `--environment ` — Specify an environment. Different environments can have different Hiera data. +- `--merge first/unique/hash/deep` — Specify the merge behavior, overriding any merge behavior from the data's `lookup_options`. +- `--knock-out-prefix ` — Used with `deep` merge. Specifies a prefix to indicate a value should be removed from the final result. +- `--sort-merged-arrays` — Used with `deep` merge. When this flag is used, all merged arrays are sorted. +- `--merge-hash-arrays` — Used with the `deep` merge strategy. When this flag is used, hashes within arrays are deep-merged with their counterparts by position. +- `--explain-options` — Explain whether a `lookup_options` hash affects this lookup, and how that hash was assembled. +- `--default ` — A value to return if Hiera can't find a value in data. Useful for emulating a call to the `lookup` function that includes a default. +- `--type ` — Assert that the value has the specified type. +- `--compile` — Perform a full catalog compilation prior to the lookup. If your hierarchy and data only use the `$facts`, `$trusted`, and `$server_facts` variables, you don't need this option. + If your Hiera configuration uses arbitrary variables set by a Puppet manifest, you need this to get accurate data. +- `--render-as s/json/yaml/binary/msgpack` — Specify the output format of the results; `s` means plain text. The default when producing a value is `yaml` and the default when producing an explanation is `s`. + +Related topics: [Creating and editing data with Hiera][editing_data]. + +## Access hash and array elements using a key.subkey notation + +Access hash and array members in Hiera using a `key.subkey` notation. + +You can access hash and array elements when doing the following things: + +- Interpolating variables into `hiera.yaml` or a data file. Many of the most commonly used variables, for example `facts` and `trusted`, are deeply nested data structures. +- Using the `lookup` function or the `puppet lookup` command. If the value of `lookup('some_key')` is a hash or array, look up a single member of it by using `lookup('some_key.subkey')`. +- Using interpolation functions that do Hiera lookups, for example `lookup` and `alias`. + +To access a single member of an array or hash: + +1. Use the name of the value followed by a period (`.`) and a subkey. + - If the value is an array, the subkey must be an integer, for example: `users.0` returns the first entry in the `users` array. + - If the value is a hash, the subkey must be the name of a key in that hash, for example, `facts.os`. + - To access values in nested data structures, you can chain subkeys together. For example, since the value of `facts.system_uptime` is a hash, you can access its `hours` key with `facts.system_uptime.hours`. + +## Hiera dotted notation + +The Hiera dotted notation does not support arbitrary expressions for subkeys; only literal keys are valid. + +A hash can include literal dots in the text of a key. For example, the value of `$trusted['extensions']` is a hash containing any certificate extensions for a node, +but some of its keys can be raw OID strings like `1.3.6.1.4.1.34380.1.2.1`. You can access those values in Hiera with the `key.subkey` notation, +but you must put quotation marks — single or double — around the affected subkey. +If the entire compound key is quoted (for example, as required by the lookup interpolation function), use the other kind of quote for the subkey, +and escape quotes (as needed by your data file format) to ensure that you don't prematurely terminate the whole string. + +For example: + +```yaml +aliased_key: "%{lookup('other_key.\"dotted.subkey\"')}" +# Or: +aliased_key: "%{lookup(\"other_key.'dotted.subkey'\")}" +``` + +> Note: Using extra quotes prevents digging into dotted keys. For example, if the lookup key contains a dot (`.`) then the entire key must be enclosed within single quotes within double quotes, +> for example, `lookup("'has.dot'")`. diff --git a/docs/_openvox_9x/hiera_config_yaml_3.md b/docs/_openvox_9x/hiera_config_yaml_3.md new file mode 100644 index 000000000..a751a03d9 --- /dev/null +++ b/docs/_openvox_9x/hiera_config_yaml_3.md @@ -0,0 +1,175 @@ +--- +layout: default +title: "Hiera: Legacy config file syntax (hiera.yaml v3)" +--- + + + +[hierarchy]: ./hiera_intro.html#hiera-hierarchies +[layers]: ./hiera_intro.html#hieras-three-config-layers +[v4]: ./hiera_config_yaml_4.html +[v5]: ./hiera_config_yaml_5.html +[merge]: ./hiera_merging.html +[confdir]: ./dirs_confdir.html +[yaml]: http://www.yaml.org/YAML_for_ruby.html +[custom_backends]: /openvox/latest/hiera_custom_backends.html +[puppetserver_gem]: /openvox-server/latest/gems.html#installing-and-removing-gems +[deep_merge_gem_docs]: https://github.com/danielsdeleo/deep_merge/blob/master/README.md +[interpolate]: ./hiera_merging.html#interpolation + +Hiera's config file is called hiera.yaml. It configures the [hierarchy][] for a given [layer][layers] of data. + +This version of Puppet supports three formats for hiera.yaml --- you can use any of them, although [v4][] and v3 are deprecated. This page is about version 3, the legacy version. + +| Format | Allowed in | Description | +| ------ | ---------- | ----------- | +| [v5][] | All three data layers | The main version of hiera.yaml, which supports all Hiera 5 features. | +| [v4][] | Environment and module layers | Deprecated. A transitional format, used in the rough draft of Hiera 5 (when we were calling it "Puppet lookup"). Doesn't support custom backends. | +| v3 | Global layer | Deprecated. The classic version of hiera.yaml, which has some problems. | + +## Important: version 3 is deprecated + +Version 3 of hiera.yaml is deprecated and not recommended. OpenVox 9.x still supports it for backward compatibility, but you should use [version 5][v5] instead. + +More importantly, it has some major problems: + +* The combinatorial hierarchy (run the whole hierarchy in one backend, then run the whole hierarchy in the next, then...) was annoying to reason about, and it made some simple workflows more difficult than necessary. +* Global configuration of deep hash merge behavior was a terrible idea. Hiera 5 has [better ways to configure it on a per-key basis.][merge] +* hiera.yaml v5 can support v3 backends, but not vice-versa. + +You should upgrade your global hiera.yaml to [version 5][v5] when you get the chance. + +## Location + +The v3 hiera.yaml file can only be used at the [global config layer][layers]. + +The default location for the global hiera.yaml is [`$confdir`][confdir]`/hiera.yaml`. Depending on your platform, that's usually at `/etc/puppetlabs/puppet/hiera.yaml` or `C:\ProgramData\PuppetLabs\puppet\etc\hiera.yaml`. + +You can use [the `hiera_config` setting](./configuration.html#hiera_config) in `puppet.conf` to change the location of the global hiera.yaml. + +## Format + +hiera.yaml v3 must be a [YAML][] hash. + +Each top-level key in the hash **must be a Ruby symbol with a colon (`:`) prefix.** The available settings are listed below, under ["Global Settings"](#global-settings) and ["Backend-Specific Settings"](#backend-specific-settings). + +### Example config file + +[example]: #example-config-file + +```yaml +--- +:backends: + - yaml + - json +:yaml: + :datadir: "/etc/puppetlabs/code/environments/%{::environment}/hieradata" +:json: + :datadir: "/etc/puppetlabs/code/environments/%{::environment}/hieradata" +:hierarchy: + - "nodes/%{::trusted.certname}" + - "virtual/%{::virtual}" + - "common" +``` + +### Default config values + +If the config file exists but has no data, Hiera uses the following default settings: + +```yaml +--- +:backends: yaml +:yaml: + # on *nix: + :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata" + # on Windows: + :datadir: "C:\ProgramData\PuppetLabs\code\environments\%{environment}\hieradata" +:hierarchy: + - "nodes/%{::trusted.certname}" + - "common" +:logger: console +:merge_behavior: native +:deep_merge_options: {} +``` + +## Global settings + +hiera.yaml v3 can contain any the following settings. If absent, they use default values as shown above. **Note that each setting must be a Ruby symbol with a colon (`:`) prefix.** + +### `:hierarchy` + +Must be a **string** or an **array of strings,** where each string is the name of a static or dynamic data source. (A dynamic source is simply one that contains a `%{variable}` interpolation token. [See "Creating Hierarchies" for more details][hierarchy].) + +The data sources in the hierarchy are checked in order, top to bottom. + +**Default value:** `["nodes/%{::trusted.certname}", "common"]` + +### `:backends` + +Must be a **string** or an **array of strings,** where each string is the name of an available Hiera backend. The built-in backends are `yaml` and `json`. Additional backends are available as add-ons. + +> **Note:** hiera.yaml v3 only supports backends written to support Hiera 3. See the Hiera 3 docs about [custom backends][custom_backends] for details. + +The list of backends is processed in order: in the [example above][example], Hiera would check the entire hierarchy in the **yaml** backend before starting again with the **json** backend. + +**Default value:** `"yaml"` + +### `:logger` + +Must be the name of an available logger, as a **string.** + +Loggers only control where warnings and debug messages are routed. You can use one of the built-in loggers, or write your own. The built-in loggers are: + +* `console` (messages go directly to STDERR) +* `puppet` (messages go to Puppet's logging system) +* `noop` (messages are silenced) + +> **Custom loggers:** You can make your own logger by providing a class called, e.g., `Hiera::Foo_logger` (in which case Hiera's internal name for the logger would be `foo`), and giving it class methods called `warn` and `debug`, each of which should accept a single string. + +**Default value:** `"console"`; note that Puppet overrides this and sets it to `"puppet"`, regardless of what's in the config file. + +### `:merge_behavior` + +Which merge behavior the `hiera_hash` function should use. **Note that this does not affect automatic class parameter lookup, the `lookup` function, or the `puppet lookup` command.** + +Must be one of the following: + +* `native` (default) --- merge top-level keys only. +* `deep` --- merge recursively; in the event of conflicting keys, allow **lower priority** values to win. You almost never want this. +* `deeper` --- merge recursively; in the event of a conflict, allow **higher priority** values to win. + +Anything but `native` requires the `deep_merge` Ruby gem to be installed. If you're using Puppet Server, you'll need to use the [`puppetserver gem`][puppetserver_gem] command to install the gem. + +For more details about hash merge lookup strategies, see ["Hash"](./hiera_merging.html#hash) and ["Deep"](./hiera_merging.html#deep) under Merge behaviors. + +### `:deep_merge_options` + +A hash of deep merging options for `hiera_hash`, if `:merge_behavior` is set to `deeper` or `deep`. **Note that this does not affect automatic class parameter lookup, the `lookup` function, or the `puppet lookup` command.** + +For example: + +```yaml +:merge_behavior: deeper +:deep_merge_options: + :knockout_prefix: '--' +``` + +Available options are documented in [the `deep_merge` gem][deep_merge_gem_docs]. + +**Default value:** An empty hash of options. + +## Backend-specific settings + +Any backend can define its own settings and read them from hiera.yaml. If present, the value of a given backend's key must be a **hash,** whose keys are the settings it uses. + +The following settings are available for the built-in backends: + +### `:yaml` and `:json` + +#### `:datadir` + +The directory in which to find data source files. This must be a string. + +You can [interpolate variables][interpolate] into the datadir using `%{variable}` interpolation tokens. This allows you to, for example, point it at `"/etc/puppetlabs/code/hieradata/%{::environment}"` to keep your production and development data entirely separate. + +**Default value:** `"/etc/puppetlabs/code/environments/%{environment}/hieradata"` on \*nix, and `"C:\ProgramData\PuppetLabs\code\environments\%{environment}\hieradata"` on Windows. diff --git a/docs/_openvox_9x/hiera_config_yaml_4.md b/docs/_openvox_9x/hiera_config_yaml_4.md new file mode 100644 index 000000000..dc2c6b350 --- /dev/null +++ b/docs/_openvox_9x/hiera_config_yaml_4.md @@ -0,0 +1,80 @@ +--- +layout: default +title: "Hiera: Legacy config file syntax (hiera.yaml v4)" +--- + +[hierarchy]: ./hiera_intro.html#hiera-hierarchies +[layers]: ./hiera_intro.html#hieras-three-config-layers +[v3]: ./hiera_config_yaml_3.html +[v5]: ./hiera_config_yaml_5.html + + +Hiera's config file is called hiera.yaml. It configures the [hierarchy][] for a given [layer][layers] of data. + +This version of Puppet supports three formats for hiera.yaml --- you can use any of them, although v4 and [v3][] are deprecated. This page is about version 4, a transitional format used in Hiera 5's experimental predecessor (Puppet lookup). + + +Format | Allowed in | Description +-------|-------------------------------|------------ +[v5][] | All three data layers | The main version of hiera.yaml, which supports all Hiera 5 features. +v4 | Environment and module layers | Deprecated. A transitional format, used in the rough draft of Hiera 5 (when we were calling it "Puppet lookup"). Doesn't support custom backends. +[v3][] | Global layer | Deprecated. The classic version of hiera.yaml, which has some problems. + +## Important: version 4 is deprecated + +Version 4 of hiera.yaml is deprecated and not recommended. OpenVox 9.x still supports it for backward compatibility, but you should use [version 5][v5] instead. + +More importantly, version 4 can't use some of Hiera 5's best new features, like custom backends. + +## `hiera.yaml` (Version 4) in a nutshell + +```yaml +# /etc/puppetlabs/code/environments/production/hiera.yaml +--- +version: 4 +datadir: data +hierarchy: + - name: "Nodes" + backend: yaml + path: "nodes/%{trusted.certname}" + + # Putting a JSON level between YAML levels like this + # was impossible in the old format. + - name: "Exported JSON nodes" + backend: json + paths: + # Puppet checks these in order. Even though this is a single + # item in the hierarchy, it acts like multiple hierarchy levels. + - "nodes/%{trusted.certname}" + - "insecure_nodes/%{facts.fqdn}" + + - name: "virtual/%{facts.virtual}" + backend: yaml + + - name: "common" + backend: yaml +``` + +The `hiera.yaml` (version 4) file goes in the main directory of a module or environment. It is a YAML hash that contains three keys: + +* `version` --- Required. Must always be `4`. +* `datadir` --- Optional. The default datadir, for any hierarchy levels that omit it. It is a relative path, from the root of the environment or module. The default is `data`. +* `hierarchy` --- Optional. A hierarchy of data sources to search, in the new format. If omitted, it defaults to a single source called `common` that uses the YAML backend. + +The `hierarchy` is an array of hashes. Unlike in classic Hiera, each hierarchy level must specify its own backend, and can optionally use a separate datadir. + +Each hierarchy level can contain the following keys: + +* `name` --- Required. An arbitrary human-readable name, used for debugging and for `puppet lookup --explain`. + + This is also used as the default `path` if you don't specify any paths. (If the name interpolates variables, Hiera will interpolate when finding data files but leave it uninterpolated when reporting the level's name.) +* `backend` --- Required. Which backend to use. Currently only `yaml` and `json` are supported. +* `path` --- Optional; mutually exclusive with `paths`. The path to a data file. Can interpolate variables, to use different files depending on a node's facts. +* `paths` --- Optional; mutually exclusive with `path`. An array of paths to data files, which can interpolate variables. This acts like multiple hierarchy levels, and is shorthand for writing consecutive levels that use the same backend and datadir. +* `datadir` --- Optional. A one-off datadir to use instead of the default one specified at top level. + +## Changes to version 4 for Hiera 5 + +In the experimental Puppet lookup, you had to use a setting to enable Hiera: for environments you had to set `environment_data_provider = hiera` in environment.conf or puppet.conf, and for modules you had to set `"data_provider":"hiera"` in metadata.json. + +That's no longer necessary. As of Puppet 4.9, Puppet automatically enables Hiera for a module or environment if a hiera.yaml file is present. This works for both v4 and v5 hiera.yaml files. diff --git a/docs/_openvox_9x/hiera_config_yaml_5.md b/docs/_openvox_9x/hiera_config_yaml_5.md new file mode 100644 index 000000000..540ed2b1d --- /dev/null +++ b/docs/_openvox_9x/hiera_config_yaml_5.md @@ -0,0 +1,368 @@ +--- +layout: default +title: "Configuring Hiera" +--- + +[hierarchy]: ./hiera_intro.html#hiera-hierarchies +[layers]: ./hiera_intro.html#hieras-three-config-layers +[confdir]: ./dirs_confdir.html +[module]: ./modules_fundamentals.html +[yaml]: http://www.yaml.org/YAML_for_ruby.html +[variables]: ./hiera_merging.html#interpolating-variables +[interpolation]: ./hiera_merging.html#interpolation +[eyaml]: https://github.com/voxpupuli/hiera-eyaml +[custom puppet function]: ./functions_basics.html +[backends]: ./hiera_custom_backends.html + +The Hiera configuration file is called hiera.yaml. It configures the hierarchy for a given layer of data. + +Related topics: [hierarchy][hierarchy], [layers][layers]. + +## Location of hiera.yaml files + +There are several hiera.yaml files in a typical deployment. Hiera uses three layers of configuration, and the module and environment layers typically have multiple instances. + +The configuration file locations for each layer: + +Layer | Location | Example +------------|-------------------------------------------------|-------- +Global | [`$confdir`][confdir]`/hiera.yaml` | `/etc/puppetlabs/puppet/hiera.yaml` `C:\ProgramData\PuppetLabs\puppet\etc\hiera.yaml` +Environment | [``](./environments_about.html)`/hiera.yaml` | `/etc/puppetlabs/code/environments/production/hiera.yaml` `C:\ProgramData\PuppetLabs\code\environments\production\hiera.yaml` +Module | [``][module]`/hiera.yaml` | `/etc/puppetlabs/code/environments/production/modules/ntp/hiera.yaml` `C:\ProgramData\PuppetLabs\code\environments\production\modules\ntp\hiera.yaml` + +> Note: You can change the location for the global layer's hiera.yaml with OpenVox's `hiera_config` setting. + +Hiera searches for data in the following order: global → environment → module. For more information, see [Hiera configuration layers][layers]. + +Related topics: [$confdir][confdir], [``][module]. + +## Config file syntax (hiera.yaml v5) + +`hiera.yaml` is a YAML file, containing a hash with up to four top-level keys: + +- `version` - Required. Must be the number 5, with no quotes. +- `hierarchy` - An array of hashes, which configures the levels of the hierarchy. +- `default_hierarchy` - An array of hashes, which sets a default hierarchy to be used only if the normal hierarchy entries do not result in a value. Only allowed in a module's hiera.yaml. +- `defaults` - A hash, which can set a default datadir, backend, and options for hierarchy levels. + +```yaml +--- +version: 5 +defaults: # Used for any hierarchy level that omits these keys. + datadir: data # This path is relative to hiera.yaml's directory. + data_hash: yaml_data # Use the built-in YAML backend. + +hierarchy: + - name: "Per-node data" # Unique human-readable name. + path: "nodes/%{trusted.certname}.yaml" # File path, relative to datadir. + # ^^^ IMPORTANT: include the file extension! + + - name: "Per-datacenter business group data" # Uses custom facts. + path: "location/%{facts.whereami}/%{facts.group}.yaml" + + - name: "Global business group data" + path: "groups/%{facts.group}.yaml" + + - name: "Per-datacenter secret data (encrypted)" + lookup_key: eyaml_lookup_key # Uses non-default backend. + path: "secrets/%{facts.whereami}.eyaml" + options: + pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem + pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem + + - name: "Per-OS defaults" + path: "os/%{facts.os.family}.yaml" + + - name: "Common data" + path: "common.yaml" +``` + +Related topics: [YAML][yaml] + +### The default configuration + +If you omit the `hierarchy` or `defaults` keys, Hiera uses the following default values: + +```yaml +--- +version: 5 +hierarchy: + - name: Common + path: common.yaml +defaults: + data_hash: yaml_data + datadir: data +``` + +These defaults are only used if the file is present and specifies `version: 5`. If hiera.yaml is absent, it disables Hiera for that layer. If it specifies a different version, different defaults apply. + +## The defaults key + +The `defaults` key sets default values for the lookup function and `datadir` keys, which lets you omit those keys in your hierarchy levels. + +The value of `defaults` must be a hash, which can have up to three keys: `datadir`, `options`, and one of the mutually exclusive lookup function keys. + +### datadir + +A default value for `datadir`, used for any file-based hierarchy level that doesn't specify its own. If not given, the `datadir` is the directory `data` in the same directory as the `hiera.yaml` configuration file. + +### options + +A default value for options, used for any hierarchy level that does not specify its own. + +### The lookup function keys + +Used for any hierarchy level that doesn't specify its own. This must be one of: + +- `data_hash` - produces a hash of key-value pairs (typically from a data file) +- `lookup_key` - produces values key by key (typically for a custom data provider) +- `data_dig` - produces values key by key (for a more advanced data provider) +- `hiera3_backend` - a data provider that calls out to a legacy Hiera 3 backend (global layer only) + +For the built-in data providers - YAML, JSON, and HOCON - the key is always `data_hash` and the value is one of `yaml_data`, `json_data`, or `hocon_data`. To set a custom data provider as the default, see the data provider documentation. Whichever key you use, the value must be the name of the custom OpenVox function that implements the lookup function. + +Related topics: [custom backends][backends], [custom OpenVox function][custom puppet function]. + +## The hierarchy key + +The `hierarchy` key configures the levels of the hierarchy. + +## The default_hierarchy key + +The `default_hierarchy` key is a top-level key. It works exactly like the hierarchy key, but its values are used only if the normal hierarchy entries in the same module, or any of the higher precedence layers (environment or global) does not result in a value. Within this default hierarchy, the normal merging rules apply. However, the `default_hierarchy` is not permitted in environment or global layers. + +Related topics: [hierarchies][hierarchy]. + +## The hierarchy key: write an array of hashes + +The value of `hierarchy` must be an array of hashes. + +1. Begin each level of the hierarchy with: + + - Two spaces of indentation. + - A hyphen (`-`). + - Another space after the hyphen. + - The first key of that level's hash. + +2. Indent the rest of the hash's keys by four spaces, so they line up with the first key. + +3. Put an empty line between hashes, to visually distinguish them. + + ```yaml + hierarchy: + - name: "Per-node data" + path: "nodes/%{trusted.certname}.yaml" + + - name: "Per-datacenter business group data" + path: "location/%{facts.whereami}/%{facts.group}.yaml" + ``` + +## Configuring a hierarchy level: built-in backends + +Hiera has three built-in backends: YAML, JSON, and HOCON. All of these use files as data sources. + +You can use any combination of these backends in a hierarchy, and can also combine them with custom backends. But if most of your data is in one file format, set default values for the `datadir` and `data_hash` keys. + +Each YAML/JSON/HOCON hierarchy level needs the following keys: + +- `name` — A unique name for this level, shown in debug messages and `--explain` output. +- `path`, `paths`, `glob`, `globs`, or `mapped_paths` (choose one) — The data files to use for this hierarchy level. + - These paths are relative to the datadir, they support variable interpolation, and they require a file extension. See "Specifying file paths" for more details. +- `data_hash` — Which backend to use. Can be omitted if you set a default. The value must be one of the following: + - `yaml_data` for YAML. + - `json_data` for JSON. + - `hocon_data` for HOCON. +- `datadir` — The directory where data files are kept. Can be omitted if you set a default. + - This path is relative to hiera.yaml's directory: if the config file is at `/etc/puppetlabs/code/environments/production/hiera.yaml` and the datadir is set to data, the full path to the data directory is `/etc/puppetlabs/code/environments/production/data`. + - In the global layer, you can optionally set the datadir to an absolute path; in the other layers, it must always be relative. + +Related topics: [variable interpolation][variables], [YAML][yaml]. + +### Specifying file paths + +Options for specifying a file path: + +Key | Data type | Expected value +--------|-----------|--------------- +`path` | String | One file path. +`paths` | Array | Any number of file paths. This acts like a sub-hierarchy: if multiple files exist, Hiera searches all of them, in the order in which they're written. +`glob` | String | One shell-like glob pattern, which might match any number of files. If multiple files are found, Hiera searches all of them in alphanumerical order. +`globs` | Array | Any number of shell-like glob patterns. If multiple files are found, Hiera searches all of them in alphanumerical order (ignoring the order of the globs). +`mapped_paths` | Array or Hash | A fact that is a collection (array or hash) of values. Hiera expands these values to produce an array of paths. + +> Note: You can only use one of these keys in a given hierarchy level. + +Explicit file extensions are required, for example, `common.yaml`, not `common`. + +File paths are relative to the `datadir`: if the full datadir is `/etc/puppetlabs/code/environments/production/data` and the file path is set to `"nodes/%{trusted.certname}.yaml"`, the full path to the file is `/etc/puppetlabs/code/environments/production/data/nodes/.yaml`. + +> Note: Hierarchy levels should interpolate variables into the path. + +Globs are implemented with Ruby's `Dir.glob` method: + +- One asterisk (`*`) matches a run of characters. +- Two asterisks (`**`) matches any depth of nested directories. +- A question mark (`?`) matches one character. +- Comma-separated lists in curly braces (`{one,two}`) match any option in the list. +- Sets of characters in square brackets (`[abcd]`) match any character in the set. +- A backslash (`\`) escapes special characters. + +Example: + +{% raw %} + +```yaml +- name: "Domain or network segment" + glob: "network/**/{%{facts.networking.domain},%{facts.networking.interfaces.en0.bindings.0.network}}.yaml" +``` + +{% endraw %} + +The `mapped_paths` key must contain three string elements, in the following order: + +- A scope variable that points to a collection of strings. +- The variable name that will be mapped to each element of the collection. +- A template where that variable can be used in interpolation expressions. + +For example, a fact named `$services` contains the array `["a", "b", "c"]`. The following configuration has the same results as if paths had been specified to be `[service/a/common.yaml, service/b/common.yaml, service/c/common.yaml]`. + +```yaml +- name: Example + mapped_paths: [services, tmp, "service/%{tmp}/common.yaml"] +``` + +Related topics: [interpolate][interpolation], [hierarchies][hierarchy]. + +## Configuring a hierarchy level: hiera-eyaml + +Hiera 5 includes a native interface for the Hiera eyaml extension, which keeps data encrypted on disk but lets OpenVox read it during catalog compilation. + +To learn how to create keys and edit encrypted files, see the [Hiera eyaml documentation](https://github.com/voxpupuli/hiera-eyaml). + +Within `hiera.yaml`, the eyaml backend resembles the standard built-in backends, with a few differences: it uses `lookup_key` instead of `data_hash`, and requires an `options` key to locate decryption keys. Note that the eyaml backend can read regular yaml files as well as yaml files with encrypted data. + +> **Important**: To use the eyaml backend, you must have the `hiera-eyaml` gem installed where OpenVox can use it. To enable eyaml on the command line and with `puppet apply`, use `sudo /opt/puppetlabs/puppet/bin/gem install hiera-eyaml`. + +Each eyaml hierarchy level needs the following keys: + +- `name` — A unique name for this level, shown in debug messages and `--explain` output. +- `lookup_key` — Which backend to use. The value must be `eyaml_lookup_key`. Use this instead of the `data_hash` setting. +- `path`, `paths`, `mapped_paths`, `glob`, or `globs` (choose one) — The data files to use for this hierarchy level. These paths are relative to the datadir, they support variable interpolation, and they require a file extension. In this case, you'll usually use `.eyaml`. They work the same way they do for the standard backends. +- `datadir` — The directory where data files are kept. Can be omitted if you set a default. Works the same way it does for the standard backends. +- `options` — A hash of options specific to `hiera-eyaml`, mostly used to configure decryption. For the default encryption method, this hash must have the following keys: + - `pkcs7_private_key` — The location of the PKCS7 private key to use. + - `pkcs7_public_key` — The location of the PKCS7 public key to use. + - If you use an alternate encryption plugin, its docs should specify which options to set. Set an `encrypt_method` option, plus some plugin-specific options to replace the `pkcs7` ones. + - You can use normal strings as keys in this hash; you don't need to use symbols. + +The file path key and the options key both support variable interpolation. + +An example hierarchy level: + +```yaml +hierarchy: + - name: "Per-datacenter secret data (encrypted)" + lookup_key: eyaml_lookup_key + path: "secrets/%{facts.whereami}.eyaml" + options: + pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem + pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem +``` + +Related topics: [Hiera eyaml][eyaml], [variable interpolation][interpolation]. + +## Configuring a hierarchy level: legacy Hiera 3 backends + +> Note: This feature is a temporary measure to let you start using new features while waiting for backend updates. + +If you rely on custom data backends designed for Hiera 3, you can use them in your global hierarchy. They are not supported at the environment or module layers. + +Each legacy hierarchy level needs the following keys: + +- `name` — A unique name for this level, shown in debug messages and `--explain` output. +- `path` or `paths` (choose one) — The data files to use for this hierarchy level. + - For file-based backends, include the file extension, even though you would have omitted it in the v3 hiera.yaml file. + - For non-file backends, don't use a file extension. +- `hiera3_backend` — The legacy backend to use. This is the same name you'd use in the v3 config file's `:backends` key. +- `datadir` — The directory where data files are kept. Set this only if your backend required a `:datadir` setting in its backend-specific options. + - This path is relative to hiera.yaml's directory: if the config file is at `/etc/puppetlabs/code/environments/production/hiera.yaml` and the datadir is set to `data`, the full path to the data directory is `/etc/puppetlabs/code/environments/production/data`. Note that Hiera v3 uses 'hieradata' instead of 'data'. + - In the global layer, you can optionally set the datadir to an absolute path. +- `options` — A hash, with any backend-specific options (other than `datadir`) required by your backend. In the v3 config, this would have been in a top-level key named after the backend. + +You can use normal strings as keys. Hiera converts them to symbols for the backend. + +The following example shows roughly equivalent v3 and v5 hiera.yaml files using legacy backends: + +```yaml +# hiera.yaml v3 +--- +:backends: + - mongodb + - xml + +:mongodb: + :connections: + :dbname: hdata + :collection: config + :host: localhost + +:xml: + :datadir: /some/other/dir + +:hierarchy: + - "%{trusted.certname}" + - "common" + + +# hiera.yaml v5 +--- +version: 5 +hierarchy: + - name: MongoDB + hiera3_backend: mongodb + paths: + - "%{trusted.certname}" + - common + options: + connections: + dbname: hdata + collection: config + host: localhost + + - name: Data in XML + hiera3_backend: xml + datadir: /some/other/dir + paths: + - "%{trusted.certname}.xml" + - common.xml +``` + +## Configuring a hierarchy level: general format + +Hiera supports custom backends. + +See the backend's documentation for configuring hierarchy levels. + +Each hierarchy level is represented by a hash which needs the following keys: + +- `name` — A unique name for this level, shown in debug messages and `--explain` output. +- A backend key, which must be one of: + - `data_hash` + - `lookup_key` + - `data_dig` - a more specialized form of `lookup_key`, suitable when the backend is for a database. `data_dig` resolves dot separated keys, whereas `lookup_key` does not. + - `hiera3_backend` (global layer only) +- A path or URI key - only if required by the backend. These keys support variable interpolation. The following path/URI keys are available: + - `path` + - `paths` + - `mapped_paths` + - `glob` + - `globs` + - `uri` + - `uris` - these paths or URIs work the same way they do for the built-in backends. Hiera handles the work of locating files, so any backend that supports `path` automatically supports `paths`, `glob`, and `globs`. + `uri` (string) and `uris` (array) can represent any kind of data source. Hiera does not ensure URIs are resolvable before calling the backend, and does not need to understand any given URI schema. A backend can omit the path/URI key, and rely wholly on the `options` key to locate its data. +- `datadir` — The directory where data files are kept: the path is relative to hiera.yaml's directory. Only required if the backend uses the `path`, `paths`, `glob`, and `globs` keys, and can be omitted if you set a default. +- `options` — A hash of extra options for the backend; for example, database credentials or the location of a decryption key. All values in the `options` hash support variable interpolation. + +Whichever key you use, the value must be the name of a function that implements the backend API. Note that the choice here is made by the implementer of the particular backend, not the user. + +Related topics: [custom OpenVox function][custom puppet function], [custom backends][backends], [variable interpolation][interpolation]. diff --git a/docs/_openvox_9x/hiera_custom_backends.md b/docs/_openvox_9x/hiera_custom_backends.md new file mode 100644 index 000000000..af69e9c09 --- /dev/null +++ b/docs/_openvox_9x/hiera_custom_backends.md @@ -0,0 +1,390 @@ +--- +layout: default +title: "Writing new data backends" +--- + +[yaml_data]: https://github.com/OpenVoxProject/openvox/blob/main/lib/puppet/functions/yaml_data.rb +[json_data]: https://github.com/OpenVoxProject/openvox/blob/main/lib/puppet/functions/json_data.rb +[hocon_data]: https://github.com/OpenVoxProject/openvox/blob/main/lib/puppet/functions/hocon_data.rb +[merging]: ./hiera_merging.html +[interpolation]: ./lang_data_string.html#interpolation +[automatic]: ./hiera_automatic.html#access-hash-and-array-elements-using-a-keysubkey-notation +[eyaml_lookup_key]: https://github.com/voxpupuli/hiera-eyaml +[puppet_functions]: ./lang_write_functions_in_puppet.html +[ruby_functions]: ./functions_ruby_overview.html +[hiera_yaml]: ./hiera_config_yaml_5.html +[struct]: ./lang_data_abstract.html#struct +[functions]: ./lang_functions.html + +You can extend Hiera to look up values in data stores, for example, a PostgreSQL database table, a custom web app, or a new kind of structured data file. + +To teach Hiera how to talk to other data sources, write a custom backend. + +> **Important**: Writing a custom backend is an advanced topic. Before proceeding, make sure you really need it. +> It is also worth asking the community whether there is one you can re-use, rather than starting from scratch. + +## Custom backends overview + +A backend is a custom Puppet function that accepts a particular set of arguments and whose return value obeys a particular format. The function can do whatever is necessary to locate its data. + +A backend function uses the modern Ruby functions API or the Puppet language. +This means you can use different versions of a Hiera backend in different environments, and you can distribute Hiera backends in OpenVox modules. + +Different types of data have different performance characteristics. +To make sure Hiera performs well with every type of data source, it supports three types of backends: `data_hash`, `lookup_key` and `data_dig`. + +### data_hash + +For data sources where it's inexpensive, performance-wise, to read the entire contents at once, like simple files on disk. Use the `data_hash` backend type if: + +* The cache is alive for the duration of one compilation. +* The data is small. +* The data can be retrieved all at once. +* Most of the data gets used. +* The data is static. + +### lookup_key + +For data sources where looking up a key is relatively expensive, performance-wise, like an HTTPS API. Use the `lookup_key` backend type if: + +* The data set is big, but only a small portion is used. +* The result can vary during the compilation. + +The `hiera-eyaml` backend is a `lookup_key` function, because decryption tends to affect performance; as a given node uses only a subset of the available secrets, it makes sense to decrypt only on-demand. + +### data_dig + +For data sources that can access arbitrary elements of hash or array values before passing anything back to Hiera, like a database. + +#### The `RichDataKey` and `RichData` types + +To simplify backend function signatures, you can use two extra data type aliases: `RichDataKey` and `RichData`. +These are only available to backend functions called by Hiera; normal functions and Puppet code cannot use them. + +Related topics: [custom Puppet function][puppet_functions], [the modern Ruby functions API][ruby_functions]. + +## data_hash backends + +A `data_hash` backend function reads an entire data source at once, and returns its contents as a hash. + +The built-in YAML, JSON, and HOCON backends are all `data_hash` functions. You can view their source on GitHub: + +* [`yaml_data.rb`][yaml_data] +* [`json_data.rb`][json_data] +* [`hocon_data.rb`][hocon_data] + +### data_hash arguments + +Hiera calls a `data_hash` function with two arguments: + +* A hash of options. The options hash will contain a `path` when the entry in `hiera.yaml` is using `path`/`paths`, `glob`/`globs`, or `mapped_paths`, + and the backend will receive one call per path to an existing file. + When the entry in `hiera.yaml` is using `uri`/`uris`, the options hash will have a `uri` key, and the backend function is called once per given URI. + When `uri`/`uris` are used, Hiera does not perform an existence check. It is up to the function to type the options parameter as wanted. +* A `Puppet::LookupContext` object. + +### data_hash return type + +The function must either call the context object's `not_found` method, or return a hash of lookup keys and their associated values. The hash can be empty. + +Puppet language example signature: + +```puppet +function mymodule::hiera_backend( + Hash $options, + Puppet::LookupContext $context, +) +``` + +Ruby example signature: + +```ruby +dispatch :hiera_backend do + param 'Hash', :options + param 'Puppet::LookupContext', :context +end +``` + +The returned hash can include the `lookup_options` key to configure merge behavior for other keys. See [Configuring merge behavior in Hiera data][merging] for more information. +Values in the returned hash can include Hiera interpolation tokens like `%{variable}` or `%{lookup('key')}`; Hiera interpolates values as needed. +This is a significant difference between `data_hash` and the other two backend types; `lookup_key` and `data_dig` functions have to explicitly handle interpolation. + +## lookup_key backends + +A `lookup_key` backend function looks up a single key and returns its value. + +The built-in `hiera_eyaml` backend is a `lookup_key` function. You can view its source at the [hiera-eyaml GitHub repository][eyaml_lookup_key]. + +### lookup_key arguments + +Hiera calls a `lookup_key` function with three arguments: + +1. A key to look up. +2. A hash of options. +3. A `Puppet::LookupContext` object. + +### lookup_key return type + +The function must either call the context object's `not_found` method, or return a value for the requested key. It may return `undef` as a value. + +Puppet language example signature: + +```puppet +function mymodule::hiera_backend( + Variant[String, Numeric] $key, + Hash $options, + Puppet::LookupContext $context, +) +``` + +Ruby example signature: + +```ruby +dispatch :hiera_backend do + param 'Variant[String, Numeric]', :key + param 'Hash', :options + param 'Puppet::LookupContext', :context +end +``` + +A `lookup_key` function can return a hash for the `lookup_options` key to configure merge behavior for other keys. See [Configuring merge behavior in Hiera data][merging] for more information. +To support Hiera interpolation tokens, for example `%{variable}` or `%{lookup('key')}` in your data, call `context.interpolate` on your values before returning them. + +Related topics: [interpolation][interpolation]. + +## data_dig backends + +A `data_dig` backend function is similar to a `lookup_key` function, but instead of looking up a single key, it looks up a single sequence of keys and subkeys. + +Hiera lets you look up individual members of hash and array values using `key.subkey` notation. Use `data_dig` types in cases where: + +* Lookups are relatively expensive. +* The data source knows how to extract elements from hash and array values. +* Users are likely to pass `key.subkey` requests to the `lookup` function to access subsets of large data structures. + +### data_dig arguments + +Hiera calls a `data_dig` function with three arguments: + +1. An array of lookup key segments, made by splitting the requested lookup key on the dot (`.`) subkey separator. For example, a lookup for `users.dbadmin.uid` results in `['users', 'dbadmin', 'uid']`. + Positive base-10 integer subkeys (for accessing array members) are converted to Integer objects, but other number subkeys remain as strings. +2. A hash of options. +3. A `Puppet::LookupContext` object. + +### data_dig return type + +The function must either call the context object's `not_found` method, or return a value for the requested sequence of key segments. +Note that returning `undef` (`nil` in Ruby) means that the key was found but that the value for that key was specified to be `undef`. + +Puppet language example signature: + +```puppet +function mymodule::hiera_backend( + Array[Variant[String, Numeric]] $segments, + Hash $options, + Puppet::LookupContext $context, +) +``` + +Ruby example signature: + +```ruby +dispatch :hiera_backend do + param 'Array[Variant[String, Numeric]]', :segments + param 'Hash', :options + param 'Puppet::LookupContext', :context +end +``` + +A `data_dig` function can return a hash for the `lookup_options` key to configure merge behavior for other keys. + +To support Hiera interpolation tokens like `%{variable}` or `%{lookup('key')}` in your data, call `context.interpolate` on your values before returning them. + +Related topics: [key.subkey notation][automatic], [Configuring merge behavior in Hiera data][merging]. + +## Hiera calling conventions for backend functions + +Hiera uses the following conventions when calling backend functions: + +* Hiera calls `data_hash` once per data source. +* Hiera calls `lookup_key` functions once per data source for every unique key lookup. +* Hiera calls `data_dig` functions once per data source for every unique sequence of key segments. + +However, a given hierarchy level can refer to multiple data sources with the `path`, `uri`, and `glob` settings. Hiera handles each hierarchy level as follows: + +* If the `path` or `glob` settings are used, Hiera determines which files exist and calls the function once for each. If no files were found, the function will not be called. +* If the `uri` settings are used, Hiera calls the function once per URI. +* If none of those settings are used, Hiera calls the function once. + +Hiera can call a function again for a given data source if the inputs change. For example, if `hiera.yaml` interpolates a local variable in a file path, +Hiera calls the function again for scopes where that variable has a different value. +This has a significant performance impact, so you should interpolate only facts, trusted facts, and server facts in the hierarchy. + +## The options hash + +Hierarchy levels are configured in the `hiera.yaml` file. When calling a backend function, Hiera passes a modified version of that configuration as a hash. + +The options hash may contain the following keys (depending on whether `path`/`glob`/`uri`/`mapped_paths` have been set): + +* `path` — The absolute path to a file on disk. It is present only if `path`, `paths`, `glob`, `globs`, or `mapped_paths` is present in the hierarchy. + Hiera will never call the function unless the file is present. +* `uri` — A URI that your function can use to locate a data source. It is present only if `uri` or `uris` is present in the hierarchy. Hiera does not verify the URI before passing it to the function. +* Every key from the hierarchy level's `options` setting. List any options your backend requires or accepts. The `path` and `uri` keys are reserved. + +> Note: If your backend uses data files, use the context object's `cached_file_data` method to read them. + +For example, the following hierarchy level in `hiera.yaml` results in several different options hashes, depending on the current node's facts and whether the files exist: + +```yaml +- name: "Secret data: per-node, per-datacenter, common" + lookup_key: eyaml_lookup_key + datadir: data + paths: + - "secrets/nodes/%{trusted.certname}.eyaml" + - "secrets/location/%{facts.whereami}.eyaml" + - "common.eyaml" + options: + pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem + pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem +``` + +The various hashes would all be similar to this: + +```ruby +{ + 'path' => '/etc/puppetlabs/code/environments/production/data/secrets/nodes/web01.example.com.eyaml', + 'pkcs7_private_key' => '/etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem', + 'pkcs7_public_key' => '/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem' +} +``` + +In your function's signature, you can validate the options hash by using the `Struct` data type to restrict its contents. +In particular, you can disable all of the `path` and `glob` settings for your backend by disallowing the `path` key in the options hash. + +Related topics: [Configuring merge behavior in Hiera data][merging], [Hiera interpolation tokens][interpolation], [hiera.yaml][hiera_yaml], [the Struct data type][struct]. + +## The Puppet::LookupContext object and methods + +To support caching and other backend needs, Hiera provides a `Puppet::LookupContext` object. + +In Ruby functions, the context object is a normal Ruby object of class `Puppet::LookupContext`, and you can call methods with standard Ruby syntax, for example `context.not_found`. + +In Puppet language functions, the context object appears as the special data type `Puppet::LookupContext` with methods attached. +You can call the context's methods using Puppet's chained function call syntax, for example, `$context.not_found`. +For methods that take a block, use Puppet's lambda syntax (parameters outside block) instead of Ruby's block syntax (parameters inside block). + +### not_found() + +Tells Hiera to halt this lookup and move on to the next data source. Call this method when your function cannot find a matching key or a given lookup. This method returns no value. + +For `data_hash` backends, return an empty hash. The empty hash will result in `not_found` and will prevent further calls to the provider. +For `lookup_key` and `data_dig` backends, use `not_found` when a requested key is not present in the data source or the data source does not exist. +Do not return `undef` or `nil` for missing keys, as these are legal values that can be set in data. + +### interpolate(value) + +Returns the provided value, but with any Hiera interpolation tokens (`%{variable}` or `%{lookup('key')}`) replaced by their value. +This lets you opt-in to allowing Hiera-style interpolation in your backend's data sources. It works recursively on arrays and hashes. Hashes can interpolate into both keys and values. + +In `data_hash` backends, support for interpolation is built in, and you do not need to call this method. + +In `lookup_key` and `data_dig` backends, call this method if you want to support interpolation. + +### environment_name() + +Returns the name of the environment, regardless of layer. + +### module_name() + +Returns the name of the module whose `hiera.yaml` called the function. Returns `undef` (in Puppet) or `nil` (in Ruby) if the function was called by the global or environment layer. + +### cache(key, value) + +Caches a value in a per-data-source private cache. It also returns the cached value. + +On future lookups in this data source, you can retrieve values by calling `cached_value(key)`. Cached values are immutable, but you can replace the value for an existing key. +Cache keys can be anything valid as a key for a Ruby hash, including `nil`. + +For example, on its first invocation for a given YAML file, the built-in `eyaml_lookup_key` backend reads the whole file and caches it, then decrypts only the specific value that was requested. +On subsequent lookups into that file, it gets the encrypted value from the cache instead of reading the file from disk again. + +The cache is useful for storing session keys or connection objects for backends that access a network service. + +Each `Puppet::LookupContext` cache lasts for the duration of the current catalog compilation. A node can't access values cached for a previous node. + +Hiera creates a separate cache for each combination of inputs for a function call. Each hierarchy level has its own cache, and hierarchy levels that use multiple paths have a separate cache for each path. + +### cache_all(hash) + +Caches all the key-value pairs from a given hash. Returns `undef` (in Puppet) or `nil` (in Ruby). + +### cached_value(key) + +Returns a previously cached value from the per-data-source private cache. Returns `undef` or `nil` if no value with this name has been cached. + +### cache_has_key(key) + +Checks whether the cache has a value for a given key yet. Returns `true` or `false`. + +### cached_entries() + +Returns everything in the per-data-source cache as an iterable object. The returned object is not a hash. +If you want a hash, use `Hash($context.all_cached())` in the Puppet language or `Hash[context.all_cached()]` in Ruby. + +### cached_file_data(path) + +Puppet syntax: + +```puppet +cached_file_data(path) |content| { ... } +``` + +Ruby syntax: + +```ruby +cached_file_data(path) {|content| ...} +``` + +For best performance, use this method to read files in Hiera backends. + +Returns the content of the specified file as a string. If an optional block is provided, it passes the content to the block and returns the block's return value. +For example, the built-in JSON backend uses a block to parse JSON and return a hash: + +```ruby +context.cached_file_data(path) do |content| + begin + JSON.parse(content) + rescue JSON::ParserError => ex + raise Puppet::DataBinding::LookupError, "Unable to parse (#{path}): #{ex.message}" + end +end +``` + +On repeated access to a given file, Hiera checks whether the file has changed on disk. If it hasn't, Hiera uses cached data instead of reading and parsing the file again. + +This method does not use the same per-data-source caches as `cache(key, value)` and similar methods. +It uses a separate cache that lasts across multiple catalog compilations, tied to the server's environment cache. + +Since the cache can outlive a given node's catalog compilation, do not do any node-specific pre-processing (like calling `context.interpolate`) in this method's block. + +### explain() { 'message' } + +Puppet syntax: + +```puppet +explain() || { 'message' } +``` + +Ruby syntax: + +```ruby +explain() { 'message' } +``` + +In both Puppet and Ruby, the provided block must take zero arguments. + +Adds a message which appears in debug messages or when using `puppet lookup --explain`. The block provided to this function must return a string. + +The `explain` method is useful for complex lookups where a function tries several different things before arriving at the value. Hiera never executes the explain block unless explain is enabled. + +Related topics: [Ruby functions][ruby_functions], [Puppet language functions][puppet_functions], [chained function call syntax][functions]. diff --git a/docs/_openvox_9x/hiera_demo.md b/docs/_openvox_9x/hiera_demo.md new file mode 100644 index 000000000..bcaca739e --- /dev/null +++ b/docs/_openvox_9x/hiera_demo.md @@ -0,0 +1,158 @@ +--- +layout: default +title: "Interactive Hiera Demo" +--- + + +This page renders Hiera configuration and data in a graphical and interactive table. +It will help you understand how data is resolved and how changing parameters alters the resolution. + +## The `hiera.yaml` config file + +This Hiera config file is located in the root of a Puppet environment `hiera.yaml`. +It describes a hierarchy relative to that environment's `datadir`. +Path names are resolved using *facts* and other variables. + + ```yaml + # /hiera.yaml + --- + version: 5 + + hierarchy: + - name: "Per-node data" # Human-readable name. + path: "nodes/%{trusted.certname}.yaml" # File path, relative to datadir. + # ^^^ IMPORTANT: include the file extension! + + - name: "Per-OS defaults" + path: "os/%{facts.os.family}.yaml" + + - name: "Common data" + path: "common.yaml" + ``` + +This file describes a hierarchy of directories and yaml files matching the resolved paths from the hierarchy. + +```console +[/etc/puppetlabs/code/environments/production]$ tree data +├── common.yaml +├── nodes +│   ├── node1.example.com.yaml +│   └── node2.example.com.yaml +└── os + ├── debian.yaml + ├── redhat.yaml + └── Windows.yaml +``` + +## Resolving data + +Each of the files in the hierarchy may contain keys and values. +When Hiera resolves keys, it will inspect these files in the order described. +The first value found wins. + +### [Demo] Understand how changes in a single fact change the resolution + +The table below represents the values that will be resolved when a catalog is compiled for `node1.example.com` on various platforms. +The layers of the hierarchy are represented by rows, and each item being resolved is a column. +Resolve each key by starting at the header and reading straight down the column until you find a value. +Use the tabs to resolve it as if `node1` were various supported platforms. +Notice the resolved path of each layer for the different platforms. + +{% tabs platform %} + + {% tab platform RedHat %} + +| Layer | `webserver` | `threads` | `backup` | `cluster` | +|-------------------------------------|-------------|-----------|------------|--------------| +| `data/nodes/node1.example.com.yaml` | | | | >>`denver`<< | +| `data/os/redhat.yaml` | >>`httpd`<< | >>6<< | | | +| `common.yaml` | `apache` | 4 | >>`true`<< | `laramie` | +| Resolved Values: | `httpd` | 6 | `true` | `denver` | +{:class="resolution"} + + {% endtab %} + + {% tab platform Ubuntu %} + +| Layer | `webserver` | `threads` | `backup` | `cluster` | +|-------------------------------------|---------------|-----------|------------|--------------| +| `data/nodes/node1.example.com.yaml` | | | | >>`denver`<< | +| `data/os/debian.yaml` | >>`apache2`<< | | | | +| `common.yaml` | `apache` | >>4<< | >>`true`<< | `laramie` | +| Resolved Values: | `apache2` | 4 | `true` | `denver` | +{:class="resolution"} + + {% endtab %} + + {% tab platform Debian %} + +| Layer | `webserver` | `threads` | `backup` | `cluster` | +|-------------------------------------|---------------|-----------|------------|--------------| +| `data/nodes/node1.example.com.yaml` | | | | >>`denver`<< | +| `data/os/debian.yaml` | >>`apache2`<< | | | | +| `common.yaml` | `apache` | >>4<< | >>`true`<< | `laramie` | +| Resolved Values: | `apache2` | 4 | `true` | `denver` | +{:class="resolution"} + + {% endtab %} + + {% tab platform Windows %} + +| Layer | `webserver` | `threads` | `backup` | `cluster` | +|-------------------------------------|--------------------|-----------|-------------|--------------| +| `data/nodes/node1.example.com.yaml` | | | | >>`denver`<< | +| `data/os/Windows.yaml` | >>`iis`<< | >>1<< | >>`false`<< | | +| `common.yaml` | `apache` | 4 | `true` | `laramie` | +| Resolved Values: | `iis` | 1 | `false` | `denver` | +{:class="resolution"} + + {% endtab %} + +{% endtabs %} + +Notice that the only layer changing and affecting the resolved values is `os/%{facts.os.family}.yaml`. +This is because the only *fact* we are changing is `os.family`. +Also note that Ubuntu and Debian resolve exactly the same; because they're both `debian` family, they both resolve to the exact same file. +{: .tip } + + +### [Demo] Understand how simultaneous changes in multiple facts change the resolution + +This table should be read just like the one above. +It represents the values resolved when compiling the same catalog for two different nodes. +`node1.example.com` is a RedHat 9 machine and `node2.example.com` is a Debian 12 machine. + +As you might expect, `node1.example.com` resolves exactly as it did above, but `node2.example.com` has a node specific overlay overriding some values. +This means that multiple layers are being resolved to a different path as you switch tabs. + +{% tabs nodes %} + +{% tab nodes node1.example.com %} + +| Layer | `webserver` | `threads` | `backup` | `cluster` | +|-------------------------------------|-------------|-----------|------------|--------------| +| `data/nodes/node1.example.com.yaml` | | | | >>`denver`<< | +| `data/os/redhat.yaml` | >>`httpd`<< | >>6<< | | | +| `common.yaml` | `apache` | 4 | >>`true`<< | `laramie` | +| Resolved Values: | `httpd` | 6 | `true` | `denver` | +{:class="resolution"} + +{% endtab %} + +{% tab nodes node2.example.com %} + +| Layer | `webserver` | `threads` | `backup` | `cluster` | +|-------------------------------------|---------------|-----------|------------|---------------| +| `data/nodes/node2.example.com.yaml` | >>`nginx`<< | >>8<< | | | +| `data/os/debian.yaml` | `apache2` | | | | +| `common.yaml` | `apache` | 4 | >>`true`<< | >>`laramie`<< | +| Resolved Values: | `nginx` | 8 | `true` | `laramie` | +{:class="resolution"} + +{% endtab %} + +{% endtabs %} + +Notice that only the most specific value for each key is resolved. +Hiera has other more complex ways of defining layers and hierarchies, but they all follow this pattern. +{: .tip } diff --git a/docs/_openvox_9x/hiera_intro.md b/docs/_openvox_9x/hiera_intro.md new file mode 100644 index 000000000..3134ea220 --- /dev/null +++ b/docs/_openvox_9x/hiera_intro.md @@ -0,0 +1,160 @@ +--- +layout: default +title: "Hiera" +--- + +[auto_lookup]: ./hiera_automatic.html#class-parameters +[codedir]: ./dirs_codedir.html +[confdir]: ./dirs_confdir.html +[v3]: ./hiera_config_yaml_3.html +[v4]: ./hiera_config_yaml_4.html +[v5]: ./hiera_config_yaml_5.html + +Hiera is a key/value lookup used for separating data from OpenVox code. + +## About Hiera + +Hiera is OpenVox's built-in key-value configuration data lookup system. + +OpenVox's strength is in reusable code. Code that serves many needs must be configurable: put site-specific information in external configuration data files, rather than in the code itself. + +OpenVox uses Hiera to do two things: + +* Store the configuration data in key-value pairs +* Look up what data a particular module needs for a given node during catalog compilation. + +This is done via: + +* Automatic Parameter Lookup for classes included in the catalog +* Explicit lookup calls + +Hiera's hierarchical lookups follow a "defaults, with overrides" pattern, meaning you specify common data once, and override it in situations where the default won't work. Hiera uses OpenVox's facts to specify data sources, so you can structure your overrides to suit your infrastructure. While using facts for this purpose is common, data-sources may well be defined without the use of facts. + +Hiera 5 comes with support for JSON, YAML, and EYAML files. + +Related topics: [Automatic Parameter Lookup][auto_lookup] + +## Hiera hierarchies + +Hiera looks up data by following a hierarchy - an ordered list of data sources. + +Hierarchies are configured in a `hiera.yaml` configuration file. Each level of the hierarchy tells Hiera how to access some kind of data source. + +### Hierarchies interpolate variables + +Most levels of a hierarchy interpolate variables into their configuration: + +`path: "os/%{facts.os.family}.yaml"` + +* The percent-and-braces `%{variable}` syntax is a Hiera interpolation token. It is similar to the OpenVox language's `${expression}` interpolation tokens. Wherever you use an interpolation token, Hiera determines the variable's value and inserts it into the hierarchy. +* `facts.os.family` uses the Hiera special `key.subkey` notation for accessing elements of hashes and arrays. It is equivalent to `$facts['os']['family']` in the OpenVox language but the 'dot' notation will produce an empty string instead of raising an error if parts of the data is missing. Make sure that an empty interpolation does not end up matching an unintended path. +* You can only interpolate values into certain parts of the config file. For more info, see the hiera.yaml format reference. +* With node-specific variables, each node gets a customized set of paths to data. The hierarchy is always the same. + +### Hiera searches the hierarchy in order + +Once Hiera replaces the variables to make a list of concrete data sources, it checks those data sources in the order they were written. +Generally, if a data source doesn't exist, or doesn't specify a value for the current key, Hiera skips it and moves on to the next source, until it finds one that exists - then it uses it. Note that this is the default merge strategy, but does not always apply, for example, Hiera can use data from all data sources and merge the result. + +Earlier data sources have priority over later ones. In the example above, the node-specific data has the highest priority, and can override data from any other level. Business group data is separated into local and global sources, with the local one overriding the global one. Common data used by all nodes always goes last. + +That's how Hiera's "defaults, with overrides" approach to data works - you specify common data at lower levels of the hierarchy, and override it at higher levels for groups of nodes with special needs. + +### Layered hierarchies + +Hiera uses layers of data with a `hiera.yaml` for each layer. + +Each layer can configure its own independent hierarchy. Before a lookup, Hiera combines them into a single super-hierarchy: global → environment → module. + +> Note: There is a fourth layer - `default_hierarchy` - that can be used in a module's `hiera.yaml`. It only comes into effect when there is no data for a key in any of the other regular hierarchies + +Assume the example above is an environment hierarchy (in the production environment). If we also had the following global hierarchy: + +```yaml +version: 5 +hierarchy: + - name: "Data exported from our old self-service config tool" + path: "selfserve/%{trusted.certname}.json" + data_hash: json_data + datadir: data +``` + +And the NTP module had the following hierarchy for default data: + +```yaml +version: 5 +hierarchy: + - name: "OS values" + path: "os/%{facts.os.name}.yaml" + - name: "Common values" + path: "common.yaml" +defaults: + data_hash: yaml_data + datadir: data +``` + +Then in a lookup for the `ntp::servers` key, `thrush.example.com` would use the following combined hierarchy: + +* `/data/selfserve/thrush.example.com.json` +* `/environments/production/data/nodes/thrush.example.com.yaml` +* `/environments/production/data/location/belfast/ops.yaml` +* `/environments/production/data/groups/ops.yaml` +* `/environments/production/data/os/Debian.yaml` +* `/environments/production/data/common.yaml` +* `/environments/production/modules/ntp/data/os/Ubuntu.yaml` +* `/environments/production/modules/ntp/data/common.yaml` + +The combined hierarchy works the same way as a layer hierarchy. Hiera skips empty data sources, and either returns the first found value or merges all found values. + +> Note: By default, datadir refers to the directory named 'data' next to the `hiera.yaml`. + +### Tips for making a good hierarchy + +* Make a short hierarchy. Data files will be easier to work with. +* Use the roles and profiles method to manage less data in Hiera. Sorting hundreds of class parameters is easier than sorting thousands. +* If the built-in facts don't provide an easy way to represent differences in your infrastructure, make custom facts. For example, create a custom datacenter fact that is based on information particular to your network layout so that each datacenter is uniquely identifiable. +* Give each environment -- production, test, development -- its own hierarchy. + +Related topics: [codedir][codedir], [confdir][confdir], [hiera.yaml][v5] + +## Hiera's three config layers + +Hiera uses three independent layers of configuration. Each layer has its own hierarchy, and they're linked into one super-hierarchy before doing a lookup. + +The three layers are searched in the following order: global → environment → module. Hiera searches every data source in the global layer's hierarchy before checking any source in the environment layer. + +### The global layer + +* The configuration file for the global layer is located, by default, in `$confdir/hiera.yaml`. You can change the location by changing the `hiera_config` setting in `puppet.conf`. +* Hiera has one global hierarchy. Since it goes before the environment layer, it's useful for temporary overrides, for example, when your ops team needs to bypass its normal change processes. +* The global layer is the only place where legacy Hiera 3 backends can be used - it's an important piece of the transition period when you migrate your backends to support Hiera 5. +* The global layer supports the following config formats: hiera.yaml v5, hiera.yaml v3 (deprecated). +* Other than the above use cases, try to avoid the global layer. All normal data should be specified in the environment layer. + +### The environment layer + +* The configuration file for the environment layer is located, by default, at `/hiera.yaml`. +* The environment layer is where most of your Hiera data hierarchy definition happens. +* Every OpenVox environment has its own hierarchy configuration, which applies to nodes in that environment. +* Supported config formats: hiera.yaml v5, hiera.yaml v3 (deprecated). + +### The module layer + +* The configuration file for a module layer is located, by default, in a module's `/hiera.yaml`. +* The module layer sets default values and merge behavior for a module's class parameters. It is a convenient alternative to the `params.pp` pattern. + * Note that to get the exact same behaviour as params.pp, the default_hierarchy should be used, as those bindings are excluded from merges. When placed in the regular hierarchy in the module's hierarchy the bindings will be merged when a merge lookup is performed. +* The module layer comes last in Hiera's lookup order, so environment data set by a user overrides the default data set by the module's author. +* Every module can have its own hierarchy configuration. You can only bind data for keys in the module's namespace. For example: + +| Lookup key | Relevant module hierarchy | +| ----------------- |:-------------------------:| +| `ntp::servers` | `ntp` | +| `jenkins::port` | `jenkins` | +| `secure_server` | `(none)` | + +* Hiera uses the `ntp` module's hierarchy when looking up `ntp::servers`, but uses the `jenkins` module's hierarchy when looking up `jenkins::port`. Hiera never checks the `ntp` module for a key beginning with `jenkins::`. +* When you use the lookup function for keys that don't have a namespace (for example, `secure_server`), the module layer is not consulted. + +The three-layer system means that each environment has its own hierarchy, and so do modules. You can make hierarchy changes on an environment-by-environment basis. Module data is also customizable. + +Related topics: [version 3][v3], [version 4][v4], [version 5][v5] diff --git a/docs/_openvox_9x/hiera_merging.md b/docs/_openvox_9x/hiera_merging.md new file mode 100644 index 000000000..96f3e7768 --- /dev/null +++ b/docs/_openvox_9x/hiera_merging.md @@ -0,0 +1,395 @@ +--- +layout: default +title: "Creating and editing data" +--- + +[lookup_command]: ./man/lookup.html +[automatic]: ./hiera_automatic.html +[data types]: ./lang_data.html +[interpolation_puppet]: ./lang_data_string.html#interpolation +[facts_hash]: ./lang_facts_and_builtin_vars.html#the-factsfact_name-hash +[trusted]: ./lang_facts_and_builtin_vars.html#trusted-facts +[server_facts]: ./lang_facts_and_builtin_vars.html#server_facts-variable +[environment]: ./environments_about.html + +## Setting the merge behavior for a lookup + +When you look up a key in Hiera, it is common for multiple data sources to have different values for it. +By default, Hiera returns the first value it finds, but it can also continue searching and merge all the found values together. + +You can set the merge behavior for a lookup in two ways: + +* At lookup time. This works with the `lookup` function, but does not support automatic class parameter lookup. +* In Hiera data, with the `lookup_options` key. This works for both manual and automatic lookups. It also lets module authors set default behavior that users can override. + +With both of these methods, specify a merge behavior as either a string (for example, `'first'`) or a hash (for example `{'strategy' => 'first'}`). +The hash syntax is useful for `deep` merges where extra options are available, but it also works with the other merge types. + +Related topics: [lookup][lookup_command], [automatic class parameter lookup][automatic]. + +## Merge behaviors + +There are four merge behaviors to choose from: `first`, `unique`, `hash`, and `deep`. +When specifying a merge behavior, use one of the following identifiers: + +* `'first'`, `{'strategy' => 'first'}`, or nothing. +* `'unique'` or `{'strategy' => 'unique'}`. +* `'hash'` or `{'strategy' => 'hash'}`. +* `'deep'` or `{'strategy' => 'deep',