Skip to main content

Changelog

Unreleased​

Installer resilience​

  • Non-interactive installs no longer abort when registry credentials are absent — nf-quickstart -y previously aborted in installers/pull-secret.sh when neither REGISTRY_USERNAME/REGISTRY_PASSWORD were set in the environment nor a ./netfoundry-pull-secret.yml was present on disk: the script called read -p with no TTY, returned immediately, and set -e then aborted the whole install. The script now detects this case, emits a yellow WARNING (explaining that the pull secret and Ziti Console Enterprise will be skipped, that operators can set the env vars or drop a netfoundry-pull-secret.yml and re-run nf-quickstart later, and that the offline/air-gapped path does not need the secret at install time because images are pre-loaded into containerd by install-images-offline.sh), and sets PULL_SECRET_SKIPPED=true. installers/ziti-console-enterprise.sh now sources pull-secret.sh instead of executing it as a subprocess so the marker propagates, and skips the ZCE helm install when set. quickstart.sh skips the ziti-console-enterprise checkpoint and omits the ZCE re-apply line from INSTALL-NOTES.txt when ZCE was skipped, so a future re-run with credentials picks up the install cleanly. Interactive runs that enter empty credentials at the prompt now follow the same soft-skip path instead of aborting.

Support stack reliability​

  • Eliminate ECK field-manager conflict on support chart upgrades — Fixes a server-side-apply conflict between Helm and the ECK operator over .spec.nodeSets / .spec.podTemplate on the in-chart Elasticsearch and Kibana CRs, which previously left the support Helm release in a failed state even when the cluster was healthy. The ZMP (Ziti Metrics Processor) controller-address enrichment step in both nf-quickstart and the offline quickstart-offline.sh, nf-upgrade --support, and the zLAN feature-enablement helper now pass --force-conflicts to helm upgrade. Operators running these paths have declared intent to win field-manager ownership for fields Helm declares; fields the operator owns but Helm does not declare (sidecars, monitoring config, etc.) are still left untouched. The advisory helm upgrade commands printed in INSTALL-NOTES.txt were updated to include the same flag so the documented re-apply step works on an existing release. (Iterated once during development: an earlier attempt replaced the enrichment-site helm upgrade with a targeted kubectl patch on deploy/ziti-metrics-processor to sidestep the conflict, but that broke ZMP startup. The chart's zmp-create-ziti-admin pre-upgrade Job is template-gated on zmp.zitiHost != "localhost:1280", so it never rendered at install time when the placeholder was still in values; the original helm upgrade at the enrichment site re-rendered the chart after the sed and the Job then fired, creating the zmp-ziti-credentials secret ZMP consumes. Skipping the re-render left the secret missing and ZMP stuck in CreateContainerConfigError. Restoring the helm upgrade plus --force-conflicts keeps that side effect intact while still resolving the ECK conflict.)
  • Offline installer: ZMP enrichment phase added — The air-gapped quickstart-offline.sh was missing the ZMP enrichment phase entirely, so every offline install completed with ziti-metrics-processor at replicas: 0 and ZITI_HOST=localhost:1280 — Elasticsearch, Kibana, Logstash, and RabbitMQ were provisioned but no Ziti events were ever processed. The offline quickstart now mirrors nf-quickstart: after the support stack and controller are installed, it replicates ziti-controller-admin-secret from the ziti namespace into support, updates support-values.yml (localhost:1280 → ${CTRL_ADDR}:${CLIENT_PORT}, replicas: 0 #zmp → replicas: 1), and runs helm upgrade -n support --install support ./helm-charts/support/ --values support-values.yml --wait --timeout 180s --force-conflicts to re-render the chart. Re-rendering fires the zmp-create-ziti-admin pre-upgrade hook (which only renders once zmp.zitiHost is no longer localhost:1280), which creates the zmp-ziti-credentials secret ZMP consumes at startup. The block is idempotent via ziti-admin-replicated and zmp-enabled checkpoints, so re-running on an existing Siemens-style box (post-install, ZMP at replicas: 0) hits the enrichment and brings ZMP up.
  • Offline bundle now ships jq — Added jq to required_packages.txt so air-gapped installs have it available at runtime. Multiple installer code paths use jq (the new ZMP secret-replication block, the zLAN feature-enable script, the Ziti Console Enterprise helper, utilities/cluster.sh, utilities/create_support_bundle.sh, and the v2→v3 controller upgrade hook). Siemens deployments worked previously only because the SIMATIC LinuxONE base image pre-installs jq; vanilla Debian/Ubuntu air-gapped customers would have hit jq: command not found on the first of those paths.

Validation​

  • Preflight: ASCII fallback for non-TTY stdout, fix RAM threshold for 16 GB hosts — Preflight output now switches to plain ASCII markers ([ OK ] [FAIL] [WARN] [SKIP]) with no ANSI escapes when stdout is not a terminal (or NO_COLOR is set, or TERM=dumb), keeping JSON-RPC and log-file capture clean. The RAM check now compares against MiB-based thresholds with tolerance for the kernel reservation gap, so a real 16 GB host (which reports ~15.56 GiB after BIOS/hypervisor carve-outs) no longer trips the WARN branch.

  • Preflight suite — nf-quickstart now runs a single visual preflight check before any heavy work (image import, k3s install, helm releases) and gives an ordered, pass/warn/fail report. The suite covers:

    • CPU / RAM / disk (k3s mode only) — enforces the documented 4 CPU / 16 GiB / 50 GB requirements; degraded values warn (and skip the support stack on CPU), insufficient values fail.
    • Host firewall (k3s mode only) — declaratively inspects the host firewall configuration (iptables, nftables, ufw, firewalld) for REJECT/DROP rules that would block inbound traffic on the controller client API and router edge ports. Replaces the earlier active bind-and-connect probe, which could not detect blocking rules because Linux routes self-addressed packets through lo, bypassing the INPUT chain — a host could have iptables -j REJECT on the port in force and still report as reachable. The declarative check reads the rule set, attributes any block to the specific firewall manager and rule, and warns when a rule references the port but its effect is ambiguous (e.g. jumps to a custom chain). Limitation: external security groups, NAT, and upstream firewalls are not observable from inside the host and are covered by the separate external-reachability reminder line.
    • DNS (non-interactive k3s only) — resolves CTRL_ADDR and RTR_ADDR from this host up front. Interactive runs verify DNS at the controller / router prompt sites via the existing dns_check, where the operator can fix the entry on the spot. BYO-cluster installs don't run host-level DNS here either — what matters there is reachability from the cluster nodes, which this host can't probe.
    • External reachability reminder (non-interactive k3s only) — informational line listing the concrete ports (tcp/CLIENT_PORT for the controller client API, tcp/RTR_EDGE_PORT for the router edge listener) the operator must verify externally. Cloud security groups, NSGs, NAT, and upstream firewalls cannot be observed from inside the guest.

    Non-interactive runs (-y) hard-exit on any FAIL with a summary line that names every failing check. Interactive runs prompt to continue (FAIL defaults to N, WARN defaults to Y) so accidental "press enter" doesn't bypass a real blocker. BYO-cluster installs emit a single host checks skipped line and proceed straight to the dependency / install phase.

1.0.0-rc3 - 2026-04-30​

Validation​

  • Controller advertise address must be a DNS name — nf-quickstart now rejects an IPv4 or IPv6 literal for CTRL_ADDR up front, before any cluster changes are made. Non-interactive runs exit with a clear error; interactive runs re-prompt until a DNS name is entered. Surfaces a longstanding requirement at input time rather than failing later during certificate issuance.

Support stack reliability​

  • Init containers wait for real Elasticsearch and Kibana readiness — Post-install hook jobs (ILM index template, Kibana imports) and the logstash deployment now block on a cluster-health probe instead of DNS resolution. Eliminates Kubernetes Job exponential backoff retries during the bootstrap window and makes the install timing deterministic.
  • Helm install timeout raised to 15 minutes for the support stack — installers/support-stack.sh now passes --timeout 15m to helm upgrade --install, preventing the post-install hook race where a healthy cluster could be reported as a failed install on slower hosts.
  • Idempotent ECK CRD apply — kubectl apply --server-side --force-conflicts replaces kubectl create -f for the ECK CRDs. Re-running the install after a partial failure (where namespaced resources were removed but cluster-scoped CRDs remained) now succeeds cleanly.

Image pinning​

  • Ziti Console Enterprise pinned to 0.1.1 — The chart no longer defaults to :latest with imagePullPolicy: Always. Pinned to 0.1.1 and IfNotPresent so an image already loaded on the node (including offline / air-gapped installs that pre-seed containerd) is reused instead of being re-pulled from Quay on every pod restart.

1.0.0-rc1 - 2026-04-15​

Breaking changes​

:::warning Ziti Controller Helm Chart v2 → v3 This release upgrades the Ziti controller Helm chart from v2 to v3, which includes PKI consolidation and a new required cluster.mode value. The upgrade process is handled automatically by the installer, but operators should be aware of the following:

  • The controller must be upgraded before the routers. Routers running versions below 1.7 will fail to connect after the controller upgrade. The upgrade script will warn if incompatible routers are detected.
  • The PKI is consolidated from separate roots for control plane, web/client APIs, and edge enrollment into a single shared root of trust. This does not require re-enrolling routers or identities.
  • cluster.mode is now required on all installations. The upgrade hook automatically sets standalone for existing installs. New installs default to cluster-init to allow adding controllers later.
  • After the controller upgrade, the controller, router, and ziti-host are automatically restarted to pick up the new certificates.

For full details on the upstream breaking change, see PKI Consolidation and Clustered Mode. :::

:::note Quay registry credentials required Ziti Console Enterprise is now installed by default with the support stack. This requires NetFoundry private container registry (Quay) credentials. Existing installations upgrading to 1.0.0 will be prompted for these credentials during the upgrade if they are not already configured. Set REGISTRY_USERNAME and REGISTRY_PASSWORD environment variables for non-interactive upgrades. :::

New features​

  • Ziti Console Enterprise — Web console for managing your Ziti network. Automatically installed with the support stack; added during upgrade for existing installs. Requires NetFoundry container registry credentials.
  • Ziti Metrics Processor (ZMP) — Real-time enrichment of Ziti network events with identity and service context before they reach Elasticsearch. ZMP uses a dedicated Ziti admin account, created automatically during install/upgrade.
  • zLAN Ziti resources managed by Helm — Config types (zfw.v1, license.v1, interfaces.v1), logstash configs, services, and service policies are now created and updated automatically by the zlan-console Helm chart on install and upgrade. Schema changes are applied automatically, and the ziti CLI is no longer required on the installer host.
  • Controller cluster support — New installs are configured with cluster-init mode and a configurable trust domain, allowing additional controllers to be added later.
  • Base path support — zLAN Console and Ziti Console Enterprise can now be served from a sub-path (e.g. /zlan, /console) via the basePath Helm value, allowing both to share the same hostname.
  • Structured logging — Installer and upgrade output is now written to timestamped log files in a logs/ directory. Kubernetes events are captured on error. nf-support-bundle collects these logs automatically.
  • Helper commands — Added nf-controller-logs and nf-router-logs for live-tailing logs. Helper commands are set up automatically during upgrades if missing.
  • HA cluster management (beta) — New nf-cluster command for managing HA controller clusters. Supports checking cluster status (nf-cluster status), adding controllers (nf-cluster add), removing controllers (nf-cluster remove), and migrating standalone installations to cluster mode (nf-cluster migrate). The command auto-detects the primary controller Helm release and handles cert-manager timing, support stack integration, and quorum warnings. See the HA cluster management guide for details.
  • Environment file for quickstart — nf-quickstart now accepts a -f flag to source environment variables from a file for non-interactive installs. A template file (env-quickstart.example) is included with all available variables.
  • Console disable flag — Set CONSOLE_DISABLED=true to skip ZAC and Ziti Console Enterprise installation.
  • Install summary — The quickstart now prints a summary of installed Helm chart versions and Ziti container images at the end of the install.

Upgrade support​

  • Automated v2→v3 upgrade hook — Handles PKI consolidation, interfaces.v1 config-type rename, cluster.mode injection, certificate propagation wait, and component restarts.
  • Support stack v0→v1 upgrade hook — Deploys ZMP and Redis, replicates admin secrets, migrates Grafana datasources, and consolidates RabbitMQ queues. Installs Ziti Console Enterprise and snapshot jobs if missing.
  • Incompatible router detection — Upgrade warns if any routers are running versions below 1.7 before proceeding with a controller upgrade.
  • Controller upgraded before router — The upgrade order now upgrades the controller first, then the router, to ensure compatibility.
  • Component-selective upgrades — nf-upgrade accepts --controller, --router, --support, --console, --zlan, --ziti-host, and --k3s flags.
  • Console Enterprise install on upgrade — If Ziti Console Enterprise is not installed, the upgrade script now offers to install it.
  • HA controller snapshot restore — Snapshot restore scripts now detect clustered controllers and handle the restore process accordingly.

Monitoring and dashboards​

  • Event namespace alignment — Updated for OpenZiti v1.4+ (e.g. fabric.circuits → circuit).
  • Grafana datasource overhaul — Human-readable datasource names, correct timestamp fields, and safe upgrade paths.
  • Dashboard consolidation — Removed multi-network dashboard variants; updated for ZMP-enriched data.
  • Logstash pipeline rewrite — Consolidated RabbitMQ queues (fabric/edge/metrics → ziti.events) with updated filters.
  • Elasticsearch keyword mappings — Index templates now include strings_as_keyword dynamic mapping to ensure .keyword sub-fields are available for dashboard aggregations.

Security​

  • Security hardening enabled by default — Restricted file permissions, credential output suppression, and secure pull secret handling are now standard for all installations. The -H flag has been retired.
  • Credentials retrieved dynamically — nf-install-notes fetches credentials live from Kubernetes secrets instead of reading from a static file.
  • Dedicated ZMP admin account — ZMP now authenticates to the Ziti controller with its own dedicated admin account instead of using the shared default admin credentials.

Container versions​

  • Elasticsearch, Kibana, Logstash, Filebeat, Metricbeat: 8.19.12
  • Grafana: 12.3.5
  • RabbitMQ: 3.13
  • Redis: 7.4
  • ZMP: 0.0.5-ba92eb4

Other changes​

  • Trust domain configuration — New installs prompt for a trust domain used in SPIFFE IDs. Configurable via TRUST_DOMAIN environment variable for non-interactive mode.
  • Offline installer configuration — Added build/offline-config.sh for generating offline installation bundles.
  • Customize Helm values guide — New documentation for modifying Helm values after installation.
  • zfw.v1 schema updated — Added ICMP protocol support.
  • k3d installer removed — installers/k3d-install.sh has been removed.

0.5.0 - 2026-03-05​

Security hardening​

  • Added -H flag to quickstart.sh for STIG-hardened installations
  • In hardened mode, NetworkPolicies are applied to the support and ziti namespaces (BYO clusters)
  • Enabled TLS certificate verification for Logstash and Grafana connections to Elasticsearch
  • Added configurable elasticsearch.tlsCaSecret Helm value for BYO Elasticsearch deployments
  • Added SHA256 integrity verification for downloaded ECK operator manifests
  • Registry pull secret output is suppressed and file permissions restricted

Upgrade script improvements​

  • Added component-specific upgrade flags: --router, --controller, --ziti-host, --support, --k3s
  • Added --skip-snapshot / -S flag to skip pre-upgrade database snapshot
  • Auto-detect offline mode from pre-downloaded Helm charts

Fixes​

  • Fixed snapshot creation and restore jobs for offline environments
  • Removed unused Docker socket mount from Metricbeat
  • Fixed .env parsing to use export instead of eval
  • Updated ziti-host Helm chart version constraint to ^1.2.0
  • Bumped support Helm chart to 0.1.5

0.4.4 - 2026-02-24​

  • Improved alignment with offline installer
  • Fixes for snapshot creation and restore jobs
  • Fixed missing zip dependency for debian and offline install packages
  • Documentation updates
  • Improved user guidance post-install and upgrade for debian package
  • Fixed OpenZiti upgrade order based on latest OpenZiti best practices (routers, then controller)

0.4.3 - 2026-02-11​

  • Fixes for nf-helpers.sh to be re-run safe
  • Updates for package installer for deb amd64 and arm64 packages
  • Added nf-restore-snapshot command for restoring controller snapshots

0.4.2 - 2025-11-14​

  • Updated installer docs with offline install and zlan options
  • Fix script directory path in nf-helpers.sh
  • Fix Helm chart apiVersion

0.4.1 - 2025-11-10​

  • Multiple fixes for zLAN installation
  • Added an OpenZiti database snapshot as a pre-upgrade step to upgrade.sh
  • Fix default router policy to better account to private routers
  • Added nf-help commands

0.4.0 - 2025-10-30​

  • Updated support stack container images to use wolfi/oss image variants
  • Added migration script for ziti-host container at ./utilities/migrate_ZET_to_helmchart.sh for legacy installs
  • Pinned Helm chart versions for OpenZiti components in .env file to ensure alignment on OpenZiti versions
  • Fix for zLAN installs - added missing interfaces.v1 config type

0.3.4 - 2025-10-28​

  • ziti-host container in the support namespace is now managed by Helm for easier maintenance and upgrades
  • NetFoundry support stack is now installed by default, the -s option can be passed to disable it
  • Added support for zLAN installation using the -z flag. Requires NetFoundry container registry secret
  • Updated charts so that all container images and pull policies are configurable

0.3.3 - 2025-09-24​

  • Improve handling of KUBECONTEXT for K3S installs
  • Fix default imagePullPolicy for support stack resources
  • Enabled OpenZiti database snapshots by default
  • Migrated documentation to public docs site at: https://netfoundry.io/docs/onprem/intro

0.3.2​

  • Added doc for FIPS installation
  • Reworked quickinstall.sh for better K8s and EKS integration
  • Added guided upgrade script at ./upgrade.sh
  • Fixes for missing KUBECONTEXT and making quickstart more re-run safe
  • Added OEM documentation at ./docs/oem.md for advanced installation use cases
  • Added support and documentation for automated backups, restore, and migration

0.3.1​

  • Updates to support ziti-controller Helm chart v2.0+
  • cert-manager and trust-manager are now installed as separate Helm charts and managed independently from the ziti-controller chart
  • Added charts for local PCV backup or S3 backup for OpenZiti boltdb database
  • Enabled local PVC backup of boltdb by default
  • Added improved support for custom helm value files
  • Added restore processes for local PVC backup and S3 backup (./utilities/restore.sh, ./utilities/s3_restore.sh)

0.3.0​

  • Moved to k3s as the default Kubernetes engine
  • Updated proxy documentation for k3s

0.2.8​

  • Cleanup of quickinstall feedback and INSTALL-NOTES.txt
  • Fix for older versions of helm that failed upon re-add of a repo
  • Documentation cleanup

0.2.7​

  • Added additional logging and diagnostic collection to installer scripts
  • Added documentation for single-node RKE2 installs
  • Added support for additional logstash outputs via helm values

0.2.6​

  • Added documentation for outbound whitelisting for installations behind a corporate proxy
  • Changed default elasticsearch nodes to 1 for a much smaller resource footprint by default
  • Updated default configuration to use ALPN support for OpenZiti, reducing the number of ports and load balancers needed
  • Added support for ARM architecture
  • Added support and documentation for minimal installs on MicroK8s and Raspberry Pi4+

0.2.5​

  • Added support for non-interactive quickstarts, use the -y flag and set the CTRL_ADDR environment variable
  • Added an uninstall.sh script that removes OpenZiti, support, and all checkpoints
  • Added a production installer - k8s-install.sh
  • Fixed time scale for Grafana OpenZiti controller dashboard showing in milliseconds when it should have showed nanoseconds