NiPyAPI 1.0 Release - Apache NiFi 2.x Compatibility #388
+112,811
−467,025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Auth: spec-driven bearerAuth; removed template-injected tokenAuth; helpers default to bearerAuth - Templates: enum allowed_values fix; skip None-check for required readOnly; removed callbacks; raw regex strings - Clients: augmented securitySchemes (temp) for NiFi/Registry; regenerated clients - Tests: full suites green (default, secure-ldap, secure-mtls) - Docs: update history; link NIFI-14852 and NIFI-14850 - Cleanup: dropped Templates feature; removed 1.x docker/localdev; pruned tests Refs: - NiFi Core bearer scheme: https://issues.apache.org/jira/browse/NIFI-14852 - OpenAPI enum inconsistencies: https://issues.apache.org/jira/browse/NIFI-14850
- tests: env-driven setup; NiFi proxy DN (C-first) policies; bucket r/w for LDAP user - security: policy retrieval handles Registry ApiException - config/utils: env-driven TLS CA; avoid stale SSLContext; sane verify_ssl defaults - certs: clean generation with clientAuth; truststore fix - docker compose, fetchers, debug script; .gitignore; history update
- Remove legacy regression scaffolding; centralize env-driven setup in conftest - Update versioning/canvas/system/parameters/security/utils tests to new flow - Drop obsolete test resources
…st target, remove env reads in utils)
… docker extra (use Docker Compose)
…cludes tests/clients; Makefile tidy
…p fix; remove mid-function defaults; Makefile docker down improvements; docs history updated; client-gen cleanup
These files should be ignored according to .gitignore line 52 and are regenerated during the build process. Cleaning up 553 temporary files that were accidentally tracked.
- Fix 'make down' to properly stop containers from all profiles - Use explicit --profile flags for single-user, secure-ldap, secure-mtls - Prevents issues where containers started with profiles weren't being detected/stopped - Add 'test-all' target for comprehensive e2e testing across all profiles - Automatically cycles through all three authentication profiles - Proper setup -> test -> teardown cycle for each profile - Includes certificate generation and infrastructure management - Fix container status display formatting in verification output - Use proper docker compose --format for clean status display - Eliminates garbled output characters Resolves infrastructure management issues after repository cleanup.
- Add .pre-commit-config.yaml with targeted scope: * Only runs on core nipyapi/ files (excludes generated API clients) * Excludes nipyapi/nifi/, nipyapi/registry/, _version.py * Prevents repository-wide editing issues from previous experience - Fix line length issues in config.py and security.py - Add pylint pragmas for legitimate broad exception catching - Remove IDE-specific workspace file (already in .gitignore) All pre-commit hooks now pass cleanly on targeted files.
- Replace ruamel.yaml with PyYAML in nipyapi/utils.py - Update requirements.txt: ruamel.yaml>=0.16.3 -> PyYAML>=6.0 - Maintain full compatibility: clean block-style output, round-trip fidelity - Update documentation references in README.rst, tests, and history - Add test-specific make target for focused testing with infrastructure - Reduces dependency count while preserving all functionality - All utils tests pass with real NiFi flow data
- Remove nipyapi/_version.py from git tracking (auto-generated by setuptools-scm) - Add nipyapi/_version.py to .gitignore per setuptools-scm best practices - Version file is automatically regenerated from git state, should not be committed
- Add test-dist make target for validating built distributions - Add resources/scripts/test_distribution.py validation script - Tests imports and basic functionality in clean virtual environment - Ensures packaged wheels actually work before release to PyPI - Update history.rst to document distribution validation features
- Remove 25+ legacy NiFi 1.x and Registry 0.x/1.x API definitions - Remove temporary development files (.auth-test, .backup, issue tracking) - Keep only current NiFi 2.5.0 and Registry 2.5.0 core API definitions - Saves ~13MB repository space (18MB -> 4.8MB in api_defs/) - Aligns with 2.x migration and removes unused legacy artifacts
Major improvements for testing and CI infrastructure: ## Environment Variable Standardization - Rename PROFILE → NIPYAPI_AUTH_MODE throughout project for clarity - Update tests/conftest.py, Makefile, and all documentation - Clear semantic meaning: authentication mode configuration - Future-extensible for additional auth modes (e.g., OIDC) ## GitHub Actions CI Workflow - Add comprehensive .github/workflows/ci.yml with 6-job matrix - Full Docker NiFi infrastructure testing (make certs, up, wait-ready, down) - Python versions: 3.9, 3.10, 3.11, 3.12 with single-user mode - Python 3.12 with secure-ldap and secure-mtls modes - Consolidated linting, coverage, and distribution tests - SSL verification overrides for testing environments ## Makefile Improvements - Update all targets to use AUTH_MODE parameter consistently - Maintain backward compatibility with make shortcuts (test-su, test-ldap, test-mtls) - Update integration testing workflow documentation ## Documentation Updates - Update docs/authentication.rst, contributing.rst, devnotes.rst - Update README.rst and docs/history.rst with changes - Consistent AUTH_MODE usage examples throughout ## Testing Enhancements - Fix distribution test warning (rename function to avoid pytest discovery) - Add pytest coverage configuration to pyproject.toml - Environment variable fixes for proper CI execution ## Infrastructure - Verified Docker compose configuration and port mappings - Certificate generation and SSL handling for all auth modes - Comprehensive integration test coverage Ready for GitHub Actions testing with full NiFi infrastructure.
- Replace inconsistent AUTH_MODE usage with NIPYAPI_AUTH_MODE throughout project - Fix authentication issues in Makefile targets (test, coverage) by adding missing environment variable - Update CI workflow to use consistent NIPYAPI_AUTH_MODE in make commands - Update all documentation examples to use NIPYAPI_AUTH_MODE consistently - Resolves test failures where Registry connection was refused due to missing auth config This ensures consistent authentication profile handling across: - Makefile targets - CI/CD workflows - Documentation examples - Test execution
- Add 10 comprehensive test cases covering all main system functions - Test both Registry (no auth) and NiFi (auth required) connectivity patterns - Validate real API response structures and data types - Test error handling for cluster operations in single-user mode - Cover authentication and fallback logic with actual infrastructure - Use real NiFi/Registry responses for baseline validation instead of mocking Functions tested: - get_system_diagnostics() - detailed system info with version data - get_nifi_version_info() - version with fallback logic - get_registry_version_info() - simple version string - get_cluster() / get_node() - proper error handling in single-user mode
- Complete overhaul of Sphinx documentation generation with custom structured approach - Replace monolithic API reference pages with modular, navigable structure (fixes #376) - Auto-generated API categorization and individual core module pages - Template-level docstring formatting fixes for Sphinx-compliant generated code - Direct GitHub source code links with line-level precision - Modern installation guide with auto-generated dependencies documentation - Zero Sphinx warnings achieved through systematic improvements - Enhanced navigation and maintainability for future development
- Fix regex patterns in generate_api_client.sh (double -> single backslash) - Issue: In zsh environment, 'augmented\\.json$' pattern failed to match augmented OpenAPI specs, causing script to use base specs without authentication schemes - Result: Generated clients had auth_settings = [] instead of ['bearerAuth'] - Regenerated both NiFi and Registry clients with correct authentication - Fixed enum validation errors in Registry models (e.g., backoff_mechanism) - Improved Mustache template docstring formatting for Sphinx compliance Authentication now works correctly for all generated API clients.
- Fix mustache template spacing around {{{notes}}} sections
- Add proper blank lines before and after notes content
- Regenerate Registry client APIs with improved docstring formatting
- Resolves 36 Sphinx warnings: 'Definition list ends without a blank line'
- Documentation now builds with 0 warnings again
- Verified functionality across all authentication profiles (single-user, secure-ldap, secure-mtls)
All tests pass: 95+114+95 = 304 total tests passed
…targets Documentation Structure Finalization: - Completed flat API structure with individual pages for all 52 APIs (39 NiFi + 13 Registry) - Auto-discovery of all 479 model classes (394 NiFi + 85 Registry) for comprehensive cross-referencing - Enhanced Google-style docstrings in generated clients with primitive type handling and method distinction - Added client architecture documentation explaining dual method structure and callbacks - Zero Sphinx warnings achieved through systematic template and manual docstring improvements - Updated installation guide with modern practices and auto-generated dependencies Build System Enhancements: - Added rebuild-all target for complete project regeneration from clean slate - Enhanced clean-all to truly remove ALL artifacts (build, clients, docs, coverage, temp files) - Optimized rebuild flow to avoid unnecessary infrastructure cycling - Leverages existing client_gen scripts for maintainable workflow This commit preserves all work before testing the comprehensive rebuild process.
Problem: Certificate regeneration while containers running caused SSL verification failures during rebuild-all and other workflows. Solution: make certs now auto-detects running containers and stops them before regenerating certificates, ensuring fresh certificate/container alignment. Benefits: - Eliminates SSL certificate timing issues during rebuild-all - Clear user feedback when containers are stopped - Prevents future certificate mismatch problems - Maintains clean, predictable workflow
- Fix secure profile port mappings: secure-ldap (9444/18444), secure-mtls (9445/18445) - Restructure with logical groupings: Operational, Low-Level, Meta targets - Add grouped help system and dependency checking - Improve certificate regeneration with auto-container-stop Fixes rebuild-all workflow blocking on secure profiles.
- Updated all NiFi API clients (28 files) with corrected auth_settings - Updated all Registry API clients (13 files) with fixed enum formatting - Applied improved mustache templates for Google-style docstrings
- Enhanced examples/fdlc.py with multi-environment workflow and security bootstrapping - Standardized password consistency across single-user and secure-ldap profiles - Added type validation and better error handling in nipyapi.versioning.save_flow_ver - Fixed SSL certificate handling and authentication patterns following conftest.py - Updated documentation with comprehensive authentication examples
- Deleted examples/console.py (166 lines of commented-out, non-functional code) - Updated examples documentation to remove console.py references - Added deprecation note to 1.0.0 release notes while preserving historical accuracy - Functionality superseded by modernized fdlc.py example
- Fix security bootstrapping: Registry policies now applied to all profiles including single-user - Enhance authentication docs to distinguish test vs production certificate usage - Add comprehensive Safari keychain guidance with security considerations - Update playground to default to single-user for optimal new user experience - Remove deprecated secure_connection.py example and legacy examples/keys/ directory - Add dedicated setup_playground.py script following conftest.py patterns
- Created docs/migration.rst with version differences, API changes, and configuration updates - Grounded migration guide in actual 0.x files (commit 0410ac3) for accurate examples - Added common migration issues: authorization failures and Registry proxy problems - Included bootstrap security policy solutions for secure profiles - Updated docs/index.rst to include migration guide in toctree - Updated docs/history.rst to document migration guide addition
- Updated docs/devnotes.rst for 2.x development practices including modern testing, client generation, and release workflows - Removed outdated Python 2.7 references and legacy build tool instructions - Improved Makefile organization and target documentation - Added structured groupings (Operational, Low-Level, Meta targets) for better developer experience
OIDC Enhancements: - Replace brittle subprocess Docker logs UUID extraction with robust JWT token parsing - Add return_token_info parameter to service_login_oidc() for backwards compatibility - Create nipyapi.utils.extract_oidc_user_identity() for industry-standard JWT sub field extraction - Fix OIDC sandbox setup to gracefully reuse existing artifacts instead of failing - Fix Registry hostname resolution for OIDC profile (localhost -> registry-oidc:18080) Professional Terminology: - Rename 'playground' to 'sandbox' throughout codebase for enterprise appropriateness - Move sandbox script from resources/scripts/ to examples/sandbox.py as user-facing example - Update all documentation, Makefile targets, and UI messages Code Quality & Tooling: - Restore missing linting targets: make lint, make flake8, make pylint, make pre-commit - Achieve 10.0/10 pylint score for core nipyapi modules (excludes generated clients) - Fix code quality issues: line length, trailing whitespace, missing timeouts - Enhance make coverage to ensure infrastructure availability before analysis - Fix coverage configuration to exclude generated API clients (industry best practice) - Add proper token validation for make coverage-upload (CI-only with helpful guidance) - Remove obsolete smoke test infrastructure (incompatible with security-first design) Infrastructure: - Add Keycloak realm configuration for OIDC testing - Remove deprecated smoke_versions.py script - Update authentication documentation with accurate OIDC setup process This commit significantly improves OIDC reliability, adopts professional terminology suitable for enterprise environments, and establishes robust code quality tooling.
…fixes - Add set_shared_ca_cert() convenience function for common CA setup - Fix authentication state restoration in all security test files - Resolve OIDC mock test token contamination issues - Improve sandbox SSL warning handling with project pathways - Organize config.py SSL functions with clear sections - All security test profiles now passing (single-user, LDAP, mTLS, OIDC) This establishes a stable foundation for the upcoming configuration architecture restructuring in the 1.x release.
- Replace sleep(0.5) with intelligent wait_to_complete polling - Consolidate test configuration to centralized active_config dictionary - Simplify bucket fixture to use ensure_registry_bucket - Rename authentication.rst to security.rst with expanded guidance - Add debug_registry.py reference for troubleshooting - Validate all auth profiles: single-user, ldap, mtls, oidc All test suites passing across authentication modes.
- **Core Infrastructure**: - Add nipyapi.profiles module with centralized configuration/authentication - Add examples/profiles.yml with pre-configured profiles for all test environments - Standardize NIPYAPI_PROFILE environment variable across all tools - **Profile Management Features**: - nipyapi.profiles.switch() handles config resolution, SSL setup, and authentication - Support for single-user, secure-ldap, secure-mtls, secure-oidc profiles - Environment variable overrides for all configuration values - Authentication method auto-detection (OIDC > mTLS > Basic precedence) - Return authentication metadata (OIDC tokens, usernames) for bootstrapping - **Enhanced Infrastructure**: - Improved wait_ready.py with detailed connection attempt logging - Enhanced is_endpoint_up() with refined SSL error handling (cert vs handshake errors) - Updated Docker compose for proper single-user HTTP configuration - Centralized URL normalization and client reset logic - **Test Suite Improvements**: - Refactored conftest.py to use profiles system instead of manual setup - Fixed authentication state corruption between tests - Enhanced LDAP tests to work with public test environment constraints - Added comprehensive profiles test coverage - **Documentation**: - Complete profiles documentation in docs/profiles.rst - Updated security.rst with profiles-based authentication guidance - Migration guide for existing authentication code - Updated all example scripts to use profiles system - **Code Quality**: - Python 3.9+ compatibility with appropriate type hints - Centralized SSL configuration and error handling - Removed deprecated debug_registry.py script - 166 tests passing across all authentication profiles This major refactoring provides a unified, maintainable approach to NiFi/Registry configuration and authentication, replacing scattered manual setup code.
…e entry - Combine 1.0.0, 1.0.1, and 1.0.2 entries into single version 1.0 (2025-08-23) - Structure entry with user-focused sections highlighting breaking changes and solutions - Emphasize AWS CLI-style profiles system and automated development workflow - Include comprehensive technical improvements list covering all major rewrite changes - Ready for feature branch release and pull request
- Replace /dev/AlmostCertainlyNotAValidWriteDevice with /nonexistent/directory/structure/that/cannot/exist/file.txt - New path fails for both root and non-root users due to missing directory structure - Resolves test failure in Docker CI environments where container runs as root
…r fit new profiles, reworked migration and quickstarts and examples for better guidance
- Add build>=1.0.0 dependency to fix 'No module named build' error - Add getenv_bool() utility function for robust boolean environment variable parsing - Update profiles.py to use getenv_bool for SSL verification settings - Add SSL environment variables (NIFI_VERIFY_SSL=0, etc.) to CI workflow for Python 3.12 - Add comprehensive tests for getenv_bool functionality Resolves CI failures on Python 3.12 with secure profiles by properly handling stricter SSL validation and environment variable parsing.
- Add environment variable debugging to verify SSL flag parsing - Add container stabilization step with docker ps output - Add failure analysis with container logs dump - Add getenv_bool parsing verification step These debug improvements will help diagnose CI-specific failures in secure-ldap and secure-mtls profiles on GitHub Actions.
- Update gen_certs.sh to create keystore/truststore files with 644 permissions - Resolves AccessDeniedException in Docker containers where NiFi runs as different user - Eliminates need for CI-specific permission workarounds - Maintains security with read-only Docker volume mounts Root cause: OpenSSL creates PKCS12 files with 600 permissions by default, but containers running as different UIDs need read access to these files. Solution creates files with proper permissions at generation time.
| # IMPORTANT: Only runs on core nipyapi/ files, excludes generated API clients | ||
| repos: | ||
| - repo: https://github.com/pre-commit/pre-commit-hooks | ||
| rev: v4.6.0 |
There was a problem hiding this comment.
Can we have isort and black support as well?
You have to have an isort config that has the black setting in it ( I can provide if you need it ).
| sphinx-apidoc -o docs/ nipyapi | ||
| wait-ready: ## wait for readiness using profile configuration; requires NIPYAPI_PROFILE=single-user|secure-ldap|secure-mtls|secure-oidc | ||
| @if [ -z "$(NIPYAPI_PROFILE)" ]; then echo "❌ NIPYAPI_PROFILE is required"; exit 1; fi | ||
| @echo "⏳ Waiting for $(NIPYAPI_PROFILE) infrastructure to be ready..." |
There was a problem hiding this comment.
python will not be available on all platforms, some will just have python3. Usually I use a variable PYTHON and default it python3, and then it can be overridden.
| $(MAKE) test-dist | ||
| @echo "=== 9/9: Generate Documentation ===" | ||
| $(MAKE) docs | ||
| @echo "✅ Comprehensive rebuild completed successfully!" |
There was a problem hiding this comment.
This is a lot of stuff, consider how it would work if you broke it down into multiple makefiles, such that you had a top level Makefile and n included .mk files that had targets related by function.
It is a trade off between working in a big file and keeping the dispersed targets in mind
| @@ -32,84 +32,168 @@ Features | |||
| -------- | |||
|
|
|||
| **Three layers of Python support for working with Apache NiFi:** | |||
There was a problem hiding this comment.
Should this say: "- Top-level Api (see examples/ in the repo)?
Also, if the examples had their own README, this would be better as a link the that
| @@ -32,84 +32,168 @@ Features | |||
| -------- | |||
|
|
|||
| **Three layers of Python support for working with Apache NiFi:** | |||
| - High-level Demos and example scripts | |||
| - Top-level examples (see `examples/` in the repo) | |||
| - Mid-level Client SDK for typical complex tasks | |||
| - Low-level Client SDKs for the full API implementation of NiFi and selected sub-projects | |||
|
|
|||
| **Functionality Highlights:** | |||
There was a problem hiding this comment.
What is a profile? Maybe a link here to a profile topic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Status:
Draft. Code needs a polish, but ready for community testing.
Summary
Major migration to Apache NiFi/Registry 2.x (tested against 2.5.0). Drops NiFi 1.x support on main branch.
This release introduces fundamental changes including a profiles-based configuration system and multiple breaking changes requiring user migration.
Key Documentation:
docs/migration.rst- Complete migration guide with breaking changes inventorydocs/profiles.rst- Profiles system configuration referencedocs/security.rst- Authentication and security setup guideBreaking Changes - Action Required
API Changes
update_run_status→update_run_status1Compatibility
New Profile Management System
Configuration Management
single-user,secure-ldap,secure-mtls,secure-oidcnipyapi.profiles.switch('profile-name')See
docs/profiles.rstfor complete configuration reference andexamples/profiles.ymlfor working configuration templates.Configuration Comparison
Development Workflow Improvements
Automation
Testing Integration
make up NIPYAPI_PROFILE=secure-ldap make test NIPYAPI_PROFILE=secure-ldapSee
Makefilefor complete list of available targets anddocs/devnotes.rstfor development workflow details.Technical Improvements
Documentation
Development Stack
Critical Configuration Changes
Port and Endpoint Changes
Function Mapping
update_run_status→update_run_status1FlowfileQueuesApi→FlowFileQueuesApiFor complete breaking changes inventory and migration steps, see
docs/migration.rst.Testing This PR
Basic Validation
Multi-Profile Testing
Example Validation
See
examples/directory for additional demonstration scripts anddocs/installation.rstfor setup guidance.Build and Distribution Testing
Package Build Validation
Distribution Smoke Test
Migration Resources
Documentation
README.rst: Updated quick start and installation guidedocs/migration.rst: Comprehensive migration guide with breaking changes inventorydocs/installation.rst: Modern installation and setup guidedocs/security.rst: Deeper details on Authentication and security Configurationexamples/fdlc.py: Modern Flow Development Life Cycle patternsexamples/sandbox.py: Multi-profile authentication demonstrationexamples/profiles.yml: Configuration templates for all authentication methodsCommon Migration Tasks
Review Areas
This is a major version release with intentional breaking changes. Comprehensive migration documentation and tooling are provided to support users upgrading from 0.x to 1.x.