Skip to content

Commit bc14fa8

Browse files
ChaffelsonChaffelson
authored
NiPyAPI 1.0 Release - Apache NiFi 2.x Compatibility (#388)
Major migration to Apache NiFi/Registry 2.x (tested against 2.5.0). Drops NiFi 1.x support on main branch. BREAKING CHANGES: - Function renaming: Upstream API changes result in operation IDs with suffixed names - Authentication and configuration overhaul aligned with modern API standards - Users must review and update authentication patterns NEW FEATURES: Profile Management System: - Extensible YAML/JSON profiles with environment variable overrides - Intelligent authentication detection: OIDC, mTLS, and Basic auth - Built-in profiles: single-user, secure-ldap, secure-mtls, secure-oidc - AWS CLI-style workflow with NIPYAPI_PROFILE environment variable Automated Development Workflow: - Comprehensive Makefile targets for development and release processes - End-to-end automation from certificates to integration tests - GitHub Actions CI with full Docker NiFi integration - Smart certificate regeneration and optimized rebuild flows Technical Improvements: - Documentation system rebuild with flat API structure - Test coverage expansion with profile-driven automation - Pre-commit checks with black, isort, flake8, and pylint - Modern dependency management with python -m build - Enhanced convenience functions and certificate handling - API augmentation system for upstream API issues Migration Support: - Comprehensive migration guide (docs/migration.rst) - Sandbox Docker environment with make sandbox - FDLC example modernized for multi-environment workflows Fixes #376 Fixes #373 Fixes #370 Fixes #240 Fixes #225
1 parent 0410ac3 commit bc14fa8

File tree

762 files changed

+114815
-467948
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

762 files changed

+114815
-467948
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [ main, maint-0.x ]
6+
pull_request:
7+
branches: [ main, maint-0.x ]
8+
9+
jobs:
10+
test-python-312-single-user:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Set up Python 3.12
17+
uses: actions/setup-python@v4
18+
with:
19+
python-version: "3.12"
20+
21+
- name: Install dependencies
22+
run: |
23+
python -m pip install --upgrade pip
24+
pip install -e .[dev]
25+
26+
- name: Lint with flake8 and pylint
27+
run: make lint
28+
29+
- name: Set up Docker Buildx
30+
uses: docker/setup-buildx-action@v3
31+
32+
- name: Generate certificates
33+
run: make certs
34+
35+
- name: Start NiFi infrastructure (single-user)
36+
run: make up NIPYAPI_PROFILE=single-user
37+
38+
- name: Wait for NiFi to be ready
39+
run: make wait-ready NIPYAPI_PROFILE=single-user
40+
41+
- name: Run integration tests with coverage
42+
run: NIPYAPI_PROFILE=single-user PYTHONPATH=${{ github.workspace }}:$PYTHONPATH pytest --cov=nipyapi --cov-report=xml --cov-report=term-missing
43+
44+
- name: Test distribution build and import
45+
run: make test-dist
46+
47+
- name: Stop NiFi infrastructure
48+
if: always()
49+
run: make down
50+
51+
- name: Upload coverage to Codecov
52+
uses: codecov/codecov-action@v4
53+
with:
54+
file: ./coverage.xml
55+
fail_ci_if_error: false
56+
verbose: true
57+
env:
58+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
59+
60+
test-python-311-single-user:
61+
runs-on: ubuntu-latest
62+
63+
steps:
64+
- uses: actions/checkout@v4
65+
66+
- name: Set up Python 3.11
67+
uses: actions/setup-python@v4
68+
with:
69+
python-version: "3.11"
70+
71+
- name: Install dependencies
72+
run: |
73+
python -m pip install --upgrade pip
74+
pip install -e .[dev]
75+
76+
- name: Set up Docker Buildx
77+
uses: docker/setup-buildx-action@v3
78+
79+
- name: Generate certificates
80+
run: make certs
81+
82+
- name: Start NiFi infrastructure (single-user)
83+
run: make up NIPYAPI_PROFILE=single-user
84+
85+
- name: Wait for NiFi to be ready
86+
run: make wait-ready NIPYAPI_PROFILE=single-user
87+
88+
- name: Run integration tests
89+
run: NIPYAPI_PROFILE=single-user PYTHONPATH=${{ github.workspace }}:$PYTHONPATH pytest
90+
91+
- name: Stop NiFi infrastructure
92+
if: always()
93+
run: make down
94+
95+
test-python-310-single-user:
96+
runs-on: ubuntu-latest
97+
98+
steps:
99+
- uses: actions/checkout@v4
100+
101+
- name: Set up Python 3.10
102+
uses: actions/setup-python@v4
103+
with:
104+
python-version: "3.10"
105+
106+
- name: Install dependencies
107+
run: |
108+
python -m pip install --upgrade pip
109+
pip install -e .[dev]
110+
111+
- name: Set up Docker Buildx
112+
uses: docker/setup-buildx-action@v3
113+
114+
- name: Generate certificates
115+
run: make certs
116+
117+
- name: Start NiFi infrastructure (single-user)
118+
run: make up NIPYAPI_PROFILE=single-user
119+
120+
- name: Wait for NiFi to be ready
121+
run: make wait-ready NIPYAPI_PROFILE=single-user
122+
123+
- name: Run integration tests
124+
run: NIPYAPI_PROFILE=single-user PYTHONPATH=${{ github.workspace }}:$PYTHONPATH pytest
125+
126+
- name: Stop NiFi infrastructure
127+
if: always()
128+
run: make down
129+
130+
test-python-39-single-user:
131+
runs-on: ubuntu-latest
132+
133+
steps:
134+
- uses: actions/checkout@v4
135+
136+
- name: Set up Python 3.9
137+
uses: actions/setup-python@v4
138+
with:
139+
python-version: "3.9"
140+
141+
- name: Install dependencies
142+
run: |
143+
python -m pip install --upgrade pip
144+
pip install -e .[dev]
145+
146+
- name: Set up Docker Buildx
147+
uses: docker/setup-buildx-action@v3
148+
149+
- name: Generate certificates
150+
run: make certs
151+
152+
- name: Start NiFi infrastructure (single-user)
153+
run: make up NIPYAPI_PROFILE=single-user
154+
155+
- name: Wait for NiFi to be ready
156+
run: make wait-ready NIPYAPI_PROFILE=single-user
157+
158+
- name: Run integration tests
159+
run: NIPYAPI_PROFILE=single-user PYTHONPATH=${{ github.workspace }}:$PYTHONPATH pytest
160+
161+
- name: Stop NiFi infrastructure
162+
if: always()
163+
run: make down
164+
165+
test-python-312-secure-ldap:
166+
runs-on: ubuntu-latest
167+
168+
steps:
169+
- uses: actions/checkout@v4
170+
171+
- name: Set up Python 3.12
172+
uses: actions/setup-python@v4
173+
with:
174+
python-version: "3.12"
175+
176+
- name: Install dependencies
177+
run: |
178+
python -m pip install --upgrade pip
179+
pip install -e .[dev]
180+
181+
- name: Set up Docker Buildx
182+
uses: docker/setup-buildx-action@v3
183+
184+
- name: Generate certificates
185+
run: make certs
186+
187+
- name: Start NiFi infrastructure (secure-ldap)
188+
run: make up NIPYAPI_PROFILE=secure-ldap
189+
190+
- name: Wait for containers to stabilize
191+
run: sleep 5 && docker ps -a
192+
193+
- name: Debug SSL environment
194+
run: |
195+
python -c "import ssl; print('OpenSSL version:', ssl.OPENSSL_VERSION)"
196+
python -c "import ssl; print('SSL context flags:', ssl.create_default_context().verify_flags)"
197+
198+
- name: Wait for NiFi to be ready
199+
run: make wait-ready NIPYAPI_PROFILE=secure-ldap
200+
201+
- name: Run integration tests
202+
run: NIPYAPI_PROFILE=secure-ldap PYTHONPATH=${{ github.workspace }}:$PYTHONPATH pytest
203+
204+
- name: Dump running containers and logs on failure
205+
if: failure()
206+
run: |
207+
echo "=== Container Status ==="
208+
docker ps -a
209+
echo "=== Container Logs ==="
210+
for container in $(docker ps -aq); do
211+
echo "--- Logs for container: $container ---"
212+
docker logs "$container" 2>&1 || echo "Failed to get logs for container $container"
213+
echo ""
214+
done
215+
216+
- name: Stop NiFi infrastructure
217+
if: always()
218+
run: make down
219+
220+
test-python-312-secure-mtls:
221+
runs-on: ubuntu-latest
222+
# Note: This test can be flakey in local CI environments
223+
224+
steps:
225+
- uses: actions/checkout@v4
226+
227+
- name: Set up Python 3.12
228+
uses: actions/setup-python@v4
229+
with:
230+
python-version: "3.12"
231+
232+
- name: Install dependencies
233+
run: |
234+
python -m pip install --upgrade pip
235+
pip install -e .[dev]
236+
237+
- name: Set up Docker Buildx
238+
uses: docker/setup-buildx-action@v3
239+
240+
- name: Generate certificates
241+
run: |
242+
make certs
243+
echo "Ensuring certificate files are fully written..."
244+
sync
245+
246+
- name: Start NiFi infrastructure (secure-mtls)
247+
run: make up NIPYAPI_PROFILE=secure-mtls
248+
249+
- name: Wait for NiFi to be ready
250+
run: make wait-ready NIPYAPI_PROFILE=secure-mtls
251+
252+
- name: Run integration tests
253+
run: NIPYAPI_PROFILE=secure-mtls PYTHONPATH=${{ github.workspace }}:$PYTHONPATH pytest
254+
255+
- name: Dump running containers and logs on failure
256+
if: failure()
257+
run: |
258+
echo "=== Container Status ==="
259+
docker ps -a
260+
echo "=== Container Logs ==="
261+
for container in $(docker ps -aq); do
262+
echo "--- Logs for container: $container ---"
263+
docker logs "$container" 2>&1 || echo "Failed to get logs for container $container"
264+
echo ""
265+
done
266+
267+
- name: Stop NiFi infrastructure
268+
if: always()
269+
run: make down

‎.gitignore‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,23 @@ pip-delete-this-directory.txt
3838

3939
# Unit test / coverage reports
4040
htmlcov/
41+
42+
# NiPyAPI generated cert artifacts (keep script, ignore outputs)
43+
resources/certs/ca/
44+
resources/certs/nifi/
45+
resources/certs/registry/
46+
resources/certs/truststore/
47+
resources/certs/client/
48+
resources/certs/*.env
49+
resources/certs/extracted/
50+
51+
# Client-gen caches and augmented OpenAPI outputs
52+
resources/client_gen/_cache/
53+
resources/client_gen/_tmp/
54+
resources/client_gen/api_defs/*.augmented.json
55+
56+
# Local scripts/helpers
57+
resources/scripts/__pycache__/
4158
.tox/
4259
.coverage
4360
.coverage.*
@@ -157,9 +174,18 @@ fabric.properties
157174

158175
# Stupid OSX exclusions
159176
/.vscode
160-
161177
.DS_Store
178+
162179
# Coverage reports
163180
coverage*
164181
.coverage
165182
htmlcov/
183+
184+
# VSCode
185+
*.code-workspace
186+
187+
# Cursor AI rules (personal AI assistant configuration)
188+
.cursorrules
189+
190+
# setuptools-scm generated version file
191+
nipyapi/_version.py

‎.pre-commit-config.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Pre-commit configuration for NiPyAPI
2+
# IMPORTANT: Only runs on core nipyapi/ files, excludes generated API clients
3+
repos:
4+
- repo: https://github.com/pre-commit/pre-commit-hooks
5+
rev: v4.6.0
6+
hooks:
7+
- id: trailing-whitespace
8+
files: ^nipyapi/.*\.py$
9+
exclude: ^nipyapi/(nifi|registry|_version\.py)
10+
- id: end-of-file-fixer
11+
files: ^nipyapi/.*\.py$
12+
exclude: ^nipyapi/(nifi|registry|_version\.py)
13+
- id: debug-statements
14+
files: ^nipyapi/.*\.py$
15+
exclude: ^nipyapi/(nifi|registry|_version\.py)
16+
17+
- repo: https://github.com/psf/black
18+
rev: 24.8.0
19+
hooks:
20+
- id: black
21+
files: ^nipyapi/.*\.py$
22+
exclude: ^nipyapi/(nifi|registry|_version\.py)
23+
24+
- repo: https://github.com/pycqa/isort
25+
rev: 5.13.2
26+
hooks:
27+
- id: isort
28+
files: ^nipyapi/.*\.py$
29+
exclude: ^nipyapi/(nifi|registry|_version\.py)
30+
31+
- repo: https://github.com/PyCQA/flake8
32+
rev: 7.3.0
33+
hooks:
34+
- id: flake8
35+
files: ^nipyapi/.*\.py$
36+
exclude: ^nipyapi/(nifi|registry|_version\.py)
37+
args: [--config=setup.cfg]
38+
39+
- repo: local
40+
hooks:
41+
- id: pylint-core-only
42+
name: pylint (core nipyapi only)
43+
entry: pylint
44+
language: system
45+
files: ^nipyapi/.*\.py$
46+
exclude: ^nipyapi/(nifi|registry|_version\.py)
47+
args: [--rcfile=pylintrc]

‎MANIFEST.in‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
include docs/authors.rst
2-
include docs/contributing.rst
3-
include docs/history.rst
41
include LICENSE
2+
include NOTICE
53
include README.rst
64
include requirements.txt
7-
include nipyapi/demo/keys/*
85

96
recursive-include nipyapi *
107
recursive-exclude * *.py[co]
118

12-
recursive-include docs *.rst conf.py Makefile make.bat
9+
# Prune docs and resources
10+
prune docs
11+
prune resources
12+
prune examples

0 commit comments

Comments
 (0)