Exception: Cannot create JSON file - 'NoneType' object has no attribute 'get'

[kmyorgy]

Subject of the issue

I am creating a Snowflake tap with a Postgres target, and getting the below error when I run the tap. This is a new pipeline; I have had one working in the other direction for several years now. (Postgres to Snowflake)

pipelinewise run_tap --tap snowflake_tap --target rds_postgres --debug

Error:
time=2023-01-05 18:10:23 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:90 log_level=DEBUG message=Parsing file at /root/.pipelinewise/rds_postgres/snowflake_tap/properties.json
time=2023-01-05 18:10:23 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:95 log_level=DEBUG message=No file at /root/.pipelinewise/rds_postgres/snowflake_tap/properties.json
time=2023-01-05 18:10:23 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:90 log_level=DEBUG message=Parsing file at /root/.pipelinewise/rds_postgres/snowflake_tap/state.json
time=2023-01-05 18:10:23 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:95 log_level=DEBUG message=No file at /root/.pipelinewise/rds_postgres/snowflake_tap/state.json
Traceback (most recent call last):
File "/app/pipelinewise/cli/pipelinewise.py", line 155, in create_filtered_tap_properties
for stream_idx, stream in enumerate(properties.get('streams', tap_properties)):
AttributeError: 'NoneType' object has no attribute 'get'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/app/.virtualenvs/pipelinewise/bin/pipelinewise", line 33, in
sys.exit(load_entry_point('pipelinewise', 'console_scripts', 'pipelinewise')())
File "/app/pipelinewise/cli/init.py", line 230, in main
getattr(ppw_instance, args.command)()
File "/app/pipelinewise/cli/pipelinewise.py", line 985, in run_tap
create_fallback=True)
File "/app/pipelinewise/cli/pipelinewise.py", line 271, in create_filtered_tap_properties
raise Exception(f'Cannot create JSON file - {exc}') from exc
Exception: Cannot create JSON file - 'NoneType' object has no attribute 'get'

Your environment

• Version of PipelineWise, e.g branch/commit #/release/tag -
  --- 2f699159f73a I am running pipelinewise in a docker container. This image was created 23 months ago. Please advise if there is a better way to show the version.

• Source type and setup: Snowflake
  General Properties

id: "snowflake_tap" # Unique identifier of the tap
name: "Snowflake Database Tap" # Name of the tap
type: "tap-snowflake" # !! THIS SHOULD NOT CHANGE !!
owner: "" # Data owner to contact
#send_alert: False # Optional: Disable all configured alerts on this tap

Source (Tap) - Snowflake connection details
db_conn:
account: "xxxxxx.us-east-1" # Snowflake host
dbname: "xxxx" # Snowflake database name
user: "xxxxxx" # Snowflake user
password: "xxxxxx" # Plain string or vault encrypted
warehouse: "XXX_WAREHOUSE" # Snowflake virtual warehouse

Destination (Target) - Target properties
target: "rds_postgres" # ID of the target connector where the data will be loaded
batch_size_rows: 20000 # Batch size for the stream to optimise load performance
stream_buffer_size: 0 # In-memory buffer size (MB) between taps and targets for asynchronous data pipes

Source to target Schema mapping
schemas:

• source_schema: "xxxx_PRD" # Source schema (aka. database) in Snowflake with tables
  target_schema: "public" # Target schema in the destination Data Warehouse

List of tables to replicate from Snowflake to a destination
- table_name: "HEALTH_REPORT_THURSDAY"

• Target type and setup: Postgres

General Properties
id: "rds_postgres" # Unique identifier of the target
name: "Postgres Data Warehouse" # Name of the target
type: "target-postgres" # !! THIS SHOULD NOT CHANGE !!

Target - Data Warehouse connection details
db_conn:
host: "xxxxs.xxxxxxxxxxxxxxxxx.us-east-1.rds.amazonaws.com" # Postgres host
port: 5432 # Postgres port
user: "xxxx" # Postgres user
password: "xxxxx" # Plain string or vault encrypted
dbname: "xxxx" # Postgres database name

Steps to reproduce

pipelinewise run_tap --tap snowflake_tap --target rds_postgres

Expected behaviour

pipeline should run

Actual behaviour

Get the above error.

Further notes

There's another issue with a similar error; in this case the user had to change versions. I am running pipelinewise in a docker container.

Since I posted this, I spun up a new instance / copy of my server, and ran: docker build -t pipelinewise:latest .
Docker Image:
Then: Recreated the tap and target YAML files, imported them, and ran the pipeline. Again, received same error regarding the tap's properties.json and state.json files being missing.

time=2023-01-06 13:47:25 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:90 log_level=DEBUG message=Parsing file at /root/.pipelinewise/rds_postgres/snowflake_tap/properties.json
time=2023-01-06 13:47:25 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:95 log_level=DEBUG message=No file at /root/.pipelinewise/rds_postgres/snowflake_tap/properties.json
time=2023-01-06 13:47:25 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:90 log_level=DEBUG message=Parsing file at /root/.pipelinewise/rds_postgres/snowflake_tap/state.json
time=2023-01-06 13:47:25 logger_name=pipelinewise.cli.utils file=/app/pipelinewise/cli/utils.py:95 log_level=DEBUG message=No file at /root/.pipelinewise/rds_postgres/snowflake_tap/state.json

Thanks!
Karen