Fix OAI Harvest Bug

What's Changed

• Fix harvest dags not working. by @dkinzer in #1639

Full Changelog: v2.3.1...v2.3.2

v2.3.1 Solrcloud backups dag and full_reindex refactor

What's Changed

• Update airflow and pipfile packages by @sensei100 in #1626
• Bl 1763 refactor catalog reindex by @dkinzer in #1625
• Bump airflow-docker-dev-setup from 84a8383 to 0ec6c04 by @dependabot in #1627
• DEVO-1116: Add dag for creating collection backups. by @dkinzer in #1628
• Retry the tasks twice before failing. by @dkinzer in #1629
• Update schedule to be 6AM UTC daily which is 1AM EST daily. by @dkinzer in #1630
• BL-1763: Fix tas_groups not implemented correctly. by @dkinzer in #1631
• Automatically check against airflow constraints in circle. by @dkinzer in #1633
• BL-1763: Fix template error by @dkinzer in #1632
• BL-1763: fix error when jinja tries to look up index. by @dkinzer in #1634
• Bl 1763 fix indexing not working by @dkinzer in #1635
• BL-1763: Move tojson call inside the template. by @dkinzer in #1636
• Bl 1763 fix indexing not working 3 by @dkinzer in #1637
• BL-1763: Add retries to tasks that run in parallel. by @dkinzer in #1638

Full Changelog: v2.3.0...v2.3.1

OAI Harvest Dag updates

What's Changed

• BL-1856 Update cron schedule for oai harvest dags by @sensei100 in #1610
• BL-1856 Try updating the cron statement by @sensei100 in #1611
• BL-1856 Try importing dag from airflow by @sensei100 in #1613
• BL-1891 Remove QA dags for az and web content (no longer used) by @sensei100 in #1615
• Bump airflow-docker-dev-setup from fe78cf2 to caed221 by @dependabot in #1614
• Update documentation for catalog reindex process by @ebtoner in #1612
• BL-1893 Use logical date rather than data_start_interval or deprecate… by @sensei100 in #1616
• BL-1856 Update the dag_ids for the OAI harvest dags by @sensei100 in #1617
• Update pip dependencies by @sensei100 in #1619
• Bump airflow-docker-dev-setup from caed221 to 09c3bd0 by @dependabot in #1618
• BL-1878: Part 2: Clear catalog caches at end of oai harvest. by @dkinzer in #1620
• BL-1894: Only send success alert when last task completes. by @dkinzer in #1621
• BL-1895: Allow pre oai harvest dag to handle no records. by @dkinzer in #1622
• BL-1897: Allow production oai harvest to handle no records. by @dkinzer in #1623
• Bump airflow-docker-dev-setup from 09c3bd0 to 84a8383 by @dependabot in #1624

Full Changelog: v2.2.3...v2.3.0

v2.2.3

What's Changed

• Bump airflow-docker-dev-setup from 9df9c0b to b8fb114 by @dependabot in #1566
• Bump jinja2 from 3.1.3 to 3.1.4 by @dependabot in #1565
• Update airflow and dependencies by @sensei100 in #1568
• Bump idna from 3.6 to 3.7 by @dependabot in #1555
• Bump pylint from 3.1.1 to 3.2.4 by @dependabot in #1594
• Bump setuptools from 69.5.1 to 70.1.1 by @dependabot in #1593
• Python 3.12.3 by @cdoyle-temple in #1597
• DEVO-1094 by @sensei100 in #1600
• Try adding url to script by @sensei100 in #1601
• Try using connection for base url by @sensei100 in #1602
• Update SimpleHttpOperator to HttpOperator by @sensei100 in #1603
• DEVO-1100 Try adding timeout variable by @sensei100 in #1605
• Add variable to task by @sensei100 in #1606
• Go back to previous web content code by @sensei100 in #1607
• Update airflow version to 2.9.3 by @sensei100 in #1608
• Bump airflow-docker-dev-setup from b8fb114 to f1b8c8e by @dependabot in #1604
• Bump airflow-docker-dev-setup from f1b8c8e to fe78cf2 by @dependabot in #1609

Full Changelog: v2.2.2...v2.2.3

Airflow updates

What's Changed

• Bump boto3 from 1.33.11 to 1.34.2 by @dependabot in #1440
• Bump moto from 4.2.11 to 4.2.12 by @dependabot in #1439
• Update README documentation by @sensei100 in #1445
• Bump pylint from 3.0.2 to 3.0.3 by @dependabot in #1431
• Bump paramiko from 3.3.1 to 3.4.0 by @dependabot in #1444
• Bump tulflow from 0.9.6 to 0.9.7 by @dependabot in #1443
• DEVO-956 Upgrade to Python 3.10 by @nomadicoder in #1446
• Bump boto3 from 1.34.5 to 1.34.6 by @dependabot in #1447
• Bump boto3 from 1.34.6 to 1.34.11 by @dependabot in #1459
• Bump pytest from 7.4.3 to 7.4.4 by @dependabot in #1458
• Bump setuptools from 69.0.2 to 69.0.3 by @dependabot in #1449
• Bump apache-airflow-providers-slack from 8.5.0 to 8.5.1 by @dependabot in #1451
• Bump apache-airflow-providers-ssh from 3.9.0 to 3.10.0 by @dependabot in #1453
• Bump apache-airflow-providers-sftp from 4.8.0 to 4.8.1 by @dependabot in #1454
• Bump apache-airflow-providers-amazon from 8.13.0 to 8.15.0 by @dependabot in #1457
• Bump numpy from 1.26.2 to 1.26.3 by @dependabot in #1460
• Bump moto from 4.2.12 to 4.2.13 by @dependabot in #1464
• Bump boto3 from 1.34.11 to 1.34.15 by @dependabot in #1465
• Bump apache-airflow-providers-amazon from 8.15.0 to 8.16.0 by @dependabot in #1466
• Bump boto3 from 1.34.15 to 1.34.16 by @dependabot in #1467
• Bump boto3 from 1.34.16 to 1.34.23 by @dependabot in #1474
• Bump pandas from 2.1.4 to 2.2.0 by @dependabot in #1475
• Bump boto3 from 1.34.23 to 1.34.25 by @dependabot in #1476
• Bump boto3 from 1.34.25 to 1.34.26 by @dependabot in #1477
• Bump boto3 from 1.34.26 to 1.34.27 by @dependabot in #1478
• Bump boto3 from 1.34.27 to 1.34.28 by @dependabot in #1479
• Bump apache-airflow-providers-http from 4.8.0 to 4.9.0 by @dependabot in #1480
• Bump apache-airflow-providers-amazon from 8.16.0 to 8.17.0 by @dependabot in #1482
• Bump pytest from 7.4.4 to 8.0.0 by @dependabot in #1484
• Bump apache-airflow-providers-slack from 8.5.1 to 8.6.0 by @dependabot in #1485
• Updates python pacakges by @sensei100 in #1511
• Bump boto3 from 1.34.44 to 1.34.45 by @dependabot in #1513
• Update airflow to 2.8.1 by @sensei100 in #1514
• Bump airflow-docker-dev-setup from 7999c71 to 9df9c0b by @dependabot in #1515
• Bump tulflow from 0.9.7 to 0.9.8 by @dependabot in #1516
• Dependabot experiment by @sensei100 in #1518
• Move requirements file to root level by @sensei100 in #1519
• Bump werkzeug from 2.2.3 to 2.3.8 by @dependabot in #1517
• Try adding quotes to constraints url by @sensei100 in #1520
• Remove requirements files by @sensei100 in #1522
• Update dependencies by @sensei100 in #1536
• Bump tulflow from 0.9.8 to 0.9.9 by @dependabot in #1538
• Bump setuptools from 69.1.1 to 69.2.0 by @dependabot in #1537
• Bump s3fs from 2024.2.0 to 2024.3.1 by @dependabot in #1543
• Bump moto from 5.0.2 to 5.0.3 by @dependabot in #1540
• Bump apache-airflow-providers-amazon from 8.18.0 to 8.19.0 by @dependabot in #1533
• Revert "Bump s3fs from 2024.2.0 to 2024.3.1" by @cdoyle-temple in #1550

New Contributors

• @cdoyle-temple made their first contribution in #1550

Full Changelog: v2.2.1...v2.2.2

Airflow upgrade

What's Changed

• Bump apache-airflow-providers-amazon from 7.4.0 to 7.4.1 by @dependabot in #1131
• Bump pytest from 7.3.0 to 7.3.1 by @dependabot in #1136
• Bump sqlparse from 0.4.3 to 0.4.4 by @dependabot in #1143
• Bump setuptools from 67.6.1 to 67.7.2 by @dependabot in #1145
• Bump pandas from 2.0.0 to 2.0.1 by @dependabot in #1146
• Bump numpy from 1.24.2 to 1.24.3 by @dependabot in #1147
• Bump importlib-metadata from 6.1.0 to 6.6.0 by @dependabot in #1148
• Bump moto from 4.1.7 to 4.1.8 by @dependabot in #1149
• Bump boto3 from 1.26.105 to 1.26.123 by @dependabot in #1157
• Bump pylint from 2.17.2 to 2.17.3 by @dependabot in #1153
• Bump flask from 2.2.4 to 2.2.5 by @dependabot in #1159
• Bump boto3 from 1.26.123 to 1.26.126 by @dependabot in #1161
• Updates airflow and fixes tests that were failing by @sensei100 in #1170
• Bump boto3 from 1.26.133 to 1.26.134 by @dependabot in #1172
• BL-1776 Update electronic notes dag to run daily by @sensei100 in #1173
• The next airflow update fails pylint, this allows for more ancestors by @sensei100 in #1174
• Bump boto3 from 1.26.134 to 1.26.135 by @dependabot in #1175
• Bump boto3 from 1.26.135 to 1.26.136 by @dependabot in #1176
• Bump apache-airflow from 2.5.3 to 2.6.1 by @dependabot in #1171
• Bump tulflow from 0.9.1 to 0.9.2 by @dependabot in #1177
• Bump moto from 4.1.9 to 4.1.10 by @dependabot in #1178
• Bump setuptools from 67.7.2 to 67.8.0 by @dependabot in #1180
• Bump requests from 2.30.0 to 2.31.0 by @dependabot in #1181
• Bump apache-airflow-providers-amazon from 8.0.0 to 8.1.0 by @dependabot in #1182
• Bump apache-airflow-providers-ssh from 3.6.0 to 3.7.0 by @dependabot in #1183
• Bump apache-airflow-providers-sftp from 4.2.4 to 4.3.0 by @dependabot in #1185
• Bump paramiko from 3.1.0 to 3.2.0 by @dependabot in #1189
• Bump apache-airflow-providers-slack from 7.2.0 to 7.3.0 by @dependabot in #1186
• Bump boto3 from 1.26.136 to 1.26.142 by @dependabot in #1191
• Bump apache-airflow-providers-ftp from 3.3.1 to 3.4.1 by @dependabot in #1192
• Bump pandas from 2.0.1 to 2.0.2 by @dependabot in #1193
• Bump apache-airflow-providers-http from 4.3.0 to 4.4.1 by @dependabot in #1194
• Bump boto3 from 1.26.142 to 1.26.143 by @dependabot in #1195
• Bump airflow-docker-dev-setup from fcaab77 to 1bbdf17 by @dependabot in #1197
• Bump boto3 from 1.26.143 to 1.26.145 by @dependabot in #1198
• Bump moto from 4.1.10 to 4.1.11 by @dependabot in #1199
• Bump boto3 from 1.26.145 to 1.26.147 by @dependabot in #1201
• Bump requests-mock from 1.10.0 to 1.11.0 by @dependabot in #1204
• Bump boto3 from 1.26.147 to 1.26.150 by @dependabot in #1205
• Bump boto3 from 1.26.150 to 1.26.151 by @dependabot in #1206
• Bump pytest from 7.3.1 to 7.3.2 by @dependabot in #1207
• Bump boto3 from 1.26.151 to 1.26.152 by @dependabot in #1208
• Bump boto3 from 1.26.152 to 1.26.153 by @dependabot in #1209
• Bump boto3 from 1.26.153 to 1.26.154 by @dependabot in #1210
• Bump boto3 from 1.26.154 to 1.26.157 by @dependabot in #1217
• Bump boto3 from 1.26.157 to 1.26.159 by @dependabot in #1220
• Bump pytest from 7.3.2 to 7.4.0 by @dependabot in #1219
• Bump apache-airflow-providers-ssh from 3.7.0 to 3.7.1 by @dependabot in #1222
• Bump apache-airflow-providers-slack from 7.3.0 to 7.3.1 by @dependabot in #1223
• Bump apache-airflow-providers-ftp from 3.4.1 to 3.4.2 by @dependabot in #1225
• Bump apache-airflow from 2.6.1 to 2.6.3 by @dependabot in #1241
• Bump pandas from 2.0.2 to 2.0.3 by @dependabot in #1231
• Bump apache-airflow-providers-http from 4.4.1 to 4.5.0 by @dependabot in #1246
• Bump moto from 4.1.11 to 4.1.14 by @dependabot in #1259
• Bump apache-airflow-providers-amazon from 8.1.0 to 8.4.0 by @dependabot in #1263
• Bump apache-airflow-providers-sftp from 4.3.0 to 4.5.0 by @dependabot in #1264
• Bump importlib-metadata from 6.6.0 to 6.8.0 by @dependabot in #1237
• Bump boto3 from 1.26.159 to 1.28.20 by @dependabot in #1268
• Bump airflow-docker-dev-setup from 1bbdf17 to d13278d by @dependabot in #1269
• Bump boto3 from 1.28.20 to 1.28.21 by @dependabot in #1270
• Bump apache-airflow-providers-slack from 7.3.1 to 7.3.2 by @dependabot in #1271
• Bump tulflow from 0.9.2 to 0.9.3 by @dependabot in #1272
• Bump numpy from 1.24.3 to 1.25.2 by @dependabot in #1273
• Bump apache-airflow-providers-ftp from 3.4.2 to 3.5.0 by @dependabot in #1276
• Bump pylint from 2.17.4 to 2.17.5 by @dependabot in #1277
• Bump paramiko from 3.2.0 to 3.3.1 by @dependabot in #1278
• Bump apache-airflow-providers-amazon from 8.4.0 to 8.5.1 by @dependabot in #1282
• Bump setuptools from 67.8.0 to 68.1.0 by @dependabot in #1286
• Bump boto3 from 1.28.21 to 1.28.28 by @dependabot in #1288
• Bump airflow-docker-dev-setup from d13278d to 61b051c by @dependabot in #1289
• Bump setuptools from 68.1.0 to 68.1.2 by @dependabot in #1291
• BL-1788 Update/add documentation for reindex dags by @sensei100 in #1283
• Bump boto3 from 1.28.28 to 1.28.62 by @dependabot in #1346
• Bump moto from 4.1.14 to 4.2.6 by @dependabot in #1348
• DEVO-890 Update slack integration by @sensei100 in #1357
• Try adding just the ti.xcom for slack messages by @sensei100 in #1358
• Try importing json by @sensei100 in #1359
• Try using key for xcom_pull by @sensei100 in #1360
• Refine key in slack text by @sensei100 in #1361
• Try using json outside the template by @sensei100 in #1362
• Fix template syntax by @sensei100 in #1363
• Bump pytest from 7.4.2 to 7.4.3 by @dependabot in #1366
• Bump pandas from 2.1.1 to 2.1.2 by @dependabot in #1368
• Bump boto3 from 1.28.69 to 1.28.73 by @dependabot in https://github....

v2.2.0

What's Changed

• DEVO-666 Add wrapt package (needed for dependabot) by @sensei100 in #1018
• Add tomli package by @sensei100 in #1019
• Set python_version to 3.9 by @sensei100 in #1020
• Bump boto3 from 1.26.53 to 1.26.58 by @dependabot in #1030
• Update submodule to use https by @sensei100 in #1047
• Bump tulflow from 0.9.0 to 0.9.1 by @dependabot in #1027
• Bump moto from 4.1.0 to 4.1.2 by @dependabot in #1040
• Bump numpy from 1.24.1 to 1.24.2 by @dependabot in #1041
• Bump cryptography from 39.0.0 to 39.0.1 by @dependabot in #1045
• Bump pylint from 2.15.10 to 2.16.2 by @dependabot in #1053
• Bump setuptools from 66.1.0 to 67.3.1 by @dependabot in #1055
• Bump apache-airflow-providers-amazon from 7.1.0 to 7.2.0 by @dependabot in #1050
• Bump apache-airflow-providers-sftp from 4.2.1 to 4.2.2 by @dependabot in #1051
• Bump apache-airflow-providers-ftp from 3.3.0 to 3.3.1 by @dependabot in #1057
• Bump setuptools from 67.3.1 to 67.3.2 by @dependabot in #1059
• Bump paramiko from 2.12.0 to 3.0.0 by @dependabot in #1023
• Bump moto from 4.1.2 to 4.1.3 by @dependabot in #1062
• Bump apache-airflow-providers-sftp from 4.2.2 to 4.2.3 by @dependabot in #1066
• Bump apache-airflow-providers-amazon from 7.2.0 to 7.2.1 by @dependabot in #1067
• Bump apache-airflow-providers-http from 4.1.1 to 4.2.0 by @dependabot in #1070
• Bump pytest from 7.2.1 to 7.2.2 by @dependabot in #1082
• Bump moto from 4.1.3 to 4.1.4 by @dependabot in #1079
• Bump apache-airflow-providers-sftp from 4.2.3 to 4.2.4 by @dependabot in #1083
• Bump apache-airflow-providers-amazon from 7.2.1 to 7.3.0 by @dependabot in #1085
• Bump apache-airflow-providers-ssh from 3.4.0 to 3.5.0 by @dependabot in #1086
• Bump setuptools from 67.3.2 to 67.6.0 by @dependabot in #1090
• Bump paramiko from 3.0.0 to 3.1.0 by @dependabot in #1094
• Bump importlib-metadata from 6.0.0 to 6.1.0 by @dependabot in #1099
• Bump moto from 4.1.4 to 4.1.5 by @dependabot in #1100
• Bump pylint from 2.16.2 to 2.17.1 by @dependabot in #1104
• Bump moto from 4.1.5 to 4.1.6 by @dependabot in #1108
• Bump setuptools from 67.6.0 to 67.6.1 by @dependabot in #1109
• Bump boto3 from 1.26.58 to 1.26.102 by @dependabot in #1112
• bump boto3 from 1.26.102 to 1.26.103 by @dependabot in #1113
• Bump pylint from 2.17.1 to 2.17.2 by @dependabot in #1115
• Bump pandas from 1.5.3 to 2.0.0 by @dependabot in #1117
• Bump boto3 from 1.26.103 to 1.26.105 by @dependabot in #1118
• Bump pytest from 7.2.2 to 7.3.0 by @dependabot in #1126
• Bump apache-airflow-providers-ssh from 3.5.0 to 3.6.0 by @dependabot in #1121
• Bump apache-airflow-providers-amazon from 7.3.0 to 7.4.0 by @dependabot in #1120
• Bump apache-airflow-providers-http from 4.2.0 to 4.3.0 by @dependabot in #1123
• Bump moto from 4.1.6 to 4.1.7 by @dependabot in #1130
• add collection name to DSpace transform by @htomren in #1140

Full Changelog: v2.1.1...v2.2.0

Update schedule intervals

What's Changed

• DEVO-571 Fix schedule_interval deprecation warning by @sensei100 in #979
• Update ssh key for circleCI by @sensei100 in #1009
• DEVO-681 Update ssh key for circleCI by @sensei100 in #1011

Full Changelog: v2.1.0...v2.1.1

DAG refactors for Airflow upgrade

What's Changed

• BL-1708 Import pendulum in all dags by @sensei100 in #925
• BL-1708 Use pendulum.now() instead of data_interval_start by @sensei100 in #926
• Update schedule to weekly by @sensei100 in #929
• BL-1707: Fix wrong permission in created sftp folder. by @dkinzer in #932
• Bump boto3 from 1.24.89 to 1.25.5 by @dependabot in #924
• Update python dependencies by @sensei100 in #970
• Bump airflow-docker-dev-setup from ec633b0 to fcaab77 by @dependabot in #972

Full Changelog: v2.0.2...v2.1.0

Update deployment process

What's Changed

• Bump airflow-docker-dev-setup from be23740 to bdfae3e by @dependabot in #891
• Update airflow version due to security vulnerability by @sensei100 in #906
• Bump airflow-docker-dev-setup from bdfae3e to ec633b0 by @dependabot in #909
• Try to only deploy dag-related tasks by @sensei100 in #910
• Bump apache-airflow from 2.4.1 to 2.4.2 by @dependabot in #916

Full Changelog: v2.0.1...v2.0.2