diff --git a/.github/workflows/artifact-publish.yml b/.github/workflows/artifact-publish.yml index c3e8a0b96..f94d16d21 100644 --- a/.github/workflows/artifact-publish.yml +++ b/.github/workflows/artifact-publish.yml @@ -7,6 +7,9 @@ on: jobs: build: runs-on: ubuntu-latest + permissions: + contents: read + packages: write steps: - name: Checkout code (with submodules) diff --git a/README.md b/README.md index 02e1229b2..ba074aac9 100644 --- a/README.md +++ b/README.md @@ -444,6 +444,23 @@ In order to utilize Confluent Cloud: #### Note +### Port Mapped Receiver Configuration +Many production deployments may not send intersection data directly from an RSU unit. This often happens if another device is responsible for generating MAP / SPaT messages or if a proxy or load balancer is used between an RSU unit and the ODE. To handle these edge cases the ODE has a configurable port-mapped receiver which can be used for receiving data on a variety of ports. The intent of this feature is to allow the ODE to continue to identify different source RSU units based upon the port the data is forwarded on. + +To use port-mapped ingest, add an entry similar to the below to the application.yaml file. The entry below creates a BSM receiver which listens on port 1234. Any BSM data received on port 1234 will be marked as having been received from the IP 1.2.3.4. + +``` +port-mapped-ingest: + sources: + - intersectipon-name: "Name" + intersection-id: 1234 + origin-ip: 1.2.3.4 + port: 1234 + type: BSM +``` + +Please note, ports added here are not automatically opened in the host docker container. Any port added here should separately be added to the running docker container. + This has only been tested with Confluent Cloud but technically all SASL authenticated Kafka brokers can be reached using this method. [Back to top](#table-of-contents) @@ -464,6 +481,8 @@ The configuration that defines this is in the jpo-utils submodule [here](jpo-uti For further documentation on configuring the MongoDB Kafka Connect image refer [to this readme](jpo-utils/README.md). + + ## Note Kafka connect is being used for MongoDB in this implementation but it can interact with many types of databases, here is further documentation for [kafka connect](https://docs.confluent.io/platform/current/connect/index.html) diff --git a/asn1_codec b/asn1_codec index 1b6dc4c0e..a51fc0a8c 160000 --- a/asn1_codec +++ b/asn1_codec @@ -1 +1 @@ -Subproject commit 1b6dc4c0eac3a741ad88ddff4fae3a005091b408 +Subproject commit a51fc0a8c1ced0654b61667f9fcb79cf799e4db1 diff --git a/docker-compose.yml b/docker-compose.yml index c4f059187..1da358793 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -64,6 +64,7 @@ services: ODE_ROCKSDB_BLOCK_SIZE: ${ODE_ROCKSDB_BLOCK_SIZE} ODE_ROCKSDB_N_MEMTABLES: ${ODE_ROCKSDB_N_MEMTABLES} ODE_ROCKSDB_MEMTABLE_SIZE: ${ODE_ROCKSDB_MEMTABLE_SIZE} + ODE_KAFKA_LISTENER_CONCURRENCY: ${ODE_KAFKA_LISTENER_CONCURRENCY} depends_on: kafka: condition: service_healthy @@ -104,6 +105,7 @@ services: ACM_LOG_TO_CONSOLE: ${ADM_LOG_TO_CONSOLE} ACM_LOG_TO_FILE: ${ADM_LOG_TO_FILE} ACM_LOG_LEVEL: ${ADM_LOG_LEVEL} + ACM_NUMBER_OF_PROCESSES: ${ADM_NUMBER_OF_PROCESSES:-1} depends_on: kafka: condition: service_healthy @@ -137,6 +139,7 @@ services: ACM_LOG_TO_CONSOLE: ${AEM_LOG_TO_CONSOLE} ACM_LOG_TO_FILE: ${AEM_LOG_TO_FILE} ACM_LOG_LEVEL: ${AEM_LOG_LEVEL} + ACM_NUMBER_OF_PROCESSES: ${AEM_NUMBER_OF_PROCESSES:-1} depends_on: kafka: condition: service_healthy diff --git a/docs/Release_notes.md b/docs/Release_notes.md index 6e9ad8d88..1a635c6c7 100644 --- a/docs/Release_notes.md +++ b/docs/Release_notes.md @@ -1,6 +1,19 @@ JPO-ODE Release Notes ---------------------------- +Version 6.0.0, released May 2026 +---------------------------------------- +### **Summary** +This release expands support for existing jpo-ode functionality and improves flexibility in deployment and data handling. JSON schemas are now bundled within the jpo-asn-pojos submodule, enabling offline usage without external dependencies. Additionally, a receiving port remapping configuration has been introduced to better identify traffic sources that may not originate from an RSU. This release also includes several minor fixes. + +Enhancements in this release: +- [ASN Hex 1609.2 Signature Retention (ITSA)](https://github.com/usdot-jpo-ode/jpo-ode/pull/596) +- [Submodule update to provide JSON schemas in resources for offline availability](https://github.com/neaeraconsulting/jpo-ode/pull/1) +- [Port remap receiver](https://github.com/neaeraconsulting/jpo-ode/pull/2) +- [Configurable kafka concurrency](https://github.com/neaeraconsulting/jpo-ode/pull/3) +- [Docker shared Kafka fix](https://github.com/neaeraconsulting/jpo-ode/pull/9) +- [Add env vars for number of ACM processes, update test process](https://github.com/neaeraconsulting/jpo-ode/pull/7) + Version 5.1.0, released October 2025 ---------------------------------------- ### **Summary** diff --git a/docs/compatibility.md b/docs/compatibility.md index 98f6598a9..3b00a9868 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -3,6 +3,7 @@ This table serves as a guide, suggesting which versions of individual submodules | [ODE (this project)](https://github.com/usdot-jpo-ode/jpo-ode/releases) | [ACM](https://github.com/usdot-jpo-ode/asn1_codec/releases) | [PPM](https://github.com/usdot-jpo-ode/jpo-cvdp/releases) | [SEC](https://github.com/usdot-jpo-ode/jpo-security-svcs/releases) | [SDWD](https://github.com/usdot-jpo-ode/jpo-sdw-depositor/releases) | [S3D](https://github.com/usdot-jpo-ode/jpo-s3-deposit/releases) | [GJConverter](https://github.com/usdot-jpo-ode/jpo-geojsonconverter/releases) | [CMonitor](https://github.com/usdot-jpo-ode/jpo-conflictmonitor/releases) | [CVisualizer](https://github.com/usdot-jpo-ode/jpo-conflictvisualizer/releases) | [CVManager](https://github.com/usdot-jpo-ode/jpo-cvmanager/releases) | [MEC](https://github.com/usdot-jpo-ode/jpo-mec-deposit/releases) | |-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------| +| 6.0.0 | 3.3.0 | 1.6.0 | 1.7.0 | 1.10.0 | 1.7.1 | 3.3.0 | 3.2.0 | N/A | 2.1.0 | 1.1.0 | | 5.1.0 | 3.2.0 | 1.6.0 | 1.7.0 | 1.10.0 | 1.7.1 | 3.2.0 | 3.1.0 | N/A | 2.0.0 | 1.0.0 | | 4.1.2 | 3.1.0 | 1.5.0 | 1.6.0 | 1.9.1 | 1.7.1 | 2.1.0 | 2.1.0 | 1.5.0 | 1.6.0 | N/A | | 4.0.0 | 3.0.0 | 1.5.0 | 1.5.0 | 1.9.0 | 1.7.0 | 2.0.0 | 2.0.0 | 1.5.0 | 1.5.0 | N/A | diff --git a/docs/release_process_quarterly.md b/docs/release_process_quarterly.md index 4d6263a7e..8f55b2fcf 100644 --- a/docs/release_process_quarterly.md +++ b/docs/release_process_quarterly.md @@ -43,6 +43,31 @@ None - [ ] messages get decoded as expected - [ ] messages get encoded as expected +#### Details on how to do tests + +Run compilation and unit tests in Docker: From the root directory of asn1_codec issue: +``` +docker build -t acm . +docker run --env-file .env --name acm -it acm +``` +In a separate shell: +``` +docker exec -it acm bash +/build/acm_tests +``` + +where the `.env` file is configured with necessary environment variables. + +To test program startup and encoding/decoding: + +Pull the release branch into the ODE submodule, and run the ODE via `docker compose up --build -d` from the jpo-ode root directory, with the following profiles set in the `.env` file: +``` +COMPOSE_PROFILES=ode,adm,aem,kafka,kafka_setup,kafka_ui +``` +To test decoding send test messages with the `udpsender_generic.py` script and verify the messages are decoded by looking at the `topic.Asn1DecoderOutput` in the kafka UI. + +To test encoding, use the Kafka UI to produce one of the decoded messages from `topic.Asn1DecoderOutput` to `topic.Asn1EncoderInput` and check the decoded output appears on `topic.Asn1EncoderOutput`. + ### 3. Project Reference Updates & Release Creation - [ ] Merge ‘release/(year)-(quarter)’ branch into ‘master’ branch for the asn1_codec project - [ ] Create a release for the asn1_codec project from the ‘master’ branch and tag the release with the version number of the release. (e.g. asn1_codec-x.x.x) @@ -220,11 +245,13 @@ None - [ ] code compiles - gradle build - [ ] `jpo-asn-j2735-2024` subproject builds via gradle - - [ ] `jpo-asn-test-generator` subproject build via gradle + - [ ] `jpo-asn-test-generator` subproject builds via gradle + - [ ] `jpo-asn-jsonschema-generator` subproject builds via gradle - maven build: - [ ] `jpo-asn-j2735-2024` subproject builds via maven - [ ] unit tests pass - [ ] test generator command line tool can generate messages + - [ ] schema generator command line tool can generate schemas ### 3. Project Reference Updates & Release Creation - [ ] Update Gradle build version number diff --git a/jpo-asn-pojos b/jpo-asn-pojos index c7d446df1..1ff018d20 160000 --- a/jpo-asn-pojos +++ b/jpo-asn-pojos @@ -1 +1 @@ -Subproject commit c7d446df1ba24d6bf7f485ce7c112423c9cf6f26 +Subproject commit 1ff018d20a64e7f0d8f57f039252618ab9e978b4 diff --git a/jpo-ode-common/pom.xml b/jpo-ode-common/pom.xml index c01fb219c..d47f8bc29 100644 --- a/jpo-ode-common/pom.xml +++ b/jpo-ode-common/pom.xml @@ -1,18 +1,20 @@ - + 4.0.0 usdot.jpo.ode jpo-ode - 5.1.0 + 6.0.0 - + jpo-ode-common jpo-ode-common JPO ODE Common Components jar - + src/main/java/us/dot/its/jpo/ode/util/GeoUtils.java @@ -21,26 +23,26 @@ src/main/java/us/dot/its/jpo/ode/inet/* - + org.json json - 20231013 + 20231013 com.fasterxml.jackson.core jackson-databind - com.fasterxml.jackson.dataformat - jackson-dataformat-xml + com.fasterxml.jackson.dataformat + jackson-dataformat-xml - + jakarta.xml.bind jakarta.xml.bind-api 4.0.0 - + javax.websocket javax.websocket-client-api @@ -52,4 +54,4 @@ 4.0.0 - + \ No newline at end of file diff --git a/jpo-ode-core/pom.xml b/jpo-ode-core/pom.xml index c5ea2bfd1..5aca2eddd 100644 --- a/jpo-ode-core/pom.xml +++ b/jpo-ode-core/pom.xml @@ -7,7 +7,7 @@ usdot.jpo.ode jpo-ode - 5.1.0 + 6.0.0 jpo-ode-core @@ -25,12 +25,12 @@ usdot.jpo.ode jpo-ode-common - 5.1.0 + 6.0.0 usdot.jpo.ode jpo-ode-plugins - 5.1.0 + 6.0.0 org.apache.httpcomponents @@ -90,13 +90,13 @@ com.networknt json-schema-validator - 1.0.76 + 1.0.88 test usdot.jpo.asn jpo-asn-j2735-2024 - 1.1.0 + 1.2.0 diff --git a/jpo-ode-plugins/pom.xml b/jpo-ode-plugins/pom.xml index 6b1324f33..8cc564282 100644 --- a/jpo-ode-plugins/pom.xml +++ b/jpo-ode-plugins/pom.xml @@ -1,5 +1,7 @@ - + 4.0.0 jpo-ode-plugins @@ -11,7 +13,7 @@ usdot.jpo.ode jpo-ode - 5.1.0 + 6.0.0 @@ -27,7 +29,7 @@ usdot.jpo.ode jpo-ode-common - 5.1.0 + 6.0.0 ${loader.path} diff --git a/sample.env b/sample.env index 77292c198..16a1f0441 100644 --- a/sample.env +++ b/sample.env @@ -63,6 +63,9 @@ DEFAULT_SNMP_PROTOCOL= ######################### # Kafka and Confluent Cloud Properties +# While not strictly required for the ODE to operate properly. Setting this value is required to allow applications outside of the ODE's docker network to connect to Kafka. +KAFKA_BOOTSTRAP_SERVERS=${DOCKER_HOST_IP}:9092 + # The type of Kafka broker to connect to. If set to "CONFLUENT", the broker will be Confluent Cloud. Otherwise, it will be a local Kafka broker. KAFKA_TYPE= KAFKA_LINGER_MS=1 @@ -142,4 +145,11 @@ ODE_TIM_INGEST_MONITORING_INTERVAL=60 # ODE STOMP Exporter for Demo UI # Set this to false to save resources when running the application in a production environement. The StompStringExporter # is exclusively used to support the Demo Console hosted at http://localhost:8080 -ODE_STOMP_EXPORTER_ENABLED=true \ No newline at end of file +ODE_STOMP_EXPORTER_ENABLED=true + +# Kafka listener concurrency +ODE_KAFKA_LISTENER_CONCURRENCY=1 + +# Number of processes for ACM +ADM_NUMBER_OF_PROCESSES=1 +AEM_NUMBER_OF_PROCESSES=1 \ No newline at end of file