Skip to content

Commit

Permalink
Merging all PSC commits to master
Browse files Browse the repository at this point in the history 
Merging all PSC commits to master
  • Loading branch information
@abhinb
abhinb authored Jan 5, 2024
2 parents c6ada06 + 0a63863 commit b0b7aa3
Show file tree
Hide file tree
Showing 82 changed files with 3,155 additions and 2,284 deletions.
34 changes: 31 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,32 @@ All environment variables and properties begin with the prefix `PRAVEGA_SENSOR_C
For a list of commonly-used configuration values, see the
[sample configuration files](pravega-sensor-collector/src/main/dist/conf).

#### Sample configuration properties


| Configuration Parameter | Value | Description |
|-------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| `CREATE_SCOPE` | `false` | Boolean value. |
| `ROUTING_KEY` | `routingkey1` | Pravega routing key |
| `ENABLE_PRAVEGA` | `true` | Boolean parameter. Default value = true |
| `pravega_client_auth_method` | `Bearer` | Authentication type to connect to Pravega client |
| `pravega_client_auth_loadDynamic` | `true` | Boolean parameter. Default value = true |
| `KEYCLOAK_SERVICE_ACCOUNT_FILE` | `/opt/Pravega-sensor-collector/PSC_Files/keycloak-project1.json` | Path for keycloak service account file |
| `PRAVEGA_SENSOR_COLLECTOR_ACCEL2_CLASS` | Raw File: `io.pravega.sensor.collector.file.rawfile.RawFileIngestService` <br> CSV file: `io.pravega.sensor.collector.file.csvfile.CsvFileIngestService` <br> Parquet file: `io.pravega.sensor.collector.file.parquet.ParquetFileIngestService` | Pravega sensor collector class package |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_FILE_SPEC` | `/opt/Pravega-sensor-collector/files1` | The application reads files for processing from a specified directory path |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_FILE_EXTENSION` | `parquet` | Types of file Example:- <br> Raw File: parquet <br> CSV file: csv <br> Parquet file: parquet | | |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_DATABASE_FILE` | `/opt/Pravega-sensor-collector/PSC_Files/datafile.db` | Directory path where database file gets created Example: /opt/database/databasefile.db |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_PRAVEGA_CONTROLLER_URI` | `tls://pravega-controller.foggy-nelson.ns.sdp.hop.lab.emc.com:443` | Pravega controller URI EX: Pravega Controller URI |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_SCOPE` | `scope1` | Scope name for Pravega sensor collector |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_STREAM` | `stream1` | Stream name for Pravega sensor collector |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_ROUTING_KEY` | `routingkey1` | Routing key for Pravega Sensor collector |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_DELETE_COMPLETED_FILES` | `false` | If true, PSC immediately delete the file soon after processing |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_TRANSACTION_TIMEOUT_MINUTES` | `2.0` | Timeout for each transaction. Default value is 2 minutes |
| `PRAVEGA_SENSOR_COLLECTOR_RAW1_CREATE_SCOPE` | `false` | If Pravega is on SDP, set this to `false`. Accept Boolean value. |
| `HADOOP_HOME` | `${HOME}/dev` | For windows, Hadoop requires native libraries on Windows to work properly. You can download `Winutils.exe` to fix this. <br> See [here](https://cwiki.apache.org/confluence/display/HADOOP2/WindowsProblems). Add the location of bin/winutils.exe in the parameter HADOOP_HOME. <br> **This is required only for Parquet file type not for CSV and Raw file ingestion type** |



### Install the Service

1. The only prerequisite on the target system is Java 11.
Expand Down Expand Up @@ -270,11 +296,13 @@ If DNS is not configured throughout your network, you may need to edit the /etc/
### Running as a Windows Service
1. Download winsw.exe from https://github.com/winsw/winsw/releases and rename it as PravegaSensorCollectorApp.exe.
1. Download winsw.exe from https://github.com/winsw/winsw/releases and rename it as PravegaSensorCollectorApp.exe. Save it in the same folder as [PravegaSensorCollectorApp.xml](windows-service/PravegaSensorCollectorApp.xml).
2. Modify PravegaSensorCollectorApp.xml. Check PRAVEGA_SENSOR_COLLECTOR_RAW1_PRAVEGA_CONTROLLER_URI.
2. Modify [PravegaSensorCollectorApp.xml](windows-service/PravegaSensorCollectorApp.xml). Check PRAVEGA_SENSOR_COLLECTOR_RAW1_PRAVEGA_CONTROLLER_URI.
4. Add the path to the jar file `pravega-sensor-collector/build/libs/pravega-sensor-collector-${APP_VERSION}.jar` generated by build-installer.sh in PravegaSensorCollectorApp.xml.
3. Install and run the service using following commands:
5. Install and run the service using following commands:
```
PravegaSensorCollectorApp.exe install
PravegaSensorCollectorApp.exe start
Expand Down
3 changes: 2 additions & 1 deletion build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,15 @@
* user guide available at https://docs.gradle.org/3.4.1/userguide/tutorial_java_projects.html
*/

configurations.all {
configurations.configureEach {
// Check for updates every build
resolutionStrategy.cacheChangingModulesFor 0, "seconds"
}

subprojects {
repositories {
mavenLocal()
mavenCentral()
maven {
url "https://oss.jfrog.org/jfrog-dependencies"
}
Expand Down
Loading

0 comments on commit b0b7aa3

Please sign in to comment.