Skip to content

Commit 4ed945b

Browse files
priyabhatnagar25Ishakaushik-cloud
priyabhatnagar25
authored and
Ishakaushik-cloud
committed
bq validation
1 parent 86d084d commit 4ed945b

22 files changed

+528
-161
lines changed

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTimeValidation.feature‎

Lines changed: 29 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,19 @@
1-
Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scenarios
1+
# Copyright © 2023 Cask Data, Inc.
2+
##
3+
## Licensed under the Apache License, Version 2.0 (the "License"); you may not
4+
## use this file except in compliance with the License. You may obtain a copy of
5+
## the License at
6+
##
7+
## http://www.apache.org/licenses/LICENSE-2.0
8+
##
9+
## Unless required by applicable law or agreed to in writing, software
10+
## distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
## WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
## License for the specific language governing permissions and limitations under
13+
# the License..
14+
15+
@CloudMySql
16+
Feature: CloudMySql sink- Verify CloudsqlMysql sink plugin design time validation scenarios
217

318
Scenario: To verify CloudMySql sink plugin validation error message with invalid database
419
Given Open Datafusion Project to configure pipeline
@@ -10,15 +25,15 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
1025
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
1126
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
1227
Then Select radio button plugin property: "instanceType" with value: "public"
13-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
28+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
1429
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
1530
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
1631
Then Enter input plugin property: "referenceName" with value: "RefName"
1732
Then Enter input plugin property: "database" with value: "TestDatabase"
1833
Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
19-
# Then Click on the Get Schema button
20-
# Then Verify the Output Schema matches the Expected Schema: "outputSchema"
21-
# Then Validate "CloudSQL MySQL" plugin properties
34+
Then Click on the Get Schema button
35+
Then Verify the Output Schema matches the Expected Schema: "outputSchema"
36+
Then Validate "CloudSQL MySQL" plugin properties
2237
Then Close the Plugin Properties page
2338
Then Navigate to the properties page of plugin: "CloudSQL MySQL2"
2439
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
@@ -30,7 +45,7 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
3045
Then Enter input plugin property: "database" with value: "invalidDatabaseName"
3146
Then Enter input plugin property: "tableName" with value: "mytable"
3247
Then Click on the Validate button
33-
# Then Verify that the Plugin is displaying an error message: "errorMessageInvalidSinkDatabase" on the header
48+
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidSinkDatabase" on the header
3449

3550
Scenario: To verify CloudMySql sink plugin validation error message with invalid tablename
3651
Given Open Datafusion Project to configure pipeline
@@ -42,26 +57,26 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
4257
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
4358
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
4459
Then Select radio button plugin property: "instanceType" with value: "public"
45-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
60+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
4661
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
4762
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
4863
Then Enter input plugin property: "referenceName" with value: "RefName"
4964
Then Enter input plugin property: "database" with value: "TestDatabase"
5065
Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
51-
# Then Click on the Get Schema button
52-
# Then Verify the Output Schema matches the Expected Schema: "outputSchema"
53-
# Then Validate "CloudSQL MySQL" plugin properties
66+
Then Click on the Get Schema button
67+
Then Verify the Output Schema matches the Expected Schema: "outputSchema"
68+
Then Validate "CloudSQL MySQL" plugin properties
5469
Then Close the Plugin Properties page
5570
Then Navigate to the properties page of plugin: "CloudSQL MySQL2"
5671
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
5772
Then Select radio button plugin property: "instanceType" with value: "public"
58-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
73+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
5974
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
6075
Then Enter input plugin property: "referenceName" with value: "invalidRefName"
6176
Then Enter input plugin property: "database" with value: "TestDatabase"
6277
Then Enter input plugin property: "tableName" with value: "Invalidtable"
6378
Then Click on the Validate button
64-
# Then Verify that the Plugin Property: "table" is displaying an in-line error message: "errorMessageInvalidTableName"
79+
Then Verify that the Plugin Property: "table" is displaying an in-line error message: "errorMessageInvalidTableName"
6580

6681
Scenario: To verify CloudMySql sink plugin validation error message with invalid reference Name
6782
Given Open Datafusion Project to configure pipeline
@@ -70,7 +85,7 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
7085
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
7186
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
7287
Then Select radio button plugin property: "instanceType" with value: "public"
73-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
88+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
7489
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
7590
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
7691
Then Enter input plugin property: "referenceName" with value: "invalidRef"
@@ -87,7 +102,7 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
87102
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
88103
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
89104
Then Select radio button plugin property: "instanceType" with value: "public"
90-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
105+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
91106
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
92107
Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
93108
Then Enter input plugin property: "referenceName" with value: "RefName"

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTimeWithMacro.feature‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,18 @@
1+
# Copyright © 2023 Cask Data, Inc.
2+
##
3+
## Licensed under the Apache License, Version 2.0 (the "License"); you may not
4+
## use this file except in compliance with the License. You may obtain a copy of
5+
## the License at
6+
##
7+
## http://www.apache.org/licenses/LICENSE-2.0
8+
##
9+
## Unless required by applicable law or agreed to in writing, software
10+
## distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
## WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
## License for the specific language governing permissions and limitations under
13+
# the License..
14+
15+
@CloudMySql
116
Feature: CloudMySql sink- Verify CloudMySql sink plugin design time macro scenarios
217

318
Scenario: To verify CloudMySql sink plugin validation with macro enabled fields for connection section
@@ -6,15 +21,15 @@ Feature: CloudMySql sink- Verify CloudMySql sink plugin design time macro scenar
621
When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
722
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
823
Then Click on the Macro button of Property: "jdbcPluginName" and set the value to: "cloudsql-mysql"
9-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
24+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
1025
Then Select radio button plugin property: "instanceType" with value: "public"
1126
Then Click on the Macro button of Property: "user" and set the value to: "username"
1227
Then Click on the Macro button of Property: "password" and set the value to: "password"
1328
Then Click on the Macro button of Property: "connectionArguments" and set the value to: "connectionArgumentsList"
1429
Then Enter input plugin property: "referenceName" with value: "sourceRef"
1530
Then Replace input plugin property: "database" with value: "TestDatabase"
1631
Then Click on the Validate button
17-
# Then Validate "CloudSQL MySQL" plugin properties
32+
Then Validate "CloudSQL MySQL" plugin properties
1833
Then Close the Plugin Properties page
1934

2035
Scenario: To verify cloudsql sink plugin validation with macro enabled fields for basic section
@@ -23,13 +38,13 @@ Feature: CloudMySql sink- Verify CloudMySql sink plugin design time macro scenar
2338
When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
2439
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
2540
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
26-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
41+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
2742
Then Select radio button plugin property: "instanceType" with value: "public"
2843
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
2944
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
3045
Then Enter input plugin property: "referenceName" with value: "sourceRef"
3146
Then Replace input plugin property: "database" with value: "TestDatabase"
3247
Then Click on the Macro button of Property: "tableName" and set the value to: "mytable"
3348
Then Click on the Validate button
34-
# Then Validate "CloudSQL MySQL" plugin properties
49+
Then Validate "CloudSQL MySQL" plugin properties
3550
Then Close the Plugin Properties page

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature‎

Lines changed: 94 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
@CloudMySql
1616
Feature: CloudMySql Sink - Run time scenarios
1717

18-
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE
18+
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
1919
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully
2020
Given Open Datafusion Project to configure pipeline
2121
When Expand Plugin group in the LHS plugins list: "Source"
@@ -24,7 +24,7 @@ Feature: CloudMySql Sink - Run time scenarios
2424
And Enter input plugin property: "referenceName" with value: "Reference"
2525
And Replace input plugin property: "project" with value: "projectId"
2626
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
27-
And Enter input plugin property: "dataset" with value: "dataset"
27+
And Replace input plugin property: "dataset" with value: "dataset"
2828
And Enter input plugin property: "table" with value: "bqSourceTable"
2929
Then Click on the Get Schema button
3030
Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
@@ -36,7 +36,7 @@ Feature: CloudMySql Sink - Run time scenarios
3636
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
3737
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
3838
Then Select radio button plugin property: "instanceType" with value: "public"
39-
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
39+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
4040
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
4141
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
4242
Then Enter input plugin property: "referenceName" with value: "RefName"
@@ -57,7 +57,98 @@ Feature: CloudMySql Sink - Run time scenarios
5757
Then Open and capture logs
5858
Then Verify the pipeline status is "Succeeded"
5959
Then Close the pipeline logs
60+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
6061

62+
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
63+
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully when connection arguments are set
64+
Given Open Datafusion Project to configure pipeline
65+
When Expand Plugin group in the LHS plugins list: "Source"
66+
When Select plugin: "BigQuery" from the plugins list as: "Source"
67+
And Navigate to the properties page of plugin: "BigQuery"
68+
And Enter input plugin property: "referenceName" with value: "Reference"
69+
And Replace input plugin property: "project" with value: "projectId"
70+
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
71+
And Replace input plugin property: "dataset" with value: "dataset"
72+
And Enter input plugin property: "table" with value: "bqSourceTable"
73+
Then Click on the Get Schema button
74+
Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
75+
Then Validate "BigQuery" plugin properties
76+
And Close the Plugin Properties page
77+
When Expand Plugin group in the LHS plugins list: "Sink"
78+
When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
79+
Then Connect plugins: "BigQuery" and "CloudSQL MySQL" to establish connection
80+
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
81+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
82+
Then Select radio button plugin property: "instanceType" with value: "public"
83+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
84+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
85+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
86+
Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
87+
Then Enter input plugin property: "referenceName" with value: "RefName"
88+
Then Enter input plugin property: "database" with value: "DatabaseName"
89+
Then Replace input plugin property: "tableName" with value: "targetTable"
90+
Then Validate "CloudSQL MySQL" plugin properties
91+
Then Close the Plugin Properties page
92+
Then Save the pipeline
93+
Then Preview and run the pipeline
94+
Then Wait till pipeline preview is in running state
95+
Then Open and capture pipeline preview logs
96+
Then Verify the preview run status of pipeline in the logs is "succeeded"
97+
Then Close the pipeline logs
98+
Then Close the preview
99+
Then Deploy the pipeline
100+
Then Run the Pipeline in Runtime
101+
Then Wait till pipeline is in running state
102+
Then Open and capture logs
103+
Then Verify the pipeline status is "Succeeded"
104+
Then Close the pipeline logs
105+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
106+
107+
108+
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
109+
Scenario: To verify data is getting transferred from BigQuery source to CloudSMySQL sink with Advanced property Connection timeout
110+
Given Open Datafusion Project to configure pipeline
111+
When Expand Plugin group in the LHS plugins list: "Source"
112+
When Select plugin: "BigQuery" from the plugins list as: "Source"
113+
And Navigate to the properties page of plugin: "BigQuery"
114+
And Enter input plugin property: "referenceName" with value: "Reference"
115+
And Replace input plugin property: "project" with value: "projectId"
116+
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
117+
And Replace input plugin property: "dataset" with value: "dataset"
118+
And Enter input plugin property: "table" with value: "bqSourceTable"
119+
Then Click on the Get Schema button
120+
Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
121+
Then Validate "BigQuery" plugin properties
122+
And Close the Plugin Properties page
123+
When Expand Plugin group in the LHS plugins list: "Sink"
124+
When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
125+
Then Connect plugins: "BigQuery" and "CloudSQL MySQL" to establish connection
126+
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
127+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
128+
Then Select radio button plugin property: "instanceType" with value: "public"
129+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
130+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
131+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
132+
Then Enter input plugin property: "referenceName" with value: "RefName"
133+
Then Enter input plugin property: "database" with value: "DatabaseName"
134+
Then Replace input plugin property: "tableName" with value: "targetTable"
135+
Then Replace input plugin property: "connectionTimeout" with value: "connectionTimeout"
136+
Then Validate "CloudSQL MySQL" plugin properties
137+
Then Close the Plugin Properties page
138+
Then Save the pipeline
139+
Then Preview and run the pipeline
140+
Then Wait till pipeline preview is in running state
141+
Then Open and capture pipeline preview logs
142+
Then Verify the preview run status of pipeline in the logs is "succeeded"
143+
Then Close the pipeline logs
144+
Then Close the preview
145+
Then Deploy the pipeline
146+
Then Run the Pipeline in Runtime
147+
Then Wait till pipeline is in running state
148+
Then Open and capture logs
149+
Then Verify the pipeline status is "Succeeded"
150+
Then Close the pipeline logs
151+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
61152

62153

63154

0 commit comments

Comments
 (0)