Checking of column has empty array #1070
                  
                    
                      mosharafMS
                    
                  
                
                  started this conversation in
                General
              
            Replies: 1 comment
-
| solved it! found a When function in the Microsoft.Spark.Sql.Functions class. This way, I'll use identical way to pyspark | 
Beta Was this translation helpful? Give feedback.
                  
                    0 replies
                  
                
            
  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment
  
        
    
Uh oh!
There was an error while loading. Please reload this page.
-
In pyspark when having an array column, I can check if the array Size is 0 and replace the column with null value like this
.withColumn('joinedColumns',when(size(df.joinedColumns)==0, None).otherwise(df.joinedColumns))
However in C# doing the same with this line
df=df.WithColumn("JoinedTables",((Size(df["JoinedTables"]) == 0) ? Lit(null) : df["JoinedTables"]));
or
df=df.WithColumn("JoinedTables",((Size(df["JoinedTables"]) == 0) ? null : df["JoinedTables"]));
It returns error
(1,34): error CS0029: Cannot implicitly convert type 'Microsoft.Spark.Sql.Column' to 'bool'
Schema is like this
|-- IsCopyStatement: boolean (nullable = true)
|-- IsDeleteStatement: boolean (nullable = true)
|-- IsInsertStatement: boolean (nullable = true)
|-- IsSelectStatement: boolean (nullable = true)
|-- IsUpdateStatement: boolean (nullable = true)
|-- JoinedColumns: array (nullable = true)
| |-- element: string (containsNull = true)
|-- JoinedTables: array (nullable = true)
| |-- element: string (containsNull = true)
|-- SQLCommand: string (nullable = true)
|-- SelectStatementFrom: array (nullable = true)
| |-- element: string (containsNull = true)
Beta Was this translation helpful? Give feedback.
All reactions