Spark Data Types

These function support supplying a spark read schema. This is particularly useful when reading data with nested arrays when you are not interested in several of the nested fields.

struct_type(sc, struct_fields)

struct_field(sc, name, data_type, nullable = FALSE)

array_type(sc, data_type, nullable = FALSE)

binary_type(sc)

boolean_type(sc)

byte_type(sc)

date_type(sc)

double_type(sc)

float_type(sc)

integer_type(sc)

numeric_type(sc)

long_type(sc)

map_type(sc, key_type, value_type, nullable = FALSE)

string_type(sc)

character_type(sc)

timestamp_type(sc)

Arguments

sc	A `spark_connection`
struct_fields	A vector or fields obtained from `struct_field()`
name	A field name to use in the output struct type
data_type	A (java) data type (e.g., `string_type()` or `double_type()`)
nullable	Logical. Describes whether field can be missing for some rows.
key_type	A (java) data type describing the map keys (usually `string_type()`)
value_type	A (java) data type describing the map values

Arguments

Contents