rialto.loader package

Submodules

rialto.loader.config_loader module

rialto.loader.config_loader.get_feature_config(path) → FeatureConfig[source]

Read yaml and parse it

Parameters:

path – config path

Returns:

Pydantic feature config object

rialto.loader.data_loader module

rialto.loader.interfaces module

class rialto.loader.interfaces.FeatureLoaderInterface[source]

Bases: object

A definition of feature loading interface

Provides functionality to read features, feature groups and selections of features according to configs. Also provides an interface to access metadata of said features.

abstract get_feature(group_name: str, feature_name: str, information_date: date)[source]

Get single feature

abstract get_feature_metadata(group_name: str, feature_name: str) → Dict[source]

Get single feature’s metadata

abstract get_features_from_cfg(path: str, information_date: date)[source]

Get features from multiple groups as defined by configuration

abstract get_group(group_name: str, information_date: date)[source]

Get feature group

abstract get_group_metadata(group_name: str) → Dict[source]

Get feature group’s metadata

abstract get_metadata_from_cfg(path: str) → Dict[source]

Get metadata from multiple groups as defined by configuration

rialto.loader.pyspark_feature_loader module

class rialto.loader.pyspark_feature_loader.PysparkFeatureLoader(spark: SparkSession, feature_schema: str, metadata_schema: str, date_column: str = 'INFORMATION_DATE')[source]

Bases: FeatureLoaderInterface

Implementation of feature loader for pyspark environment

class KeyMap(df, key)

Bases: tuple

df

Alias for field number 0

key

Alias for field number 1

get_feature(group_name: str, feature_name: str, information_date: date) → DataFrame[source]

Get single feature

Parameters:

  • group_name – feature group name

  • feature_name – feature name

  • information_date – selected date

Returns:

A dataframe containing feature group key and selected feature

get_feature_metadata(group_name: str, feature_name: str) → FeatureMetadata[source]

Get single features metadata

Parameters:

  • group_name – feature group name

  • feature_name – feature name

Returns:

metadata dictionary

get_features_from_cfg(path: str, information_date: date) → DataFrame[source]

Get multiple features across many groups, fetches latest features in relation to the provided date

Parameters:

  • path – configuration location

  • information_date – date for extraction

get_group(group_name: str, information_date: date) → DataFrame[source]

Get feature group

Parameters:

  • group_name – feature group name

  • information_date – selected date

Returns:

A dataframe containing feature group key

get_group_fs_name(group_name: str) → str[source]

Return groups file system name

If given name matches databricks path, i.e. has two dot separators, do nothing. Else assume it’s a class name and search for fs name in metadata. :param group_name: Group name :return: group filesystem name

get_group_metadata(group_name: str) → GroupMetadata[source]

Get feature groups metadata

Parameters:

group_name – feature group name

Returns:

metadata dictionary

get_metadata_from_cfg(path: str) → Dict[str, FeatureMetadata][source]

Get multiple features metadata from config

Parameters:

path – configuration path

Returns:

dictionary feature_name : FeatureMetadata

read_group(group: str, information_date: date) → DataFrame[source]

Read a feature group by getting the latest partition by date

Parameters:

  • group – group name

  • information_date – partition date

Returns:

dataframe

Module contents