rialto.runner package

Subpackages

• rialto.runner.reporting package
  • Submodules
  • rialto.runner.reporting.bookkeeper module
    • BookKeeper
      • BookKeeper.add()
  • rialto.runner.reporting.mailer module
    • Mailer
      • Mailer.create_message()
      • Mailer.send_mail()
  • rialto.runner.reporting.record module
    • Record
      • Record.date
      • Record.exception
      • Record.get_schema()
      • Record.job
      • Record.reason
      • Record.records
      • Record.run_timestamp
      • Record.status
      • Record.target
      • Record.time
      • Record.to_spark_row()
  • rialto.runner.reporting.tracker module
    • Tracker
      • Tracker.add()
      • Tracker.report_by_mail()
  • Module contents
• rialto.runner.services package
  • Submodules
  • rialto.runner.services.config_loader module
    • ConfigLoader
      • ConfigLoader.load_yaml()
  • rialto.runner.services.config_overrides module
    • override_config()
  • rialto.runner.services.data_checker module
    • DataChecker
      • DataChecker.check_date()
      • DataChecker.check_range()
      • DataChecker.check_written()
  • rialto.runner.services.date_manager module
    • DateManager
      • DateManager.all_dates()
      • DateManager.date_subtract()
      • DateManager.get_date_from()
      • DateManager.get_date_until()
      • DateManager.get_execution_and_partition_dates()
      • DateManager.str_to_date()
  • rialto.runner.services.executor module
    • PipelineExecutor
      • PipelineExecutor.execute()
  • rialto.runner.services.result_mapper module
    • TaskResultMapper
      • TaskResultMapper.already_complete()
      • TaskResultMapper.dependencies_incomplete()
      • TaskResultMapper.exception()
      • TaskResultMapper.interrupted()
      • TaskResultMapper.success()
  • rialto.runner.services.table module
    • Table
      • Table.from_dependency_config()
      • Table.from_target_config()
      • Table.get_all_partition_columns()
      • Table.get_schema_path()
      • Table.get_table_path()
  • rialto.runner.services.task_registry module
    • PipelineDependency
      • PipelineDependency.complete
      • PipelineDependency.date_from
      • PipelineDependency.date_until
      • PipelineDependency.table
    • PipelineTask
      • PipelineTask.completion
      • PipelineTask.config
      • PipelineTask.dependencies
      • PipelineTask.dependencies_complete
      • PipelineTask.error
      • PipelineTask.error_trace
      • PipelineTask.execution_date
      • PipelineTask.name
      • PipelineTask.partition_date
      • PipelineTask.precheck_failed
      • PipelineTask.result
      • PipelineTask.target
    • TaskRegistry
      • TaskRegistry.add_task()
      • TaskRegistry.log_status()
  • rialto.runner.services.task_status_checker module
    • TaskStatusChecker
      • TaskStatusChecker.check_completion()
      • TaskStatusChecker.check_pipeline_dependencies()
  • rialto.runner.services.writer module
    • DatabricksWriter
      • DatabricksWriter.write()
    • Writer
      • Writer.write()
  • Module contents

Submodules

rialto.runner.engine module

class rialto.runner.engine.RunnerEngine(services: RunnerServices, rerun: bool, skip_dependencies: bool)

Bases: object

Orchestrates pipeline execution lifecycle and task tracking

check_tasks() → None

Check task completion and dependency status

debug_first_task(op: str = None) → DataFrame

Debug mode: execute first task and return result

dry_run_execution(op: str = None) → None

Execute pre-run checks without task execution

finalize() → None

Send final reports via mail/bookkeeping

log_task_status() → None

Log summary of task statuses

register_tasks(pipelines: List[PipelineConfig]) → None

Register tasks for all pipelines and date combinations

run(op: str = None) → None

Execute all tasks

run_tasks() → None

Execute runnable tasks with per-task error isolation

select_pipelines(op: str = None) → List[PipelineConfig]

Select pipelines to run based on operation name

rialto.runner.runner module

class rialto.runner.runner.Runner(spark: SparkSession, config_path: str, run_date: str = None, rerun: bool = False, op: str = None, skip_dependencies: bool = False, overrides: Dict = None, merge_schema: bool = False, services: RunnerServices = None)

Bases: object

Entry point for pipeline execution orchestration (beginner-friendly API)

dry_run()

Dry run - log status of pipelines without executing

rialto.runner.runner_services module

class rialto.runner.runner_services.DefaultRunnerServices

Bases: object

Factory for default Runner services composition

static build(spark: SparkSession, config_path: str, run_date: str = None, merge_schema: bool = False, overrides: dict = None) → RunnerServices

Build default services for Runner.

Parameters:

• spark – SparkSession instance

• config_path – Path to pipeline configuration YAML

• run_date – Override run date (optional)

• merge_schema – Enable schema merging in writer

• overrides – Configuration overrides

Returns:

RunnerServices bundle

class rialto.runner.runner_services.RunnerServices(config: PipelinesConfig, date_manager: DateManager, writer: DatabricksWriter, data_checker: DataChecker, task_checker: TaskStatusChecker, registry: TaskRegistry, executor: PipelineExecutor, tracker: Tracker)

Bases: object

Bundle of collaborator services for Runner orchestration

config: PipelinesConfig

data_checker: DataChecker

date_manager: DateManager

executor: PipelineExecutor

registry: TaskRegistry

task_checker: TaskStatusChecker

tracker: Tracker

writer: DatabricksWriter

rialto.runner.transformation module

class rialto.runner.transformation.Transformation

Bases: object

Interface for feature implementation

abstractmethod run(reader: DataReader, run_date: date, spark: SparkSession = None, config: PipelineConfig = None, metadata_manager: MetadataManager = None, feature_loader: PysparkFeatureLoader = None) → DataFrame

Run the transformation

Parameters:

• reader – data store api object

• run_date – date

• spark – spark session

• config – pipeline config

• metadata_manager – metadata manager

• feature_loader – feature loader

Returns:

dataframe

rialto.runner.utils module

rialto.runner.utils.find_dependency(config: PipelineConfig, name: str)

Get dependency from config

Parameters:

• config – Pipeline configuration

• name – Dependency name

Returns:

Dependency object

Module contents