Orchestration of Data Pipelines
Parse and validate tags then create and populate MaestroPipelineList
Build a schedule table
Convert a duration string to number of seconds
Creates a new maestro project
Create a new orchestrator
Create a new pipeline in a pipelines directory
Get the artifacts (return values) of the pipelines in a MaestroSchedul...
Generate a sequence of run times for a pipeline
Get the schedule from a MaestroSchedule object
Get the statuses of the pipelines in a MaestroSchedule object
Manually run a pipeline regardless of schedule
Checks whether a DAG is valid (no loops)
Retrieve latest maestro build errors
Retrieve latest maestro pipeline errors
Retrieve latest maestro pipeline messages
Retrieve latest maestro pipeline warnings
cli output for generate schedule table
maestro package
Class for an individual maestro pipeline A pipeline is defined as a si...
Class for a list of MaestroPipelines A MaestroPipelineList is created ...
Class for a schedule of pipelines
Parse a time string
Run a schedule
Visualize the schedule as a DAG
Suggest orchestrator frequency based on a schedule
Framework for creating and orchestrating data pipelines. Organize, orchestrate, and monitor multiple pipelines in a single project. Use tags to decorate functions with scheduling parameters and configuration.
Useful links