← Back to Browse
Data Processing
PipelineX
Build ETL data pipelines with a declarative config. Transform, validate, and load data at scale.
★★★★☆ 4.3
145 reviews
4.6k
installs
v1.2.0
MIT
Install
npx skilldock install pipeline-x
About
PipelineX provides a DAG-based pipeline runner for data transformation tasks. Define your sources, transformations, and destinations in YAML. Supports streaming and batch modes, schema validation at each step, automatic retry with exponential backoff, and detailed execution logs.
Usage Example
# pipeline.yaml
source:
type: postgres
query: "SELECT * FROM orders WHERE date > :lastRun"
transform:
- deduplicate: { key: order_id }
- enrich: { lookup: customers, on: customer_id }
destination:
type: warehouse
table: analytics.orders