3.4 Scratchpad

Use Scratchpad blocks to experiment and write code you do not necessarily need for your production pipeline. Scratchpad blocks are your playground for experimentation, prototyping, and exploratory data analysis. They provide a safe space to test ideas without affecting your production pipeline flow.

When to use scratchpad blocks

Scratchpad blocks excel in several scenarios:

Data exploration: When you receive a new dataset and need to understand its structure, quality, and characteristics before building formal transformation logic.

Prototype development: Testing transformation ideas before implementing them in official transformer blocks.

Debugging: Investigating data issues or pipeline problems by examining intermediate results.

Ad-hoc analysis: Answering one-off business questions that don't require formal pipeline integration.

Key characteristics

Scratchpad blocks aren't used when executing a pipeline. This means they won't slow down or interfere with your production data flows, making them perfect for experimentation.

Unlike other block types, scratchpads don't automatically pass data to downstream blocks. Instead, you can manually access data from other blocks.

Conclusion

Scratchpad blocks are invaluable tools that enable data engineers to experiment, explore, and innovate without compromising production pipeline integrity. By providing a safe environment for data exploration, prototype development, and debugging activities, scratchpads bridge the gap between experimentation and production-ready code. Their isolation from pipeline execution ensures that your analytical curiosity and testing efforts never interfere with critical data flows, making them essential for maintaining both productivity and reliability in your data engineering projects.