3.1 Data loaders

Data loaders are ready-made templates designed to seamlessly link up with a multitude of data sources, spanning from Postgres, BigQuery, Redshift, and S3 to various others. They handle the complexity of connecting to different systems, authenticating, and retrieving data in the proper format for downstream processing.

Data loaders in Mage support various data sources including:

• Databases: PostgreSQL, MySQL, BigQuery, Snowflake, Redshift

• Cloud storage: Amazon S3, Google Cloud Storage, Azure Blob Storage

• APIs: REST APIs, GraphQL endpoints, custom web services

• File systems: Local files, CSV, JSON, Parquet formats

• Streaming: Kafka, Kinesis, real-time data feeds

Data loader templates

Mage provides pre-built templates for common data sources, eliminating the need to write connection logic from scratch. These templates include proper error handling, authentication, and optimization for different data types. When you create a new data loader block, you can choose from templates like:

• API: For connecting to REST endpoints and web services

• Database: For SQL-based connections with built-in query optimization

• File: For local and remote file access with format detection

• Custom: For building your own data loading logic

Best practices for data loaders

Keep connections configurable: Use the io_config.yaml file to store connection details rather than hardcoding them in your data loader blocks.

Handle errors gracefully: Always include error handling for network timeouts, authentication failures, and data format issues.

Validate data early: Add basic data validation in your data loaders to catch issues before they propagate through your pipeline.

Document your sources: Include comments describing the data source, expected format, and any special considerations for future maintainers.

Conclusion

Data loaders serve as the crucial entry points that bring external data into your Mage pipelines, handling the complex tasks of authentication, connection management, and data retrieval across diverse systems. With pre-built templates for databases, APIs, cloud storage, and streaming sources, Mage eliminates the boilerplate code typically required for data ingestion while providing robust error handling and optimization features. By following best practices like configurable connections, early data validation, and comprehensive documentation, you can build reliable data loaders that form the solid foundation for all downstream processing in your data engineering workflows.