How-to Guides

Use these guides when you know what you want to achieve and need a concrete recipe. Each guide uses canonical imports and links to the Reference for exact signatures and detailed behavior.

Filesystem and storage

• Work with Filesystems - create filesystems, path confinement, caching, and extended I/O methods.
• Configure Cloud Storage - set up AWS, GCP, and Azure providers from environment variables, structured classes, or URIs.

Datasets

• Read and Write Datasets - read and write JSON, CSV, and Parquet files, plus dataset handlers for partitioned parquet.
• Merge Datasets - incremental insert, update, and upsert merges across the DuckDB and PyArrow backends.
• Maintain Parquet Datasets - compaction, deduplication, repartitioning, and optimization with typed plans and results.
• Merge Operations Examples - end-to-end merge scenarios and result interpretation.

Performance and memory

• Optimize Performance - caching, parallel processing, and type optimization.
• Memory-Constrained Environments - memory monitoring, chunked processing, and streaming merge controls.
• Adaptive Key Tracking - tiered, memory-bounded key tracking for deduplication and merge.

Multi-key operations

• Multi-Key Examples - composite-key merge and deduplication recipes.
• Multi-Key Performance - how composite keys behave and when to use them.

Queries and file management

• Use SQL Filters - translate SQL WHERE clauses into PyArrow and Polars filter expressions.
• Sync and Manage Files - synchronize files and directories across filesystems.