ADR-0001: Import Layering Rules for Package Architecture

Historical document. This ADR records the import layering rules established during the pre-0.22 refactor. The package boundaries it describes (common, core, datasets, sql) still hold, but some module placements it references have since changed: schema, type, and polars utilities moved from common to datasets in 0.22.x, and common.misc was split into common.parallel, common.sync, and common.partitions. For the current layout, see Architecture Overview. For the import-path migration, see Move Your Package Imports.

Status

Accepted

Context

The fsspeckit package was refactored to use a domain-driven architecture with clear package boundaries. During the refactor, several critical issues were identified:

1. Circular Import: The core.filesystem module had a circular import that prevented the package from loading
2. Layering Violations: The core.ext.parquet module imported from datasets.pyarrow, violating architectural layering
3. Code Duplication: Schema utilities were duplicated across common.schema and datasets.pyarrow.schema

These issues undermined the architectural benefits of the refactor and made the codebase difficult to maintain.

Decision

We will establish strict import layering rules with automated enforcement to prevent future violations.

Package Structure

fsspeckit/
├── common/          # Shared utilities (lowest level)
├── core/            # Core filesystem and I/O operations
├── datasets/        # Backend-specific dataset operations
├── sql/             # SQL-related functionality
└── utils/           # Backwards compatibility façade

Dependency Rules

1. fsspeckit.common (lowest level)
2. No dependencies on other fsspeckit packages
3. Can be imported by any package

4. Contains shared utilities, schemas, and common functionality

5. fsspeckit.core (core level)

6. May import from: fsspeckit.common
7. May import from: fsspeckit.storage_options
8. May import from: External packages (fsspec, pyarrow, etc.)

9. MUST NOT import from: fsspeckit.datasets, fsspeckit.sql

10. fsspeckit.datasets (backend level)

11. May import from: fsspeckit.common
12. May import from: fsspeckit.core
13. May import from: fsspeckit.storage_options
14. May import from: External packages

15. MUST NOT import from: fsspeckit.sql

16. fsspeckit.sql (highest level)

17. May import from: fsspeckit.common
18. May import from: fsspeckit.core
19. May import from: fsspeckit.datasets

20. May import from: External packages

21. fsspeckit.utils (façade layer)

22. Re-exports symbols from all packages for backwards compatibility
23. Does not contain implementation (only re-exports)
24. No layering restrictions (sits outside the dependency graph)

Dependency Flow Diagram

        External Packages
               ↓
        ┌──────────────┐
        │ fsspeckit    │
        │   .common    │ ← Lowest level
        └──────┬───────┘
               ↓
        ┌──────────────┐
        │ fsspeckit    │
        │   .core      │ ← Core level
        └──────┬───────┘
               ↓
        ┌──────────────┐
        │ fsspeckit    │
        │ .datasets    │ ← Backend level
        └──────┬───────┘
               ↓
        ┌──────────────┐
        │  fsspeckit   │
        │   .sql       │ ← Highest level
        └──────┬───────┘
               ↑
        ┌──────────────┐
        │ fsspeckit    │
        │   .utils     │ ← Façade layer
        └──────────────┘

Enforcement Mechanism

1. Automated CI Check: A Python script (scripts/check_layering.py) validates import layering on every PR
2. Fast Failure: The check runs early in CI to fail quickly on violations
3. Clear Error Messages: Violations report file, line number, and the violating import
4. Documentation: Layering rules documented in CONTRIBUTING.md

Implementation

The layering check script (scripts/check_layering.py): - Parses all Python files in the codebase - Extracts import statements - Validates against the layering rules - Reports violations with file paths and line numbers - Exits with error code if violations found

CI integration (.github/workflows/ci.yml): - Runs layering check before tests - Fails CI on violations - Runs on all Python versions in the test matrix

Rationale

Why Strict Layering?

1. Maintainability: Clear dependencies make code easier to understand and modify
2. Testability: Packages with fewer dependencies are easier to test in isolation
3. Reusability: Lower-level packages can be reused without pulling in higher-level dependencies
4. Prevent Circular Imports: Strict rules prevent circular dependency chains
5. Architectural Clarity: Enforced boundaries make the architecture visible and enforceable

Why This Specific Layering?

1. common at the bottom: Shared utilities should not depend on anything else
2. core above common: Core functionality uses shared utilities
3. datasets above core: Backend implementations use core functionality
4. sql at the top: SQL features can use everything else

Why Automated Enforcement?

1. Prevents Regression: Manual reviews can miss violations
2. Fast Feedback: Developers get immediate feedback
3. Enforceable: Rules in code are clearer than rules in documentation
4. Educational: Violations teach developers about the architecture

Alternatives Considered

Alternative 1: No Layering Rules

• Pros: Maximum flexibility for developers
• Cons: Circular imports return, code becomes tangled
• Decision: Rejected - leads to the problems we're trying to solve

Alternative 2: Documentation Only

• Pros: Easy to implement
• Cons: Rules are not enforced, easy to violate accidentally
• Decision: Rejected - insufficient for maintaining architectural integrity

Alternative 3: Linting Rules (ruff, pylint)

• Pros: Integrates with existing tooling
• Cons: Complex to configure for specific package-level rules
• Decision: Could be added in the future, but custom script is simpler now

Alternative 4: Runtime Validation

• Pros: Catches violations at runtime
• Cons: Violations only caught when code runs, not during CI
• Decision: Rejected - we want to catch violations before they reach production

Consequences

Positive

1. Architectural Integrity: Package boundaries are maintained
2. Easier Maintenance: Dependencies are clear and enforced
3. Better Testability: Packages can be tested in isolation
4. Improved Code Quality: Prevents anti-patterns like circular imports
5. Developer Onboarding: New developers learn the architecture through enforcement

Negative

1. Initial Learning Curve: Developers need to understand layering rules
2. Occasional Friction: Some features may require refactoring to fit the rules
3. Maintenance Overhead: The check script needs to be maintained
4. Reduced Flexibility: Cannot take shortcuts that violate layering

Migration Path

For existing code: 1. Phase 1: Identify violations using the check script 2. Phase 2: Refactor code to move functionality to appropriate layers 3. Phase 3: Use re-exports in utils for backwards compatibility 4. Phase 4: Enforce rules in CI

References

• Migration Guide
• CONTRIBUTING.md

Date

2025-12-08

Authors

• fsspeckit team