Skip to content

Gsoc/week2 intermediate representation - #6

Open
Abhishek-Kumar-Rai5 wants to merge 3 commits into
PecanProject:mainfrom
Abhishek-Kumar-Rai5:gsoc/week2-intermediate-representation
Open

Gsoc/week2 intermediate representation#6
Abhishek-Kumar-Rai5 wants to merge 3 commits into
PecanProject:mainfrom
Abhishek-Kumar-Rai5:gsoc/week2-intermediate-representation

Conversation

@Abhishek-Kumar-Rai5

Copy link
Copy Markdown
Collaborator

Completes #3

Summary

This PR implements the Week 2 Intermediate Representation (IR) layer for the BETYdb extraction pipeline.

The implementation introduces the core IR models, supporting value types, dataset container, graph-level validation, and the IR builder entry point. This layer serves as the structured representation between the normalized document layer and the later extraction/export stages.

Package Structure

src/
└── betydb_extraction/
    ├── ir/
    │   ├── __init__.py
    │   ├── dataset.py
    │   ├── identifiers.py
    │   │
    │   ├── support/
    │   │   ├── __init__.py
    │   │   ├── enums.py
    │   │   ├── extraction_source.py
    │   │   ├── extracted_field.py
    │   │   ├── quantity.py
    │   │   ├── temporal.py
    │   │   └── statistics.py
    │   │
    │   ├── entities/
    │   │   ├── __init__.py
    │   │   ├── citation.py
    │   │   ├── site.py
    │   │   ├── species.py
    │   │   ├── method.py
    │   │   ├── treatment.py
    │   │   ├── management.py
    │   │   └── observation.py
    │   │
    │   └── validation/
    │       ├── __init__.py
    │       ├── errors.py
    │       └── referential_integrity.py
    │
    └── ir_builder/
        ├── __init__.py
        └── factory.py

Changes

IR package

  • Added the ir/ package and supporting module structure.
  • Implemented common support types (ExtractedField, quantities, temporal values, statistics, provenance, etc.).
  • Added identifier utilities used throughout the IR.

Entity models

Implemented the core IR entities:

  • Citation
  • Site
  • Species
  • Method
  • Treatment
  • Management
  • Observation

Dataset

  • Added the IRDataset aggregate model.
  • Implemented dataset-level validation and serialization behavior.
  • Enforced entity ordering and dataset consistency checks.

Validation

Added whole-graph validation for:

  • ID uniqueness
  • Referential integrity
  • Treatment and management relationships
  • Observation consistency
  • Dataset containment
  • Other IR-level invariants defined for the intermediate representation

IR Builder

  • Added the initial ir_builder package.
  • Implemented the builder entry point for constructing an IRDataset from the normalized document layer.
  • This PR only introduces the builder interface required for the current milestone; extraction logic will be implemented in subsequent work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant