Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
151 lines (130 loc) · 6.77 KB

ARCHITECTURE.md

File metadata and controls

151 lines (130 loc) · 6.77 KB
Raw

Architecture of SHARE/trove

This document is a starting point and reference to familiarize yourself with this codebase.

Bird's eye view

In short, SHARE/trove holds metadata records that describe things and makes those records available for searching, browsing, and subscribing.

overview of shtrove: metadata records in, search/browse/subscribe out

Parts

a slightly simplified look at the tangles of communication between different parts of the system, as currently implemented:

graph LR;
    subgraph shtrove;
        subgraph web[api/web server];
            ingest;
            search;
            browse;
            rss;
            atom;
            oaipmh;
        end;
        worker["background worker (celery)"];
        indexer["indexer daemon"];
        rabbitmq["task queue (rabbitmq)"];
        postgres["database (postgres)"];
        elasticsearch;
        web---rabbitmq;
        web---postgres;
        web---elasticsearch;
        worker---rabbitmq;
        worker---postgres;
        worker---elasticsearch;
        indexer---rabbitmq;
        indexer---postgres;
        indexer---elasticsearch;
    end;
    source["metadata source (e.g. osf.io backend)"];
    user["web user, either by browsing directly or via web app (like osf.io)"];
    subscribers["feed subscription tools"];
    source-->ingest;
    user-->search;
    user-->browse;
    subscribers-->rss;
    subscribers-->atom;
    subscribers-->oaipmh;
Loading

/trove/ingest

a slightly simplified look at how metadata records are ingested, as currently implemented:

sequenceDiagram
    participant ms as metadata source
    box shtrove
    participant ss as web server
    participant sd as db (postgres)
    participant sw as worker (celery)
    participant sq as queues (rabbitmq)
    participant si as indexer
    participant se as elasticsearch
    end
    ms ->> ss: POST /trove/ingest
    ss ->> sd: save ResourceIdentifier(s)
    ss ->> sd: save Indexcard
    ss ->> sd: save ResourceDescription(s)
    ss ->> sq: enqueue derive task
    ss ->> ms: 201 CREATED (success!)
    sq -->> sw: receive derive task
    sd <<-->> sw: load ResourceDescription(s)
    sw ->> sd: save DerivedIndexcards
    sw ->> sq: enqueue indexer message
    sq -->> si: bulk receive messages
    sd <<-->> si: bulk load metadata records
    si ->> se: bulk index
Loading

/trove/index-card-search

a slightly simplified look at how search requests are served, as currently implemented:

sequenceDiagram
    participant c as client
    box shtrove
    participant ss as web server
    participant sd as db (postgres)
    participant se as elasticsearch
    end
    c ->> ss: GET /trove/index-card-search
    ss <<-->> se: query for result ids (and context)
    ss <<-->> sd: load metadata records
    ss ->> c: respond/stream search results
Loading

Code map

A brief look at important areas of code as they happen to exist now.

  • trove: django app for rdf-based apis
    • trove.digestive_tract: most of what happens after ingestion
      • stores records and identifiers in the database
      • initiates indexing
    • trove.extract: parsing ingested metadata records into resource descriptions
    • trove.derive: from a given resource description, create special non-rdf serializations
    • trove.render: from an api response modeled as rdf graph, render the requested mediatype
    • trove.models: database models for identifiers and resource descriptions
    • trove.trovesearch: builds rdf-graph responses for trove search apis (using IndexStrategy implementations from share.search)
    • trove.vocab: identifies and describes concepts used elsewhere
      • trove.vocab.trove: describes types, properties, and api paths in the trove api
      • trove.vocab.osfmap: describes metadata from osf.io (currently the only metadata ingested)
    • trove.openapi: generate openapi json for the trove api from thesaurus in trove.vocab.trove
  • share: django app with search indexes and remnants of sharev2
    • share.models: database models for external sources, users, and other system book-keeping
    • share.oaipmh: provide data via OAI-PMH
    • share.search: all interaction with elasticsearch
      • share.search.index_strategy: abstract base class IndexStrategy with multiple implementations, for different approaches to indexing the same data
      • share.search.daemon: the "indexer daemon", an optimized background worker for batch-processing updates and sending to all active index strategies
      • share.search.index_messenger: for sending messages to the indexer daemon
  • api: django app with remnants of the legacy sharev2 api
    • api.views.feeds: allows custom RSS and Atom feeds
    • otherwise, subject to possible deprecation
  • osf_oauth2_adapter: django app for login via osf.io
  • project: the actual django project
    • default settings at project.settings
    • pulls together code from other directories implemented as django apps (share, trove, api, and osf_oauth2_adapter)

Cross-cutting concerns

Resource descriptions

Uses the resource description framework:

  • the content of each ingested metadata record is an rdf graph focused on a specific resource
  • all api responses from trove views are (experimentally) modeled as rdf graphs, which may be rendered a variety of ways

Identifiers

Whenever feasible, use full IRI strings (utf-8) to identify resources, concepts, types, and properties that may be exposed outwardly (without converting to URI or using to send requests). Prefer using open, standard, well-defined namespaces wherever possible (DCAT is a good place to start; see trove.vocab.namespaces for others already in use). When app-specific concepts must be defined, use the TROVE namespace (https://share.osf.io/vocab/2023/trove/).

A notable exception (non-URI identifier) is the "source-unique identifier" or "suid" -- essentially a two-tuple (source, identifier) that uniquely and persistently identifies a metadata record in a source repository. This identifier may be any string value, provided by the external source.

Conventions

(an incomplete list)

  • local variables prefixed with underscore (to consistently distinguish between internal-only names and those imported/built-in)
  • prefer full type annotations in python code, wherever reasonably feasible

Why this?

inspired by this writeup and this example architecture document