flexmeasures.data.services.data_sources

Functions

flexmeasures.data.services.data_sources.get_data_generator(source: DataSource | None, model: str, config: dict, save_config: bool, data_generator_type: Type[DG]) → DG | None
flexmeasures.data.services.data_sources.get_first_matching_source(query: Select) → DataSource | None

Return the matching data source with the lowest id, or None if there is no match.

Tolerating multiple matches is a form of defense in depth: while uniqueness of data sources is enforced at the database level, a database may already contain (near-)duplicate rows from before that enforcement (for example, rows created by concurrently running jobs on a fresh database, or rows that differ only in fields the caller did not filter on, such as attributes). Rather than letting such rows fail every lookup with MultipleResultsFound, we deterministically pick the oldest row and log a warning.

flexmeasures.data.services.data_sources.get_or_create_source(source: User | str, source_type: str | None = None, model: str | None = None, version: str | None = None, attributes: dict | None = None, account: Account | None = None, flush: bool = True) → DataSource
flexmeasures.data.services.data_sources.get_source_or_none(source: int | str, source_type: str | None = None) → DataSource | None
Parameters:

  • source – source id

  • source_type – optionally, filter by source type

flexmeasures.data.services.data_sources.insert_source_race_safely(new_source: DataSource, query: Select) → DataSource

Insert a new data source, returning the winning row if we lose an insert race.

The insert happens within a SAVEPOINT, so that losing a race against a concurrent session creating the same source (e.g. parallel workers scheduling against a fresh database, whose get-or-create logic all found no source yet) doesn’t poison the enclosing transaction. Committing the savepoint flushes, which assigns an id so that the new source can be referenced in the current db session.

Parameters:

  • new_source – the (not yet added) data source to insert

  • query – the query with which to re-fetch the winning row, should our insert hit a uniqueness conflict