Skip to content

Check destination connectivity and schema compatibility before starting sync #26

@kmacrow

Description

@kmacrow

Issue

  • Currently the transfer orchestration layer waits until the source is completely unloaded before checking that the destination is viable
  • For small syncs this isn't a huge deal, but for large transfers you could burn a lot of time / bandwidth / cycles unloading data before failing because of an issue with the destination

Proposed solution

  • Run some "preflight checks" on the destination before unloading the source data, fail early if needed
  • It would be ideal to check routability, auth and schema compat before starting

Additional context

  • Checking schema compatibility could be tricky because currently we don't load the source schema until the source read starts - so the orchestrator has no opportunity to pre-check against the destination
  • We could factor that out into a new interface and require all sources to support a get_streams() and all destinations to support a check_streams()
  • Similarly we don't have a test_connect() in the destination connector interface right now, but that would be straightforward to add - destination connection checks are actually just a tiny transfer of a mock dataset that simulates the whole process end-to-end

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions