You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the transfer orchestration layer waits until the source is completely unloaded before checking that the destination is viable
For small syncs this isn't a huge deal, but for large transfers you could burn a lot of time / bandwidth / cycles unloading data before failing because of an issue with the destination
Proposed solution
Run some "preflight checks" on the destination before unloading the source data, fail early if needed
It would be ideal to check routability, auth and schema compat before starting
Additional context
Checking schema compatibility could be tricky because currently we don't load the source schema until the source read starts - so the orchestrator has no opportunity to pre-check against the destination
We could factor that out into a new interface and require all sources to support a get_streams() and all destinations to support a check_streams()
Similarly we don't have a test_connect() in the destination connector interface right now, but that would be straightforward to add - destination connection checks are actually just a tiny transfer of a mock dataset that simulates the whole process end-to-end
Issue
Proposed solution
Additional context
get_streams()and all destinations to support acheck_streams()test_connect()in the destination connector interface right now, but that would be straightforward to add - destination connection checks are actually just a tiny transfer of a mock dataset that simulates the whole process end-to-end