Skip to content

Return bulk export result with history#16

Open
sagargg wants to merge 4 commits intofeature/support_cloudflare_r2from
fix/datastore_bulk_export_history
Open

Return bulk export result with history#16
sagargg wants to merge 4 commits intofeature/support_cloudflare_r2from
fix/datastore_bulk_export_history

Conversation

@sagargg
Copy link
Copy Markdown
Member

@sagargg sagargg commented May 15, 2024

  1. Create a lookup table _query_history_lookup in BigQuery to store query history.
  2. Store encoded queries and table last modified timestamps in the query history lookup table.
  3. Create a history entry if the query does not exist in the lookup table. If it exists, return the result from the history.

If the query history exists, the current response time is ~6 seconds, but returning directly from BigQuery takes an additional 5 seconds due to the extra logic we added.

@sagargg sagargg requested a review from shubham-mahajan May 15, 2024 10:37
@sagargg sagargg changed the base branch from master to feature/support_cloudflare_r2 May 22, 2024 10:38
Copy link
Copy Markdown
Member

@shubham-mahajan shubham-mahajan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sagargg sagargg force-pushed the fix/datastore_bulk_export_history branch 2 times, most recently from 047ab90 to 468bd00 Compare May 22, 2024 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants