Skip to content

Docs: Clarify Spark branch write precedence over WAP branch#15917

Open
yadavay-amzn wants to merge 1 commit intoapache:mainfrom
yadavay-amzn:fix/15916-spark-writes-branch-docs
Open

Docs: Clarify Spark branch write precedence over WAP branch#15917
yadavay-amzn wants to merge 1 commit intoapache:mainfrom
yadavay-amzn:fix/15916-spark-writes-branch-docs

Conversation

@yadavay-amzn
Copy link
Copy Markdown

After #15288, the docs in spark-writes.md no longer match the actual behavior for writing to branches.

Changes:

  • Updated the branch creation note: the target branch may now be created during writes (previously docs said it must exist beforehand)
  • Replaced the incorrect statement "WAP branch and branch identifier cannot both be specified" with the actual precedence rules from the code:
    1. Identifier and option branches can't conflict — if both set, they must match
    2. Identifier and option branches take priority over the session WAP branch
    3. If neither is set and WAP is enabled, the WAP branch is used
  • Added note that WAP ID and WAP branch cannot be set at the same time

Closes #15916

After apache#15288, the identifier and option branches take priority over the
session WAP branch. The docs previously stated that WAP branch and branch
identifier cannot both be specified, which is no longer accurate.

Also updated the note about branch creation during writes.
@barry3406
Copy link
Copy Markdown

Verified the precedence list against SparkWriteBuilder (the source quoted in #15916) and the wording matches the actual logic. One small thing — since this is the second time the doc has drifted from the code (first was after #15288), might be worth a one-line <!-- KEEP IN SYNC: SparkWriteBuilder.branchOption --> HTML comment near the precedence list so the next person touching that code path remembers to update the docs. Optional — the change as it stands is already correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: Clarify Spark branch write precedence over WAP branch

2 participants