Skip to content

Follow url metadata redirects #133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
femincan wants to merge 26 commits into
base: main
Choose a base branch
from
+1,583 −110
Open

Follow url metadata redirects #133

Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
b25ec33
refactor: extract robots check as a separate function
femincan Mar 21, 2026
2cecac5
Merge branch 'main' into feat/url-metadata-follow-redirects
femincan Mar 24, 2026
a28eb03
refactor: replace console.debug with console.log
femincan Mar 24, 2026
e30f069
feat: follow redirects in fetchAsBotStream
femincan Mar 26, 2026
55e618f
fix: correct redirect count condition in fetchAsBotStream
femincan Mar 26, 2026
adb5875
feat: add repeatTimes parameter to mockEndpoint
femincan Mar 26, 2026
fa5d085
test: add fetchAsBotStream tests
femincan Mar 26, 2026
e74ebe1
test: fix tests to align with new fetchAsBotStream behavior
femincan Mar 27, 2026
cbdee20
test: remove unused mockEndpoints in fetchAsBot.test.ts to prevent te…
femincan Mar 27, 2026
a5fa24a
refactor: use one fetch call instead of two duplicated ones to detect…
femincan Mar 28, 2026
61cdf24
test: remove unnecessary repeatTimes property from mockEndpoint calls…
femincan Mar 28, 2026
8922604
test: clean up and refactor fetchAsBotStream tests
femincan Mar 28, 2026
3f036cb
test: initialize mockAgent in beforeEach and assert for pending inter…
femincan Mar 29, 2026
9a9f670
fix: make method property of fetchAsBot more strict by forcing only u…
femincan Mar 29, 2026
8e04fdb
refactor: organize and refactor fetchAsBotStream function
femincan Apr 5, 2026
2a9e76d
feat: validate redirect protocol for redirects in fetchAsBotStream
femincan Apr 5, 2026
e730b7e
fix: update currentUrl type to URL and use origin directly in URL.parse
femincan Apr 5, 2026
185f908
feat: update fetchAsBot to match implementation with fetchAsBotStream
femincan Apr 5, 2026
09538e2
fix: move robots check inside the while loop to check for redirection…
femincan Apr 5, 2026
2bb6666
refactor: update fetchAsBot to use parsedUrl for consistency
femincan Apr 5, 2026
0dc0d34
test: disable real network connections in tests to ensure all request…
femincan Apr 8, 2026
8391734
fix: handle empty newLocation in redirects and add support for relati…
femincan Apr 8, 2026
14b9d71
fix: update robots access check to use currentUrl in fetchAsBotStream
femincan Apr 8, 2026
9206210
test: add tests for redirects in fetchAsBot and fetchAsBotStream
femincan Apr 8, 2026
d644702
test: add tests to ensure redirects are not followed when followRedir…
femincan Apr 9, 2026
25e227d
Merge branch 'main' into feat/url-metadata-follow-redirects
femincan Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading