feat: support http range header #10
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
On mobile but wanted to give a shout that ive got a refactor going that cleans this up a bunch |
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
achingbrain
reviewed
Mar 5, 2024
Done a first pass, will look again after the refactor lands. |
…port FYI: some tests were failing after `npm run reset` and `npm run test` was run. I had to fix them manually.
|
FYI: some tests were failing when merging main into byte-range PR due to new ipns things. changes at |
achingbrain
reviewed
Mar 15, 2024
achingbrain
approved these changes
Mar 15, 2024
Co-authored-by: Alex Potsides <[email protected]>
SgtPooki
commented
Mar 15, 2024
| * - Content-Range: <unit> <byteStart>-<byteEnd>/<byteSize> | ||
| * - Content-Range: <unit> <byteStart>-<byteEnd>/‍* | ||
| */ | ||
| // - Content-Range: <unit> */<byteSize> // this is purposefully not in jsdoc block |
There was a problem hiding this comment.
@achingbrain is this better than all as single line comments?
| } catch (err) { | ||
| this.log.error('error parsing resource %s', resource, err) | ||
|
|
||
| return badRequestResponse('Invalid resource') |
There was a problem hiding this comment.
Less throwing, more bad fetch responses
|
commenting elsewhere since the thread (#10 (comment)) was collapsed: we could solve both concerns by setting internalFileSize so we know which value we set vs consumers.. but that could get confusing. i'll leave as is for now. i've got _isRangeRequest updated locally, and have isValidRangeRequest updated as well. So now we have: inaccessable properties outside the ByteRangeContext class:
|
|
🎉 This PR is included in version 1.2.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.7.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
SgtPooki
added a commit
that referenced
this pull request
Mar 18, 2024
* feat: implement new ipns record&answer properties * fix: parseUrlString response includes defined ttl, set ttl if ipnsCached * test: fix firefox failure * feat: support http range header (#10) * chore: limit body parameters to the types used * chore: add response-header helper and tests * feat: add range header parsing support * feat: verified-fetch supports range-requests * test: fix dns test asserting test failure since we are catching it now * fix: return 500 error when streaming unixfs content throws * fix: cleanup code and unexecuting tests hiding errors * chore: some cleanup and code coverage * tmp: most things working * fix: stream slicing and test correctness * chore: fixed some ByteRangeContext tests * test: add back header helpers * fix: unixfs tests are passing * fix: range-requests on raw content * feat: tests are passing moved transform stream over to https://github.com/SgtPooki/streams * chore: log string casing * chore: use 502 response instead of 500 * chore: use libp2p/interface for types in src * chore: failing to create range resp logs error * chore: Apply suggestions from code review * chore: fix broken tests from github PR patches (my own) * chore: re-enable stream tests for ByteRangeContext * chore: clean up getBody a bit * chore: ByteRangeContext getBody cleanup * chore: apply suggestions from code review Co-authored-by: Alex Potsides <[email protected]> * fix: getSlicedBody uses correct types * chore: remove extra stat call * chore: fix jsdoc with '*/' * chore: fileSize is public property, but should not be used * test: fix blob comparisons that broke or were never worjing properly * chore: Update byte-range-context.ts Co-authored-by: Alex Potsides <[email protected]> * chore: jsdoc cleanup * Revert "chore: fileSize is public property, but should not be used" This reverts commit 46dc133. * chore: jsdoc comments explaining .fileSize use * chore: isRangeRequest is public * chore: getters/setters update * chore: remove unnecessary _contentRangeHeaderValue * chore: ByteRangeContext uses setFileSize and getFileSize * chore: remove .stat changes that are no longer needed --------- Co-authored-by: Alex Potsides <[email protected]> * chore(release): 1.2.0 [skip ci] ## @helia/verified-fetch [1.2.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.1.3...@helia/verified-fetch-1.2.0) (2024-03-15) ### Features * support http range header ([#10](#10)) ([9f5078a](9f5078a)) ### Trivial Changes * fix build ([#22](#22)) ([01261fe](01261fe)) * chore(release): 1.7.0 [skip ci] ## @helia/verified-fetch-interop [1.7.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-interop-1.6.0...@helia/verified-fetch-interop-1.7.0) (2024-03-15) ### Dependencies * **@helia/verified-fetch:** upgraded to 1.2.0 * chore: apply pr comments * fix: some ipns ttl precision cleanup --------- Co-authored-by: Alex Potsides <[email protected]> Co-authored-by: semantic-release-bot <[email protected]>
SgtPooki
added a commit
that referenced
this pull request
Mar 21, 2024
* fix: set cache-control header conditionally Fixes #17 * chore: bump deps * feat: implement new ipns record&answer properties (#23) * feat: implement new ipns record&answer properties * fix: parseUrlString response includes defined ttl, set ttl if ipnsCached * test: fix firefox failure * feat: support http range header (#10) * chore: limit body parameters to the types used * chore: add response-header helper and tests * feat: add range header parsing support * feat: verified-fetch supports range-requests * test: fix dns test asserting test failure since we are catching it now * fix: return 500 error when streaming unixfs content throws * fix: cleanup code and unexecuting tests hiding errors * chore: some cleanup and code coverage * tmp: most things working * fix: stream slicing and test correctness * chore: fixed some ByteRangeContext tests * test: add back header helpers * fix: unixfs tests are passing * fix: range-requests on raw content * feat: tests are passing moved transform stream over to https://github.com/SgtPooki/streams * chore: log string casing * chore: use 502 response instead of 500 * chore: use libp2p/interface for types in src * chore: failing to create range resp logs error * chore: Apply suggestions from code review * chore: fix broken tests from github PR patches (my own) * chore: re-enable stream tests for ByteRangeContext * chore: clean up getBody a bit * chore: ByteRangeContext getBody cleanup * chore: apply suggestions from code review Co-authored-by: Alex Potsides <[email protected]> * fix: getSlicedBody uses correct types * chore: remove extra stat call * chore: fix jsdoc with '*/' * chore: fileSize is public property, but should not be used * test: fix blob comparisons that broke or were never worjing properly * chore: Update byte-range-context.ts Co-authored-by: Alex Potsides <[email protected]> * chore: jsdoc cleanup * Revert "chore: fileSize is public property, but should not be used" This reverts commit 46dc133. * chore: jsdoc comments explaining .fileSize use * chore: isRangeRequest is public * chore: getters/setters update * chore: remove unnecessary _contentRangeHeaderValue * chore: ByteRangeContext uses setFileSize and getFileSize * chore: remove .stat changes that are no longer needed --------- Co-authored-by: Alex Potsides <[email protected]> * chore(release): 1.2.0 [skip ci] ## @helia/verified-fetch [1.2.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.1.3...@helia/verified-fetch-1.2.0) (2024-03-15) ### Features * support http range header ([#10](#10)) ([9f5078a](9f5078a)) ### Trivial Changes * fix build ([#22](#22)) ([01261fe](01261fe)) * chore(release): 1.7.0 [skip ci] ## @helia/verified-fetch-interop [1.7.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-interop-1.6.0...@helia/verified-fetch-interop-1.7.0) (2024-03-15) ### Dependencies * **@helia/verified-fetch:** upgraded to 1.2.0 * chore: apply pr comments * fix: some ipns ttl precision cleanup --------- Co-authored-by: Alex Potsides <[email protected]> Co-authored-by: semantic-release-bot <[email protected]> * chore: add matchUrlGroups typeguard * chore: remove unnecessary headerValue != null check * test: remove unnecessary redefinition of verifiedFetch --------- Co-authored-by: Daniel N <[email protected]> Co-authored-by: Russell Dempsey <[email protected]> Co-authored-by: Alex Potsides <[email protected]> Co-authored-by: semantic-release-bot <[email protected]>
github-actions bot
pushed a commit
that referenced
this pull request
Mar 21, 2024
## @helia/verified-fetch [1.3.0](https://github.com/ipfs/helia-verified-fetch/compare/@helia/verified-fetch-1.2.1...@helia/verified-fetch-1.3.0) (2024-03-21) ### Features * abort signals are respected ([#26](#26)) ([30148fe](30148fe)) ### Bug Fixes * set cache-control header correctly ([#19](#19)) ([114f3a4](114f3a4)), closes [#17](#17) [#23](#23) [#10](#10)
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
github-actions bot
pushed a commit
that referenced
this pull request
Oct 14, 2024
## 1.0.0 (2024-10-14) ### ⚠ BREAKING CHANGES * upgrade to helia v5 (#107) ### Features * abort signals are respected ([#26](#26)) ([30148fe](30148fe)) * allow use as a singleton ([#413](https://github.com/ipfs/helia-verified-fetch/issues/413)) ([7cbeed0](7cbeed0)) * create @helia/verified-fetch ([#392](https://github.com/ipfs/helia-verified-fetch/issues/392)) ([06d36fe](06d36fe)) * customize ipns dnsResolvers ([#445](https://github.com/ipfs/helia-verified-fetch/issues/445)) ([8f60822](8f60822)) * download tars from @helia/verified-fetch ([#442](https://github.com/ipfs/helia-verified-fetch/issues/442)) ([70ddd00](70ddd00)) * require content-type parser to set content-type ([#423](https://github.com/ipfs/helia-verified-fetch/issues/423)) ([b39d07c](b39d07c)) * support `Accept` header in `@helia/verified-fetch` ([#438](https://github.com/ipfs/helia-verified-fetch/issues/438)) ([54c4383](54c4383)) * support downloading car files from @helia/verified-fetch ([#441](https://github.com/ipfs/helia-verified-fetch/issues/441)) ([703980c](703980c)) * support http range header ([#10](#10)) ([9f5078a](9f5078a)) * support IPFS/IPNS paths, Gateways, etc ([#4](#4)) ([e7f1816](e7f1816)) * support redirects for UnixFS directories ([#5](#5)) ([4601d46](4601d46)) * support requesting raw IPNS records in @helia/verified-fetch ([#443](https://github.com/ipfs/helia-verified-fetch/issues/443)) ([e92086a](e92086a)) * upgrade to helia v5 ([#107](#107)) ([91a6473](91a6473)) * use blockstore sessions ([#50](#50)) ([541dd64](541dd64)) * use the waterworks trustless gateway by default ([#94](#94)) ([2db8d1e](2db8d1e)) ### Bug Fixes * @helia/verified-fetch init args are optional ([#412](https://github.com/ipfs/helia-verified-fetch/issues/412)) ([e6d41a0](e6d41a0)) * aborted signals throw libp2p AbortError ([#30](#30)) ([4575791](4575791)) * add interop tests and update project config ([fdc83b8](fdc83b8)) * allow contentTypeParser with Helia instance ([#427](https://github.com/ipfs/helia-verified-fetch/issues/427)) ([b5c95e6](b5c95e6)) * append query path to path resolved from IPNS name ([#3](#3)) ([fd86e6a](fd86e6a)) * byte range request end should never equal file size ([#24](#24)) ([aafc567](aafc567)) * content-type response header hints how to process response ([#426](https://github.com/ipfs/helia-verified-fetch/issues/426)) ([8b78e79](8b78e79)) * decodeURIComponent on path parts ([#40](#40)) ([f628cf6](f628cf6)) * do not coerce `undefined` to `null` for JSON serialization ([#2](#2)) ([d36ce29](d36ce29)) * gateway conformance improvements ([#85](#85)) ([7281078](7281078)) * gateway conformance tests ([#81](#81)) ([d0a3b6b](d0a3b6b)) * identity CIDs use contentTypeParser ([#49](#49)) ([3014498](3014498)) * implicit accept header can be overridden by format query ([#36](#36)) ([75c0b75](75c0b75)) * missing paths returns 404 instead of 502 ([#59](#59)) ([291a054](291a054)), closes [#53](#53) * pass url and body to badRequestResponse ([#44](#44)) ([cc228e6](cc228e6)) * prevent interop timeouts with fast fixture loading ([#73](#73)) ([a43d994](a43d994)) * reduce dagPb and dagCbor handler complexity ([#45](#45)) ([3b41752](3b41752)) * remove redundant abort controller ([#41](#41)) ([04b220d](04b220d)) * remove stubs from verified-fetch tests ([#431](https://github.com/ipfs/helia-verified-fetch/issues/431)) ([cbab8b0](cbab8b0)) * set cache-control header correctly ([#19](#19)) ([114f3a4](114f3a4)), closes [#17](#17) [#23](#23) [#10](#10) * support https?://<dnsLink>.ipns.<gateway> urls ([#16](#16)) ([0ece19a](0ece19a)) * ttl and caching for ipns urls ([#34](#34)) ([44ac5a1](44ac5a1)) * unixfs dir redirect ([#33](#33)) ([32ca87f](32ca87f)) * update @helia/ipns and dns config ([#18](#18)) ([9f88c54](9f88c54)) * use CID object for verified-fetch progress events ([#425](https://github.com/ipfs/helia-verified-fetch/issues/425)) ([b010184](b010184)) * use ipfs-unixfs-exporter directly ([#42](#42)) ([4532bf1](4532bf1)) * verified-fetch etag header ([#434](https://github.com/ipfs/helia-verified-fetch/issues/434)) ([f9b1ffe](f9b1ffe)) * verified-fetch header cache-control typo ([#433](https://github.com/ipfs/helia-verified-fetch/issues/433)) ([ea39b48](ea39b48)) * walking dag-cbor paths ([#39](#39)) ([99668ce](99668ce)) * X-Ipfs-Path is set correctly ([#46](#46)) ([5bb6685](5bb6685)) ### Trivial Changes * add tests for directory redirects for gateways ([#15](#15)) ([269609d](269609d)) * fix build ([#22](#22)) ([01261fe](01261fe)) * initial import ([4b4fb8c](4b4fb8c)) * **release:** 1.0.0 [skip ci] ([ef5b6a5](ef5b6a5)), closes [#68](#68) * **release:** 1.0.0 [skip ci] ([a584a1a](a584a1a)) * **release:** 1.0.0 [skip ci] ([5207a64](5207a64)) * **release:** 1.0.1 [skip ci] ([5499724](5499724)), closes [#81](#81) * **release:** 1.0.1 [skip ci] ([8d2fdda](8d2fdda)), closes [#2](#2) * **release:** 1.0.2 [skip ci] ([104af2d](104af2d)), closes [#3](#3) * **release:** 1.1.0 [skip ci] ([e300af2](e300af2)) * **release:** 1.1.0 [skip ci] ([f4fcb4d](f4fcb4d)), closes [#4](#4) [#5](#5) [#6](#6) * **release:** 1.1.0 [skip ci] ([b9b959b](b9b959b)) * **release:** 1.1.1 [skip ci] ([8ff0b8b](8ff0b8b)), closes [#83](#83) * **release:** 1.1.1 [skip ci] ([6f2c6bd](6f2c6bd)), closes [#15](#15) [#14](#14) * **release:** 1.1.2 [skip ci] ([6161aba](6161aba)), closes [#85](#85) * **release:** 1.1.2 [skip ci] ([4e1abc0](4e1abc0)), closes [#16](#16) * **release:** 1.1.3 [skip ci] ([14c2302](14c2302)), closes [#18](#18) * **release:** 1.10.0 [skip ci] ([4589c26](4589c26)) * **release:** 1.10.1 [skip ci] ([140bab3](140bab3)), closes [#33](#33) * **release:** 1.11.0 [skip ci] ([39bb14e](39bb14e)) * **release:** 1.12.0 [skip ci] ([50682bf](50682bf)) * **release:** 1.13.0 [skip ci] ([e00d41c](e00d41c)) * **release:** 1.14.0 [skip ci] ([2521838](2521838)) * **release:** 1.15.0 [skip ci] ([a38a0c6](a38a0c6)) * **release:** 1.15.1 [skip ci] ([f59e862](f59e862)), closes [#47](#47) * **release:** 1.16.0 [skip ci] ([0a18c44](0a18c44)) * **release:** 1.17.0 [skip ci] ([f5299cf](f5299cf)) * **release:** 1.18.0 [skip ci] ([7771af2](7771af2)) * **release:** 1.19.0 [skip ci] ([089635d](089635d)) * **release:** 1.2.0 [skip ci] ([befd270](befd270)) * **release:** 1.2.0 [skip ci] ([e836abf](e836abf)), closes [#10](#10) [#22](#22) * **release:** 1.2.0 [skip ci] ([77d5e9e](77d5e9e)) * **release:** 1.2.1 [skip ci] ([718644b](718644b)), closes [#24](#24) * **release:** 1.20.0 [skip ci] ([abaaeab](abaaeab)) * **release:** 1.21.0 [skip ci] ([aeb5a89](aeb5a89)) * **release:** 1.21.1 [skip ci] ([f7ac2e7](f7ac2e7)), closes [#73](#73) * **release:** 1.22.0 [skip ci] ([6d4b26c](6d4b26c)), closes [#50](#50) * **release:** 1.23.0 [skip ci] ([754e219](754e219)) * **release:** 1.24.0 [skip ci] ([c1a795f](c1a795f)) * **release:** 1.24.1 [skip ci] ([9f8fb86](9f8fb86)), closes [#85](#85) [#54](#54) * **release:** 1.25.0 [skip ci] ([3d8bb78](3d8bb78)) * **release:** 1.3.0 [skip ci] ([43c41e4](43c41e4)), closes [#26](#26) [#17](#17) [#23](#23) [#10](#10) * **release:** 1.3.0 [skip ci] ([bf8af38](bf8af38)) * **release:** 1.3.1 [skip ci] ([0bf5847](0bf5847)), closes [#30](#30) * **release:** 1.3.10 [skip ci] ([6768fbd](6768fbd)), closes [#51](#51) * **release:** 1.3.11 [skip ci] ([8606b15](8606b15)) * **release:** 1.3.12 [skip ci] ([969a2e0](969a2e0)), closes [#42](#42) * **release:** 1.3.13 [skip ci] ([a180547](a180547)), closes [#53](#53) [#46](#46) * **release:** 1.3.14 [skip ci] ([31f238a](31f238a)), closes [#63](#63) * **release:** 1.3.2 [skip ci] ([2e141ca](2e141ca)), closes [#33](#33) * **release:** 1.3.3 [skip ci] ([19574cf](19574cf)), closes [#34](#34) * **release:** 1.3.4 [skip ci] ([7b06a56](7b06a56)), closes [#40](#40) * **release:** 1.3.5 [skip ci] ([554bb3e](554bb3e)), closes [#41](#41) * **release:** 1.3.6 [skip ci] ([489a1d6](489a1d6)), closes [#36](#36) * **release:** 1.3.7 [skip ci] ([d0114c1](d0114c1)), closes [#39](#39) * **release:** 1.3.8 [skip ci] ([0a37856](0a37856)), closes [#44](#44) [#47](#47) * **release:** 1.3.9 [skip ci] ([768e3ff](768e3ff)), closes [#49](#49) * **release:** 1.4.0 [skip ci] ([dab598a](dab598a)), closes [#50](#50) * **release:** 1.4.0 [skip ci] ([e008b20](e008b20)) * **release:** 1.4.1 [skip ci] ([e3c0a68](e3c0a68)), closes [#68](#68) [#72](#72) [#72](#72) * **release:** 1.4.2 [skip ci] ([2633d31](2633d31)), closes [#45](#45) * **release:** 1.4.3 [skip ci] ([924a6df](924a6df)), closes [#85](#85) * **release:** 1.5.0 [skip ci] ([4885fde](4885fde)), closes [#94](#94) * **release:** 1.5.0 [skip ci] ([71d613d](71d613d)) * **release:** 1.5.1 [skip ci] ([4e6775c](4e6775c)), closes [#12](#12) * **release:** 1.6.0 [skip ci] ([03d0409](03d0409)) * **release:** 1.7.0 [skip ci] ([8bf9c9f](8bf9c9f)) * **release:** 1.8.0 [skip ci] ([a240056](a240056)) * **release:** 1.9.0 [skip ci] ([8715740](8715740)) * start running CI again ([#64](#64)) ([9399f95](9399f95)) * stub blocks interface in tests ([#418](https://github.com/ipfs/helia-verified-fetch/issues/418)) ([a06f64b](a06f64b)) * unskip IPNS test ([#6](#6)) ([76485a4](76485a4)) * Update .github/pull_request_template.md [skip ci] ([5b825e6](5b825e6)) * Update .github/workflows/stale.yml [skip ci] ([3e40ad0](3e40ad0)) * update project config ([#419](https://github.com/ipfs/helia-verified-fetch/issues/419)) ([cb9dc90](cb9dc90)) ### Documentation * add link to blog post and ready-to-run example ([#63](#63)) ([696ed57](696ed57)) * add link to the blog post and example ([#87](#87)) ([f30f979](f30f979)) * change cidv0 to cidv1 in the readme ([#105](#105)) ([060e726](060e726)) * fix readme link ([#51](#51)) ([8a41c57](8a41c57)) * generate readme ([#70](#70)) ([a75567a](a75567a)), closes [#68](#68) [#72](#72) [#72](#72) * move the package docs to root of repo ([#1](#1)) ([8fb4a76](8fb4a76)) * update api docs link in readme ([#14](#14)) ([d615633](d615633)) * update documented default value ([161a470](161a470)), closes [/github.com/multiformats/js-dns/blob/a56c9e0b953d644392cf10fd0792757da0d61c32/src/resolvers/default.browser.ts#L6-L7](https://github.com/ipfs//github.com/multiformats/js-dns/blob/a56c9e0b953d644392cf10fd0792757da0d61c32/src/resolvers/default.browser.ts/issues/L6-L7) ### Dependencies * bump aegir from 42.2.11 to 44.1.4 ([#108](#108)) ([e36fbff](e36fbff)) * bump kubo from 0.26.0 to 0.27.0 ([#12](#12)) ([92cad49](92cad49)) * bump kubo from 0.27.0 to 0.28.0 ([#54](#54)) ([3579844](3579844)) * updating all deps ([#47](#47)) ([6d0ffd8](6d0ffd8)) ### Tests * add gateway conformance tests ([#67](#67)) ([30958fb](30958fb)), closes [#68](#68) * use successRate for gateway conformance tests ([#83](#83)) ([5f71a33](5f71a33))
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
Add Range header support.
Description
Fixes #9
Notes & open questions
handling ReadableStreamstests, but these are covered byrange requeststests.. going to try to fix quickly and if I can't i'll remove them.Change checklist