Skip to content

Commit

Permalink
Merge branch 'dev' into 'master'
Browse files Browse the repository at this point in the history
Merge features from dev to master

Closes #5

See merge request tgeorg/vo-scraper!12
  • Loading branch information
gteufelberger committed Feb 21, 2022
2 parents a9b7cdb + a8d0797 commit 990af02
Show file tree
Hide file tree
Showing 3 changed files with 31 additions and 19 deletions.
7 changes: 2 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ Example:

https://video.ethz.ch/.episode-video.json?recordId=3f6dee77-396c-4e2e-a312-a41a457b319f

This file contains links to all available video streams (usually 1080p, 720p, and 360p). Note that if a lecture requires a login, this file will only be accessible if you a cookie with a valid login-token!
This file contains links to all available video streams (usually 1080p, 720p, and 360p). Note that if a lecture requires a login, this file will only be accessible if you have a cookie with a valid login-token!

The link to the video stream looks something like this:

Expand Down Expand Up @@ -193,14 +193,11 @@ In both cases we get back a cookie which we then can include when requesting the
3. If the lecture is password protected, make sure you use the correct credentials. Most protected lectures require your NETHZ credentials while some use a custom username and password.
4. Make sure you're running the newest version of the scraper by re-downloading the script from the repository. There might have been an update.
5. Check whether other lectures still work. Maybe the site was updated which broke the scraper.
6. Enable the debug flag with `-v` and see whether any of the additional information now provided is helpful.
6. Enable the debug flag with `--verbose` and see whether any of the additional information now provided is helpful.
7. Check "[How does it acquire the videos?](#how_it_works)" and see whether you can manually reach the video in your browser following the steps described there.
8. After having tried all that without success, feel free to open up a new issue. Make sure to explain what you have tried and what the results were. There is no guarantee I will respond within reasonable time as I'm a busy student myself. If you can fix the issue yourself, feel free to open a merge request with the fix.


### Q: Can you fix *X*? Can you implement feature *Y*?

#### A: Feel free to open an issue [here](https://gitlab.ethz.ch/tgeorg/vo-scraper/issues). Merge requests are always welcome but subject to my own moderation.
***

Loosely based on https://gitlab.ethz.ch/dominik/infk-vorlesungsscraper
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.0.1
3.1.0
41 changes: 28 additions & 13 deletions vo-scraper.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@
gitlab_issue_page = gitlab_repo_page + "issues"
gitlab_changelog_page = gitlab_repo_page + "-/tags/v"
remote_version_link = gitlab_repo_page + "raw/master/VERSION"
program_version = '3.0.1'
program_version = '3.1.0'

# For web requests
user_agent = 'Mozilla/5.0'
Expand Down Expand Up @@ -376,6 +376,20 @@ def get_user_choice(max_episode_number):

return choice

def resolution_from_input(resolution):
# Turn named resolution into number
if resolution.lower() == "4k": resolution = "2160p"
if resolution.lower() == "2k": resolution = "1440p"
if resolution.lower() == "fullhd": resolution = "1080p"
if resolution.lower() == "hd": resolution = "720p"
if resolution.lower() == "sd": resolution = "480"
if resolution.lower() == "high": resolution = "1080p"
if resolution.lower() == "medium": resolution = "720p"
if resolution.lower() == "low": resolution = "360p"

# Parse the given video resolution
return int(str(resolution).replace("p", ""))


def get_video_src_link_for_resolution(video_json_data, video_quality):
"""
Expand Down Expand Up @@ -406,16 +420,7 @@ def get_video_src_link_for_resolution(video_json_data, video_quality):
elif video_quality == "highest":
quality_index = 0
else:
# Turn named resolution into number
if video_quality.lower() == "4k":
video_quality = "2160p"
if video_quality.lower() == "2k":
video_quality = "1440p"
if video_quality.lower() == "hd":
video_quality = "1080p"

# Parse the given video resolution
video_quality_parsed = int(str(video_quality).replace("p", ""))
video_quality_parsed = resolution_from_input(video_quality)

# Subtract requested from available resolutions to get the closest one
list_of_quality_diff = [(x[0], abs(video_quality_parsed - x[2])) for x in resolutions]
Expand Down Expand Up @@ -549,7 +554,7 @@ def vo_scrapper(vo_link, video_quality, user, passw):
video_json_data = json.loads(r.text)

# Get video src url from json based on resolution
video_src_link, available_video_quality = get_video_src_link_for_resolution(video_json_data, video_quality)
video_src_link, available_video_quality = get_video_src_link_for_resolution(video_json_data, video_quality)

lecture_title = vo_json_data['title']
episode_title = vo_json_data["episodes"][item_nr]["title"]
Expand Down Expand Up @@ -883,7 +888,7 @@ def setup_arg_parser():
parser.add_argument(
"-q", "--quality",
default='HD',
help="Select a specific video resolution. Either specify a height directly like `1080p` or use the keywords `HD`, `2K`, and `4K`. The scraper will try to download the video closest to the specified resolution. Additionally you can also use `highest` and `lowest` to always download the highest or lowest quality respectively.",
help="Select a specific video resolution. Either specify a height directly like `1080p` or use the keywords `FullHD`, `2K`, and `4K`. The scraper will try to download the video closest to the specified resolution. Additionally you can also use `highest` and `lowest` to always download the highest or lowest quality respectively.",
)
parser.add_argument(
"-sc", "--skip-connection-check",
Expand Down Expand Up @@ -962,6 +967,8 @@ def remove_illegal_characters(str):
with open(PARAMETER_FILE) as f:
# Read file and remove trailing whitespaces and newlines
parameters = [x.strip() for x in f.readlines()]
# Remove comments i.e. lines starting with `#`
parameters = [line for line in parameters if not line.startswith('#')]
# Split strings with spaces
parameters = [words for segments in parameters for words in segments.split()]
# Add parameters list
Expand Down Expand Up @@ -1005,6 +1012,14 @@ def remove_illegal_characters(str):
else:
print_information("Update check skipped.", verbose_only=True)

# Print selected quality
if video_quality == "lowest" or video_quality == "highest":
quality_string = video_quality
else:
quality_string = str(resolution_from_input(video_quality)) + 'p'
print_information("Selected quality for downloads: " + quality_string)
print_information("")

# Run scraper for every link provided to get video sources for each episode
for (link, user, password) in lecture_objects:
print_information("Currently selected: " + link, verbose_only=True)
Expand Down

0 comments on commit 990af02

Please sign in to comment.