fix: temp-fix - #1894
Conversation
5c771e5 to
61f3a9a
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The history listing and episode-matching logic has confirmed correctness issues (subshell/background wait misuse, racey globals, and regex-based episode matching) that can produce wrong or incomplete results.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates ani-cli’s scraping layer to work with HiAnime again, preserves season switching, and adds a Miruro-backed variant script with improved stream selection/referrer handling.
Changes:
- Replaces the previous provider scraping logic in
ani-cliwith a HiAnime-based flow (search/desc/episodes/servers → m3u8 selection). - Adds a new
ani-cli-miruroscript that uses Miruro’s API backend and implements provider prioritization + HLS/MP4 probing. - Adds a
.gitignoreentry to avoid committingscraper/node_modules/.
File summaries
| File | Description |
|---|---|
| ani-cli-miruro | New Miruro-backed CLI script including episode/source fetching and playback logic. |
| ani-cli | Updates main scraper to HiAnime, adds season switching, refactors menus, and adjusts history/playback flows. |
| .gitignore | Ignores scraper/node_modules/. |
Review details
Suppressed comments (4)
ani-cli:340
backup_old_histappends to${histfile}.newwithout truncating/creating it first, and then unconditionallymvs it. If${histfile}.newalready exists (e.g., from a previous interrupted run) or the history is empty, this can duplicate entries or fail the migration.
backup_old_hist() {
backupfile="${histfile}.v4"
while IFS=" " read -r ep_no anime_id anime_title; do
case "$anime_id" in
*-*) printf "%s\t%s\t%s\n" "$ep_no" "$anime_id" "$anime_title" >>"${histfile}.new" ;;
ani-cli:435
canon_ep_nois derived via a regex-basedsedmatch on$ep_no. For episode numbers like12.5,.is a regex wildcard and can produce an incorrect line number, which then breaks ani-skip episode mapping.
This issue also appears on line 441 of the same file.
canon_ep_no="$(printf "%s" "$ep_list" | sed -n "/^${ep_no}$/=")"
ani-cli:442
- Episode validation and
canon_ep_nocalculation use regex-basedgrep/sedmatching on$ep_no. With non-integer episode numbers (e.g.12.5), this can accept the wrong episode and compute an incorrectcanon_ep_no. Prefer fixed-string matching.
printf "%s" "$ep_list" | grep -q "^$ep_no$" || die "Invalid episode!"
canon_ep_no="$(printf "%s" "$ep_list" | sed -n "/^${ep_no}$/=")"
ani-cli:599
- In the history path,
process_hist_entryis started as a background job inside a command substitution, butwaitis executed outside that subshell. The parent shell cannot wait for jobs started in the command-substitution subshell, so the list can be incomplete/truncated. Movewaitinside the same command substitution (and pass args to avoid data races).
anime_list=$(while read -r ep_no anime_id anime_title; do process_hist_entry & done <"$histfile")
wait
- Files reviewed: 1/1 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| hianime_desc() { | ||
| #shellcheck disable=SC2059 | ||
| _page="$(anidb_curl "$(printf "$desc_api" "$1")" | tr '\n' ' ' | sed 's|<a href|\n<a href|g')" | ||
| mal_id="$(printf "%s" "$_page" | sed -nE 's|.*https://myanimelist.net/anime/([0-9]+)/.*|\1|p')" | ||
| seasons="$(printf "%s" "$_page" | sed -n '/>Seasons</,/>Details</p' | sed -nE 's|.*anime/([^"]+-[0-9]+)"[^>]*title="([^"]+)".*|\1\t\2|p' | sed -e "s|'|'|g")" | ||
| _page="$(hianime_curl "$(printf "$desc_api" "$1")")" | ||
| mal_id="$(printf "%s" "$_page" | sed -n 's|.*myanimelist\.net/anime/\([0-9][0-9]*\).*|\1|p')" | ||
| # Related entries are grouped in the season list on HiAnime pages. |
| _lang="jpn" | ||
| [ "$2" = "dub" ] && _lang="eng" | ||
| hianime_m3u8() { | ||
| _ep_id=$(printf "%s" "$episode_maps" | sed -n "s|^\([0-9][0-9]*\)\t${1}$|\1|p") |
| process_hist_entry() { | ||
| ep_list=$(anidb_episodes "$anime_id" | cut -f 2) | ||
| ep_list=$(hianime_episodes "$anime_id" | cut -f 2) | ||
| ep_no=$(printf "%s" "$ep_list" | sed -n "/^${ep_no}$/{n;p;}") 2>/dev/null | ||
| [ -n "$ep_no" ] && printf "%s\t%s - episode %s\n" "$anime_id" "$anime_title" "$ep_no" | ||
| } |
61f3a9a to
872bfbc
Compare
|
Why is copilot AI here??? Wtf man |
|
@Dhairya3391 if this fixes the outage, bump it to v5.1 pls |
872bfbc to
9a00809
Compare
|
Oh, you brought your own copilot, that's fair |
9a00809 to
f659d8c
Compare
|
@port19x anything else needed? it works as of now as fix but can't rely on this because site is not the best looking. |
|
@Dhairya3391 why ping pystardust lol. |
|
Please add your used AI tool as a coauthor, I'll give this a review pass in codex. |
f659d8c to
2f1d1c2
Compare
|
I don't think anidb is coming back. This site also doesn't look very reliable. This is the other candidate that I could successfully scrape without being hls-blocked https://anizone.to/ |
2f1d1c2 to
b39fe44
Compare
|
Want me to come into the discord again for the next provider? |
|
yeah we'll need to look at reliable providers again |
|
I dont know how much productive it would be but we could communicate better i guess. |
|
I'm billionairelover |
b39fe44 to
44f7eeb
Compare
Co-authored-by: Codex <codex@openai.com>
44f7eeb to
685cb77
Compare
Tsagar3
left a comment
There was a problem hiding this comment.
Fix: subtitle tracks are dropped in the HiAnime flow
I tested this PR and found that episodes play without subtitles. Here's the root cause and a minimal fix.
Root cause
The HiAnime embed payload (the blurb decoded from window.__P) carries the subtitles outside the HLS playlist:
{
"download_url": "/download/mal/21/1/sub",
"src": "https://hls2.aniwatchtv.uk/v/.../master.m3u8",
"subtitles": [
{ "lang": "en", "label": "English", "default": true, "src": "https://hls2.aniwatchtv.uk/v/.../subs/6e4a0p3k8ehe9eee.vtt" }
],
"sprite_vtt": ""
}The master.m3u8 (and the resulting index.m3u8) only contains #EXT-X-STREAM-INF variants — there are no #EXT-X-MEDIA:TYPE=SUBTITLES tracks. ffprobe on a segment confirms only h264 video + aac audio.
deobfuscate_blob only extracts the stream src (the .m3u8), so the separate .vtt file referenced by "subtitles" is discarded. mpv then never receives a subtitle track, hence no subs.
Fix
deobfuscate_blobalso extracts the.vttsrc from thesubtitlesarray.hianime_m3u8persists it to a temp file (reusing the existinghistfile-based tempfile pattern), cleared when the episode has no subs.get_video_linkreads it intosub_link.play_episodepasses--sub-file="$sub_link"to mpv (flatpak too) when present.
Verified with mpv against the extracted VTT:
● Subs --sid=1 '6e4a0p3k8ehe9eee.vtt' (webvtt) [external] and it fetches the .vtt with the correct Referer.
Shell quirk worth knowing
The decoded JSON has no trailing newline, and a lone printf "%b" | sed -n 's/.../p' may then emit its output without a trailing newline. Two consecutive extracts in a subshell can therefore concatenate into a single line (master.m3u8https://...vtt). The patch wraps each extract in printf "%s\n" "$( ... )" and filters blank lines downstream so the two URLs always land on their own lines.
Suggested diff
@@ cleanup() @@
cleanup() {
- tput sgr0 # clears colors
- rm -f "${histfile}.new" # remove temp logfile
+ tput sgr0 # clears colors
+ rm -f "${histfile}.new" # remove temp logfile
+ rm -f "${histfile}.subs" # remove temp subtitle url
}
@@ deobfuscate_blob() @@
- printf "%b" "$_output" | sed -n 's|.*"src":"\([^"]*\.m3u8[^"]*\)".*|\1|p'
+ printf "%s\n" "$(printf "%b" "$_output" | sed -n 's|.*"src":"\([^"]*\.m3u8[^"]*\)".*|\1|p')"
+ printf "%s\n" "$(printf "%b" "$_output" | sed -n 's|.*"subtitles":\[[^]]*"src":"\([^"]*\.vtt[^"]*\)".*|\1|p')"
)
@@ hianime_m3u8() - "5. Deobfuscate" @@
- _m3u8_master="$(deobfuscate_blob "$_blob")"
+ _deobfuscated="$(printf "%s\n" "$(deobfuscate_blob "$_blob")" | sed '/^$/d')"
+ _m3u8_master="$(printf "%s\n" "$_deobfuscated" | sed -n '1p')"
+ _subs="$(printf "%s\n" "$_deobfuscated" | sed -n '2p')"
[ -z "$_m3u8_master" ] && return 1
+ if [ -n "$_subs" ]; then
+ printf "%s\n" "$_subs" >"${histfile}.subs"
+ else
+ rm -f "${histfile}.subs"
+ fi
@@ get_video_link() @@
links=$(hianime_m3u8 "$ep_no" "$mode" | sort -g -r -s)
[ -z "$links" ] && die "No sources found for $mode!"
info "hianime.at links fetched"
+ sub_link="$(cat "${histfile}.subs" 2>/dev/null)"
select_quality "$quality"
@@ play_episode() @@
[ -z "$video_link" ] && get_video_link
+ sub_file_flag=""
+ [ -n "$sub_link" ] && sub_file_flag="--sub-file=$sub_link"
# wait for the previous player to close (in range selection)
wait
# shellcheck disable=SC2086
case "$player_function" in
- debug) printf "All links:\n%s\nSelected link:\n%s\n" "$links" "$video_link" ;;
+ debug) printf "All links:\n%s\nSelected link:\n%s\nSubtitles:\n%s\n" "$links" "$video_link" "$sub_link" ;;
...
- *flatpak*mpv*) flatpak run io.mpv.Mpv --http-header-fields="Referer: https://zokoanime.video/" $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 & ;;
+ *flatpak*mpv*) flatpak run io.mpv.Mpv --http-header-fields="Referer: https://zokoanime.video/" $sub_file_flag $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 & ;;
*mpv*)
if [ "$no_detach" = 0 ]; then
- nohup $player_function --http-header-fields="Referer: https://zokoanime.video/" $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 &
+ nohup $player_function --http-header-fields="Referer: https://zokoanime.video/" $sub_file_flag $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 &
else
- $player_function --http-header-fields="Referer: https://zokoanime.video/" $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link"
+ $player_function --http-header-fields="Referer: https://zokoanime.video/" $sub_file_flag $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link"Note: subs are currently only wired into the mpv paths; VLC and Syncplay would need --sub-file equivalents if so desired.
Happy to turn this into a commit on the branch if it helps.
- deobfuscate_blob extracts both m3u8 and subtitles .vtt src (proper newline handling for printf/sed quirk)
- hianime_m3u8 persists .vtt to ${histfile}.subs, cleans on failures
- get_video_link downloads VTT locally with Referer, fallback to remote URL
- play_episode passes --sub-file for mpv/flatpak, --mpv-sub-files for iina (alias fix), :sub-file for vlc, --sub-file via syncplay mpv passthrough
- cleanup removes temp subs/vtt, download fetches vtt, replay preserves subs
Fixes subs not loading where master.m3u8 has no #EXT-X-MEDIA tracks (verified mpv --sid=1 external webvtt)
|
added sub fix, it's bit janky but works for now |
|
Still a bug in deobfuscate, last line: The sed regex stretches is greedy, ignores languages of interest and thus extends to the end of array of subtitles: it always and only reports the vtt of the last-listed language: To instead force english, this scrap works (plus some debug on stderr; line-splitting in sed really isn't my forte): Likely the language selection grep should be moved to the caller, rather than hidden in deobfuscate, i.e. deobfuscate in that case would return lines 2..n being the languages to egrep | head -1 against. Maybe an ordered list of regexes, (say ':' separated like in $PATH) would be better, as that would permit the user to specify the languages with an order of precedence. OT: how does language selection work anyway in ani-cli, say for a dub? I didn't see anything when skimming for any use of $lang/$_lang or http headers (and $_lang is the dub/sub selection instead)? |
|
Good catch @jakobi — I can confirm the greedy match. I reproduced it with a JSON mirroring the real embed (3 languages, English marked The problem is The clean fix is to emit one line per subtitle entry and let the caller pick: --- deobfuscate_blob
- printf "%s\n" "$(printf "%b" "$_output" | sed -n 's|.*"subtitles":\[[^]]*"src":"\([^"]*\.vtt[^"]*\)".*|\1|p')"
+ printf "%s\n" "$(printf "%b" "$_output" | tr '}' '\n' | sed -n 's|.*"lang":"\([^"]*\)".*"label":"\([^"]*\)".*"src":"\([^"]*\.vtt[^"]*\)".*|\1\t\2\t\3|p')"
+
+# now emits: lang<TAB>label<TAB>src , one per language
--- hianime_m3u8 (deobfuscate step)
_deobfuscated="$(printf "%s\n" "$(deobfuscate_blob "$_blob")" | sed '/^$/d')"
_m3u8_master="$(printf "%s\n" "$_deobfuscated" | sed -n '1p')"
- _subs="$(printf "%s\n" "$_deobfuscated" | sed -n '2p')"
+ _sub_entries="$(printf "%s\n" "$_deobfuscated" | sed -n '2,$p')"
+ _sub_langs="${ANI_CLI_SUB_LANG:-en}"
+ _subs="$(printf "%s\n" "$_sub_entries" | grep -iE "^$(printf '%s' "$_sub_langs" | tr ',:' '|')[[:space:]]" | head -n1 | cut -f 3)"
+ [ -z "$_subs" ] && _subs="$(printf "%s\n" "$_sub_entries" | head -n1 | cut -f 3)"Selection logic (verified):
Re And to answer your OT: language selection in ani-cli today is only sub vs dub — |
anidb.app has been behind a 503 maintenance page for several days, which made every episode launch fail. Playback now falls back to hianime.at when anidb.app cannot serve an episode. anidb.app stays primary and its resolution path is unchanged. AllAnime still supplies search, episode lists, and every tracking hook; hianime.at only resolves stream links for an already selected title and episode. The hianime scraping (endpoints, HD-1 embed, base64(json XOR "otaku-embed-v1") player config) is derived from pystardust/ani-cli#1894 and #1897; see NOTICE.md. Also in this change: - Cloudflare detection no longer treats a bare challenges.cloudflare.com reference as an interstitial. Providers embed the Turnstile widget on ordinary pages, so hianime.at was rejected on every request. anidb.app's maintenance page is now reported as maintenance instead of a browser challenge. - Subtitle track selection prefers the default and then a genuine English label, so it cannot land on a Spanish track that is tagged lang=en. hianime.at does not burn subtitles in, so the external WebVTT is handed to the player. - hls_playlist_links labels a stream 1080p instead of Hls when RESOLUTION is last on the EXT-X-STREAM-INF line, so -q matches again for both providers. ANI_CLI_ANIDB=0 skips the primary provider, ANI_CLI_HIANIME=0 disables the fallback, and ANI_CLI_HIANIME_BASE retargets the fallback domain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0183q3i6yuiLUGn7acwLRsMs
- Dynamically select ZokoAnime server, adapting to HiAnime's server rename - Filter out Top 10 sidebar items from search results so invalid searches show no results - Support subtitle language preference (ANI_CLI_SUB_LANG, default en) and avoid greedy regex - Stream subtitles directly to players without temporary files - Escape subtitle URLs for IINA and use :input-slave for VLC - Forward skip_flag to syncplay and ensure mal_id is set before ani-skip runs - Prevent history collisions between v5.0 and HiAnime by checking full slug - Avoid global variable races in concurrent history processing - Use fixed-string matching for episode numbers to handle decimals like 12.5 - Optimize string parsing with POSIX parameter expansion and remove hianime_desc - Update documentation and man page references to hianime
Fixes the HiAnime provider after recent server changes, cleans up subtitle handling, and resolves several edge-case bugs.
Summary of Changes
:input-slavefor VLC.ANI_CLI_SUB_LANG) instead of always grabbing the last track in the list.ani-skipruns so--skipworks as expected.12.5so the dot isn't treated as a wildcard.printf | sed) with built-in shell parameter expansions, removes the unused description endpoint request, and increases curl max-time to 15s for large episode lists like One Piece.