Skip to content

fix: temp-fix - #1894

Open
Dhairya3391 wants to merge 3 commits into
pystardust:masterfrom
Dhairya3391:temp-fix
Open

fix: temp-fix#1894
Dhairya3391 wants to merge 3 commits into
pystardust:masterfrom
Dhairya3391:temp-fix

Conversation

@Dhairya3391

@Dhairya3391 Dhairya3391 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Fixes the HiAnime provider after recent server changes, cleans up subtitle handling, and resolves several edge-case bugs.

Summary of Changes

  • Fix HiAnime Server Rename: HiAnime renamed its video player server from HD-1 to ZokoAnime. The scraper now dynamically finds the ZokoAnime stream, with a fallback if server names change again.
  • Clean Search Results: Strips out the site's 'Top 10' sidebar so searches only return matching titles instead of appending ~30 unrelated shows. Invalid queries now correctly show 'No results found!'.
  • Subtitles Without Temp Files: Passes remote subtitle links directly to players (mpv, IINA, VLC, Syncplay) instead of downloading temp files. Colon-escapes URLs for IINA and uses :input-slave for VLC.
  • Subtitle Language Preference: Selects English by default (configurable via ANI_CLI_SUB_LANG) instead of always grabbing the last track in the list.
  • Intro Skipping: Ensures the anime ID is loaded before ani-skip runs so --skip works as expected.
  • History Fixes: Validates the full title slug on HiAnime so older v5.0 history won't play the wrong show. Passes explicit arguments to background history workers to eliminate variable races.
  • Decimal Episode Numbers: Uses exact text matching instead of regular expressions for episode numbers like 12.5 so the dot isn't treated as a wildcard.
  • Speed & Cleanup: Replaces subprocess calls (printf | sed) with built-in shell parameter expansions, removes the unused description endpoint request, and increases curl max-time to 15s for large episode lists like One Piece.
  • Documentation: Updates man pages and documentation files from anidb to hianime.

Copilot AI lite review requested due to automatic review settings September 5, 2026 13:25
@Dhairya3391
Dhairya3391 force-pushed the temp-fix branch 2 times, most recently from 5c771e5 to 61f3a9a Compare September 5, 2026 13:29
@Dhairya3391 Dhairya3391 changed the title temp-fix fix: temp-fix Sep 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The history listing and episode-matching logic has confirmed correctness issues (subshell/background wait misuse, racey globals, and regex-based episode matching) that can produce wrong or incomplete results.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates ani-cli’s scraping layer to work with HiAnime again, preserves season switching, and adds a Miruro-backed variant script with improved stream selection/referrer handling.

Changes:

  • Replaces the previous provider scraping logic in ani-cli with a HiAnime-based flow (search/desc/episodes/servers → m3u8 selection).
  • Adds a new ani-cli-miruro script that uses Miruro’s API backend and implements provider prioritization + HLS/MP4 probing.
  • Adds a .gitignore entry to avoid committing scraper/node_modules/.
File summaries
File Description
ani-cli-miruro New Miruro-backed CLI script including episode/source fetching and playback logic.
ani-cli Updates main scraper to HiAnime, adds season switching, refactors menus, and adjusts history/playback flows.
.gitignore Ignores scraper/node_modules/.
Review details

Suppressed comments (4)

ani-cli:340

  • backup_old_hist appends to ${histfile}.new without truncating/creating it first, and then unconditionally mvs it. If ${histfile}.new already exists (e.g., from a previous interrupted run) or the history is empty, this can duplicate entries or fail the migration.
backup_old_hist() {
    backupfile="${histfile}.v4"
    while IFS="	" read -r ep_no anime_id anime_title; do
        case "$anime_id" in
            *-*) printf "%s\t%s\t%s\n" "$ep_no" "$anime_id" "$anime_title" >>"${histfile}.new" ;;

ani-cli:435

  • canon_ep_no is derived via a regex-based sed match on $ep_no. For episode numbers like 12.5, . is a regex wildcard and can produce an incorrect line number, which then breaks ani-skip episode mapping.

This issue also appears on line 441 of the same file.

            canon_ep_no="$(printf "%s" "$ep_list" | sed -n "/^${ep_no}$/=")"

ani-cli:442

  • Episode validation and canon_ep_no calculation use regex-based grep/sed matching on $ep_no. With non-integer episode numbers (e.g. 12.5), this can accept the wrong episode and compute an incorrect canon_ep_no. Prefer fixed-string matching.
        printf "%s" "$ep_list" | grep -q "^$ep_no$" || die "Invalid episode!"
        canon_ep_no="$(printf "%s" "$ep_list" | sed -n "/^${ep_no}$/=")"

ani-cli:599

  • In the history path, process_hist_entry is started as a background job inside a command substitution, but wait is executed outside that subshell. The parent shell cannot wait for jobs started in the command-substitution subshell, so the list can be incomplete/truncated. Move wait inside the same command substitution (and pass args to avoid data races).
        anime_list=$(while read -r ep_no anime_id anime_title; do process_hist_entry & done <"$histfile")
        wait
  • Files reviewed: 1/1 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread ani-cli Outdated
Comment on lines +199 to +203
hianime_desc() {
#shellcheck disable=SC2059
_page="$(anidb_curl "$(printf "$desc_api" "$1")" | tr '\n' ' ' | sed 's|<a href|\n<a href|g')"
mal_id="$(printf "%s" "$_page" | sed -nE 's|.*https://myanimelist.net/anime/([0-9]+)/.*|\1|p')"
seasons="$(printf "%s" "$_page" | sed -n '/>Seasons</,/>Details</p' | sed -nE 's|.*anime/([^"]+-[0-9]+)"[^>]*title="([^"]+)".*|\1\t\2|p' | sed -e "s|&#039;|'|g")"
_page="$(hianime_curl "$(printf "$desc_api" "$1")")"
mal_id="$(printf "%s" "$_page" | sed -n 's|.*myanimelist\.net/anime/\([0-9][0-9]*\).*|\1|p')"
# Related entries are grouped in the season list on HiAnime pages.
Comment thread ani-cli Outdated
_lang="jpn"
[ "$2" = "dub" ] && _lang="eng"
hianime_m3u8() {
_ep_id=$(printf "%s" "$episode_maps" | sed -n "s|^\([0-9][0-9]*\)\t${1}$|\1|p")
Comment thread ani-cli
Comment on lines 347 to 351
process_hist_entry() {
ep_list=$(anidb_episodes "$anime_id" | cut -f 2)
ep_list=$(hianime_episodes "$anime_id" | cut -f 2)
ep_no=$(printf "%s" "$ep_list" | sed -n "/^${ep_no}$/{n;p;}") 2>/dev/null
[ -n "$ep_no" ] && printf "%s\t%s - episode %s\n" "$anime_id" "$anime_title" "$ep_no"
}
@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Why is copilot AI here??? Wtf man

@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

@Dhairya3391 if this fixes the outage, bump it to v5.1 pls

@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Oh, you brought your own copilot, that's fair

@Dhairya3391

Dhairya3391 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

@port19x anything else needed? it works as of now as fix but can't rely on this because site is not the best looking.
edit : misclicked on tag

@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

@Dhairya3391 why ping pystardust lol.
We can keep this around for now, but we'll wait a little for if anidb comes back up after maintenance

@port19x port19x changed the title fix: temp-fix fix: replace anidb with hianime provider Sep 5, 2026
@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Please add your used AI tool as a coauthor, I'll give this a review pass in codex.
Too fat and too early for me to manually read all the code

@71zenith

71zenith commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

I don't think anidb is coming back. This site also doesn't look very reliable. This is the other candidate that I could successfully scrape without being hls-blocked https://anizone.to/
But this is also another medium sized self hosted library and could go down as easily as anidb.

@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Hey @71zenith, can you look at my PR over at #1895

@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Want me to come into the discord again for the next provider?

@Dhairya3391

Copy link
Copy Markdown
Contributor Author

yeah we'll need to look at reliable providers again

@71zenith

71zenith commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

I dont know how much productive it would be but we could communicate better i guess.

@port19x

port19x commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

I'm billionairelover

@Tsagar3 Tsagar3 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix: subtitle tracks are dropped in the HiAnime flow

I tested this PR and found that episodes play without subtitles. Here's the root cause and a minimal fix.

Root cause

The HiAnime embed payload (the blurb decoded from window.__P) carries the subtitles outside the HLS playlist:

{
  "download_url": "/download/mal/21/1/sub",
  "src": "https://hls2.aniwatchtv.uk/v/.../master.m3u8",
  "subtitles": [
    { "lang": "en", "label": "English", "default": true, "src": "https://hls2.aniwatchtv.uk/v/.../subs/6e4a0p3k8ehe9eee.vtt" }
  ],
  "sprite_vtt": ""
}

The master.m3u8 (and the resulting index.m3u8) only contains #EXT-X-STREAM-INF variants — there are no #EXT-X-MEDIA:TYPE=SUBTITLES tracks. ffprobe on a segment confirms only h264 video + aac audio.

deobfuscate_blob only extracts the stream src (the .m3u8), so the separate .vtt file referenced by "subtitles" is discarded. mpv then never receives a subtitle track, hence no subs.

Fix

  • deobfuscate_blob also extracts the .vtt src from the subtitles array.
  • hianime_m3u8 persists it to a temp file (reusing the existing histfile-based tempfile pattern), cleared when the episode has no subs.
  • get_video_link reads it into sub_link.
  • play_episode passes --sub-file="$sub_link" to mpv (flatpak too) when present.

Verified with mpv against the extracted VTT:
● Subs --sid=1 '6e4a0p3k8ehe9eee.vtt' (webvtt) [external] and it fetches the .vtt with the correct Referer.

Shell quirk worth knowing

The decoded JSON has no trailing newline, and a lone printf "%b" | sed -n 's/.../p' may then emit its output without a trailing newline. Two consecutive extracts in a subshell can therefore concatenate into a single line (master.m3u8https://...vtt). The patch wraps each extract in printf "%s\n" "$( ... )" and filters blank lines downstream so the two URLs always land on their own lines.

Suggested diff

@@ cleanup() @@
 cleanup() {
-    tput sgr0               # clears colors
-    rm -f "${histfile}.new" # remove temp logfile
+    tput sgr0                    # clears colors
+    rm -f "${histfile}.new"      # remove temp logfile
+    rm -f "${histfile}.subs"     # remove temp subtitle url
 }

@@ deobfuscate_blob() @@
-    printf "%b" "$_output" | sed -n 's|.*"src":"\([^"]*\.m3u8[^"]*\)".*|\1|p'
+    printf "%s\n" "$(printf "%b" "$_output" | sed -n 's|.*"src":"\([^"]*\.m3u8[^"]*\)".*|\1|p')"
+    printf "%s\n" "$(printf "%b" "$_output" | sed -n 's|.*"subtitles":\[[^]]*"src":"\([^"]*\.vtt[^"]*\)".*|\1|p')"
 )

@@ hianime_m3u8() - "5. Deobfuscate" @@
-    _m3u8_master="$(deobfuscate_blob "$_blob")"
+    _deobfuscated="$(printf "%s\n" "$(deobfuscate_blob "$_blob")" | sed '/^$/d')"
+    _m3u8_master="$(printf "%s\n" "$_deobfuscated" | sed -n '1p')"
+    _subs="$(printf "%s\n" "$_deobfuscated" | sed -n '2p')"
     [ -z "$_m3u8_master" ] && return 1
+    if [ -n "$_subs" ]; then
+        printf "%s\n" "$_subs" >"${histfile}.subs"
+    else
+        rm -f "${histfile}.subs"
+    fi

@@ get_video_link() @@
     links=$(hianime_m3u8 "$ep_no" "$mode" | sort -g -r -s)
     [ -z "$links" ] && die "No sources found for $mode!"
     info "hianime.at links fetched"
+    sub_link="$(cat "${histfile}.subs" 2>/dev/null)"
     select_quality "$quality"

@@ play_episode() @@
     [ -z "$video_link" ] && get_video_link
+    sub_file_flag=""
+    [ -n "$sub_link" ] && sub_file_flag="--sub-file=$sub_link"
     # wait for the previous player to close (in range selection)
     wait
     # shellcheck disable=SC2086
     case "$player_function" in
-        debug) printf "All links:\n%s\nSelected link:\n%s\n" "$links" "$video_link" ;;
+        debug) printf "All links:\n%s\nSelected link:\n%s\nSubtitles:\n%s\n" "$links" "$video_link" "$sub_link" ;;
         ...
-        *flatpak*mpv*) flatpak run io.mpv.Mpv --http-header-fields="Referer: https://zokoanime.video/" $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 & ;;
+        *flatpak*mpv*) flatpak run io.mpv.Mpv --http-header-fields="Referer: https://zokoanime.video/" $sub_file_flag $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 & ;;
         *mpv*)
             if [ "$no_detach" = 0 ]; then
-                nohup $player_function --http-header-fields="Referer: https://zokoanime.video/" $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 &
+                nohup $player_function --http-header-fields="Referer: https://zokoanime.video/" $sub_file_flag $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link" >/dev/null 2>&1 &
             else
-                $player_function --http-header-fields="Referer: https://zokoanime.video/" $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link"
+                $player_function --http-header-fields="Referer: https://zokoanime.video/" $sub_file_flag $skip_flag $player_extra_flags --force-media-title="${anime_title} Episode ${ep_no}" "$video_link"

Note: subs are currently only wired into the mpv paths; VLC and Syncplay would need --sub-file equivalents if so desired.

Happy to turn this into a commit on the branch if it helps.

- deobfuscate_blob extracts both m3u8 and subtitles .vtt src (proper newline handling for printf/sed quirk)
- hianime_m3u8 persists .vtt to ${histfile}.subs, cleans on failures
- get_video_link downloads VTT locally with Referer, fallback to remote URL
- play_episode passes --sub-file for mpv/flatpak, --mpv-sub-files for iina (alias fix), :sub-file for vlc, --sub-file via syncplay mpv passthrough
- cleanup removes temp subs/vtt, download fetches vtt, replay preserves subs

Fixes subs not loading where master.m3u8 has no #EXT-X-MEDIA tracks (verified mpv --sid=1 external webvtt)
@Dhairya3391

Copy link
Copy Markdown
Contributor Author

added sub fix, it's bit janky but works for now

@jakobi

jakobi commented Sep 6, 2026

Copy link
Copy Markdown

Still a bug in deobfuscate, last line:

The sed regex stretches is greedy, ignores languages of interest and thus extends to the end of array of subtitles: it always and only reports the vtt of the last-listed language:

printf "%s\n" "$(printf "%b" "$_output" | \
   sed -n 's|.*"subtitles" \[[^]]*"src":"\([^"]*\.vtt[^"]*\)".*|\1|p')"

To instead force english, this scrap works (plus some debug on stderr; line-splitting in sed really isn't my forte):

# match against label, do avoid capturing parens in the regex
export _langre="english|german" 
printf "%b" "$_output" | perl -e '
   undef $/; $_=<>; 
   s/.*"subtitles":\[([^]]*).*/$1/s and
     /"label":"[^"]*(?:$ENV{_langre}).*?"src":"([^"]+)/si and 
    printf "%s\n", "$1" and
    printf MAIN::STDERR "%s\n", "$1";
'

Likely the language selection grep should be moved to the caller, rather than hidden in deobfuscate, i.e. deobfuscate in that case would return lines 2..n being the languages to egrep | head -1 against.

Maybe an ordered list of regexes, (say ':' separated like in $PATH) would be better, as that would permit the user to specify the languages with an order of precedence.

OT: how does language selection work anyway in ani-cli, say for a dub? I didn't see anything when skimming for any use of $lang/$_lang or http headers (and $_lang is the dub/sub selection instead)?

@Tsagar3

Tsagar3 commented Sep 6, 2026

Copy link
Copy Markdown

Good catch @jakobi — I can confirm the greedy match. I reproduced it with a JSON mirroring the real embed (3 languages, English marked "default":true):

# current sed
.*"subtitles":\[[^]]*"src":"..."
        -> always returns the LAST-listed language (es.vtt in my test)

The problem is [^]]* swallows the whole array and the trailing .* backtracks to the last "src":" before the closing ], so language order (and "default") is ignored. It just happened to work in my earlier test because One Piece ep 1 lists a single language.

The clean fix is to emit one line per subtitle entry and let the caller pick:

--- deobfuscate_blob
-    printf "%s\n" "$(printf "%b" "$_output" | sed -n 's|.*"subtitles":\[[^]]*"src":"\([^"]*\.vtt[^"]*\)".*|\1|p')"
+    printf "%s\n" "$(printf "%b" "$_output" | tr '}' '\n' | sed -n 's|.*"lang":"\([^"]*\)".*"label":"\([^"]*\)".*"src":"\([^"]*\.vtt[^"]*\)".*|\1\t\2\t\3|p')"
+
+# now emits: lang<TAB>label<TAB>src , one per language

--- hianime_m3u8 (deobfuscate step)
     _deobfuscated="$(printf "%s\n" "$(deobfuscate_blob "$_blob")" | sed '/^$/d')"
     _m3u8_master="$(printf "%s\n" "$_deobfuscated" | sed -n '1p')"
-    _subs="$(printf "%s\n" "$_deobfuscated" | sed -n '2p')"
+    _sub_entries="$(printf "%s\n" "$_deobfuscated" | sed -n '2,$p')"
+    _sub_langs="${ANI_CLI_SUB_LANG:-en}"
+    _subs="$(printf "%s\n" "$_sub_entries" | grep -iE "^$(printf '%s' "$_sub_langs" | tr ',:' '|')[[:space:]]" | head -n1 | cut -f 3)"
+    [ -z "$_subs" ] && _subs="$(printf "%s\n" "$_sub_entries" | head -n1 | cut -f 3)"

Selection logic (verified):

  • picks the first entry whose lang matches ANI_CLI_SUB_LANG (default en, comma/colon-separated list gives ordered precedence, i.e. @jakobi's PATH-style idea for free)
  • falls back to the first-listed entry when nothing matches (keeps the pre-fix default=true behaviour when English is listed first, like the real embed)
  • everything downstream (sub_link, local VTT download, --sub-file/VLC/Syncplay flags) is untouched since it just consumes _subs

Re grep -E/case-insensitivity: the script already uses grep -E for quality selection, so this is consistent. If you want label-based matching too, extending the alternation to label is trivial.

And to answer your OT: language selection in ani-cli today is only sub vs dub — --dub sets mode=dub, which picks the dub server/stream via data-type on the servers list. The subtitle language itself wasn't user-selectable before this; ANI_CLI_SUB_LANG adds exactly that knob. For dub streams the .vtt is the dialogue subtitle track (usually en), which the default covers.

GenshinPapi added a commit to GenshinPapi/AniAutoWatchList that referenced this pull request Sep 7, 2026
anidb.app has been behind a 503 maintenance page for several days, which made
every episode launch fail. Playback now falls back to hianime.at when anidb.app
cannot serve an episode.

anidb.app stays primary and its resolution path is unchanged. AllAnime still
supplies search, episode lists, and every tracking hook; hianime.at only resolves
stream links for an already selected title and episode.

The hianime scraping (endpoints, HD-1 embed, base64(json XOR "otaku-embed-v1")
player config) is derived from pystardust/ani-cli#1894 and #1897; see NOTICE.md.

Also in this change:

- Cloudflare detection no longer treats a bare challenges.cloudflare.com
  reference as an interstitial. Providers embed the Turnstile widget on ordinary
  pages, so hianime.at was rejected on every request. anidb.app's maintenance
  page is now reported as maintenance instead of a browser challenge.
- Subtitle track selection prefers the default and then a genuine English label,
  so it cannot land on a Spanish track that is tagged lang=en. hianime.at does
  not burn subtitles in, so the external WebVTT is handed to the player.
- hls_playlist_links labels a stream 1080p instead of Hls when RESOLUTION is last
  on the EXT-X-STREAM-INF line, so -q matches again for both providers.

ANI_CLI_ANIDB=0 skips the primary provider, ANI_CLI_HIANIME=0 disables the
fallback, and ANI_CLI_HIANIME_BASE retargets the fallback domain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183q3i6yuiLUGn7acwLRsMs
- Dynamically select ZokoAnime server, adapting to HiAnime's server rename
- Filter out Top 10 sidebar items from search results so invalid searches show no results
- Support subtitle language preference (ANI_CLI_SUB_LANG, default en) and avoid greedy regex
- Stream subtitles directly to players without temporary files
- Escape subtitle URLs for IINA and use :input-slave for VLC
- Forward skip_flag to syncplay and ensure mal_id is set before ani-skip runs
- Prevent history collisions between v5.0 and HiAnime by checking full slug
- Avoid global variable races in concurrent history processing
- Use fixed-string matching for episode numbers to handle decimals like 12.5
- Optimize string parsing with POSIX parameter expansion and remove hianime_desc
- Update documentation and man page references to hianime
@Dhairya3391 Dhairya3391 changed the title fix: replace anidb with hianime provider fix: temp-fix Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants