feat: rewrite scrapeThread to use TweetDetail GraphQL API by nj-io · Pull Request #17 · nirholas/XActions

nj-io · 2026-04-05T09:02:35Z

Summary

Replaces the DOM-based scrapeThread with direct TweetDetail GraphQL API calls. X doesn't render self-reply threads as article[data-testid="tweet"] elements in the DOM, causing the current scraper to return empty results — especially for tweets with many replies.

Why the DOM approach fails

X lazy-loads replies and often doesn't render the author's own thread continuation as DOM elements
For viral tweets (2000+ replies), self-replies get pushed out of the initial page entirely
screen_name moved from user.legacy to user.core in X's GraphQL schema, breaking author detection

New approach

Navigates to the tweet page (needed for session cookies)
Calls TweetDetail GraphQL API directly from the page context via fetch()
Gets full_text from the API (no truncation, no "Show more" clicking needed)
note_tweet support for long-form posts (>280 chars)
Filters conversation entries to self-reply chain only (author replying to themselves)
Sorts chronologically

Shared helpers

Introduces helper functions designed for reuse by a future scrapePost tool:

Helper	Purpose
`fetchTweetDetail(page, tweetId)`	Calls TweetDetail GraphQL API from page context
`parseTweetResult(result)`	Extracts rich data: text, media (images + best video URL), X Articles (title + cover + URL), cards (link previews), external URLs, engagement stats
`parseThreadFromEntries(entries, author, tweetId)`	Detects self-reply thread chain
`extractEntries(data)`	Extracts timeline entries from GraphQL response
`unwrapResult(result)`	Handles `TweetWithVisibilityResults` wrapper
`getScreenName(result)`	Gets screen_name from both `user.core` and `user.legacy` paths

Backward compatibility

scrapeThread return shape is unchanged — internal fields (media, article, card, urls, quotedTweetId, inReplyTo) are stripped before returning, so existing consumers see the same { id, text, author, timestamp, url, isMainAuthor, platform } shape.

Relation to other PRs

Supersedes Fix Twitter thread scraping to stop returning empty results #12 which patches the DOM approach — this replaces DOM scraping entirely
Foundation for a future scrapePost/x_read_post PR that will use the shared helpers to read any post with full rich data and recursive quote tweet resolution

Test plan

2-tweet thread (Aakash Gupta) — returns both tweets
15-tweet thread (Breedlove) — returns full thread with media
Single tweet (Karpathy "LLM Knowledge Bases") — returns 1 tweet correctly
Backward compatible return shape (no new fields exposed)

🤖 Generated with Claude Code

Replace DOM-based thread scraping with direct GraphQL API calls. X doesn't render self-reply threads as article elements in the DOM, causing empty results — especially for high-engagement tweets. The new approach: - Calls TweetDetail GraphQL API from the page context using session cookies - Gets full_text (no truncation, no "Show more" needed) - note_tweet support for long-form posts - Filters to self-reply chain only (author replying to themselves) - Chronological sorting Also introduces shared helpers for future use by scrapePost: - fetchTweetDetail() — GraphQL API caller - parseTweetResult() — rich data extraction (text, media, article, card, external URLs, engagement stats) - parseThreadFromEntries() — thread chain detection - extractEntries(), unwrapResult(), getScreenName() Fixes: - screen_name moved from user.legacy to user.core in X's GraphQL schema - Self-replies missing from API response for viral tweets (2000+ replies) now handled gracefully (returns available tweets) Supersedes nirholas#12 which patches the DOM approach — this replaces it entirely. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vercel · 2026-04-05T09:02:40Z

@nj-io is attempting to deploy a commit to the kaivocmenirehtacgmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

New scrapePost() function and x_read_post MCP tool that reads any tweet URL with full rich data and recursive quote tweet resolution. Features: - Single tweets or threads (auto-detected via self-reply chain) - Rich data per tweet: text, media (images + best-quality video URL), X Articles (title + cover image + URL), cards (link previews), external URLs (Substack, GitHub, etc.), engagement stats - Recursive quote tweet resolution — if a quoted tweet is itself a thread, or contains its own quote tweet, those are fetched too (up to 5 levels deep) - Uses shared helpers from scrapeThread rewrite (fetchTweetDetail, parseTweetResult, parseThreadFromEntries) Return shape: { thread: [{ id, author, text, timestamp, url, media, article, card, urls, replies, retweets, likes, views, quotedPost? }] } Where quotedPost has the same { thread: [...] } shape, recursively. Depends on: nirholas#17 (scrapeThread GraphQL rewrite with shared helpers) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Replace uniform randomDelay (1-3s) with log-normal distribution (2-7s base + 8% distraction spikes of 8-20s) - Add checkAuth() guard after page navigation — fails fast on expired cookies - Add randomDelay before each fetchTweetDetail API call to simulate human browsing between tweet reads

New scrapePost() function and x_read_post MCP tool that reads any tweet URL with full rich data and recursive quote tweet resolution. Features: - Single tweets or threads (auto-detected via self-reply chain) - Rich data per tweet: text, media (images + best-quality video URL), X Articles (title + cover image + URL), cards (link previews), external URLs (Substack, GitHub, etc.), engagement stats - Recursive quote tweet resolution — if a quoted tweet is itself a thread, or contains its own quote tweet, those are fetched too (up to 5 levels deep) - Human-like delays between API calls (inherited from fetchTweetDetail) - Auth check on navigation (inherited from shared helpers) Depends on: nirholas#17 (scrapeThread GraphQL rewrite with shared helpers) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… filtering Replace the broken DOM-based x_get_likes with a proper scraper using the Likes GraphQL API (cursor-based pagination). - 50 tweets in ~14s, 200 in ~49s (was capped at ~25 with DOM scraping) - Rich data via parseTweetResult (text, media, articles, cards, URLs, engagement) - JSONL output to ~/.xactions/exports/ — progress survives crashes - from/to timestamp filters with early exit on reverse chronological data - Removes x_get_likes from xeepyTools, routes through local-tools.js Depends on: nirholas#17 (shared helpers: parseTweetResult, checkAuth, randomDelay) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…likes Two new scrapers and MCP tools: scrapeLikedTweets — GraphQL-based likes scraper: - Cursor pagination via Likes API (50 tweets in 14s, 200 in 49s) - JSONL output to ~/.xactions/exports/ - from/to timestamp filtering with early exit - Rich data via parseTweetResult discoverLikes — interleaved fetch + deep read: - Fetches likes via API, deep-reads each via scrapePost - Human-like pacing: 3-8s between pages, 2-5s before reads, 5-15s after - Produces two JSONL files: likes index + deep reads - ~38s per tweet average (5 likes = 190s) Both remove x_get_likes from xeepyTools and delete the old DOM handler. Depends on: nirholas#17 (shared helpers), nirholas#18 (scrapePost) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

nj-io · 2026-04-07T05:56:16Z

Superseded — resubmitting as clean PRs from current codebase.

nj-io requested a review from nirholas as a code owner April 5, 2026 09:02

nj-io mentioned this pull request Apr 5, 2026

feat: add x_read_post tool for full rich tweet reading with recursive quote tweets #18

Closed

7 tasks

nj-io mentioned this pull request Apr 6, 2026

feat: GraphQL-based scrapeLikedTweets with JSONL output #20

Closed

6 tasks

nj-io mentioned this pull request Apr 6, 2026

feat: add x_get_likes (GraphQL) and x_discover_likes with human-like deep reads #21

Closed

5 tasks

nj-io closed this Apr 7, 2026

nj-io mentioned this pull request Apr 7, 2026

feat: GraphQL-based scrapeThread, scrapePost, x_read_post, and shared infrastructure #23

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: rewrite scrapeThread to use TweetDetail GraphQL API#17

feat: rewrite scrapeThread to use TweetDetail GraphQL API#17
nj-io wants to merge 2 commits intonirholas:mainfrom
nj-io:feat/thread-graphql-rewrite

nj-io commented Apr 5, 2026

Uh oh!

vercel bot commented Apr 5, 2026

Uh oh!

nj-io commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

nj-io commented Apr 5, 2026

Summary

Why the DOM approach fails

New approach

Shared helpers

Backward compatibility

Relation to other PRs

Test plan

Uh oh!

vercel bot commented Apr 5, 2026

Uh oh!

nj-io commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant