Autoclick Support

[ikreymer]

Adds support for autoclick behavior:

• Adds new autoclick behavior option to --behaviors, but not enabling by default
• Adds support for new exposed function __bx_addSet which allows autoclick behavior to persist state about links that have already been clicked to avoid duplicates, only used if link has an href
• Adds a new pageFinished flag on the worker state.
• Adds a on('dialog') handler to reject onbeforeunload page navigations, when in behavior (page not finished), but accept when page is finished - to allow navigation away only when behaviors are done
• Update to browsertrix-behaviors 0.7.0, which supports autoclick
• Add --clickSelector option to customize elements that will be clicked, defaulting to a.
• Add --linkSelector as alias for --selectLinks for consistency
• Unknown options for --behaviors printed as warnings, instead of hard exit, for forward compatibility for new behavior types in the future

Fixes #728, also #216, #665, #31

[ikreymer]

Some examples of sites where the autoclick behavior is helpful:

• https://www.bs.ch/themen -- all links involve a distinct ajax call, in page clicking otherwise not working. Test with --behaviors autoclick
• https://globalchange.gov/our-work/highlights - clicking on page links results in ajax call, pages nav not working w/o. Test with --behaviors autoclick
• https://www.wonderplugin.com/wordpress-lightbox/ - clicking on lightbox links results in custom behavior that is not crawled otherwise. Can be tested with --behaviors autoclick --clickSelector --clickSelector a.wplightbox

[tw4l]

Thanks for the sites to test out! I can confirm that it's working in the ways you describe for each, but I'm also noticing a whole lot of these messages in the logs for all of the autoclick-enabled crawls:

{"timestamp":"2024-12-04T21:08:45.377Z","logLevel":"warn","context":"behavior","message":"Behavior run partially failed","details":{"reason":{"type":"exception","message":"Protocol error (Runtime.evaluate): Execution context was destroyed.","stack":"ProtocolError: Protocol error (Runtime.evaluate): Execution context was destroyed.\n    at <instance_members_initializer> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:90:14)\n    at new Callback (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:94:16)\n    at CallbackRegistry.create (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:20:26)\n    at Connection._rawSend (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)\n    at CdpCDPSession.send (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)\n    at Browser.evaluateWithCLI (file:///app/dist/util/browser.js:208:56)\n    at file:///app/dist/crawler.js:747:89\n    at Array.map (<anonymous>)\n    at Crawler.runBehaviors (file:///app/dist/crawler.js:747:61)\n    at Crawler.doPostLoadActions (file:///app/dist/crawler.js:669:49)"},"page":"https://www.bs.ch/themen","workerid":0}}

I haven't caught exactly why yet, but worth looking into.

For selectors I like the idea of a --clickLinks arg!

[ikreymer]

https://www.bs.ch/themen

Thanks for the sites to test out! I can confirm that it's working in the ways you describe for each, but I'm also noticing a whole lot of these messages in the logs for all of the autoclick-enabled crawls:

{"timestamp":"2024-12-04T21:08:45.377Z","logLevel":"warn","context":"behavior","message":"Behavior run partially failed","details":{"reason":{"type":"exception","message":"Protocol error (Runtime.evaluate): Execution context was destroyed.","stack":"ProtocolError: Protocol error (Runtime.evaluate): Execution context was destroyed.\n    at <instance_members_initializer> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:90:14)\n    at new Callback (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:94:16)\n    at CallbackRegistry.create (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:20:26)\n    at Connection._rawSend (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:86:26)\n    at CdpCDPSession.send (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:63:33)\n    at Browser.evaluateWithCLI (file:///app/dist/util/browser.js:208:56)\n    at file:///app/dist/crawler.js:747:89\n    at Array.map (<anonymous>)\n    at Crawler.runBehaviors (file:///app/dist/crawler.js:747:61)\n    at Crawler.doPostLoadActions (file:///app/dist/crawler.js:669:49)"},"page":"https://www.bs.ch/themen","workerid":0}}

I haven't caught exactly why yet, but worth looking into.

Good catch! Turns out blocking the page unload was not actually working, and actually workeropts were being unnecessarily duplicated, see: 8a4e4db - should be fixed now!

[tw4l]

Thanks for that change! :) Not sure whether to actually approve this PR since we won't be merging this as-is with everything hardcoded and still need to add the selector flag, but based on testing I think this behavior is working as intended and good to add!

[ikreymer]

Ready for final review, not enabling autoclick by default for now.