Make header-generator compatible with Cloudflare workers (edge runtimes)

[labithiotis]

Currently its not possible to use header-generator HeaderGenerator class inside Cloudflare workers as we get a the following error:

Error generating random headers with lib: ReferenceError: __dirname is not defined 

I would be nice to use this library inside Cloudflare workers (or other runtimes that don't have all of NodeJs's FS APIs).

Looking at the code there are two places which are causing this issue;

1. Loading JSON files which could be resolved by adding resolveJsonModule to tsconfig.json.
   https://github.com/apify/fingerprint-suite/blob/master/packages/header-generator/src/header-generator.ts#L215
2. Which is a tricky problem, is with passing in a string path to BayesianNetwork that uses another library called adm-zip, maybe this code can be made optional so it can work in Cloudflare workers??
   https://github.com/apify/fingerprint-suite/blob/master/packages/header-generator/src/header-generator.ts#L236

[barjin]

Hello and thank you for your interest in this project!

As you've correctly pointed out, the broken __dirname resolution is only the first of many problems.

The HeaderGenerator class loads the probabilistic models from .zip files bundled in the package (in the data-files folder). These are then internally unzipped using adm-zip and used to generate the random header sets.

Since Cloudflare Workers don't provide the fs API, there is no way of loading the probabilistic model for us. This feature cannot be made optional, as it stands in the center of what this package is doing.

I see some ways through (e.g. making the JSON and zip files into Javascript code exporting the contents as constants), but we currently don't have capacity for such changes (and it might be a bit more complicated than this, especially with the automatic model updates). Thinking about it, it might also introduce performance issues with parsing such large files (maybe not). All in all, this is sadly not on our immediate roadmap.

Community PRs welcome, though!

[labithiotis]

@barjin I did look at making a PR, but quickly ran into an issue where the library loads zip files as you mentioned. In the end, I just made a simple GCP Cloud Functions that returns headers for CF worker.

I should have closed the issue when I realised it was a trivial change. I will close the issue now.

[barjin]

Thank you for your detailed insight. I'll reopen this issue, as this question actually keeps popping up quite often (see e.g. here or various attempts at running Crawlee in different serverless environments. If we want to make our dependents bundleable and runnable in such environments, we'd likely need to solve this first.