Contributor

Member

Thanks for the PR. Can you please give some info about the motivation for this.

Contributor Author

We are using HtmlUnit massively parallel and using a fresh WebClient for each request to avoid side effects. Each WebClient creates a JavaScript event loop thread, even if JavaScript is disabled. So I implemented a BackgroundJavaScriptFactory which returns a JavaScriptExecutor which uses a thread pool.

To not be forced to use a fixed BackgroundJavaScriptFactory for all threads, I added BackgroundJavaScriptFactory manager which uses a ThreadLocal to provide the wanted BackgroundJavaScriptFactory for each thread. This is currently not possible, because BackgroundJavaScriptFactory uses JavaScriptFunctionJob which is just package visible. Instead of just fixing this by using JavaScriptJob as return value of createJavaScriptJob(), I did the bigger more clean solution to use an interface. This removes the need to derive from BackgroundJavaScriptFactory for alternate implementations.

Member

Hi Markus,
working currently on disabling the js support more general

Each WebClient creates a JavaScript event loop thread, even if JavaScript is disabled.

With the latest snapshot build available i have introduced a new constructor for the WebClient new WebClient(getBrowserVersion(), false, null, -1). The false here means to start no js engine at all (and even do not create any js peer objects). This should save a lot of memory in your scenario. But there is also a drawback; most of the code is not written for this scenario - you might face exceptions at various places.

But it will be a great help if you can try the latest snapshot and report the exceptions you got here. Will fix this step by step....

Thanks

Member

@markusheiden do you still need this interface or did my suggestion the trick?

Contributor Author

Short answer: No, we don't need it anymore, because we no longer use HtmlUnit.

Long answer:

Sorry, I overlooked your change. I did implement this before via a custom JavaScriptEngine which does not start Javascript at all (almost all methods implemented empty, it just creates and initializes a Window). We never experienced problems with that solution.

The interface was intended for an alternative JavascriptBackgroundFactory which uses a thread pool for Javascript execution to speed up the start time of the Javascript engine. We never tested that in production though, because we did not need Javascript then.

Anyway we used HtmlUnit as a crawler and since we need to crawl single page app based portals, the Javascript level of HtmlUnit is no longer sufficient. We migrated to a Puppeteer (Chrome) based solution.

Member

Thanks for the feedback.

Hope you had at least some fun with HtmlUnit and now have success with puppeteer.