feat: add Session binding capability via session_id in Request

[Mantisus]

Description

• Add strict binding of a Request to a specific Session. If the Session is not available in the SessionPool, an error will be raised for the Request which can be handled in the failed_request_handler.

Issues

• Closes: Add support for request to use a specific session #1076

Testing

Added tests to verify functionality:

• Binding to a valid session
• Binding to a non-existent session
• Catching error in failed_request_handler

[Pijukatel]

Nice, I have just two small comments.

[Pijukatel]

I see you are following the same pattern as already exists, but I think the cast is not necessary if we redefine model like this:

    user_data: Annotated[
        UserData,
        Field(alias='userData', default_factory=UserData),
        PlainValidator(user_data_adapter.validate_python),
        PlainSerializer(
            lambda instance: user_data_adapter.dump_python(
                instance,
                by_alias=True,
                exclude_none=True,
                exclude_unset=True,
                exclude_defaults=True,
            )
        ),
    ] = UserData()

I tried tests and type check and it seems to work fine like this.
@janbuchar was there any specific reason not covered by unit tests to have it defined as dict[str, JsonSerializable] and not as UserData ?

[janbuchar]

The reason for that is explained in the comment - it's for user convenience. With the dict[str, JsonSerializable], you don't have to import UserData when constructing a Request - anything json-serializable will do.

I don't want to sacrifice that just to avoid some trivial casts.

[Pijukatel]

You do not have to import UserData even when you define your model with user_data: Annotated[UserData.... It still allows you to initialize with dict[str, JsonSerializable]. Mypy is fine with that as well.

So with our current project setup, to me this seems like unnecessary missleading type + several casts for no benefit.

[janbuchar]

Mypy being fine with that sounds like a bug to me, UserData is a "bigger" type than dict[str, JsonSerializable.

[Pijukatel]

So what is the real benefit of this in the end? (import argument not being applicable)

[Pijukatel]

We do not use this mypy settigns:

[tool.pydantic-mypy]
init_typed = true

therefore input to our models can be whatever from mypy perspective

[Pijukatel]

So we should either use this settings and then there is some justification for this. Or if we do not use the settings, then this has no benefit at all.

[janbuchar]

You get precise type checking for user_data when creating a new Request object and the required type is understandable without any further investigation (dict of json-serializable values).

I don't understand why the import argument shouldn't apply btw.

[Pijukatel]

I don't understand why the import argument shouldn't apply btw.

Because there is no extra import needed in either case. Try to run it if you don't believe me.

[janbuchar]

I tried 🙂

1. if you use Request.from_url(), it works OK because it has **kwargs: Any
2. if you construct it directly (Request(url='https://crawlee.dev', user_data={'hello': 'darkness'}, unique_key="", id="")), it's tricky
   a. mypy with init_typed is probably fine with it
   b. pyright (what vscode uses) is not fine with it
   c. a different mypy configuration that our end users might use may or may not be fine with it

[vdusek]

Could we please cover this in the docs? 🙏 Maybe Session management? Or find a better place. Thanks.