Custom Client Constructors

[austinjp17]

Added two new YahooConnector constructors. One using a parameter passed client, giving the user complete discretion over the client build. The second a shortcut to create a proxy-tunneled client.

After a week or so testing I ran into persistent 429 responses, setting the user agent fixed this for me. I assume the proxy would as well but didn't try it.

[austinjp17]

It would be great as well if reqwest was updated to the latest version. I caught it right after submitting the pr.

[andber1]

I have the same issue as you described in #42. The solution with setting the user agent also works for me (maybe a short help with this info should be added to new_w_client?). Thanks!
However, your solution does not compile with feature "blocking" enabled:

error[E0308]: mismatched types
265 |             client,
    |             ^^^^^^ expected `reqwest::blocking::Client`, found `reqwest::Client`

Furthermore, should reqwest be re-exported so that one does not have to fiddle with the correct version?

[xemwebe]

The tests seem to suffer from the same problem, we would need another solution here too.

[wlk0]

You just need to specify a user agent to fix the 429 response issue:

    pub fn new() -> YahooConnector {        
        YahooConnector {
            client: Client::builder().user_agent( "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36").build().unwrap(),
            url: YCHART_URL,
            search_url: YSEARCH_URL,
        }
    }

[xemwebe]

Here, use something like .map_err(...)? instead. It is no good practice to let a library panic.

[xemwebe]

Use Client::builder() instead of reqwest::Client::builder(). Client is already imported and direct reference here uses only the non-blocking Client implementation.

[xemwebe]

Same as above

[xemwebe]

Please add a comment here.

[xemwebe]

Thank your for your pull request. I have added some comments

[xemwebe]

You just need to specify a user agent to fix the 429 response issue:

    pub fn new() -> YahooConnector {        
        YahooConnector {
            client: Client::builder().user_agent( "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36").build().unwrap(),
            url: YCHART_URL,
            search_url: YSEARCH_URL,
        }
    }

I have adopted your suggestion and it does seem to solve the problem

[nighostchris]

@xemwebe Will there be a new version published with this fix soon?

[xemwebe]

@xemwebe Will there be a new version published with this fix soon?

Just did so.

[xemwebe]

@austinjp17: The 429 response issue seems to solved by specifying a user agent. If you are still interested in this PR (e.g. for adding the additional constructors), please update your PR to solve the open issues, otherwise I would reject this PR.

[austinjp17]

Using the new constructor with the predefined agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" worked again for a few days then returned to 429s. I am far from a networking professional but imagine it's because every crate user is requesting from under the same agent. I have returned to using a unique app id string as the agent and haven't had any request rejections since. Let me clean it up and I'll update.

[xemwebe]

Using the new constructor with the predefined agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" worked again for a few days then returned to 429s. I am far from a networking professional but imagine it's because every crate user is requesting from under the same agent. I have returned to using a unique app id string as the agent and haven't had any request rejections since. Let me clean it up and I'll update.

I may be that yahoo takes some measures to identify and block users who try to download an excessive amount of data. Maybe some randomization is required, or using your own custom user agent (which is possible now).

[xemwebe]

Are you still interested in the PR or could this be closed?

[austinjp17]

A unique user agent cleaned it all up for me, can be closed. Appreciate your effort on this