Fixed retry of PowerBIDatasetRefreshOperator when dataset refresh wasn't directly available

[Ohashiro]

Fixes: #44618

Changes

This PR adds 2 changes:

• a retry mechanism on the API request to get the refresh status, to fix the bug when the API is not yet up-to-date with the triggered refreshId
• a new separation between the refresh trigger and the status polling

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[dabla]

@Ohashiro really nice work here well done!

[dabla]

@Ohashiro If you have time to apply changes from comments, then I think you could remove the draft en we merge this PR.

[dabla]

LGTM good job @Ohashiro thx a lot!

[Ohashiro]

Thanks for all your feedbacks and your review!
Before merging, I just wanted to test the behavior of the xcom_push for the key powerbi_dataset_refresh_status. I think that we push this message only when the refresh is successful but I think that we should also push when the refresh fails

[dabla]

Thanks for all your feedbacks and your review! Before merging, I just wanted to test the behavior of the xcom_push for the key powerbi_dataset_refresh_status. I think that we push this message only when the refresh is successful but I think that we should also push when the refresh fails

Then it's more safe to put it back to draft, and set it back to ready once you've done the testing.

[Ohashiro]

Hi @dabla I made a quick change to push the powerbi_dataset_refresh_status even when the refresh fails. Should I let you review this commit before setting the PR as "Ready for review"?

[dabla]

@Ohashiro do not forget to remove the draft option

[Ohashiro]

Sorry, where is the "draft option"? I thought it was "ready" as is

[dabla]

Sorry, where is the "draft option"? I thought it was "ready" as is

It should be somewhere at the bottom, I cannot see it unfortunately. It's indeed the "Ready for review" button at bottom, now your PR is still marked as "Draft".

[Ohashiro]

I don't know, there might be a bug somewhere, I clicked on the "Ready for review" button a few hours ago and the PR appears "Open" to me.

I rebased main to see if it refreshes something and solves the display issue

[dabla]

I don't know, there might be a bug somewhere, I clicked on the "Ready for review" button a few hours ago and the PR appears "Open" to me.

I rebased main to see if it refreshes something and solves the display issue

Now it's fine.

@potiuk or @eladkal I've just reviewed this PR, if approved by you guys could it be merged when you have time as this fixes an issue with the PowerBIDatasetRefreshOperator.

[dabla]

@Ohashiro Could you rename the title of the PR into something like:

"Fixed retry of PowerBIDatasetRefreshOperator when dataset refresh wasn't directly available"

[Ohashiro]

Hi @dabla ! Do you know if there's something missing on this PR?

[dabla]

Hi @dabla ! Do you know if there's something missing on this PR?

No I just think they are being busy working on Airflow 3. Could @potiuk or @eladkal look to merge this please, it’s a bugfix which I already reviewed.

[potiuk]

No I just think they are being busy working on Airflow 3

Yes. THAT

[Ohashiro]

OK I understand, thank you very much!

[dabla]

OK I understand, thank you very much!

Thank you @Ohashiro for fixing this issue.