Offline support for Network page #8332

hrajwade96 · 2024-09-21T09:19:46Z

Adding offline support to Network page.

issue link - #4470

List which issues are fixed by this PR.

Please add a note to packages/devtools_app/release_notes/NEXT_RELEASE_NOTES.md if your change requires release notes. Otherwise, add the 'release-notes-not-required' label to the PR.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read the Flutter Style Guide recently, and have followed its advice.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or there is a reason for not adding tests.

If you need help, consider asking for help on Discord.

This reverts commit 0c1f414.

kenzieschmoll · 2024-09-30T21:30:15Z

@hrajwade96 is this PR ready for review or is this still a work in progress?

hrajwade96 · 2024-10-01T17:13:00Z

@hrajwade96 is this PR ready for review or is this still a work in progress?

@kenzieschmoll just finishing up few things, I will mark it ready soon

…into network_screen_offline_support

packages/devtools_app/lib/src/screens/network/network_screen.dart

packages/devtools_app/lib/src/screens/network/network_controller.dart

packages/devtools_app/lib/src/screens/network/offline_network_data.dart

packages/devtools_app/lib/src/screens/network/network_screen.dart

packages/devtools_app/lib/src/screens/network/offline_network_data.dart

kenzieschmoll

Just a couple more comments. Thank you for all your work on this change!

kenzieschmoll · 2024-11-21T18:37:59Z

Looks like there are several failing checks. Please also add an entry to NEXT_RELEASE_NOTES.md so that the release notes check passes.

kenzieschmoll · 2024-11-21T19:41:49Z

You will likely have several merge conflicts after pulling in the tip of the master branch. A change just landed that upgraded the Flutter version used in DevTools, which updated to a new version of the dart formatter. Here's what I recommend to make this merge simpler for you:

Merge the master branch into your branch.
For all conflicts, accept your revision of the file..
Run dt update-flutter-sdk. This will update the flutter sdk contained at tool/flutter-sdk to the version used on the DevTools CI.
Run tool/flutter-sdk/bin/dart format from the devtools_app directory. This will format your files with your changes to use the new formatter.

…fline_support # Conflicts: # packages/devtools_app/lib/src/screens/network/har_data_entry.dart # packages/devtools_app/lib/src/screens/network/network_screen.dart # packages/devtools_app/lib/src/shared/http/http_request_data.dart

…fline_support # Conflicts: # packages/devtools_app/lib/src/screens/network/network_screen.dart # packages/devtools_app/test/screens/network/offline_data_test.dart # packages/devtools_app/test/screens/network/sample_network_offline_data.json

hrajwade96 · 2025-01-05T17:50:22Z

You will likely have several merge conflicts after pulling in the tip of the master branch. A change just landed that upgraded the Flutter version used in DevTools, which updated to a new version of the dart formatter. Here's what I recommend to make this merge simpler for you:

Merge the master branch into your branch.

For all conflicts, accept your revision of the file..

Run dt update-flutter-sdk. This will update the flutter sdk contained at tool/flutter-sdk to the version used on the DevTools CI.

Run tool/flutter-sdk/bin/dart format from the devtools_app directory. This will format your files with your changes to use the new formatter.

@kenzieschmoll can you run the checks again, I have done the above steps, formatted the code and updated the release notes as well.

hrajwade96 · 2025-01-06T18:04:02Z

You will likely have several merge conflicts after pulling in the tip of the master branch. A change just landed that upgraded the Flutter version used in DevTools, which updated to a new version of the dart formatter. Here's what I recommend to make this merge simpler for you:

Merge the master branch into your branch.

For all conflicts, accept your revision of the file..

Run dt update-flutter-sdk. This will update the flutter sdk contained at tool/flutter-sdk to the version used on the DevTools CI.

Run tool/flutter-sdk/bin/dart format from the devtools_app directory. This will format your files with your changes to use the new formatter.

@kenzieschmoll can you run the checks again, I have done the above steps, formatted the code and updated the release notes as well.

I pushed a minor change that I had missed pushing earlier which was the primary cause of the checks failing.
@kenzieschmoll can you re-run

kenzieschmoll · 2025-01-13T18:14:29Z

Looks like you still have some failing tests. Based on the errors it looks like you may need to set a global in the test setup for the failing tests: setGlobal(OfflineDataController, OfflineDataController());

You can verify the tests pass locally by running flutter test test/screens/network/

kenzieschmoll · 2025-01-14T20:43:23Z

Getting closer. Another failing test: devtools/packages/devtools_app/test/shared/framework/visible_screens_test.dart.

Consider running all tests locally flutter test test/ to ensure tests are passing.

hrajwade96 · 2025-01-15T08:01:26Z

Getting closer. Another failing test: devtools/packages/devtools_app/test/shared/framework/visible_screens_test.dart.

Consider running all tests locally flutter test test/ to ensure tests are passing.

Yes I've fixed this one, I ran locally now there's just one test (below one) failing locally but mostly it's timezone bound, hence although it's failing locally, it should pass here.

    test('TimestampColumn', () {
      final column = TimestampColumn();
      final getRequest = findRequestById('1');

      // The hours field may be unreliable since it depends on the timezone the
      // test is running in.
      expect(column.getDisplayValue(getRequest), contains(':45:26.279'));
    });

kenzieschmoll · 2025-01-15T17:03:05Z

A couple things look like they are still missing from this PR. I checked the code out locally and noticed that we do not have a way to export the offline network data. This is because the OpenSaveButtonGroup was not added to _NetworkProfilerControlsState. Please apply this patch to the _NetworkProfilerControlsState.build method:

    return Row(
      children: [
        if (!widget.offline) ...[
          StartStopRecordingButton(
            recording: _recording,
            onPressed:
                () async => await widget.controller.togglePolling(!_recording),
            tooltipOverride:
                _recording
                    ? 'Stop recording network traffic'
                    : 'Resume recording network traffic',
            minScreenWidthForTextBeforeScaling: double.infinity,
            gaScreen: gac.network,
            gaSelection: _recording ? gac.pause : gac.resume,
          ),
          const SizedBox(width: denseSpacing),
          ClearButton(
            minScreenWidthForTextBeforeScaling:
                _NetworkProfilerControls._includeTextWidth,
            gaScreen: gac.network,
            gaSelection: gac.clear,
            onPressed: widget.controller.clear,
          ),
          const SizedBox(width: denseSpacing),
          DownloadButton(
            tooltip: 'Download as .har file',
            minScreenWidthForTextBeforeScaling:
                _NetworkProfilerControls._includeTextWidth,
            onPressed: widget.controller.exportAsHarFile,
            gaScreen: gac.network,
            gaSelection: gac.NetworkEvent.downloadAsHar.name,
          ),
          const SizedBox(width: denseSpacing),
          // TODO(kenz): fix focus issue when state is refreshed
          Expanded(
            child: SearchField<NetworkController>(
              searchController: widget.controller,
              searchFieldEnabled: hasRequests,
              searchFieldWidth:
                  screenWidth <= MediaSize.xs
                      ? defaultSearchFieldWidth
                      : wideSearchFieldWidth,
            ),
          ),
          const SizedBox(width: denseSpacing),
          Expanded(
            child: StandaloneFilterField<NetworkRequest>(
              controller: widget.controller,
              filteredItem: 'request',
            ),
          ),
          const SizedBox(width: denseSpacing),
          OpenSaveButtonGroup(
            screenId: ScreenMetaData.performance.id,
            onSave: widget.controller.exportData,
          ),
        ],
      ],
    );

Once I added these buttons and then attempted to export network data, I am seeing an exception:

══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
The following HttpProfileRequestError was thrown while handling a gesture:
SocketException: Connection refused (OS Error: Connection refused, errno = 61), address = 127.0.0.1,
port = 53461.

When the exception was thrown, this was the stack:
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 307:3        throw_
errors.dart:307
packages/vm_service/src/dart_io_extensions.dart 565:7                              [_returnIfNoError]
dart_io_extensions.dart:565
packages/vm_service/src/dart_io_extensions.dart 542:40                             get headers
dart_io_extensions.dart:542
packages/devtools_app/src/shared/http/http_request_data.dart 387:41                HttpProfileRequestDataExtension.toJson
http_request_data.dart:387
packages/devtools_app/src/shared/http/http_request_data.dart 375:50                HttpProfileRequestExtension.toJson
http_request_data.dart:375
packages/devtools_app/src/shared/http/http_request_data.dart 81:44                 toJson
http_request_data.dart:81
packages/devtools_app/src/screens/network/offline_network_data.dart 87:39          <fn>
offline_network_data.dart:87
dart-sdk/lib/internal/iterable.dart 442:31                                         elementAt
iterable.dart:442
dart-sdk/lib/internal/iterable.dart 371:26                                         moveNext
iterable.dart:371
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 1127:20  next
operations.dart:1127
dart-sdk/lib/_internal/js_dev_runtime/patch/core_patch.dart 358:14                 of
core_patch.dart:358
dart-sdk/lib/internal/iterable.dart 224:7                                          toList
iterable.dart:224
packages/devtools_app/src/screens/network/offline_network_data.dart 87:39          toJson
offline_network_data.dart:87
packages/devtools_app/src/screens/network/network_controller.dart 470:24           prepareOfflineScreenData
network_controller.dart:470
packages/devtools_app/src/shared/offline/offline_data.dart 188:34                  exportData
offline_data.dart:188
packages/devtools_app/src/shared/ui/file_import.dart 63:21                         <fn>
file_import.dart:63
packages/flutter/src/material/ink_well.dart 1185:21                                handleTap

hrajwade96 · 2025-01-15T17:40:24Z

A couple things look like they are still missing from this PR. I checked the code out locally and noticed that we do not have a way to export the offline network data. This is because the OpenSaveButtonGroup was not added to _NetworkProfilerControlsState. Please apply this patch to the _NetworkProfilerControlsState.build method:

    return Row(
      children: [
        if (!widget.offline) ...[
          StartStopRecordingButton(
            recording: _recording,
            onPressed:
                () async => await widget.controller.togglePolling(!_recording),
            tooltipOverride:
                _recording
                    ? 'Stop recording network traffic'
                    : 'Resume recording network traffic',
            minScreenWidthForTextBeforeScaling: double.infinity,
            gaScreen: gac.network,
            gaSelection: _recording ? gac.pause : gac.resume,
          ),
          const SizedBox(width: denseSpacing),
          ClearButton(
            minScreenWidthForTextBeforeScaling:
                _NetworkProfilerControls._includeTextWidth,
            gaScreen: gac.network,
            gaSelection: gac.clear,
            onPressed: widget.controller.clear,
          ),
          const SizedBox(width: denseSpacing),
          DownloadButton(
            tooltip: 'Download as .har file',
            minScreenWidthForTextBeforeScaling:
                _NetworkProfilerControls._includeTextWidth,
            onPressed: widget.controller.exportAsHarFile,
            gaScreen: gac.network,
            gaSelection: gac.NetworkEvent.downloadAsHar.name,
          ),
          const SizedBox(width: denseSpacing),
          // TODO(kenz): fix focus issue when state is refreshed
          Expanded(
            child: SearchField<NetworkController>(
              searchController: widget.controller,
              searchFieldEnabled: hasRequests,
              searchFieldWidth:
                  screenWidth <= MediaSize.xs
                      ? defaultSearchFieldWidth
                      : wideSearchFieldWidth,
            ),
          ),
          const SizedBox(width: denseSpacing),
          Expanded(
            child: StandaloneFilterField<NetworkRequest>(
              controller: widget.controller,
              filteredItem: 'request',
            ),
          ),
          const SizedBox(width: denseSpacing),
          OpenSaveButtonGroup(
            screenId: ScreenMetaData.performance.id,
            onSave: widget.controller.exportData,
          ),
        ],
      ],
    );

Once I added these buttons and then attempted to export network data, I am seeing an exception:

══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
The following HttpProfileRequestError was thrown while handling a gesture:
SocketException: Connection refused (OS Error: Connection refused, errno = 61), address = 127.0.0.1,
port = 53461.

When the exception was thrown, this was the stack:
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 307:3        throw_
errors.dart:307
packages/vm_service/src/dart_io_extensions.dart 565:7                              [_returnIfNoError]
dart_io_extensions.dart:565
packages/vm_service/src/dart_io_extensions.dart 542:40                             get headers
dart_io_extensions.dart:542
packages/devtools_app/src/shared/http/http_request_data.dart 387:41                HttpProfileRequestDataExtension.toJson
http_request_data.dart:387
packages/devtools_app/src/shared/http/http_request_data.dart 375:50                HttpProfileRequestExtension.toJson
http_request_data.dart:375
packages/devtools_app/src/shared/http/http_request_data.dart 81:44                 toJson
http_request_data.dart:81
packages/devtools_app/src/screens/network/offline_network_data.dart 87:39          <fn>
offline_network_data.dart:87
dart-sdk/lib/internal/iterable.dart 442:31                                         elementAt
iterable.dart:442
dart-sdk/lib/internal/iterable.dart 371:26                                         moveNext
iterable.dart:371
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 1127:20  next
operations.dart:1127
dart-sdk/lib/_internal/js_dev_runtime/patch/core_patch.dart 358:14                 of
core_patch.dart:358
dart-sdk/lib/internal/iterable.dart 224:7                                          toList
iterable.dart:224
packages/devtools_app/src/screens/network/offline_network_data.dart 87:39          toJson
offline_network_data.dart:87
packages/devtools_app/src/screens/network/network_controller.dart 470:24           prepareOfflineScreenData
network_controller.dart:470
packages/devtools_app/src/shared/offline/offline_data.dart 188:34                  exportData
offline_data.dart:188
packages/devtools_app/src/shared/ui/file_import.dart 63:21                         <fn>
file_import.dart:63
packages/flutter/src/material/ink_well.dart 1185:21                                handleTap

@kenzieschmoll, I wanted to clarify my understanding based on your comment:

This PR focuses solely on viewing Network data in offline mode. Importing offline data back into DevTools is planned in the upcoming PR, as we discussed earlier (If you suggest to add it as part of this PR itself we can add it here).
Exporting HAR files already has a dedicated Download CTA, so I assumed adding 'OpenSaveButtonGroup' wasn't necessary at this stage.
Additionally, as previously discussed, export functionality is not supported while in offline mode.
Let me know if there are any changes to this or we can go ahead with this

kenzieschmoll · 2025-01-15T19:15:50Z

This PR focuses solely on viewing Network data in offline mode

By this do you mean viewing data when the Network screen is showing, and then DevTools loses connection to the running app? In other words, the "review history" feature?

Importing offline data back into DevTools is planned in the upcoming PR

Okay this sounds good. Can we turn the feature flag to off then until this support is added?

How much work is left to do to support exporting the data as JSON? I would expect there is a small bit of serialization work left to ensure the data can be serialized and deserialized to JSON, but the majority of the logic is contained in this PR. Is that accurate?

(edited) UPDATE:
I have a follow on I'd like to submit after this PR. This will essentially pave the way for you to implement the full offline support for the network screen (which is exporting the data as json and being able to re-import it as json). This follow on PR I have put to gather will combine the download buttons to let the user choose which format they'd like to download data in.

This would be behind a new experiment flag networkSaveLoad so that the network disconnect experience can still be enabled while the full offline support is under development.

kenzieschmoll · 2025-01-15T20:23:27Z

I dug a little deeper on the errors I am encountering. These exceptions occur even for the "review history" behavior. Have you tested with requests that have an error? Some of the fields on HttpProfileRequestData (from the vm_service package) will throw an exception if the request has an error (hasError). We need to make sure that the serialization logic gracefully handles all of these cases.

We should also have test coverage for this case.

hrajwade96 · 2025-01-19T12:51:17Z

I dug a little deeper on the errors I am encountering. These exceptions occur even for the "review history" behavior. Have you tested with requests that have an error? Some of the fields on HttpProfileRequestData (from the vm_service package) will throw an exception if the request has an error (hasError). We need to make sure that the serialization logic gracefully handles all of these cases.

We should also have test coverage for this case.

Ok will check this case and also add test coverage for this

hrajwade96 · 2025-01-19T12:58:34Z

By this do you mean viewing data when the Network screen is showing, and then DevTools loses connection to the running app? In other words, the "review history" feature?

Yes I meant the "review history" feature.

Okay this sounds good. Can we turn the feature flag to off then until this support is added?

Sure, so as per your next comment I think you are suggesting to enable the review history (with this PR) and add another flag which gives two options to choose the format for export, and also an import button (in the next PR).

(edited) UPDATE: I have a follow on I'd like to submit after this PR. This will essentially pave the way for you to implement the full offline support for the network screen (which is exporting the data as json and being able to re-import it as json). This follow on PR I have put to gather will combine the download buttons to let the user choose which format they'd like to download data in.

This would be behind a new experiment flag networkSaveLoad so that the network disconnect experience can still be enabled while the full offline support is under development.

Sure, I'll pick this up in the next PR. Could you elaborate on the key idea behind adding the option to export it as a '.json' file?

added initial implementation

fe11d5b

hrajwade96 requested review from bkonyi, kenzieschmoll and a team as code owners September 21, 2024 09:19

hrajwade96 marked this pull request as draft September 21, 2024 09:19

hrajwade96 changed the title ~~added initial implementation~~ Offline support for Network page Sep 21, 2024

hrajwade96 added 3 commits September 29, 2024 18:20

adding constants file

6c140b4

updated NEXT_RELEASE_NOTES.md

0c1f414

Revert "updated NEXT_RELEASE_NOTES.md"

e7eeecb

This reverts commit 0c1f414.

hrajwade96 and others added 2 commits October 2, 2024 22:46

used constants, added util for header size calc, added params

c5ec6f6

Merge branch 'master' into network_screen_offline_support

131cde6

hrajwade96 marked this pull request as ready for review October 2, 2024 17:19

hrajwade96 added 3 commits October 2, 2024 22:56

param renamed, used _log

2a9c4cb

Merge remote-tracking branch 'origin/network_screen_offline_support' …

4f473be

…into network_screen_offline_support

added copyright header

ae58738

kenzieschmoll reviewed Oct 2, 2024

View reviewed changes

packages/devtools_app/lib/src/screens/network/network_screen.dart Outdated Show resolved Hide resolved

kenzieschmoll reviewed Oct 2, 2024

View reviewed changes

packages/devtools_app/lib/src/screens/network/network_controller.dart Outdated Show resolved Hide resolved

hrajwade96 added 5 commits October 3, 2024 18:35

setup changes implemented

2bfbe1c

put class name in square brackets

857cfbe

added isNullOrEmpty check

7150436

added exit offline cta, hiding other controls, removed json string

da11a67

fix for request selection

941ec88

kenzieschmoll reviewed Oct 7, 2024

View reviewed changes

hrajwade96 added 5 commits October 8, 2024 09:09

comments resolved

852b144

removing getFullRequestData section

433af38

using updateOrAddAll for setting data

4644b00

Wrapped with OfflineAwareControls, removed if-else.

6fd98d0

removed addAutoDisposeListener as we are using OfflineAwareControls

665ea0d

kenzieschmoll reviewed Nov 21, 2024

View reviewed changes

removed repetitive expect statements

e7b5c90

kenzieschmoll approved these changes Nov 21, 2024

View reviewed changes

hrajwade96 added 6 commits November 22, 2024 09:17

minor fix

4f31b92

added timelineMicrosOffset to offline data

04c8b41

added timelineMicrosOffset to tests

ab8c519

release notes updated

2a9cacf

hrajwade96 force-pushed the network_screen_offline_support branch from aebe0d8 to 2a9cacf Compare December 27, 2024 07:03

code reformatted

4298078

hrajwade96 added 2 commits January 6, 2025 23:20

lint fixes - fixed import, removed unused import

a538a74

comment added for ignore

e05cd02

lint fix: removed redundant async

ae52105

setting globals for test cases

9f6fd00

kenzieschmoll mentioned this pull request Jan 14, 2025

License update - Batch 21 #8719

Draft

7 tasks

fixed visible screens test

560d5a6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offline support for Network page #8332

Offline support for Network page #8332

hrajwade96 commented Sep 21, 2024 •

edited

Loading

kenzieschmoll commented Sep 30, 2024

hrajwade96 commented Oct 1, 2024

kenzieschmoll left a comment

kenzieschmoll commented Nov 21, 2024

kenzieschmoll commented Nov 21, 2024

hrajwade96 commented Jan 5, 2025

hrajwade96 commented Jan 6, 2025 •

edited

Loading

kenzieschmoll commented Jan 13, 2025 •

edited

Loading

kenzieschmoll commented Jan 14, 2025 •

edited

Loading

hrajwade96 commented Jan 15, 2025 •

edited

Loading

kenzieschmoll commented Jan 15, 2025

hrajwade96 commented Jan 15, 2025 •

edited

Loading

kenzieschmoll commented Jan 15, 2025 •

edited

Loading

kenzieschmoll commented Jan 15, 2025

hrajwade96 commented Jan 19, 2025

hrajwade96 commented Jan 19, 2025

Offline support for Network page #8332

Are you sure you want to change the base?

Offline support for Network page #8332

Conversation

hrajwade96 commented Sep 21, 2024 • edited Loading

Pre-launch Checklist

kenzieschmoll commented Sep 30, 2024

hrajwade96 commented Oct 1, 2024

kenzieschmoll left a comment

Choose a reason for hiding this comment

kenzieschmoll commented Nov 21, 2024

kenzieschmoll commented Nov 21, 2024

hrajwade96 commented Jan 5, 2025

hrajwade96 commented Jan 6, 2025 • edited Loading

kenzieschmoll commented Jan 13, 2025 • edited Loading

kenzieschmoll commented Jan 14, 2025 • edited Loading

hrajwade96 commented Jan 15, 2025 • edited Loading

kenzieschmoll commented Jan 15, 2025

hrajwade96 commented Jan 15, 2025 • edited Loading

kenzieschmoll commented Jan 15, 2025 • edited Loading

kenzieschmoll commented Jan 15, 2025

hrajwade96 commented Jan 19, 2025

hrajwade96 commented Jan 19, 2025

hrajwade96 commented Sep 21, 2024 •

edited

Loading

hrajwade96 commented Jan 6, 2025 •

edited

Loading

kenzieschmoll commented Jan 13, 2025 •

edited

Loading

kenzieschmoll commented Jan 14, 2025 •

edited

Loading

hrajwade96 commented Jan 15, 2025 •

edited

Loading

hrajwade96 commented Jan 15, 2025 •

edited

Loading

kenzieschmoll commented Jan 15, 2025 •

edited

Loading