From 61a0f1e105b725d3abb6b40af2455ede15a2d708 Mon Sep 17 00:00:00 2001 From: Malcolm Slaney Date: Sun, 7 Nov 2021 17:24:16 -0800 Subject: [PATCH 001/872] Be more clear about train_step and test_step Be more clear that the train_step and test_step are also moved to the accelerator. This was not clear before. This makes it explicit, which should help people understanding TPU efficiency. --- site/en/guide/distributed_training.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb index b115d68afd0..34b5441fafb 100644 --- a/site/en/guide/distributed_training.ipynb +++ b/site/en/guide/distributed_training.ipynb @@ -508,7 +508,7 @@ "Here's what you need to change in your code:\n", "\n", "1. Create an instance of the appropriate `tf.distribute.Strategy`.\n", - "2. Move the creation of Keras model, optimizer and metrics inside `strategy.scope`.\n", + "2. Move the creation of Keras model, optimizer and metrics inside `strategy.scope`. Thus the code in the model's run(), train_step(), and test_step() will all be distributed and executed on the accelerator(s).\n", "\n", "TensorFlow distribution strategies support all types of Keras models—[Sequential](/keras/sequential_model.ipynb), [Functional](/keras/functional.ipynb), and [subclassed](/keras/custom_layers_and_models.ipynb).\n", "\n", From 061a8b5b7081bffbdea890d2460bdda7e3489700 Mon Sep 17 00:00:00 2001 From: sushreebarsa <84765720+sushreebarsa@users.noreply.github.com> Date: Sun, 12 Dec 2021 22:54:46 +0530 Subject: [PATCH 002/872] Update autoencoder.ipynb Changed the code from encoded_imgs = autoencoder.encoder(x_test).numpy() to encoded_imgs = autoencoder.encoder(x_test_noisy).numpy() Fixes #53357 --- site/en/tutorials/generative/autoencoder.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/generative/autoencoder.ipynb b/site/en/tutorials/generative/autoencoder.ipynb index d2af1c3a345..6d69f9532c3 100644 --- a/site/en/tutorials/generative/autoencoder.ipynb +++ b/site/en/tutorials/generative/autoencoder.ipynb @@ -492,7 +492,7 @@ }, "outputs": [], "source": [ - "encoded_imgs = autoencoder.encoder(x_test).numpy()\n", + "encoded_imgs = autoencoder.encoder(x_test_noisy).numpy()\n", "decoded_imgs = autoencoder.decoder(encoded_imgs).numpy()" ] }, From 999302f4b223ab13750301a9502d338f38fa46b4 Mon Sep 17 00:00:00 2001 From: Malcolm Slaney Date: Fri, 31 Dec 2021 12:45:43 -0800 Subject: [PATCH 003/872] Update distributed_training.ipynb Made the references to the run, train_step and test_step methods more clear. [Do I need to be more explicit that the Keras model is a (sub) class and it does the work with the run, train_step and test_step methods, all of which might be subclassed for a specific model implementation?] --- site/en/guide/distributed_training.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb index 34b5441fafb..129e3f155cf 100644 --- a/site/en/guide/distributed_training.ipynb +++ b/site/en/guide/distributed_training.ipynb @@ -508,7 +508,7 @@ "Here's what you need to change in your code:\n", "\n", "1. Create an instance of the appropriate `tf.distribute.Strategy`.\n", - "2. Move the creation of Keras model, optimizer and metrics inside `strategy.scope`. Thus the code in the model's run(), train_step(), and test_step() will all be distributed and executed on the accelerator(s).\n", + "2. Move the creation of Keras model, optimizer and metrics inside `strategy.scope`. Thus the code in the model's `run()`, `train_step()`, and `test_step()` methods will all be distributed and executed on the accelerator(s).\n", "\n", "TensorFlow distribution strategies support all types of Keras models—[Sequential](/keras/sequential_model.ipynb), [Functional](/keras/functional.ipynb), and [subclassed](/keras/custom_layers_and_models.ipynb).\n", "\n", From 5b8909ec134de310b116b6ac00ebfca93544de42 Mon Sep 17 00:00:00 2001 From: tilakrayal <81610181+tilakrayal@users.noreply.github.com> Date: Fri, 11 Feb 2022 15:24:05 +0530 Subject: [PATCH 004/872] Update csv.ipynb --- site/en/tutorials/load_data/csv.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/load_data/csv.ipynb b/site/en/tutorials/load_data/csv.ipynb index c0691070730..dc13711fbec 100644 --- a/site/en/tutorials/load_data/csv.ipynb +++ b/site/en/tutorials/load_data/csv.ipynb @@ -230,7 +230,7 @@ " layers.Dense(1)\n", "])\n", "\n", - "abalone_model.compile(loss = tf.losses.MeanSquaredError(),\n", + "abalone_model.compile(loss = tf.keras.losses.MeanSquaredError(),\n", " optimizer = tf.optimizers.Adam())" ] }, From dbc999f857d459edc60d7e7aeb19154efc415d42 Mon Sep 17 00:00:00 2001 From: gdarkwah <61522557+gdarkwah@users.noreply.github.com> Date: Wed, 2 Mar 2022 13:15:56 -0600 Subject: [PATCH 005/872] update to checkpoint callback options (save_frequency) introduced the number of batches ('n_batches') option for the save frequency instead of 'batch_size'. Using 'batch_size' works in this tutorial because the length of the training data is 1000 which coincidentally results in a rounded value of ~32 when it is divided by the 'batch_size'. In cases when the number of samples is not 1000, this will result in the model saving at different epoch frequencies other than after every 5 epochs. the definition of 'save_freq' (https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint#args) clearly refers to the number of batches ('n_batches' in this context) and not the number of samples in a batch ('batch_size'). --- site/en/tutorials/keras/save_and_load.ipynb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/site/en/tutorials/keras/save_and_load.ipynb b/site/en/tutorials/keras/save_and_load.ipynb index 6e48a08a4a6..a75e01116ba 100644 --- a/site/en/tutorials/keras/save_and_load.ipynb +++ b/site/en/tutorials/keras/save_and_load.ipynb @@ -385,12 +385,17 @@ "\n", "batch_size = 32\n", "\n", + "# calculate the number of batches per epoch\n", + "import math\n", + "n_batches = len(train_images) / batch_size\n", + "n_batches = math.ceil(n_batches) # round up the number of batches to the nearest whole integer\n", + "\n", "# Create a callback that saves the model's weights every 5 epochs\n", "cp_callback = tf.keras.callbacks.ModelCheckpoint(\n", " filepath=checkpoint_path, \n", " verbose=1, \n", " save_weights_only=True,\n", - " save_freq=5*batch_size)\n", + " save_freq=5*n_batches)\n", "\n", "# Create a new model instance\n", "model = create_model()\n", From 47d3c4bc0462984ec34444698711874539a383da Mon Sep 17 00:00:00 2001 From: kristenrq <58997898+kristenrq@users.noreply.github.com> Date: Mon, 7 Mar 2022 09:56:26 -0800 Subject: [PATCH 006/872] Create Contribute to TF SIGs page --- site/en/community/contribute/sigs.md | 97 ++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 site/en/community/contribute/sigs.md diff --git a/site/en/community/contribute/sigs.md b/site/en/community/contribute/sigs.md new file mode 100644 index 00000000000..2a6ac07775e --- /dev/null +++ b/site/en/community/contribute/sigs.md @@ -0,0 +1,97 @@ +# Contribute to TensorFlow Special Interest Groups (SIGs) + +The TensorFlow Special Interest Groups (TF SIGs) organize community contributions to key parts of the TensorFlow ecosystem. SIG leads and members work together to build and support important TensorFlow use cases. + +SIGs are led by members of the open source community, including industry collaborators and [Machine Learning Google Developer Experts](https://developers.google.com/community/experts) (ML GDEs). TensorFlow's success is due in large part to their hard work and contributions. + +We encourage you to join a SIG working on the area of TensorFlow's ecosystem you care most about. Not all SIGs will have the same level of energy, breadth of scope, or governance models — browse our [SIG charters](https://github.com/tensorflow/community/tree/master/sigs) to learn more. Stay connected with SIG leads and members on the [TensorFlow Forum](https://discuss.tensorflow.org/c/special-interest-groups/8), where you can subscribe to preferred [tags](https://discuss.tensorflow.org/tags) and learn more about the regular SIG meetings. + +## SIG Addons + +SIG Addons builds and maintains a repository of community contributions that conform to well-established API patterns, but implement new functionality not available in core TensorFlow. + +TensorFlow natively supports a large number of operators, layers, metrics, losses, optimizers, and more. However, in a fast-moving field like ML, there are many new developments that cannot be integrated into core TensorFlow (because their broad applicability is not yet clear, or it is mostly used by a smaller subset of the community). SIG Addons enables users to introduce new extensions to the TensorFlow ecosystem in a sustainable manner. + +SIG Addons on GitHub Contributing Discuss on the Forum + +## SIG Build + +SIG Build improves and extends the TensorFlow build process. SIG Build maintains a repository showcasing resources, guides, tools, and builds contributed by the community, for the community. + +SIG Build on GitHub Contributing Discuss on the Forum + +## SIG IO + +SIG IO maintains TensorFlow I/O, a collection of file systems and file formats that are not available in TensorFlow's built-in support. + +SIG IO on GitHub Contributing Discuss on the Forum + +## SIG JVM + +SIG JVM maintains the TF Java bindings to let users use JVM for building, training and running machine learning models. + +Java and other JVM languages, such as Scala or Kotlin, are frequently used in small-to-large enterprises all over the world, which makes TensorFlow a strategic choice for adopting machine learning at a large scale. + +SIG JVM on GitHub Contributing Discuss on the Forum + +## SIG Models + +SIG Models focuses on enabling contributions to the state-of-the-art model implementation in TensorFlow 2, and sharing best practices of using TensorFlow 2 for state-of-the-art research. Subgroups orient around different machine learning applications (Vision, NLP, etc.). + +SIG Models host discussions and collaborations around the [TensorFlow Model Garden](https://github.com/tensorflow/models) and [TensorFlow Hub](https://tfhub.dev). Learn how to contribute on GitHub below, or discuss [Research & Models](https://discuss.tensorflow.org/c/research-models/26) on the Forum. + +TensorFlow Model Garden on GitHub Contributing + +TensorFlow Hub on GitHub Contributing + +## SIG Micro + +SIG Micro discusses and shares updates on [TensorFlow Lite for Microcontrollers](https://www.tensorflow.org/lite/microcontrollers), a port of TensorFlow Lite designed to run machine learning models on DSPs, microcontrollers and other devices with limited memory. + +TensorFlow Lite Micro on GitHub Contributing Discuss on the Forum + +## SIG MLIR + +SIG MLIR maintains [MLIR](https://mlir.llvm.org/) dialects and utilities for TensorFlow, XLA and TF Lite, providing high performance compilers and optimization techniques that can be applied to TensorFlow graphs and code generation. Their overarching goal is to create common intermediate representation (IR) that reduces the cost to bring up new hardware, and improve usability for existing TensorFlow users. + +SIG MLIR on GitHub Contributing Discuss on the Forum + +## SIG Networking + +SIG Networking maintains the TensorFlow Networking repository for platform-specific networking extensions to core TensorFlow and related utilities. + +SIG Networking on GitHub Discuss on the Forum + +## SIG Recommenders + +SIG Recommenders maintains a collection of projects related to large-scale recommendation systems built upon TensorFlow contributed and maintained by the community. Those contributions are complementary to TensorFlow Core and TensorFlow Recommenders. + +SIG Recommenders on GitHub Contributing Discuss on the Forum + +## SIG Rust + +SIG Rust maintains idiomatic Rust language bindings for TensorFlow. + +SIG Rust on GitHub Contributing Discuss on the Forum + +## SIG TensorBoard + +SIG TensorBoard maintains a suite of web applications for inspecting and understanding TensorFlow runs and graphs. + +SIG TensorBoard on GitHub Contributing Discuss on the Forum + +## SIG TF.js + +SIG TF.js facilitates community-contributed components to TensorFlow.js and offers project support through the SIG. + +TensorFlow.js on GitHub Contributing Discuss on the Forum + +## SIG TFX-Addons + +SIG TFX-Addons accelerates the sharing of customizations and additions to meet the needs of production ML, expand the vision, and help drive new directions for [TensorFlow Extended (TFX)](https://www.tensorflow.org/tfx) and the ML community. + +SIG TFX-Addons on GitHub Contributing Discuss on the Forum + +## New SIGs + +Didn't find what you were looking for? If you believe there is a strong need for a new TensorFlow SIG, please read the [SIG playbook](https://www.tensorflow.org/community/sig_playbook) and follow instructions on how to propose it to our contributor community. From 4e6ba4ab2c0541607bfd4da1cc70584e166f2b9e Mon Sep 17 00:00:00 2001 From: mohantym <86464649+mohantym@users.noreply.github.com> Date: Tue, 8 Mar 2022 11:01:36 +0530 Subject: [PATCH 007/872] Fixed Broken link on "Saving and Restoring" Fixed Broken link on "Saving and Restoring" with website link. --- site/en/guide/checkpoint.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/checkpoint.ipynb b/site/en/guide/checkpoint.ipynb index a47c1bf1b0a..7e602cffb06 100644 --- a/site/en/guide/checkpoint.ipynb +++ b/site/en/guide/checkpoint.ipynb @@ -139,7 +139,7 @@ "## Saving from `tf.keras` training APIs\n", "\n", "See the [`tf.keras` guide on saving and\n", - "restoring](./keras/overview.ipynb#save_and_restore).\n", + "restoring](https://www.tensorflow.org/guide/saved_model).\n", "\n", "`tf.keras.Model.save_weights` saves a TensorFlow checkpoint. " ] From 54aab3f15cab40db213bc868a0e06598831b96fd Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 8 Mar 2022 15:36:02 +0000 Subject: [PATCH 008/872] Add link to TF.js in Contribute to TF SIGs guide --- site/en/community/contribute/sigs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/community/contribute/sigs.md b/site/en/community/contribute/sigs.md index 2a6ac07775e..67779f6c404 100644 --- a/site/en/community/contribute/sigs.md +++ b/site/en/community/contribute/sigs.md @@ -82,7 +82,7 @@ SIG TensorBoard maintains a suite of web applications for inspecting and underst ## SIG TF.js -SIG TF.js facilitates community-contributed components to TensorFlow.js and offers project support through the SIG. +SIG TF.js facilitates community-contributed components to [TensorFlow.js](https://www.tensorflow.org/js) and offers project support through the SIG. TensorFlow.js on GitHub Contributing Discuss on the Forum From 59c2c45b9ae1d552f1a975b200e799b63b320af5 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 8 Mar 2022 15:37:14 +0000 Subject: [PATCH 009/872] Add link to TensorBoard site in Contribute to TF SIGs page --- site/en/community/contribute/sigs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/community/contribute/sigs.md b/site/en/community/contribute/sigs.md index 67779f6c404..71c7c174310 100644 --- a/site/en/community/contribute/sigs.md +++ b/site/en/community/contribute/sigs.md @@ -76,7 +76,7 @@ SIG Rust maintains idiomatic Rust language bindings for TensorFlow. ## SIG TensorBoard -SIG TensorBoard maintains a suite of web applications for inspecting and understanding TensorFlow runs and graphs. +SIG [TensorBoard](https://www.tensorflow.org/tensorboard) maintains a suite of web applications for inspecting and understanding TensorFlow runs and graphs. SIG TensorBoard on GitHub Contributing Discuss on the Forum From 1d032c11e07372cc56db68350182e18dbd74e472 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 8 Mar 2022 15:38:20 +0000 Subject: [PATCH 010/872] Add links to TF Core and Recommenders in Contribute to TF SIGs page --- site/en/community/contribute/sigs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/community/contribute/sigs.md b/site/en/community/contribute/sigs.md index 71c7c174310..159f17cc1cb 100644 --- a/site/en/community/contribute/sigs.md +++ b/site/en/community/contribute/sigs.md @@ -64,7 +64,7 @@ SIG Networking maintains the TensorFlow Networking repository for platform-speci ## SIG Recommenders -SIG Recommenders maintains a collection of projects related to large-scale recommendation systems built upon TensorFlow contributed and maintained by the community. Those contributions are complementary to TensorFlow Core and TensorFlow Recommenders. +SIG Recommenders maintains a collection of projects related to large-scale recommendation systems built upon TensorFlow contributed and maintained by the community. Those contributions are complementary to [TensorFlow Core](https://www.tensorflow.org/overview) and [TensorFlow Recommenders](https://www.tensorflow.org/recommenders). SIG Recommenders on GitHub Contributing Discuss on the Forum From d89bf5935bd41e7f5cdfafe8dad7af6b5889f640 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 8 Mar 2022 16:52:23 +0000 Subject: [PATCH 011/872] Fix link to Using the SavedModel format guide in Training checkpoints guide --- site/en/guide/checkpoint.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/checkpoint.ipynb b/site/en/guide/checkpoint.ipynb index 7e602cffb06..ed727deab4d 100644 --- a/site/en/guide/checkpoint.ipynb +++ b/site/en/guide/checkpoint.ipynb @@ -139,7 +139,7 @@ "## Saving from `tf.keras` training APIs\n", "\n", "See the [`tf.keras` guide on saving and\n", - "restoring](https://www.tensorflow.org/guide/saved_model).\n", + "restoring](saved_model.ipynb).\n", "\n", "`tf.keras.Model.save_weights` saves a TensorFlow checkpoint. " ] From ab7b652244d67a245fcd5a947d082fca276d3578 Mon Sep 17 00:00:00 2001 From: kristenrq <58997898+kristenrq@users.noreply.github.com> Date: Tue, 8 Mar 2022 11:36:02 -0800 Subject: [PATCH 012/872] Copy update under SIG TensorBoard --- site/en/community/contribute/sigs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/community/contribute/sigs.md b/site/en/community/contribute/sigs.md index 159f17cc1cb..aa98dc32c75 100644 --- a/site/en/community/contribute/sigs.md +++ b/site/en/community/contribute/sigs.md @@ -76,9 +76,9 @@ SIG Rust maintains idiomatic Rust language bindings for TensorFlow. ## SIG TensorBoard -SIG [TensorBoard](https://www.tensorflow.org/tensorboard) maintains a suite of web applications for inspecting and understanding TensorFlow runs and graphs. +SIG TensorBoard facilitates discussion around [TensorBoard](https://www.tensorflow.org/tensorboard), a suite of web applications for inspecting and understanding TensorFlow runs and graphs. -SIG TensorBoard on GitHub Contributing Discuss on the Forum +TensorBoard on GitHub Contributing Discuss on the Forum ## SIG TF.js From c8bb6a2a2449bcbcbac1b3edfc0819f0ba37e59e Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Wed, 9 Mar 2022 01:09:45 +0000 Subject: [PATCH 013/872] Update link to Keras Save and load models link in Checkpointing guide --- site/en/guide/checkpoint.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/checkpoint.ipynb b/site/en/guide/checkpoint.ipynb index ed727deab4d..fb3b45437f7 100644 --- a/site/en/guide/checkpoint.ipynb +++ b/site/en/guide/checkpoint.ipynb @@ -139,7 +139,7 @@ "## Saving from `tf.keras` training APIs\n", "\n", "See the [`tf.keras` guide on saving and\n", - "restoring](saved_model.ipynb).\n", + "restoring](https://www.tensorflow.org/guide/keras/save_and_serialize).\n", "\n", "`tf.keras.Model.save_weights` saves a TensorFlow checkpoint. " ] From 383c338d8a37cf3ad5bd55b1111f009babd04afd Mon Sep 17 00:00:00 2001 From: Mark McDonald Date: Wed, 9 Mar 2022 01:24:56 -0800 Subject: [PATCH 014/872] Add YAML pre-processing function for nesting dotted packages. Python docs emit `tf.foo`, this utility function nests that as `tf` > `foo`. The `gen_java` directory isn't a great place for this, so I've got a separate change out to move this module. PiperOrigin-RevId: 433418091 --- .../api_generator/gen_java/processing.py | 50 ++++++++++++++++++- .../api_generator/gen_java/processing_test.py | 48 ++++++++++++++++++ 2 files changed, 96 insertions(+), 2 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/gen_java/processing.py b/tools/tensorflow_docs/api_generator/gen_java/processing.py index fed6ee0724c..74cbcca9e13 100644 --- a/tools/tensorflow_docs/api_generator/gen_java/processing.py +++ b/tools/tensorflow_docs/api_generator/gen_java/processing.py @@ -14,9 +14,13 @@ # limitations under the License. # ============================================================================== """Tools for processing generated Java documentation.""" -from typing import Any, Iterable, Mapping, Sequence +from typing import Any, Iterable, Mapping, MutableMapping, MutableSequence -Toc = Mapping[str, Sequence[Mapping[str, Any]]] +# TODO(b/193033225): If possible, this should be a TypedDict. If not, using the +# real protos might make things a little cleaner. +TocEntry = MutableMapping[str, Any] +Section = MutableSequence[TocEntry] +Toc = Mapping[str, Section] def add_package_headings(toc: Toc, root_pkgs: Iterable[str], @@ -43,6 +47,48 @@ def add_package_headings(toc: Toc, root_pkgs: Iterable[str], return {'toc': new_toc} +def nest_toc(toc: Toc) -> Toc: + """Nests a flat TOC into a tree structure based on common packages.""" + new_toc = [] + + # We only look at the first level for flat package names. + entries_by_title = {e['title']: e for e in toc['toc']} + for title, entry in entries_by_title.items(): + target_entry = _nest_toc_entry(title, new_toc) + + # Populate the target entry with the original entry, sans title. + # (pytype suppressed due to inferring .keys() as a List) + fields = entry.keys() - {'title'} # pytype: disable=unsupported-operands + target_entry.update({f: entry[f] for f in fields}) + + # Clean up empty sections + if not target_entry.get('section'): + target_entry.pop('section', None) + + return {'toc': new_toc} + + +def _nest_toc_entry(title: str, section: Section) -> TocEntry: + """Nest the title (split by .) into the TOC. Creating hierarchy as needed.""" + pkg, *maybe_rest = title.split('.', 1) + + for entry in section: + if entry.get('title') == pkg: + target_entry = entry + if 'section' not in target_entry: + target_entry['section'] = [] + break + else: + target_entry = {'title': pkg, 'section': []} + section.append(target_entry) + + if not maybe_rest: + return target_entry + else: + rest = maybe_rest[0] + return _nest_toc_entry(rest, target_entry['section']) + + def sort_toc(toc: Toc, labels: Iterable[str]) -> Toc: """Pre-sort the TOC entries by `labels`.""" new_toc = [] diff --git a/tools/tensorflow_docs/api_generator/gen_java/processing_test.py b/tools/tensorflow_docs/api_generator/gen_java/processing_test.py index 7778b5d3a01..b7b43094c7f 100644 --- a/tools/tensorflow_docs/api_generator/gen_java/processing_test.py +++ b/tools/tensorflow_docs/api_generator/gen_java/processing_test.py @@ -128,6 +128,54 @@ def test_multiple_roots(self): } self.assertDictEqual(actual_toc, expected_toc) + def test_nesting_toc(self): + toc_in = { + 'toc': [{ + 'title': 'tf_lite.support', + 'path': '/tflite/support.html', + }, { + 'title': 'tf_lite.support.cls', + 'path': '/tflite/support/cls.html', + }, { + 'title': 'tf_lite.task.things', + 'path': '/tflite/task/things.html', + }, { + 'title': 'tf_other.widgets', + 'path': '/tfother/widgets.html', + }] + } + actual_toc = processing.nest_toc(toc_in) + expected_toc = { + 'toc': [{ + 'title': + 'tf_lite', + 'section': [{ + 'title': + 'support', + 'path': + '/tflite/support.html', + 'section': [{ + 'title': 'cls', + 'path': '/tflite/support/cls.html' + }] + }, { + 'title': + 'task', + 'section': [{ + 'title': 'things', + 'path': '/tflite/task/things.html' + }] + }] + }, { + 'title': 'tf_other', + 'section': [{ + 'title': 'widgets', + 'path': '/tfother/widgets.html' + }] + }] + } + self.assertEqual(actual_toc['toc'], expected_toc['toc']) + def test_ordering(self): toc_in = { 'toc': [ From 6dc4b26c2a5c4b80e323d6beba7684a2dd83de6c Mon Sep 17 00:00:00 2001 From: Mark McDonald Date: Wed, 9 Mar 2022 01:34:16 -0800 Subject: [PATCH 015/872] Move `processing` YAML post-proc out of `gen_java`. I intend on using this for some of the Python TOC processing, so keeping it under the Java gen dir doesn't make a lot of sense. PiperOrigin-RevId: 433419504 --- tools/tensorflow_docs/api_generator/__init__.py | 1 + .../tensorflow_docs/api_generator/gen_java/__init__.py | 6 +++--- .../{gen_java/processing.py => toc_processing.py} | 0 .../processing_test.py => toc_processing_test.py} | 10 +++++----- 4 files changed, 9 insertions(+), 8 deletions(-) rename tools/tensorflow_docs/api_generator/{gen_java/processing.py => toc_processing.py} (100%) rename tools/tensorflow_docs/api_generator/{gen_java/processing_test.py => toc_processing_test.py} (94%) diff --git a/tools/tensorflow_docs/api_generator/__init__.py b/tools/tensorflow_docs/api_generator/__init__.py index 75b29e5ce1f..63704e38eeb 100644 --- a/tools/tensorflow_docs/api_generator/__init__.py +++ b/tools/tensorflow_docs/api_generator/__init__.py @@ -19,6 +19,7 @@ from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser +from tensorflow_docs.api_generator import toc_processing from tensorflow_docs.api_generator import public_api from tensorflow_docs.api_generator import traverse from tensorflow_docs.api_generator import utils diff --git a/tools/tensorflow_docs/api_generator/gen_java/__init__.py b/tools/tensorflow_docs/api_generator/gen_java/__init__.py index a513595dd79..52dfd9a1872 100644 --- a/tools/tensorflow_docs/api_generator/gen_java/__init__.py +++ b/tools/tensorflow_docs/api_generator/gen_java/__init__.py @@ -20,7 +20,7 @@ import subprocess from typing import Iterable, Mapping, Optional, Union -from tensorflow_docs.api_generator.gen_java import processing +from tensorflow_docs.api_generator import toc_processing import yaml @@ -78,7 +78,7 @@ def gen_java_docs( yaml_content = yaml_path.read_text() yaml_data = yaml.safe_load(yaml_content) if section_labels: - yaml_data = processing.add_package_headings(yaml_data, root_pkgs, - section_labels) + yaml_data = toc_processing.add_package_headings(yaml_data, root_pkgs, + section_labels) yaml_content = yaml.dump(yaml_data, Dumper=Formatter) yaml_path.write_text(yaml_content) diff --git a/tools/tensorflow_docs/api_generator/gen_java/processing.py b/tools/tensorflow_docs/api_generator/toc_processing.py similarity index 100% rename from tools/tensorflow_docs/api_generator/gen_java/processing.py rename to tools/tensorflow_docs/api_generator/toc_processing.py diff --git a/tools/tensorflow_docs/api_generator/gen_java/processing_test.py b/tools/tensorflow_docs/api_generator/toc_processing_test.py similarity index 94% rename from tools/tensorflow_docs/api_generator/gen_java/processing_test.py rename to tools/tensorflow_docs/api_generator/toc_processing_test.py index b7b43094c7f..595d7b9e7c5 100644 --- a/tools/tensorflow_docs/api_generator/gen_java/processing_test.py +++ b/tools/tensorflow_docs/api_generator/toc_processing_test.py @@ -17,7 +17,7 @@ from absl.testing import absltest -from tensorflow_docs.api_generator.gen_java import processing +from tensorflow_docs.api_generator import toc_processing class ProcessingTest(absltest.TestCase): @@ -55,7 +55,7 @@ def test_toc_package_sections(self): 'org.tf': 'RootLabel', 'org.tf.foo': 'FooPackage', } - actual_toc = processing.add_package_headings(toc_in, ['org.tf'], labels) + actual_toc = toc_processing.add_package_headings(toc_in, ['org.tf'], labels) expected_toc = { 'toc': [ { @@ -109,7 +109,7 @@ def test_multiple_roots(self): }] } roots = ['org.tf', 'com.google'] - actual_toc = processing.add_package_headings(toc_in, roots, {}) + actual_toc = toc_processing.add_package_headings(toc_in, roots, {}) expected_toc = { 'toc': [ { @@ -144,7 +144,7 @@ def test_nesting_toc(self): 'path': '/tfother/widgets.html', }] } - actual_toc = processing.nest_toc(toc_in) + actual_toc = toc_processing.nest_toc(toc_in) expected_toc = { 'toc': [{ 'title': @@ -194,7 +194,7 @@ def test_ordering(self): ] } labels = ['org.tf.b', 'com.google.c', 'org.tf'] - actual_toc = processing.sort_toc(toc_in, labels) + actual_toc = toc_processing.sort_toc(toc_in, labels) expected_toc = { 'toc': [ { From 78819d47fabc55fb492279d14865375f7a5d04b5 Mon Sep 17 00:00:00 2001 From: Xinyi Wang Date: Wed, 9 Mar 2022 08:07:58 -0800 Subject: [PATCH 016/872] Update the tutorial (and remove a few typos). PiperOrigin-RevId: 433481314 --- .../parameter_server_training.ipynb | 48 +++++++++++++++---- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/site/en/tutorials/distribute/parameter_server_training.ipynb b/site/en/tutorials/distribute/parameter_server_training.ipynb index 5130c5f90d0..309210bef35 100644 --- a/site/en/tutorials/distribute/parameter_server_training.ipynb +++ b/site/en/tutorials/distribute/parameter_server_training.ipynb @@ -421,7 +421,15 @@ }, "source": [ "The code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n", - "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "w60PuWrWwBD4" + }, + "source": [ "### Model construction and compiling\n", "\n", "Now, you will create a `tf.keras.Model`—a trivial `tf.keras.models.Sequential` model for demonstration purposes—followed by a `Model.compile` call to incorporate components, such as an optimizer, metrics, or parameters such as `steps_per_execution`:" @@ -520,11 +528,13 @@ "source": [ "### Set up the data\n", "\n", - "First, write a function that creates a dataset that includes preprocessing logic implemented by [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers).\n", + "First, write a function that creates a dataset.\n", + "\n", + "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `strategy.scope()`** like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline. \n", "\n", - "You will create these layers outside the `dataset_fn` but apply the transformation inside the `dataset_fn`, since you will wrap the `dataset_fn` into a `tf.function`, which doesn't allow variables to be created inside it.\n", + "If you don't follow the above procedure, creating the layers might create Tensorflow states which will be lifted out of the `tf.function` to the coordinator, and thus accessing them on workers would incur repetitive RPC calls between coordinator and workers and causes significant slowdown. \n", "\n", - "Note: There is a known performance implication when using lookup table resources, which layers, such as `tf.keras.layers.StringLookup`, employ. Refer to the [Known limitations](#known_limitations) section for more information." + "Placing the layers under `strategy.scope()` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Please refer to [this tutorial](https://www.tensorflow.org/tutorials/distribute/input#data_preprocessing) for more information on data preprocessing with distributed input." ] }, { @@ -891,7 +901,7 @@ "source": [ "### Inline evaluation\n", "\n", - "In this method, the coordinator alternates between training and evaluation and thus it is called it _inline evaluation_.\n", + "In this method, the coordinator alternates between training and evaluation, and thus it is called _inline evaluation_.\n", "\n", "There are several benefits of inline evaluation. For example:\n", "\n", @@ -1031,9 +1041,9 @@ "\n", "In a real production environment, you will run all tasks in different processes on different machines. The simplest way to configure cluster information on each task is to set `\"TF_CONFIG\"` environment variables and use a `tf.distribute.cluster_resolver.TFConfigClusterResolver` to parse `\"TF_CONFIG\"`.\n", "\n", - "For a general description about `\"TF_CONFIG\"` environment variables, refer to the [Distributed training](https://www.tensorflow.org/guide/distributed_training#setting_up_tf_config_environment_variable) guide.\n", + "For a general description of `\"TF_CONFIG\"` environment variables, refer to the [Distributed training](https://www.tensorflow.org/guide/distributed_training#setting_up_tf_config_environment_variable) guide.\n", "\n", - "If you start your training tasks using Kubernetes or other configuration templates, it is very likely that these templates have already set `“TF_CONFIG\"` for you." + "If you start your training tasks using Kubernetes or other configuration templates, likely, these templates have already set `“TF_CONFIG\"` for you." ] }, { @@ -1217,7 +1227,7 @@ "2. Avoid creating a hotspot variable that is required by all parameter servers in a single step if possible. For example, use a constant learning rate or subclass `tf.keras.optimizers.schedules.LearningRateSchedule` in optimizers since the default behavior is that the learning rate will become a variable placed on a particular parameter server and requested by all other parameter servers in each step.\n", "3. Shuffle your large vocabularies before passing them to Keras preprocessing layers.\n", "\n", - "Another possible reason for performance issues is the coordinator. Your first implementation of `schedule`/`join` is Python-based and thus may have threading overhead. Also the latency between the coordinator and the workers can be large. If this is the case,\n", + "Another possible reason for performance issues is the coordinator. The implementation of `schedule`/`join` is Python-based and thus may have threading overhead. Also, the latency between the coordinator and the workers can be large. If this is the case,\n", "\n", "- For `Model.fit`, you can set `steps_per_execution` argument provided at `Model.compile` to a value larger than 1.\n", "\n", @@ -1261,15 +1271,32 @@ "- It is not supported to load a saved_model via `tf.saved_model.load` containing sharded variables. Note loading such a saved_model using TensorFlow Serving is expected to work.\n", "- It is not supported to load a checkpoint containing sharded optimizer slot variables into a different number of shards.\n", "- It is not supported to recover from parameter server failure without restarting the coordinator task.\n", - "- Usage of `tf.lookup.StaticHashTable` (which is commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`) results in resources placed on the coordinator at this time with parameter server training. This has performance implications for lookup RPCs from workers to the coordinator. This is a current high priority to address.\n", + "- Creation of `tf.lookup.StaticHashTable`, commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`, should be placed under `strategy.scope()`. Otherwise, resources will be placed on the coordinator, and lookup RPCs from workers to the coordinator incur performance implications. \n", "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2MKBF0RPSvzB" + }, + "source": [ "### `Model.fit` specifics\n", "\n", "- `steps_per_epoch` argument is required in `Model.fit`. You can select a value that provides appropriate intervals in an epoch.\n", "- `ParameterServerStrategy` does not have support for custom callbacks that have batch-level calls for performance reasons. You should convert those calls into epoch-level calls with suitably picked `steps_per_epoch`, so that they are called every `steps_per_epoch` number of steps. Built-in callbacks are not affected: their batch-level calls have been modified to be performant. Supporting batch-level calls for `ParameterServerStrategy` is being planned.\n", "- For the same reason, unlike other strategies, progress bar and metrics are logged only at epoch boundaries.\n", "- `run_eagerly` is not supported.\n", - "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wvY-mg35Sx5L" + }, + "source": [ "### Custom training loop specifics\n", "\n", "- `ClusterCoordinator.schedule` doesn't support visitation guarantees for a dataset.\n", @@ -1283,6 +1310,7 @@ "colab": { "collapsed_sections": [], "name": "parameter_server_training.ipynb", + "provenance": [], "toc_visible": true }, "kernelspec": { From 8fb96646927cef090d6c6458c9ea837c39d98674 Mon Sep 17 00:00:00 2001 From: kristenrq <58997898+kristenrq@users.noreply.github.com> Date: Fri, 11 Mar 2022 09:32:41 -0800 Subject: [PATCH 017/872] Update site/en/community/contribute/sigs.md Co-authored-by: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> --- site/en/community/contribute/sigs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/community/contribute/sigs.md b/site/en/community/contribute/sigs.md index aa98dc32c75..b736ec5919a 100644 --- a/site/en/community/contribute/sigs.md +++ b/site/en/community/contribute/sigs.md @@ -76,7 +76,7 @@ SIG Rust maintains idiomatic Rust language bindings for TensorFlow. ## SIG TensorBoard -SIG TensorBoard facilitates discussion around [TensorBoard](https://www.tensorflow.org/tensorboard), a suite of web applications for inspecting and understanding TensorFlow runs and graphs. +SIG TensorBoard facilitates discussion around [TensorBoard](https://www.tensorflow.org/tensorboard)—a suite of tools for inspecting, debugging and optimizing TensorFlow programs. TensorBoard on GitHub Contributing Discuss on the Forum From 49b6621f6fbe26a1ab65d0c019ab22b0d612163c Mon Sep 17 00:00:00 2001 From: Mark McDonald Date: Mon, 14 Mar 2022 20:17:20 -0700 Subject: [PATCH 018/872] Make `code_url_prefix` optional. Existing tests (e.g. `pretty_docs_test`) already cover optional URLs, so make it official through the API. This allows doc generation scripts to skip the top source links per `base_dir`. PiperOrigin-RevId: 434640091 --- tools/tensorflow_docs/api_generator/generate_lib.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index 4b6ee11f3cd..bd25bf6aae5 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -705,7 +705,7 @@ def __init__( root_title: str, py_modules: Sequence[Tuple[str, Any]], base_dir: Optional[Sequence[Union[str, pathlib.Path]]] = None, - code_url_prefix: Union[str, Sequence[str]] = (), + code_url_prefix: Union[Optional[str], Sequence[Optional[str]]] = (), search_hints: bool = True, site_path: str = 'api_docs/python', private_map: Optional[Dict[str, str]] = None, @@ -774,7 +774,7 @@ def __init__( if not self._base_dir: raise ValueError('`base_dir` cannot be empty') - if isinstance(code_url_prefix, str): + if isinstance(code_url_prefix, str) or code_url_prefix is None: code_url_prefix = (code_url_prefix,) self._code_url_prefix = tuple(code_url_prefix) if not self._code_url_prefix: From 2047bac087c506a054894271b49d29eed8270211 Mon Sep 17 00:00:00 2001 From: Kris Tonthat <3449962+ktonthat@users.noreply.github.com> Date: Tue, 15 Mar 2022 12:40:14 -0700 Subject: [PATCH 019/872] Update basics.ipynb Correct language issues in the tensors and operations page: "This enables a more interactive frontend to TensorFlow, the details of which we will discuss much later." 1. "This" should not be used without an object for clarity: "This enables" --> "This xxxx enables" 2. Use second person ("you") instead of "we" https://developers.google.com/style/person?hl=en 3. Inclusive language: native -> built-in Existing URLs containing the issue: https://www.tensorflow.org/tutorials/customization/basics --- site/en/tutorials/customization/basics.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb index 314738300e3..fa13409feaa 100644 --- a/site/en/tutorials/customization/basics.ipynb +++ b/site/en/tutorials/customization/basics.ipynb @@ -84,7 +84,7 @@ "source": [ "## Import TensorFlow\n", "\n", - "To get started, import the `tensorflow` module. As of TensorFlow 2, eager execution is turned on by default. This enables a more interactive frontend to TensorFlow, the details of which we will discuss much later." + "To get started, import the `tensorflow` module. As of TensorFlow 2, eager execution is turned on by default. Eager execution enables a more interactive frontend to TensorFlow, which you will later explore in more detail." ] }, { @@ -106,7 +106,7 @@ "source": [ "## Tensors\n", "\n", - "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations ([tf.add](https://www.tensorflow.org/api_docs/python/tf/add), [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/matmul), [tf.linalg.inv](https://www.tensorflow.org/api_docs/python/tf/linalg/inv) etc.) that consume and produce `tf.Tensor`s. These operations automatically convert native Python types, for example:\n" + "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations ([tf.add](https://www.tensorflow.org/api_docs/python/tf/add), [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/matmul), [tf.linalg.inv](https://www.tensorflow.org/api_docs/python/tf/linalg/inv) etc.) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types, for example:\n" ] }, { From fd8ebe8e140abb8ba3cea84ef31a535e17fac54c Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Tue, 15 Mar 2022 17:43:34 -0700 Subject: [PATCH 020/872] Internal change PiperOrigin-RevId: 434904150 --- site/en/community/contribute/docs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/site/en/community/contribute/docs.md b/site/en/community/contribute/docs.md index 772cbd6627a..29b2b5c9550 100644 --- a/site/en/community/contribute/docs.md +++ b/site/en/community/contribute/docs.md @@ -167,21 +167,21 @@ when you submit your pull request. Add a remote: 
-git remote add upstream git@github.com:tensorflow/docs.git
+git remote add upstream git@github.com:tensorflow/docs.git
 
 # View remote repos
 git remote -v
 origin    git@github.com:username/docs.git (fetch)
 origin    git@github.com:username/docs.git (push)
-upstream  git@github.com:tensorflow/docs.git (fetch)
-upstream  git@github.com:tensorflow/docs.git (push)
+upstream  git@github.com:tensorflow/docs.git (fetch)
+upstream  git@github.com:tensorflow/docs.git (push)
To update: 
 git checkout master
-git pull upstream master
+git pull upstream master
 
 git push  # Push changes to your GitHub account (defaults to origin)
From 2fd6dce63811452fa377a233ce30d58cefb70901 Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Wed, 16 Mar 2022 03:40:03 -0700 Subject: [PATCH 021/872] Update the TensorFlow Decision Forests section in the public canned estimator migration guide (https://www.tensorflow.org/guide/migrate/canned_estimators). PiperOrigin-RevId: 435004032 --- site/en/guide/migrate/canned_estimators.ipynb | 126 ++++++++++++++---- 1 file changed, 101 insertions(+), 25 deletions(-) diff --git a/site/en/guide/migrate/canned_estimators.ipynb b/site/en/guide/migrate/canned_estimators.ipynb index 413c8116639..66d688b7676 100644 --- a/site/en/guide/migrate/canned_estimators.ipynb +++ b/site/en/guide/migrate/canned_estimators.ipynb @@ -587,40 +587,100 @@ "id": "m3EVq388eOKF" }, "source": [ - "In TensorFlow 2, the closest pre-packaged substitute for a model generated by `tf.estimator.BoostedTreesEstimator` is one created using `tfdf.keras.GradientBoostedTreesModel`, which creates a sequentially-trained sequence of shallow decision trees, each designed to \"learn\" from errors made by its predecessors in the sequence.\n", + "In TensorFlow 2, `tf.estimator.BoostedTreesEstimator` is replaced by [tfdf.keras.GradientBoostedTreesModel](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel#attributes) from the [TensorFlow Decision Forests](https://www.tensorflow.org/decision_forests) package.\n", "\n", - "`GradientBoostedTreesModel` provides more options for customization, allowing for the specification of everything from basic depth constraints to early stopping conditions. See [here](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel#attributes) for more `GradientBoostedTreesModel` attribute details." + "TensorFlow Decision Forests provides various advantages over the `tf.estimator.BoostedTreesEstimator`, notably regarding quality, speed, ease of use and flexibility. To learn about TensorFlow Decision Forests, start with the [beginner colab](https://www.tensorflow.org/decision_forests/tutorials/beginner_colab).\n", + "\n", + "The following example shows how to train a Gradient Boosted Trees model using TensorFlow 2:" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UB90fXJdVWC5" + }, + "source": [ + "Install TensorFlow Decision Forests." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "JLS_2vKKeOKF" + "id": "9097mTCIVVE9" }, "outputs": [], "source": [ - "gbt_model = tfdf.keras.GradientBoostedTreesModel(\n", - " task=tfdf.keras.Task.CLASSIFICATION)\n", - "gbt_model.compile(metrics=['mse', 'accuracy'])" + "!pip install tensorflow_decision_forests" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "B1qTdAS-VpXk" + }, + "source": [ + "Create a TensorFlow dataset. Note that Decision Forests support natively many types of features and do not need pre-processing." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "sZZSM7_VeOKF" + "id": "jkjFHmDTVswY" }, "outputs": [], "source": [ - "train_df, eval_df = x_train.copy(), x_eval.copy()\n", - "train_df['survived'], eval_df['survived'] = y_train, y_eval\n", + "train_dataframe = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/train.csv')\n", + "eval_dataframe = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/eval.csv')\n", "\n", - "train_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(train_df, label='survived')\n", - "eval_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(eval_df, label='survived')\n", - "\n", - "gbt_model.fit(train_dataset)\n", - "gbt_model.evaluate(eval_dataset, return_dict=True)" + "# Convert the Pandas Dataframes into TensorFlow datasets.\n", + "train_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(train_dataframe, label=\"survived\")\n", + "eval_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(eval_dataframe, label=\"survived\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7fPa-LfDWDzB" + }, + "source": [ + "Train the model on the `train_dataset` dataset." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "JO0yCH9hWPvJ" + }, + "outputs": [], + "source": [ + "# Use the default hyper-parameters of the model.\n", + "gbt_model = tfdf.keras.GradientBoostedTreesModel()\n", + "gbt_model.fit(train_dataset)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2Y5xm29AWGxt" + }, + "source": [ + "Evaluate the quality of the model on the `eval_dataset` dataset." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "JLS_2vKKeOKF" + }, + "outputs": [], + "source": [ + "gbt_model.compile(metrics=['accuracy'])\n", + "gbt_evaluation = gbt_model.evaluate(eval_dataset, return_dict=True)\n", + "print(gbt_evaluation)" ] }, { @@ -629,36 +689,52 @@ "id": "Z22UJ5SUqToQ" }, "source": [ - "In TensorFlow 2, there is also another available TFDF substitute for a model generated by `tf.estimator.BoostedTreesEstimator` - `tfdf.keras.RandomForestModel`. `RandomForestModel` creates a robust, overfitting-resistant learner consisting of a voting population of deep decision trees, each trained on random subsets of the input training dataset.\n", + "Gradient Boosted Trees is just one of the many decision forests algorithms avaiable in TensorFlow Decision Forests. For example, Random Forests (available as [tfdf.keras.GradientBoostedTreesModel](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/RandomForestModel) is very resistant to overfitting) while CART (available as [tfdf.keras.CartModel](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/CartModel)) is great for model interpretation.\n", "\n", - "`RandomForestModel` and `GradientBoostedTreesModel` provide similarly extensive levels of customization. Choosing between them is problem-specific and dependent on your task or application.\n", - "\n", - "Check the API docs for more information on the [`RandomForestModel`](https://https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/RandomForestModel#attributes) and [`GradientBoostedTreesModel`](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel#attributes) attribute." + "In the next example, we train and plot a Random Forest model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "027bGnCork_W" + "id": "W3slOhn4Zi9X" }, "outputs": [], "source": [ - "rf_model = tfdf.keras.RandomForestModel(\n", - " task=tfdf.keras.Task.CLASSIFICATION)\n", - "rf_model.compile(metrics=['mse', 'accuracy'])" + "# Train a Random Forest model\n", + "rf_model = tfdf.keras.RandomForestModel()\n", + "rf_model.fit(train_dataset)\n", + "\n", + "# Evaluate the Random Forest model\n", + "rf_model.compile(metrics=['accuracy'])\n", + "rf_evaluation = rf_model.evaluate(eval_dataset, return_dict=True)\n", + "print(rf_evaluation)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Z0QYolhoZb_k" + }, + "source": [ + "Finaly, in the next example, we train and evaluate a CART model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "Tc7KtjMlryn_" + "id": "027bGnCork_W" }, "outputs": [], "source": [ - "rf_model.fit(train_dataset)\n", - "rf_model.evaluate(eval_dataset, return_dict=True)" + "# Train a CART model\n", + "cart_model = tfdf.keras.CartModel()\n", + "cart_model.fit(train_dataset)\n", + "\n", + "# Plot the CART model\n", + "tfdf.model_plotter.plot_model_in_colab(cart_model, max_depth=2)" ] } ], From e51ae97c932ad8c8a6f9fcdbe4b31e6c45a90e64 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 16 Mar 2022 05:07:48 -0700 Subject: [PATCH 022/872] Drop autogenerated dataclass docstrings. These can be huge and unreadable. PiperOrigin-RevId: 435018511 --- tools/tensorflow_docs/api_generator/parser.py | 25 ++++++++++++++++--- .../api_generator/parser_test.py | 20 +++++++++++++++ 2 files changed, 41 insertions(+), 4 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py index 5b34130a438..190abc975be 100644 --- a/tools/tensorflow_docs/api_generator/parser.py +++ b/tools/tensorflow_docs/api_generator/parser.py @@ -108,16 +108,23 @@ def _get_raw_docstring(py_object): Returns: The docstring, or the empty string if no docstring was found. """ + obj_type = obj_type_lib.ObjType.get(py_object) - if obj_type_lib.ObjType.get(py_object) is obj_type_lib.ObjType.TYPE_ALIAS: + if obj_type is obj_type_lib.ObjType.TYPE_ALIAS: if inspect.getdoc(py_object) != inspect.getdoc(py_object.__origin__): result = inspect.getdoc(py_object) else: result = '' - elif obj_type_lib.ObjType.get(py_object) is not obj_type_lib.ObjType.OTHER: - result = inspect.getdoc(py_object) or '' - else: + elif obj_type is obj_type_lib.ObjType.CLASS: + if dataclasses.is_dataclass(py_object): + result = _get_dataclass_docstring(py_object) + else: + result = inspect.getdoc(py_object) or '' + elif obj_type is obj_type_lib.ObjType.OTHER: result = '' + else: + result = inspect.getdoc(py_object) or '' + if result is None: result = '' @@ -129,6 +136,16 @@ def _get_raw_docstring(py_object): return result +def _get_dataclass_docstring(py_object): + result = inspect.getdoc(py_object) or '' + + if (result.startswith(f'{py_object.__name__}(') and result.endswith(')') and + '\n' not in result): + # This is probably an autogenerated dataclass docstring. + # We don't want it. These are not formatted and can be huge and unreadable. + result = '' + return result + class _AddDoctestFences(object): """Adds ``` fences around doctest caret blocks >>> that don't have them.""" CARET_BLOCK_RE = re.compile( diff --git a/tools/tensorflow_docs/api_generator/parser_test.py b/tools/tensorflow_docs/api_generator/parser_test.py index 6507645f1ea..81aa1d4e11e 100644 --- a/tools/tensorflow_docs/api_generator/parser_test.py +++ b/tools/tensorflow_docs/api_generator/parser_test.py @@ -1044,6 +1044,26 @@ def test_strip_pylintandpyformat(self): strip_todos = parser._StripPylintAndPyformat() self.assertEqual(expected, strip_todos(input_str)) + def test_get_dataclass_docstring(self): + + @dataclasses.dataclass + class MyClass(): + """docstring""" + a: int + b: float + + self.assertEqual(parser._get_raw_docstring(MyClass), 'docstring') + + def test_get_dataclass_docstring_no_autogen_docstring(self): + + @dataclasses.dataclass + class MyClass(): + a: int + b: float + + self.assertEmpty(parser._get_raw_docstring(MyClass)) + + if __name__ == '__main__': absltest.main() From f4c71d8b0106d203807670bf01a4599b5f7816f2 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 16 Mar 2022 05:15:18 -0700 Subject: [PATCH 023/872] Add a doc-generator script for the full TensorFlow models package. This will likely replace the adjacent build_nlp_api_docs and build_vision_api_docs scripts. PiperOrigin-RevId: 435019769 --- tools/tensorflow_docs/api_generator/public_api.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py index b6b5239693e..c81db3ed1a1 100644 --- a/tools/tensorflow_docs/api_generator/public_api.py +++ b/tools/tensorflow_docs/api_generator/public_api.py @@ -185,7 +185,11 @@ def visit_Import(self, node): # pylint: disable=invalid-name def visit_ImportFrom(self, node): # pylint: disable=invalid-name self._add_imported_symbol(node) - source = inspect.getsource(obj) + try: + source = inspect.getsource(obj) + except OSError: + # inspect raises an OSError for empty module files. + return [] tree = ast.parse(source) visitor = ImportNodeVisitor() visitor.visit(tree) From de1b7e61197676e2d1e8d88b9186e01da3db9b48 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Fri, 18 Mar 2022 09:18:20 -0700 Subject: [PATCH 024/872] Separate the signature-structure from the printing logic. A `inspect.Signature` object contains a return_annotation and a dictionary of parameters. each parameter in the dictionary knows its argtype (positional, arg, vararg, kwarg, kwarg-only, varkwarg), its default, and its annotation. The signature object is responsible for converting this structure to a printable string. The `TfSignature` object works the same way, except that it expects (value, source) pairs for defaults and annotations. Output changes: - Fixes arg order for when a key-word argument comes after a vararg, like `tfm.vision.layers.Conv3D(*args, use_buffered_input=False)`. - The text used for dataclass defaults is shown instead of the value (when possible) - `timeout_s:int=60*60` displays 60*60 instead of 3600. - For dataclasses using factories the defaults are now shown as `dataclasses.field(default_factory=list)` instead of ``. - Fixes missing annotations for dataclasses using inheritance. Code changes: * generate_lib.py - `page_info.build` needs to happen in a `reference_resolver.temp_prefix` context so I've moved that inside docs_for_object. This could probably be cleaner. * parser.py - uses `signature_lib.strip_obj_addresses` to cleanup the reprs os objects printed to class and module "other attributes" tables. * signature.py - Merged the annotation and defaults ast-extractors. - Moved "strip_object_addresses" to a stand-alone function. - Converted all the signature code to create `inspect.Signature` style objects where the defaults and annotations all contain (value, text) pairs. * cleared all the lint warning. PiperOrigin-RevId: 435656377 --- .../api_generator/generate_lib.py | 9 +- tools/tensorflow_docs/api_generator/parser.py | 6 +- .../api_generator/parser_test.py | 30 +- .../api_generator/pretty_docs/base_page.py | 20 +- .../api_generator/pretty_docs/class_page.py | 6 +- .../pretty_docs/docs_for_object.py | 4 + .../pretty_docs/type_alias_page.py | 2 +- .../api_generator/signature.py | 487 ++++++++++-------- .../api_generator/signature_test.py | 168 ++++-- 9 files changed, 448 insertions(+), 284 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index bd25bf6aae5..35873c5d825 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -33,7 +33,6 @@ from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib from tensorflow_docs.api_generator import signature from tensorflow_docs.api_generator import traverse -from tensorflow_docs.api_generator.pretty_docs import base_page from tensorflow_docs.api_generator.pretty_docs import docs_for_object @@ -563,15 +562,13 @@ def write_docs( path = output_dir / parser.documentation_path(full_name) - text = page_info.build() - try: path.parent.mkdir(exist_ok=True, parents=True) - path.write_text(text, encoding='utf-8') + path.write_text(page_info.text, encoding='utf-8') num_docs_output += 1 - except OSError: + except OSError as e: raise OSError('Cannot write documentation for ' - f'{full_name} to {path.parent}') + f'{full_name} to {path.parent}') from e duplicates = parser_config.duplicates.get(full_name, []) if not duplicates: diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py index 190abc975be..67e070f0b72 100644 --- a/tools/tensorflow_docs/api_generator/parser.py +++ b/tools/tensorflow_docs/api_generator/parser.py @@ -18,7 +18,6 @@ import dataclasses import enum import inspect -import os import pathlib import posixpath import pprint @@ -31,6 +30,7 @@ from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import obj_type as obj_type_lib +from tensorflow_docs.api_generator import signature as signature_lib @dataclasses.dataclass @@ -125,7 +125,6 @@ def _get_raw_docstring(py_object): else: result = inspect.getdoc(py_object) or '' - if result is None: result = '' @@ -146,6 +145,7 @@ def _get_dataclass_docstring(py_object): result = '' return result + class _AddDoctestFences(object): """Adds ``` fences around doctest caret blocks >>> that don't have them.""" CARET_BLOCK_RE = re.compile( @@ -528,6 +528,8 @@ def _get_other_member_doc( class_full_name = f'{module}.{class_name}' info = f'Instance of `{class_full_name}`' + if info is not None: + info = signature_lib.strip_obj_addresses(info) parts = [info, description] parts = [item for item in parts if item is not None] diff --git a/tools/tensorflow_docs/api_generator/parser_test.py b/tools/tensorflow_docs/api_generator/parser_test.py index 81aa1d4e11e..d61df5f5950 100644 --- a/tools/tensorflow_docs/api_generator/parser_test.py +++ b/tools/tensorflow_docs/api_generator/parser_test.py @@ -18,8 +18,6 @@ import collections import dataclasses import inspect -import os -import tempfile import textwrap from typing import List, Union @@ -236,8 +234,8 @@ def test_docs_for_class(self): self.assertIs(method_infos['a_method'].py_object, TestClass.a_method) # Make sure that the signature is extracted properly and omits self. - self.assertEqual(['arg='default''], - method_infos['a_method'].signature.arguments) + self.assertEqual('(\n arg='default'\n)', + str(method_infos['a_method'].signature)) self.assertEqual(method_infos['static_method'].decorators, ['staticmethod']) self.assertEqual(method_infos['class_method'].decorators, ['classmethod']) @@ -503,8 +501,8 @@ def test_docs_for_function(self): inspect.getdoc(test_function).split('\n')[0], page_info.doc.brief) # Make sure the extracted signature is good. - self.assertEqual(['unused_arg', 'unused_kwarg='default''], - page_info.signature.arguments) + self.assertEqual('(\n unused_arg, unused_kwarg='default'\n)', + str(page_info.signature)) def test_docs_for_function_with_kwargs(self): index = {'test_function_with_args_kwargs': test_function_with_args_kwargs} @@ -536,8 +534,8 @@ def test_docs_for_function_with_kwargs(self): page_info.doc.brief) # Make sure the extracted signature is good. - self.assertEqual(['unused_arg', '*unused_args', '**unused_kwargs'], - page_info.signature.arguments) + self.assertEqual('(\n unused_arg, *unused_args, **unused_kwargs\n)', + str(page_info.signature)) def test_parse_md_docstring(self): @@ -771,9 +769,9 @@ def test_strips_default_arg_memory_address(self): py_object=ConcreteMutableMapping, parser_config=parser_config) - pop_default_arg = page_info.methods[0].signature.arguments[1] - self.assertNotIn('object at 0x', pop_default_arg) - self.assertIn('<object>', pop_default_arg) + output = str(page_info.methods[0].signature) + self.assertNotIn('object at 0x', output) + self.assertIn('<object object>', output) @parameterized.named_parameters( ('mutable_mapping', 'ConcreteMutableMapping', '__contains__', @@ -930,8 +928,6 @@ class A(): self.assertEqual('Instance of `tf.test.A`', result) - - def testIsClasssAttr(self): result = parser.is_class_attr('test_module.test_function', {'test_module': test_module}) @@ -994,8 +990,6 @@ def test_split_title_blocks(self): '\nSome tensors, with the same type as the input.\n') self.assertLen(returns.items, 2) - - def test_strip_todos(self): input_str = ("""# TODO(blah) blah @@ -1048,11 +1042,11 @@ def test_get_dataclass_docstring(self): @dataclasses.dataclass class MyClass(): - """docstring""" + """Docstring!""" a: int b: float - self.assertEqual(parser._get_raw_docstring(MyClass), 'docstring') + self.assertEqual(parser._get_raw_docstring(MyClass), 'Docstring!') def test_get_dataclass_docstring_no_autogen_docstring(self): @@ -1063,7 +1057,5 @@ class MyClass(): self.assertEmpty(parser._get_raw_docstring(MyClass)) - - if __name__ == '__main__': absltest.main() diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py index 5ae46d443eb..fb99ef12712 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py @@ -13,10 +13,12 @@ # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== +"""Base classes for page construction.""" import abc +import functools import pathlib import textwrap -from typing import Any, Callable, ClassVar, Dict, List, NamedTuple, Optional, Sequence, Tuple, Type +from typing import Any, ClassVar, Dict, List, NamedTuple, Optional, Sequence, Tuple, Type from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_controls @@ -99,6 +101,9 @@ class PageInfo: aliases: A list of full-name for all aliases for this object. doc: A list of objects representing the docstring. These can all be converted to markdown using str(). + search_hints: If true include metadata search hints, else include a + "robots: noindex" + text: The resulting page text. """ DEFAULT_BUILDER_CLASS: ClassVar[Type[PageBuilder]] = TemplatePageBuilder @@ -116,6 +121,8 @@ def __init__( py_object: The object being documented. extra_docs: Extra docs for symbols like public constants(list, tuple, etc) that need to be added to the markdown pages created. + search_hints: If true include metadata search hints, else include a + "robots: noindex" """ self.full_name = full_name self.py_object = py_object @@ -132,9 +139,14 @@ def collect_docs(self, parser_config: config.ParserConfig): def build(self) -> str: """Builds the documentation.""" - cls = self.DEFAULT_BUILDER_CLASS - return cls(self).build() + text = cls(self).build() + self._text = text + return text + + @property + def text(self): + return self._text def __eq__(self, other): if isinstance(other, PageInfo): @@ -268,7 +280,7 @@ def build_other_members(other_members: List[MemberInfo], title: str): def build_signature(name: str, - signature: signature_lib.SignatureComponents, + signature: signature_lib.TfSignature, decorators: Optional[Sequence[str]], type_alias: bool = False) -> str: """Returns a markdown code block containing the function signature. diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py index cb315754385..8367f3d55d0 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py @@ -13,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== +"""Page builder classes for class pages.""" import collections import dataclasses import itertools @@ -465,13 +466,13 @@ class MethodInfo(NamedTuple): py_object: Any doc: parser.DocstringInfo url: str - signature: signature_lib.SignatureComponents + signature: signature_lib.TfSignature decorators: List[str] defined_in: Optional[parser.FileLocation] @classmethod def from_member_info(cls, method_info: base_page.MemberInfo, - signature: signature_lib.SignatureComponents, + signature: signature_lib.TfSignature, decorators: List[str], defined_in: Optional[parser.FileLocation]): """Upgrades a `base_page.MemberInfo` to a `MethodInfo`.""" @@ -598,6 +599,7 @@ def _create_class_doc(doc): def _method_sort(method): + """Create a sort-key tuple for a method based on its name.""" # All private methods will be at the end of the list in an alphabetically # sorted order. All dunder methods will be above private methods and below # public methods. Public methods will be at the top in an alphabetically diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py index 0b073eee466..0185983bad8 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py @@ -60,6 +60,8 @@ def docs_for_object( parser_config: A `config.ParserConfig` object. extra_docs: Extra docs for symbols like public constants(list, tuple, etc) that need to be added to the markdown pages created. + search_hints: If true include metadata search hints, else include a + "robots: noindex". page_builder_classes: An optional dict of `{ObjectType:Type[PageInfo]}` for overriding the default page builder classes. @@ -114,4 +116,6 @@ def docs_for_object( page_info.set_defined_in(parser.get_defined_in(py_object, parser_config)) + page_info.build() + return page_info diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py index c984f6f2db1..6ae93d5a879 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py @@ -13,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== +"""Bage builder classes for type alias pages.""" import textwrap import typing from typing import Any, List, Dict @@ -132,7 +133,6 @@ def collect_docs(self, parser_config) -> None: assert self.signature is None linker = signature_lib.FormatArguments( - type_annotations={}, parser_config=parser_config, func_full_name=self.full_name) diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index ae66e8b052b..8c30d7c230c 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -16,6 +16,7 @@ """Turn Python docstrings into Markdown for TensorFlow documentation.""" import ast +import copy import dataclasses import enum import functools @@ -25,53 +26,85 @@ import textwrap import typing -from typing import Any, Dict, List, Iterable, NamedTuple, Optional, Union +from typing import Any, Callable, Dict, List, Tuple, Type import astor from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import public_api +EMPTY = inspect.Signature.empty + + +def _source_from_ast(node: ast.AST) -> str: + return astor.to_source(node).strip().replace('"""', "'") -class _TypeAnnotationExtractor(ast.NodeVisitor): - """Extracts the type annotations by parsing the AST of a function.""" + +class _BaseDefaultAndAnnotationExtractor(ast.NodeVisitor): + """A base class for extracting annotations and defaults from the AST.""" + _PAREN_NUMBER_RE = re.compile(r'^\((True|False|[0-9.e-]+)\)') def __init__(self): - self.annotation_dict = {} - self.arguments_typehint_exists = False - self.return_typehint_exists = False + self.annotations = {} + self.defaults = {} + self.return_annotation = EMPTY + + def _preprocess_default(self, val: ast.AST) -> str: + text_default_val = ( + _source_from_ast(val).replace('\t', '\\t').replace('\n', '\\n')) + text_default_val = self._PAREN_NUMBER_RE.sub('\\1', text_default_val) + return text_default_val + + +class _CallableDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor + ): + """Extracts the type annotations by parsing the AST of a function.""" def visit_FunctionDef(self, node) -> None: # pylint: disable=invalid-name """Visits the `FunctionDef` node in AST tree and extracts the typehints.""" # Capture the return type annotation. if node.returns: - self.annotation_dict['return'] = astor.to_source( - node.returns).strip().replace('"""', '"') - self.return_typehint_exists = True + self.return_annotation = _source_from_ast(node.returns) # Capture the args type annotation. for arg in node.args.args: if arg.annotation: - self.annotation_dict[arg.arg] = astor.to_source( - arg.annotation).strip().replace('"""', '"') + self.annotations[arg.arg] = _source_from_ast(arg.annotation) self.arguments_typehint_exists = True # Capture the kwarg only args type annotation. for kwarg in node.args.kwonlyargs: if kwarg.annotation: - self.annotation_dict[kwarg.arg] = astor.to_source( - kwarg.annotation).strip().replace('"""', '"') + self.annotations[kwarg.arg] = _source_from_ast(kwarg.annotation) self.arguments_typehint_exists = True + # From https://docs.python.org/3/library/ast.html#ast.arguments: + # `defaults` is a list of default values for arguments that can be passed + # positionally. If there are fewer defaults, they correspond to the last + # n arguments. + + last_n_pos_args = node.args.args[-1 * len(node.args.defaults):] + for arg, default_val in zip(last_n_pos_args, node.args.defaults): + if default_val is not None: + text_default_val = self._preprocess_default(default_val) + self.defaults[arg.arg] = text_default_val + + for kwarg, default_val in zip(node.args.kwonlyargs, node.args.kw_defaults): + if default_val is not None: + text_default_val = self._preprocess_default(default_val) + self.defaults[kwarg.arg] = text_default_val + -class _DataclassTypeAnnotationExtractor(ast.NodeVisitor): +class _DataclassDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor + ): """Extracts the type annotations by parsing the AST of a dataclass.""" def __init__(self): - self.annotation_dict = {} - self.arguments_typehint_exists = False - self.return_typehint_exists = False + super().__init__() + self.annotations = {} + self.defaults = {} + self.return_annotation = EMPTY def visit_ClassDef(self, node) -> None: # pylint: disable=invalid-name # Don't visit all nodes. Only visit top-level AnnAssign nodes so that @@ -82,44 +115,18 @@ def visit_ClassDef(self, node) -> None: # pylint: disable=invalid-name def visit_AnnAssign(self, node) -> None: # pylint: disable=invalid-name """Vists an assignment with a type annotation. Dataclasses is an example.""" - arg = astor.to_source(node.target).strip() - anno = astor.to_source(node.annotation).strip() - self.annotation_dict[arg] = anno - self.arguments_typehint_exists = True + arg = _source_from_ast(node.target) + self.annotations[arg] = _source_from_ast(node.annotation) + if node.value is not None: + self.defaults[arg] = self._preprocess_default(node.value) -class _ASTDefaultValueExtractor(ast.NodeVisitor): - """Extracts the default values by parsing the AST of a function.""" - _PAREN_NUMBER_RE = re.compile(r'^\((True|False|[0-9.e-]+)\)') +_OBJECT_MEMORY_ADDRESS_RE = re.compile(r'<(?P.+?) at 0x[\da-f]+>') - def __init__(self): - self.ast_args_defaults = {} - self.ast_kw_only_defaults = {} - def _preprocess(self, val) -> str: - text_default_val = astor.to_source(val).strip().replace( - '\t', '\\t').replace('\n', '\\n').replace('"""', "'") - text_default_val = self._PAREN_NUMBER_RE.sub('\\1', text_default_val) - return text_default_val - - def visit_FunctionDef(self, node) -> None: # pylint: disable=invalid-name - """Visits the `FunctionDef` node and extracts the default values.""" - - # From https://docs.python.org/3/library/ast.html#ast.arguments: - # `defaults` is a list of default values for arguments that can be passed - # positionally. If there are fewer defaults, they correspond to the last - # n arguments. - last_n_pos_args = node.args.args[-1 * len(node.args.defaults):] - for arg, default_val in zip(last_n_pos_args, node.args.defaults): - if default_val is not None: - text_default_val = self._preprocess(default_val) - self.ast_args_defaults[arg.arg] = text_default_val - - for kwarg, default_val in zip(node.args.kwonlyargs, node.args.kw_defaults): - if default_val is not None: - text_default_val = self._preprocess(default_val) - self.ast_kw_only_defaults[kwarg.arg] = text_default_val +def strip_obj_addresses(text): + return _OBJECT_MEMORY_ADDRESS_RE.sub(r'<\g>', text) class FormatArguments(object): @@ -133,8 +140,6 @@ class FormatArguments(object): 'saver_pb2.SaverDef': 'tf.train.SaverDef', } - _OBJECT_MEMORY_ADDRESS_RE = re.compile(r'<(?P.+) object at 0x[\da-f]+>') - # A regular expression capturing a python identifier. _IDENTIFIER_RE = r'[a-zA-Z_]\w*' @@ -157,11 +162,9 @@ class FormatArguments(object): def __init__( self, - type_annotations: Dict[str, str], parser_config: config.ParserConfig, func_full_name: str, ) -> None: - self._type_annotations = type_annotations self._reverse_index = parser_config.reverse_index self._reference_resolver = parser_config.reference_resolver # func_full_name is used to calculate the relative path. @@ -216,7 +219,6 @@ def _get_non_builtin_ast_types(self, ast_typehint: str) -> List[str]: Returns: List of non-builtin ast types. """ - non_builtin_ast_types = [] for single_type, _ in self._INDIVIDUAL_TYPES_RE.findall(ast_typehint): if (not single_type or single_type in self._TYPING or @@ -298,8 +300,9 @@ def _replace_internal_names(self, default_text: str) -> str: return public_name + default_text[len(internal_name):] return default_text - def format_return(self, return_anno: Any) -> str: - return self.preprocess(self._type_annotations['return'], return_anno) + def format_return(self, return_anno: Tuple[Any, str]) -> str: + value, source = return_anno + return self.preprocess(source, value) def format_args(self, args: List[inspect.Parameter]) -> List[str]: """Creates a text representation of the args in a method/function. @@ -314,23 +317,24 @@ def format_args(self, args: List[inspect.Parameter]) -> List[str]: args_text_repr = [] for arg in args: - arg_name = arg.name - if arg_name in self._type_annotations: - typeanno = self.preprocess(self._type_annotations[arg_name], - arg.annotation) - args_text_repr.append(f'{arg_name}: {typeanno}') + typeanno = None + if arg.annotation is not EMPTY: + value, source = arg.annotation + if source is not None: + typeanno = self.preprocess(source, value) + + if typeanno: + args_text_repr.append(f'{arg.name}: {typeanno}') else: - args_text_repr.append(f'{arg_name}') + args_text_repr.append(f'{arg.name}') return args_text_repr - def format_kwargs(self, kwargs: List[inspect.Parameter], - ast_defaults: Dict[str, str]) -> List[str]: + def format_kwargs(self, kwargs: List[inspect.Parameter]) -> List[str]: """Creates a text representation of the kwargs in a method/function. Args: kwargs: List of kwargs to format. - ast_defaults: Default values extracted from the function's AST tree. Returns: Formatted kwargs with type annotations if they exist. @@ -339,60 +343,143 @@ def format_kwargs(self, kwargs: List[inspect.Parameter], kwargs_text_repr = [] for kwarg in kwargs: - kname = kwarg.name - ast_default = ast_defaults.get(kname) - default_val = kwarg.default - - if id(default_val) in self._reverse_index: - default_text = self._reverse_index[id(default_val)] - elif ast_default is not None: - default_text = ast_default - if default_text != repr(default_val): - default_text = self._replace_internal_names(default_text) - # Kwarg without default value. - elif default_val is kwarg.empty: - kwargs_text_repr.extend(self.format_args([kwarg])) - continue - else: - # Strip object memory addresses to avoid unnecessary doc churn. - default_text = self._OBJECT_MEMORY_ADDRESS_RE.sub( - r'<\g>', repr(default_val)) - default_text = html.escape(str(default_text)) + default_text = None + if kwarg.default is not EMPTY: + default_val, default_source = kwarg.default + + if id(default_val) in self._reverse_index: + default_text = self._reverse_index[id(default_val)] + elif default_source is not None: + default_text = default_source + if default_text != repr(default_val): + default_text = self._replace_internal_names(default_text) + # Kwarg without default value. + elif default_val is EMPTY: + kwargs_text_repr.extend(self.format_args([kwarg])) + continue + else: + # Strip object memory addresses to avoid unnecessary doc churn. + default_text = strip_obj_addresses(repr(default_val)) + default_text = html.escape(str(default_text)) # Format the kwargs to add the type annotation and default values. - if kname in self._type_annotations: - typeanno = self.preprocess(self._type_annotations[kname], - kwarg.annotation) - kwargs_text_repr.append(f'{kname}: {typeanno} = {default_text}') + typeanno = None + if kwarg.annotation is not EMPTY: + anno_value, anno_source = kwarg.annotation + if anno_source is not None: + typeanno = self.preprocess(anno_source, anno_value) + + if typeanno is not None and default_text is not None: + kwargs_text_repr.append(f'{kwarg.name}: {typeanno} = {default_text}') + elif default_text is not None: + kwargs_text_repr.append(f'{kwarg.name}={default_text}') + elif typeanno is not None: + kwargs_text_repr.append(f'{kwarg.name}: {typeanno}') else: - kwargs_text_repr.append(f'{kname}={default_text}') + kwargs_text_repr.append(kwarg.name) return kwargs_text_repr -class SignatureComponents(NamedTuple): - """Contains the components that make up the signature of a function/method.""" +class TfSignature(inspect.Signature): + """A custom version of `inspect.Signature`.""" + + def __init__(self, parameters, *, return_annotation, parser_config, + func_full_name): + super().__init__(parameters, return_annotation=return_annotation) + self.parser_config = parser_config + self.func_full_name = func_full_name - arguments: List[str] - arguments_typehint_exists: bool - return_typehint_exists: bool - return_type: Optional[str] = None + def __copy__(self): + return TfSignature( + list(self.parameters.values()), + return_annotation=self.return_annotation, + parser_config=self.parser_config, + func_full_name=self.func_full_name) + + def __deepcopy__(self, memo): + return TfSignature( + copy.deepcopy(list(self.parameters.values()), memo), + return_annotation=copy.deepcopy(self.return_annotation, memo), + parser_config=copy.deepcopy(self.parser_config, memo), + func_full_name=copy.deepcopy(self.func_full_name, memo)) def __str__(self): + # separate the args by type + pos_only_args = [] + args = [] + kwargs = [] + only_kwargs = [] + varargs = None + varkwargs = None + + for index, param in enumerate(self.parameters.values()): + kind = param.kind + default = param.default + + if kind == param.POSITIONAL_ONLY: + pos_only_args.append(param) + elif default is EMPTY and kind == param.POSITIONAL_OR_KEYWORD: + args.append(param) + elif default is not EMPTY and kind == param.POSITIONAL_OR_KEYWORD: + kwargs.append(param) + elif kind == param.VAR_POSITIONAL: + varargs = (index, param) + elif kind == param.KEYWORD_ONLY: + only_kwargs.append(param) + elif kind == param.VAR_KEYWORD: + varkwargs = param + + # Build the text representation. + all_args_list = [] + + formatter = FormatArguments( + parser_config=self.parser_config, func_full_name=self.func_full_name) + + if pos_only_args: + all_args_list.extend(formatter.format_args(pos_only_args)) + all_args_list.append('/') + + if args: + all_args_list.extend(formatter.format_args(args)) + + if kwargs: + all_args_list.extend(formatter.format_kwargs(kwargs)) + + if only_kwargs: + if varargs is None: + all_args_list.append('*') + all_args_list.extend(formatter.format_kwargs(only_kwargs)) + + if varargs is not None: + all_args_list.insert(varargs[0], '*' + varargs[1].name) + + if varkwargs is not None: + all_args_list.append('**' + varkwargs.name) + + return_annotation_text = '' + if self.return_annotation is not EMPTY: + if EMPTY not in self.return_annotation: + return_annotation_text = formatter.format_return(self.return_annotation) + arguments_signature = '' - if self.arguments: - str_signature = ',\n'.join(self.arguments) + has_any_annotations = any( + v.annotation is not EMPTY for v in self.parameters.values()) + if all_args_list: + str_signature = ',\n'.join(all_args_list) # If there is no type annotation on arguments, then wrap the entire # signature to width 80. - if not self.arguments_typehint_exists: + if not has_any_annotations: str_signature = textwrap.fill(str_signature, width=80) + arguments_signature = '\n' + textwrap.indent( str_signature, prefix=' ') + '\n' full_signature = f'({arguments_signature})' - if self.return_typehint_exists: - full_signature += f' -> {self.return_type}' - + if return_annotation_text: + full_signature = f'({arguments_signature}) -> {return_annotation_text}' + else: + full_signature = f'({arguments_signature})' return full_signature @@ -404,7 +491,7 @@ class FuncType(enum.Enum): def get_method_type(method, name, is_dataclass): - + """Determine the type of callable.""" if isinstance(method, classmethod): func_type = FuncType.CLASSMETHOD elif name == '__new__': @@ -430,7 +517,7 @@ def generate_signature( parser_config: config.ParserConfig, func_full_name: str, func_type: FuncType = FuncType.FUNCTION, -) -> SignatureComponents: +) -> TfSignature: """Given a function, returns a list of strings representing its args. This function uses `__name__` for callables if it is available. This can lead @@ -441,8 +528,8 @@ def generate_signature( Args: func: A function, method, or functools.partial to extract the signature for. - parser_config: `config.ParserConfig` for the method/function whose signature is - generated. + parser_config: `config.ParserConfig` for the method/function whose signature + is generated. func_full_name: The full name of a function whose signature is generated. func_type: Type of the current `func`. This is required because there isn't a clear distinction between function and method being passed to @@ -453,117 +540,115 @@ def generate_signature( Returns: A `SignatureComponents` NamedTuple. """ - - all_args_list = [] - try: sig = inspect.signature(func) - sig_values = sig.parameters.values() - return_anno = sig.return_annotation except (ValueError, TypeError): - sig_values = [] - return_anno = None + sig = inspect.signature(lambda: None) + + params = list(sig.parameters.values()) + + # Drop `self` + if params: + first = params[0] + if first.kind != first.VAR_POSITIONAL: + if func_type == FuncType.METHOD: + # - Skip the first arg for regular methods. + # - Some wrapper methods forget `self` and just use `(*args, **kwargs)`. + # That's still valid, don't drop `*args`. + # - For classmethods the `cls` arg already bound here (it's not in + # `params`). + # - For regular functions (or staticmethods) you never need to skip. + params.pop(0) + + sig = sig.replace(parameters=params) if dataclasses.is_dataclass(func): - type_annotation_visitor = _DataclassTypeAnnotationExtractor() + sig = sig.replace(return_annotation=EMPTY) + extract_fn = _extract_dataclass_defaults_and_annotations else: - type_annotation_visitor = _TypeAnnotationExtractor() + extract_fn = _extract_callable_defaults_and_annotations - ast_defaults_visitor = _ASTDefaultValueExtractor() + (annotation_source_dict, defaults_source_dict, + return_annotation_source) = extract_fn(func) - try: - func_source = textwrap.dedent(inspect.getsource(func)) + # Replace everything with either `EMPTY` or (value, source) pairs. + new_params = [] + for name, param in sig.parameters.items(): + default = param.default + if default is not EMPTY: + default = (default, defaults_source_dict.get(name, None)) + + annotation = param.annotation + if annotation is not EMPTY: + annotation = (annotation, annotation_source_dict.get(name, None)) + + param = param.replace(default=default, annotation=annotation) + new_params.append(param) + + return_annotation = sig.return_annotation + if return_annotation is not EMPTY: + return_annotation = (return_annotation, return_annotation_source) + + sig = TfSignature( + parameters=new_params, + return_annotation=return_annotation, + parser_config=parser_config, + func_full_name=func_full_name) + + return sig + + +AnnotsDefaultsReturns = Tuple[Dict[str, str], Dict[str, str], Any] + + +def _extract_dataclass_defaults_and_annotations( + func: Type[object]) -> AnnotsDefaultsReturns: + """Extract ast defaults and annotations form a dataclass.""" + stack = [c for c in func.__mro__ if dataclasses.is_dataclass(c)] + + annotation_source_dict = {} + defaults_source_dict = {} + return_annotation_source = EMPTY + + # Iterate over the classes in reverse order so precedence works. + for cls in reversed(stack): + func_source = textwrap.dedent(inspect.getsource(cls)) func_ast = ast.parse(func_source) # Extract the type annotation from the parsed ast. - type_annotation_visitor.visit(func_ast) - ast_defaults_visitor.visit(func_ast) - except Exception: # pylint: disable=broad-except - # A wide-variety of errors can be thrown here. - pass + ast_visitor = _DataclassDefaultAndAnnotationExtractor() + ast_visitor.visit(func_ast) - type_annotations = type_annotation_visitor.annotation_dict - arguments_typehint_exists = type_annotation_visitor.arguments_typehint_exists - return_typehint_exists = type_annotation_visitor.return_typehint_exists + annotation_source_dict.update(ast_visitor.annotations) + defaults_source_dict.update(ast_visitor.defaults) - ############################################################################# - # Process the information about the func. - ############################################################################# + return annotation_source_dict, defaults_source_dict, return_annotation_source - pos_only_args = [] - args = [] - kwargs = [] - only_kwargs = [] - varargs = None - varkwargs = None - for index, param in enumerate(sig_values): - kind = param.kind - default = param.default +def _extract_callable_defaults_and_annotations( + func: Callable[..., Any]) -> AnnotsDefaultsReturns: + """Extract ast defaults and annotations form a standard callable.""" - if (index == 0 and func_type == FuncType.METHOD and - kind != param.VAR_POSITIONAL): - # - Skip the first arg for regular methods. - # - Some wrapper methods forget `self` and just use `(*args, **kwargs)`. - # That's still valid, don't drop `*args`. - # - For classmethods the `cls` arg already bound here (it's not in - # `sig_values`). - # - For regular functions (or staticmethods) you never need to skip. - continue - elif kind == param.POSITIONAL_ONLY: - pos_only_args.append(param) - elif default is param.empty and kind == param.POSITIONAL_OR_KEYWORD: - args.append(param) - elif default is not param.empty and kind == param.POSITIONAL_OR_KEYWORD: - kwargs.append(param) - elif kind == param.VAR_POSITIONAL: - varargs = (index, param) - elif kind == param.KEYWORD_ONLY: - only_kwargs.append(param) - elif kind == param.VAR_KEYWORD: - varkwargs = param - - ############################################################################# - # Build the text representation of Args and Kwargs. - ############################################################################# - - formatter = FormatArguments( - type_annotations, parser_config, func_full_name=func_full_name) - - if pos_only_args: - all_args_list.extend(formatter.format_args(pos_only_args)) - all_args_list.append('/') - - if args: - all_args_list.extend(formatter.format_args(args)) - - if kwargs: - all_args_list.extend( - formatter.format_kwargs(kwargs, ast_defaults_visitor.ast_args_defaults)) - - if only_kwargs: - if varargs is None: - all_args_list.append('*') - all_args_list.extend( - formatter.format_kwargs(only_kwargs, - ast_defaults_visitor.ast_kw_only_defaults)) - - if varargs is not None: - all_args_list.insert(varargs[0], '*' + varargs[1].name) - - if varkwargs is not None: - all_args_list.append('**' + varkwargs.name) - - if return_anno and return_anno is not sig.empty and type_annotations.get( - 'return', None): - return_type = formatter.format_return(return_anno) + ast_visitor = _CallableDefaultAndAnnotationExtractor() + + annotation_source_dict = {} + defaults_source_dict = {} + return_annotation_source = EMPTY + + try: + func_source = textwrap.dedent(inspect.getsource(func)) + func_ast = ast.parse(func_source) + # Extract the type annotation from the parsed ast. + ast_visitor.visit(func_ast) + except Exception: # pylint: disable=broad-except + # A wide-variety of errors can be thrown here. + pass else: - return_type = 'None' + annotation_source_dict = ast_visitor.annotations + defaults_source_dict = ast_visitor.defaults + return_annotation_source = ast_visitor.return_annotation + + return annotation_source_dict, defaults_source_dict, return_annotation_source - return SignatureComponents( - arguments=all_args_list, - arguments_typehint_exists=arguments_typehint_exists, - return_typehint_exists=return_typehint_exists, - return_type=return_type) def extract_decorators(func: Any) -> List[str]: """Extracts the decorators on top of functions/methods. @@ -582,7 +667,7 @@ def __init__(self): def visit_FunctionDef(self, node): # pylint: disable=invalid-name for dec in node.decorator_list: - self.decorator_list.append(astor.to_source(dec).strip()) + self.decorator_list.append(_source_from_ast(dec)) visitor = ASTDecoratorExtractor() diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index b843ba7776b..3e89d53941f 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -27,8 +27,8 @@ from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib from tensorflow_docs.api_generator import signature -from tensorflow_docs.api_generator.pretty_docs import type_alias_page from tensorflow_docs.api_generator.pretty_docs import class_page +from tensorflow_docs.api_generator.pretty_docs import type_alias_page @dataclasses.dataclass @@ -81,7 +81,7 @@ def example_fun(arg=self.known_object): # pylint: disable=unused-argument parser_config=self.parser_config, func_full_name='', func_type=signature.FuncType.FUNCTION) - self.assertEqual(sig.arguments, ['arg=location.of.object.in.api']) + self.assertEqual('(\n arg=location.of.object.in.api\n)', str(sig)) def test_literals(self): @@ -102,21 +102,23 @@ def example_fun( parser_config=self.parser_config, func_full_name='', func_type=signature.FuncType.FUNCTION) - self.assertEqual(sig.arguments, [ - 'self', 'cls', 'a=5', 'b=5.0', 'c=None', 'd=True', - 'e='hello'', 'f=(1, (2, 3))' - ]) + + expected = textwrap.dedent("""\ + ( + self, cls, a=5, b=5.0, c=None, d=True, e='hello', f=(1, (2, 3)) + )""") + self.assertEqual(expected, str(sig)) def test_dotted_name(self): # pylint: disable=g-bad-name - class a(object): + class a: - class b(object): + class b: - class c(object): + class c: - class d(object): + class d: def __init__(self, *args): pass @@ -133,9 +135,9 @@ def example_fun(arg1=a.b.c.d, arg2=a.b.c.d(1, 2), arg3=e['f']): # pylint: disab parser_config=self.parser_config, func_full_name='', func_type=signature.FuncType.FUNCTION) - self.assertEqual( - sig.arguments, - ['arg1=a.b.c.d', 'arg2=a.b.c.d(1, 2)', 'arg3=e['f']']) + expected = ('(\n arg1=a.b.c.d, arg2=a.b.c.d(1, 2), ' + 'arg3=e['f']\n)') + self.assertEqual(expected, str(sig)) def test_compulsory_kwargs_without_defaults(self): @@ -147,13 +149,14 @@ def example_fun(x, z, a=True, b='test', *, c, y=None, d, **kwargs) -> bool: # p parser_config=self.parser_config, func_full_name='', func_type=signature.FuncType.FUNCTION) - self.assertEqual(sig.arguments, [ - 'x', 'z', 'a=True', 'b='test'', '*', 'c', 'y=None', 'd', - '**kwargs' - ]) - self.assertEqual(sig.return_type, 'bool') - self.assertEqual(sig.arguments_typehint_exists, False) - self.assertEqual(sig.return_typehint_exists, True) + self.assertEqual( + list(sig.parameters.keys()), + ['x', 'z', 'a', 'b', 'c', 'y', 'd', 'kwargs']) + expected = textwrap.dedent("""\ + ( + x, z, a=True, b='test', *, c, y=None, d, **kwargs + ) -> bool""") + self.assertEqual(expected, str(sig)) def test_compulsory_kwargs_without_defaults_with_args(self): @@ -165,12 +168,13 @@ def example_fun(x, z, cls, *args, a=True, b='test', y=None, c, **kwargs): # pyl parser_config=self.parser_config, func_full_name='', func_type=signature.FuncType.FUNCTION) - self.assertEqual(sig.arguments, [ - 'x', 'z', 'cls', '*args', 'a=True', 'b='test'', 'y=None', 'c', - '**kwargs' - ]) - self.assertEqual(sig.arguments_typehint_exists, False) - self.assertEqual(sig.return_typehint_exists, False) + self.assertEqual( + list(sig.parameters.keys()), + ['x', 'z', 'cls', 'args', 'a', 'b', 'y', 'c', 'kwargs']) + self.assertEqual( + str(sig), + '(\n x, z, cls, *args, a=True, b='test', y=None, c, **kwargs\n)' + ) def test_type_annotations(self): # pylint: disable=unused-argument @@ -196,19 +200,18 @@ def example_fun(self, func_full_name='', func_type=signature.FuncType.METHOD, ) - self.assertEqual(sig.arguments, [ - 'x: List[str]', - 'z: int', - 'a: Union[List[str], str, int] = None', - 'b: str = 'test'', - '*', - 'y: bool = False', - 'c: Callable[..., int]', - '**kwargs', - ]) - self.assertEqual(sig.return_type, 'None') - self.assertEqual(sig.arguments_typehint_exists, True) - self.assertEqual(sig.return_typehint_exists, True) + expected = textwrap.dedent("""\ + ( + x: List[str], + z: int, + a: Union[List[str], str, int] = None, + b: str = 'test', + *, + y: bool = False, + c: Callable[..., int], + **kwargs + ) -> None""") + self.assertEqual(expected, str(sig)) def test_dataclasses_type_annotations(self): @@ -218,16 +221,16 @@ def test_dataclasses_type_annotations(self): func_full_name='', func_type=signature.FuncType.FUNCTION) - self.assertEqual(sig.arguments, [ - 'x: List[str]', - 'z: int', - 'c: List[int] = <factory>', - 'a: Union[List[str], str, int] = None', - 'b: str = 'test'', - 'y: bool = False', - ]) - self.assertEqual(sig.return_type, 'None') - self.assertEqual(sig.arguments_typehint_exists, True) + expected = textwrap.dedent("""\ + ( + x: List[str], + z: int, + c: List[int] = dataclasses.field(default_factory=list), + a: Union[List[str], str, int] = None, + b: str = 'test', + y: bool = False + )""") + self.assertEqual(expected, str(sig)) @parameterized.named_parameters( ('deep_objects', Union[Dict[str, Dict[bool, signature.extract_decorators]], @@ -367,6 +370,73 @@ def __init__(self, x: Optional[Union[int, str]]): self.assertEqual('(\n x: Optional[Union[int, str]]\n)', str(info.methods[0].signature)) + def test_dataclass_default_uses_ast_repr(self): + + @dataclasses.dataclass + class MyClass: + a: float = 1 / 9 + + sig = signature.generate_signature( + MyClass, func_full_name='MyClass', parser_config=self.parser_config) + + expected = '(\n a: float = (1 / 9)\n)' + self.assertEqual(expected, str(sig)) + + def test_dataclass_inheritance_sees_parent(self): + const = 3.14159 + + @dataclasses.dataclass + class Parent: + a: int = 60 * 60 + b: float = 1 / 9 + + @dataclasses.dataclass + class Child(Parent): + b: float = 2 / 9 + c: float = const + + sig = signature.generate_signature( + Child, func_full_name='Child', parser_config=self.parser_config) + expected = textwrap.dedent("""\ + ( + a: int = (60 * 60), + b: float = (2 / 9), + c: float = const + )""") + self.assertEqual(expected, str(sig)) + + def test_vararg_before_kwargonly_consistent_order(self): + + def my_fun(*args, a=1, **kwargs): # pylint: disable=unused-argument + pass + + sig = signature.generate_signature( + my_fun, func_full_name='my_fun', parser_config=self.parser_config) + expected = '(\n *args, a=1, **kwargs\n)' + self.assertEqual(expected, str(sig)) + + def test_class_vararg_before_kwargonly_consistent_order(self): + + class MyClass: + + def __init__(*args, a=1, **kwargs): # pylint: disable=no-method-argument + pass + + sig = signature.generate_signature( + MyClass, func_full_name='MyClass', parser_config=self.parser_config) + expected = '(\n *args, a=1, **kwargs\n)' + self.assertEqual(expected, str(sig)) + + def test_strip_address(self): + + class What: + pass + + w = What() + + expected = ('<__main__.TestGenerateSignature.test_strip_address.' + '.What object>') + self.assertEqual(expected, signature.strip_obj_addresses(str(w))) if __name__ == '__main__': absltest.main() From 93899e9ff830ec773b25ae5da5d90418fb030589 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Sun, 20 Mar 2022 20:06:37 -0700 Subject: [PATCH 025/872] Remove the TfSignature func_full_name arg -> it's unused. Also split out `_tfrepr` from `_get_other_member_doc`. Pure refactor-> no output changes. PiperOrigin-RevId: 436100334 --- tools/tensorflow_docs/api_generator/parser.py | 15 +++++++-- .../api_generator/pretty_docs/base_page.py | 2 ++ .../api_generator/pretty_docs/class_page.py | 2 +- .../pretty_docs/function_page.py | 1 - .../pretty_docs/type_alias_page.py | 4 +-- .../api_generator/signature.py | 33 ++++++++----------- .../api_generator/signature_test.py | 17 +++------- 7 files changed, 34 insertions(+), 40 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py index 67e070f0b72..e147ae25ee4 100644 --- a/tools/tensorflow_docs/api_generator/parser.py +++ b/tools/tensorflow_docs/api_generator/parser.py @@ -492,7 +492,18 @@ def _get_other_member_doc( if description is None and extra_docs is not None: description = extra_docs.get(id(obj), None) + value_repr = _tfrepr(obj, parser_config) + + parts = [value_repr, description] + parts = [item for item in parts if item is not None] + + return '\n\n'.join(parts) + + +def _tfrepr(obj, parser_config): + """Convert an object to a string for display.""" info = None + if isinstance(obj, dict): # pprint.pformat (next block) doesn't sort dicts until python 3.8 items = [ @@ -530,10 +541,8 @@ def _get_other_member_doc( if info is not None: info = signature_lib.strip_obj_addresses(info) - parts = [info, description] - parts = [item for item in parts if item is not None] - return '\n\n'.join(parts) + return info def parse_md_docstring( diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py index fb99ef12712..fd42d9a575b 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py @@ -132,6 +132,7 @@ def __init__( self._defined_in = None self._aliases = None self._doc = None + self._text = None def collect_docs(self, parser_config: config.ParserConfig): """Collects additional information from the `config.ParserConfig`.""" @@ -198,6 +199,7 @@ def set_doc(self, doc: parser.DocstringInfo): self._doc = doc + class MemberInfo(NamedTuple): """Describes an attribute of a class or module.""" short_name: str diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py index 8367f3d55d0..69cac531d07 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py @@ -276,7 +276,7 @@ def _add_method( member_info.short_name, is_dataclass) signature = signature_lib.generate_signature( - py_obj, parser_config, member_info.full_name, func_type=func_type) + py_obj, parser_config, func_type=func_type) decorators = signature_lib.extract_decorators(member_info.py_object) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py index a44ae8768b1..2af6443d975 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py @@ -81,7 +81,6 @@ def collect_docs(self, parser_config): self._signature = signature_lib.generate_signature( self.py_object, parser_config, - self.full_name, func_type=signature_lib.FuncType.FUNCTION, ) self._decorators = signature_lib.extract_decorators(self.py_object) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py index 6ae93d5a879..876fb8fb112 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py @@ -132,9 +132,7 @@ def collect_docs(self, parser_config) -> None: """ assert self.signature is None - linker = signature_lib.FormatArguments( - parser_config=parser_config, - func_full_name=self.full_name) + linker = signature_lib.FormatArguments(parser_config=parser_config) sig_args = [] if self.py_object.__origin__: diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index 8c30d7c230c..8782f5b7b8d 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -163,15 +163,9 @@ class FormatArguments(object): def __init__( self, parser_config: config.ParserConfig, - func_full_name: str, ) -> None: self._reverse_index = parser_config.reverse_index self._reference_resolver = parser_config.reference_resolver - # func_full_name is used to calculate the relative path. - self._func_full_name = func_full_name - - self._is_fragment = self._reference_resolver._is_fragment.get( - self._func_full_name, None) def get_link(self, obj_full_name: str) -> str: return self._reference_resolver.python_link( @@ -384,25 +378,30 @@ def format_kwargs(self, kwargs: List[inspect.Parameter]) -> List[str]: class TfSignature(inspect.Signature): """A custom version of `inspect.Signature`.""" - def __init__(self, parameters, *, return_annotation, parser_config, - func_full_name): + def __init__(self, parameters, *, return_annotation, parser_config): super().__init__(parameters, return_annotation=return_annotation) self.parser_config = parser_config - self.func_full_name = func_full_name + + def replace(self, **kwargs): + attrs = { + 'parameters': self.parameters, + 'return_annotation': self.return_annotation, + 'parser_config': self.parser_config, + } + attrs.update(kwargs) + return type(self)(**attrs) def __copy__(self): return TfSignature( list(self.parameters.values()), return_annotation=self.return_annotation, - parser_config=self.parser_config, - func_full_name=self.func_full_name) + parser_config=self.parser_config) def __deepcopy__(self, memo): return TfSignature( copy.deepcopy(list(self.parameters.values()), memo), return_annotation=copy.deepcopy(self.return_annotation, memo), - parser_config=copy.deepcopy(self.parser_config, memo), - func_full_name=copy.deepcopy(self.func_full_name, memo)) + parser_config=copy.deepcopy(self.parser_config, memo)) def __str__(self): # separate the args by type @@ -433,8 +432,7 @@ def __str__(self): # Build the text representation. all_args_list = [] - formatter = FormatArguments( - parser_config=self.parser_config, func_full_name=self.func_full_name) + formatter = FormatArguments(parser_config=self.parser_config) if pos_only_args: all_args_list.extend(formatter.format_args(pos_only_args)) @@ -515,7 +513,6 @@ def get_method_type(method, name, is_dataclass): def generate_signature( func: Any, parser_config: config.ParserConfig, - func_full_name: str, func_type: FuncType = FuncType.FUNCTION, ) -> TfSignature: """Given a function, returns a list of strings representing its args. @@ -530,7 +527,6 @@ def generate_signature( func: A function, method, or functools.partial to extract the signature for. parser_config: `config.ParserConfig` for the method/function whose signature is generated. - func_full_name: The full name of a function whose signature is generated. func_type: Type of the current `func`. This is required because there isn't a clear distinction between function and method being passed to `generate_signature`. Sometimes methods are detected as function by @@ -592,8 +588,7 @@ def generate_signature( sig = TfSignature( parameters=new_params, return_annotation=return_annotation, - parser_config=parser_config, - func_full_name=func_full_name) + parser_config=parser_config) return sig diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index 3e89d53941f..a8b3296293a 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -79,7 +79,6 @@ def example_fun(arg=self.known_object): # pylint: disable=unused-argument sig = signature.generate_signature( example_fun, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.FUNCTION) self.assertEqual('(\n arg=location.of.object.in.api\n)', str(sig)) @@ -100,7 +99,6 @@ def example_fun( sig = signature.generate_signature( example_fun, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.FUNCTION) expected = textwrap.dedent("""\ @@ -133,7 +131,6 @@ def example_fun(arg1=a.b.c.d, arg2=a.b.c.d(1, 2), arg3=e['f']): # pylint: disab sig = signature.generate_signature( example_fun, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.FUNCTION) expected = ('(\n arg1=a.b.c.d, arg2=a.b.c.d(1, 2), ' 'arg3=e['f']\n)') @@ -147,7 +144,6 @@ def example_fun(x, z, a=True, b='test', *, c, y=None, d, **kwargs) -> bool: # p sig = signature.generate_signature( example_fun, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.FUNCTION) self.assertEqual( list(sig.parameters.keys()), @@ -166,7 +162,6 @@ def example_fun(x, z, cls, *args, a=True, b='test', y=None, c, **kwargs): # pyl sig = signature.generate_signature( example_fun, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.FUNCTION) self.assertEqual( list(sig.parameters.keys()), @@ -197,7 +192,6 @@ def example_fun(self, sig = signature.generate_signature( TestMethodSig.example_fun, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.METHOD, ) expected = textwrap.dedent("""\ @@ -218,7 +212,6 @@ def test_dataclasses_type_annotations(self): sig = signature.generate_signature( ExampleDataclass, parser_config=self.parser_config, - func_full_name='', func_type=signature.FuncType.FUNCTION) expected = textwrap.dedent("""\ @@ -377,7 +370,7 @@ class MyClass: a: float = 1 / 9 sig = signature.generate_signature( - MyClass, func_full_name='MyClass', parser_config=self.parser_config) + MyClass, parser_config=self.parser_config) expected = '(\n a: float = (1 / 9)\n)' self.assertEqual(expected, str(sig)) @@ -395,8 +388,7 @@ class Child(Parent): b: float = 2 / 9 c: float = const - sig = signature.generate_signature( - Child, func_full_name='Child', parser_config=self.parser_config) + sig = signature.generate_signature(Child, parser_config=self.parser_config) expected = textwrap.dedent("""\ ( a: int = (60 * 60), @@ -410,8 +402,7 @@ def test_vararg_before_kwargonly_consistent_order(self): def my_fun(*args, a=1, **kwargs): # pylint: disable=unused-argument pass - sig = signature.generate_signature( - my_fun, func_full_name='my_fun', parser_config=self.parser_config) + sig = signature.generate_signature(my_fun, parser_config=self.parser_config) expected = '(\n *args, a=1, **kwargs\n)' self.assertEqual(expected, str(sig)) @@ -423,7 +414,7 @@ def __init__(*args, a=1, **kwargs): # pylint: disable=no-method-argument pass sig = signature.generate_signature( - MyClass, func_full_name='MyClass', parser_config=self.parser_config) + MyClass, parser_config=self.parser_config) expected = '(\n *args, a=1, **kwargs\n)' self.assertEqual(expected, str(sig)) From 2b73dc4ece856e1e1b1b9a68e64df43e2534abb1 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Mon, 21 Mar 2022 05:39:54 -0700 Subject: [PATCH 026/872] Switch between one-line and one-arg-per-line based on length. If it doesn't fit on one line, put one arg per line. It was "if there are type annotations put one arg per line", but I think this makes more sense. PiperOrigin-RevId: 436180555 --- tools/tensorflow_docs/api_generator/signature.py | 5 ++--- tools/tensorflow_docs/api_generator/signature_test.py | 6 ++---- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index 8782f5b7b8d..0cc91b59b7f 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -465,9 +465,8 @@ def __str__(self): v.annotation is not EMPTY for v in self.parameters.values()) if all_args_list: str_signature = ',\n'.join(all_args_list) - # If there is no type annotation on arguments, then wrap the entire - # signature to width 80. - if not has_any_annotations: + # If it fits on one line flatten it. + if len(str_signature) + 4 < 80: str_signature = textwrap.fill(str_signature, width=80) arguments_signature = '\n' + textwrap.indent( diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index a8b3296293a..f52521bc4c2 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -345,7 +345,7 @@ class Cls6: b: Optional[str] info = self._setup_class_info(Cls6, '__init__') - self.assertEqual('(\n a: Optional[int],\n b: Optional[str]\n)', + self.assertEqual('(\n a: Optional[int], b: Optional[str]\n)', str(info.methods[0].signature)) def test_signature_dataclass_custom_init(self): @@ -391,9 +391,7 @@ class Child(Parent): sig = signature.generate_signature(Child, parser_config=self.parser_config) expected = textwrap.dedent("""\ ( - a: int = (60 * 60), - b: float = (2 / 9), - c: float = const + a: int = (60 * 60), b: float = (2 / 9), c: float = const )""") self.assertEqual(expected, str(sig)) From 814cdac2da4e076b2180a2b2ccc2c6d28033452b Mon Sep 17 00:00:00 2001 From: kristenrq <58997898+kristenrq@users.noreply.github.com> Date: Mon, 21 Mar 2022 09:02:17 -0700 Subject: [PATCH 027/872] Update _toc.yaml --- site/en/community/_toc.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/site/en/community/_toc.yaml b/site/en/community/_toc.yaml index 074ecf72531..1a81d38cb01 100644 --- a/site/en/community/_toc.yaml +++ b/site/en/community/_toc.yaml @@ -34,5 +34,7 @@ toc: - heading: "Community" - title: "Contribute to the community" path: /community/contribute/community + - title: "Contribute to SIGs" + path: /community/contribute/sigs - title: "RFC process" path: /community/contribute/rfc_process From 960699fe84615f1fa1ec8bb00fd9a308c2fc2950 Mon Sep 17 00:00:00 2001 From: kristenrq <58997898+kristenrq@users.noreply.github.com> Date: Mon, 21 Mar 2022 09:04:41 -0700 Subject: [PATCH 028/872] Add link to SIGs page --- site/en/community/contribute/community.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/site/en/community/contribute/community.md b/site/en/community/contribute/community.md index d980b88ebde..bb2b2035d1b 100644 --- a/site/en/community/contribute/community.md +++ b/site/en/community/contribute/community.md @@ -39,6 +39,8 @@ Most communication happens on the TensorFlow Forum. The following mailing lists * [announce@tensorflow.org](mailto:announce@tensorflow.org) — All major releases and important announcements are sent to this mailing group. We recommend that you join this list if you depend on TensorFlow in any way. * [developers@tensorflow.org](mailto:developers@tensorflow.org) — Discussion for developers who are contributing to TensorFlow. +For more information on project-specific communication, visit the [Contribute to SIGs](https://tensorflow.org/community/contribute/sigs) page. + ### Blog and social media The [TensorFlow Blog](http://blog.tensorflow.org/) is full of great content both from our team at Google and the broader community. We'd love to hear what you have to say, so if you would like to submit an article for review, please contact us at tensorflow-blog@google.com. Note that we receive many great submissions, and setting expectations, we can only publish a few. From ead75e10b1f27b6b761f4707bc60779227c13e1a Mon Sep 17 00:00:00 2001 From: Yilei Yang Date: Tue, 22 Mar 2022 03:58:12 -0700 Subject: [PATCH 029/872] Remove unused comments related to Python 2 compatibility. PiperOrigin-RevId: 436433782 --- tools/tensorflow_docs/api_generator/__init__.py | 1 - tools/tensorflow_docs/api_generator/config.py | 1 - tools/tensorflow_docs/api_generator/doc_controls.py | 1 - tools/tensorflow_docs/api_generator/doc_controls_test.py | 1 - tools/tensorflow_docs/api_generator/doc_generator_visitor.py | 1 - .../tensorflow_docs/api_generator/doc_generator_visitor_test.py | 1 - tools/tensorflow_docs/api_generator/generate_lib.py | 1 - tools/tensorflow_docs/api_generator/generate_lib_test.py | 1 - tools/tensorflow_docs/api_generator/obj_type.py | 1 - tools/tensorflow_docs/api_generator/parser.py | 1 - tools/tensorflow_docs/api_generator/parser_test.py | 1 - tools/tensorflow_docs/api_generator/public_api.py | 1 - tools/tensorflow_docs/api_generator/public_api_test.py | 1 - tools/tensorflow_docs/api_generator/reference_resolver.py | 1 - tools/tensorflow_docs/api_generator/reference_resolver_test.py | 1 - tools/tensorflow_docs/api_generator/signature.py | 1 - tools/tensorflow_docs/api_generator/signature_test.py | 1 - tools/tensorflow_docs/api_generator/test_module1.py | 1 - tools/tensorflow_docs/api_generator/test_module2.py | 1 - tools/tensorflow_docs/api_generator/toc_processing.py | 1 - tools/tensorflow_docs/api_generator/toc_processing_test.py | 1 - tools/tensorflow_docs/api_generator/traverse.py | 1 - tools/tensorflow_docs/api_generator/traverse_test.py | 1 - tools/tensorflow_docs/api_generator/utils.py | 1 - 24 files changed, 24 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/__init__.py b/tools/tensorflow_docs/api_generator/__init__.py index 63704e38eeb..325a9eb075e 100644 --- a/tools/tensorflow_docs/api_generator/__init__.py +++ b/tools/tensorflow_docs/api_generator/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2018 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/config.py b/tools/tensorflow_docs/api_generator/config.py index e699952a427..45ace4ee794 100644 --- a/tools/tensorflow_docs/api_generator/config.py +++ b/tools/tensorflow_docs/api_generator/config.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/doc_controls.py b/tools/tensorflow_docs/api_generator/doc_controls.py index dc08e1d9e7e..5cdeaf6eafc 100644 --- a/tools/tensorflow_docs/api_generator/doc_controls.py +++ b/tools/tensorflow_docs/api_generator/doc_controls.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2018 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/doc_controls_test.py b/tools/tensorflow_docs/api_generator/doc_controls_test.py index ca46b64b3dd..9aeadc21915 100644 --- a/tools/tensorflow_docs/api_generator/doc_controls_test.py +++ b/tools/tensorflow_docs/api_generator/doc_controls_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2018 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index 4e6f785871e..2686ca57ce8 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py index b346909251e..cea7bf2ddc7 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index 35873c5d825..83cfa50ac67 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/generate_lib_test.py b/tools/tensorflow_docs/api_generator/generate_lib_test.py index e1dc9f35da9..df5d2cfd733 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib_test.py +++ b/tools/tensorflow_docs/api_generator/generate_lib_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/obj_type.py b/tools/tensorflow_docs/api_generator/obj_type.py index 897f3f30235..fb1b36a05ad 100644 --- a/tools/tensorflow_docs/api_generator/obj_type.py +++ b/tools/tensorflow_docs/api_generator/obj_type.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py index e147ae25ee4..2ca49c05dbf 100644 --- a/tools/tensorflow_docs/api_generator/parser.py +++ b/tools/tensorflow_docs/api_generator/parser.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/parser_test.py b/tools/tensorflow_docs/api_generator/parser_test.py index d61df5f5950..14cb4f55d8c 100644 --- a/tools/tensorflow_docs/api_generator/parser_test.py +++ b/tools/tensorflow_docs/api_generator/parser_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py index c81db3ed1a1..c39026e5e33 100644 --- a/tools/tensorflow_docs/api_generator/public_api.py +++ b/tools/tensorflow_docs/api_generator/public_api.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/public_api_test.py b/tools/tensorflow_docs/api_generator/public_api_test.py index 140acd42e61..901a048809a 100644 --- a/tools/tensorflow_docs/api_generator/public_api_test.py +++ b/tools/tensorflow_docs/api_generator/public_api_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/reference_resolver.py b/tools/tensorflow_docs/api_generator/reference_resolver.py index 45957e3c75c..04969ee2e64 100644 --- a/tools/tensorflow_docs/api_generator/reference_resolver.py +++ b/tools/tensorflow_docs/api_generator/reference_resolver.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/reference_resolver_test.py b/tools/tensorflow_docs/api_generator/reference_resolver_test.py index 91bf8acd868..db938ecf457 100644 --- a/tools/tensorflow_docs/api_generator/reference_resolver_test.py +++ b/tools/tensorflow_docs/api_generator/reference_resolver_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index 0cc91b59b7f..499035fefb6 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index f52521bc4c2..09a9a6c228f 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/test_module1.py b/tools/tensorflow_docs/api_generator/test_module1.py index b48f4bc99eb..421f96ddd20 100644 --- a/tools/tensorflow_docs/api_generator/test_module1.py +++ b/tools/tensorflow_docs/api_generator/test_module1.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2018 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/test_module2.py b/tools/tensorflow_docs/api_generator/test_module2.py index 1461eff856d..068fc17dcd9 100644 --- a/tools/tensorflow_docs/api_generator/test_module2.py +++ b/tools/tensorflow_docs/api_generator/test_module2.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2018 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/toc_processing.py b/tools/tensorflow_docs/api_generator/toc_processing.py index 74cbcca9e13..a11ed1e9eee 100644 --- a/tools/tensorflow_docs/api_generator/toc_processing.py +++ b/tools/tensorflow_docs/api_generator/toc_processing.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2021 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/toc_processing_test.py b/tools/tensorflow_docs/api_generator/toc_processing_test.py index 595d7b9e7c5..087f69e626d 100644 --- a/tools/tensorflow_docs/api_generator/toc_processing_test.py +++ b/tools/tensorflow_docs/api_generator/toc_processing_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2021 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/traverse.py b/tools/tensorflow_docs/api_generator/traverse.py index 2f5c0c012c6..7e3049b2707 100644 --- a/tools/tensorflow_docs/api_generator/traverse.py +++ b/tools/tensorflow_docs/api_generator/traverse.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/traverse_test.py b/tools/tensorflow_docs/api_generator/traverse_test.py index f5b89bce8b2..98ca7629678 100644 --- a/tools/tensorflow_docs/api_generator/traverse_test.py +++ b/tools/tensorflow_docs/api_generator/traverse_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/utils.py b/tools/tensorflow_docs/api_generator/utils.py index 1c9c62bc49b..f90ae9e6c28 100644 --- a/tools/tensorflow_docs/api_generator/utils.py +++ b/tools/tensorflow_docs/api_generator/utils.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); From d542a8e0300dbb21794685fb5dc4a0156674f969 Mon Sep 17 00:00:00 2001 From: Yilei Yang Date: Tue, 22 Mar 2022 03:58:16 -0700 Subject: [PATCH 030/872] Remove unused comments related to Python 2 compatibility. PiperOrigin-RevId: 436433787 --- tools/tensorflow_docs/api_generator/pretty_docs/__init__.py | 1 - tools/tensorflow_docs/api_generator/pretty_docs/base_page.py | 1 - tools/tensorflow_docs/api_generator/pretty_docs/class_page.py | 1 - .../tensorflow_docs/api_generator/pretty_docs/docs_for_object.py | 1 - tools/tensorflow_docs/api_generator/pretty_docs/function_page.py | 1 - tools/tensorflow_docs/api_generator/pretty_docs/module_page.py | 1 - .../api_generator/pretty_docs/pretty_docs_test.py | 1 - .../tensorflow_docs/api_generator/pretty_docs/type_alias_page.py | 1 - tools/tensorflow_docs/tools/nblint/__main__.py | 1 - tools/tensorflow_docs/tools/nblint/decorator.py | 1 - tools/tensorflow_docs/tools/nblint/fix.py | 1 - tools/tensorflow_docs/tools/nblint/linter.py | 1 - tools/tensorflow_docs/vis/__init__.py | 1 - tools/tensorflow_docs/vis/embed.py | 1 - tools/tensorflow_docs/vis/webp_animation.py | 1 - tools/tensorflow_docs/vis/webp_test.py | 1 - 16 files changed, 16 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/__init__.py b/tools/tensorflow_docs/api_generator/pretty_docs/__init__.py index 110e0d25747..b98e8d8ce8c 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/__init__.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py index fd42d9a575b..34913a0177c 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py index 69cac531d07..525b986cc91 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py index 0185983bad8..783717ebdfe 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py index 2af6443d975..73d596cbc5c 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py index a937b92dd8a..d9d40bb9d89 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py b/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py index 9970f1f559f..0c9514cbb44 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py index 876fb8fb112..daef3c77a61 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2022 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nblint/__main__.py b/tools/tensorflow_docs/tools/nblint/__main__.py index 47cf94e4718..0fbac6f4caa 100644 --- a/tools/tensorflow_docs/tools/nblint/__main__.py +++ b/tools/tensorflow_docs/tools/nblint/__main__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # pylint: disable=invalid-name # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # diff --git a/tools/tensorflow_docs/tools/nblint/decorator.py b/tools/tensorflow_docs/tools/nblint/decorator.py index 9922509f1a8..408fef3d969 100644 --- a/tools/tensorflow_docs/tools/nblint/decorator.py +++ b/tools/tensorflow_docs/tools/nblint/decorator.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nblint/fix.py b/tools/tensorflow_docs/tools/nblint/fix.py index ab9021d8de4..34c603c1ca7 100644 --- a/tools/tensorflow_docs/tools/nblint/fix.py +++ b/tools/tensorflow_docs/tools/nblint/fix.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nblint/linter.py b/tools/tensorflow_docs/tools/nblint/linter.py index 8ccfc660e55..b3df30af09a 100644 --- a/tools/tensorflow_docs/tools/nblint/linter.py +++ b/tools/tensorflow_docs/tools/nblint/linter.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/vis/__init__.py b/tools/tensorflow_docs/vis/__init__.py index a8a4e6eed04..9043810d541 100644 --- a/tools/tensorflow_docs/vis/__init__.py +++ b/tools/tensorflow_docs/vis/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/vis/embed.py b/tools/tensorflow_docs/vis/embed.py index 4ff8623021e..720440cd7c4 100644 --- a/tools/tensorflow_docs/vis/embed.py +++ b/tools/tensorflow_docs/vis/embed.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/vis/webp_animation.py b/tools/tensorflow_docs/vis/webp_animation.py index 9e6e5a9c03c..ae6a8713d4f 100644 --- a/tools/tensorflow_docs/vis/webp_animation.py +++ b/tools/tensorflow_docs/vis/webp_animation.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/vis/webp_test.py b/tools/tensorflow_docs/vis/webp_test.py index a3cc4000d8f..0bc1dd28aed 100644 --- a/tools/tensorflow_docs/vis/webp_test.py +++ b/tools/tensorflow_docs/vis/webp_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); From c0c4ab3446e236893dfab72b7296f635ddbafccf Mon Sep 17 00:00:00 2001 From: Yilei Yang Date: Tue, 22 Mar 2022 03:59:46 -0700 Subject: [PATCH 031/872] Remove unused comments related to Python 2 compatibility. PiperOrigin-RevId: 436433948 --- tools/templates/build_docs.py | 1 - tools/tensorflow_docs/api_generator/compat_test/__init__.py | 1 - tools/tensorflow_docs/api_generator/compat_test/estimator.py | 1 - tools/tensorflow_docs/api_generator/gen_java/__init__.py | 1 - tools/tensorflow_docs/api_generator/report/__init__.py | 1 - tools/tensorflow_docs/api_generator/report/linter.py | 1 - tools/tensorflow_docs/api_generator/report/linter_test.py | 1 - tools/tensorflow_docs/api_generator/report/schema/__init__.py | 1 - tools/tensorflow_docs/api_generator/report/utils.py | 1 - tools/tensorflow_docs/tools/nbfmt/notebook_utils.py | 1 - tools/tensorflow_docs/tools/nblint/style/google.py | 1 - tools/tensorflow_docs/tools/nblint/style/tensorflow.py | 1 - tools/tensorflow_docs/tools/nblint/style/tensorflow_docs_l10n.py | 1 - 13 files changed, 13 deletions(-) diff --git a/tools/templates/build_docs.py b/tools/templates/build_docs.py index 179e864390b..5ac97f46ce3 100644 --- a/tools/templates/build_docs.py +++ b/tools/templates/build_docs.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/compat_test/__init__.py b/tools/tensorflow_docs/api_generator/compat_test/__init__.py index ff91b4d0c8f..dba383f4c29 100644 --- a/tools/tensorflow_docs/api_generator/compat_test/__init__.py +++ b/tools/tensorflow_docs/api_generator/compat_test/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2017 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/compat_test/estimator.py b/tools/tensorflow_docs/api_generator/compat_test/estimator.py index 4df0efa24b6..0b54303ca39 100644 --- a/tools/tensorflow_docs/api_generator/compat_test/estimator.py +++ b/tools/tensorflow_docs/api_generator/compat_test/estimator.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2017 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/gen_java/__init__.py b/tools/tensorflow_docs/api_generator/gen_java/__init__.py index 52dfd9a1872..a8b1bd19b7f 100644 --- a/tools/tensorflow_docs/api_generator/gen_java/__init__.py +++ b/tools/tensorflow_docs/api_generator/gen_java/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/report/__init__.py b/tools/tensorflow_docs/api_generator/report/__init__.py index 668bd75faa8..efaab0c5923 100644 --- a/tools/tensorflow_docs/api_generator/report/__init__.py +++ b/tools/tensorflow_docs/api_generator/report/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/report/linter.py b/tools/tensorflow_docs/api_generator/report/linter.py index 6d23bc52362..aade7e4e50d 100644 --- a/tools/tensorflow_docs/api_generator/report/linter.py +++ b/tools/tensorflow_docs/api_generator/report/linter.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/report/linter_test.py b/tools/tensorflow_docs/api_generator/report/linter_test.py index 1692cd4e2ad..c363c327498 100644 --- a/tools/tensorflow_docs/api_generator/report/linter_test.py +++ b/tools/tensorflow_docs/api_generator/report/linter_test.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/report/schema/__init__.py b/tools/tensorflow_docs/api_generator/report/schema/__init__.py index c6f920e7d27..efacf041752 100644 --- a/tools/tensorflow_docs/api_generator/report/schema/__init__.py +++ b/tools/tensorflow_docs/api_generator/report/schema/__init__.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/api_generator/report/utils.py b/tools/tensorflow_docs/api_generator/report/utils.py index 18cac306439..97063b3be2e 100644 --- a/tools/tensorflow_docs/api_generator/report/utils.py +++ b/tools/tensorflow_docs/api_generator/report/utils.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py b/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py index 2ccb10ba20f..577f2e1e18e 100644 --- a/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py +++ b/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nblint/style/google.py b/tools/tensorflow_docs/tools/nblint/style/google.py index 8c4f262633c..ef2e56a43ca 100644 --- a/tools/tensorflow_docs/tools/nblint/style/google.py +++ b/tools/tensorflow_docs/tools/nblint/style/google.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nblint/style/tensorflow.py b/tools/tensorflow_docs/tools/nblint/style/tensorflow.py index fd4ce078211..c192deabce2 100644 --- a/tools/tensorflow_docs/tools/nblint/style/tensorflow.py +++ b/tools/tensorflow_docs/tools/nblint/style/tensorflow.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tools/tensorflow_docs/tools/nblint/style/tensorflow_docs_l10n.py b/tools/tensorflow_docs/tools/nblint/style/tensorflow_docs_l10n.py index 4806b6090cf..84ed779b795 100644 --- a/tools/tensorflow_docs/tools/nblint/style/tensorflow_docs_l10n.py +++ b/tools/tensorflow_docs/tools/nblint/style/tensorflow_docs_l10n.py @@ -1,4 +1,3 @@ -# Lint as: python3 # Copyright 2020 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); From 9edcd05306c98755f26b9ac7738a8dd5ee0e268f Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Tue, 22 Mar 2022 17:44:07 -0700 Subject: [PATCH 032/872] Link signature defaults for known objects to API locations. When a default contains an instance, prefer to link to the **class** if available. Most of the time if you link to an instance it's a dataclass default. For the main TF API this mainly links simple defaults like enums, dtypes and activation functions back to their class pages. For tensorflow_models this makes the pages for complex nested configs like [OptimizerConfig](https://github.com/tensorflow/models/blob/3d0e12fdd61f1e9e0515b5caed2a33635ae0c8c3/official/modeling/optimization/configs/optimization_config.py#L32) more useful by linking all the nested dataclass instance pages back to their class' page. + drop some TF1 special cases. PiperOrigin-RevId: 436614177 --- .../api_generator/reference_resolver.py | 4 +- .../api_generator/signature.py | 86 ++++++++++--------- .../api_generator/signature_test.py | 22 ++++- 3 files changed, 66 insertions(+), 46 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/reference_resolver.py b/tools/tensorflow_docs/api_generator/reference_resolver.py index 04969ee2e64..91f898b91bf 100644 --- a/tools/tensorflow_docs/api_generator/reference_resolver.py +++ b/tools/tensorflow_docs/api_generator/reference_resolver.py @@ -290,7 +290,7 @@ def one_ref(match): return '\n'.join(fixed_lines) - def python_link(self, link_text, ref_full_name): + def python_link(self, link_text: str, ref_full_name: Optional[str] = None): """Resolve a "`tf.symbol`" reference to a link. This will pick the canonical location for duplicate symbols. @@ -302,6 +302,8 @@ def python_link(self, link_text, ref_full_name): Returns: A link to the documentation page of `ref_full_name`. """ + if ref_full_name is None: + ref_full_name = link_text link_text = html.escape(link_text, quote=True) url = self.reference_to_url(ref_full_name) diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index 499035fefb6..4a38e4ce9a2 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -25,7 +25,7 @@ import textwrap import typing -from typing import Any, Callable, Dict, List, Tuple, Type +from typing import Any, Callable, Dict, List, Optional, Tuple, Type import astor @@ -131,14 +131,6 @@ def strip_obj_addresses(text): class FormatArguments(object): """Formats the arguments and adds type annotations if they exist.""" - _INTERNAL_NAMES = { - 'ops.GraphKeys': 'tf.GraphKeys', - '_ops.GraphKeys': 'tf.GraphKeys', - 'init_ops.zeros_initializer': 'tf.zeros_initializer', - 'init_ops.ones_initializer': 'tf.ones_initializer', - 'saver_pb2.SaverDef': 'tf.train.SaverDef', - } - # A regular expression capturing a python identifier. _IDENTIFIER_RE = r'[a-zA-Z_]\w*' @@ -166,9 +158,11 @@ def __init__( self._reverse_index = parser_config.reverse_index self._reference_resolver = parser_config.reference_resolver - def get_link(self, obj_full_name: str) -> str: + def get_link(self, + link_text: str, + obj_full_name: Optional[str] = None) -> str: return self._reference_resolver.python_link( - link_text=obj_full_name, ref_full_name=obj_full_name) + link_text=link_text, ref_full_name=obj_full_name) def _extract_non_builtin_types(self, arg_obj: Any, non_builtin_types: List[Any]) -> List[Any]: @@ -252,25 +246,54 @@ def _linkify(self, non_builtin_map: Dict[str, Any], match) -> str: return self.get_link(obj_full_name) - def preprocess(self, ast_typehint: str, obj_anno: Any) -> str: + def maybe_add_link(self, source: str, value: Any) -> str: + """Return a link to an object's api page if found. + + Args: + source: The source string from the code. + value: The value of the object. + + Returns: + The original string with maybe an HTML link added. + """ + cls = type(value) + + value_name = self._reverse_index.get(id(value), None) + cls_name = self._reverse_index.get(id(cls), None) + + if cls_name is not None: + # It's much more common for the class to be documented than the instance. + # and the class page will provide better docs. + before = source.split('(')[0] + cls_short_name = cls_name.split('.')[-1] + if before.endswith(cls_short_name): + # Yes, this is a guess but it will usually be right. + return self.get_link(source, cls_name) + + if value_name is not None: + return self.get_link(value_name, value_name) + + return source + + def preprocess(self, string: str, value: Any) -> str: """Links type annotations to its page if it exists. Args: - ast_typehint: AST extracted type annotation. - obj_anno: Type annotation object. + string: AST extracted type annotation. + value: Type annotation object. Returns: Linked type annotation if the type annotation object exists. """ # If the object annotations exists in the reverse_index, get the link # directly for the entire annotation. - obj_anno_full_name = self._reverse_index.get(id(obj_anno), None) + obj_anno_full_name = self._reverse_index.get(id(value), None) if obj_anno_full_name is not None: return self.get_link(obj_anno_full_name) - non_builtin_ast_types = self._get_non_builtin_ast_types(ast_typehint) + non_builtin_ast_types = self._get_non_builtin_ast_types(string) try: - non_builtin_type_objs = self._extract_non_builtin_types(obj_anno, []) + non_builtin_type_objs = self._extract_non_builtin_types(value, []) except RecursionError: non_builtin_type_objs = {} @@ -282,16 +305,7 @@ def preprocess(self, ast_typehint: str, obj_anno: Any) -> str: non_builtin_map = dict(zip(non_builtin_ast_types, non_builtin_type_objs)) partial_func = functools.partial(self._linkify, non_builtin_map) - return self._INDIVIDUAL_TYPES_RE.sub(partial_func, ast_typehint) - - def _replace_internal_names(self, default_text: str) -> str: - full_name_re = f'^{self._IDENTIFIER_RE}(.{self._IDENTIFIER_RE})+' - match = re.match(full_name_re, default_text) - if match: - for internal_name, public_name in self._INTERNAL_NAMES.items(): - if match.group(0).startswith(internal_name): - return public_name + default_text[len(internal_name):] - return default_text + return self._INDIVIDUAL_TYPES_RE.sub(partial_func, string) def format_return(self, return_anno: Tuple[Any, str]) -> str: value, source = return_anno @@ -339,21 +353,11 @@ def format_kwargs(self, kwargs: List[inspect.Parameter]) -> List[str]: default_text = None if kwarg.default is not EMPTY: default_val, default_source = kwarg.default + if default_source is None: + default_source = strip_obj_addresses(repr(default_val)) + default_source = html.escape(default_source) - if id(default_val) in self._reverse_index: - default_text = self._reverse_index[id(default_val)] - elif default_source is not None: - default_text = default_source - if default_text != repr(default_val): - default_text = self._replace_internal_names(default_text) - # Kwarg without default value. - elif default_val is EMPTY: - kwargs_text_repr.extend(self.format_args([kwarg])) - continue - else: - # Strip object memory addresses to avoid unnecessary doc churn. - default_text = strip_obj_addresses(repr(default_val)) - default_text = html.escape(str(default_text)) + default_text = self.maybe_add_link(default_source, default_val) # Format the kwargs to add the type annotation and default values. typeanno = None diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index 09a9a6c228f..0090f49cfd9 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -50,9 +50,16 @@ def setUp(self): super().setUp() self.known_object = object() reference_resolver = reference_resolver_lib.ReferenceResolver( - duplicate_of={}, + link_prefix='/', + duplicate_of={ + 'tfdocs.api_generator.signature.extract_decorators': + 'tfdocs.api_generator.signature.extract_decorators', + 'location.of.object.in.api': + 'location.of.object.in.api', + }, is_fragment={ - 'tfdocs.api_generator.signature.extract_decorators': False + 'location.of.object.in.api': False, + 'tfdocs.api_generator.signature.extract_decorators': False, }, py_module_names=[]) self.parser_config = config.ParserConfig( @@ -60,7 +67,12 @@ def setUp(self): duplicates={}, duplicate_of={}, tree={}, - index={}, + index={ + 'location.of.object.in.api': + self.known_object, + 'tfdocs.api_generator.signature.extract_decorators': + signature.extract_decorators + }, reverse_index={ id(self.known_object): 'location.of.object.in.api', @@ -79,7 +91,9 @@ def example_fun(arg=self.known_object): # pylint: disable=unused-argument example_fun, parser_config=self.parser_config, func_type=signature.FuncType.FUNCTION) - self.assertEqual('(\n arg=location.of.object.in.api\n)', str(sig)) + self.assertEqual( + '(\n arg=location.of.object.in.api\n)', + str(sig)) def test_literals(self): From 629312db15be659ab5701d2c1309d2e9506d9a05 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mustafa=20=C3=87avu=C5=9Fo=C4=9Flu?= <65891055+mustafacavusoglu@users.noreply.github.com> Date: Wed, 23 Mar 2022 19:24:30 +0300 Subject: [PATCH 033/872] label to labels --- site/en/guide/data.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb index fd3232be1b9..508e6064a57 100644 --- a/site/en/guide/data.ipynb +++ b/site/en/guide/data.ipynb @@ -728,7 +728,7 @@ "source": [ "for images, label in ds.take(1):\n", " print('images.shape: ', images.shape)\n", - " print('labels.shape: ', labels.shape)\n" + " print('labels.shape: ', label.shape)\n" ] }, { From cd7e008785fcdf76528ef4ee4ef9a4a580443abf Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 23 Mar 2022 10:18:03 -0700 Subject: [PATCH 034/872] Update data.ipynb --- site/en/guide/data.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb index 508e6064a57..0641c2ef6d3 100644 --- a/site/en/guide/data.ipynb +++ b/site/en/guide/data.ipynb @@ -728,7 +728,7 @@ "source": [ "for images, label in ds.take(1):\n", " print('images.shape: ', images.shape)\n", - " print('labels.shape: ', label.shape)\n" + " print('label.shape: ', label.shape)\n" ] }, { From 2b8ed607c81ac186ddf4f5c0b98806ea8249e869 Mon Sep 17 00:00:00 2001 From: Markus Hinsche Date: Wed, 23 Mar 2022 18:23:24 +0100 Subject: [PATCH 035/872] Update outdated link to Git development workflow --- site/en/community/contribute/code.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/community/contribute/code.md b/site/en/community/contribute/code.md index 92dc59b1cea..790e1c72e71 100644 --- a/site/en/community/contribute/code.md +++ b/site/en/community/contribute/code.md @@ -162,7 +162,7 @@ up to date. Additional `git` and GitHub resources: * [Git documentation](https://git-scm.com/documentation) -* [Git development workflow](https://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html) +* [Git development workflow](https://docs.scipy.org/doc/numpy/dev/development_workflow.html) * [Resolving merge conflicts](https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/). From 446ad92f2976755a3270265171b4a52e1788faf0 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 23 Mar 2022 11:32:07 -0700 Subject: [PATCH 036/872] Update notebook instructions. PiperOrigin-RevId: 436789596 --- tools/templates/notebook.ipynb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tools/templates/notebook.ipynb b/tools/templates/notebook.ipynb index 3cf831e5b28..b172323d9cf 100644 --- a/tools/templates/notebook.ipynb +++ b/tools/templates/notebook.ipynb @@ -143,15 +143,14 @@ "* Save the notebook with the table of contents open.\n", "* Use one `H1` header for the title.\n", "* Include the button-bar immediately after the `H1`.\n", - "* Avoid using `H1` headers for section titles. Use `H2` and `H3` instead.\n", "* Headers that are`H4` and below are not visible in the navigation\n", "bar of [tensorflow.org](http://www.tensorflow.org).\n", "* Include an overview section before any code.\n", "* Put all your installs and imports in a setup section.\n", - "* Always include the `__future__` imports.\n", - "* Write Python 3 compatible code.\n", "* Keep code and text cells as brief as possible.\n", - "* Avoid leaving an empty cell at the end of the notebook." + "* Break text cells at headings\n", + "* Break code cells between \"building\" and \"running\", and between \"printing one result\" and \"printing another result\".\n", + "* Necessary but uninteresting code (like plotting logic) should be hidden in a toggleable code cell by putting `#@title` as the first line." ] }, { @@ -166,6 +165,7 @@ "* Use the [Google Python Style Guide](http://google.github.io/styleguide/pyguide.html), where applicable.\n", "* tensorflow.org doesn't support interactive plots.\n", "* Keep examples quick. Use small datasets, or small slices of datasets. Don't train to convergence, train until it's obvious it's making progress.\n", + "* If you define a function, run it and show us what it does before using it in another function.\n", "* Demonstrate small parts before combining them into something more complex, like this:" ] }, @@ -240,8 +240,7 @@ "\n", "* Use the highest level API that gets the job done (unless the goal is to demonstrate the low level API).\n", "* Use `keras.Sequential` > keras functional api > keras model subclassing > ...\n", - "* Use `model.fit` > `model.train_on_batch` > manual `GradientTapes`.\n", - "* Use eager-style code.\n", + "* Use `model.fit` > `model.train_step` > manual `GradientTapes`.\n", "* Use `tensorflow_datasets` and `tf.data` where possible.\n", "* When using pre-trained models, prefer models from [tfhub.dev](https://tfhub.dev) where possible.\n", "* Avoid `compat.v1`." @@ -284,6 +283,7 @@ "Tce3stUlHN0L" ], "name": "notebook.ipynb", + "provenance": [], "toc_visible": true }, "kernelspec": { From 82d4b67294e283adc7247426444914521a16a73b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mustafa=20=C3=87avu=C5=9Fo=C4=9Flu?= <65891055+mustafacavusoglu@users.noreply.github.com> Date: Wed, 23 Mar 2022 21:36:46 +0300 Subject: [PATCH 037/872] update (typo fix) --- site/en/guide/data.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb index 0641c2ef6d3..6421484aafd 100644 --- a/site/en/guide/data.ipynb +++ b/site/en/guide/data.ipynb @@ -726,9 +726,9 @@ }, "outputs": [], "source": [ - "for images, label in ds.take(1):\n", + "for images, labels in ds.take(1):\n", " print('images.shape: ', images.shape)\n", - " print('label.shape: ', label.shape)\n" + " print('labels.shape: ', labels.shape)\n" ] }, { From 7272e37d10bf01c6704bd3357252a52abbe3a6ec Mon Sep 17 00:00:00 2001 From: Faizan Muhammad Date: Wed, 23 Mar 2022 15:44:08 -0700 Subject: [PATCH 038/872] Update Public Documentation for Tracing Protocol PiperOrigin-RevId: 436847893 --- site/en/guide/function.ipynb | 115 +++++++++++++++++++++++----- site/en/guide/intro_to_graphs.ipynb | 2 +- 2 files changed, 97 insertions(+), 20 deletions(-) diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb index 7a60e993784..e39c65234b1 100644 --- a/site/en/guide/function.ipynb +++ b/site/en/guide/function.ipynb @@ -360,20 +360,18 @@ "source": [ "#### Rules of tracing\n", "\n", - "A `Function` determines whether to reuse a traced `ConcreteFunction` by computing a **cache key** from an input's args and kwargs. A **cache key** is a key that identifies a `ConcreteFunction` based on the input args and kwargs of the `Function` call, according to the following rules (which may change):\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "h62XoXho6EWN" - }, - "source": [ - "- The key generated for a `tf.Tensor` is its shape and dtype.\n", - "- The key generated for a `tf.Variable` is a unique variable id.\n", - "- The key generated for a Python primitive (like `int`, `float`, `str`) is its value. \n", - "- The key generated for nested `dict`s, `list`s, `tuple`s, `namedtuple`s, and [`attr`](https://www.attrs.org/en/stable/)s is the flattened tuple of leaf-keys (see `nest.flatten`). (As a result of this flattening, calling a concrete function with a different nesting structure than the one used during tracing will result in a TypeError).\n", - "- For all other Python types the key is unique to the object. This way a function or method is traced independently for each instance it is called with.\n" + "When called, a `Function` matches the call arguments to existing `ConcreteFunction`s using `tf.types.experimental.TraceType` of each argument. If a matching `ConcreteFunction` is found, the call is dispatched to it. If no match is found, a new `ConcreteFunction` is traced. \n", + "\n", + "If multiple matches are found, the most specific signature is chosen. Matching is done by [subtyping](https://en.wikipedia.org/wiki/Subtyping), much like normal function calls in C++ or Java, for instance. For example, `TensorShape([1, 2])` is a subtype of `TensorShape([None, None])` and so a call to the tf.function with `TensorShape([1, 2])` can be dispatched to the `ConcreteFunction` produced with `TensorShape([None, None])` but if a `ConcreteFunction` with `TensorShape([1, None])` also exists then it will prioritized since it is more specific.\n", + "\n", + "The `TraceType` is determined from input arguments as follows:\n", + "* For `Tensor`, the type is parameterized by the `Tensor`'s `dtype` and `shape`; ranked shapes are a subtype of unranked shapes; fixed dimensions are a subtype of unknown dimensions\n", + "* For `Variable`, the type is similar to `Tensor`, but also includes a unique resource ID of the variable, necessary to correctly wire control dependencies\n", + "* For Python primitive values, the type corresponds to the **value** itself. For example, the `TraceType` of the value `3` is `LiteralTraceType<3>`, not `int`.\n", + "* For Python ordered containers such as `list` and `tuple`, etc., the type is parameterized by the types of their elements; for example, the type of `[1, 2]` is `ListTraceType, LiteralTraceType<2>>` and the type for `[2, 1]` is `ListTraceType, LiteralTraceType<1>>` which is different.\n", + "* For Python mappings such as `dict`, the type is also a mapping from the same keys but to the types of values instead the actual values. For example, the type of `{1: 2, 3: 4}`, is `MappingTraceType<>>, >>>`. However, unlike ordered containers, `{1: 2, 3: 4}` and `{3: 4, 1: 2}` have equivalent types.\n", + "* For Python objects which implement the `__tf_tracing_type__` method, the type is whatever that method returns\n", + "* For any other Python objects, the type is a generic `TraceType` which uses the object's Python equality and hashing for matching. (Note: It relies on [weakref](https://docs.python.org/3/library/weakref.html) to the object and hence only works as long as the object is in scope/not deleted.)\n" ] }, { @@ -382,7 +380,7 @@ "id": "GNNN4lgRzpIs" }, "source": [ - "Note: Cache keys are based on the `Function` input parameters so changes to global and [free variables](https://docs.python.org/3/reference/executionmodel.html#binding-of-names) alone will not create a new trace. See [this section](#depending_on_python_global_and_free_variables) for recommended practices when dealing with Python global and free variables." + "Note: `TraceType` is based on the `Function` input parameters so changes to global and [free variables](https://docs.python.org/3/reference/executionmodel.html#binding-of-names) alone will not create a new trace. See [this section](#depending_on_python_global_and_free_variables) for recommended practices when dealing with Python global and free variables." ] }, { @@ -391,7 +389,7 @@ "id": "PEDwbumO32Wh" }, "source": [ - "#### Controlling retracing\n", + "### Controlling retracing\n", "\n", "Retracing, which is when your `Function` creates more than one trace, helps ensures that TensorFlow generates correct graphs for each set of inputs. However, tracing is an expensive operation! If your `Function` retraces a new graph for every call, you'll find that your code executes more slowly than if you didn't use `tf.function`.\n", "\n", @@ -404,7 +402,7 @@ "id": "EUtycWJa34TT" }, "source": [ - "- Specify `input_signature` in `tf.function` to limit tracing." + "#### Pass a fixed `input_signature` to `tf.function`" ] }, { @@ -436,7 +434,7 @@ "id": "ocxX-HVk7P2o" }, "source": [ - "- Specify a \\[None\\] dimension in `tf.TensorSpec` to allow for flexibility in trace reuse.\n", + "#### Use unknown dimensions for flexibility\n", "\n", " Since TensorFlow matches tensors based on their shape, using a `None` dimension as a wildcard will allow `Function`s to reuse traces for variably-sized input. Variably-sized input can occur if you have sequences of different length, or images of different sizes for each batch (See the [Transformer](../tutorials/text/transformer.ipynb) and [Deep Dream](../tutorials/generative/deepdream.ipynb) tutorials for example)." ] @@ -465,7 +463,7 @@ "id": "AY5oiQN0XIyA" }, "source": [ - "- Cast Python arguments to Tensors to reduce retracing.\n", + "#### Pass tensors instead of python literals\n", "\n", " Often, Python arguments are used to control hyperparameters and graph constructions - for example, `num_layers=10` or `training=True` or `nonlinearity='relu'`. So, if the Python argument changes, it makes sense that you'd have to retrace the graph.\n", "\n", @@ -525,6 +523,85 @@ "tf.function(f)()" ] }, + { + "cell_type": "markdown", + "metadata": { + "id": "-tZoWrA6INvc" + }, + "source": [ + "#### Use the tracing protocol\n", + "\n", + "Where possible, you should prefer converting the Python type into a [`tf.experimental.ExtensionType`](www.tensorflow.org/api_docs/python/tf/experimental/ExtensionType) instead. Moreover, the `TraceType` of an `ExtensionType` is the `tf.TypeSpec` associated with it. Therefore, if needed, you can simply [override](https://www.tensorflow.org/guide/extension_type#customizing_the_extensiontypes_typespec) the default `tf.TypeSpec` to take control of an `ExtensionType`'s `Tracing Protocol`.\n", + "\n", + "Otherwise, for direct control over when `Function` should retrace in regards to a particular Python type, you can implement the `Tracing Protocol` for it yourself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gZkIh7UaIKc6" + }, + "outputs": [], + "source": [ + "@tf.function\n", + "def get_mixed_flavor(fruit_a, fruit_b):\n", + " return fruit_a.flavor + fruit_b.flavor\n", + "\n", + "class Fruit:\n", + " flavor = tf.constant([0, 0])\n", + "\n", + "class Apple(Fruit):\n", + " flavor = tf.constant([1, 2])\n", + "\n", + "class Mango(Fruit):\n", + " flavor = tf.constant([3, 4])\n", + "\n", + "# As described in the above rules, a generic TraceType for `Apple` and `Mango`\n", + "# is generated (and a corresponding ConcreteFunction is traced) but it fails to \n", + "# match the second function call since the first pair of Apple() and Mango() \n", + "# have gone out out of scope by then and deleted.\n", + "get_mixed_flavor(Apple(), Mango()) # Traces a new concrete function\n", + "get_mixed_flavor(Apple(), Mango()) # Traces a new concrete function again\n", + "\n", + "# However, we, as the designers of the `Fruit` class, know that each subclass\n", + "# has a fixed flavor and we can reuse an existing traced concrete function if\n", + "# it was the same subclass. Avoiding such unnecessary tracing of concrete\n", + "# functions can have significant performance benefits.\n", + "\n", + "class FruitTraceType(tf.types.experimental.TraceType):\n", + " def __init__(self, fruit_type):\n", + " self.fruit_type = fruit_type\n", + "\n", + " def is_subtype_of(self, other):\n", + " return (type(other) is FruitTraceType and\n", + " self.fruit_type is other.fruit_type)\n", + "\n", + " def most_specific_common_supertype(self, others):\n", + " return self if all(self == other for other in others) else None\n", + "\n", + " def __eq__(self, other):\n", + " return type(other) is FruitTraceType and self.fruit_type == other.fruit_type\n", + " \n", + " def __hash__(self):\n", + " return hash(self.fruit_type)\n", + "\n", + "class FruitWithTraceType:\n", + "\n", + " def __tf_tracing_type__(self, context):\n", + " return FruitTraceType(type(self))\n", + "\n", + "class AppleWithTraceType(FruitWithTraceType):\n", + " flavor = tf.constant([1, 2])\n", + "\n", + "class MangoWithTraceType(FruitWithTraceType):\n", + " flavor = tf.constant([3, 4])\n", + "\n", + "# Now if we try calling it again:\n", + "get_mixed_flavor(AppleWithTraceType(), MangoWithTraceType()) # Traces a new concrete function\n", + "get_mixed_flavor(AppleWithTraceType(), MangoWithTraceType()) # Re-uses the traced concrete function" + ] + }, { "cell_type": "markdown", "metadata": { diff --git a/site/en/guide/intro_to_graphs.ipynb b/site/en/guide/intro_to_graphs.ipynb index f34a20a8420..19b5c5f432e 100644 --- a/site/en/guide/intro_to_graphs.ipynb +++ b/site/en/guide/intro_to_graphs.ipynb @@ -333,7 +333,7 @@ "\n", "A `tf.Graph` is specialized to a specific type of inputs (for example, tensors with a specific [`dtype`](https://www.tensorflow.org/api_docs/python/tf/dtypes/DType) or objects with the same [`id()`](https://docs.python.org/3/library/functions.html#id])).\n", "\n", - "Each time you invoke a `Function` with new `dtypes` and shapes in its arguments, `Function` creates a new `tf.Graph` for the new arguments. The `dtypes` and shapes of a `tf.Graph`'s inputs are known as an **input signature** or just a **signature**.\n", + "Each time you invoke a `Function` with a set of arguments that can't be handled by any of its existing graphs (such as arguments with new `dtypes` or incompatible shapes), `Function` creates a new `tf.Graph` specialized to those new arguments. The type specification of a `tf.Graph`'s inputs is known as its **input signature** or just a **signature**. For more information regarding when a new `tf.Graph` is generated and how that can be controlled, see the [rules of retracing](https://www.tensorflow.org/guide/function#rules_of_tracing).\n", "\n", "The `Function` stores the `tf.Graph` corresponding to that signature in a `ConcreteFunction`. **A `ConcreteFunction` is a wrapper around a `tf.Graph`.**\n" ] From 0f28394716522f1a4d60b8e3350f1f9bef352790 Mon Sep 17 00:00:00 2001 From: Olzhas Akpambetov Date: Wed, 23 Mar 2022 16:56:05 -0700 Subject: [PATCH 039/872] Lint Parameter server strategy guide PiperOrigin-RevId: 436863751 --- .../parameter_server_training.ipynb | 43 ++++++++----------- 1 file changed, 19 insertions(+), 24 deletions(-) diff --git a/site/en/tutorials/distribute/parameter_server_training.ipynb b/site/en/tutorials/distribute/parameter_server_training.ipynb index 309210bef35..0edbd218744 100644 --- a/site/en/tutorials/distribute/parameter_server_training.ipynb +++ b/site/en/tutorials/distribute/parameter_server_training.ipynb @@ -108,7 +108,7 @@ "- Multiple _worker_ jobs (job name `worker`); and\n", "- Multiple _parameter server_ jobs (job name `ps`)\n", "\n", - "While the _coordinator_ creates resources, dispatches training tasks, writes checkpoints, and deals with task failures, _workers_ and _parameter servers_ run `tf.distribute.Server` that listen for requests from the coordinator." + "The _coordinator_ creates resources, dispatches training tasks, writes checkpoints, and deals with task failures. The _workers_ and _parameter servers_ run `tf.distribute.Server` instances that listen for requests from the coordinator." ] }, { @@ -348,15 +348,15 @@ }, "source": [ "When a `variable_partitioner` is passed in and if you create a variable directly\n", - "under `strategy.scope()`, it will become a container type with a `variables`\n", - "property which provides access to the list of shards. In most cases, this\n", + "under `Strategy.scope`, it will become a container type with a `variables`\n", + "property, which provides access to the list of shards. In most cases, this\n", "container will be automatically converted to a Tensor by concatenating all the\n", "shards. As a result, it can be used as a normal variable. On the other hand,\n", "some TensorFlow methods such as `tf.nn.embedding_lookup` provide efficient\n", "implementation for this container type and in these methods automatic\n", "concatenation will be avoided.\n", "\n", - "Please see the API docs of `tf.distribute.experimental.ParameterServerStrategy` for more details." + "Refer to the API docs of `tf.distribute.experimental.ParameterServerStrategy` for more details." ] }, { @@ -384,7 +384,7 @@ "\n", "Note that it is recommended to shuffle and repeat the data with parameter server training, and specify `steps_per_epoch` in `fit` call so the library knows the epoch boundaries.\n", "\n", - "Please see the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input#usage_2) tutorial for more information about the `InputContext` argument." + "Refer to the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input#usage_2) tutorial for more information about the `InputContext` argument." ] }, { @@ -420,8 +420,7 @@ "id": "v_jhF70K7zON" }, "source": [ - "The code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n", - "\n" + "The code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n" ] }, { @@ -463,7 +462,7 @@ "\n", "- `ModelCheckpoint`: to save the model weights.\n", "- `BackupAndRestore`: to make sure the training progress is automatically backed up, and recovered if the cluster experiences unavailability (such as abort or preemption); or\n", - "- `TensorBoard`: to save the progress reports into summary files, which get visualized in TensorBoard tool.\n", + "- `TensorBoard`: to save the progress reports into summary files, which can be visualized in the TensorBoard tool.\n", "\n", "Note: Due to performance consideration, custom callbacks cannot have batch level callbacks overridden when used with `ParameterServerStrategy`. Please modify your custom callbacks to make them epoch level calls, and adjust `steps_per_epoch` to a suitable value. In addition, `steps_per_epoch` is a required argument for `Model.fit` when used with `ParameterServerStrategy`." ] @@ -530,11 +529,11 @@ "\n", "First, write a function that creates a dataset.\n", "\n", - "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `strategy.scope()`** like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline. \n", + "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `Strategy.scope`** like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline.\n", "\n", - "If you don't follow the above procedure, creating the layers might create Tensorflow states which will be lifted out of the `tf.function` to the coordinator, and thus accessing them on workers would incur repetitive RPC calls between coordinator and workers and causes significant slowdown. \n", + "If you don't follow the above procedure, creating the layers might create Tensorflow states which will be lifted out of the `tf.function` to the coordinator. Thus, accessing them on workers would incur repetitive RPC calls between coordinator and workers, and cause significant slowdown.\n", "\n", - "Placing the layers under `strategy.scope()` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Please refer to [this tutorial](https://www.tensorflow.org/tutorials/distribute/input#data_preprocessing) for more information on data preprocessing with distributed input." + "Placing the layers under `Strategy.scope` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Refer to _Data preprocessing_ in the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input) tutorial for more information on data preprocessing with distributed input." ] }, { @@ -644,7 +643,7 @@ "source": [ "### Build the model\n", "\n", - "Next, create the model and other objects. Make sure to create all variables under `strategy.scope`." + "Next, create the model and other objects. Make sure to create all variables under `Strategy.scope`." ] }, { @@ -655,7 +654,7 @@ }, "outputs": [], "source": [ - "# These variables created under the `strategy.scope` will be placed on parameter\n", + "# These variables created under the `Strategy.scope` will be placed on parameter\n", "# servers in a round-robin fashion.\n", "with strategy.scope():\n", " # Create the model. The input needs to be compatible with Keras processing layers.\n", @@ -875,9 +874,9 @@ "source": [ "### More about dataset creation\n", "\n", - "The dataset in the above code is created using the `ClusterCoordinator.create_per_worker_dataset` API). It creates one dataset per worker and returns a container object. You can call the `iter` method on it to create a per-worker iterator. The per-worker iterator contains one iterator per worker and the corresponding slice of a worker will be substituted in the input argument of the function passed to the `ClusterCoordinator.schedule` method before the function is executed on a particular worker.\n", + "The dataset in the above code is created using the `ClusterCoordinator.create_per_worker_dataset` API. It creates one dataset per worker and returns a container object. You can call the `iter` method on it to create a per-worker iterator. The per-worker iterator contains one iterator per worker and the corresponding slice of a worker will be substituted in the input argument of the function passed to the `ClusterCoordinator.schedule` method before the function is executed on a particular worker.\n", "\n", - "Currently, the `ClusterCoordinator.schedule` method assumes workers are equivalent and thus assumes the datasets on different workers are the same except they may be shuffled differently if they contain a `Dataset.shuffle` operation. Because of this, it is also recommended that the datasets to be repeated indefinitely and you schedule a finite number of steps instead of relying on the `OutOfRangeError` from a dataset.\n", + "The `ClusterCoordinator.schedule` method assumes workers are equivalent and thus assumes the datasets on different workers are the same (except that they may be shuffled differently). Because of this, it is also recommended to repeat datasets, and schedule a finite number of steps instead of relying on the `OutOfRangeError` from a dataset.\n", "\n", "Another important note is that `tf.data` datasets don’t support implicit serialization and deserialization across task boundaries. So it is important to create the whole dataset inside the function passed to `ClusterCoordinator.create_per_worker_dataset`." ] @@ -1227,9 +1226,9 @@ "2. Avoid creating a hotspot variable that is required by all parameter servers in a single step if possible. For example, use a constant learning rate or subclass `tf.keras.optimizers.schedules.LearningRateSchedule` in optimizers since the default behavior is that the learning rate will become a variable placed on a particular parameter server and requested by all other parameter servers in each step.\n", "3. Shuffle your large vocabularies before passing them to Keras preprocessing layers.\n", "\n", - "Another possible reason for performance issues is the coordinator. The implementation of `schedule`/`join` is Python-based and thus may have threading overhead. Also, the latency between the coordinator and the workers can be large. If this is the case,\n", + "Another possible reason for performance issues is the coordinator. The implementation of `schedule`/`join` is Python-based and thus may have threading overhead. Also, the latency between the coordinator and the workers can be large. If this is the case:\n", "\n", - "- For `Model.fit`, you can set `steps_per_execution` argument provided at `Model.compile` to a value larger than 1.\n", + "- For `Model.fit`, you can set the `steps_per_execution` argument provided at `Model.compile` to a value larger than 1.\n", "\n", "- For a custom training loop, you can pack multiple steps into a single `tf.function`:\n", "\n", @@ -1271,9 +1270,7 @@ "- It is not supported to load a saved_model via `tf.saved_model.load` containing sharded variables. Note loading such a saved_model using TensorFlow Serving is expected to work.\n", "- It is not supported to load a checkpoint containing sharded optimizer slot variables into a different number of shards.\n", "- It is not supported to recover from parameter server failure without restarting the coordinator task.\n", - "- Creation of `tf.lookup.StaticHashTable`, commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`, should be placed under `strategy.scope()`. Otherwise, resources will be placed on the coordinator, and lookup RPCs from workers to the coordinator incur performance implications. \n", - "\n", - "\n" + "- Creation of `tf.lookup.StaticHashTable`, commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`, should be placed under `Strategy.scope`. Otherwise, resources will be placed on the coordinator, and lookup RPCs from workers to the coordinator incur performance implications.\n" ] }, { @@ -1286,9 +1283,8 @@ "\n", "- `steps_per_epoch` argument is required in `Model.fit`. You can select a value that provides appropriate intervals in an epoch.\n", "- `ParameterServerStrategy` does not have support for custom callbacks that have batch-level calls for performance reasons. You should convert those calls into epoch-level calls with suitably picked `steps_per_epoch`, so that they are called every `steps_per_epoch` number of steps. Built-in callbacks are not affected: their batch-level calls have been modified to be performant. Supporting batch-level calls for `ParameterServerStrategy` is being planned.\n", - "- For the same reason, unlike other strategies, progress bar and metrics are logged only at epoch boundaries.\n", - "- `run_eagerly` is not supported.\n", - "\n" + "- For the same reason, unlike other strategies, progress bars and metrics are logged only at epoch boundaries.\n", + "- `run_eagerly` is not supported.\n" ] }, { @@ -1310,7 +1306,6 @@ "colab": { "collapsed_sections": [], "name": "parameter_server_training.ipynb", - "provenance": [], "toc_visible": true }, "kernelspec": { From 2da2058d30167b38bad7496c7e2df14313b79fc6 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Thu, 24 Mar 2022 04:25:59 -0700 Subject: [PATCH 040/872] Upgrade the class-defaults extractor to work on without annotations. Now this can work on any class (maybe modules too), not only `dataclasses` PiperOrigin-RevId: 436961154 --- .../api_generator/signature.py | 64 ++++++++++--------- .../api_generator/signature_test.py | 22 +++++++ 2 files changed, 56 insertions(+), 30 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index 4a38e4ce9a2..2ac88f11b54 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -54,9 +54,13 @@ def _preprocess_default(self, val: ast.AST) -> str: text_default_val = self._PAREN_NUMBER_RE.sub('\\1', text_default_val) return text_default_val + def extract(self, obj: Any): + obj_source = textwrap.dedent(inspect.getsource(obj)) + obj_ast = ast.parse(obj_source) + self.visit(obj_ast) -class _CallableDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor - ): + +class _ArgDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor): """Extracts the type annotations by parsing the AST of a function.""" def visit_FunctionDef(self, node) -> None: # pylint: disable=invalid-name @@ -95,8 +99,7 @@ def visit_FunctionDef(self, node) -> None: # pylint: disable=invalid-name self.defaults[kwarg.arg] = text_default_val -class _DataclassDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor - ): +class _ClassDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor): """Extracts the type annotations by parsing the AST of a dataclass.""" def __init__(self): @@ -111,6 +114,8 @@ def visit_ClassDef(self, node) -> None: # pylint: disable=invalid-name for sub in node.body: if isinstance(sub, ast.AnnAssign): self.visit_AnnAssign(sub) + elif isinstance(sub, ast.Assign): + self.visit_Assign(sub) def visit_AnnAssign(self, node) -> None: # pylint: disable=invalid-name """Vists an assignment with a type annotation. Dataclasses is an example.""" @@ -120,6 +125,20 @@ def visit_AnnAssign(self, node) -> None: # pylint: disable=invalid-name if node.value is not None: self.defaults[arg] = self._preprocess_default(node.value) + def visit_Assign(self, node) -> None: # pylint: disable=invalid-name + """Vists an assignment with a type annotation. Dataclasses is an example.""" + names = [_source_from_ast(t) for t in node.targets] + if node.value is not None: + val = self._preprocess_default(node.value) + for name in names: + self.defaults[name] = val + + def extract(self, cls): + # Iterate over the classes in reverse order so each class overwrites it's + # parents. Skip `object`. + for cls in reversed(cls.__mro__[:-1]): + super().extract(cls) + _OBJECT_MEMORY_ADDRESS_RE = re.compile(r'<(?P.+?) at 0x[\da-f]+>') @@ -562,9 +581,9 @@ def generate_signature( if dataclasses.is_dataclass(func): sig = sig.replace(return_annotation=EMPTY) - extract_fn = _extract_dataclass_defaults_and_annotations + extract_fn = _extract_class_defaults_and_annotations else: - extract_fn = _extract_callable_defaults_and_annotations + extract_fn = _extract_arg_defaults_and_annotations (annotation_source_dict, defaults_source_dict, return_annotation_source) = extract_fn(func) @@ -598,44 +617,29 @@ def generate_signature( AnnotsDefaultsReturns = Tuple[Dict[str, str], Dict[str, str], Any] -def _extract_dataclass_defaults_and_annotations( - func: Type[object]) -> AnnotsDefaultsReturns: +def _extract_class_defaults_and_annotations( + cls: Type[object]) -> AnnotsDefaultsReturns: """Extract ast defaults and annotations form a dataclass.""" - stack = [c for c in func.__mro__ if dataclasses.is_dataclass(c)] - - annotation_source_dict = {} - defaults_source_dict = {} - return_annotation_source = EMPTY - - # Iterate over the classes in reverse order so precedence works. - for cls in reversed(stack): - func_source = textwrap.dedent(inspect.getsource(cls)) - func_ast = ast.parse(func_source) - # Extract the type annotation from the parsed ast. - ast_visitor = _DataclassDefaultAndAnnotationExtractor() - ast_visitor.visit(func_ast) + ast_visitor = _ClassDefaultAndAnnotationExtractor() + ast_visitor.extract(cls) - annotation_source_dict.update(ast_visitor.annotations) - defaults_source_dict.update(ast_visitor.defaults) - - return annotation_source_dict, defaults_source_dict, return_annotation_source + return (ast_visitor.annotations, ast_visitor.defaults, + ast_visitor.return_annotation) -def _extract_callable_defaults_and_annotations( +def _extract_arg_defaults_and_annotations( func: Callable[..., Any]) -> AnnotsDefaultsReturns: """Extract ast defaults and annotations form a standard callable.""" - ast_visitor = _CallableDefaultAndAnnotationExtractor() + ast_visitor = _ArgDefaultAndAnnotationExtractor() annotation_source_dict = {} defaults_source_dict = {} return_annotation_source = EMPTY try: - func_source = textwrap.dedent(inspect.getsource(func)) - func_ast = ast.parse(func_source) # Extract the type annotation from the parsed ast. - ast_visitor.visit(func_ast) + ast_visitor.extract(func) except Exception: # pylint: disable=broad-except # A wide-variety of errors can be thrown here. pass diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index 0090f49cfd9..028cd0273f1 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -408,6 +408,28 @@ class Child(Parent): )""") self.assertEqual(expected, str(sig)) + def test_extract_non_annotated(self): + + const = 1234 + + class A: + a = 60 * 60 + b = 1 / 9 + + class B(A): + b = 2 / 9 + c = const + + ast_extractor = signature._ClassDefaultAndAnnotationExtractor() + ast_extractor.extract(B) + + self.assertEqual({ + 'a': '(60 * 60)', + 'b': '(2 / 9)', + 'c': 'const' + }, ast_extractor.defaults) + + def test_vararg_before_kwargonly_consistent_order(self): def my_fun(*args, a=1, **kwargs): # pylint: disable=unused-argument From f74813d5e7c968343e358a8ce1747f232fea409b Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Fri, 25 Mar 2022 06:29:31 -0700 Subject: [PATCH 041/872] Fix broken caching for custom page_info classes. This is why the raw_ops page is broken: https://www.tensorflow.org/api_docs/python/tf/raw_ops PiperOrigin-RevId: 437232519 --- .../api_generator/generate_lib.py | 2 +- .../api_generator/pretty_docs/base_page.py | 49 ++++++++++++++++--- .../pretty_docs/docs_for_object.py | 31 +----------- 3 files changed, 43 insertions(+), 39 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index 83cfa50ac67..da44a5135cd 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -563,7 +563,7 @@ def write_docs( try: path.parent.mkdir(exist_ok=True, parents=True) - path.write_text(page_info.text, encoding='utf-8') + path.write_text(page_info.page_text, encoding='utf-8') num_docs_output += 1 except OSError as e: raise OSError('Cannot write documentation for ' diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py index 34913a0177c..764cdd0ca36 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py @@ -14,8 +14,9 @@ # ============================================================================== """Base classes for page construction.""" import abc -import functools +import os import pathlib +import posixpath import textwrap from typing import Any, ClassVar, Dict, List, NamedTuple, Optional, Sequence, Tuple, Type @@ -103,6 +104,7 @@ class PageInfo: search_hints: If true include metadata search hints, else include a "robots: noindex" text: The resulting page text. + page_text: The cached result. """ DEFAULT_BUILDER_CLASS: ClassVar[Type[PageBuilder]] = TemplatePageBuilder @@ -122,6 +124,7 @@ def __init__( that need to be added to the markdown pages created. search_hints: If true include metadata search hints, else include a "robots: noindex" + """ self.full_name = full_name self.py_object = py_object @@ -131,22 +134,53 @@ def __init__( self._defined_in = None self._aliases = None self._doc = None - self._text = None + self._page_text = None def collect_docs(self, parser_config: config.ParserConfig): """Collects additional information from the `config.ParserConfig`.""" pass + def docs_for_object(self, parser_config): + duplicate_names = parser_config.duplicates.get(self.full_name, []) + if self.full_name in duplicate_names: + duplicate_names.remove(self.full_name) + + relative_path = os.path.relpath( + path='.', + start=os.path.dirname(parser.documentation_path(self.full_name)) or '.') + + # Convert from OS-specific path to URL/POSIX path. + relative_path = posixpath.join(*relative_path.split(os.path.sep)) + + with parser_config.reference_resolver.temp_prefix(relative_path): + self.set_doc( + parser.parse_md_docstring( + self.py_object, + self.full_name, + parser_config, + self._extra_docs, + )) + + self.collect_docs(parser_config) + + self.set_aliases(duplicate_names) + + self.set_defined_in(parser.get_defined_in(self.py_object, parser_config)) + + self._page_text = self.build() + + return self._page_text + def build(self) -> str: """Builds the documentation.""" cls = self.DEFAULT_BUILDER_CLASS - text = cls(self).build() - self._text = text - return text + return cls(self).build() @property - def text(self): - return self._text + def page_text(self): + if self._page_text is None: + self._page_text = self.build() + return self._page_text def __eq__(self, other): if isinstance(other, PageInfo): @@ -198,7 +232,6 @@ def set_doc(self, doc: parser.DocstringInfo): self._doc = doc - class MemberInfo(NamedTuple): """Describes an attribute of a class or module.""" short_name: str diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py index 783717ebdfe..ae1cb5ec6b7 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py @@ -13,15 +13,11 @@ # limitations under the License. # ============================================================================== """Create a `pretty_docs.base_page.PageInfo` from a python object.""" -import os -import posixpath - from typing import Any, Dict, Optional, Type from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_controls from tensorflow_docs.api_generator import obj_type as obj_type_lib -from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator.pretty_docs import base_page from tensorflow_docs.api_generator.pretty_docs import class_page from tensorflow_docs.api_generator.pretty_docs import function_page @@ -75,9 +71,6 @@ def docs_for_object( # Which other aliases exist for the object referenced by full_name? main_name = parser_config.reference_resolver.py_main_name(full_name) - duplicate_names = parser_config.duplicates.get(main_name, []) - if main_name in duplicate_names: - duplicate_names.remove(main_name) if page_builder_classes is None: page_builder_classes = _DEFAULT_PAGE_BUILDER_CLASSES @@ -93,28 +86,6 @@ def docs_for_object( search_hints=search_hints, extra_docs=extra_docs) - relative_path = os.path.relpath( - path='.', - start=os.path.dirname(parser.documentation_path(full_name)) or '.') - - # Convert from OS-specific path to URL/POSIX path. - relative_path = posixpath.join(*relative_path.split(os.path.sep)) - - with parser_config.reference_resolver.temp_prefix(relative_path): - page_info.set_doc( - parser.parse_md_docstring( - py_object, - full_name, - parser_config, - extra_docs, - )) - - page_info.collect_docs(parser_config) - - page_info.set_aliases(duplicate_names) - - page_info.set_defined_in(parser.get_defined_in(py_object, parser_config)) - - page_info.build() + page_info.docs_for_object(parser_config) return page_info From c1278c14dd73d850196ec502591a5e51909604ca Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Fri, 25 Mar 2022 11:07:15 -0700 Subject: [PATCH 042/872] Update Distributed Input tutorial: fix typos, linting, add some minor details, and add virtual device setup to make notebook outputs better match likely usage. PiperOrigin-RevId: 437290238 --- site/en/tutorials/distribute/input.ipynb | 134 ++++++++++++++--------- 1 file changed, 82 insertions(+), 52 deletions(-) diff --git a/site/en/tutorials/distribute/input.ipynb b/site/en/tutorials/distribute/input.ipynb index 151ac8007a4..073cdd66725 100644 --- a/site/en/tutorials/distribute/input.ipynb +++ b/site/en/tutorials/distribute/input.ipynb @@ -84,7 +84,7 @@ "id": "MM6W__qraV55" }, "source": [ - "## Distributed Datasets" + "## Distributed datasets" ] }, { @@ -93,8 +93,8 @@ "id": "lNy9GxjSlMKQ" }, "source": [ - "To use `tf.distribute` APIs to scale, it is recommended that users use `tf.data.Dataset` to represent their input. `tf.distribute` has been made to work efficiently with `tf.data.Dataset` (for example, automatic prefetch of data onto each accelerator device) with performance optimizations being regularly incorporated into the implementation. If you have a use case for using something other than `tf.data.Dataset`, please refer a later [section](#tensorinputs) in this guide.\n", - "In a non distributed training loop, users first create a `tf.data.Dataset` instance and then iterate over the elements. For example:\n" + "To use `tf.distribute` APIs to scale, use `tf.data.Dataset` to represent their input. `tf.distribute` works efficiently with `tf.data.Dataset`—for example, via automatic prefetching onto each accelerator device and regular performance updates. If you have a use case for using something other than `tf.data.Dataset`, please refer to the [Tensor inputs section](#tensorinputs) in this guide.\n", + "In a non-distributed training loop, first create a `tf.data.Dataset` instance and then iterate over the elements. For example:\n" ] }, { @@ -114,6 +114,34 @@ "print(tf.__version__)" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6cnilUtmKwpa" + }, + "outputs": [], + "source": [ + "# Simulate multiple CPUs with virtual devices\n", + "N_VIRTUAL_DEVICES = 2\n", + "physical_devices = tf.config.list_physical_devices(\"CPU\")\n", + "tf.config.set_logical_device_configuration(\n", + " physical_devices[0], [tf.config.LogicalDeviceConfiguration() for _ in range(N_VIRTUAL_DEVICES)])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zd4l1ySeLRk1" + }, + "outputs": [], + "source": [ + "print(\"Available devices:\")\n", + "for i, device in enumerate(tf.config.list_logical_devices()):\n", + " print(\"%d) %s\" % (i, device))" + ] + }, { "cell_type": "code", "execution_count": null, @@ -214,14 +242,14 @@ " * Replica 1:[0, 1]\n", " * Replica 2:[2, 3]\n", " * Batch 2:\n", - " * Replica 2: [4]\n", + " * Replica 1: [4]\n", " * Replica 2: [5]\n", "\n", "\n", "\n", "* `tf.data.Dataset.range(4).batch(4)`\n", " * Without distribution:\n", - " * Batch 1: [[0], [1], [2], [3]]\n", + " * Batch 1: [0, 1, 2, 3]\n", " * With distribution over 5 replicas:\n", " * Batch 1:\n", " * Replica 1: [0]\n", @@ -246,7 +274,7 @@ "\n", "Note: The above examples only illustrate how a global batch is split on different replicas. It is not advisable to depend on the actual values that might end up on each replica as it can change depending on the implementation.\n", "\n", - "Rebatching the dataset has a space complexity that increases linearly with the number of replicas. This means that for the multi worker training use case the input pipeline can run into OOM errors. " + "Rebatching the dataset has a space complexity that increases linearly with the number of replicas. This means that for the multi-worker training use case the input pipeline can run into OOM errors. " ] }, { @@ -257,7 +285,7 @@ "source": [ "##### Sharding\n", "\n", - "`tf.distribute` also autoshards the input dataset in multi worker training with `MultiWorkerMirroredStrategy` and `TPUStrategy`. Each dataset is created on the CPU device of the worker. Autosharding a dataset over a set of workers means that each worker is assigned a subset of the entire dataset (if the right `tf.data.experimental.AutoShardPolicy` is set). This is to ensure that at each step, a global batch size of non overlapping dataset elements will be processed by each worker. Autosharding has a couple of different options that can be specified using `tf.data.experimental.DistributeOptions`. Note that there is no autosharding in multi worker training with `ParameterServerStrategy`, and more information on dataset creation with this strategy can be found in the [Parameter Server Strategy tutorial](parameter_server_training.ipynb). " + "`tf.distribute` also autoshards the input dataset in multi-worker training with `MultiWorkerMirroredStrategy` and `TPUStrategy`. Each dataset is created on the CPU device of the worker. Autosharding a dataset over a set of workers means that each worker is assigned a subset of the entire dataset (if the right `tf.data.experimental.AutoShardPolicy` is set). This is to ensure that at each step, a global batch size of non-overlapping dataset elements will be processed by each worker. Autosharding has a couple of different options that can be specified using `tf.data.experimental.DistributeOptions`. Note that there is no autosharding in multi-worker training with `ParameterServerStrategy`, and more information on dataset creation with this strategy can be found in the [ParameterServerStrategy tutorial](parameter_server_training.ipynb). " ] }, { @@ -268,7 +296,7 @@ }, "outputs": [], "source": [ - "dataset = tf.data.Dataset.from_tensors(([1.],[1.])).repeat(64).batch(16)\n", + "dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(64).batch(16)\n", "options = tf.data.Options()\n", "options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA\n", "dataset = dataset.with_options(options)" @@ -358,7 +386,7 @@ "source": [ "#### Usage\n", "\n", - "This API takes an input function and returns a `tf.distribute.DistributedDataset` instance. The input function that users pass in has a `tf.distribute.InputContext` argument and should return a `tf.data.Dataset` instance. With this API, `tf.distribute` does not make any further changes to the user’s `tf.data.Dataset` instance returned from the input function. It is the responsibility of the user to batch and shard the dataset. `tf.distribute` calls the input function on the CPU device of each of the workers. Apart from allowing users to specify their own batching and sharding logic, this API also demonstrates better scalability and performance compared to `tf.distribute.Strategy.experimental_distribute_dataset` when used for multi worker training." + "This API takes an input function and returns a `tf.distribute.DistributedDataset` instance. The input function that users pass in has a `tf.distribute.InputContext` argument and should return a `tf.data.Dataset` instance. With this API, `tf.distribute` does not make any further changes to the user’s `tf.data.Dataset` instance returned from the input function. It is the responsibility of the user to batch and shard the dataset. `tf.distribute` calls the input function on the CPU device of each of the workers. Apart from allowing users to specify their own batching and sharding logic, this API also demonstrates better scalability and performance compared to `tf.distribute.Strategy.experimental_distribute_dataset` when used for multi-worker training." ] }, { @@ -373,11 +401,11 @@ "\n", "def dataset_fn(input_context):\n", " batch_size = input_context.get_per_replica_batch_size(global_batch_size)\n", - " dataset = tf.data.Dataset.from_tensors(([1.],[1.])).repeat(64).batch(16)\n", + " dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(64).batch(16)\n", " dataset = dataset.shard(\n", - " input_context.num_input_pipelines, input_context.input_pipeline_id)\n", + " input_context.num_input_pipelines, input_context.input_pipeline_id)\n", " dataset = dataset.batch(batch_size)\n", - " dataset = dataset.prefetch(2) # This prefetches 2 batches per device.\n", + " dataset = dataset.prefetch(2) # This prefetches 2 batches per device.\n", " return dataset\n", "\n", "dist_dataset = mirrored_strategy.distribute_datasets_from_function(dataset_fn)" @@ -411,7 +439,7 @@ "source": [ "##### Sharding\n", "\n", - "The `tf.distribute.InputContext` object that is implicitly passed as an argument to the user’s input function is created by `tf.distribute` under the hood. It has information about the number of workers, current worker id etc. This input function can handle sharding as per policies set by the user using these properties that are part of the `tf.distribute.InputContext` object.\n" + "The `tf.distribute.InputContext` object that is implicitly passed as an argument to the user’s input function is created by `tf.distribute` under the hood. It has information about the number of workers, current worker ID etc. This input function can handle sharding as per policies set by the user using these properties that are part of the `tf.distribute.InputContext` object.\n" ] }, { @@ -422,7 +450,7 @@ "source": [ "##### Prefetching\n", "\n", - "`tf.distribute` does not add a prefetch transformation at the end of the `tf.data.Dataset` returned by the user provided input function." + "`tf.distribute` does not add a prefetch transformation at the end of the `tf.data.Dataset` returned by the user-provided input function, so you explicitly call `Dataset.prefetch` in the example above." ] }, { @@ -442,7 +470,7 @@ "id": "dL3XbI1gzEjO" }, "source": [ - "## Distributed Iterators" + "## Distributed iterators" ] }, { @@ -452,7 +480,7 @@ }, "source": [ "Similar to non-distributed `tf.data.Dataset` instances, you will need to create an iterator on the `tf.distribute.DistributedDataset` instances to iterate over it and access the elements in the `tf.distribute.DistributedDataset`.\n", - "The following are the ways in which you can create an `tf.distribute.DistributedIterator` and use it to train your model:\n" + "The following are the ways in which you can create a `tf.distribute.DistributedIterator` and use it to train your model:\n" ] }, { @@ -486,7 +514,7 @@ "global_batch_size = 16\n", "mirrored_strategy = tf.distribute.MirroredStrategy()\n", "\n", - "dataset = tf.data.Dataset.from_tensors(([1.],[1.])).repeat(100).batch(global_batch_size)\n", + "dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(100).batch(global_batch_size)\n", "dist_dataset = mirrored_strategy.experimental_distribute_dataset(dataset)\n", "\n", "@tf.function\n", @@ -536,14 +564,16 @@ "id": "UpJXIlxjqPYg" }, "source": [ - "With `next()` or `tf.distribute.DistributedIterator.get_next()`, if the `tf.distribute.DistributedIterator` has reached its end, an OutOfRange error will be thrown. The client can catch the error on python side and continue doing other work such as checkpointing and evaluation. However, this will not work if you are using a host training loop (i.e., run multiple steps per `tf.function`), which looks like:\n", + "With `next` or `tf.distribute.DistributedIterator.get_next`, if the `tf.distribute.DistributedIterator` has reached its end, an OutOfRange error will be thrown. The client can catch the error on python side and continue doing other work such as checkpointing and evaluation. However, this will not work if you are using a host training loop (i.e., run multiple steps per `tf.function`), which looks like:\n", + "\n", "```\n", "@tf.function\n", "def train_fn(iterator):\n", " for _ in tf.range(steps_per_loop):\n", " strategy.run(step_fn, args=(next(iterator),))\n", "```\n", - " `train_fn` contains multiple steps by wrapping the step body inside a `tf.range`. In this case, different iterations in the loop with no dependency could start in parallel, so an OutOfRange error can be triggered in later iterations before the computation of previous iterations finishes. Once an OutOfRange error is thrown, all the ops in the function will be terminated right away. If this is some case that you would like to avoid, an alternative that does not throw an OutOfRange error is `tf.distribute.DistributedIterator.get_next_as_optional()`. `get_next_as_optional` returns a `tf.experimental.Optional` which contains the next element or no value if the `tf.distribute.DistributedIterator` has reached to an end." + "\n", + "This example `train_fn` contains multiple steps by wrapping the step body inside a `tf.range`. In this case, different iterations in the loop with no dependency could start in parallel, so an OutOfRange error can be triggered in later iterations before the computation of previous iterations finishes. Once an OutOfRange error is thrown, all the ops in the function will be terminated right away. If this is some case that you would like to avoid, an alternative that does not throw an OutOfRange error is `tf.distribute.DistributedIterator.get_next_as_optional`. `get_next_as_optional` returns a `tf.experimental.Optional` which contains the next element or no value if the `tf.distribute.DistributedIterator` has reached an end." ] }, { @@ -554,10 +584,10 @@ }, "outputs": [], "source": [ - "# You can break the loop with get_next_as_optional by checking if the Optional contains value\n", + "# You can break the loop with `get_next_as_optional` by checking if the `Optional` contains a value\n", "global_batch_size = 4\n", "steps_per_loop = 5\n", - "strategy = tf.distribute.MirroredStrategy(devices=[\"GPU:0\", \"CPU:0\"])\n", + "strategy = tf.distribute.MirroredStrategy()\n", "\n", "dataset = tf.data.Dataset.range(9).batch(global_batch_size)\n", "distributed_iterator = iter(strategy.experimental_distribute_dataset(dataset))\n", @@ -568,7 +598,7 @@ " optional_data = distributed_iterator.get_next_as_optional()\n", " if not optional_data.has_value():\n", " break\n", - " per_replica_results = strategy.run(lambda x:x, args=(optional_data.get_value(),))\n", + " per_replica_results = strategy.run(lambda x: x, args=(optional_data.get_value(),))\n", " tf.print(strategy.experimental_local_results(per_replica_results))\n", "train_fn(distributed_iterator)" ] @@ -579,7 +609,7 @@ "id": "LaclbKnqzLjf" }, "source": [ - "## Using `element_spec` property" + "## Using the `element_spec` property" ] }, { @@ -588,7 +618,7 @@ "id": "Z1YvXqOpwy08" }, "source": [ - "If you pass the elements of a distributed dataset to a `tf.function` and want a `tf.TypeSpec` guarantee, you can specify the `input_signature` argument of the `tf.function`. The output of a distributed dataset is `tf.distribute.DistributedValues` which can represent the input to a single device or multiple devices. To get the `tf.TypeSpec` corresponding to this distributed value you can use the `element_spec` property of the distributed dataset or distributed iterator object." + "If you pass the elements of a distributed dataset to a `tf.function` and want a `tf.TypeSpec` guarantee, you can specify the `input_signature` argument of the `tf.function`. The output of a distributed dataset is `tf.distribute.DistributedValues` which can represent the input to a single device or multiple devices. To get the `tf.TypeSpec` corresponding to this distributed value, you can use `tf.distribute.DistributedDataset.element_spec` or `tf.distribute.DistributedIterator.element_spec`." ] }, { @@ -604,7 +634,7 @@ "steps_per_epoch = 5\n", "mirrored_strategy = tf.distribute.MirroredStrategy()\n", "\n", - "dataset = tf.data.Dataset.from_tensors(([1.],[1.])).repeat(100).batch(global_batch_size)\n", + "dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(100).batch(global_batch_size)\n", "dist_dataset = mirrored_strategy.experimental_distribute_dataset(dataset)\n", "\n", "@tf.function(input_signature=[dist_dataset.element_spec])\n", @@ -627,7 +657,7 @@ "id": "-OAa6svUzuWm" }, "source": [ - "## Data Preprocessing" + "## Data preprocessing" ] }, { @@ -636,9 +666,9 @@ "id": "pSMrs3kJQexW" }, "source": [ - "So far, we have discussed how to distribute a `tf.data.Dataset`. Yet before the data is ready for the model, we have the crucial step of preprocessing the data, e.g., cleansing, transforming, augmenting. Two sets of those handy tools are:\n", + "So far, you have learned how to distribute a `tf.data.Dataset`. Yet before the data is ready for the model, it needs to be preprocessed, for example by cleansing, transforming, and augmenting it. Two sets of those handy tools are:\n", "\n", - "* [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers): a set of Keras layers that allow developers to build Keras-native input processing pipelines. Some Keras preprocessing layers contain non-trainable states, which can be set on initialization or [\"adapted\"](https://www.tensorflow.org/guide/keras/preprocessing_layers#the_adapt_method). When distributing stateful preprocessing layers, we want the states replicated to all workers. To use these layers, you can either make them part of the model or apply them to the datasets.\n", + "* [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers): a set of Keras layers that allow developers to build Keras-native input processing pipelines. Some Keras preprocessing layers contain non-trainable states, which can be set on initialization or `adapt`ed (refer to the `adapt` section of the [Keras preprocessing layers guide](https://www.tensorflow.org/guide/keras/preprocessing_layers)). When distributing stateful preprocessing layers, the states should be replicated to all workers. To use these layers, you can either make them part of the model or apply them to the datasets.\n", "\n", "* [TensorFlow Transform (tf.Transform)](https://www.tensorflow.org/tfx/transform/get_started): a library for TensorFlow that allows you to define both instance-level and full-pass data transformation through data preprocessing pipelines. Tensorflow Transform has two phases. The first is the Analyze phase, where the raw training data is analyzed in a full-pass process to compute the statistics needed for the transformations, and the transformation logic is generated as instance-level operations. The second is the Transform phase, where the raw training data is transformed in an instance-level process.\n", "\n" @@ -670,9 +700,9 @@ "source": [ "### Best Practice with tf.distribute\n", "\n", - "Working with both tools involves initializing the transformation logic to apply to data, which might create Tensorflow resources. We would want these resources or states replicated to all workers to save inter-workers or worker-coordinator communication. To do so, we recommend you create Keras preprocessing layers, `tft.TFTransformOutput.transform_features_layer`, or `tft.TransformFeaturesLayer` under `tf.distribute.Strategy.scope()`, just like you would for any other Keras layers.\n", + "Working with both tools involves initializing the transformation logic to apply to data, which might create Tensorflow resources. These resources or states should be replicated to all workers to save inter-workers or worker-coordinator communication. To do so, you are recommended to create Keras preprocessing layers, `tft.TFTransformOutput.transform_features_layer`, or `tft.TransformFeaturesLayer` under `tf.distribute.Strategy.scope`, just like you would for any other Keras layers.\n", "\n", - "We will demonstrate examples with the high-level Keras `Model.fit` API and the custom training loop separately." + "The following examples demonstrate usage of the `tf.distribute.Strategy` API with the high-level Keras `Model.fit` API and with a custom training loop separately." ] }, { @@ -685,16 +715,16 @@ "\n", "**Preprocessing layers and large vocabularies**\n", "\n", - "When dealing with large vocabularies (over one gigabyte) in a multi-worker setting (i.e., `tf.distribute.MultiWorkerMirroredStrategy`, `tf.distribute.experimental.ParameterServerStrategy`, `tf.distribute.TPUStrategy`), we recommend saving the vocabulary to a static file accessible from all workers (e.g., with Cloud Storage). This will reduce the time spent replicating the vocabulary to all workers during training.\n", + "When dealing with large vocabularies (over one gigabyte) in a multi-worker setting (for example, `tf.distribute.MultiWorkerMirroredStrategy`, `tf.distribute.experimental.ParameterServerStrategy`, `tf.distribute.TPUStrategy`), it is recommended to save the vocabulary to a static file accessible from all workers (for example, with Cloud Storage). This will reduce the time spent replicating the vocabulary to all workers during training.\n", "\n", - "**Preprocessing in data pipeline vs. in model**\n", + "**Preprocessing in the `tf.data` pipeline versus in the model**\n", "\n", - "While Keras preprocessing layers can be applied either as part of the model or directly to a `tf.data.Dataset`, each of the options come with their edge:\n", + "While Keras preprocessing layers can be applied either as part of the model or directly to a `tf.data.Dataset`, each of the options come with their edge:\n", "\n", - "* Applying in the model makes your model portable, and it helps reduce the training/serving skew. ([more details](https://www.tensorflow.org/guide/keras/preprocessing_layers#benefits_of_doing_preprocessing_inside_the_model_at_inference_time))\n", - "* Applying in the `tf.data` pipeline allows prefetching or offloading to the CPU, which generally gives better performance when using accelerators.\n", + "* Applying the preprocessing layers within the model makes your model portable, and it helps reduce the training/serving skew. (For more details, refer to the _Benefits of doing preprocessing inside the model at inference time_ section in the [Working with preprocessing layers guide](https://www.tensorflow.org/guide/keras/preprocessing_layers#benefits_of_doing_preprocessing_inside_the_model_at_inference_time))\n", + "* Applying within the `tf.data` pipeline allows prefetching or offloading to the CPU, which generally gives better performance when using accelerators.\n", "\n", - "When running on TPU, users should almost always place preprocessing layers in the `tf.data` pipeline, as not all layers support TPU, and string ops do not execute on TPU. (The two exceptions are `Normalization` and `Rescaling`, which run fine on TPU and are commonly used as the first layer is an image model.)" + "When running on one or more TPUs, users should almost always place Keras preprocessing layers in the `tf.data` pipeline, as not all layers support TPUs, and string ops do not execute on TPUs. (The two exceptions are `tf.keras.layers.Normalization` and `tf.keras.layers.Rescaling`, which run fine on TPUs and are commonly used as the first layer in an image model.)" ] }, { @@ -712,7 +742,7 @@ "id": "NhRB2Xe8B6bX" }, "source": [ - "Users of Keras `Model.fit` do not need to distribute data with `tf.distribute.Strategy.experimental_distribute_dataset` nor `tf.distribute.Strategy.distribute_datasets_from_function` themselves. Check out the [Working with Preprocessing Layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) guide and [Distributed Training with Keras](https://www.tensorflow.org/tutorials/distribute/keras) guide for details. A shortened example may look as below:\n", + "When using Keras `Model.fit`, you do not need to distribute data with `tf.distribute.Strategy.experimental_distribute_dataset` nor `tf.distribute.Strategy.distribute_datasets_from_function` themselves. Check out the [Working with preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) guide and the [Distributed training with Keras](https://www.tensorflow.org/tutorials/distribute/keras) guide for details. A shortened example may look as below:\n", "\n", "```\n", "strategy = tf.distribute.MirroredStrategy()\n", @@ -771,7 +801,7 @@ "id": "r2PX1QH_OwU3" }, "source": [ - "When writing a [custom training loop](https://www.tensorflow.org/tutorials/distribute/custom_training), you will distribute your data with either the `tf.distribute.Strategy.experimental_distribute_dataset` API or the `tf.distribute.Strategy.distribute_datasets_from_function` API. If you distribute your dataset through `tf.distribute.Strategy.experimental_distribute_dataset`, applying these preprocessing APIs in your data pipeline will lead the resources automatically co-located with the data pipeline to avoid remote resource access. Thus we will only demonstrate examples with `tf.distribute.Strategy.distribute_datasets_from_function`, in which case it is crucial to place initialization of these APIs under `strategy.scope()` for efficiency:" + "When writing a [custom training loop](https://www.tensorflow.org/tutorials/distribute/custom_training), you will distribute your data with either the `tf.distribute.Strategy.experimental_distribute_dataset` API or the `tf.distribute.Strategy.distribute_datasets_from_function` API. If you distribute your dataset through `tf.distribute.Strategy.experimental_distribute_dataset`, applying these preprocessing APIs in your data pipeline will lead the resources automatically co-located with the data pipeline to avoid remote resource access. Thus the examples here will all use `tf.distribute.Strategy.distribute_datasets_from_function`, in which case it is crucial to place initialization of these APIs under `strategy.scope()` for efficiency:" ] }, { @@ -866,7 +896,7 @@ "id": "3_IQxRXxQWof" }, "source": [ - "## Partial Batches" + "## Partial batches" ] }, { @@ -875,7 +905,7 @@ "id": "hW2_gVkiztUG" }, "source": [ - "Partial batches are encountered when `tf.data.Dataset` instances that users create may contain batch sizes that are not evenly divisible by the number of replicas or when the cardinality of the dataset instance is not divisible by the batch size. This means that when the dataset is distributed over multiple replicas, the `next` call on some iterators will result in an OutOfRangeError. To handle this use case, `tf.distribute` returns dummy batches of batch size 0 on replicas that do not have any more data to process.\n" + "Partial batches are encountered when: 1) `tf.data.Dataset` instances that users create may contain batch sizes that are not evenly divisible by the number of replicas; or 2) when the cardinality of the dataset instance is not divisible by the batch size. This means that when the dataset is distributed over multiple replicas, the `next` call on some iterators will result in an `tf.errors.OutOfRangeError`. To handle this use case, `tf.distribute` returns dummy batches of batch size `0` on replicas that do not have any more data to process.\n" ] }, { @@ -884,9 +914,9 @@ "id": "rqutdpqtPcCH" }, "source": [ - "For the single worker case, if data is not returned by the `next` call on the iterator, dummy batches of 0 batch size are created and used along with the real data in the dataset. In the case of partial batches, the last global batch of data will contain real data alongside dummy batches of data. The stopping condition for processing data now checks if any of the replicas have data. If there is no data on any of the replicas, an OutOfRange error is thrown.\n", + "For the single-worker case, if the data is not returned by the `next` call on the iterator, dummy batches of 0 batch size are created and used along with the real data in the dataset. In the case of partial batches, the last global batch of data will contain real data alongside dummy batches of data. The stopping condition for processing data now checks if any of the replicas have data. If there is no data on any of the replicas, you will get a `tf.errors.OutOfRangeError`.\n", "\n", - "For the multi worker case, the boolean value representing presence of data on each of the workers is aggregated using cross replica communication and this is used to identify if all the workers have finished processing the distributed dataset. Since this involves cross worker communication there is some performance penalty involved.\n" + "For the multi-worker case, the boolean value representing presence of data on each of the workers is aggregated using cross replica communication and this is used to identify if all the workers have finished processing the distributed dataset. Since this involves cross worker communication there is some performance penalty involved.\n" ] }, { @@ -904,11 +934,11 @@ "id": "Nx4jyN_Az-Dy" }, "source": [ - "* When using `tf.distribute.Strategy.experimental_distribute_dataset` APIs with a multiple worker setup, users pass a `tf.data.Dataset` that reads from files. If the `tf.data.experimental.AutoShardPolicy` is set to `AUTO` or `FILE`, the actual per step batch size may be smaller than the user defined global batch size. This can happen when the remaining elements in the file are less than the global batch size. Users can either exhaust the dataset without depending on the number of steps to run or set `tf.data.experimental.AutoShardPolicy` to `DATA` to work around it.\n", + "* When using `tf.distribute.Strategy.experimental_distribute_dataset` APIs with a multi-worker setup, you pass a `tf.data.Dataset` that reads from files. If the `tf.data.experimental.AutoShardPolicy` is set to `AUTO` or `FILE`, the actual per-step batch size may be smaller than the one you defined for the global batch size. This can happen when the remaining elements in the file are less than the global batch size. You can either exhaust the dataset without depending on the number of steps to run, or set `tf.data.experimental.AutoShardPolicy` to `DATA` to work around it.\n", "\n", "* Stateful dataset transformations are currently not supported with `tf.distribute` and any stateful ops that the dataset may have are currently ignored. For example, if your dataset has a `map_fn` that uses `tf.random.uniform` to rotate an image, then you have a dataset graph that depends on state (i.e the random seed) on the local machine where the python process is being executed.\n", "\n", - "* Experimental `tf.data.experimental.OptimizationOptions` that are disabled by default can in certain contexts -- such as when used together with `tf.distribute` -- cause a performance degradation. You should only enable them after you validate that they benefit the performance of your workload in a distribute setting.\n", + "* Experimental `tf.data.experimental.OptimizationOptions` that are disabled by default can in certain contexts—such as when used together with `tf.distribute`—cause a performance degradation. You should only enable them after you validate that they benefit the performance of your workload in a distribute setting.\n", "\n", "* Please refer to [this guide](https://www.tensorflow.org/guide/data_performance) for how to optimize your input pipeline with `tf.data` in general. A few additional tips:\n", " * If you have multiple workers and are using `tf.data.Dataset.list_files` to create a dataset from all files matching one or more glob patterns, remember to set the `seed` argument or set `shuffle=False` so that each worker shard the file consistently.\n", @@ -934,7 +964,7 @@ "source": [ "* The order in which the data is processed by the workers when using `tf.distribute.experimental_distribute_dataset` or `tf.distribute.distribute_datasets_from_function` is not guaranteed. This is typically required if you are using `tf.distribute` to scale prediction. You can however insert an index for each element in the batch and order outputs accordingly. The following snippet is an example of how to order outputs.\n", "\n", - "Note: `tf.distribute.MirroredStrategy()` is used here for the sake of convenience. We only need to reorder inputs when we are using multiple workers and `tf.distribute.MirroredStrategy` is used to distribute training on a single worker." + "Note: `tf.distribute.MirroredStrategy` is used here for the sake of convenience. You only need to reorder inputs when you are using multiple workers, but `tf.distribute.MirroredStrategy` is used to distribute training on a single worker." ] }, { @@ -979,7 +1009,7 @@ }, "source": [ "\n", - "## How do I distribute my data if I am not using a canonical tf.data.Dataset instance?" + "## Tensor inputs instead of tf.data" ] }, { @@ -995,8 +1025,8 @@ "### Use experimental_distribute_values_from_function for arbitrary tensor inputs\n", "`strategy.run` accepts `tf.distribute.DistributedValues` which is the output of\n", "`next(iterator)`. To pass the tensor values, use\n", - "`experimental_distribute_values_from_function` to construct\n", - "`tf.distribute.DistributedValues` from raw tensors." + "`tf.distribute.Strategy.experimental_distribute_values_from_function` to construct\n", + "`tf.distribute.DistributedValues` from raw tensors. The user will have to specify their own batching and sharding logic in the input function with this option, which can be done using the `tf.distribute.experimental.ValueContext` input object." ] }, { @@ -1008,14 +1038,13 @@ "outputs": [], "source": [ "mirrored_strategy = tf.distribute.MirroredStrategy()\n", - "worker_devices = mirrored_strategy.extended.worker_devices\n", "\n", "def value_fn(ctx):\n", - " return tf.constant(1.0)\n", + " return tf.constant(ctx.replica_id_in_sync_group)\n", "\n", "distributed_values = mirrored_strategy.experimental_distribute_values_from_function(value_fn)\n", "for _ in range(4):\n", - " result = mirrored_strategy.run(lambda x:x, args=(distributed_values,))\n", + " result = mirrored_strategy.run(lambda x: x, args=(distributed_values,))\n", " print(result)" ] }, @@ -1058,7 +1087,8 @@ "dist_dataset = mirrored_strategy.experimental_distribute_dataset(dataset)\n", "iterator = iter(dist_dataset)\n", "for _ in range(4):\n", - " mirrored_strategy.run(lambda x:x, args=(next(iterator),))" + " result = mirrored_strategy.run(lambda x: x, args=(next(iterator),))\n", + " print(result)" ] } ], From 8aea6cc53d01185bac7eadc631b22ed5cf5be010 Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Tue, 29 Mar 2022 15:22:18 -0700 Subject: [PATCH 043/872] Update distributed model saving/loading tutorial: Add example showing how saving works after calling `.fit`. Also fix some typos, linting, adding some minor details. PiperOrigin-RevId: 438140288 --- .../tutorials/distribute/save_and_load.ipynb | 164 +++++++++++++----- 1 file changed, 123 insertions(+), 41 deletions(-) diff --git a/site/en/tutorials/distribute/save_and_load.ipynb b/site/en/tutorials/distribute/save_and_load.ipynb index 6b398b372ee..79e09b961b6 100644 --- a/site/en/tutorials/distribute/save_and_load.ipynb +++ b/site/en/tutorials/distribute/save_and_load.ipynb @@ -71,7 +71,9 @@ "source": [ "## Overview\n", "\n", - "It's common to save and load a model during training. There are two sets of APIs for saving and loading a keras model: a high-level API, and a low-level API. This tutorial demonstrates how you can use the SavedModel APIs when using `tf.distribute.Strategy`. To learn about SavedModel and serialization in general, please read the [saved model guide](../../guide/saved_model.ipynb), and the [Keras model serialization guide](https://www.tensorflow.org/guide/keras/save_and_serialize). Let's start with a simple example: " + "This tutorial demonstrates how you can save and load models in a SavedModel format with `tf.distribute.Strategy` during or after training. There are two kinds of APIs for saving and loading a Keras model: high-level (`tf.keras.Model.save` and `tf.keras.models.load_model`) and low-level (`tf.saved_model.save` and `tf.saved_model.load`).\n", + "\n", + "To learn about SavedModel and serialization in general, please read the [saved model guide](../../guide/saved_model.ipynb), and the [Keras model serialization guide](https://www.tensorflow.org/guide/keras/save_and_serialize). Let's start with a simple example: " ] }, { @@ -102,7 +104,7 @@ "id": "qqapWj98ptNV" }, "source": [ - "Prepare the data and model using `tf.distribute.Strategy`:" + "Load and prepare the data with TensorFlow Datasets and `tf.data`, and create the model using `tf.distribute.MirroredStrategy`:" ] }, { @@ -116,7 +118,7 @@ "mirrored_strategy = tf.distribute.MirroredStrategy()\n", "\n", "def get_data():\n", - " datasets, ds_info = tfds.load(name='mnist', with_info=True, as_supervised=True)\n", + " datasets = tfds.load(name='mnist', as_supervised=True)\n", " mnist_train, mnist_test = datasets['train'], datasets['test']\n", "\n", " BUFFER_SIZE = 10000\n", @@ -157,7 +159,7 @@ "id": "qmU4Y3feS9Na" }, "source": [ - "Train the model: " + "Train the model with `tf.keras.Model.fit`: " ] }, { @@ -181,11 +183,11 @@ "source": [ "## Save and load the model\n", "\n", - "Now that you have a simple model to work with, let's take a look at the saving/loading APIs. \n", - "There are two sets of APIs available:\n", + "Now that you have a simple model to work with, let's explore the saving/loading APIs. \n", + "There are two kinds of APIs available:\n", "\n", - "* High level keras `model.save` and `tf.keras.models.load_model`\n", - "* Low level `tf.saved_model.save` and `tf.saved_model.load`\n" + "* High-level (Keras): `Model.save` and `tf.keras.models.load_model`\n", + "* Low-level: `tf.saved_model.save` and `tf.saved_model.load`\n" ] }, { @@ -194,7 +196,7 @@ "id": "FX_IF2F1tvFs" }, "source": [ - "### The Keras APIs" + "### The Keras API" ] }, { @@ -203,7 +205,7 @@ "id": "O8xfceg4Z3H_" }, "source": [ - "Here is an example of saving and loading a model with the Keras APIs:" + "Here is an example of saving and loading a model with the Keras API:" ] }, { @@ -214,7 +216,7 @@ }, "outputs": [], "source": [ - "keras_model_path = \"/tmp/keras_save\"\n", + "keras_model_path = '/tmp/keras_save'\n", "model.save(keras_model_path)" ] }, @@ -245,9 +247,9 @@ "id": "gYAnskzorda-" }, "source": [ - "After restoring the model, you can continue training on it, even without needing to call `compile()` again, since it is already compiled before saving. The model is saved in the TensorFlow's standard `SavedModel` proto format. For more information, please refer to [the guide to `saved_model` format](../../guide/saved_model.ipynb).\n", + "After restoring the model, you can continue training on it, even without needing to call `Model.compile` again, since it was already compiled before saving. The model is saved in TensorFlow's standard `SavedModel` proto format. For more information, please refer to [the guide to `SavedModel` format](../../guide/saved_model.ipynb).\n", "\n", - "Now to load the model and train it using a `tf.distribute.Strategy`:" + "Now, restore the model and train it using a `tf.distribute.Strategy`:" ] }, { @@ -258,7 +260,7 @@ }, "outputs": [], "source": [ - "another_strategy = tf.distribute.OneDeviceStrategy(\"/cpu:0\")\n", + "another_strategy = tf.distribute.OneDeviceStrategy('/cpu:0')\n", "with another_strategy.scope():\n", " restored_keras_model_ds = tf.keras.models.load_model(keras_model_path)\n", " restored_keras_model_ds.fit(train_dataset, epochs=2)" @@ -270,7 +272,7 @@ "id": "PdiiPmL5tQk5" }, "source": [ - "As you can see, loading works as expected with `tf.distribute.Strategy`. The strategy used here does not have to be the same strategy used before saving. " + "As the `Model.fit` output shows, loading works as expected with `tf.distribute.Strategy`. The strategy used here does not have to be the same strategy used before saving. " ] }, { @@ -279,7 +281,7 @@ "id": "3CrXIbmFt0f6" }, "source": [ - "### The `tf.saved_model` APIs" + "### The `tf.saved_model` API" ] }, { @@ -288,7 +290,7 @@ "id": "HtGzPp6et4Em" }, "source": [ - "Now let's take a look at the lower level APIs. Saving the model is similar to the keras API:" + "Saving the model with lower-level API is similar to the Keras API:" ] }, { @@ -300,7 +302,7 @@ "outputs": [], "source": [ "model = get_model() # get a fresh model\n", - "saved_model_path = \"/tmp/tf_save\"\n", + "saved_model_path = '/tmp/tf_save'\n", "tf.saved_model.save(model, saved_model_path)" ] }, @@ -310,7 +312,7 @@ "id": "q1QNRYcwuRll" }, "source": [ - "Loading can be done with `tf.saved_model.load()`. However, since it is an API that is on the lower level (and hence has a wider range of use cases), it does not return a Keras model. Instead, it returns an object that contain functions that can be used to do inference. For example:" + "Loading can be done with `tf.saved_model.load`. However, since it is a lower-level API (and hence has a wider range of use cases), it does not return a Keras model. Instead, it returns an object that contain functions that can be used to do inference. For example:" ] }, { @@ -321,7 +323,7 @@ }, "outputs": [], "source": [ - "DEFAULT_FUNCTION_KEY = \"serving_default\"\n", + "DEFAULT_FUNCTION_KEY = 'serving_default'\n", "loaded = tf.saved_model.load(saved_model_path)\n", "inference_func = loaded.signatures[DEFAULT_FUNCTION_KEY]" ] @@ -332,7 +334,7 @@ "id": "x65l7AaHUZCA" }, "source": [ - "The loaded object may contain multiple functions, each associated with a key. The `\"serving_default\"` is the default key for the inference function with a saved Keras model. To do an inference with this function: " + "The loaded object may contain multiple functions, each associated with a key. The `\"serving_default\"` key is the default key for the inference function with a saved Keras model. To do inference with this function: " ] }, { @@ -375,7 +377,9 @@ "\n", " # Calling the function in a distributed manner\n", " for batch in dist_predict_dataset:\n", - " another_strategy.run(inference_func,args=(batch,))" + " result = another_strategy.run(inference_func, args=(batch,))\n", + " print(result)\n", + " break" ] }, { @@ -384,7 +388,7 @@ "id": "hWGSukoyw3fF" }, "source": [ - "Calling the restored function is just a forward pass on the saved model (predict). What if yout want to continue training the loaded function? Or embed the loaded function into a bigger model? A common practice is to wrap this loaded object to a Keras layer to achieve this. Luckily, [TF Hub](https://www.tensorflow.org/hub) has [hub.KerasLayer](https://github.com/tensorflow/hub/blob/master/tensorflow_hub/keras_layer.py) for this purpose, shown here:" + "Calling the restored function is just a forward pass on the saved model (`tf.keras.Model.predict`). What if you want to continue training the loaded function? Or what if you need to embed the loaded function into a bigger model? A common practice is to wrap this loaded object into a Keras layer to achieve this. Luckily, [TF Hub](https://www.tensorflow.org/hub) has [`hub.KerasLayer`](https://github.com/tensorflow/hub/blob/master/tensorflow_hub/keras_layer.py) for this purpose, shown here:" ] }, { @@ -421,7 +425,7 @@ "id": "Oe1z_OtSJlu2" }, "source": [ - "As you can see, `hub.KerasLayer` wraps the result loaded back from `tf.saved_model.load()` into a Keras layer that can be used to build another model. This is very useful for transfer learning. " + "In the above example, Tensorflow Hub's `hub.KerasLayer` wraps the result loaded back from `tf.saved_model.load` into a Keras layer that is used to build another model. This is very useful for transfer learning. " ] }, { @@ -439,11 +443,11 @@ "id": "GC6GQ9HDLxD6" }, "source": [ - "For saving, if you are working with a keras model, it is almost always recommended to use the Keras's `model.save()` API. If what you are saving is not a Keras model, then the lower level API is your only choice. \n", + "For saving, if you are working with a Keras model, use the Keras `Model.save` API unless you need the additional control allowed by the low-level API. If what you are saving is not a Keras model, then the lower-level API, `tf.saved_model.save`, is your only choice. \n", "\n", - "For loading, which API you use depends on what you want to get from the loading API. If you cannot (or do not want to) get a Keras model, then use `tf.saved_model.load()`. Otherwise, use `tf.keras.models.load_model()`. Note that you can get a Keras model back only if you saved a Keras model. \n", + "For loading, your API choice depends on what you want to get from the model loading API. If you cannot (or do not want to) get a Keras model, then use `tf.saved_model.load`. Otherwise, use `tf.keras.models.load_model`. Note that you can get a Keras model back only if you saved a Keras model. \n", "\n", - "It is possible to mix and match the APIs. You can save a Keras model with `model.save`, and load a non-Keras model with the low-level API, `tf.saved_model.load`. " + "It is possible to mix and match the APIs. You can save a Keras model with `Model.save`, and load a non-Keras model with the low-level API, `tf.saved_model.load`. " ] }, { @@ -456,11 +460,11 @@ "source": [ "model = get_model()\n", "\n", - "# Saving the model using Keras's save() API\n", - "model.save(keras_model_path) \n", + "# Saving the model using Keras `Model.save`\n", + "model.save(keras_model_path)\n", "\n", "another_strategy = tf.distribute.MirroredStrategy()\n", - "# Loading the model using lower level API\n", + "# Loading the model using the lower-level API\n", "with another_strategy.scope():\n", " loaded = tf.saved_model.load(keras_model_path)" ] @@ -471,7 +475,7 @@ "id": "0Z7lSj8nZiW5" }, "source": [ - "### Saving/Loading from local device" + "### Saving/Loading from a local device" ] }, { @@ -480,7 +484,7 @@ "id": "NVAjWcosZodw" }, "source": [ - "When saving and loading from a local io device while running remotely, for example using a cloud TPU, the option `experimental_io_device` must be used to set the io device to localhost." + "When saving and loading from a local I/O device while training on remote devices—for example, when using a Cloud TPU—you must use the option `experimental_io_device` in `tf.saved_model.SaveOptions` and `tf.saved_model.LoadOptions` to set the I/O device to `localhost`. For example:" ] }, { @@ -494,7 +498,7 @@ "model = get_model()\n", "\n", "# Saving the model to a path on localhost.\n", - "saved_model_path = \"/tmp/tf_save\"\n", + "saved_model_path = '/tmp/tf_save'\n", "save_options = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\n", "model.save(saved_model_path, options=save_options)\n", "\n", @@ -517,14 +521,10 @@ { "cell_type": "markdown", "metadata": { - "id": "Tzog2ti7YYgy" + "id": "2cCSZrD7VCxe" }, "source": [ - "A special case is when you have a Keras model that does not have well-defined inputs. For example, a Sequential model can be created without any input shapes (`Sequential([Dense(3), ...]`). Subclassed models also do not have well-defined inputs after initialization. In this case, you should stick with the lower level APIs on both saving and loading, otherwise you will get an error. \n", - "\n", - "To check if your model has well-defined inputs, just check if `model.inputs` is `None`. If it is not `None`, you are all good. Input shapes are automatically defined when the model is used in `.fit`, `.evaluate`, `.predict`, or when calling the model (`model(inputs)`). \n", - "\n", - "Here is an example:" + "One special case is when you create Keras models in certain ways, and then save them before training. For example:" ] }, { @@ -536,6 +536,7 @@ "outputs": [], "source": [ "class SubclassedModel(tf.keras.Model):\n", + " \"\"\"Example model defined by subclassing `tf.keras.Model`.\"\"\"\n", "\n", " output_name = 'output_layer'\n", "\n", @@ -548,8 +549,89 @@ " return self._dense_layer(inputs)\n", "\n", "my_model = SubclassedModel()\n", - "# my_model.save(keras_model_path) # ERROR! \n", - "tf.saved_model.save(my_model, saved_model_path)" + "try:\n", + " my_model.save(keras_model_path)\n", + "except ValueError as e:\n", + " print(f'{type(e).__name__}: ', *e.args)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "D4qMyXFDSPDO" + }, + "source": [ + "A SavedModel saves the `tf.types.experimental.ConcreteFunction` objects generated when you trace a `tf.function` (check _When is a Function tracing?_ in the [Introduction to graphs and tf.function](../../guide/intro_to_graphs.ipynb) guide to learn more). If you get a `ValueError` like this it's because `Model.save` was not able to find or create a traced `ConcreteFunction`.\n", + "\n", + "**Caution:** You shouldn't save a model without at least one `ConcreteFunction`, since the low-level API will otherwise generate a SavedModel with no `ConcreteFunction` signatures ([learn more](../../guide/saved_model.ipynb) about the SavedModel format). For example:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "064SE47mYDj8" + }, + "outputs": [], + "source": [ + "tf.saved_model.save(my_model, saved_model_path)\n", + "x = tf.saved_model.load(saved_model_path)\n", + "x.signatures" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LRTxlASJX-cY" + }, + "source": [ + "\n", + "Usually the model's forward pass—the `call` method—will be traced automatically when the model is called for the first time, often via the Keras `Model.fit` method. A `ConcreteFunction` can also be generated by the Keras [Sequential](https://www.tensorflow.org/guide/keras/sequential_model) and [Functional](https://www.tensorflow.org/guide/keras/functional) APIs, if you set the input shape, for example, by making the first layer either a `tf.keras.layers.InputLayer` or another layer type, and passing it the `input_shape` keyword argument. \n", + "\n", + "To verify if your model has any traced `ConcreteFunction`s, check if `Model.save_spec` is `None`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xAXise4eR0YJ" + }, + "outputs": [], + "source": [ + "print(my_model.save_spec() is None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "G2G_FQrWJAO3" + }, + "source": [ + "Let's use `tf.keras.Model.fit` to train the model, and notice that the `save_spec` gets defined and model saving will work:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cv5LTi0zDkKS" + }, + "outputs": [], + "source": [ + "BATCH_SIZE_PER_REPLICA = 4\n", + "BATCH_SIZE = BATCH_SIZE_PER_REPLICA * mirrored_strategy.num_replicas_in_sync\n", + "\n", + "dataset_size = 100\n", + "dataset = tf.data.Dataset.from_tensors(\n", + " (tf.range(5, dtype=tf.float32), tf.range(5, dtype=tf.float32))\n", + " ).repeat(dataset_size).batch(BATCH_SIZE)\n", + "\n", + "my_model.compile(optimizer='adam', loss='mean_squared_error')\n", + "my_model.fit(dataset, epochs=2)\n", + "\n", + "print(my_model.save_spec() is None)\n", + "my_model.save(keras_model_path)" ] } ], From 63d3f5b5dedd0e25b6bf656f311f1bf988455e7d Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 30 Mar 2022 04:39:25 -0700 Subject: [PATCH 044/872] Standardize the use of inspect.getsource PiperOrigin-RevId: 438269328 --- .../api_generator/get_source.py | 40 +++++++++++++++++++ tools/tensorflow_docs/api_generator/parser.py | 10 ++--- .../api_generator/public_api.py | 8 ++-- .../api_generator/report/linter.py | 13 ++---- .../api_generator/signature.py | 19 +++++---- 5 files changed, 61 insertions(+), 29 deletions(-) create mode 100644 tools/tensorflow_docs/api_generator/get_source.py diff --git a/tools/tensorflow_docs/api_generator/get_source.py b/tools/tensorflow_docs/api_generator/get_source.py new file mode 100644 index 00000000000..d85c5ef88e6 --- /dev/null +++ b/tools/tensorflow_docs/api_generator/get_source.py @@ -0,0 +1,40 @@ +# Copyright 2015 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +"""Simple get_source.""" +import inspect +import textwrap + +from typing import Any, Optional, Sequence, Tuple + + +def get_source(py_object: Any) -> Optional[str]: + if py_object is not None: + try: + return textwrap.dedent(inspect.getsource(py_object)) + except Exception: # pylint: disable=broad-except + # A wide-variety of errors can be thrown here. + pass + return None + + +def get_source_lines( + py_object: Any) -> Tuple[Optional[Sequence[str]], Optional[int]]: + if py_object is not None: + try: + return inspect.getsourcelines(py_object) + except Exception: # pylint: disable=broad-except + # A wide-variety of errors can be thrown here. + pass + return None, None diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py index 2ca49c05dbf..5dd25b8f90d 100644 --- a/tools/tensorflow_docs/api_generator/parser.py +++ b/tools/tensorflow_docs/api_generator/parser.py @@ -28,6 +28,7 @@ from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_generator_visitor +from tensorflow_docs.api_generator import get_source from tensorflow_docs.api_generator import obj_type as obj_type_lib from tensorflow_docs.api_generator import signature as signature_lib @@ -653,15 +654,14 @@ def get_defined_in( if code_url_prefix is None: return None - try: - lines, start_line = inspect.getsourcelines(py_object) + lines, start_line = get_source.get_source_lines(py_object) + if start_line is None: + end_line = None + else: end_line = start_line + len(lines) - 1 if 'MACHINE GENERATED' in lines[0]: # don't link to files generated by tf_export return None - except (IOError, TypeError, IndexError): - start_line = None - end_line = None # In case this is compiled, point to the original if posix_rel_path_str.endswith('.pyc'): diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py index c39026e5e33..c4846900c55 100644 --- a/tools/tensorflow_docs/api_generator/public_api.py +++ b/tools/tensorflow_docs/api_generator/public_api.py @@ -23,6 +23,7 @@ from tensorflow_docs.api_generator import doc_controls from tensorflow_docs.api_generator import doc_generator_visitor +from tensorflow_docs.api_generator import get_source _TYPING_IDS = frozenset( id(obj) @@ -184,11 +185,10 @@ def visit_Import(self, node): # pylint: disable=invalid-name def visit_ImportFrom(self, node): # pylint: disable=invalid-name self._add_imported_symbol(node) - try: - source = inspect.getsource(obj) - except OSError: - # inspect raises an OSError for empty module files. + source = get_source.get_source(obj) + if source is None: return [] + tree = ast.parse(source) visitor = ImportNodeVisitor() visitor.visit(tree) diff --git a/tools/tensorflow_docs/api_generator/report/linter.py b/tools/tensorflow_docs/api_generator/report/linter.py index aade7e4e50d..6752d9bee88 100644 --- a/tools/tensorflow_docs/api_generator/report/linter.py +++ b/tools/tensorflow_docs/api_generator/report/linter.py @@ -23,19 +23,12 @@ import astor +from tensorflow_docs.api_generator import get_source from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator.pretty_docs import base_page from tensorflow_docs.api_generator.report.schema import api_report_generated_pb2 as api_report_pb2 -def _get_source(py_object: Any) -> Optional[str]: - if py_object is not None: - try: - source = textwrap.dedent(inspect.getsource(py_object)) - return source - except Exception: # pylint: disable=broad-except - return None - return None def _count_empty_param(items: List[Tuple[str, Optional[str]]]) -> int: @@ -183,7 +176,7 @@ def lint_returns( Returns: A filled `ReturnLint` proto object. """ - source = _get_source(page_info.py_object) + source = get_source.get_source(page_info.py_object) return_visitor = ReturnVisitor() if source is not None: @@ -243,7 +236,7 @@ def lint_raises(page_info: base_page.PageInfo) -> api_report_pb2.RaisesLint: # Extract the raises from the source code. raise_visitor = RaiseVisitor() - source = _get_source(page_info.py_object) + source = get_source.get_source(page_info.py_object) if source is not None: try: raise_visitor.visit(ast.parse(source)) diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py index 2ac88f11b54..cc6d46b0d62 100644 --- a/tools/tensorflow_docs/api_generator/signature.py +++ b/tools/tensorflow_docs/api_generator/signature.py @@ -30,6 +30,7 @@ import astor from tensorflow_docs.api_generator import config +from tensorflow_docs.api_generator import get_source from tensorflow_docs.api_generator import public_api EMPTY = inspect.Signature.empty @@ -55,9 +56,10 @@ def _preprocess_default(self, val: ast.AST) -> str: return text_default_val def extract(self, obj: Any): - obj_source = textwrap.dedent(inspect.getsource(obj)) - obj_ast = ast.parse(obj_source) - self.visit(obj_ast) + obj_source = get_source.get_source(obj) + if obj_source is not None: + obj_ast = ast.parse(obj_source) + self.visit(obj_ast) class _ArgDefaultAndAnnotationExtractor(_BaseDefaultAndAnnotationExtractor): @@ -672,14 +674,11 @@ def visit_FunctionDef(self, node): # pylint: disable=invalid-name visitor = ASTDecoratorExtractor() - try: - # Note: inspect.getsource doesn't include the decorator lines on classes, - # this won't work for classes until that's fixed. - func_source = textwrap.dedent(inspect.getsource(func)) + # Note: inspect.getsource doesn't include the decorator lines on classes, + # this won't work for classes until that's fixed. + func_source = get_source.get_source(func) + if func_source is not None: func_ast = ast.parse(func_source) visitor.visit(func_ast) - except Exception: # pylint: disable=broad-except - # A wide-variety of errors can be thrown here. - pass return visitor.decorator_list From a5dd8de8d4f26b58da4b11ba7c9c0c7dab2af939 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 30 Mar 2022 05:15:11 -0700 Subject: [PATCH 045/872] Clarify docs for `explicit_package_contents_filter`. PiperOrigin-RevId: 438274763 --- .../api_generator/public_api.py | 78 +++++++++++++++---- .../api_generator/public_api_test.py | 15 ++++ 2 files changed, 78 insertions(+), 15 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py index c4846900c55..9603d85ba54 100644 --- a/tools/tensorflow_docs/api_generator/public_api.py +++ b/tools/tensorflow_docs/api_generator/public_api.py @@ -18,8 +18,11 @@ import inspect import os import pathlib +import textwrap +import types import typing -from typing import Any, Callable, List, Sequence, Tuple +from typing import Any, Callable, List, Sequence, Tuple, Union + from tensorflow_docs.api_generator import doc_controls from tensorflow_docs.api_generator import doc_generator_visitor @@ -167,7 +170,7 @@ def util_2 return filtered_children -def _get_imported_symbols(obj): +def _get_imported_symbols(obj: Union[str, types.ModuleType]): """Returns a list of symbol names imported by the given `obj`.""" class ImportNodeVisitor(ast.NodeVisitor): @@ -177,7 +180,13 @@ def __init__(self): self.imported_symbols = [] def _add_imported_symbol(self, node): - self.imported_symbols.extend([alias.name for alias in node.names]) + for alias in node.names: + name = alias.asname or alias.name + if name == '*': + continue + if '.' in name: + continue + self.imported_symbols.append(name) def visit_Import(self, node): # pylint: disable=invalid-name self._add_imported_symbol(node) @@ -185,7 +194,10 @@ def visit_Import(self, node): # pylint: disable=invalid-name def visit_ImportFrom(self, node): # pylint: disable=invalid-name self._add_imported_symbol(node) - source = get_source.get_source(obj) + if isinstance(obj, str): + source = textwrap.dedent(obj) + else: + source = get_source.get_source(obj) if source is None: return [] @@ -197,21 +209,57 @@ def visit_ImportFrom(self, node): # pylint: disable=invalid-name def explicit_package_contents_filter(path: Sequence[str], parent: Any, children: Children) -> Children: - """Filter modules to only include explicit contents. - - This function returns the children explicitly included by this module, meaning - that it will exclude: + """Filter submodules, only keep what's explicitly included. - * Modules in a package not explicitly imported by the package (submodules - are implicitly injected into their parent's namespace). - * Modules imported by a module that is not a package. + This filter only affects the visibility of **modules**. Other objects are not + affected. This filter is useful if you explicitly define your API in the packages of - your library, but do not expliticly define that API in the `__all__` variable - of each module. The purpose is to make it easier to maintain that API. + your library (the __init__.py files), but do not expliticly define that API + in the `__all__` variable of each module. The purpose is to make it easier to + maintain that API. + + **This filter makes it so that modules are only documented where they are + explicitly imported in an __init__.py** + + ### Packages + + Lots of imports **indirectly** inject modules into package namespaces, this + filter helps you ignore those. Anywhere you `import pkg.sub1` it will inject + `sub1` into the `pkg` namsspace. + + When filtering a package it only keeps modules that are **directly** + impotrted in the package. This code, injects `[sub0, sub1, sub2, sub3, sub4, + sub_sub1, *]` into the pkg namespace: + + pkg/__init__.py + + ``` + import sub0 + import pkg.sub1 + from pkg import sub2 + from pkg.sub3 import sub_sub1 + from pkg.sub4 import * + ``` + + But this filter will only keep the modules `[sub0, sub2, sub_sub1]` in the + docs for `pkg`. + + ### Regular modules + + For regular modules all modules in the namespace are assumed to be + implementation details and/or documented in their source location. For example + in this package: + + ``` + pkg/ + __init__.py + sub1.py + sub2.py + ``` - Note: This filter does work with wildcard imports, however it is generally not - recommended to use wildcard imports. + If you `import sub2` in `__init__.py` `sub2` will documented in `pkg` + But if you `import sub2` in `sub1.py` `sub2` will not be documented in `sub1` Args: path: A tuple of names forming the path to the object. diff --git a/tools/tensorflow_docs/api_generator/public_api_test.py b/tools/tensorflow_docs/api_generator/public_api_test.py index 901a048809a..1b801787a14 100644 --- a/tools/tensorflow_docs/api_generator/public_api_test.py +++ b/tools/tensorflow_docs/api_generator/public_api_test.py @@ -172,6 +172,21 @@ def test_explicit_package_contents_filter_removes_modules_imported_by_modules( # Assert that the filtered_members do not include a module named `inspect`. self.assertNotIn('inspect', [name for name, _ in filtered_members]) + def test_get_imported_symbols(self): + source = """ + import sub0 + import pkg.sub1 + from pkg import sub2 + from pkg.sub3 import sub_sub1 + from pkg.sub4 import * + from pkg import sub5 as r1 + from pkg import sub6 as r2, sub7, sub8 as r3 + + """ + imported = public_api._get_imported_symbols(source) + self.assertCountEqual( + ['sub0', 'sub2', 'sub_sub1', 'r1', 'r2', 'sub7', 'r3'], imported) + def test_ignore_typing(self): children_before = [('a', 1), ('b', 3), ('c', typing.List)] children_after = public_api.ignore_typing('ignored', 'ignored', From 75cf485555783746bc3e594957f6468f46e27bad Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 30 Mar 2022 05:23:25 -0700 Subject: [PATCH 046/872] Add a toc entry classes to simplify toc generation. This will give a 400 line cleanup in generate_lib.py a few changes down the road. PiperOrigin-RevId: 438275925 --- tools/tensorflow_docs/api_generator/toc.py | 160 ++++++++++++++++++ .../tensorflow_docs/api_generator/toc_test.py | 79 +++++++++ 2 files changed, 239 insertions(+) create mode 100644 tools/tensorflow_docs/api_generator/toc.py create mode 100644 tools/tensorflow_docs/api_generator/toc_test.py diff --git a/tools/tensorflow_docs/api_generator/toc.py b/tools/tensorflow_docs/api_generator/toc.py new file mode 100644 index 00000000000..800eb0062bf --- /dev/null +++ b/tools/tensorflow_docs/api_generator/toc.py @@ -0,0 +1,160 @@ +# Copyright 2022 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +"""Classes for generating the TOC.""" + +import contextlib +import dataclasses +import enum +import os + +from typing import Any, IO, Iterator, List, Optional, Tuple, Union + +import yaml + + +class _TocDumper(yaml.SafeDumper): + + def ignore_aliases(self, data): + """Don't output references for duplicated objects (usually strings).""" + return True + + +def _dict_representer(dumper: yaml.SafeDumper, data: Any): + """Represent the object as a dict created of (key, value) pairs.""" + return dumper.represent_dict(iter(data)) + + +def _use_yaml_dict_representer(cls): + """Register the class's as using a `_dict_representer`.""" + yaml.add_representer(cls, representer=_dict_representer, Dumper=_TocDumper) + return cls + + +def _str_enum_representer(dumper: yaml.SafeDumper, data: Any): + """Represent a str-Enum as a string.""" + return dumper.represent_str(data.value) + + +def _use_yaml_str_enum_representer(cls): + """Register the class as using `_str_enum_representer`.""" + yaml.add_representer( + cls, representer=_str_enum_representer, Dumper=_TocDumper) + return cls + + +@_use_yaml_str_enum_representer +class Status(enum.Enum): + """Represents a page status.""" + ALPHA = 'alpha' + BETA = 'beta' + DEPRECATED = 'deprecated' + EXPERIMENTAL = 'experimental' + EXTERNAL = 'external' + LIMITED = 'limited' + NEW = 'new' + NIGHTLY = 'nightly' + PREVIEW = 'preview' + UNSUPPORTED = 'unsupported' + + +@_use_yaml_str_enum_representer +class HeadingStyle(enum.Enum): + """Represents a Heading Style.""" + ACCORDION = 'accordion' + DIVIDER = 'divider' + + +class Entry: + """Base class for toc entries.""" + + def replace(self, **kwargs): + new_kwargs = dict(self) + new_kwargs.update(kwargs) + return type(self)(**new_kwargs) + + def __iter__(self) -> Iterator[Tuple[str, Any]]: + """Support `dict(entry)` for yaml output.""" + for key, value in self.__dict__.items(): + if value is not None: + yield (key, value) + + +@_use_yaml_dict_representer +@dataclasses.dataclass +class Heading(Entry): + """A toc heading.""" + heading: str + style: Optional[HeadingStyle] = None + + +@_use_yaml_dict_representer +@dataclasses.dataclass +class Section(Entry): + """A toc section.""" + title: str + section: List[Entry] + status: Optional[Status] = None + + def __iter__(self) -> Iterator[Tuple[str, Any]]: + """Support `dict(entry)` for yaml output.""" + yield 'title', self.title + if self.status is not None: + yield 'status', self.status + yield 'section', self.section + + +@_use_yaml_dict_representer +@dataclasses.dataclass +class Link(Entry): + """Represents toc page-link.""" + title: str + path: str + status: Optional[Status] = None + + def __iter__(self) -> Iterator[Tuple[str, Any]]: + """Support `dict(entry)` for yaml output.""" + yield 'title', self.title + if self.status is not None: + yield 'status', self.status + yield 'path', self.path + + +@_use_yaml_dict_representer +class Break(Entry): + """Represents a toc whitesoace break.""" + + def __init__(self): + self.__dict__['break'] = True + + +@_use_yaml_dict_representer +@dataclasses.dataclass +class Toc(Entry): + """Represents the top-level `toc` element in included files.""" + toc: List[Entry] + + @contextlib.contextmanager + def _maybe_open(self, file: Union[os.PathLike, IO]) -> Iterator[IO]: + if isinstance(file, os.PathLike): + with open(file, 'w') as stream: + yield stream + else: + stream = file + yield stream + + def write(self, file: Union[os.PathLike, IO]) -> None: + with self._maybe_open(file) as stream: + yaml.dump( + self, stream=stream, default_flow_style=False, Dumper=_TocDumper) diff --git a/tools/tensorflow_docs/api_generator/toc_test.py b/tools/tensorflow_docs/api_generator/toc_test.py new file mode 100644 index 00000000000..b5033517ab8 --- /dev/null +++ b/tools/tensorflow_docs/api_generator/toc_test.py @@ -0,0 +1,79 @@ +# Copyright 2022 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== + +import io +import textwrap +import types + +from tensorflow_docs.api_generator import toc as toc_lib +from tensorflow_docs.api_generator import doc_generator_visitor +from absl.testing import absltest + + +class TestToc(absltest.TestCase): + + def test_toc_write(self): + link = toc_lib.Link(title='A link', path='/path/to/link') + link2 = toc_lib.Link( + title='Another link', + path='/path/to/link2', + status=toc_lib.Status.EXTERNAL) + + subsection = toc_lib.Section( + title='A subsection', + section=[link], + status=toc_lib.Status.EXPERIMENTAL) + + toc = toc_lib.Toc([ + # pyformat: disable + toc_lib.Heading('Hello'), + link, + link2, + toc_lib.Break(), + subsection + ]) + + stream = io.StringIO() + toc.write(stream) + + expected = textwrap.dedent("""\ + toc: + - heading: Hello + - title: A link + path: /path/to/link + - title: Another link + status: external + path: /path/to/link2 + - break: true + - title: A subsection + status: experimental + section: + - title: A link + path: /path/to/link + """) + + self.assertEqual(expected, stream.getvalue()) + + def test_replace(self): + link = toc_lib.Link(title='A link', path='/path/to/link') + new_link = link.replace(status=toc_lib.Status.NEW, title='New title.') + + expected = toc_lib.Link( + title='New title.', path='/path/to/link', status=toc_lib.Status.NEW) + self.assertEqual(expected, new_link) + + +if __name__ == '__main__': + absltest.main() From 65bd57a9f73429dfd798d17e0be0c73abbdb35fe Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 30 Mar 2022 05:31:18 -0700 Subject: [PATCH 047/872] Simplify test object construction. Manually building the indices is complicated and error prone. Never do it manually. PiperOrigin-RevId: 438277327 --- .../api_generator/doc_generator_visitor.py | 17 +- .../api_generator/generate_lib.py | 36 +- .../api_generator/generate_lib_test.py | 69 +-- .../api_generator/parser_test.py | 502 +++++++----------- .../api_generator/report/linter_test.py | 127 ++--- .../api_generator/signature_test.py | 69 ++- 6 files changed, 290 insertions(+), 530 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index 2686ca57ce8..b3318be95be 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -205,7 +205,7 @@ def __init__(self): self._duplicates: Dict[str, List[str]] = None self._duplicate_of: Dict[str, str] = None - self._path_tree = PathTree() + self.path_tree = PathTree() @property def index(self): @@ -242,7 +242,6 @@ def reverse_index(self): Returns: The `id(object)` to full name map. """ - self._maybe_find_duplicates() return self._reverse_index @property @@ -256,7 +255,6 @@ def duplicate_of(self): Returns: The map from duplicate name to preferred name. """ - self._maybe_find_duplicates() return self._duplicate_of @property @@ -273,7 +271,6 @@ def duplicates(self): Returns: The map from main name to list of all duplicate names. """ - self._maybe_find_duplicates() return self._duplicates def __call__(self, parent_path, parent, children): @@ -301,8 +298,8 @@ class or module. parent_name = '.'.join(parent_path) self._index[parent_name] = parent self._tree[parent_name] = [] - if parent_path not in self._path_tree: - self._path_tree[parent_path] = parent + if parent_path not in self.path_tree: + self.path_tree[parent_path] = parent if not (inspect.ismodule(parent) or inspect.isclass(parent)): raise TypeError('Unexpected type in visitor -- ' @@ -310,7 +307,7 @@ class or module. for name, child in children: child_path = parent_path + (name,) - self._path_tree[child_path] = child + self.path_tree[child_path] = child full_name = '.'.join([parent_name, name]) if parent_name else name self._index[full_name] = child @@ -379,7 +376,7 @@ def _score_name(self, name): return (defining_class_score, experimental_score, keras_score, module_length_score, name) - def _maybe_find_duplicates(self): + def build(self): """Compute data structures containing information about duplicates. Find duplicates in `index` and decide on one to be the "main" name. @@ -411,7 +408,7 @@ def _maybe_find_duplicates(self): # symbol (incl. itself). duplicates = {} - for path, node in self._path_tree.items(): + for path, node in self.path_tree.items(): if not path: continue full_name = node.full_name @@ -420,7 +417,7 @@ def _maybe_find_duplicates(self): if full_name in duplicates: continue - aliases = self._path_tree.nodes_for_obj(py_object) + aliases = self.path_tree.nodes_for_obj(py_object) # maybe_singleton types can't be looked up by object. if not aliases: aliases = [node] diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index da44a5135cd..ddb6dff9abb 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -39,26 +39,6 @@ import yaml -try: - # TODO(markdaoust) delete this when the warning is in a stable release. - _estimator = importlib.import_module( - 'tensorflow_estimator.python.estimator.estimator') - - if doc_controls.get_inheritable_header(_estimator.Estimator) is None: - _add_header = doc_controls.inheritable_header("""\ - Warning: Estimators are not recommended for new code. Estimators run - `v1.Session`-style code which is more difficult to write correctly, and - can behave unexpectedly, especially when combined with TF 2 code. - Estimators do fall under our - [compatibility guarantees](https://tensorflow.org/guide/versions), but - will receive no fixes other than security vulnerabilities. See the - [migration guide](https://tensorflow.org/guide/migrate) for details. - """) - _add_header(_estimator.Estimator) -except ImportError: - pass - - # Used to add a collections.OrderedDict representer to yaml so that the # dump doesn't contain !!OrderedDict yaml tags. # Reference: https://stackoverflow.com/a/21048064 @@ -687,6 +667,7 @@ def extract(py_modules, traverse.traverse(py_module, visitors, short_name) + accumulator.build() return accumulator @@ -818,12 +799,17 @@ def run_extraction(self): Returns: """ - return extract( + visitor = extract( py_modules=self._py_modules, base_dir=self._base_dir, private_map=self._private_map, visitor_cls=self._visitor_cls, callbacks=self._callbacks) + reference_resolver = self.make_reference_resolver(visitor) + + # Write the api docs. + parser_config = self.make_parser_config(visitor, reference_resolver) + return parser_config def build(self, output_dir): """Build all the docs. @@ -838,11 +824,7 @@ def build(self, output_dir): workdir = pathlib.Path(tempfile.mkdtemp()) # Extract the python api from the _py_modules - visitor = self.run_extraction() - reference_resolver = self.make_reference_resolver(visitor) - - # Write the api docs. - parser_config = self.make_parser_config(visitor, reference_resolver) + parser_config = self.run_extraction() work_py_dir = workdir / 'api_docs/python' write_docs( output_dir=str(work_py_dir), @@ -859,7 +841,7 @@ def build(self, output_dir): ) if self.api_cache: - reference_resolver.to_json_file( + parser_config.reference_resolver.to_json_file( str(work_py_dir / self._short_name.replace('.', '/') / '_api_cache.json')) diff --git a/tools/tensorflow_docs/api_generator/generate_lib_test.py b/tools/tensorflow_docs/api_generator/generate_lib_test.py index df5d2cfd733..b7a2f03374e 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib_test.py +++ b/tools/tensorflow_docs/api_generator/generate_lib_test.py @@ -19,6 +19,7 @@ import sys import tempfile import textwrap +import types from absl import flags from absl.testing import absltest @@ -77,59 +78,21 @@ def setUp(self): def get_test_objects(self): # These are all mutable objects, so rebuild them for each test. # Don't cache the objects. - module = sys.modules[__name__] - - index = { - 'tf': - sys, # Can be any module, this test doesn't care about content. - 'tf.TestModule': - module, - 'tf.test_function': - test_function, - 'tf.TestModule.test_function': - test_function, - 'tf.TestModule.TestClass': - TestClass, - 'tf.TestModule.TestClass.ChildClass': - TestClass.ChildClass, - 'tf.TestModule.TestClass.ChildClass.GrandChildClass': - TestClass.ChildClass.GrandChildClass, - } - - tree = { - 'tf': ['TestModule', 'test_function'], - 'tf.TestModule': ['test_function', 'TestClass'], - 'tf.TestModule.TestClass': ['ChildClass'], - 'tf.TestModule.TestClass.ChildClass': ['GrandChildClass'], - 'tf.TestModule.TestClass.ChildClass.GrandChildClass': [] - } - - duplicate_of = {'tf.test_function': 'tf.TestModule.test_function'} - - duplicates = { - 'tf.TestModule.test_function': [ - 'tf.test_function', 'tf.TestModule.test_function' - ] - } - - base_dir = os.path.dirname(__file__) - - visitor = DummyVisitor(index, duplicate_of) - - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf'], link_prefix='api_docs/python') - - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates=duplicates, - duplicate_of=duplicate_of, - tree=tree, - index=index, - reverse_index={}, - base_dir=base_dir, - code_url_prefix='/') - - return reference_resolver, parser_config + tf = types.ModuleType('tf') + tf.__file__ = __file__ + tf.TestModule = types.ModuleType('module') + tf.test_function = test_function + tf.TestModule.test_function = test_function + tf.TestModule.TestClass = TestClass + + generator = generate_lib.DocGenerator( + root_title='TensorFlow', + py_modules=[('tf', tf)], + code_url_prefix='https://tensorflow.org/') + + parser_config = generator.run_extraction() + + return parser_config.reference_resolver, parser_config def test_write(self): _, parser_config = self.get_test_objects() diff --git a/tools/tensorflow_docs/api_generator/parser_test.py b/tools/tensorflow_docs/api_generator/parser_test.py index 14cb4f55d8c..0e11291f22b 100644 --- a/tools/tensorflow_docs/api_generator/parser_test.py +++ b/tools/tensorflow_docs/api_generator/parser_test.py @@ -18,6 +18,7 @@ import dataclasses import inspect import textwrap +import types from typing import List, Union @@ -27,6 +28,7 @@ from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_controls +from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib from tensorflow_docs.api_generator.pretty_docs import docs_for_object @@ -186,40 +188,21 @@ def foo(self): '`tf.not.a.real.symbol`', result) def test_docs_for_class(self): + m = types.ModuleType('m') + m.__file__ = __file__ + m.TestClass = TestClass - index = { - 'TestClass': TestClass, - 'TestClass.a_method': TestClass.a_method, - 'TestClass.a_property': TestClass.a_property, - 'TestClass.ChildClass': TestClass.ChildClass, - 'TestClass.static_method': TestClass.static_method, - 'TestClass.class_method': TestClass.class_method, - 'TestClass.CLASS_MEMBER': TestClass.CLASS_MEMBER, - } - - visitor = DummyVisitor(index=index, duplicate_of={}) - - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - tree = { - 'TestClass': [ - 'a_method', 'class_method', 'static_method', 'a_property', - 'ChildClass', 'CLASS_MEMBER' - ] - } - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='TestClass', py_object=TestClass, parser_config=parser_config) + full_name='m.TestClass', + py_object=TestClass, + parser_config=parser_config) # Make sure the brief docstring is present self.assertEqual( @@ -248,68 +231,46 @@ def test_docs_for_class(self): self.assertIs(TestClass.ChildClass, page_info.classes[0].py_object) def test_dataclass_attributes_table(self): + m = types.ModuleType('m') + m.__file__ = __file__ + m.ExampleDataclass = ExampleDataclass - index = { - 'ExampleDataclass': ExampleDataclass, - } + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - visitor = DummyVisitor(index=index, duplicate_of={}) - - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = {'ExampleDataclass': []} - - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='ExampleDataclass', + full_name='m.ExampleDataclass', py_object=ExampleDataclass, parser_config=parser_config) self.assertCountEqual(['a', 'b', 'c', 'x', 'y', 'z'], [name for name, value in page_info.attr_block.items]) - def test_namedtuple_field_order(self): + def test_namedtuple_field_order_respects_hidden(self): namedtupleclass = collections.namedtuple( 'namedtupleclass', ['z', 'y', 'x', 'hidden', 'w', 'v', 'u']) - index = { - 'namedtupleclass': namedtupleclass, - 'namedtupleclass.u': namedtupleclass.u, - 'namedtupleclass.v': namedtupleclass.v, - 'namedtupleclass.w': namedtupleclass.w, - 'namedtupleclass.x': namedtupleclass.x, - 'namedtupleclass.y': namedtupleclass.y, - 'namedtupleclass.z': namedtupleclass.z, - } + m = types.ModuleType('m') + m.__file__ = __file__ + m.namedtupleclass = namedtupleclass - visitor = DummyVisitor(index=index, duplicate_of={}) + def hide(path, parent, children): + return [(name, value) for name, value in children if name != 'hidden'] - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = {'namedtupleclass': {'u', 'v', 'w', 'x', 'y', 'z'}} - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org', + callbacks=[hide]) + + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='namedtupleclass', + full_name='m.namedtupleclass', py_object=namedtupleclass, parser_config=parser_config) @@ -337,32 +298,19 @@ class Child(Parent): def a_method(self, arg='default'): pass - index = { - 'Child': Child, - 'Child.a_method': Child.a_method, - } + m = types.ModuleType('m') + m.__file__ = __file__ + m.Child = Child - visitor = DummyVisitor(index=index, duplicate_of={}) + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = { - 'Child': ['a_method'], - } - - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='Child', py_object=Child, parser_config=parser_config) + full_name='m.Child', py_object=Child, parser_config=parser_config) # Make sure the `a_method` is not present self.assertEmpty(page_info.methods) @@ -389,33 +337,19 @@ class ChildMessage(CMessage, Message, MessageMeta): def my_method(self): pass - index = { - 'ChildMessage': ChildMessage, - 'ChildMessage.hidden': ChildMessage.hidden, - 'ChildMessage.hidden2': ChildMessage.hidden2, - 'ChildMessage.hidden3': ChildMessage.hidden3, - 'ChildMessage.my_method': ChildMessage.my_method, - } - - visitor = DummyVisitor(index=index, duplicate_of={}) - - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) + m = types.ModuleType('m') + m.__file__ = __file__ + m.ChildMessage = ChildMessage - tree = {'ChildMessage': ['hidden', 'hidden2', 'hidden3', 'my_method']} + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='ChildMessage', + full_name='m.ChildMessage', py_object=ChildMessage, parser_config=parser_config) @@ -423,42 +357,21 @@ def my_method(self): self.assertEqual('my_method', page_info.methods[0].short_name) def test_docs_for_module(self): + m = types.ModuleType('m') + m.__file__ = __file__ + m.test_function = test_function + m.test_function_with_args_kwargs = test_function_with_args_kwargs + m.TestClass = TestClass - index = { - 'TestModule': - test_module, - 'TestModule.test_function': - test_function, - 'TestModule.test_function_with_args_kwargs': - test_function_with_args_kwargs, - 'TestModule.TestClass': - TestClass, - } + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - visitor = DummyVisitor(index=index, duplicate_of={}) - - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = { - 'TestModule': [ - 'TestClass', 'test_function', 'test_function_with_args_kwargs' - ] - } - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='TestModule', - py_object=test_module, - parser_config=parser_config) + full_name='m', py_object=test_module, parser_config=parser_config) # Make sure the brief docstring is present self.assertEqual( @@ -472,23 +385,16 @@ def test_docs_for_module(self): self.assertEqual({TestClass}, classes) def test_docs_for_function(self): - index = {'test_function': test_function} + m = types.ModuleType('m') + m.__file__ = __file__ + m.test_function = test_function - visitor = DummyVisitor(index=index, duplicate_of={}) + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = {'': ['test_function']} - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( full_name='test_function', @@ -504,23 +410,16 @@ def test_docs_for_function(self): str(page_info.signature)) def test_docs_for_function_with_kwargs(self): - index = {'test_function_with_args_kwargs': test_function_with_args_kwargs} + m = types.ModuleType('m') + m.__file__ = __file__ + m.test_function_with_args_kwargs = test_function_with_args_kwargs - visitor = DummyVisitor(index=index, duplicate_of={}) + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = {'': ['test_function_with_args_kwargs']} - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( full_name='test_function_with_args_kwargs', @@ -574,29 +473,26 @@ class HasOneMember(object): def foo(self): pass - duplicate_of = {'tf.third': 'tf.fourth'} - index = { - 'tf': test_module, - 'tf.fancy': test_function_with_fancy_docstring, - 'tf.reference': HasOneMember, - 'tf.reference.foo': HasOneMember.foo, - 'tf.third': HasOneMember, - 'tf.fourth': HasOneMember - } + class HasOneMember2(object): + + def foo(self): + pass - visitor = DummyVisitor(index=index, duplicate_of=duplicate_of) + tf = types.ModuleType('tf') + tf.__file__ = __file__ + tf.fancy = test_function_with_fancy_docstring + tf.reference = HasOneMember + tf.third = HasOneMember2 + tf.fourth = HasOneMember2 - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf'], link_prefix='../..') - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree={}, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('tf', tf)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() + parser_config.reference_resolver = ( + parser_config.reference_resolver.with_prefix('/')) doc_info = parser.parse_md_docstring( test_function_with_fancy_docstring, @@ -653,37 +549,33 @@ def test_downgrade_h1_docstrings(self): self.assertIn('\nRaises:', doc) def test_generate_index(self): + m = types.ModuleType('m') + m.__file__ = __file__ + m.TestClass = TestClass + m.test_function = test_function + m.submodule = types.ModuleType('submodule') + m.submodule.test_function = test_function - index = { - 'tf': test_module, - 'tf.TestModule': test_module, - 'tf.test_function': test_function, - 'tf.TestModule.test_function': test_function, - 'tf.TestModule.TestClass': TestClass, - 'tf.TestModule.TestClass.a_method': TestClass.a_method, - 'tf.TestModule.TestClass.a_property': TestClass.a_property, - 'tf.TestModule.TestClass.ChildClass': TestClass.ChildClass, - } - duplicate_of = {'tf.TestModule.test_function': 'tf.test_function'} - - visitor = DummyVisitor(index=index, duplicate_of=duplicate_of) + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) + parser_config = generator.run_extraction() docs = parser.generate_global_index( - 'TestLibrary', index=index, reference_resolver=reference_resolver) + 'TestLibrary', + index=parser_config.index, + reference_resolver=parser_config.reference_resolver) # Make sure duplicates and non-top-level symbols are in the index, but # methods and properties are not. self.assertNotIn('a_method', docs) self.assertNotIn('a_property', docs) - self.assertIn('TestModule.TestClass', docs) - self.assertIn('TestModule.TestClass.ChildClass', docs) - self.assertIn('TestModule.test_function', docs) - # Leading backtick to make sure it's included top-level. - # This depends on formatting, but should be stable. - self.assertIn('tf.test_function', docs) + self.assertIn('m.TestClass', docs) + self.assertIn('m.TestClass.ChildClass', docs) + self.assertIn('m.submodule.test_function', docs) + self.assertIn('m.submodule.test_function', docs) def test_getsource_indexerror_resilience(self): """Validates that parser gracefully handles IndexErrors. @@ -692,44 +584,19 @@ def test_getsource_indexerror_resilience(self): why this happens, but it consistently repros on the `get` method of collections.MutableMapping subclasses. """ + m = types.ModuleType('m') + m.__file__ = __file__ + m.ConcreteMutableMapping = ConcreteMutableMapping - # This isn't the full set of APIs from MutableMapping, but sufficient for - # testing. - index = { - 'ConcreteMutableMapping': - ConcreteMutableMapping, - 'ConcreteMutableMapping.__init__': - ConcreteMutableMapping.__init__, - 'ConcreteMutableMapping.__getitem__': - ConcreteMutableMapping.__getitem__, - 'ConcreteMutableMapping.__setitem__': - ConcreteMutableMapping.__setitem__, - 'ConcreteMutableMapping.values': - ConcreteMutableMapping.values, - 'ConcreteMutableMapping.get': - ConcreteMutableMapping.get - } - visitor = DummyVisitor(index=index, duplicate_of={}) - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') - tree = { - 'ConcreteMutableMapping': [ - '__init__', '__getitem__', '__setitem__', 'values', 'get' - ] - } - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='ConcreteMutableMapping', + full_name='m.ConcreteMutableMapping', py_object=ConcreteMutableMapping, parser_config=parser_config) @@ -744,31 +611,21 @@ def test_strips_default_arg_memory_address(self): See: `help(collections.MutableMapping.pop)` """ - index = { - 'ConcreteMutableMapping': ConcreteMutableMapping, - 'ConcreteMutableMapping.pop': ConcreteMutableMapping.pop - } - visitor = DummyVisitor(index=index, duplicate_of={}) - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = {'ConcreteMutableMapping': ['pop']} - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + m = types.ModuleType('m') + m.__file__ = __file__ + m.fun = lambda x=object(): x + + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() page_info = docs_for_object.docs_for_object( - full_name='ConcreteMutableMapping', - py_object=ConcreteMutableMapping, - parser_config=parser_config) + full_name='m.fun', py_object=m.fun, parser_config=parser_config) - output = str(page_info.methods[0].signature) + output = str(page_info.signature) self.assertNotIn('object at 0x', output) self.assertIn('<object object>', output) @@ -802,21 +659,16 @@ def test_empty_defined_in(self, cls, method, py_object): method: The class method name to generate docs for. py_object: The python object for the specified cls.method. """ + m = types.ModuleType('m') + m.__file__ = __file__ + m.ConcreteMutableMapping = ConcreteMutableMapping - visitor = DummyVisitor(index={}, duplicate_of={}) - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf']) - - tree = {cls: [method]} - parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index={}, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() function_info = docs_for_object.docs_for_object( full_name='%s.%s' % (cls, method), @@ -834,15 +686,16 @@ class A(): a = A() a.__doc__ = 'Object doc' - parser_config = config.ParserConfig( - reference_resolver=None, - duplicates={}, - duplicate_of={}, - tree={}, - index={}, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + m = types.ModuleType('m') + m.__file__ = __file__ + m.a = a + + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() result = parser._get_other_member_doc(a, parser_config, {}) @@ -889,15 +742,16 @@ class A(): a = A() - parser_config = config.ParserConfig( - reference_resolver=None, - duplicates={}, - duplicate_of={}, - tree={}, - index={}, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + m = types.ModuleType('m') + m.__file__ = __file__ + m.a = a + + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() result = parser._get_other_member_doc(a, parser_config, {}) expected = textwrap.dedent("""\ @@ -913,19 +767,21 @@ class A(): a = A() - parser_config = config.ParserConfig( - reference_resolver=None, - duplicates={}, - duplicate_of={}, - tree={}, - index={}, - reverse_index={id(A): 'tf.test.A'}, - base_dir='/', - code_url_prefix='/') + m = types.ModuleType('m') + m.__file__ = __file__ + m.A = A + m.a = a + + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() result = parser._get_other_member_doc(a, parser_config, {}) - self.assertEqual('Instance of `tf.test.A`', result) + self.assertEqual('Instance of `m.A`', result) def testIsClasssAttr(self): result = parser.is_class_attr('test_module.test_function', diff --git a/tools/tensorflow_docs/api_generator/report/linter_test.py b/tools/tensorflow_docs/api_generator/report/linter_test.py index c363c327498..c06d262bf07 100644 --- a/tools/tensorflow_docs/api_generator/report/linter_test.py +++ b/tools/tensorflow_docs/api_generator/report/linter_test.py @@ -16,11 +16,13 @@ import copy +import types from typing import Optional from absl.testing import absltest from tensorflow_docs.api_generator import config +from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib from tensorflow_docs.api_generator.pretty_docs import docs_for_object @@ -113,100 +115,74 @@ def method_one(self, x: str) -> Optional[str]: class LinterTest(absltest.TestCase): - def setUp(self): - super(LinterTest, self).setUp() - index = { - 'TestClass': TestClass, - 'TestClass.__init__': TestClass.__init__, - 'TestClass.method_one': TestClass.method_one, - 'TestClass.temp_c': TestClass.temp_c, - } - tree = { - 'TestClass': ['__init__', 'method_one', 'temp_c'], - } - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=DummyVisitor(index=index, duplicate_of={}), - py_module_names=['tf'], - ) - self.parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree=tree, - index=index, - reverse_index={}, - base_dir='/', - code_url_prefix='/') + def _build_page_info(self): + m = types.ModuleType('m') + m.__file__ = __file__ + m.TestClass = TestClass - def test_class_raises_lint(self): - class_page_info = docs_for_object.docs_for_object( - full_name='TestClass', + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() + return docs_for_object.docs_for_object( + full_name='m.TestClass', py_object=TestClass, - parser_config=self.parser_config) - class_page_info_before = copy.deepcopy(class_page_info) + parser_config=parser_config) + + def _make_report(self): + page_info = self._build_page_info() + + test_api_report = utils.ApiReport() + test_api_report.fill_metrics(page_info) + return test_api_report + + def test_fill_report_doesnt_edit_page(self): + page1 = self._build_page_info() + page2 = self._build_page_info() test_api_report = utils.ApiReport() - test_api_report.fill_metrics(class_page_info) - self.assertEqual(class_page_info_before, class_page_info) + test_api_report.fill_metrics(page2) + + self.assertEqual(page1, page2) + + def test_class_raises_lint(self): + test_api_report = self._make_report() for test_report in test_api_report.api_report.symbol_metric: - if (test_report.symbol_name == 'TestClass' and + if (test_report.symbol_name == 'm.TestClass' and test_report.object_type == api_report_pb2.ObjectType.CLASS): self.assertEqual(test_report.raises_lint.num_raises_defined, 2) self.assertEqual(test_report.raises_lint.total_raises_in_code, 2) def test_method_return_lint(self): - class_page_info = docs_for_object.docs_for_object( - full_name='TestClass', - py_object=TestClass, - parser_config=self.parser_config) - - class_page_info_before = copy.deepcopy(class_page_info) - - test_api_report = utils.ApiReport() - test_api_report.fill_metrics(class_page_info) - self.assertEqual(class_page_info_before, class_page_info) + test_api_report = self._make_report() for test_report in test_api_report.api_report.symbol_metric: - if (test_report.symbol_name == 'TestClass.method_one' and + if (test_report.symbol_name == 'm.TestClass.method_one' and test_report.object_type == api_report_pb2.ObjectType.METHOD): self.assertTrue(test_report.return_lint.returns_defined) def test_description_lint(self): - class_page_info = docs_for_object.docs_for_object( - full_name='TestClass', - py_object=TestClass, - parser_config=self.parser_config) - class_page_info_before = copy.deepcopy(class_page_info) - - test_api_report = utils.ApiReport() - test_api_report.fill_metrics(class_page_info) - self.assertEqual(class_page_info_before, class_page_info) + test_api_report = self._make_report() for test_report in test_api_report.api_report.symbol_metric: - if (test_report.symbol_name == 'TestClass' and + if (test_report.symbol_name == 'm.TestClass' and test_report.object_type == api_report_pb2.ObjectType.CLASS): self.assertEqual(test_report.desc_lint.len_brief, 2) self.assertEqual(test_report.desc_lint.len_long_desc, 54) - if (test_report.symbol_name == 'TestClass.method_one' and + if (test_report.symbol_name == 'm.TestClass.method_one' and test_report.object_type == api_report_pb2.ObjectType.METHOD): self.assertEqual(test_report.desc_lint.len_brief, 4) self.assertEqual(test_report.desc_lint.len_long_desc, 10) def test_parameter_lint(self): - class_page_info = docs_for_object.docs_for_object( - full_name='TestClass', - py_object=TestClass, - parser_config=self.parser_config) - class_page_info_before = copy.deepcopy(class_page_info) - - test_api_report = utils.ApiReport() - test_api_report.fill_metrics(class_page_info) - self.assertEqual(class_page_info_before, class_page_info) + test_api_report = self._make_report() for test_report in test_api_report.api_report.symbol_metric: - if (test_report.symbol_name == 'TestClass' and + if (test_report.symbol_name == 'm.TestClass' and test_report.object_type == api_report_pb2.ObjectType.CLASS): self.assertEqual(test_report.parameter_lint.num_empty_param_desc_args, 2) @@ -216,7 +192,7 @@ def test_parameter_lint(self): 1) self.assertEqual(test_report.parameter_lint.total_attr_param, 2) - if (test_report.symbol_name == 'TestClass.method_one' and + if (test_report.symbol_name == 'm.TestClass.method_one' and test_report.object_type == api_report_pb2.ObjectType.METHOD): self.assertEqual(test_report.parameter_lint.num_empty_param_desc_args, 0) @@ -227,30 +203,25 @@ def test_parameter_lint(self): self.assertEqual(test_report.parameter_lint.total_attr_param, 0) def test_example_lint(self): - class_page_info = docs_for_object.docs_for_object( - full_name='TestClass', - py_object=TestClass, - parser_config=self.parser_config) - class_page_info_before = copy.deepcopy(class_page_info) - - test_api_report = utils.ApiReport() - test_api_report.fill_metrics(class_page_info) - self.assertEqual(class_page_info_before, class_page_info) + test_api_report = self._make_report() for test_report in test_api_report.api_report.symbol_metric: - if (test_report.symbol_name == 'TestClass' and + if (test_report.symbol_name == 'm.TestClass' and test_report.object_type == api_report_pb2.ObjectType.CLASS): self.assertEqual(test_report.usage_example_lint.num_doctest, 2) self.assertEqual(test_report.usage_example_lint.num_untested_examples, 1) - self.assertEqual(test_report.package_group, 'TestClass') + self.assertEqual( + 'm', + test_report.package_group, + ) - if (test_report.symbol_name == 'TestClass.method_one' and + if (test_report.symbol_name == 'm.TestClass.method_one' and test_report.object_type == api_report_pb2.ObjectType.METHOD): self.assertEqual(test_report.usage_example_lint.num_doctest, 0) self.assertEqual(test_report.usage_example_lint.num_untested_examples, 1) - self.assertEqual(test_report.package_group, 'TestClass') + self.assertEqual('m', test_report.package_group) if __name__ == '__main__': diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index 028cd0273f1..a0eddca2f82 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -16,6 +16,7 @@ import dataclasses import textwrap +import types from typing import Callable, Dict, List, Optional, Union @@ -23,6 +24,7 @@ from absl.testing import parameterized from tensorflow_docs.api_generator import config +from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib from tensorflow_docs.api_generator import signature @@ -49,51 +51,40 @@ class TestGenerateSignature(parameterized.TestCase, absltest.TestCase): def setUp(self): super().setUp() self.known_object = object() - reference_resolver = reference_resolver_lib.ReferenceResolver( - link_prefix='/', - duplicate_of={ - 'tfdocs.api_generator.signature.extract_decorators': - 'tfdocs.api_generator.signature.extract_decorators', - 'location.of.object.in.api': - 'location.of.object.in.api', - }, - is_fragment={ - 'location.of.object.in.api': False, - 'tfdocs.api_generator.signature.extract_decorators': False, - }, - py_module_names=[]) - self.parser_config = config.ParserConfig( - reference_resolver=reference_resolver, - duplicates={}, - duplicate_of={}, - tree={}, - index={ - 'location.of.object.in.api': - self.known_object, - 'tfdocs.api_generator.signature.extract_decorators': - signature.extract_decorators - }, - reverse_index={ - id(self.known_object): - 'location.of.object.in.api', - id(signature.extract_decorators): - 'tfdocs.api_generator.signature.extract_decorators', - }, - base_dir='/', - code_url_prefix='/') + + m = types.ModuleType('m') + m.__file__ = __file__ + m.extract_decorators = signature.extract_decorators + m.submodule = types.ModuleType('submodule') + m.submodule.known = self.known_object + + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('m', m)], + code_url_prefix='https://tensorflow.org') + + self.parser_config = generator.run_extraction() + def test_known_object(self): def example_fun(arg=self.known_object): # pylint: disable=unused-argument pass + self.parser_config.reference_resolver = ( + self.parser_config.reference_resolver.with_prefix('/')) + sig = signature.generate_signature( example_fun, parser_config=self.parser_config, func_type=signature.FuncType.FUNCTION) - self.assertEqual( - '(\n arg=location.of.object.in.api\n)', - str(sig)) + + expected = textwrap.dedent("""\ + ( + arg=m.submodule.known + )""") + + self.assertEqual(expected, str(sig)) def test_literals(self): @@ -244,11 +235,11 @@ def test_dataclasses_type_annotations(self): List[Dict[int, signature.extract_decorators]]], textwrap.dedent("""\ Union[ - dict[str, dict[bool, tfdocs.api_generator.signature.extract_decorators]], + dict[str, dict[bool, m.extract_decorators]], int, bool, - tfdocs.api_generator.signature.extract_decorators, - list[dict[int, tfdocs.api_generator.signature.extract_decorators]] + m.extract_decorators, + list[dict[int, m.extract_decorators]] ]""")), ('callable_ellipsis_sig', Union[Callable[..., int], str], textwrap.dedent("""\ @@ -260,7 +251,7 @@ def test_dataclasses_type_annotations(self): float], int], textwrap.dedent("""\ Union[ - Callable[[bool, tfdocs.api_generator.signature.extract_decorators], float], + Callable[[bool, m.extract_decorators], float], int ]""")), ('callable_without_args', Union[None, dict, str, Callable], From 0d2801bde33137031f1a5bfd17d6f8f677e815c5 Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Wed, 30 Mar 2022 11:14:20 -0700 Subject: [PATCH 048/872] Add M1 guidance to install pages. PiperOrigin-RevId: 438349806 --- site/en/install/pip.html | 60 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/site/en/install/pip.html b/site/en/install/pip.html index 9736a46eb89..9a7265725a9 100644 --- a/site/en/install/pip.html +++ b/site/en/install/pip.html @@ -91,6 +91,19 @@

Ubuntu

macOS

+

Install using the Homebrew package manager:

 /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
@@ -143,7 +156,52 @@ 
2. Create a virtual environment (recommended)

 
 

 

-
Ubuntu / macOS

+
Ubuntu

+

+  Create a new virtual environment by choosing a Python interpreter and making a
+  ./venv directory to hold it:
+

+
python3 -m venv --system-site-packages ./venv

+

+  Activate the virtual environment using a shell-specific command:
+

+
source ./venv/bin/activate  # sh, bash, or zsh

+
. ./venv/bin/activate.fish  # fish

+
source ./venv/bin/activate.csh  # csh or tcsh

+
+

+  When the virtual environment is active, your shell prompt is prefixed with (venv).
+

+

+  Install packages within a virtual environment without affecting the host system
+  setup. Start by upgrading pip:
+

+
+pip install --upgrade pip
+
+pip list  # show packages installed within the virtual environment
+

+

+  And to exit the virtual environment later:
+

+
deactivate  # don't exit until you're done using TensorFlow

+

+
+

+
macOS

+
 

   Create a new virtual environment by choosing a Python interpreter and making a
   ./venv directory to hold it:

From ee52c35b32a20f02c03567c6b9cdc0c28624aeab Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 30 Mar 2022 18:25:00 -0700
Subject: [PATCH 049/872] Add a simplification for ast.parse

This was failing on the wrapped lines in Dataset.map (and all subclasses) because textwrap.dedent didn't remove the leading space.

PiperOrigin-RevId: 438447039
---
 .../tensorflow_docs/api_generator/get_source.py  | 15 +++++++++++++++
 .../tensorflow_docs/api_generator/public_api.py  |  8 ++------
 .../api_generator/report/linter.py               | 16 +++++++++-------
 tools/tensorflow_docs/api_generator/signature.py | 12 +++++-------
 4 files changed, 31 insertions(+), 20 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/get_source.py b/tools/tensorflow_docs/api_generator/get_source.py
index d85c5ef88e6..8ea0b02014a 100644
--- a/tools/tensorflow_docs/api_generator/get_source.py
+++ b/tools/tensorflow_docs/api_generator/get_source.py
@@ -13,12 +13,27 @@
 # limitations under the License.
 # ==============================================================================
 """Simple get_source."""
+import ast
 import inspect
 import textwrap
 
 from typing import Any, Optional, Sequence, Tuple
 
 
+def get_ast(py_object) -> Optional[ast.AST]:
+  if isinstance(py_object, str):
+    source = textwrap.dedent(py_object)
+  else:
+    source = get_source(py_object)
+  if source is None:
+    return None
+
+  try:
+    return ast.parse(source)
+  except Exception:  # pylint: disable=broad-except
+    return None
+
+
 def get_source(py_object: Any) -> Optional[str]:
   if py_object is not None:
     try:
diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py
index 9603d85ba54..925e5415856 100644
--- a/tools/tensorflow_docs/api_generator/public_api.py
+++ b/tools/tensorflow_docs/api_generator/public_api.py
@@ -194,14 +194,10 @@ def visit_Import(self, node):  # pylint: disable=invalid-name
     def visit_ImportFrom(self, node):  # pylint: disable=invalid-name
       self._add_imported_symbol(node)
 
-  if isinstance(obj, str):
-    source = textwrap.dedent(obj)
-  else:
-    source = get_source.get_source(obj)
-  if source is None:
+  tree = get_source.get_ast(obj)
+  if tree is None:
     return []
 
-  tree = ast.parse(source)
   visitor = ImportNodeVisitor()
   visitor.visit(tree)
   return visitor.imported_symbols
diff --git a/tools/tensorflow_docs/api_generator/report/linter.py b/tools/tensorflow_docs/api_generator/report/linter.py
index 6752d9bee88..44760208c73 100644
--- a/tools/tensorflow_docs/api_generator/report/linter.py
+++ b/tools/tensorflow_docs/api_generator/report/linter.py
@@ -176,12 +176,13 @@ def lint_returns(
   Returns:
     A filled `ReturnLint` proto object.
   """
-  source = get_source.get_source(page_info.py_object)
-
   return_visitor = ReturnVisitor()
-  if source is not None:
+
+  source = get_source.get_source(page_info.py_object)
+  obj_ast = get_source.get_ast(page_info.py_object)
+  if obj_ast is not None:
     try:
-      return_visitor.visit(ast.parse(source))
+      return_visitor.visit(obj_ast)
     except Exception:  # pylint: disable=broad-except
       pass
 
@@ -236,12 +237,13 @@ def lint_raises(page_info: base_page.PageInfo) -> api_report_pb2.RaisesLint:
 
   # Extract the raises from the source code.
   raise_visitor = RaiseVisitor()
-  source = get_source.get_source(page_info.py_object)
-  if source is not None:
+  obj_ast = get_source.get_ast(page_info.py_object)
+  if obj_ast is not None:
     try:
-      raise_visitor.visit(ast.parse(source))
+      raise_visitor.visit(obj_ast)
     except Exception:  # pylint: disable=broad-except
       pass
+
   raises_lint.total_raises_in_code = len(raise_visitor.total_raises)
 
   # Extract the raises defined in the docstring.
diff --git a/tools/tensorflow_docs/api_generator/signature.py b/tools/tensorflow_docs/api_generator/signature.py
index cc6d46b0d62..51221602319 100644
--- a/tools/tensorflow_docs/api_generator/signature.py
+++ b/tools/tensorflow_docs/api_generator/signature.py
@@ -56,9 +56,8 @@ def _preprocess_default(self, val: ast.AST) -> str:
     return text_default_val
 
   def extract(self, obj: Any):
-    obj_source = get_source.get_source(obj)
-    if obj_source is not None:
-      obj_ast = ast.parse(obj_source)
+    obj_ast = get_source.get_ast(obj)
+    if obj_ast is not None:
       self.visit(obj_ast)
 
 
@@ -674,11 +673,10 @@ def visit_FunctionDef(self, node):  # pylint: disable=invalid-name
 
   visitor = ASTDecoratorExtractor()
 
-  # Note: inspect.getsource doesn't include the decorator lines on classes,
+  # Note: get_source doesn't include the decorator lines on classes,
   # this won't work for classes until that's fixed.
-  func_source = get_source.get_source(func)
-  if func_source is not None:
-    func_ast = ast.parse(func_source)
+  func_ast = get_source.get_ast(func)
+  if func_ast is not None:
     visitor.visit(func_ast)
 
   return visitor.decorator_list

From 10313cbebd0560d87a0c91d399dccf1e06e8fc33 Mon Sep 17 00:00:00 2001
From: Bing Hu 
Date: Thu, 31 Mar 2022 14:29:54 -0700
Subject: [PATCH 050/872] Review and update public doc "Distributed training
 with Keras"

PiperOrigin-RevId: 438662329
---
 site/en/tutorials/distribute/keras.ipynb | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/site/en/tutorials/distribute/keras.ipynb b/site/en/tutorials/distribute/keras.ipynb
index c75e5f88af5..31a0dd7f386 100644
--- a/site/en/tutorials/distribute/keras.ipynb
+++ b/site/en/tutorials/distribute/keras.ipynb
@@ -76,7 +76,7 @@
         "\n",
         "You will use the `tf.keras` APIs to build the model and `Model.fit` for training it. (To learn about distributed training with a custom training loop and the `MirroredStrategy`, check out [this tutorial](custom_training.ipynb).)\n",
         "\n",
-        "`MirroredStrategy` trains your model on multiple GPUs on a single machine. For _synchronous training on many GPUs on multiple workers_, use the `tf.distribute.MultiWorkerMirroredStrategy` [with the Keras Model.fit](multi_worker_with_keras.ipynb) or [a custom training loop](multi_worker_with_ctl.ipynb). For other options, refer to the [Distributed training guide](../../guide/distributed_training.ipynb).\n",
+        "`MirroredStrategy` trains your model on multiple GPUs on a single machine. For _synchronous training on many GPUs on multiple workers_, use the `tf.distribute.MultiWorkerMirroredStrategy` with the [Keras Model.fit](multi_worker_with_keras.ipynb) or [a custom training loop](multi_worker_with_ctl.ipynb). For other options, refer to the [Distributed training guide](../../guide/distributed_training.ipynb).\n",
         "\n",
         "To learn about various other strategies, there is the [Distributed training with TensorFlow](../../guide/distributed_training.ipynb) guide."
       ]
@@ -289,7 +289,7 @@
         "id": "1BnQYQTpB3YA"
       },
       "source": [
-        "Create and compile the Keras model in the context of `Strategy.scope`:"
+        "Within the context of `Strategy.scope`, create and compile the model using the Keras API:"
       ]
     },
     {
@@ -329,13 +329,16 @@
         "id": "YOXO5nvvK3US"
       },
       "source": [
-        "Define the following `tf.keras.callbacks`:\n",
+        "Define the following [Keras Callbacks](https://www.tensorflow.org/guide/keras/train_and_evaluate):\n",
         "\n",
         "- `tf.keras.callbacks.TensorBoard`: writes a log for TensorBoard, which allows you to visualize the graphs.\n",
         "- `tf.keras.callbacks.ModelCheckpoint`: saves the model at a certain frequency, such as after every epoch.\n",
+        "- `tf.keras.callbacks.BackupAndRestore`: provides the fault tolerance functionality by backing up the model and current epoch number. Learn more in the _Fault tolerance_ section of the [Multi-worker training with Keras](multi_worker_with_keras.ipynb) tutorial.\n",
         "- `tf.keras.callbacks.LearningRateScheduler`: schedules the learning rate to change after, for example, every epoch/batch.\n",
         "\n",
-        "For illustrative purposes, add a custom callback called `PrintLR` to display the *learning rate* in the notebook."
+        "For illustrative purposes, add a [custom callback](https://www.tensorflow.org/guide/keras/custom_callback) called `PrintLR` to display the *learning rate* in the notebook.\n",
+        "\n",
+        "**Note:** Use the `BackupAndRestore` callback instead of `ModelCheckpoint` as the main mechanism to restore the training state upon a restart from a job failure. Since `BackupAndRestore` only supports eager mode, in graph mode consider using `ModelCheckpoint`."
       ]
     },
     {
@@ -382,8 +385,8 @@
         "# Define a callback for printing the learning rate at the end of each epoch.\n",
         "class PrintLR(tf.keras.callbacks.Callback):\n",
         "  def on_epoch_end(self, epoch, logs=None):\n",
-        "    print('\\nLearning rate for epoch {} is {}'.format(epoch + 1,\n",
-        "                                                      model.optimizer.lr.numpy()))"
+        "    print('\\nLearning rate for epoch {} is {}'.format(",
+        "        epoch + 1, model.optimizer.lr.numpy()))"
       ]
     },
     {
@@ -419,7 +422,7 @@
         "id": "6EophnOAB3YD"
       },
       "source": [
-        "Now, train the model in the usual way by calling `Model.fit` on the model and passing in the dataset created at the beginning of the tutorial. This step is the same whether you are distributing the training or not."
+        "Now, train the model in the usual way by calling Keras `Model.fit` on the model and passing in the dataset created at the beginning of the tutorial. This step is the same whether you are distributing the training or not."
       ]
     },
     {
@@ -535,7 +538,7 @@
         "id": "Xa87y_A0vRma"
       },
       "source": [
-        "Export the graph and the variables to the platform-agnostic SavedModel format using `Model.save`. After your model is saved, you can load it with or without the `Strategy.scope`."
+        "Export the graph and the variables to the platform-agnostic SavedModel format using Keras `Model.save`. After your model is saved, you can load it with or without the `Strategy.scope`."
       ]
     },
     {
@@ -626,7 +629,7 @@
         "\n",
         "More examples that use different distribution strategies with the Keras `Model.fit` API:\n",
         "\n",
-        "1. The [Solve GLUE tasks using BERT on TPU](https://www.tensorflow.org/text/tutorials/bert_glue) tutorial uses `tf.distribute.MirroredStrategy` for training on GPUs and `tf.distribute.TPUStrategy`—on TPUs.\n",
+        "1. The [Solve GLUE tasks using BERT on TPU](https://www.tensorflow.org/text/tutorials/bert_glue) tutorial uses `tf.distribute.MirroredStrategy` for training on GPUs and `tf.distribute.TPUStrategy` on TPUs.\n",
         "1. The [Save and load a model using a distribution strategy](save_and_load.ipynb) tutorial demonstates how to use the SavedModel APIs with `tf.distribute.Strategy`.\n",
         "1. The [official TensorFlow models](https://github.com/tensorflow/models/tree/master/official) can be configured to run multiple distribution strategies.\n",
         "\n",

From a3fbfca0af5773cec68c17c1b1ea6cb21d35f791 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Fri, 1 Apr 2022 07:00:37 -0700
Subject: [PATCH 051/872] Move special case out of library into user-script.

PiperOrigin-RevId: 438814932
---
 .../tensorflow_docs/api_generator/generate_lib.py  | 14 --------------
 .../api_generator/generate_lib_test.py             |  3 ---
 2 files changed, 17 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py
index ddb6dff9abb..2480324f373 100644
--- a/tools/tensorflow_docs/api_generator/generate_lib.py
+++ b/tools/tensorflow_docs/api_generator/generate_lib.py
@@ -15,7 +15,6 @@
 """Generate tensorflow.org style API Reference docs for a Python module."""
 
 import collections
-import importlib
 import inspect
 import os
 import pathlib
@@ -576,24 +575,11 @@ def write_docs(
     toc_gen = GenerateToc(module_children)
     toc_dict = toc_gen.generate()
 
-    # Replace the overview path *only* for 'TensorFlow' to
-    # `/api_docs/python/tf_overview`. This will be redirected to
-    # `/api_docs/python/tf`.
-    toc_values = toc_dict['toc'][0]
-    if toc_values['title'] == 'tf':
-      section = toc_values['section'][0]
-      section['path'] = str(site_path / 'tf_overview')
-
     leftnav_toc = output_dir / root_module_name / '_toc.yaml'
     with open(leftnav_toc, 'w') as toc_file:
       yaml.dump(toc_dict, toc_file, default_flow_style=False)
 
   if redirects and gen_redirects:
-    if yaml_toc and toc_values['title'] == 'tf':
-      redirects.append({
-          'from': str(site_path / 'tf_overview'),
-          'to': str(site_path / 'tf'),
-      })
     redirects_dict = {
         'redirects': sorted(redirects, key=lambda redirect: redirect['from'])
     }
diff --git a/tools/tensorflow_docs/api_generator/generate_lib_test.py b/tools/tensorflow_docs/api_generator/generate_lib_test.py
index b7a2f03374e..5673cab3f42 100644
--- a/tools/tensorflow_docs/api_generator/generate_lib_test.py
+++ b/tools/tensorflow_docs/api_generator/generate_lib_test.py
@@ -114,9 +114,6 @@ def test_write(self):
             'redirects': [{
                 'from': '/api_docs/python/tf/test_function',
                 'to': '/api_docs/python/tf/TestModule/test_function'
-            }, {
-                'from': '/api_docs/python/tf_overview',
-                'to': '/api_docs/python/tf'
             }]
         })
 

From a541d7e37ce125738e2d9b0c04b03be83ae37e5d Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Fri, 1 Apr 2022 10:10:06 -0700
Subject: [PATCH 052/872] Add an ApiTree data-structure.

doc_generator_visitor -> build the ApiTree from the PathTree.
toc -> add code to generate the toc from the ApiTree (not used yet).

PiperOrigin-RevId: 438850818
---
 tools/tensorflow_docs/api_generator/config.py |   5 +-
 .../api_generator/doc_generator_visitor.py    | 221 +++++++++++++-----
 .../doc_generator_visitor_test.py             | 195 ++++++++++++++--
 .../api_generator/generate_lib.py             |   8 +-
 tools/tensorflow_docs/api_generator/toc.py    | 123 +++++++++-
 .../tensorflow_docs/api_generator/toc_test.py |  71 +++++-
 6 files changed, 539 insertions(+), 84 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/config.py b/tools/tensorflow_docs/api_generator/config.py
index 45ace4ee794..0df0d515dd3 100644
--- a/tools/tensorflow_docs/api_generator/config.py
+++ b/tools/tensorflow_docs/api_generator/config.py
@@ -18,7 +18,7 @@ class ParserConfig(object):
   """Stores all indexes required to parse the docs."""
 
   def __init__(self, reference_resolver, duplicates, duplicate_of, tree, index,
-               reverse_index, base_dir, code_url_prefix):
+               reverse_index, path_tree, base_dir, code_url_prefix):
     """Object with the common config for docs_for_object() calls.
 
     Args:
@@ -42,11 +42,10 @@ def __init__(self, reference_resolver, duplicates, duplicate_of, tree, index,
     self.tree = tree
     self.reverse_index = reverse_index
     self.index = index
+    self.path_tree = path_tree
     self.base_dir = base_dir
     self.code_url_prefix = code_url_prefix
 
   def py_name_to_object(self, full_name):
     """Return the Python object for a Python symbol name."""
     return self.index[full_name]
-
-
diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
index b3318be95be..8fb2564edb6 100644
--- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
+++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
@@ -16,9 +16,13 @@
 
 import collections
 import dataclasses
+import functools
 import inspect
 
-from typing import Any, Dict, List, Optional, Mapping, Tuple
+from typing import Any, Dict, List, Optional, Tuple
+
+from tensorflow_docs.api_generator import obj_type as obj_type_lib
+
 
 ApiPath = Tuple[str, ...]
 
@@ -75,6 +79,9 @@ def __repr__(self):
 
   __str__ = __repr__
 
+  def __eq__(self, other):
+    raise ValueError("Don't try to compare these")
+
   @property
   def short_name(self) -> str:
     return self.path[-1]
@@ -84,70 +91,38 @@ def full_name(self) -> str:
     return '.'.join(self.path)
 
 
-class PathTree(Mapping[ApiPath, PathTreeNode]):
+class PathTree(Dict[ApiPath, PathTreeNode]):
   """An index/tree of all object-paths in the API.
 
   Items must be inserted in order, from root to leaf.
 
-  Acts as a Dict[ApiPath, PathTreeNode].
 
   Attributes:
     root: The root `PathTreeNode`
   """
 
   def __init__(self):
-    root = PathTreeNode(path=(), py_object=None, parent=None)
-    self._index: Dict[ApiPath, PathTreeNode] = {(): root}
+    root = PathTreeNode(path=(), py_object=None, parent=None, children={})
+    super().__setitem__((), root)
 
     self.root: PathTreeNode = root
     self._nodes_for_id: Dict[int, List[PathTreeNode]] = (
         collections.defaultdict(list))
 
-  def keys(self):
-    """Returns the paths currently contained in the tree."""
-    return self._index.keys()
-
-  def __iter__(self):
-    return iter(self._index)
-
-  def __len__(self):
-    return len(self._index)
-
-  def values(self):
-    """Returns the path-nodes for each node currently in the tree."""
-    return self._index.values()
-
-  def items(self):
-    """Returns the (path, node) pairs for each node currently in the tree."""
-    return self._index.items()
-
-  def __contains__(self, path: ApiPath) -> bool:
-    """Returns `True` if path exists in the tree.
-
-    Args:
-      path: A tuple of strings, the api path to the object.
-
-    Returns:
-      True if `path` exists in the tree.
-    """
-    return path in self._index
-
-  def __getitem__(self, path: ApiPath) -> PathTreeNode:
-    """Fetch an item from the tree.
-
-    Args:
-      path: A tuple of strings, the api path to the object.
-
-    Returns:
-      A `PathTreeNode`.
+  def __eq__(self, other):
+    raise ValueError("Don't try to compare these")
 
-    Raises:
-      KeyError: If no node can be found at that path.
-    """
-    return self._index[path]
+  def iter_nodes(self):
+    """Iterate over the nodes in BFS order."""
+    stack = collections.deque([self.root])
+    while stack:
+      children = list(stack.popleft().children.values())
+      yield from children
+      stack.extend(children)
 
-  def get(self, path: ApiPath, default=None):
-    return self._index.get(path, default)
+  def __contains__(self, path: ApiPath) -> bool:  # pylint: disable=useless-super-delegation
+    # TODO(b/184563451): remove
+    return super().__contains__(path)
 
   def __setitem__(self, path: ApiPath, obj: Any):
     """Add an object to the tree.
@@ -157,16 +132,17 @@ def __setitem__(self, path: ApiPath, obj: Any):
       obj: The python object.
     """
     parent_path = path[:-1]
-    parent = self._index[parent_path]
+    parent = self[parent_path]
 
     node = PathTreeNode(path=path, py_object=obj, parent=parent)
 
-    self._index[path] = node
+    super().__setitem__(path, node)
     if not maybe_singleton(obj):
       # We cannot use the duplicate mechanism for some constants, since e.g.,
       # id(c1) == id(c2) with c1=1, c2=1. This isn't problematic since constants
       # have no usable docstring and won't be documented automatically.
-      self.nodes_for_obj(obj).append(node)
+      nodes = self.nodes_for_obj(obj)
+      nodes.append(node)
     parent.children[node.short_name] = node
 
   def nodes_for_obj(self, py_object) -> List[PathTreeNode]:
@@ -315,7 +291,7 @@ class or module.
 
     return children
 
-  def _score_name(self, name):
+  def _score_name(self, name: str):
     """Return a tuple of scores indicating how to sort for the best name.
 
     This function is meant to be used as the `key` to the `sorted` function.
@@ -443,3 +419,144 @@ def build(self):
     self._duplicate_of = duplicate_of
     self._duplicates = duplicates
     self._reverse_index = reverse_index
+
+
+@dataclasses.dataclass(repr=False)
+class ApiTreeNode(PathTreeNode):
+  aliases: List[ApiPath] = dataclasses.field(default_factory=list)
+
+  @property
+  def obj_type(self) -> obj_type_lib.ObjType:
+    return obj_type_lib.ObjType.get(self.py_object)
+
+
+class ApiTree(Dict[ApiPath, ApiTreeNode]):
+  """Public API index.
+
+  Items must be inserted in order from root to leaves.
+
+  Lookup a path-tuple to fetch a node:
+
+  ```
+  node = index[path]
+  ```
+
+  Use the `node_from_obj` method to lookup the node for a python object:
+
+  ```
+  node = index.node_from_obj(obj)
+  ```
+
+  Remember that `maybe_singelton` (numbers, strings, tuples) classes can't be
+  looked up this way.
+
+  To build a tree, nodes must be inserted in tree order starting from the root.
+
+
+  Attributes:
+    root: The root `ApiFileNode` of the tree.
+  """
+
+  def __init__(self):
+    root = ApiTreeNode(
+        path=(), py_object=None, parent=None, aliases=[()])  # type: ignore
+    self.root = root
+    super().__setitem__((), root)
+    self._nodes = []
+    self._node_for_object = {}
+
+  def __eq__(self, other):
+    raise ValueError("Don't try to compare these")
+
+  def node_for_object(self, obj: Any) -> Optional[ApiTreeNode]:
+    if maybe_singleton(obj):
+      return None
+    return self._node_for_object.get(id(obj), None)
+
+  def __contains__(self, path: ApiPath) -> bool:  # pylint: disable=useless-super-delegation
+    # TODO(b/184563451): remove
+    return super().__contains__(path)
+
+  def iter_nodes(self):
+    """Iterate over the nodes in BFS order."""
+    stack = collections.deque([self.root])
+    while stack:
+      children = list(stack.popleft().children.values())
+      yield from children
+      stack.extend(children)
+
+  def __setitem__(self, *args, **kwargs):
+    raise TypeError('Use .insert instead of setitem []')
+
+  def insert(self, path: ApiPath, py_object: Any, aliases: List[ApiPath]):
+    """Add an object to the index."""
+    assert path not in self, 'A path was inserted twice.'
+
+    parent_path = path[:-1]
+    parent = self[parent_path]
+
+    node = ApiTreeNode(
+        path=path, py_object=py_object, aliases=aliases, parent=parent)
+
+    super().__setitem__(path, node)
+    self._nodes.append(node)
+    for alias in aliases:
+      if alias == path:
+        continue
+      assert alias not in self
+      super().__setitem__(alias, node)
+
+    self._node_for_object[id(node.py_object)] = node
+
+    parent.children[node.short_name] = node
+
+  @classmethod
+  def from_path_tree(cls, path_tree: PathTree, score_name_fn) -> 'ApiTree':
+    """Create an ApiTree from an PathTree.
+
+    Args:
+      path_tree: The `PathTree` to convert.
+      score_name_fn: The name scoring function.
+
+    Returns:
+      an `ApiIndex`, created from `path_tree`.
+    """
+    self = cls()
+
+    active_nodes = collections.deque(path_tree.root.children.values())
+    while active_nodes:
+      current_node = active_nodes.popleft()
+      if current_node.path in self:
+        continue
+
+      duplicate_nodes = set(
+          # Singelton objects will return [].
+          path_tree.nodes_for_obj(current_node.py_object))
+      # Add the current node in case it's a singelton.
+      duplicate_nodes.add(current_node)
+
+      parents = [node.parent for node in duplicate_nodes]
+
+      # Choose the master name with a lexical sort on the tuples returned by
+      # by _score_name.
+      if not all(parent.path in self for parent in parents):
+        # rewind
+        active_nodes.appendleft(current_node)
+        # do each duplicate's immediate parents first.
+        for parent in parents:
+          if parent.path in self:
+            continue
+          active_nodes.appendleft(parent)
+        continue
+      # If we've made it here, the immediate parents of each of the paths have
+      # been processed, so now we can choose its master name.
+      aliases = [node.path for node in duplicate_nodes]
+
+      master_path = min(['.'.join(a) for a in aliases], key=score_name_fn)
+      master_path = tuple(master_path.split('.'))
+
+      self.insert(master_path, current_node.py_object, aliases)
+
+      active_nodes.extend(current_node.children.values())
+
+    return self
diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py
index cea7bf2ddc7..c6900ec77a8 100644
--- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py
+++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py
@@ -15,13 +15,17 @@
 """Tests for tools.docs.doc_generator_visitor."""
 
 import argparse
+import inspect
+import io
 import os
+import textwrap
 import types
 
 from absl.testing import absltest
 
 from tensorflow_docs.api_generator import doc_generator_visitor
 from tensorflow_docs.api_generator import generate_lib
+from tensorflow_docs.api_generator import toc as toc_lib
 
 
 class NoDunderVisitor(doc_generator_visitor.DocGeneratorVisitor):
@@ -44,6 +48,7 @@ def test_call_module(self):
 
     self.assertEqual({'doc_generator_visitor': ['DocGeneratorVisitor']},
                      visitor.tree)
+
     self.assertEqual({
         'doc_generator_visitor': doc_generator_visitor,
         'doc_generator_visitor.DocGeneratorVisitor':
@@ -51,19 +56,24 @@ def test_call_module(self):
     }, visitor.index)
 
   def test_call_class(self):
+
+    class ExampleClass:
+
+      def example_method(self):
+        pass
+
     visitor = doc_generator_visitor.DocGeneratorVisitor()
     visitor(
-        ('DocGeneratorVisitor',), doc_generator_visitor.DocGeneratorVisitor,
-        [('index', doc_generator_visitor.DocGeneratorVisitor.reverse_index)])
+        parent_path=('ExampleClass',),
+        parent=ExampleClass,
+        children=[('example_method', ExampleClass.example_method)])
 
-    self.assertEqual({'DocGeneratorVisitor': ['index']},
-                     visitor.tree)
-    self.assertEqual({
-        'DocGeneratorVisitor':
-            doc_generator_visitor.DocGeneratorVisitor,
-        'DocGeneratorVisitor.index':
-            doc_generator_visitor.DocGeneratorVisitor.reverse_index
-    }, visitor.index)
+    self.assertEqual({'ExampleClass': ['example_method']}, visitor.tree)
+    self.assertEqual(
+        {
+            'ExampleClass': ExampleClass,
+            'ExampleClass.example_method': ExampleClass.example_method,
+        }, visitor.index)
 
   def test_call_raises(self):
     visitor = doc_generator_visitor.DocGeneratorVisitor()
@@ -135,9 +145,8 @@ class Parent(object):
         private_map={},
         visitor_cls=NoDunderVisitor)
 
-    self.assertEqual(
-        sorted(['tf.contrib.Parent', 'tf.submodule.Parent']),
-        visitor.duplicates['tf.submodule.Parent'])
+    self.assertCountEqual(['tf.contrib.Parent', 'tf.submodule.Parent'],
+                          visitor.duplicates['tf.submodule.Parent'])
 
     self.assertEqual({
         'tf.contrib.Parent': 'tf.submodule.Parent',
@@ -169,11 +178,8 @@ class Child(Parent):
         private_map={},
         visitor_cls=NoDunderVisitor)
 
-    self.assertEqual(
-        sorted([
-            'tf.Parent.obj1',
-            'tf.Child.obj1',
-        ]), visitor.duplicates['tf.Parent.obj1'])
+    self.assertCountEqual(['tf.Parent.obj1', 'tf.Child.obj1'],
+                          visitor.duplicates['tf.Parent.obj1'])
 
     self.assertEqual({
         'tf.Child.obj1': 'tf.Parent.obj1',
@@ -204,9 +210,8 @@ class Parent(object):
         private_map={},
         visitor_cls=NoDunderVisitor)
 
-    self.assertEqual(
-        sorted(['tf.Parent', 'tf.submodule.submodule2.Parent']),
-        visitor.duplicates['tf.Parent'])
+    self.assertCountEqual(['tf.Parent', 'tf.submodule.submodule2.Parent'],
+                          visitor.duplicates['tf.Parent'])
 
     self.assertEqual({
         'tf.submodule.submodule2.Parent': 'tf.Parent'
@@ -316,7 +321,153 @@ def test_duplicate_singleton(self):
     tree[('tf', 'sub2')] = tf.sub2
     tree[('tf', 'sub2', 'thing')] = tf.sub2.thing
 
-    self.assertEmpty(tree.nodes_for_obj(tf.sub.thing), [])
+    found = tree.nodes_for_obj(tf.sub.thing)
+    self.assertIsNotNone(found)
+    self.assertEmpty(found)
+
+
+class ApiTreeTest(absltest.TestCase):
+
+  def _make_fake_module(self) -> types.ModuleType:
+
+    class Parent:
+
+      def method1(self):
+        pass
+
+      def method2(self):
+        pass
+
+    class Child(Parent):
+
+      def method2(self):
+        pass
+
+      def method3(self):
+        pass
+
+    class Outer(object):
+      attribute = object()
+
+      class Nested(object):
+        pass
+
+    fun1 = lambda x: x
+    fun2 = lambda x: x
+
+    tf = types.ModuleType('tf')
+    tf.__file__ = __file__
+    tf.Parent = Parent
+    tf.Outer = Outer
+    tf.fun1 = fun1
+    tf.sub1 = types.ModuleType('sub1')
+    tf.sub1.Parent = Parent
+    tf.sub2 = types.ModuleType('sub2')
+    tf.sub2.Child = Child
+    tf.sub2.fun2 = fun2
+    tf.sub1.sub2 = tf.sub2
+
+    return tf
+
+  def test_api_tree(self):
+    seven = 7
+    tf = self._make_fake_module()
+
+    api_tree = doc_generator_visitor.ApiTree()
+    api_tree.insert(path=('tf',), py_object=tf, aliases=[('tf',)])
+    api_tree.insert(
+        path=('tf', 'Parent'),
+        py_object=tf.Parent,
+        aliases=[('tf', 'Parent'), ('tf', 'Parent2')])
+    api_tree.insert(
+        path=('tf', 'seven'), py_object=seven, aliases=[('tf', 'seven')])
+
+    # A node can be looked up by any alias
+    self.assertIs(api_tree[('tf', 'Parent')], api_tree[('tf', 'Parent2')])
+    # Nodes only show up once when iterating
+    self.assertEqual([
+        api_tree[('tf',)], api_tree[('tf', 'Parent')], api_tree[('tf', 'seven')]
+    ], list(api_tree.iter_nodes()))
+    # Test lookup by object.
+    self.assertIs(api_tree[('tf', 'Parent')],
+                  api_tree.node_for_object(tf.Parent))
+    # You can't lookup things that maybe singeltons.
+    self.assertIs(api_tree[('tf', 'seven')].py_object, seven)
+    self.assertIsNone(api_tree.node_for_object(seven))
+
+  def test_from_path_tree(self):
+    tf = self._make_fake_module()
+
+    path_tree = doc_generator_visitor.PathTree()
+    path_tree[('tf',)] = tf
+    path_tree[('tf', 'Parent')] = tf.Parent
+    path_tree[('tf', 'Parent2')] = tf.Parent
+
+    result = doc_generator_visitor.ApiTree.from_path_tree(
+        path_tree, score_name_fn=lambda name: name)
+
+    expected = doc_generator_visitor.ApiTree()
+    expected.insert(path=('tf',), py_object=tf, aliases=[('tf',)])
+    expected.insert(
+        path=('tf', 'Parent'),
+        py_object=tf.Parent,
+        aliases=[('tf', 'Parent'), ('tf', 'Parent2')])
+
+    result = sorted(result.iter_nodes(), key=lambda node: node.path)
+    expected = sorted(expected.iter_nodes(), key=lambda node: node.path)
+
+    # Circular references make it hard to compare trees or nodes.
+    for e, r in zip(result, expected):
+      self.assertEqual(e.path, r.path)
+      self.assertIs(e.py_object, r.py_object)
+      self.assertCountEqual(e.aliases, r.aliases)
+      self.assertCountEqual(e.children.keys(), r.children.keys())
+
+  def test_api_tree_toc_integration(self):
+    tf = self._make_fake_module()
+
+    visitor = generate_lib.extract([('tf', tf)],
+                                   base_dir=os.path.dirname(tf.__file__),
+                                   private_map={},
+                                   visitor_cls=NoDunderVisitor)
+
+    api_tree = doc_generator_visitor.ApiTree.from_path_tree(
+        visitor.path_tree, visitor._score_name)
+
+    toc = toc_lib.TocBuilder(site_path='/').build(api_tree)
+
+    stream = io.StringIO()
+    toc.write(stream)
+
+    expected = textwrap.dedent("""\
+        toc:
+        - title: tf
+          section:
+          - title: Overview
+            path: /tf
+          - title: Outer
+            path: /tf/Outer
+          - title: Outer.Nested
+            path: /tf/Outer/Nested
+          - title: fun1
+            path: /tf/fun1
+          - title: sub1
+            section:
+            - title: Overview
+              path: /tf/sub1
+            - title: Parent
+              path: /tf/sub1/Parent
+          - title: sub2
+            section:
+            - title: Overview
+              path: /tf/sub2
+            - title: Child
+              path: /tf/sub2/Child
+            - title: fun2
+              path: /tf/sub2/fun2
+        """)
+
+    self.assertEqual(expected, stream.getvalue())
 
 
 if __name__ == '__main__':
diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py
index 2480324f373..90f9c10d35c 100644
--- a/tools/tensorflow_docs/api_generator/generate_lib.py
+++ b/tools/tensorflow_docs/api_generator/generate_lib.py
@@ -766,7 +766,9 @@ def make_reference_resolver(self, visitor):
     return reference_resolver_lib.ReferenceResolver.from_visitor(
         visitor, py_module_names=[self._short_name])
 
-  def make_parser_config(self, visitor, reference_resolver):
+  def make_parser_config(self,
+                         visitor: doc_generator_visitor.DocGeneratorVisitor):
+    reference_resolver = self.make_reference_resolver(visitor)
     return config.ParserConfig(
         reference_resolver=reference_resolver,
         duplicates=visitor.duplicates,
@@ -774,6 +776,7 @@ def make_parser_config(self, visitor, reference_resolver):
         tree=visitor.tree,
         index=visitor.index,
         reverse_index=visitor.reverse_index,
+        path_tree=visitor.path_tree,
         base_dir=self._base_dir,
         code_url_prefix=self._code_url_prefix)
 
@@ -791,10 +794,9 @@ def run_extraction(self):
         private_map=self._private_map,
         visitor_cls=self._visitor_cls,
         callbacks=self._callbacks)
-    reference_resolver = self.make_reference_resolver(visitor)
 
     # Write the api docs.
-    parser_config = self.make_parser_config(visitor, reference_resolver)
+    parser_config = self.make_parser_config(visitor)
     return parser_config
 
   def build(self, output_dir):
diff --git a/tools/tensorflow_docs/api_generator/toc.py b/tools/tensorflow_docs/api_generator/toc.py
index 800eb0062bf..873e2cb804d 100644
--- a/tools/tensorflow_docs/api_generator/toc.py
+++ b/tools/tensorflow_docs/api_generator/toc.py
@@ -18,9 +18,15 @@
 import dataclasses
 import enum
 import os
+import pathlib
 
 from typing import Any, IO, Iterator, List, Optional, Tuple, Union
 
+from tensorflow_docs.api_generator import doc_controls
+from tensorflow_docs.api_generator import doc_generator_visitor
+from tensorflow_docs.api_generator import obj_type as obj_type_lib
+from tensorflow_docs.api_generator import signature
+
 import yaml
 
 
@@ -146,7 +152,7 @@ class Toc(Entry):
   toc: List[Entry]
 
   @contextlib.contextmanager
-  def _maybe_open(self, file: Union[os.PathLike, IO]) -> Iterator[IO]:
+  def _maybe_open(self, file: Union[os.PathLike, IO[str]]) -> Iterator[IO[str]]:
     if isinstance(file, os.PathLike):
       with open(file, 'w') as stream:
         yield stream
@@ -154,7 +160,120 @@ def _maybe_open(self, file: Union[os.PathLike, IO]) -> Iterator[IO]:
       stream = file
       yield stream
 
-  def write(self, file: Union[os.PathLike, IO]) -> None:
+  def write(self, file: Union[os.PathLike, IO[str]]) -> None:
     with self._maybe_open(file) as stream:
       yaml.dump(
           self, stream=stream, default_flow_style=False, Dumper=_TocDumper)
+
+
+class TocBuilder:
+  """A class to build a Toc from an ApiTree."""
+
+  def __init__(self, site_path):
+    self.site_path = pathlib.Path(site_path)
+
+  def build(self, api_tree: doc_generator_visitor.ApiTree) -> Toc:
+    """Create a `Toc` from an `ApiTree`."""
+    entries = []
+    for child in api_tree.root.children.values():
+      entries.extend(self._entries_from_api_node(child))
+    return Toc(toc=entries)
+
+  def _entries_from_api_node(
+      self, api_node: doc_generator_visitor.ApiTreeNode) -> List[Entry]:
+    """Converts an ApiTreeNode to a list of toc entries."""
+    obj_type = api_node.obj_type
+
+    if obj_type is obj_type_lib.ObjType.MODULE:
+      return [self._make_section(api_node)]
+    if obj_type is obj_type_lib.ObjType.CLASS:
+      return self._flat_class_entries(api_node)
+    if obj_type in [
+        obj_type_lib.ObjType.CALLABLE, obj_type_lib.ObjType.TYPE_ALIAS
+    ]:
+      return [self._make_link(api_node)]
+    else:
+      return []
+
+  def _make_link(self,
+                 api_node: doc_generator_visitor.ApiTreeNode,
+                 title: Optional[str] = None) -> Link:
+
+    docpath = pathlib.Path(self.site_path, *api_node.path)
+    title = title or api_node.short_name
+    return Link(
+        title=title, path=str(docpath), status=self._make_status(api_node))
+
+  def _make_section(self,
+                    api_node: doc_generator_visitor.ApiTreeNode) -> Section:
+    """Create a `toc.Section` from a module's ApiTreeNode."""
+    overview = self._make_overview(api_node)
+    entries = []
+    for child in api_node.children.values():
+      entries.extend(self._entries_from_api_node(child))
+    entries = sorted(entries, key=self._section_order_key)
+    entries = [overview] + entries
+
+    status = self._make_status(api_node)
+    return Section(title=api_node.short_name, section=entries, status=status)
+
+  def _make_overview(self, api_node: doc_generator_visitor.ApiTreeNode):
+    docpath = pathlib.Path(self.site_path, *api_node.path)
+    return Link(title='Overview', path=str(docpath))
+
+  def _section_order_key(self, entry: Entry) -> Tuple[bool, str]:
+    title = getattr(entry, 'title', None)
+    is_section = isinstance(entry, Section)
+
+    return (is_section, title)
+
+  def _flat_class_entries(self,
+                          api_node: doc_generator_visitor.ApiTreeNode,
+                          title: Optional[str] = None) -> List[Entry]:
+    """Returns entries for both `Class` and `Class.Nested`."""
+    title = title or api_node.short_name
+    entries = [self._make_link(api_node, title=title)]
+    for name, child_node in api_node.children.items():
+      if child_node.obj_type in [
+          obj_type_lib.ObjType.CLASS, obj_type_lib.ObjType.MODULE
+      ]:
+        subtitle = f'{title}.{name}'
+        entries.extend(self._flat_class_entries(child_node, title=subtitle))
+
+    return entries
+
+  def _make_status(self, api_node: doc_generator_visitor.ApiTreeNode):
+    """Returns the toc.Status of an ApiTreeNode."""
+    if self._is_deprecated(api_node):
+      return Status.DEPRECATED
+    if self._is_experimental(api_node):
+      return Status.EXPERIMENTAL
+    return None
+
+  def _is_experimental(self, api_node: doc_generator_visitor.ApiTreeNode):
+    return 'experimental' in api_node.short_name.lower()
+
+  def _is_deprecated(self, api_node: doc_generator_visitor.ApiTreeNode):
+    """Checks if an object is deprecated or not.
+
+    Each deprecated function has a `_tf_decorator.decorator_name` attribute.
+    Check the docstring of that function to confirm if the function was
+    indeed deprecated. If a different deprecation setting was used on the
+    function, then "THIS FUNCTION IS DEPRECATED" substring won't be inserted
+    into the docstring of that function by the decorator.
+
+    Args:
+      api_node: The node to evaluate.
+
+    Returns:
+      True if depreacted else False.
+    """
+    if doc_controls.is_deprecated(api_node.py_object):
+      return True
+
+    decorator_list = signature.extract_decorators(api_node.py_object)
+    if any('deprecat' in dec for dec in decorator_list):
+      docstring = getattr(api_node.py_object, '__doc__') or ''
+      return 'THIS FUNCTION IS DEPRECATED' in docstring
+
+    return False
diff --git a/tools/tensorflow_docs/api_generator/toc_test.py b/tools/tensorflow_docs/api_generator/toc_test.py
index b5033517ab8..77189d80565 100644
--- a/tools/tensorflow_docs/api_generator/toc_test.py
+++ b/tools/tensorflow_docs/api_generator/toc_test.py
@@ -17,10 +17,11 @@
 import textwrap
 import types
 
-from tensorflow_docs.api_generator import toc as toc_lib
-from tensorflow_docs.api_generator import doc_generator_visitor
 from absl.testing import absltest
 
+from tensorflow_docs.api_generator import doc_generator_visitor
+from tensorflow_docs.api_generator import toc as toc_lib
+
 
 class TestToc(absltest.TestCase):
 
@@ -74,6 +75,72 @@ def test_replace(self):
         title='New title.', path='/path/to/link', status=toc_lib.Status.NEW)
     self.assertEqual(expected, new_link)
 
+  def _make_tree(self) -> doc_generator_visitor.ApiTree:
+    api_tree = doc_generator_visitor.ApiTree()
+    api_tree.insert(
+        path=('module',), py_object=types.ModuleType('module'), aliases=[])
+    api_tree.insert(path=('module', 'func1'), py_object=lambda x: x, aliases=[])
+    api_tree.insert(
+        path=('module', 'Class'),
+        py_object=types.new_class('Class'),
+        aliases=[])
+    api_tree.insert(
+        path=('module', 'Class', 'method'), py_object=lambda x: x, aliases=[])
+    api_tree.insert(
+        path=('module', 'Class', 'NestedClass'),
+        py_object=types.new_class('NestedClass'),
+        aliases=[])
+    api_tree.insert(
+        path=('module', 'Class', 'NestedClass', 'method2'),
+        py_object=lambda x: x,
+        aliases=[])
+    api_tree.insert(
+        path=('module', 'Class', 'constant'),
+        py_object='Just a string.',
+        aliases=[])
+    api_tree.insert(
+        path=('module', 'submodule'),
+        py_object=types.ModuleType('submodule'),
+        aliases=[])
+    api_tree.insert(
+        path=('module', 'submodule', 'func2'),
+        py_object=lambda x: x,
+        aliases=[])
+    api_tree.insert(
+        path=('module', 'submodule', 'constant'),
+        py_object='Another string.',
+        aliases=[])
+    return api_tree
+
+  def test_toc_builder(self):
+    api_tree = self._make_tree()
+    builder = toc_lib.TocBuilder('/path/in/site')
+    toc = builder.build(api_tree)
+    stream = io.StringIO()
+    toc.write(stream)
+
+    expected = textwrap.dedent("""\
+        toc:
+        - title: module
+          section:
+          - title: Overview
+            path: /path/in/site/module
+          - title: Class
+            path: /path/in/site/module/Class
+          - title: Class.NestedClass
+            path: /path/in/site/module/Class/NestedClass
+          - title: func1
+            path: /path/in/site/module/func1
+          - title: submodule
+            section:
+            - title: Overview
+              path: /path/in/site/module/submodule
+            - title: func2
+              path: /path/in/site/module/submodule/func2
+        """)
+
+    self.assertEqual(expected, stream.getvalue())
+
 
 if __name__ == '__main__':
   absltest.main()

From 02f808a53848a32cd3c6a36eafdc901b4cf9fc26 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Sun, 3 Apr 2022 19:39:49 -0700
Subject: [PATCH 053/872] Switch score_name to take a tuple instead of a
 string.

This will be necessary (with the new toc generator) when generating docs for something like `tf.lite` where the root module name contains a dot. Splittling on `.` gives a path ('tf','lite') which doesn't exist when generating just `tf.lite`.

PiperOrigin-RevId: 439213594
---
 .../api_generator/doc_generator_visitor.py    | 73 ++++++++++++-------
 .../api_generator/pretty_docs/base_page.py    |  2 +-
 2 files changed, 48 insertions(+), 27 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
index 8fb2564edb6..184f70854f2 100644
--- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
+++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
@@ -19,7 +19,7 @@
 import functools
 import inspect
 
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, Optional, NamedTuple, Tuple
 
 from tensorflow_docs.api_generator import obj_type as obj_type_lib
 
@@ -291,7 +291,14 @@ class or module.
 
     return children
 
-  def _score_name(self, name: str):
+  class NameScore(NamedTuple):
+    defining_class_score: int
+    experimental_score: int
+    keras_score: int
+    module_length_score: int
+    path: ApiPath
+
+  def _score_name(self, path: ApiPath) -> NameScore:
     """Return a tuple of scores indicating how to sort for the best name.
 
     This function is meant to be used as the `key` to the `sorted` function.
@@ -307,18 +314,18 @@ def _score_name(self, name: str):
       name: Fallback, sorts lexicographically on the full_name.
 
     Args:
-      name: the full name to score, for example `tf.estimator.Estimator`
+      path: APiPath to score, for example `('tf','estimator','Estimator')`
 
     Returns:
       A tuple of scores. When sorted the preferred name will have the lowest
       value.
     """
-    parts = name.split('.')
-    short_name = parts[-1]
-    if len(parts) == 1:
-      return (-99, -99, -99, -99, short_name)
+    py_object = self.path_tree[path].py_object
+    if len(path) == 1:
+      return self.NameScore(-99, -99, -99, -99, path)
 
-    container = self._index.get('.'.join(parts[:-1]), name)
+    short_name = path[-1]
+    container = self.path_tree[path[:-1]].py_object
 
     defining_class_score = 1
     if inspect.isclass(container):
@@ -327,30 +334,44 @@ def _score_name(self, name: str):
         defining_class_score = -1
 
     experimental_score = -1
-    if 'contrib' in parts or any('experimental' in part for part in parts):
+    if 'contrib' in path or any('experimental' in part for part in path):
       experimental_score = 1
 
     keras_score = 1
-    if 'keras' in parts:
+    if 'keras' in path:
       keras_score = -1
 
-    while parts:
-      container = self._index['.'.join(parts)]
+    if inspect.ismodule(py_object):
+      # prefer short paths for modules
+      module_length_score = len(path)
+    else:
+      module_length_score = self._get_module_length_score(path)
+
+    return self.NameScore(
+        defining_class_score=defining_class_score,
+        experimental_score=experimental_score,
+        keras_score=keras_score,
+        module_length_score=module_length_score,
+        path=path)
+
+  def _get_module_length_score(self, path):
+    partial_path = list(path)
+    while partial_path:
+      container = self.path_tree[tuple(partial_path[:-1])].py_object
+      partial_path.pop()
       if inspect.ismodule(container):
         break
-      parts.pop()
 
-    module_length = len(parts)
+    module_length = len(partial_path)
 
-    if len(parts) == 2:
+    if module_length == 2:
       # `tf.submodule.thing` is better than `tf.thing`
       module_length_score = -1
     else:
       # shorter is better
       module_length_score = module_length
 
-    return (defining_class_score, experimental_score, keras_score,
-            module_length_score, name)
+    return module_length_score
 
   def build(self):
     """Compute data structures containing information about duplicates.
@@ -398,19 +419,20 @@ def build(self):
       if not aliases:
         aliases = [node]
 
-      names = [alias.full_name for alias in aliases]
+      name_tuples = [alias.path for alias in aliases]
 
-      names = sorted(names)
       # Choose the main name with a lexical sort on the tuples returned by
       # by _score_name.
-      main_name = min(names, key=self._score_name)
+      main_name_tuple = min(name_tuples, key=self._score_name)
+      main_name = '.'.join(main_name_tuple)
 
-      if names:
-        duplicates[main_name] = list(names)
+      names = ['.'.join(name_tuple) for name_tuple in name_tuples]
+      if name_tuples:
+        duplicates[main_name] = sorted(names)
 
-      names.remove(main_name)
       for name in names:
-        duplicate_of[name] = main_name
+        if name != main_name:
+          duplicate_of[name] = main_name
 
       # Set the reverse index to the canonical name.
       if not maybe_singleton(py_object):
@@ -552,8 +574,7 @@ def from_path_tree(cls, path_tree: PathTree, score_name_fn) -> 'ApiTree':
       # been processed, so now we can choose its master name.
       aliases = [node.path for node in duplicate_nodes]
 
-      master_path = min(['.'.join(a) for a in aliases], key=score_name_fn)
-      master_path = tuple(master_path.split('.'))
+      master_path = min(aliases, key=score_name_fn)
 
       self.insert(master_path, current_node.py_object, aliases)
 
diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py
index 764cdd0ca36..66e2bcd9569 100644
--- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py
+++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py
@@ -57,7 +57,7 @@ def top_source_link(self):
     return top_source_link(self.page_info.defined_in)
 
   def build_collapsable_aliases(self):
-    return build_collapsable_aliases(self.page_info.aliases)
+    return build_collapsable_aliases(sorted(self.page_info.aliases))
 
   def top_compat(self):
     return build_top_compat(self.page_info, h_level=2)

From 5eac44723aa7d315e025449033b84d9bdc33ecb6 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 4 Apr 2022 16:27:20 -0700
Subject: [PATCH 054/872] Simplify in generate_lib with the ApiTree and Toc
 classes.

- config.py: Attach the api_tree to the parser_config.
- doc_generator_visitor.py: Use the api_tree to generate the toc.
- toc.py: move root's submodules from children to peers.

PiperOrigin-RevId: 439439106
---
 tools/tensorflow_docs/api_generator/config.py |   8 +-
 .../api_generator/doc_generator_visitor.py    |  25 +-
 .../api_generator/generate_lib.py             | 419 +-----------------
 .../api_generator/reference_resolver.py       |   3 +
 tools/tensorflow_docs/api_generator/toc.py    |  68 ++-
 .../tensorflow_docs/api_generator/toc_test.py |  29 ++
 6 files changed, 141 insertions(+), 411 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/config.py b/tools/tensorflow_docs/api_generator/config.py
index 0df0d515dd3..50b8b86b8b4 100644
--- a/tools/tensorflow_docs/api_generator/config.py
+++ b/tools/tensorflow_docs/api_generator/config.py
@@ -17,8 +17,9 @@
 class ParserConfig(object):
   """Stores all indexes required to parse the docs."""
 
-  def __init__(self, reference_resolver, duplicates, duplicate_of, tree, index,
-               reverse_index, path_tree, base_dir, code_url_prefix):
+  def __init__(self, *, reference_resolver, duplicates, duplicate_of, tree,
+               index, reverse_index, path_tree, api_tree, base_dir,
+               code_url_prefix):
     """Object with the common config for docs_for_object() calls.
 
     Args:
@@ -32,6 +33,8 @@ def __init__(self, reference_resolver, duplicates, duplicate_of, tree, index,
         members. Used to populate the members section of a class or module page.
       index: A `dict` mapping full names to objects.
       reverse_index: A `dict` mapping object ids to full names.
+      path_tree: A PathTree datastructure to manage all the API paths.
+      api_tree: A PathTree datastructure to manage all the API objects.
       base_dir: A base path that is stripped from file locations written to the
         docs.
       code_url_prefix: A Url to pre-pend to the links to file locations.
@@ -43,6 +46,7 @@ def __init__(self, reference_resolver, duplicates, duplicate_of, tree, index,
     self.reverse_index = reverse_index
     self.index = index
     self.path_tree = path_tree
+    self.api_tree = api_tree
     self.base_dir = base_dir
     self.code_url_prefix = code_url_prefix
 
diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
index 184f70854f2..b063fb9221c 100644
--- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
+++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
@@ -16,7 +16,7 @@
 
 import collections
 import dataclasses
-import functools
+import enum
 import inspect
 
 from typing import Any, Dict, List, Optional, NamedTuple, Tuple
@@ -182,6 +182,7 @@ def __init__(self):
     self._duplicate_of: Dict[str, str] = None
 
     self.path_tree = PathTree()
+    self.api_tree = None
 
   @property
   def index(self):
@@ -390,6 +391,8 @@ def build(self):
     if self._reverse_index is not None:
       return
 
+    self.api_tree = ApiTree.from_path_tree(self.path_tree, self._score_name)
+
     # Maps the id of a symbol to its fully qualified name. For symbols that have
     # several aliases, this map contains the first one found.
     # We use id(py_object) to get a hashable value for py_object. Note all
@@ -445,12 +448,32 @@ def build(self):
 
 @dataclasses.dataclass(repr=False)
 class ApiTreeNode(PathTreeNode):
+  """A node in the ApiTree."""
   aliases: List[ApiPath] = dataclasses.field(default_factory=list)
 
   @property
   def obj_type(self) -> obj_type_lib.ObjType:
     return obj_type_lib.ObjType.get(self.py_object)
 
+  class OutputType(enum.Enum):
+    PAGE = 'page'
+    FRAGMENT = 'fragment'
+
+  def output_type(self) -> OutputType:
+    obj_type = obj_type_lib.ObjType.get(self.py_object)
+
+    if obj_type in (obj_type_lib.ObjType.CLASS, obj_type_lib.ObjType.MODULE):
+      return self.OutputType.PAGE
+    elif obj_type in (obj_type_lib.ObjType.CALLABLE,
+                      obj_type_lib.ObjType.TYPE_ALIAS):
+      parent_type = obj_type_lib.ObjType.get(self.parent.py_object)
+      if parent_type is obj_type_lib.ObjType.CLASS:
+        return self.OutputType.FRAGMENT
+      else:
+        return self.OutputType.PAGE
+    else:
+      return self.OutputType.FRAGMENT
+
 
 class ApiTree(Dict[ApiPath, ApiTreeNode]):
   """Public API index.
diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py
index 90f9c10d35c..e7590663490 100644
--- a/tools/tensorflow_docs/api_generator/generate_lib.py
+++ b/tools/tensorflow_docs/api_generator/generate_lib.py
@@ -15,7 +15,6 @@
 """Generate tensorflow.org style API Reference docs for a Python module."""
 
 import collections
-import inspect
 import os
 import pathlib
 import shutil
@@ -24,12 +23,11 @@
 from typing import Any, Dict, List, Optional, Sequence, Tuple, Type, Union
 
 from tensorflow_docs.api_generator import config
-from tensorflow_docs.api_generator import doc_controls
 from tensorflow_docs.api_generator import doc_generator_visitor
 from tensorflow_docs.api_generator import parser
 from tensorflow_docs.api_generator import public_api
 from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib
-from tensorflow_docs.api_generator import signature
+from tensorflow_docs.api_generator import toc as toc_lib
 from tensorflow_docs.api_generator import traverse
 
 from tensorflow_docs.api_generator.pretty_docs import docs_for_object
@@ -57,368 +55,11 @@ def dict_constructor(loader, node):
 yaml.add_constructor(_mapping_tag, dict_constructor)
 
 
-class TocNode(object):
-  """Represents a node in the TOC.
-
-  Attributes:
-    full_name: Name of the module.
-    short_name: The last path component.
-    py_object: Python object of the module.
-    path: Path to the module's page on tensorflow.org relative to
-      tensorflow.org.
-    experimental: Whether the module is experimental or not.
-    deprecated: Whether the module is deprecated or not.
-  """
-
-  def __init__(self, module: str, py_object: Any, path: str):
-    self._module = module
-    self._py_object = py_object
-    self._path = path
-
-  @property
-  def full_name(self):
-    return self._module
-
-  @property
-  def short_name(self):
-    return self.full_name.split('.')[-1]
-
-  @property
-  def py_object(self):
-    return self._py_object
-
-  @property
-  def path(self):
-    return self._path
-
-  @property
-  def experimental(self):
-    return 'experimental' in self.short_name
-
-  _DEPRECATED_STRING = 'THIS FUNCTION IS DEPRECATED'
-
-  @property
-  def deprecated(self):
-    """Checks if the module is deprecated or not.
-
-    Special case is `tf.contrib`. It doesn't have the _tf_decorator attribute
-    but that module should be marked as deprecated.
-
-    Each deprecated function has a `_tf_decorator.decorator_name` attribute.
-    Check the docstring of that function to confirm if the function was
-    indeed deprecated. If a different deprecation setting was used on the
-    function, then "THIS FUNCTION IS DEPRECATED" substring won't be inserted
-    into the docstring of that function by the decorator.
-
-    Returns:
-      True if depreacted else False.
-    """
-    if doc_controls.is_deprecated(self.py_object):
-      return True
-
-    if 'tf.contrib' in self.full_name:
-      return True
-
-    # Instead of only checking the docstring, checking for the decorator
-    # provides an additional level of certainty about the correctness of the
-    # the application of `status: deprecated`.
-    decorator_list = signature.extract_decorators(self.py_object)
-    if any('deprecat' in dec for dec in decorator_list):
-      return self._check_docstring()
-
-    return False
-
-  def _check_docstring(self):
-    # Only add the deprecated status if the function is deprecated. There are
-    # other settings that should be ignored like deprecate_args, etc.
-    docstring = self.py_object.__doc__
-    return docstring is not None and self._DEPRECATED_STRING in docstring
-
-
-class Module(TocNode):
-  """Represents a single module and its children and submodules.
-
-  Attributes:
-    full_name: Name of the module.
-    short_name: The last path component.
-    py_object: Python object of the module.
-    title: Title of the module in _toc.yaml
-    path: Path to the module's page on tensorflow.org relative to
-      tensorflow.org.
-    children: List of attributes on the module.
-    submodules: List of submodules in the module.
-    experimental: Whether the module is experimental or not.
-    deprecated: Whether the module is deprecated or not.
-  """
-
-  def __init__(self, module, py_object, path):
-    super(Module, self).__init__(module, py_object, path)
-
-    self._children = []
-    self._submodules = []
-
-  @property
-  def title(self):
-    if self.full_name.count('.') > 1:
-      title = self.full_name.split('.')[-1]
-    else:
-      title = self.full_name
-    return title
-
-  @property
-  def children(self):
-    return sorted(self._children, key=lambda x: x.full_name)
-
-  @property
-  def submodules(self):
-    return self._submodules
-
-  def add_children(self, children):
-    if not isinstance(children, list):
-      children = [children]
-
-    self._children.extend(children)
-
-  def add_submodule(self, sub_mod):
-    self._submodules.append(sub_mod)
-
-
-class ModuleChild(TocNode):
-  """Represents a child of a module.
-
-  Attributes:
-    full_name: Name of the child.
-    short_name: The last path component.
-    py_object: Python object of the child.
-    title: Title of the module in _toc.yaml
-    path: Path to the module's page on tensorflow.org relative to
-      tensorflow.org.
-    experimental: Whether the module child is experimental or not.
-    deprecated: Whether the module is deprecated or not.
-  """
-
-  def __init__(self, name, py_object, parent, path):
-    self._parent = parent
-    super(ModuleChild, self).__init__(name, py_object, path)
-
-  @property
-  def title(self):
-    return self.full_name[len(self._parent) + 1:]
-
-
-class GenerateToc(object):
-  """Generates a data structure that defines the structure of _toc.yaml."""
-
-  def __init__(self, modules):
-    self._modules = modules
-
-  def _create_graph(self):
-    """Creates a graph to allow a dfs traversal on it to generate the toc.
-
-    Each graph key contains a module and its value is an object of `Module`
-    class. That module object contains a list of submodules.
-
-    Example low-level structure of the graph is as follows:
-
-    {
-      'module1': [submodule1, submodule2],
-      'submodule1': [sub1-submodule1],
-      'sub1-submodule1': [],
-      'submodule2': [],
-      'module2': [],
-      'module3': [submodule4],
-      'submodule4': [sub1-submodule4],
-      'sub1-submodule4': [sub1-sub1-submodule4],
-      'sub1-sub1-submodule4': []
-    }
-
-    Returns:
-      A tuple of (graph, base_modules). Base modules is returned because while
-      creating a nested list of dictionaries, the top level should only contain
-      the base modules.
-    """
-
-    # Sort the modules in case-insensitive alphabetical order.
-    sorted_modules = sorted(self._modules.keys(), key=lambda a: a.lower())
-    toc_base_modules = []
-
-    toc_graph = {}
-    min_dots = min(module.count('.') for module in sorted_modules)
-    min_docs = max(min_dots, 1)
-
-    for module in sorted_modules:
-      mod = self._modules[module]
-
-      # Add the module to the graph.
-      toc_graph[module] = mod
-
-      # If the module's name contains more than one dot, it is not a base level
-      # module. Hence, add it to its parents submodules list.
-      if module.count('.') > min_docs:
-        # For example, if module is `tf.keras.applications.densenet` then its
-        # parent is `tf.keras.applications`.
-        parent_module = '.'.join(module.split('.')[:-1])
-        parent_mod_obj = toc_graph.get(parent_module, None)
-        if parent_mod_obj is not None:
-          parent_mod_obj.add_submodule(mod)
-      else:
-        toc_base_modules.append(module)
-
-    return toc_graph, toc_base_modules
-
-  def _generate_children(self, mod, is_parent_deprecated):
-    """Creates a list of dictionaries containing child's title and path.
-
-    For example: The dictionary created will look this this in _toc.yaml.
-
-    ```
-    children_list = [{'title': 'Overview', 'path': '/tf/app'},
-                     {'title': 'run', 'path': '/tf/app/run'}]
-    ```
-
-    The above list will get converted to the following yaml syntax.
-
-    ```
-    - title: Overview
-      path: /tf/app
-    - title: run
-      path: /tf/app/run
-    ```
-
-    Args:
-      mod: A module object.
-      is_parent_deprecated: Bool, Whether the parent is deprecated or not.
-
-    Returns:
-      A list of dictionaries containing child's title and path.
-    """
-
-    children_list = []
-    children_list.append(
-        collections.OrderedDict([('title', 'Overview'), ('path', mod.path)]))
-
-    for child in mod.children:
-      child_yaml_content = [('title', child.title), ('path', child.path)]
-
-      # Set `status: deprecated` only if the parent's status is not
-      # deprecated.
-      if child.deprecated and not is_parent_deprecated:
-        child_yaml_content.insert(1, ('status', 'deprecated'))
-      elif child.experimental:
-        child_yaml_content.insert(1, ('status', 'experimental'))
-
-      children_list.append(collections.OrderedDict(child_yaml_content))
-
-    return children_list
-
-  def _dfs(self, mod, visited, is_parent_deprecated):
-    """Does a dfs traversal on the graph generated.
-
-    This creates a nested dictionary structure which is then dumped as .yaml
-    file. Each submodule's dictionary of title and path is nested under its
-    parent module.
-
-    For example, `tf.keras.app.net` will be nested under `tf.keras.app` which
-    will be nested under `tf.keras`. Here's how the nested dictionaries will
-    look when its dumped as .yaml.
-
-    ```
-    - title: tf.keras
-      section:
-      - title: Overview
-        path: /tf/keras
-      - title: app
-        section:
-        - title: Overview
-          path: /tf/keras/app
-        - title: net
-          section:
-          - title: Overview
-            path: /tf/keras/app/net
-    ```
-
-    The above nested structure is what the dfs traversal will create in form
-    of lists of dictionaries.
-
-    Args:
-      mod: A module object.
-      visited: A dictionary of modules visited by the dfs traversal.
-      is_parent_deprecated: Bool, Whether any parent is deprecated or not.
-
-    Returns:
-      A dictionary containing the nested data structure.
-    """
-
-    visited[mod.full_name] = True
-
-    # parent_exp is set to the current module because the current module is
-    # the parent for its children.
-    children_list = self._generate_children(
-        mod, is_parent_deprecated or mod.deprecated)
-
-    # generate for submodules within the submodule.
-    for submod in mod.submodules:
-      if not visited[submod.full_name]:
-        sub_mod_dict = self._dfs(submod, visited, is_parent_deprecated or
-                                 mod.deprecated)
-        children_list.append(sub_mod_dict)
-
-    # If the parent module is not experimental, then add the experimental
-    # status to the submodule.
-    submod_yaml_content = [('title', mod.title), ('section', children_list)]
-
-    # If the parent module is not deprecated, then add the deprecated
-    # status to the submodule. If the parent is deprecated, then setting its
-    # status to deprecated in _toc.yaml propagates to all its children and
-    # submodules.
-    if mod.deprecated and not is_parent_deprecated:
-      submod_yaml_content.insert(1, ('status', 'deprecated'))
-    elif mod.experimental:
-      submod_yaml_content.insert(1, ('status', 'experimental'))
-
-    return collections.OrderedDict(submod_yaml_content)
-
-  def generate(self) -> Dict[str, Any]:
-    """Generates the final toc.
-
-    Returns:
-      A list of dictionaries which will be dumped into .yaml file.
-    """
-
-    toc = []
-    toc_graph, toc_base_modules = self._create_graph()
-    visited = {node: False for node in toc_graph.keys()}
-
-    # Sort in alphabetical case-insensitive order.
-    toc_base_modules = sorted(toc_base_modules, key=lambda a: a.lower())
-    for module in toc_base_modules:
-      module_obj = toc_graph[module]
-      # Generate children of the base module.
-      section = self._generate_children(
-          module_obj, is_parent_deprecated=module_obj.deprecated)
-
-      # DFS traversal on the submodules.
-      for sub_mod in module_obj.submodules:
-        sub_mod_list = self._dfs(
-            sub_mod, visited, is_parent_deprecated=module_obj.deprecated)
-        section.append(sub_mod_list)
-
-      module_yaml_content = [('title', module_obj.title), ('section', section)]
-      if module_obj.deprecated:
-        module_yaml_content.insert(1, ('status', 'deprecated'))
-      elif module_obj.experimental:
-        module_yaml_content.insert(1, ('status', 'experimental'))
-
-      toc.append(collections.OrderedDict(module_yaml_content))
-
-    return {'toc': toc}
-
-
 def write_docs(
     *,
     output_dir: Union[str, pathlib.Path],
     parser_config: config.ParserConfig,
-    yaml_toc: bool,
+    yaml_toc: Union[bool, Type[toc_lib.TocBuilder]],
     root_module_name: str,
     root_title: str = 'TensorFlow',
     search_hints: bool = True,
@@ -470,11 +111,6 @@ def write_docs(
                      f"    output_dir='{output_dir}'")
   output_dir.mkdir(parents=True, exist_ok=True)
 
-  # These dictionaries are used for table-of-contents generation below
-  # They will contain, after the for-loop below::
-  #  - module name(string):classes and functions the module contains(list)
-  module_children = {}
-
   # Collect redirects for an api _redirects.yaml file.
   redirects = []
 
@@ -483,43 +119,13 @@ def write_docs(
 
   # Parse and write Markdown pages, resolving cross-links (`tf.symbol`).
   num_docs_output = 0
-  for full_name in sorted(parser_config.index.keys(), key=lambda k: k.lower()):
-    py_object = parser_config.index[full_name]
-
-    if full_name in parser_config.duplicate_of:
-      continue
+  for node in parser_config.api_tree.iter_nodes():
+    full_name = node.full_name
+    py_object = node.py_object
 
-    # Methods constants are only documented only as part of their parent's page.
-    if parser_config.reference_resolver.is_fragment(full_name):
+    if node.output_type() is node.OutputType.FRAGMENT:
       continue
 
-    # Remove the extension from the path.
-    docpath, _ = os.path.splitext(parser.documentation_path(full_name))
-
-    # For a module, remember the module for the table-of-contents
-    if inspect.ismodule(py_object):
-      if full_name in parser_config.tree:
-        mod_obj = Module(
-            module=full_name,
-            py_object=py_object,
-            path=str(site_path / docpath))
-        module_children[full_name] = mod_obj
-    # For something else that's documented,
-    # figure out what module it lives in
-    else:
-      subname = str(full_name)
-      while True:
-        subname = subname[:subname.rindex('.')]
-        if inspect.ismodule(parser_config.index[subname]):
-          module_name = parser_config.duplicate_of.get(subname, subname)
-          child_mod = ModuleChild(
-              name=full_name,
-              py_object=py_object,
-              parent=module_name,
-              path=str(site_path / docpath))
-          module_children[module_name].add_children(child_mod)
-          break
-
     # Generate docs for `py_object`, resolving references.
     try:
       page_info = docs_for_object.docs_for_object(
@@ -572,12 +178,12 @@ def write_docs(
                      '`base_dir`.')
 
   if yaml_toc:
-    toc_gen = GenerateToc(module_children)
-    toc_dict = toc_gen.generate()
+    if isinstance(yaml_toc, bool):
+      yaml_toc = toc_lib.FlatModulesTocBuilder
+    toc = yaml_toc(site_path).build(parser_config.api_tree)
 
-    leftnav_toc = output_dir / root_module_name / '_toc.yaml'
-    with open(leftnav_toc, 'w') as toc_file:
-      yaml.dump(toc_dict, toc_file, default_flow_style=False)
+    toc_path = output_dir / root_module_name / '_toc.yaml'
+    toc.write(toc_path)
 
   if redirects and gen_redirects:
     redirects_dict = {
@@ -677,7 +283,7 @@ def __init__(
       .DocGeneratorVisitor,
       api_cache: bool = True,
       callbacks: Optional[List[public_api.ApiFilter]] = None,
-      yaml_toc: bool = True,
+      yaml_toc: Union[bool, Type[toc_lib.TocBuilder]] = True,
       gen_redirects: bool = True,
       gen_report: bool = True,
       extra_docs: Optional[Dict[int, str]] = None,
@@ -777,6 +383,7 @@ def make_parser_config(self,
         index=visitor.index,
         reverse_index=visitor.reverse_index,
         path_tree=visitor.path_tree,
+        api_tree=visitor.api_tree,
         base_dir=self._base_dir,
         code_url_prefix=self._code_url_prefix)
 
diff --git a/tools/tensorflow_docs/api_generator/reference_resolver.py b/tools/tensorflow_docs/api_generator/reference_resolver.py
index 91f898b91bf..e7cb889f16c 100644
--- a/tools/tensorflow_docs/api_generator/reference_resolver.py
+++ b/tools/tensorflow_docs/api_generator/reference_resolver.py
@@ -383,6 +383,9 @@ def _one_ref(self, match, full_name=None):
                                'tf.contrib' not in full_name):
       string = self._partial_symbols_dict.get(string, string)
 
+    if not string:
+      return match.group(0)
+
     try:
       if string.startswith('tensorflow::'):
         # C++ symbol
diff --git a/tools/tensorflow_docs/api_generator/toc.py b/tools/tensorflow_docs/api_generator/toc.py
index 873e2cb804d..65572bfabba 100644
--- a/tools/tensorflow_docs/api_generator/toc.py
+++ b/tools/tensorflow_docs/api_generator/toc.py
@@ -181,6 +181,7 @@ def build(self, api_tree: doc_generator_visitor.ApiTree) -> Toc:
 
   def _entries_from_api_node(
       self, api_node: doc_generator_visitor.ApiTreeNode) -> List[Entry]:
+
     """Converts an ApiTreeNode to a list of toc entries."""
     obj_type = api_node.obj_type
 
@@ -205,7 +206,8 @@ def _make_link(self,
         title=title, path=str(docpath), status=self._make_status(api_node))
 
   def _make_section(self,
-                    api_node: doc_generator_visitor.ApiTreeNode) -> Section:
+                    api_node: doc_generator_visitor.ApiTreeNode,
+                    title: Optional[str] = None) -> Section:
     """Create a `toc.Section` from a module's ApiTreeNode."""
     overview = self._make_overview(api_node)
     entries = []
@@ -215,7 +217,8 @@ def _make_section(self,
     entries = [overview] + entries
 
     status = self._make_status(api_node)
-    return Section(title=api_node.short_name, section=entries, status=status)
+    return Section(
+        title=title or api_node.short_name, section=entries, status=status)
 
   def _make_overview(self, api_node: doc_generator_visitor.ApiTreeNode):
     docpath = pathlib.Path(self.site_path, *api_node.path)
@@ -277,3 +280,64 @@ def _is_deprecated(self, api_node: doc_generator_visitor.ApiTreeNode):
       return 'THIS FUNCTION IS DEPRECATED' in docstring
 
     return False
+
+
+class FlatModulesTocBuilder(TocBuilder):
+  """Builds a toc where the top level submodules are peers (not children).
+
+  The base TocBuilder does this:
+
+  ```
+  module:
+    thing1
+    sub1:
+      thing2
+    sub2:
+      thing3
+  ```
+
+  This one outputs:
+
+  ```
+  module:
+    thing1
+  module.sub1:
+    thing2
+  module.sub2:
+    thing3
+  ```
+  """
+
+  def build(self, api_tree: doc_generator_visitor.ApiTree) -> Toc:
+    entries = []
+    for module_node in api_tree.root.children.values():
+      assert module_node.obj_type is obj_type_lib.ObjType.MODULE
+      entries.extend(self._flat_module_entries(module_node))
+
+    return Toc(toc=entries)
+
+  def _flat_module_entries(self,
+                           api_node: doc_generator_visitor.ApiTreeNode,
+                           title: Optional[str] = None) -> List[Section]:
+    """For top-level modules, place the submodules as peers."""
+    title = title or api_node.short_name
+
+    overview = self._make_link(api_node, title='Overview')
+    entries = []
+    submodule_sections = []
+    for name, child_node in api_node.children.items():
+      if child_node.obj_type is obj_type_lib.ObjType.MODULE:
+        subtitle = f'{title}.{name}'
+        submodule_sections.append(
+            self._make_section(child_node, title=subtitle))
+      else:
+        entries.extend(self._entries_from_api_node(child_node))
+
+    entries = sorted(entries, key=self._section_order_key)
+    entries.insert(0, overview)
+
+    submodule_sections = sorted(submodule_sections, key=self._section_order_key)
+
+    status = self._make_status(api_node)
+    module_section = Section(title=title, section=entries, status=status)
+    return [module_section] + submodule_sections
diff --git a/tools/tensorflow_docs/api_generator/toc_test.py b/tools/tensorflow_docs/api_generator/toc_test.py
index 77189d80565..beb7a452ec0 100644
--- a/tools/tensorflow_docs/api_generator/toc_test.py
+++ b/tools/tensorflow_docs/api_generator/toc_test.py
@@ -141,6 +141,35 @@ def test_toc_builder(self):
 
     self.assertEqual(expected, stream.getvalue())
 
+  def test_flat_modules_builder(self):
+    api_tree = self._make_tree()
+    builder = toc_lib.FlatModulesTocBuilder('/path/in/site')
+    toc = builder.build(api_tree)
+    stream = io.StringIO()
+    toc.write(stream)
+
+    expected = textwrap.dedent("""\
+        toc:
+        - title: module
+          section:
+          - title: Overview
+            path: /path/in/site/module
+          - title: Class
+            path: /path/in/site/module/Class
+          - title: Class.NestedClass
+            path: /path/in/site/module/Class/NestedClass
+          - title: func1
+            path: /path/in/site/module/func1
+        - title: module.submodule
+          section:
+          - title: Overview
+            path: /path/in/site/module/submodule
+          - title: func2
+            path: /path/in/site/module/submodule/func2
+        """)
+
+    self.assertEqual(expected, stream.getvalue())
+
 
 if __name__ == '__main__':
   absltest.main()

From 7a471e1881de2767d3374797710414ff787a0328 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 6 Apr 2022 17:02:32 -0700
Subject: [PATCH 055/872] Add logging to the api traversal filtering.

I often find myself stepping through here with a debugger.
Just write some logs instead.

This will help any investigation into why a symbol was or was not included in the resulting API.

PiperOrigin-RevId: 439970734
---
 tools/tensorflow_docs/api_generator/traverse.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/tools/tensorflow_docs/api_generator/traverse.py b/tools/tensorflow_docs/api_generator/traverse.py
index 7e3049b2707..8dabdb4d63f 100644
--- a/tools/tensorflow_docs/api_generator/traverse.py
+++ b/tools/tensorflow_docs/api_generator/traverse.py
@@ -16,8 +16,12 @@
 import inspect
 import sys
 
+import logging
 from google.protobuf.message import Message as ProtoMessage
 
+# To see the logs pass: --logger_levels=tensorflow_docs:DEBUG --alsologtostderr
+_LOGGER = logging.getLogger(__name__)
+
 __all__ = ['traverse']
 
 
@@ -162,9 +166,18 @@ def _traverse_internal(root, visitors, stack, path):
     filtered_children.append((name, child))
   children = filtered_children
 
+  _LOGGER.debug('path: %s', path)
+  _LOGGER.debug('children: %s', [n for n, c in children])
   # Apply all callbacks, allowing each to filter the children
   for visitor in visitors:
-    children = visitor(path, root, list(children))
+    old_names = [n for n, c in children]
+    children = visitor(path, root, children)
+    children = list(children)
+    new_names = [n for n, c in children]
+
+    if old_names != new_names:
+      _LOGGER.debug('filter: %s', visitor)
+      _LOGGER.debug('children: %s', new_names)
 
   for name, child in children:
     # Break cycles

From 4ab7c5e786585a10ef70a769e6fc170da2141559 Mon Sep 17 00:00:00 2001
From: James Mullenbach 
Date: Thu, 7 Apr 2022 08:35:54 -0700
Subject: [PATCH 056/872] Update ParameterServerStrategy tutorial: add details
 on SidecarEvaluator API, remove DatasetCreator as a requirement, update known
 limitations, update links, linting, light copy-editing.

PiperOrigin-RevId: 440114337
---
 .../parameter_server_training.ipynb           | 247 +++++++++++-------
 1 file changed, 150 insertions(+), 97 deletions(-)

diff --git a/site/en/tutorials/distribute/parameter_server_training.ipynb b/site/en/tutorials/distribute/parameter_server_training.ipynb
index 0edbd218744..88c21c58601 100644
--- a/site/en/tutorials/distribute/parameter_server_training.ipynb
+++ b/site/en/tutorials/distribute/parameter_server_training.ipynb
@@ -74,7 +74,7 @@
         "\n",
         "A parameter server training cluster consists of _workers_ and _parameter servers_. Variables are created on parameter servers and they are read and updated by workers in each step. By default, workers read and update these variables independently without synchronizing with each other. This is why sometimes parameter server-style training is called _asynchronous training_.\n",
         "\n",
-        "In TensorFlow 2, parameter server training is powered by the `tf.distribute.experimental.ParameterServerStrategy` class, which distributes the training steps to a cluster that scales up to thousands of workers (accompanied by parameter servers)."
+        "In TensorFlow 2, parameter server training is powered by the `tf.distribute.ParameterServerStrategy` class, which distributes the training steps to a cluster that scales up to thousands of workers (accompanied by parameter servers)."
       ]
     },
     {
@@ -87,9 +87,9 @@
         "\n",
         "There are two main supported training methods:\n",
         "\n",
-        "- The Keras `Model.fit` API, which is recommended when you prefer a high-level abstraction and handling of training.\n",
-        "- A custom training loop (you can refer to [Custom training](https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough#train_the_model), [Writing a training loop from scratch\n",
-        "](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) and [Custom training loop with Keras and MultiWorkerMirroredStrategy](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_ctl) for more details.) Custom loop training is recommended when you prefer to define the details of their training loop."
+        "- The Keras `Model.fit` API: if you prefer a high-level abstraction and handling of training. This is generally recommended if you are training a `tf.keras.Model`.\n",
+        "- A custom training loop: if you prefer to define the details of your training loop (you can refer to guides on [Custom training](../customization/custom_training_walkthrough.ipynb), [Writing a training loop from scratch\n",
+        "](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) and [Custom training loop with Keras and MultiWorkerMirroredStrategy](multi_worker_with_ctl.ipynb) for more details)."
       ]
     },
     {
@@ -100,12 +100,12 @@
       "source": [
         "### A cluster with jobs and tasks\n",
         "\n",
-        "Regardless of the API of choice (`Model.fit` or a custom training loop), distributed training in TensorFlow 2 involves: a `'cluster'` with several `'jobs'`, and each of the jobs may have one or more `'tasks'`.\n",
+        "Regardless of the API of choice (`Model.fit` or a custom training loop), distributed training in TensorFlow 2 involves a `'cluster'` with several `'jobs'`, and each of the jobs may have one or more `'tasks'`.\n",
         "\n",
         "When using parameter server training, it is recommended to have:\n",
         "\n",
         "- One _coordinator_ job (which has the job name `chief`)\n",
-        "- Multiple _worker_ jobs (job name `worker`); and\n",
+        "- Multiple _worker_ jobs (job name `worker`)\n",
         "- Multiple _parameter server_ jobs (job name `ps`)\n",
         "\n",
         "The _coordinator_ creates resources, dispatches training tasks, writes checkpoints, and deals with task failures. The _workers_ and _parameter servers_ run `tf.distribute.Server` instances that listen for requests from the coordinator."
@@ -117,10 +117,9 @@
         "id": "oLV1FbpLtqtB"
       },
       "source": [
-        "### Parameter server training with `Model.fit` API\n",
+        "### Parameter server training with the `Model.fit` API\n",
         "\n",
-        "Parameter server training with the `Model.fit` API requires the coordinator to use a `tf.distribute.experimental.ParameterServerStrategy` object, and a `tf.keras.utils.experimental.DatasetCreator` as the input. Similar to `Model.fit` usage with no strategy, or with other strategies, the workflow involves creating and compiling the model, preparing the callbacks, followed by\n",
-        "a `Model.fit` call."
+        "Parameter server training with the `Model.fit` API requires the coordinator to use a `tf.distribute.ParameterServerStrategy` object. Similar to `Model.fit` usage with no strategy, or with other strategies, the workflow involves creating and compiling the model, preparing the callbacks, and calling `Model.fit`."
       ]
     },
     {
@@ -131,12 +130,12 @@
       "source": [
         "### Parameter server training with a custom training loop\n",
         "\n",
-        "With custom training loops, the `tf.distribute.experimental.coordinator.ClusterCoordinator` class is the key component used for the coordinator.\n",
+        "With custom training loops, the `tf.distribute.coordinator.ClusterCoordinator` class is the key component used for the coordinator.\n",
         "\n",
         "- The `ClusterCoordinator` class needs to work in conjunction with a `tf.distribute.Strategy` object.\n",
-        "- This `tf.distribute.Strategy` object is needed to provide the information of the cluster and is used to define a training step, as demonstrated in [Custom training with tf.distribute.Strategy](https://www.tensorflow.org/tutorials/distribute/custom_training#training_loop).\n",
+        "- This `tf.distribute.Strategy` object is needed to provide the information of the cluster and is used to define a training step, as demonstrated in [Custom training with tf.distribute.Strategy](custom_training.ipynb).\n",
         "- The `ClusterCoordinator` object then dispatches the execution of these training steps to remote workers.\n",
-        "- For parameter server training, the `ClusterCoordinator` needs to work with a `tf.distribute.experimental.ParameterServerStrategy`.\n",
+        "- For parameter server training, the `ClusterCoordinator` needs to work with a `tf.distribute.ParameterServerStrategy`.\n",
         "\n",
         "The most important API provided by the `ClusterCoordinator` object is `schedule`:\n",
         "\n",
@@ -144,7 +143,7 @@
         "- The queued functions will be dispatched to remote workers in background threads and their `RemoteValue`s will be filled asynchronously.\n",
         "- Since `schedule` doesn’t require worker assignment, the `tf.function` passed in can be executed on any available worker.\n",
         "- If the worker it is executed on becomes unavailable before its completion, the function will be retried on another available worker.\n",
-        "- Because of this fact and the fact that function execution is not atomic, a function may be executed more than once.\n",
+        "- Because of this fact and the fact that function execution is not atomic, a single function call may be executed more than once.\n",
         "\n",
         "In addition to dispatching remote functions, the `ClusterCoordinator` also helps\n",
         "to create datasets on all the workers and rebuild these datasets when a worker recovers from failure."
@@ -196,9 +195,9 @@
       "source": [
         "## Cluster setup\n",
         "\n",
-        "As mentioned above, a parameter server training cluster requires a coordinator task that runs your training program, one or several workers and parameter server tasks that run TensorFlow servers—`tf.distribute.Server`—and possibly an additional evaluation task that runs side-car evaluation (see the side-car evaluation section below). The requirements to set them up are:\n",
+        "As mentioned above, a parameter server training cluster requires a coordinator task that runs your training program, one or several workers and parameter server tasks that run TensorFlow servers—`tf.distribute.Server`—and possibly an additional evaluation task that runs sidecar evaluation (refer to the [sidecar evaluation section](#sidecar_evaluation) below). The requirements to set them up are:\n",
         "\n",
-        "- The coordinator task needs to know the addresses and ports of all other TensorFlow servers except the evaluator.\n",
+        "- The coordinator task needs to know the addresses and ports of all other TensorFlow servers, except the evaluator.\n",
         "- The workers and parameter servers need to know which port they need to listen to. For the sake of simplicity, you can usually pass in the complete cluster information when creating TensorFlow servers on these tasks.\n",
         "- The evaluator task doesn’t have to know the setup of the training cluster. If it does, it should not attempt to connect to the training cluster.\n",
         "- Workers and parameter servers should have task types as `\"worker\"` and `\"ps\"`, respectively. The coordinator should use `\"chief\"` as the task type for legacy reasons.\n",
@@ -214,7 +213,7 @@
       "source": [
         "### In-process cluster\n",
         "\n",
-        "You will start by creating several TensorFlow servers in advance and connect to them later. Note that this is only for the purpose of this tutorial's demonstration, and in real training the servers will be started on `\"worker\"` and `\"ps\"` machines."
+        "You will start by creating several TensorFlow servers in advance and you will connect to them later. Note that this is only for the purpose of this tutorial's demonstration, and in real training the servers will be started on `\"worker\"` and `\"ps\"` machines."
       ]
     },
     {
@@ -276,9 +275,9 @@
         "id": "pX_91OByt0J2"
       },
       "source": [
-        "The in-process cluster setup is frequently used in unit testing, such as [here](https://github.com/tensorflow/tensorflow/blob/7621d31921c2ed979f212da066631ddfda37adf5/tensorflow/python/distribute/coordinator/cluster_coordinator_test.py#L437).\n",
+        "The in-process cluster setup is frequently used in unit testing, such as [here](https://github.com/tensorflow/tensorflow/blob/eb4c40fc91da260199fa2aed6fe67d36ad49fafd/tensorflow/python/distribute/coordinator/cluster_coordinator_test.py#L447).\n",
         "\n",
-        "Another option for local testing is to launch processes on the local machine—check out [Multi-worker training with Keras](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras) for an example of this approach."
+        "Another option for local testing is to launch processes on the local machine—check out [Multi-worker training with Keras](multi_worker_with_keras.ipynb) for an example of this approach."
       ]
     },
     {
@@ -289,7 +288,7 @@
       "source": [
         "## Instantiate a ParameterServerStrategy\n",
         "\n",
-        "Before you dive into the training code, let's instantiate a `ParameterServerStrategy` object. Note that this is needed regardless of whether you are proceeding with `Model.fit` or a custom training loop. The `variable_partitioner` argument will be explained in the [Variable sharding section](#variable-sharding)."
+        "Before you dive into the training code, let's instantiate a `tf.distribute.ParameterServerStrategy` object. Note that this is needed regardless of whether you are proceeding with `Model.fit` or a custom training loop. The `variable_partitioner` argument will be explained in the [Variable sharding section](#variable_sharding)."
       ]
     },
     {
@@ -305,7 +304,7 @@
         "        min_shard_bytes=(256 << 10),\n",
         "        max_shards=NUM_PS))\n",
         "\n",
-        "strategy = tf.distribute.experimental.ParameterServerStrategy(\n",
+        "strategy = tf.distribute.ParameterServerStrategy(\n",
         "    cluster_resolver,\n",
         "    variable_partitioner=variable_partitioner)"
       ]
@@ -328,7 +327,8 @@
         "### Variable sharding\n",
         "\n",
         "Variable sharding refers to splitting a variable into multiple smaller\n",
-        "variables, which are called _shards_. Variable sharding may be useful to distribute the network load when accessing these shards. It is also useful to distribute computation and storage of a normal variable across multiple parameter servers.\n",
+        "variables, which are called _shards_. Variable sharding may be useful to distribute the network load when accessing these shards. It is also useful to distribute computation and storage of a normal variable across multiple parameter servers, for example, when using very large embeddings\n",
+        "that may not fit in a single machine's memory.\n",
         "\n",
         "To enable variable sharding, you can pass in a `variable_partitioner` when\n",
         "constructing a `ParameterServerStrategy` object. The `variable_partitioner` will\n",
@@ -337,7 +337,7 @@
         "`variable_partitioner`s are provided such as\n",
         "`tf.distribute.experimental.partitioners.MinSizePartitioner`. It is recommended to use size-based partitioners like\n",
         "`tf.distribute.experimental.partitioners.MinSizePartitioner` to avoid\n",
-        "partitioning small variables, which could have negative impact on model training\n",
+        "partitioning small variables, which could have a negative impact on model training\n",
         "speed."
       ]
     },
@@ -347,8 +347,8 @@
         "id": "1--SxlxtsOb7"
       },
       "source": [
-        "When a `variable_partitioner` is passed in and if you create a variable directly\n",
-        "under `Strategy.scope`, it will become a container type with a `variables`\n",
+        "When a `variable_partitioner` is passed in, and you create a variable directly\n",
+        "under `Strategy.scope`, the variable will become a container type with a `variables`\n",
         "property, which provides access to the list of shards. In most cases, this\n",
         "container will be automatically converted to a Tensor by concatenating all the\n",
         "shards. As a result, it can be used as a normal variable. On the other hand,\n",
@@ -356,7 +356,7 @@
         "implementation for this container type and in these methods automatic\n",
         "concatenation will be avoided.\n",
         "\n",
-        "Refer to the API docs of `tf.distribute.experimental.ParameterServerStrategy` for more details."
+        "Refer to the API docs of `tf.distribute.ParameterServerStrategy` for more details."
       ]
     },
     {
@@ -368,7 +368,7 @@
         "## Training with `Model.fit`\n",
         "\n",
         "\n",
-        "Keras provides an easy-to-use training API via `Model.fit` that handles the training loop under the hood, with the flexibility of overridable `train_step`, and callbacks, which provide functionalities such as checkpoint saving or summary saving for TensorBoard. With `Model.fit`, the same training code can be used for other strategies with a simple swap of the strategy object."
+        "Keras provides an easy-to-use training API via `Model.fit` that handles the training loop under the hood, with the flexibility of an overridable `train_step`, and callbacks which provide functionalities such as checkpoint saving or summary saving for TensorBoard. With `Model.fit`, the same training code can be used with other strategies with a simple swap of the strategy object."
       ]
     },
     {
@@ -379,12 +379,14 @@
       "source": [
         "### Input data\n",
         "\n",
-        "`Model.fit` with parameter server training requires that the input data be\n",
-        "provided in a callable that takes a single argument of type `tf.distribute.InputContext`, and returns a `tf.data.Dataset`. Then, create a `tf.keras.utils.experimental.DatasetCreator` object that takes such `callable`, and an optional `tf.distribute.InputOptions` object via `input_options` argument.\n",
+        "Keras `Model.fit` with `tf.distribute.ParameterServerStrategy` can take input data in the form of a `tf.data.Dataset`, `tf.distribute.DistributedDataset`, or a `tf.keras.utils.experimental.DatasetCreator`, with `Dataset` being the recommended option for ease of use. If you encounter memory issues using `Dataset`, however, you may need to use `DatasetCreator` with a callable `dataset_fn` argument (refer to the `tf.keras.utils.experimental.DatasetCreator` API documentation for details).\n",
         "\n",
-        "Note that it is recommended to shuffle and repeat the data with parameter server training, and specify `steps_per_epoch` in `fit` call so the library knows the epoch boundaries.\n",
+        "If you transform your dataset into a `tf.data.Dataset`, you should use `Dataset.shuffle` and `Dataset.repeat`, as demonstrated in the code example below. \n",
         "\n",
-        "Refer to the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input#usage_2) tutorial for more information about the `InputContext` argument."
+        "- Keras `Model.fit` with parameter server training assumes that each worker receives the same dataset, except when it is shuffled differently. Therefore, by calling `Dataset.shuffle`, you ensure more even iterations over the data.\n",
+        "- Because workers do not synchronize, they may finish processing their datasets at different times. Therefore, the easiest way to define epochs with parameter server training is to use `Dataset.repeat`—which repeats a dataset indefinitely when called without an argument—and specify the `steps_per_epoch` argument in the `Model.fit` call. \n",
+        "\n",
+        "Refer to the \"Training workflows\" section of the [tf.data guide](../../guide/data.ipynb) for more details on `shuffle` and `repeat`."
       ]
     },
     {
@@ -395,23 +397,14 @@
       },
       "outputs": [],
       "source": [
-        "def dataset_fn(input_context):\n",
-        "  global_batch_size = 64\n",
-        "  batch_size = input_context.get_per_replica_batch_size(global_batch_size)\n",
-        "\n",
-        "  x = tf.random.uniform((10, 10))\n",
-        "  y = tf.random.uniform((10,))\n",
-        "\n",
-        "  dataset = tf.data.Dataset.from_tensor_slices((x, y)).shuffle(10).repeat()\n",
-        "  dataset = dataset.shard(\n",
-        "      input_context.num_input_pipelines,\n",
-        "      input_context.input_pipeline_id)\n",
-        "  dataset = dataset.batch(batch_size)\n",
-        "  dataset = dataset.prefetch(2)\n",
+        "global_batch_size = 64\n",
         "\n",
-        "  return dataset\n",
+        "x = tf.random.uniform((10, 10))\n",
+        "y = tf.random.uniform((10,))\n",
         "\n",
-        "dc = tf.keras.utils.experimental.DatasetCreator(dataset_fn)"
+        "dataset = tf.data.Dataset.from_tensor_slices((x, y)).shuffle(10).repeat()\n",
+        "dataset = dataset.batch(global_batch_size)\n",
+        "dataset = dataset.prefetch(2)"
       ]
     },
     {
@@ -420,7 +413,7 @@
         "id": "v_jhF70K7zON"
       },
       "source": [
-        "The code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n"
+        "If you instead create your dataset with `tf.keras.utils.experimental.DatasetCreator`, the code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n"
       ]
     },
     {
@@ -431,7 +424,7 @@
       "source": [
         "### Model construction and compiling\n",
         "\n",
-        "Now, you will create a `tf.keras.Model`—a trivial `tf.keras.models.Sequential` model for demonstration purposes—followed by a `Model.compile` call to incorporate components, such as an optimizer, metrics, or parameters such as `steps_per_execution`:"
+        "Now, you will create a `tf.keras.Model`—a trivial `tf.keras.models.Sequential` model for demonstration purposes—followed by a `Model.compile` call to incorporate components, such as an optimizer, metrics, and other parameters such as `steps_per_execution`:"
       ]
     },
     {
@@ -445,7 +438,7 @@
         "with strategy.scope():\n",
         "  model = tf.keras.models.Sequential([tf.keras.layers.Dense(10)])\n",
         "\n",
-        "  model.compile(tf.keras.optimizers.SGD(), loss='mse', steps_per_execution=10)"
+        "  model.compile(tf.keras.optimizers.SGD(), loss=\"mse\", steps_per_execution=10)"
       ]
     },
     {
@@ -458,13 +451,13 @@
         "\n",
         " \n",
         "\n",
-        "Before you call `model.fit` for the actual training, let's prepare the needed callbacks for common tasks, such as:\n",
+        "Before you call Keras `Model.fit` for the actual training, prepare any needed [callbacks](https://www.tensorflow.org/guide/keras/train_and_evaluate) for common tasks, such as:\n",
         "\n",
-        "- `ModelCheckpoint`: to save the model weights.\n",
-        "- `BackupAndRestore`: to make sure the training progress is automatically backed up, and recovered if the cluster experiences unavailability (such as abort or preemption); or\n",
-        "- `TensorBoard`: to save the progress reports into summary files, which can be visualized in the TensorBoard tool.\n",
+        "- `tf.keras.callbacks.ModelCheckpoint`: saves the model at a certain frequency, such as after every epoch.\n",
+        "- `tf.keras.callbacks.BackupAndRestore`: provides fault tolerance by backing up the model and current epoch number, if the cluster experiences unavailability (such as abort or preemption). You can then restore the training state upon a restart from a job failure, and continue training from the beginning of the interrupted epoch.\n",
+        "- `tf.keras.callbacks.TensorBoard`: periodically writes model logs in summary files that can be visualized in the TensorBoard tool.\n",
         "\n",
-        "Note: Due to performance consideration, custom callbacks cannot have batch level callbacks overridden when used with `ParameterServerStrategy`. Please modify your custom callbacks to make them epoch level calls, and adjust `steps_per_epoch` to a suitable value. In addition, `steps_per_epoch` is a required argument for `Model.fit` when used with `ParameterServerStrategy`."
+        "Note: Due to performance considerations, custom callbacks cannot have batch level callbacks overridden when used with `ParameterServerStrategy`. Please modify your custom callbacks to make them epoch level calls, and adjust `steps_per_epoch` to a suitable value. In addition, `steps_per_epoch` is a required argument for `Model.fit` when used with `ParameterServerStrategy`."
       ]
     },
     {
@@ -475,10 +468,10 @@
       },
       "outputs": [],
       "source": [
-        "working_dir = '/tmp/my_working_dir'\n",
-        "log_dir = os.path.join(working_dir, 'log')\n",
-        "ckpt_filepath = os.path.join(working_dir, 'ckpt')\n",
-        "backup_dir = os.path.join(working_dir, 'backup')\n",
+        "working_dir = \"/tmp/my_working_dir\"\n",
+        "log_dir = os.path.join(working_dir, \"log\")\n",
+        "ckpt_filepath = os.path.join(working_dir, \"ckpt\")\n",
+        "backup_dir = os.path.join(working_dir, \"backup\")\n",
         "\n",
         "callbacks = [\n",
         "    tf.keras.callbacks.TensorBoard(log_dir=log_dir),\n",
@@ -486,7 +479,7 @@
         "    tf.keras.callbacks.BackupAndRestore(backup_dir=backup_dir),\n",
         "]\n",
         "\n",
-        "model.fit(dc, epochs=5, steps_per_epoch=20, callbacks=callbacks)"
+        "model.fit(dataset, epochs=5, steps_per_epoch=20, callbacks=callbacks)"
       ]
     },
     {
@@ -497,7 +490,7 @@
       "source": [
         "### Direct usage with `ClusterCoordinator` (optional)\n",
         "\n",
-        "Even if you choose the `Model.fit` training path, you can optionally instantiate a `tf.distribute.experimental.coordinator.ClusterCoordinator` object to schedule other functions you would like to be executed on the workers. See the [Training with a custom training loop](#training_with_custom_training_loop) section for more details and examples."
+        "Even if you choose the `Model.fit` training path, you can optionally instantiate a `tf.distribute.coordinator.ClusterCoordinator` object to schedule other functions you would like to be executed on the workers. Refer to the [Training with a custom training loop](#training_with_custom_training_loop) section for more details and examples."
       ]
     },
     {
@@ -510,11 +503,11 @@
         "\n",
         " \n",
         "\n",
-        "Using custom training loops with `tf.distribute.Strategy` provides great flexibility to define training loops. With the `ParameterServerStrategy` defined above (as `strategy`), you will use a `tf.distribute.experimental.coordinator.ClusterCoordinator` to dispatch the execution of training steps to remote workers.\n",
+        "Using custom training loops with `tf.distribute.Strategy` provides great flexibility to define training loops. With the `ParameterServerStrategy` defined above (as `strategy`), you will use a `tf.distribute.coordinator.ClusterCoordinator` to dispatch the execution of training steps to remote workers.\n",
         "\n",
-        "Then, you will create a model, define a dataset and a step function, as you have done in the training loop with other `tf.distribute.Strategy`s. You can find more details in the [Custom training with tf.distribute.Strategy](https://www.tensorflow.org/tutorials/distribute/custom_training) tutorial.\n",
+        "Then, you will create a model, define a dataset, and define a step function, as you have done in the training loop with other `tf.distribute.Strategy`s. You can find more details in the [Custom training with tf.distribute.Strategy](custom_training.ipynb) tutorial.\n",
         "\n",
-        "To ensure efficient dataset prefetching, use the recommended distributed dataset creation APIs mentioned in the [Dispatch training steps to remote workers](https://www.tensorflow.org/tutorials/distribute/parameter_server_training#dispatch_training_steps_to_remote_workers) section below. Also, make sure to call `Strategy.run` inside `worker_fn` to take full advantage of GPUs allocated to workers. The rest of the steps are the same for training with or without GPUs.\n",
+        "To ensure efficient dataset prefetching, use the recommended distributed dataset creation APIs mentioned in the [Dispatch training steps to remote workers](#dispatch_training_steps_to_remote_workers) section below. Also, make sure to call `Strategy.run` inside `worker_fn` to take full advantage of GPUs allocated to workers. The rest of the steps are the same for training with or without GPUs.\n",
         "\n",
         "Let’s create these components in the following steps:\n"
       ]
@@ -529,11 +522,11 @@
         "\n",
         "First, write a function that creates a dataset.\n",
         "\n",
-        "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `Strategy.scope`** like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline.\n",
+        "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `Strategy.scope`**, like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline.\n",
         "\n",
         "If you don't follow the above procedure, creating the layers might create Tensorflow states which will be lifted out of the `tf.function` to the coordinator. Thus, accessing them on workers would incur repetitive RPC calls between coordinator and workers, and cause significant slowdown.\n",
         "\n",
-        "Placing the layers under `Strategy.scope` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Refer to _Data preprocessing_ in the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input) tutorial for more information on data preprocessing with distributed input."
+        "Placing the layers under `Strategy.scope` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Refer to _Data preprocessing_ in the [Distributed input](input.ipynb) tutorial for more information on data preprocessing with distributed input."
       ]
     },
     {
@@ -677,7 +670,7 @@
         "id": "iyuxiqCQU50m"
       },
       "source": [
-        "Let's confirm that the use of `FixedShardsPartitioner` split all variables into two shards and each shard was assigned to different parameter servers:"
+        "Let's confirm that the use of `FixedShardsPartitioner` split all variables into two shards and that each shard was assigned to a different parameter server:"
       ]
     },
     {
@@ -720,7 +713,7 @@
         "    with tf.GradientTape() as tape:\n",
         "      pred = model(batch_data, training=True)\n",
         "      per_example_loss = tf.keras.losses.BinaryCrossentropy(\n",
-        "              reduction=tf.keras.losses.Reduction.NONE)(labels, pred)\n",
+        "          reduction=tf.keras.losses.Reduction.NONE)(labels, pred)\n",
         "      loss = tf.nn.compute_average_loss(per_example_loss)\n",
         "      gradients = tape.gradient(loss, model.trainable_variables)\n",
         "\n",
@@ -753,7 +746,7 @@
         "### Dispatch training steps to remote workers\n",
         " \n",
         "\n",
-        "After all the computations are defined by `ParameterServerStrategy`, you will use the `tf.distribute.experimental.coordinator.ClusterCoordinator` class to create resources and distribute the training steps to remote workers.\n",
+        "After all the computations are defined by `ParameterServerStrategy`, you will use the `tf.distribute.coordinator.ClusterCoordinator` class to create resources and distribute the training steps to remote workers.\n",
         "\n",
         "Let’s first create a `ClusterCoordinator` object and pass in the strategy object:"
       ]
@@ -766,7 +759,7 @@
       },
       "outputs": [],
       "source": [
-        "coordinator = tf.distribute.experimental.coordinator.ClusterCoordinator(strategy)"
+        "coordinator = tf.distribute.coordinator.ClusterCoordinator(strategy)"
       ]
     },
     {
@@ -775,7 +768,7 @@
         "id": "-xRIgKxciOSe"
       },
       "source": [
-        "Then, create a per-worker dataset and an iterator. In the `per_worker_dataset_fn` below, wrapping the `dataset_fn` into `strategy.distribute_datasets_from_function` is recommended to allow efficient prefetching to GPUs seamlessly."
+        "Then, create a per-worker dataset and an iterator using the `ClusterCoordinator.create_per_worker_dataset` API, which replicates the dataset to all workers. In the `per_worker_dataset_fn` below, wrapping the `dataset_fn` into `strategy.distribute_datasets_from_function` is recommended to allow efficient prefetching to GPUs seamlessly."
       ]
     },
     {
@@ -814,15 +807,15 @@
       },
       "outputs": [],
       "source": [
-        "num_epoches = 4\n",
+        "num_epochs = 4\n",
         "steps_per_epoch = 5\n",
-        "for i in range(num_epoches):\n",
+        "for i in range(num_epochs):\n",
         "  accuracy.reset_states()\n",
         "  for _ in range(steps_per_epoch):\n",
         "    coordinator.schedule(step_fn, args=(per_worker_iterator,))\n",
         "  # Wait at epoch boundaries.\n",
         "  coordinator.join()\n",
-        "  print (\"Finished epoch %d, accuracy is %f.\" % (i, accuracy.result().numpy()))"
+        "  print(\"Finished epoch %d, accuracy is %f.\" % (i, accuracy.result().numpy()))"
       ]
     },
     {
@@ -843,7 +836,7 @@
       "outputs": [],
       "source": [
         "loss = coordinator.schedule(step_fn, args=(per_worker_iterator,))\n",
-        "print (\"Final loss is %f\" % loss.fetch())"
+        "print(\"Final loss is %f\" % loss.fetch())"
       ]
     },
     {
@@ -863,7 +856,7 @@
         "  # Do something like logging metrics or writing checkpoints.\n",
         "```\n",
         "\n",
-        "For the complete training and serving workflow for this particular example, please check out this [test](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/distribute/parameter_server_training_test.py).\n"
+        "For the complete training and serving workflow for this particular example, please check out this [test](https://github.com/keras-team/keras/blob/master/keras/integration_test/parameter_server_keras_preprocessing_test.py).\n"
       ]
     },
     {
@@ -876,9 +869,9 @@
         "\n",
         "The dataset in the above code is created using the `ClusterCoordinator.create_per_worker_dataset` API. It creates one dataset per worker and returns a container object. You can call the `iter` method on it to create a per-worker iterator. The per-worker iterator contains one iterator per worker and the corresponding slice of a worker will be substituted in the input argument of the function passed to the `ClusterCoordinator.schedule` method before the function is executed on a particular worker.\n",
         "\n",
-        "The `ClusterCoordinator.schedule` method assumes workers are equivalent and thus assumes the datasets on different workers are the same (except that they may be shuffled differently). Because of this, it is also recommended to repeat datasets, and schedule a finite number of steps instead of relying on the `OutOfRangeError` from a dataset.\n",
+        "The `ClusterCoordinator.schedule` method assumes workers are equivalent and thus assumes the datasets on different workers are the same (except that they may be shuffled differently). Because of this, it is also recommended to repeat datasets, and schedule a finite number of steps instead of relying on receiving an `OutOfRangeError` from a dataset.\n",
         "\n",
-        "Another important note is that `tf.data` datasets don’t support implicit serialization and deserialization across task boundaries. So it is important to create the whole dataset inside the function passed to `ClusterCoordinator.create_per_worker_dataset`."
+        "Another important note is that `tf.data` datasets don’t support implicit serialization and deserialization across task boundaries. So it is important to create the whole dataset inside the function passed to `ClusterCoordinator.create_per_worker_dataset`. The `create_per_worker_dataset` API can also directly take a `tf.data.Dataset` or `tf.distribute.DistributedDataset` as input."
       ]
     },
     {
@@ -889,7 +882,7 @@
       "source": [
         "## Evaluation\n",
         "\n",
-        "There is more than one way to define and run an evaluation loop in distributed training. Each has its own pros and cons as described below. The inline evaluation method is recommended if you don't have a preference."
+        "The two main approaches to performing evaluation with `tf.distribute.ParameterServerStrategy` training are inline evaluation and sidecar evaluation. Each has its own pros and cons as described below. The inline evaluation method is recommended if you don't have a preference."
       ]
     },
     {
@@ -905,7 +898,7 @@
         "There are several benefits of inline evaluation. For example:\n",
         "\n",
         "- It can support large evaluation models and evaluation datasets that a single task cannot hold.\n",
-        "- The evaluation results can be used to make decisions for training the next epoch.\n",
+        "- The evaluation results can be used to make decisions for training the next epoch, for example, whether to stop training early.\n",
         "\n",
         "There are two ways to implement inline evaluation: direct evaluation and distributed evaluation.\n",
         "\n",
@@ -921,7 +914,7 @@
       "outputs": [],
       "source": [
         "eval_dataset = tf.data.Dataset.from_tensor_slices(\n",
-        "      feature_and_label_gen(num_examples=16)).map(\n",
+        "    feature_and_label_gen(num_examples=16)).map(\n",
         "          lambda x: (\n",
         "              {\"features\": feature_preprocess_stage(x[\"features\"])},\n",
         "              label_preprocess_stage(x[\"label\"])\n",
@@ -934,7 +927,7 @@
         "  actual_pred = tf.cast(tf.greater(pred, 0.5), tf.int64)\n",
         "  eval_accuracy.update_state(labels, actual_pred)\n",
         "\n",
-        "print (\"Evaluation accuracy: %f\" % eval_accuracy.result())"
+        "print(\"Evaluation accuracy: %f\" % eval_accuracy.result())"
       ]
     },
     {
@@ -982,7 +975,7 @@
         "for _ in range(eval_steps_per_epoch):\n",
         "  coordinator.schedule(eval_step, args=(per_worker_eval_iterator,))\n",
         "coordinator.join()\n",
-        "print (\"Evaluation accuracy: %f\" % eval_accuracy.result())"
+        "print(\"Evaluation accuracy: %f\" % eval_accuracy.result())"
       ]
     },
     {
@@ -991,7 +984,7 @@
         "id": "cKrQktZX5z7a"
       },
       "source": [
-        "Note: Currently, the `schedule` and `join` methods of `tf.distribute.experimental.coordinator.ClusterCoordinator` don’t support visitation guarantee or exactly-once semantics. In other words, there is no guarantee that all evaluation examples in a dataset will be evaluated exactly once; some may not be visited and some may be evaluated multiple times. Visitation guarantee on evaluation dataset is being worked on."
+        "Note: The `schedule` and `join` methods of `tf.distribute.coordinator.ClusterCoordinator` don’t support visitation guarantees or exactly-once semantics. In other words, there is no guarantee that all evaluation examples in a dataset will be evaluated exactly once; some may not be visited and some may be evaluated multiple times. The `tf.data` service API can be used to provide exactly-once visitation for evaluation when using `ParameterServerStrategy` (refer to the _Dynamic Sharding_ section of the `tf.data.experimental.service` API documentation)."
       ]
     },
     {
@@ -1000,9 +993,69 @@
         "id": "H40X-9Gs3i7_"
       },
       "source": [
-        "### Side-car evaluation\n",
+        "### Sidecar evaluation\n",
+        "\n",
         "\n",
-        "Another method is called _side-car evaluation_ where you create a dedicated evaluator task that repeatedly reads checkpoints and runs evaluation on a latest checkpoint. It allows your training program to finish early if you don't need to change your training loop based on evaluation results. However, it requires an additional evaluator task and periodic checkpointing to trigger evaluation. Following is a possible side-car evaluation loop:\n",
+        "Another method for defining and running an evaluation loop in `tf.distribute.ParameterServerStrategy` training is called _sidecar evaluation_, in which you create a dedicated evaluator task that repeatedly reads checkpoints and runs evaluation on the latest checkpoint (refer to [this guide](../../guide/checkpoint.ipynb) for more details on checkpointing). The chief and worker tasks do not spend any time on evaluation, so for a fixed number of iterations the overall training time should be shorter than using other evaluation methods. However, it requires an additional evaluator task and periodic checkpointing to trigger evaluation."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "HonyjnXK9-ys"
+      },
+      "source": [
+        "To write an evaluation loop for sidecar evaluation, you have two\n",
+        "options: \n",
+        "\n",
+        "1. Use the `tf.keras.utils.SidecarEvaluator` API.\n",
+        "2. Create a custom evaluation loop. \n",
+        "\n",
+        "Refer to the `tf.keras.utils.SidecarEvaluator` API documentation for more details on option 1."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "U_c0EiwB88OG"
+      },
+      "source": [
+        "Sidecar evaluation is supported only with a single task. This means:\n",
+        "\n",
+        "*   It is guaranteed that each example is evaluated once. In the event the\n",
+        "    evaluator is preempted or restarted, it simply restarts the\n",
+        "    evaluation loop from the latest checkpoint, and the partial evaluation\n",
+        "    progress made before the restart is discarded.\n",
+        "\n",
+        "*   However, running evaluation on a single task implies that a full evaluation\n",
+        "    can possibly take a long time.\n",
+        "\n",
+        "*   If the size of the model is too large to fit into an evaluator's memory,\n",
+        "    single sidecar evaluation is not applicable."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "VNJoWVc797B1"
+      },
+      "source": [
+        "Another caveat is that the `tf.keras.utils.SidecarEvaluator` implementation, and the custom\n",
+        "evaluation loop below, may skip some checkpoints because it always picks up the\n",
+        "latest checkpoint available, and during an evaluation epoch, multiple\n",
+        "checkpoints can be produced from the training cluster. You can write a custom\n",
+        "evaluation loop that evaluates every checkpoint, but it is not covered in this\n",
+        "tutorial. On the other hand, it may sit idle if checkpoints are produced less\n",
+        "frequently than how long it takes to run evaluation."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "G5jopxBd85Ji"
+      },
+      "source": [
+        "A custom evaluation loop provides more control over the details, such as choosing which checkpoint to evaluate, or providing any additional logic to run along with evaluation. The following is a possible custom sidecar evaluation loop:\n",
         "\n",
         "```python\n",
         "checkpoint_dir = ...\n",
@@ -1022,7 +1075,7 @@
         "  eval_model.evaluate(eval_data)\n",
         "\n",
         "  # Evaluation finishes when it has evaluated the last epoch.\n",
-        "  if latest_checkpoint.endswith('-{}'.format(train_epoches)):\n",
+        "  if latest_checkpoint.endswith('-{}'.format(train_epochs)):\n",
         "    break\n",
         "```"
       ]
@@ -1040,7 +1093,7 @@
         "\n",
         "In a real production environment, you will run all tasks in different processes on different machines. The simplest way to configure cluster information on each task is to set `\"TF_CONFIG\"` environment variables and use a `tf.distribute.cluster_resolver.TFConfigClusterResolver` to parse `\"TF_CONFIG\"`.\n",
         "\n",
-        "For a general description of `\"TF_CONFIG\"` environment variables, refer to the [Distributed training](https://www.tensorflow.org/guide/distributed_training#setting_up_tf_config_environment_variable) guide.\n",
+        "For a general description of `\"TF_CONFIG\"` environment variables, refer to \"Setting up the `TF_CONFIG` environment variable\" in the [Distributed training](../../guide/distributed_training.ipynb) guide.\n",
         "\n",
         "If you start your training tasks using Kubernetes or other configuration templates, likely, these templates have already set `“TF_CONFIG\"` for you."
       ]
@@ -1053,7 +1106,7 @@
       "source": [
         "### Set the `\"TF_CONFIG\"` environment variable\n",
         "\n",
-        "Suppose you have 3 workers and 2 parameter servers, the `\"TF_CONFIG\"` of worker 1 can be:\n",
+        "Suppose you have 3 workers and 2 parameter servers. Then the `\"TF_CONFIG\"` of worker 1 can be:\n",
         "\n",
         "```python\n",
         "os.environ[\"TF_CONFIG\"] = json.dumps({\n",
@@ -1095,12 +1148,12 @@
         "if cluster_resolver.task_type in (\"worker\", \"ps\"):\n",
         "  # Start a TensorFlow server and wait.\n",
         "elif cluster_resolver.task_type == \"evaluator\":\n",
-        "  # Run side-car evaluation\n",
+        "  # Run sidecar evaluation\n",
         "else:\n",
         "  # Run the coordinator.\n",
         "```\n",
         "\n",
-        "The following code starts a TensorFlow server and waits:\n",
+        "The following code starts a TensorFlow server and waits, useful for the `\"worker\"` and `\"ps\"` roles:\n",
         "\n",
         "```python\n",
         "# Set the environment variable to allow reporting worker and ps failure to the\n",
@@ -1134,7 +1187,7 @@
       "source": [
         "### Worker failure\n",
         "\n",
-        "`tf.distribute.experimental.coordinator.ClusterCoordinator` or `Model.fit` provide built-in fault tolerance for worker failure. Upon worker recovery, the previously provided dataset function (either to `ClusterCoordinator.create_per_worker_dataset` for a custom training loop, or `tf.keras.utils.experimental.DatasetCreator` for `Model.fit`) will be invoked on the workers to re-create the datasets."
+        "Both the `tf.distribute.coordinator.ClusterCoordinator` custom training loop and `Model.fit` approaches provide built-in fault tolerance for worker failure. Upon worker recovery, the `ClusterCoordinator` invokes dataset re-creation on the workers."
       ]
     },
     {
@@ -1178,7 +1231,7 @@
         "global_steps = int(optimizer.iterations.numpy())\n",
         "starting_epoch = global_steps // steps_per_epoch\n",
         "\n",
-        "for _ in range(starting_epoch, num_epoches):\n",
+        "for _ in range(starting_epoch, num_epochs):\n",
         "  for _ in range(steps_per_epoch):\n",
         "    coordinator.schedule(step_fn, args=(per_worker_iterator,))\n",
         "  coordinator.join()\n",
@@ -1218,9 +1271,9 @@
       "source": [
         "## Performance improvement\n",
         "\n",
-        "There are several possible reasons if you see performance issues when you train with `ParameterServerStrategy` and `ClusterResolver`.\n",
+        "There are several possible reasons you may experience performance issues when you train with `tf.distribute.ParameterServerStrategy` and `tf.distribute.coordinator.ClusterCoordinator`.\n",
         "\n",
-        "One common reason is parameter servers have unbalanced load and some heavily-loaded parameter servers have reached capacity. There can also be multiple root causes. Some simple methods to mitigate this issue are to:\n",
+        "One common reason is that the parameter servers have unbalanced load and some heavily-loaded parameter servers have reached capacity. There can also be multiple root causes. Some simple methods to mitigate this issue are to:\n",
         "\n",
         "1. Shard your large model variables via specifying a `variable_partitioner` when constructing a `ParameterServerStrategy`.\n",
         "2. Avoid creating a hotspot variable that is required by all parameter servers in a single step if possible. For example, use a constant learning rate or subclass `tf.keras.optimizers.schedules.LearningRateSchedule` in optimizers since the default behavior is that the learning rate will become a variable placed on a particular parameter server and requested by all other parameter servers in each step.\n",
@@ -1247,7 +1300,7 @@
         "\n",
         "As the library is optimized further, hopefully most users won't have to manually pack steps in the future.\n",
         "\n",
-        "In addition, a small trick for performance improvement is to schedule functions without a return value as explained in the handling task failure section above."
+        "In addition, a small trick for performance improvement is to schedule functions without a return value as explained in the [handling task failure section](#handling_task_failure) above."
       ]
     },
     {
@@ -1267,8 +1320,7 @@
         "- `os.environment[\"grpc_fail_fast\"]=\"use_caller\"` is needed on every task including the coordinator, to make fault tolerance work properly.\n",
         "- Synchronous parameter server training is not supported.\n",
         "- It is usually necessary to pack multiple steps into a single function to achieve optimal performance.\n",
-        "- It is not supported to load a saved_model via `tf.saved_model.load` containing sharded variables. Note loading such a saved_model using TensorFlow Serving is expected to work.\n",
-        "- It is not supported to load a checkpoint containing sharded optimizer slot variables into a different number of shards.\n",
+        "- It is not supported to load a saved_model via `tf.saved_model.load` containing sharded variables. Note loading such a saved_model using TensorFlow Serving is expected to work (refer to the [serving tutorial](https://www.tensorflow.org/tfx/tutorials/serving/rest_simple) for details).\n",
         "- It is not supported to recover from parameter server failure without restarting the coordinator task.\n",
         "- Creation of `tf.lookup.StaticHashTable`, commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`, should be placed under `Strategy.scope`. Otherwise, resources will be placed on the coordinator, and lookup RPCs from workers to the coordinator incur performance implications.\n"
       ]
@@ -1296,7 +1348,7 @@
         "### Custom training loop specifics\n",
         "\n",
         "- `ClusterCoordinator.schedule` doesn't support visitation guarantees for a dataset.\n",
-        "- When `ClusterCoordinator.create_per_worker_dataset` is used, the whole dataset must be created inside the function passed to it.\n",
+        "- When `ClusterCoordinator.create_per_worker_dataset` is used with a callable as input, the whole dataset must be created inside the function passed to it.\n",
         "- `tf.data.Options` is ignored in a dataset created by `ClusterCoordinator.create_per_worker_dataset`."
       ]
     }
@@ -1306,6 +1358,7 @@
     "colab": {
       "collapsed_sections": [],
       "name": "parameter_server_training.ipynb",
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 44a3c7bafb44232d25a1e42a6be686ad4c6b67c7 Mon Sep 17 00:00:00 2001
From: Bing Hu 
Date: Thu, 7 Apr 2022 15:32:48 -0700
Subject: [PATCH 057/872] Review and update public doc "Custom training with
 tf.distribute.Strategy"

PiperOrigin-RevId: 440216715
---
 .../distribute/custom_training.ipynb          | 66 ++++++++-----------
 1 file changed, 29 insertions(+), 37 deletions(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index da45c340b1a..9dbb2d3fad5 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -68,9 +68,9 @@
         "id": "FbVhjPpzn6BM"
       },
       "source": [
-        "This tutorial demonstrates how to use [`tf.distribute.Strategy`](https://www.tensorflow.org/guide/distributed_training) with custom training loops. We will train a simple CNN model on the fashion MNIST dataset. The fashion MNIST dataset contains 60000 train images of size 28 x 28 and 10000 test images of size 28 x 28.\n",
+        "This tutorial demonstrates how to use `tf.distribute.Strategy` — a TensorFlow API that provides an abstraction for [distributing your training](../../guide/distributed_training.ipynb) across multiple processing units (GPUs, multiple machines, or TPUs) — with custom training loops. In this example, you will train a simple convolutional neural network on the [Fashion MNIST dataset](https://github.com/zalandoresearch/fashion-mnist) containing 70,000 images of size 28 x 28.\n",
         "\n",
-        "We are using custom training loops to train our model because they give us flexibility and a greater control on training. Moreover, it is easier to debug the model and the training loop."
+        "[Custom training loops](../customization/custom_training_walkthrough.ipynb) provide flexibility and a greater control on training. They also make it is easier to debug the model and the training loop."
       ]
     },
     {
@@ -97,7 +97,7 @@
         "id": "MM6W__qraV55"
       },
       "source": [
-        "## Download the fashion MNIST dataset"
+        "## Download the Fashion MNIST dataset"
       ]
     },
     {
@@ -112,14 +112,14 @@
         "\n",
         "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
         "\n",
-        "# Adding a dimension to the array -> new shape == (28, 28, 1)\n",
-        "# We are doing this because the first layer in our model is a convolutional\n",
+        "# Add a dimension to the array -> new shape == (28, 28, 1)\n",
+        "# This is done because the first layer in our model is a convolutional\n",
         "# layer and it requires a 4D input (batch_size, height, width, channels).\n",
         "# batch_size dimension will be added later on.\n",
         "train_images = train_images[..., None]\n",
         "test_images = test_images[..., None]\n",
         "\n",
-        "# Getting the images in [0, 1] range.\n",
+        "# Scale the images to the [0, 1] range.\n",
         "train_images = train_images / np.float32(255)\n",
         "test_images = test_images / np.float32(255)"
       ]
@@ -141,13 +141,13 @@
       "source": [
         "How does `tf.distribute.MirroredStrategy` strategy work?\n",
         "\n",
-        "*   All the variables and the model graph is replicated on the replicas.\n",
+        "*   All the variables and the model graph are replicated across the replicas.\n",
         "*   Input is evenly distributed across the replicas.\n",
         "*   Each replica calculates the loss and gradients for the input it received.\n",
         "*   The gradients are synced across all the replicas by summing them.\n",
         "*   After the sync, the same update is made to the copies of the variables on each replica.\n",
         "\n",
-        "Note: You can put all the code below inside a single scope. We are dividing it into several code cells for illustration purposes.\n"
+        "Note: You can put all the code below inside a single scope. This example divides it into several code cells for illustration purposes.\n"
       ]
     },
     {
@@ -158,8 +158,8 @@
       },
       "outputs": [],
       "source": [
-        "# If the list of devices is not specified in the\n",
-        "# `tf.distribute.MirroredStrategy` constructor, it will be auto-detected.\n",
+        "# If the list of devices is not specified in\n",
+        "# `tf.distribute.MirroredStrategy` constructor, they will be auto-detected.\n",
         "strategy = tf.distribute.MirroredStrategy()"
       ]
     },
@@ -171,7 +171,7 @@
       },
       "outputs": [],
       "source": [
-        "print ('Number of devices: {}'.format(strategy.num_replicas_in_sync))"
+        "print('Number of devices: {}'.format(strategy.num_replicas_in_sync))"
       ]
     },
     {
@@ -183,15 +183,6 @@
         "## Setup input pipeline"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "0Qb6nDgxiN_n"
-      },
-      "source": [
-        "Export the graph and the variables to the platform-agnostic SavedModel format. After your model is saved, you can load it with or without the scope."
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -240,7 +231,7 @@
       "source": [
         "## Create the model\n",
         "\n",
-        "Create a model using `tf.keras.Sequential`. You can also use the Model Subclassing API to do this."
+        "Create a model using `tf.keras.Sequential`. You can also use the [Model Subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models) or the [functional API](https://www.tensorflow.org/guide/keras/functional) to do this."
       ]
     },
     {
@@ -286,14 +277,14 @@
       "source": [
         "## Define the loss function\n",
         "\n",
-        "Normally, on a single machine with 1 GPU/CPU, loss is divided by the number of examples in the batch of input.\n",
+        "Normally, on a single machine with single GPU/CPU, loss is divided by the number of examples in the batch of input.\n",
         "\n",
         "*So, how should the loss be calculated when using a `tf.distribute.Strategy`?*\n",
         "\n",
         "* For an example, let's say you have 4 GPU's and a batch size of 64. One batch of input is distributed\n",
         "across the replicas (4 GPUs), each replica getting an input of size 16.\n",
         "\n",
-        "* The model on each replica does a forward pass with its respective input and calculates the loss. Now, instead of dividing the loss by the number of examples in its respective input (BATCH_SIZE_PER_REPLICA = 16), the loss should be divided by the GLOBAL_BATCH_SIZE (64)."
+        "* The model on each replica does a forward pass with its respective input and calculates the loss. Now, instead of dividing the loss by the number of examples in its respective input (`BATCH_SIZE_PER_REPLICA` = 16), the loss should be divided by the `GLOBAL_BATCH_SIZE` (64)."
       ]
     },
     {
@@ -315,10 +306,10 @@
       "source": [
         "*How to do this in TensorFlow?*\n",
         "\n",
-        "* If you're writing a custom training loop, as in this tutorial, you should sum the per example losses and divide the sum by the GLOBAL_BATCH_SIZE: \n",
+        "* If you're writing a custom training loop, as in this tutorial, you should sum the per example losses and divide the sum by the `GLOBAL_BATCH_SIZE`: \n",
         "`scale_loss = tf.reduce_sum(loss) * (1. / GLOBAL_BATCH_SIZE)`\n",
         "or you can use `tf.nn.compute_average_loss` which takes the per example loss,\n",
-        "optional sample weights, and GLOBAL_BATCH_SIZE as arguments and returns the scaled loss.\n",
+        "optional sample weights, and `GLOBAL_BATCH_SIZE` as arguments and returns the scaled loss.\n",
         "\n",
         "* If you are using regularization losses in your model then you need to scale\n",
         "the loss value by number of replicas. You can do this by using the `tf.nn.scale_regularization_loss` function.\n",
@@ -351,7 +342,7 @@
       "outputs": [],
       "source": [
         "with strategy.scope():\n",
-        "  # Set reduction to `none` so we can do the reduction afterwards and divide by\n",
+        "  # Set reduction to `NONE` so you can do the reduction afterwards and divide by\n",
         "  # global batch size.\n",
         "  loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
         "      from_logits=True,\n",
@@ -484,9 +475,9 @@
         "\n",
         "  template = (\"Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, \"\n",
         "              \"Test Accuracy: {}\")\n",
-        "  print (template.format(epoch+1, train_loss,\n",
-        "                         train_accuracy.result()*100, test_loss.result(),\n",
-        "                         test_accuracy.result()*100))\n",
+        "  print(template.format(epoch + 1, train_loss,\n",
+        "                         train_accuracy.result() * 100, test_loss.result(),\n",
+        "                         test_accuracy.result() * 100))\n",
         "\n",
         "  test_loss.reset_states()\n",
         "  train_accuracy.reset_states()\n",
@@ -501,7 +492,7 @@
       "source": [
         "Things to note in the example above:\n",
         "\n",
-        "* We are iterating over the `train_dist_dataset` and `test_dist_dataset` using  a `for x in ...` construct.\n",
+        "* Iterate over the `train_dist_dataset` and `test_dist_dataset` using  a `for x in ...` construct.\n",
         "* The scaled loss is the return value of the `distributed_train_step`. This value is aggregated across replicas using the `tf.distribute.Strategy.reduce` call and then across batches by summing the return value of the `tf.distribute.Strategy.reduce` calls.\n",
         "* `tf.keras.Metrics` should be updated inside `train_step` and `test_step` that gets executed by `tf.distribute.Strategy.run`.\n",
         "*`tf.distribute.Strategy.run` returns results from each local replica in the strategy, and there are multiple ways to consume this result. You can do `tf.distribute.Strategy.reduce` to get an aggregated value. You can also do `tf.distribute.Strategy.experimental_local_results` to get the list of values contained in the result, one per local replica.\n"
@@ -570,8 +561,8 @@
         "for images, labels in test_dataset:\n",
         "  eval_step(images, labels)\n",
         "\n",
-        "print ('Accuracy after restoring the saved model without strategy: {}'.format(\n",
-        "    eval_accuracy.result()*100))"
+        "print('Accuracy after restoring the saved model without strategy: {}'.format(\n",
+        "    eval_accuracy.result() * 100))"
       ]
     },
     {
@@ -606,7 +597,7 @@
         "  average_train_loss = total_loss / num_batches\n",
         "\n",
         "  template = (\"Epoch {}, Loss: {}, Accuracy: {}\")\n",
-        "  print (template.format(epoch+1, average_train_loss, train_accuracy.result()*100))\n",
+        "  print(template.format(epoch + 1, average_train_loss, train_accuracy.result() * 100))\n",
         "  train_accuracy.reset_states()"
       ]
     },
@@ -617,7 +608,7 @@
       },
       "source": [
         "### Iterating inside a tf.function\n",
-        "You can also iterate over the entire input `train_dist_dataset` inside a tf.function using the `for x in ...` construct or by creating iterators like we did above. The example below demonstrates wrapping one epoch of training in a tf.function and iterating over `train_dist_dataset` inside the function."
+        "You can also iterate over the entire input `train_dist_dataset` inside a `tf.function` using the `for x in ...` construct or by creating iterators like you did above. The example below demonstrates wrapping one epoch of training with a `@tf.function` decorator and iterating over `train_dist_dataset` inside the function."
       ]
     },
     {
@@ -643,7 +634,7 @@
         "  train_loss = distributed_train_epoch(train_dist_dataset)\n",
         "\n",
         "  template = (\"Epoch {}, Loss: {}, Accuracy: {}\")\n",
-        "  print (template.format(epoch+1, train_loss, train_accuracy.result()*100))\n",
+        "  print(template.format(epoch + 1, train_loss, train_accuracy.result() * 100))\n",
         "\n",
         "  train_accuracy.reset_states()"
       ]
@@ -658,7 +649,7 @@
         "\n",
         "Note: As a general rule, you should use `tf.keras.Metrics` to track per-sample values and avoid values that have been aggregated within a replica.\n",
         "\n",
-        "We do *not* recommend using `tf.metrics.Mean` to track the training loss across different replicas, because of the loss scaling computation that is carried out.\n",
+        "Because of the loss scaling computation that is carried out, it's not recommended to use `tf.metrics.Mean` to track the training loss across different replicas.\n",
         "\n",
         "For example, if you run a training job with the following characteristics:\n",
         "* Two replicas\n",
@@ -699,7 +690,8 @@
         "## Next steps\n",
         "\n",
         "*   Try out the new `tf.distribute.Strategy` API on your models.\n",
-        "*   Visit the [Performance section](../../guide/function.ipynb) in the guide to learn more about other strategies and [tools](../../guide/profiler.md) you can use to optimize the performance of your TensorFlow models."
+        "*   Visit the [Better performance with tf.function](../../guide/function.ipynb) and [TensorFlow Profiler](../../guide/profiler.md) guide to learn more about tools to optimize the performance of your TensorFlow models.\n",
+        "*   The [Distributed training in TensorFlow](../../guide/distributed_training.ipynb) guide provides an overview of the available distribution strategies."
       ]
     }
   ],

From 9b3d081efba20decde0c25b32820ccf550528d4a Mon Sep 17 00:00:00 2001
From: Nathan Luehr 
Date: Fri, 8 Apr 2022 12:19:43 -0500
Subject: [PATCH 058/872] Simplify GPU install instructions.

Now that TensorRT is available in the regular CUDA apt repos, the
'machine-learning' repos are no longer needed in the GPU setup instructions.
---
 site/en/install/gpu.md | 49 ++++++++++++------------------------------
 1 file changed, 14 insertions(+), 35 deletions(-)

diff --git a/site/en/install/gpu.md b/site/en/install/gpu.md
index 76008867312..041241f3bed 100644
--- a/site/en/install/gpu.md
+++ b/site/en/install/gpu.md
@@ -104,28 +104,17 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i
 sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
 sudo apt-get update
 
-wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
-
-sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
-sudo apt-get update
-
-wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/libnvinfer7_7.1.3-1+cuda11.0_amd64.deb
-sudo apt install ./libnvinfer7_7.1.3-1+cuda11.0_amd64.deb
-sudo apt-get update
-
 # Install development and runtime libraries (~4GB)
 sudo apt-get install --no-install-recommends \
-    cuda-11-0 \
-    libcudnn8=8.0.4.30-1+cuda11.0  \
-    libcudnn8-dev=8.0.4.30-1+cuda11.0
+    cuda-11-2 \
+    libcudnn8=8.1.0.77-1+cuda11.2  \
+    libcudnn8-dev=8.1.0.77-1+cuda11.2 \
+    libnvinfer8=8.2.4-1+cuda11.4 \
+    libnvinfer-dev=8.2.4-1+cuda11.4 \
+    libnvinfer-plugin8=8.2.4-1+cuda11.4 \
+    libnvinfer-plugin-dev=8.2.4-1+cuda11.4
 
 # Reboot. Check that GPUs are visible using the command: nvidia-smi
-
-# Install TensorRT. Requires that libcudnn8 is installed above.
-sudo apt-get install -y --no-install-recommends libnvinfer7=7.1.3-1+cuda11.0 \
-    libnvinfer-dev=7.1.3-1+cuda11.0 \
-    libnvinfer-plugin7=7.1.3-1+cuda11.0
-
#### Ubuntu 16.04 (CUDA 11.0) @@ -139,29 +128,19 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/ /" sudo apt-get update -wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb -sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb -sudo apt-get update -wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnvinfer7_7.1.3-1+cuda11.0_amd64.deb -sudo apt install ./libnvinfer7_7.1.3-1+cuda11.0_amd64.deb -sudo apt-get update # Install development and runtime libraries (~4GB) sudo apt-get install --no-install-recommends \ - cuda-11-0 \ - libcudnn8=8.0.4.30-1+cuda11.0 \ - libcudnn8-dev=8.0.4.30-1+cuda11.0 + cuda-11-2 \ + libcudnn8=8.1.0.77-1+cuda11.2 \ + libcudnn8-dev=8.1.0.77-1+cuda11.2 \ + libnvinfer8=8.0.1-1+cuda11.3 \ + libnvinfer-dev=8.0.1-1+cuda11.3 \ + libnvinfer-plugin8=8.0.1-1+cuda11.3 \ + libnvinfer-plugin-dev=8.0.1-1+cuda11.3 # Reboot. Check that GPUs are visible using the command: nvidia-smi - -# Install TensorRT. Requires that libcudnn7 is installed above. -sudo apt-get install -y --no-install-recommends \ - libnvinfer7=7.1.3-1+cuda11.0 \ - libnvinfer-dev=7.1.3-1+cuda11.0 \ - libnvinfer-plugin7=7.1.3-1+cuda11.0 \ - libnvinfer-plugin-dev=7.1.3-1+cuda11.0 - From 279bbdda56bf41dd60e99155cb6b727f538bc52d Mon Sep 17 00:00:00 2001 From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com> Date: Sat, 9 Apr 2022 20:11:36 +0530 Subject: [PATCH 059/872] Updated correct path for checkpoint and HDF5 links Updated links with the correct path for the checkpoint link in the "What are these files section?" and also for HDF5 in the "Manually saved weights section. Earlier, the checkpoint link was showing the Saved model link and HDF5 file was showing the tensorflow.js conversion model. --- site/en/tutorials/keras/save_and_load.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/keras/save_and_load.ipynb b/site/en/tutorials/keras/save_and_load.ipynb index 6e48a08a4a6..dd297c4a14f 100644 --- a/site/en/tutorials/keras/save_and_load.ipynb +++ b/site/en/tutorials/keras/save_and_load.ipynb @@ -485,7 +485,7 @@ "id": "JtdYhvWnH2ib" }, "source": [ - "The above code stores the weights to a collection of [checkpoint](https://www.tensorflow.org/guide/saved_model#save_and_restore_variables)-formatted files that contain only the trained weights in a binary format. Checkpoints contain:\n", + "The above code stores the weights to a collection of [checkpoint](https://www.tensorflow.org/guide/checkpoint)-formatted files that contain only the trained weights in a binary format. Checkpoints contain:\n", "* One or more shards that contain your model's weights.\n", "* An index file that indicates which weights are stored in which shard.\n", "\n", @@ -500,7 +500,7 @@ "source": [ "## Manually save weights\n", "\n", - "Manually saving weights with the `Model.save_weights` method. By default, `tf.keras`—and `save_weights` in particular—uses the TensorFlow [checkpoint](../../guide/checkpoint.ipynb) format with a `.ckpt` extension (saving in [HDF5](https://js.tensorflow.org/tutorials/import-keras.html) with a `.h5` extension is covered in the [Save and serialize models](https://www.tensorflow.org/guide/keras/save_and_serialize#weights-only_saving_in_savedmodel_format) guide):" + "Manually saving weights with the `Model.save_weights` method. By default, `tf.keras`—and `save_weights` in particular—uses the TensorFlow [checkpoint](../../guide/checkpoint.ipynb) format with a `.ckpt` extension (saving in [HDF5](https://www.tensorflow.org/guide/keras/save_and_serialize#hdf5_format) with a `.h5` extension is covered in the [Save and serialize models](https://www.tensorflow.org/guide/keras/save_and_serialize#weights-only_saving_in_savedmodel_format) guide):" ] }, { From 30eeb1b77f7f6c29abe9f648d8970fad3a2f40f4 Mon Sep 17 00:00:00 2001 From: Nik Schaefer Date: Sat, 9 Apr 2022 11:57:24 -0500 Subject: [PATCH 060/872] Fix grammatical error in custom_layer docs --- site/en/tutorials/customization/custom_layers.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/customization/custom_layers.ipynb b/site/en/tutorials/customization/custom_layers.ipynb index 8214537210f..8bfe0a01b09 100644 --- a/site/en/tutorials/customization/custom_layers.ipynb +++ b/site/en/tutorials/customization/custom_layers.ipynb @@ -103,7 +103,7 @@ "\n", "Most of the time when writing code for machine learning models you want to operate at a higher level of abstraction than individual operations and manipulation of individual variables.\n", "\n", - "Many machine learning models are expressible as the composition and stacking of relatively simple layers, and TensorFlow provides both a set of many common layers as a well as easy ways for you to write your own application-specific layers either from scratch or as the composition of existing layers.\n", + "Many machine learning models are expressible as the composition and stacking of relatively simple layers, and TensorFlow provides both a set of many common layers as well as easy ways for you to write your own application-specific layers either from scratch or as the composition of existing layers.\n", "\n", "TensorFlow includes the full [Keras](https://keras.io) API in the tf.keras package, and the Keras layers are very useful when building your own models.\n" ] From eff29eeacc846afeca2dd8be985c1381a2ff37ce Mon Sep 17 00:00:00 2001 From: 372046933 <372046933@users.noreply.github.com> Date: Sun, 10 Apr 2022 19:20:55 +0800 Subject: [PATCH 061/872] Update create_op.md Replace `tensorflow/core/lib/core/errors.h` with`tensorflow/core/platform/errors.h` and `tensorflow/core/lib/core/status.h` with `tensorflow/core/platform/status.h` --- site/en/guide/create_op.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/site/en/guide/create_op.md b/site/en/guide/create_op.md index 26ca8910225..5f35812c272 100644 --- a/site/en/guide/create_op.md +++ b/site/en/guide/create_op.md @@ -524,10 +524,10 @@ This asserts that the input is a vector, and returns having set the of a tensor in [`tensorflow/core/framework/tensor_shape.h`](https://www.tensorflow.org/code/tensorflow/core/framework/tensor_shape.h) * The error itself, which is represented by a `Status` object, see - [`tensorflow/core/lib/core/status.h`](https://www.tensorflow.org/code/tensorflow/core/lib/core/status.h). A + [`tensorflow/core/platform/status.h`](https://www.tensorflow.org/code/tensorflow/core/platform/status.h). A `Status` has both a type (frequently `InvalidArgument`, but see the list of types) and a message. Functions for constructing an error may be found in - [`tensorflow/core/lib/core/errors.h`][validation-macros]. + [`tensorflow/core/platform/errors.h`][validation-macros]. Alternatively, if you want to test whether a `Status` object returned from some function is an error, and if so return it, use @@ -1497,7 +1497,7 @@ of building TensorFlow from source. [standard_ops-py]:https://www.tensorflow.org/code/tensorflow/python/ops/standard_ops.py [standard_ops-cc]:https://www.tensorflow.org/code/tensorflow/cc/ops/standard_ops.h [python-BUILD]:https://www.tensorflow.org/code/tensorflow/python/BUILD -[validation-macros]:https://www.tensorflow.org/code/tensorflow/core/lib/core/errors.h +[validation-macros]:https://www.tensorflow.org/code/tensorflow/core/platform/errors.h [op_def_builder]:https://www.tensorflow.org/code/tensorflow/core/framework/op_def_builder.h [register_types]:https://www.tensorflow.org/code/tensorflow/core/framework/register_types.h [FinalizeAttr]:https://www.tensorflow.org/code/tensorflow/core/framework/op_def_builder.cc From 3a6a7a3322de8620f0031ae3a0a4b62e40d1d7f2 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Mon, 11 Apr 2022 04:22:20 -0700 Subject: [PATCH 062/872] normalize the public_api filters More testabole -> more tests. PiperOrigin-RevId: 440850613 --- .../api_generator/generate_lib.py | 30 ++- .../api_generator/public_api.py | 227 ++++++++++++++---- .../api_generator/public_api_test.py | 123 +++++----- .../api_generator/test_module1.py | 25 -- .../api_generator/test_module2.py | 31 --- .../tensorflow_docs/api_generator/traverse.py | 125 +--------- .../api_generator/traverse_test.py | 12 - 7 files changed, 265 insertions(+), 308 deletions(-) delete mode 100644 tools/tensorflow_docs/api_generator/test_module1.py delete mode 100644 tools/tensorflow_docs/api_generator/test_module2.py diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index e7590663490..e378bb2f9c6 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -215,7 +215,8 @@ def extract(py_modules, base_dir, private_map, visitor_cls=doc_generator_visitor.DocGeneratorVisitor, - callbacks=None): + callbacks=None, + include_default_callbacks=True): """Walks the module contents, returns an index of all visited objects. The return value is an instance of `self._visitor_cls`, usually: @@ -235,6 +236,9 @@ def extract(py_modules, `PublicApiFilter` and the accumulator (`DocGeneratorVisitor`). The primary use case for these is to filter the list of children (see: `public_api.local_definitions_filter`) + include_default_callbacks: When true the long list of standard + visitor-callbacks are included. When false, only the `callbacks` argument + is used. Returns: The accumulator (`DocGeneratorVisitor`) @@ -246,16 +250,28 @@ def extract(py_modules, raise ValueError("only pass one [('name',module)] pair in py_modules") short_name, py_module = py_modules[0] - api_filter = public_api.PublicAPIFilter( - base_dir=base_dir, - private_map=private_map) - - accumulator = visitor_cls() # The objects found during traversal, and their children are passed to each # of these visitors in sequence. Each visitor returns the list of children # to be passed to the next visitor. - visitors = [api_filter, public_api.ignore_typing] + callbacks + [accumulator] + if include_default_callbacks: + visitors = [ + # filter the api. + public_api.FailIfNestedTooDeep(10), + public_api.filter_module_all, + public_api.add_proto_fields, + public_api.filter_builtin_modules, + public_api.filter_private_symbols, + public_api.FilterBaseDirs(base_dir), + public_api.FilterPrivateMap(private_map), + public_api.filter_doc_controls_skip, + public_api.ignore_typing + ] + else: + visitors = [] + + accumulator = visitor_cls() + visitors = visitors + callbacks + [accumulator] traverse.traverse(py_module, visitors, short_name) diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py index 925e5415856..ee327ee5c4f 100644 --- a/tools/tensorflow_docs/api_generator/public_api.py +++ b/tools/tensorflow_docs/api_generator/public_api.py @@ -15,26 +15,29 @@ """Visitor restricting traversal to only the public tensorflow API.""" import ast +import dataclasses import inspect import os +import sys import pathlib import textwrap import types import typing -from typing import Any, Callable, List, Sequence, Tuple, Union - +from typing import Any, Callable, Dict, Iterable, List, Sequence, Tuple, Union from tensorflow_docs.api_generator import doc_controls from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import get_source +from google.protobuf.message import Message as ProtoMessage + _TYPING_IDS = frozenset( id(obj) for obj in typing.__dict__.values() if not doc_generator_visitor.maybe_singleton(obj)) -Children = List[Tuple[str, Any]] +Children = Iterable[Tuple[str, Any]] ApiFilter = Callable[[Tuple[str, ...], Any, Children], Children] @@ -300,67 +303,195 @@ def explicit_package_contents_filter(path: Sequence[str], parent: Any, ]) -class PublicAPIFilter(object): - """Visitor to use with `traverse` to filter just the public API.""" - - def __init__(self, base_dir, private_map=None): - """Constructor. - - Args: - base_dir: The directory to take source file paths relative to. - private_map: A mapping from dotted path like "tf.symbol" to a list of - names. Included names will not be listed at that location. - """ - self._base_dir = base_dir - self._private_map = private_map or {} - - def _is_private(self, path, parent, name, obj): - """Returns whether a name is private or not.""" +@dataclasses.dataclass +class FailIfNestedTooDeep: + max_depth: int - # Skip objects blocked by doc_controls. - if doc_controls.should_skip(obj): - return True + def __call__(self, path: Sequence[str], parent: Any, + children: Children) -> Children: + if inspect.ismodule(parent) and len(path) > 10: + raise RuntimeError('Modules nested too deep:\n\n{}\n\nThis is likely a ' + 'problem with an accidental public import.'.format( + '.'.join(path))) + return children - if isinstance(parent, type): - if doc_controls.should_skip_class_attr(parent, name): - return True - if doc_controls.should_doc_private(obj): - return False +@dataclasses.dataclass +class FilterBaseDirs: + base_dirs: Sequence[pathlib.Path] - if inspect.ismodule(obj): - mod_base_dirs = get_module_base_dirs(obj) + def __call__(self, path: Sequence[str], parent: Any, + children: Children) -> Children: + for name, child in children: + if not inspect.ismodule(child): + yield name, child + continue + mod_base_dirs = get_module_base_dirs(child) # This check only handles normal packages/modules. Namespace-package # contents will get filtered when the submodules are checked. if len(mod_base_dirs) == 1: mod_base_dir = mod_base_dirs[0] # Check that module is in one of the `self._base_dir`s - if not any(base in mod_base_dir.parents for base in self._base_dir): - return True + if not any(base in mod_base_dir.parents for base in self.base_dirs): + continue + yield name, child + - # Skip objects blocked by the private_map - if name in self._private_map.get('.'.join(path), []): - return True +@dataclasses.dataclass +class FilterPrivateMap: + private_map: Dict[str, List[str]] + def __call__(self, path: Sequence[str], parent: Any, + children: Children) -> Children: + if self.private_map is None: + yield from children + + for name, child in children: + if name in self.private_map.get('.'.join(path), []): + continue + yield (name, child) + + +def filter_private_symbols(path: Sequence[str], parent: Any, + children: Children) -> Children: + del path + del parent + for name, child in children: # Skip "_" hidden attributes if name.startswith('_') and name not in ALLOWED_DUNDER_METHODS: - return True + if not doc_controls.should_doc_private(child): + continue + yield (name, child) - return False - def __call__(self, path: Sequence[str], parent: Any, - children: Children) -> Children: - """Visitor interface, see `traverse` for details.""" +def filter_doc_controls_skip(path: Sequence[str], parent: Any, + children: Children) -> Children: + del path + for name, child in children: + if doc_controls.should_skip(child): + continue + if isinstance(parent, type): + if doc_controls.should_skip_class_attr(parent, name): + continue + yield (name, child) - # Avoid long waits in cases of pretty unambiguous failure. - if inspect.ismodule(parent) and len(path) > 10: - raise RuntimeError('Modules nested too deep:\n\n{}\n\nThis is likely a ' - 'problem with an accidental public import.'.format( - '.'.join(path))) - # Remove things that are not visible. - children = [(child_name, child_obj) - for child_name, child_obj in list(children) - if not self._is_private(path, parent, child_name, child_obj)] +def filter_module_all(path: Sequence[str], parent: Any, + children: Children) -> Children: + """Filters module children based on the "__all__" arrtibute. + + Args: + path: API to this symbol + parent: The object + children: A list of (name, object) pairs. + + Returns: + `children` filtered to respect __all__ + """ + del path + if not (inspect.ismodule(parent) and hasattr(parent, '__all__')): + return children + module_all = set(parent.__all__) + children = [(name, value) for (name, value) in children if name in module_all] + + return children + + +def add_proto_fields(path: Sequence[str], parent: Any, + children: Children) -> Children: + """Add properties to Proto classes, so they can be documented. + + Warning: This inserts the Properties into the class so the rest of the system + is unaffected. This patching is acceptable because there is never a reason to + run other tensorflow code in the same process as the doc generator. + + Args: + path: API to this symbol + parent: The object + children: A list of (name, object) pairs. + + Returns: + `children` with proto fields added as properties. + """ + del path + if not inspect.isclass(parent) or not issubclass(parent, ProtoMessage): + return children + descriptor = getattr(parent, 'DESCRIPTOR', None) + if descriptor is None: return children + fields = descriptor.fields + if not fields: + return children + + field = fields[0] + # Make the dictionaries mapping from int types and labels to type and + # label names. + field_types = { + getattr(field, name): name + for name in dir(field) + if name.startswith('TYPE') + } + + labels = { + getattr(field, name): name + for name in dir(field) + if name.startswith('LABEL') + } + + field_properties = {} + + for field in fields: + name = field.name + doc_parts = [] + + label = labels[field.label].lower().replace('label_', '') + if label != 'optional': + doc_parts.append(label) + + type_name = field_types[field.type] + if type_name == 'TYPE_MESSAGE': + type_name = field.message_type.name + elif type_name == 'TYPE_ENUM': + type_name = field.enum_type.name + else: + type_name = type_name.lower().replace('type_', '') + + doc_parts.append(type_name) + doc_parts.append(name) + doc = '`{}`'.format(' '.join(doc_parts)) + prop = property(fget=lambda x: x, doc=doc) + field_properties[name] = prop + + for name, prop in field_properties.items(): + setattr(parent, name, prop) + + children = dict(children) + children.update(field_properties) + children = sorted(children.items(), key=lambda item: item[0]) + + return children + + +def filter_builtin_modules(path: Sequence[str], parent: Any, + children: Children) -> Children: + """Filters module children to remove builtin modules. + + Args: + path: API to this symbol + parent: The object + children: A list of (name, object) pairs. + + Returns: + `children` with all builtin modules removed. + """ + del path + del parent + # filter out 'builtin' modules + filtered_children = [] + for name, child in children: + # Do not descend into built-in modules + if inspect.ismodule(child) and child.__name__ in sys.builtin_module_names: + continue + filtered_children.append((name, child)) + return filtered_children diff --git a/tools/tensorflow_docs/api_generator/public_api_test.py b/tools/tensorflow_docs/api_generator/public_api_test.py index 1b801787a14..2d785ef41e3 100644 --- a/tools/tensorflow_docs/api_generator/public_api_test.py +++ b/tools/tensorflow_docs/api_generator/public_api_test.py @@ -15,6 +15,7 @@ """Tests for tensorflow.tools.common.public_api.""" import inspect +import pathlib import types import typing @@ -41,60 +42,24 @@ def __call__(self, path, parent, children): self.last_children = list(children) # Make a copy to preserve state. return children - def test_call_forward(self): - visitor = self.TestVisitor() - - api_visitors = [public_api.PublicAPIFilter(base_dir='/'), visitor] - - path = ('tf', 'test') - parent = 'dummy' - children = [('name1', 'thing1'), ('name2', 'thing2')] - - for api_visitor in api_visitors: - children = api_visitor(path, parent, children) - - self.assertEqual(set([( - 'tf', - 'test', - )]), visitor.symbols) - self.assertEqual('dummy', visitor.last_parent) - self.assertEqual([('name1', 'thing1'), ('name2', 'thing2')], - visitor.last_children) - - def test_private_child_removal(self): - visitor = self.TestVisitor() - api_visitors = [ - public_api.PublicAPIFilter(base_dir='/'), - visitor, - ] - - children = [('name1', 'thing1'), ('_name2', 'thing2')] - path = ('tf', 'test') - parent = 'dummy' - for api_visitor in api_visitors: - children = api_visitor(path, parent, children) - - # Make sure the private symbols are removed before the visitor is called. - self.assertEqual([('name1', 'thing1')], visitor.last_children) - self.assertEqual([('name1', 'thing1')], children) - - def test_private_map_child_removal(self): - visitor = self.TestVisitor() - - api_visitors = [ - public_api.PublicAPIFilter( - base_dir='/', private_map={'tf.test': ['mock']}), visitor - ] - - children = [('name1', 'thing1'), ('mock', 'thing2')] - path = ('tf', 'test') - parent = 'dummy' - - for api_visitor in api_visitors: - children = api_visitor(path, parent, children) - # Make sure private aliases are removed. - self.assertEqual([('name1', 'thing1')], visitor.last_children) - self.assertEqual([('name1', 'thing1')], children) + def test_filter_private_symbols(self): + module = types.ModuleType('module') + module.a = 1 + module._b = 2 + + result = public_api.filter_private_symbols(('module'), module, + [('a', module.a), + ('_b', module._b)]) + self.assertEqual([('a', module.a)], list(result)) + + def test_private_map_filter(self): + private_map_filter = public_api.FilterPrivateMap({'tf.test': ['mock']}) + result = private_map_filter( + path=('tf', 'test'), + parent='dummy', + children=[('name1', 'thing1'), ('mock', 'thing2')]) + + self.assertEqual([('name1', 'thing1')], list(result)) def test_local_definitions_filter(self): tf = types.ModuleType('tf') @@ -197,17 +162,53 @@ def test_ignore_class_attr(self): class MyClass: + def method(self): + pass + @doc_controls.do_not_doc_inheritable - def my_method(self): + def hidden_method(self): pass - private = public_api.PublicAPIFilter._is_private( - self=None, + class SubClass(MyClass): + + def hidden_method(self): + 'still hidden' + + result = public_api.filter_doc_controls_skip( path=('a', 'b'), - parent=MyClass, - name='my_method', - obj=MyClass.my_method) - self.assertTrue(private) + parent=SubClass, + children=[('method', SubClass.method), + ('hidden_method', SubClass.hidden_method)]) + + self.assertEqual([('method', MyClass.method)], list(result)) + + def test_filter_all(self): + module = types.ModuleType('module') + module.__all__ = ['a'] + module.a = 1 + module.b = 2 + + result = public_api.filter_module_all(('module'), module, [('a', module.a), + ('b', module.b)]) + self.assertEqual([('a', module.a)], list(result)) + + def test_filter_base_dirs(self): + module = types.ModuleType('module') + module.__file__ = '/1/2/3/module' + module.a = 1 + module.sub1 = types.ModuleType('sub1') + module.sub1.__file__ = '/1/2/3/4/sub1' + module.sub2 = types.ModuleType('sub2') + module.sub2.__file__ = '/1/2/bad/sub2' + + my_filter = public_api.FilterBaseDirs(base_dirs=[pathlib.Path('/1/2/3/')]) + + result = my_filter( + path=('module',), + parent=module, + children=[('a', module.a), ('sub1', module.sub1), + ('sub2', module.sub2)]) + self.assertEqual([('a', module.a), ('sub1', module.sub1)], list(result)) if __name__ == '__main__': diff --git a/tools/tensorflow_docs/api_generator/test_module1.py b/tools/tensorflow_docs/api_generator/test_module1.py deleted file mode 100644 index 421f96ddd20..00000000000 --- a/tools/tensorflow_docs/api_generator/test_module1.py +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A module target for TraverseTest.test_module.""" -from tensorflow_docs.api_generator import test_module2 - - -class ModuleClass1(object): - - def __init__(self): - self._m2 = test_module2.ModuleClass2() - - def __model_class1_method__(self): - pass diff --git a/tools/tensorflow_docs/api_generator/test_module2.py b/tools/tensorflow_docs/api_generator/test_module2.py deleted file mode 100644 index 068fc17dcd9..00000000000 --- a/tools/tensorflow_docs/api_generator/test_module2.py +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2018 The TensorFlow Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================== -"""A module target for TraverseTest.test_module.""" - - -class ModuleClass2(object): - - def __init__(self): - pass - - def __model_class1_method__(self): - pass - - -class Hidden(object): - pass - - -__all__ = ['ModuleClass2'] diff --git a/tools/tensorflow_docs/api_generator/traverse.py b/tools/tensorflow_docs/api_generator/traverse.py index 8dabdb4d63f..5c461756e2b 100644 --- a/tools/tensorflow_docs/api_generator/traverse.py +++ b/tools/tensorflow_docs/api_generator/traverse.py @@ -17,7 +17,6 @@ import sys import logging -from google.protobuf.message import Message as ProtoMessage # To see the logs pass: --logger_levels=tensorflow_docs:DEBUG --alsologtostderr _LOGGER = logging.getLogger(__name__) @@ -25,123 +24,6 @@ __all__ = ['traverse'] -def _filter_module_all(path, root, children): - """Filters module children based on the "__all__" arrtibute. - - Args: - path: API to this symbol - root: The object - children: A list of (name, object) pairs. - - Returns: - `children` filtered to respect __all__ - """ - del path - if not (inspect.ismodule(root) and hasattr(root, '__all__')): - return children - module_all = set(root.__all__) - children = [(name, value) for (name, value) in children if name in module_all] - - return children - - -def _add_proto_fields(path, root, children): - """Add properties to Proto classes, so they can be documented. - - Warning: This inserts the Properties into the class so the rest of the system - is unaffected. This patching is acceptable because there is never a reason to - run other tensorflow code in the same process as the doc generator. - - Args: - path: API to this symbol - root: The object - children: A list of (name, object) pairs. - - Returns: - `children` with proto fields added as properties. - """ - del path - if not inspect.isclass(root) or not issubclass(root, ProtoMessage): - return children - - descriptor = getattr(root, 'DESCRIPTOR', None) - if descriptor is None: - return children - fields = descriptor.fields - if not fields: - return children - - field = fields[0] - # Make the dictionaries mapping from int types and labels to type and - # label names. - types = { - getattr(field, name): name - for name in dir(field) - if name.startswith('TYPE') - } - - labels = { - getattr(field, name): name - for name in dir(field) - if name.startswith('LABEL') - } - - field_properties = {} - - for field in fields: - name = field.name - doc_parts = [] - - label = labels[field.label].lower().replace('label_', '') - if label != 'optional': - doc_parts.append(label) - - type_name = types[field.type] - if type_name == 'TYPE_MESSAGE': - type_name = field.message_type.name - elif type_name == 'TYPE_ENUM': - type_name = field.enum_type.name - else: - type_name = type_name.lower().replace('type_', '') - - doc_parts.append(type_name) - doc_parts.append(name) - doc = '`{}`'.format(' '.join(doc_parts)) - prop = property(fget=lambda x: x, doc=doc) - field_properties[name] = prop - - for name, prop in field_properties.items(): - setattr(root, name, prop) - - children = dict(children) - children.update(field_properties) - children = sorted(children.items(), key=lambda item: item[0]) - - return children - - -def _filter_builtin_modules(path, root, children): - """Filters module children to remove builtin modules. - - Args: - path: API to this symbol - root: The object - children: A list of (name, object) pairs. - - Returns: - `children` with all builtin modules removed. - """ - del path - del root - # filter out 'builtin' modules - filtered_children = [] - for name, child in children: - # Do not descend into built-in modules - if inspect.ismodule(child) and child.__name__ in sys.builtin_module_names: - continue - filtered_children.append((name, child)) - return filtered_children - def _traverse_internal(root, visitors, stack, path): """Internal helper for traverse.""" @@ -220,9 +102,4 @@ def traverse(root, visitors, root_name): arguments, and returns a list of accepted children. root_name: The short-name of the root module. """ - base_visitors = [ - _filter_module_all, - _add_proto_fields, - _filter_builtin_modules - ] - _traverse_internal(root, base_visitors + visitors, [], (root_name,)) + _traverse_internal(root, visitors, [], (root_name,)) diff --git a/tools/tensorflow_docs/api_generator/traverse_test.py b/tools/tensorflow_docs/api_generator/traverse_test.py index 98ca7629678..4bf3cf9454e 100644 --- a/tools/tensorflow_docs/api_generator/traverse_test.py +++ b/tools/tensorflow_docs/api_generator/traverse_test.py @@ -15,8 +15,6 @@ """Tests for Python module traversal.""" from absl.testing import absltest -from tensorflow_docs.api_generator import test_module1 -from tensorflow_docs.api_generator import test_module2 from tensorflow_docs.api_generator import traverse @@ -43,16 +41,6 @@ class Cyclist(object): traverse.traverse(Cyclist, [visitor], root_name='root_name') # We simply want to make sure we terminate. - def test_module(self): - visitor = TestVisitor() - traverse.traverse(test_module1, [visitor], root_name='root_name') - - called = [parent for _, parent, _ in visitor.call_log] - - self.assertIn(test_module1.ModuleClass1, called) - self.assertIn(test_module2.ModuleClass2, called) - self.assertNotIn(test_module2.Hidden, called) - def test_class(self): visitor = TestVisitor() traverse.traverse(TestVisitor, [visitor], root_name='root_name') From 7aabefbf38515581e2998e1928ba95c981023577 Mon Sep 17 00:00:00 2001 From: Hun-soo Jung Date: Mon, 11 Apr 2022 20:33:49 +0900 Subject: [PATCH 063/872] Correct plot labels of AUPRC The AUPRC plot takes precision for x-axis and recall for y-axis. --- site/en/tutorials/structured_data/imbalanced_data.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/structured_data/imbalanced_data.ipynb b/site/en/tutorials/structured_data/imbalanced_data.ipynb index df9f33417cc..ffce3927e0b 100644 --- a/site/en/tutorials/structured_data/imbalanced_data.ipynb +++ b/site/en/tutorials/structured_data/imbalanced_data.ipynb @@ -986,8 +986,8 @@ " precision, recall, _ = sklearn.metrics.precision_recall_curve(labels, predictions)\r\n", "\r\n", " plt.plot(precision, recall, label=name, linewidth=2, **kwargs)\r\n", - " plt.xlabel('Recall')\r\n", - " plt.ylabel('Precision')\r\n", + " plt.xlabel('Precision')\r\n", + " plt.ylabel('Recall')\r\n", " plt.grid(True)\r\n", " ax = plt.gca()\r\n", " ax.set_aspect('equal')" From ac5d51c3e68cc07bbdb2989e2c7d9e04da5cd7f5 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Mon, 11 Apr 2022 23:50:53 +0100 Subject: [PATCH 064/872] Update doc links in Save and load models tutorial --- site/en/tutorials/keras/save_and_load.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/keras/save_and_load.ipynb b/site/en/tutorials/keras/save_and_load.ipynb index dd297c4a14f..47cc3af4d16 100644 --- a/site/en/tutorials/keras/save_and_load.ipynb +++ b/site/en/tutorials/keras/save_and_load.ipynb @@ -485,7 +485,7 @@ "id": "JtdYhvWnH2ib" }, "source": [ - "The above code stores the weights to a collection of [checkpoint](https://www.tensorflow.org/guide/checkpoint)-formatted files that contain only the trained weights in a binary format. Checkpoints contain:\n", + "The above code stores the weights to a collection of [checkpoint](../../guide/checkpoint.ipynb)-formatted files that contain only the trained weights in a binary format. Checkpoints contain:\n", "* One or more shards that contain your model's weights.\n", "* An index file that indicates which weights are stored in which shard.\n", "\n", @@ -500,7 +500,7 @@ "source": [ "## Manually save weights\n", "\n", - "Manually saving weights with the `Model.save_weights` method. By default, `tf.keras`—and `save_weights` in particular—uses the TensorFlow [checkpoint](../../guide/checkpoint.ipynb) format with a `.ckpt` extension (saving in [HDF5](https://www.tensorflow.org/guide/keras/save_and_serialize#hdf5_format) with a `.h5` extension is covered in the [Save and serialize models](https://www.tensorflow.org/guide/keras/save_and_serialize#weights-only_saving_in_savedmodel_format) guide):" + "To save weights manually, use `tf.keras.Model.save_weights`. By default, `tf.keras`—and the `Model.save_weights` method in particular—uses the TensorFlow [Checkpoint](../../guide/checkpoint.ipynb) format with a `.ckpt` extension. To save in the HDF5 format with a `.h5` extension, refer to the [Save and load models](https://www.tensorflow.org/guide/keras/save_and_serialize) guide." ] }, { From 9b8b87831877332ec197b2c9558ce3c07e526641 Mon Sep 17 00:00:00 2001 From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com> Date: Thu, 14 Apr 2022 11:43:07 +0530 Subject: [PATCH 065/872] Updated deprecated attributes with the new one Updated and replaced the deprecated attributes max_tokens with "num_tokens" and vocab_size with "vocabulary_size" to remove the warnings from output. --- site/en/tutorials/load_data/csv.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/load_data/csv.ipynb b/site/en/tutorials/load_data/csv.ipynb index dc13711fbec..6112ce0a393 100644 --- a/site/en/tutorials/load_data/csv.ipynb +++ b/site/en/tutorials/load_data/csv.ipynb @@ -554,7 +554,7 @@ " continue\n", " \n", " lookup = layers.StringLookup(vocabulary=np.unique(titanic_features[name]))\n", - " one_hot = layers.CategoryEncoding(max_tokens=lookup.vocab_size())\n", + " one_hot = layers.CategoryEncoding(num_tokens=lookup.vocabulary_size())\n", "\n", " x = lookup(input)\n", " x = one_hot(x)\n", From 0c4f1bb9408e6a39bfbb60cd6a6c552a9ccec0c5 Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Mon, 18 Apr 2022 16:05:44 -0700 Subject: [PATCH 066/872] Clarify some wording in the tf.Tensor guide. PiperOrigin-RevId: 442656079 --- site/en/guide/tensor.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/tensor.ipynb b/site/en/guide/tensor.ipynb index ac27fb83ab8..b24ee3efef9 100644 --- a/site/en/guide/tensor.ipynb +++ b/site/en/guide/tensor.ipynb @@ -393,7 +393,7 @@ "* **Shape**: The length (number of elements) of each of the axes of a tensor.\n", "* **Rank**: Number of tensor axes. A scalar has rank 0, a vector has rank 1, a matrix is rank 2.\n", "* **Axis** or **Dimension**: A particular dimension of a tensor.\n", - "* **Size**: The total number of items in the tensor, the product shape vector.\n" + "* **Size**: The total number of items in the tensor, the product of the shape vector's elements.\n" ] }, { From 6b36a12896791ee5ae874cf1b04992d576fb568d Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Tue, 19 Apr 2022 02:31:38 -0700 Subject: [PATCH 067/872] Don't duplicate the keras path (or any path). PiperOrigin-RevId: 442754247 --- tools/tensorflow_docs/api_generator/doc_generator_visitor.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index b063fb9221c..48efc97f893 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -131,6 +131,8 @@ def __setitem__(self, path: ApiPath, obj: Any): path: A tuple of strings. obj: The python object. """ + assert path not in self + parent_path = path[:-1] parent = self[parent_path] From 300d9f5b3395076a3385e246576466bae3f4e40d Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Tue, 19 Apr 2022 04:44:14 -0700 Subject: [PATCH 068/872] Protect against inconsistent path selection. PiperOrigin-RevId: 442778183 --- .../api_generator/doc_generator_visitor.py | 31 ++++++++-- .../doc_generator_visitor_test.py | 60 +++++++++++++++---- 2 files changed, 73 insertions(+), 18 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index 48efc97f893..8beeb3eefca 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -19,7 +19,7 @@ import enum import inspect -from typing import Any, Dict, List, Optional, NamedTuple, Tuple +from typing import Any, Dict, List, Optional, NamedTuple, Sequence, Tuple from tensorflow_docs.api_generator import obj_type as obj_type_lib @@ -584,7 +584,7 @@ def from_path_tree(cls, path_tree: PathTree, score_name_fn) -> 'ApiTree': parents = [node.parent for node in duplicate_nodes] - # Choose the master name with a lexical sort on the tuples returned by + # Choose the priority name with a lexical sort on the tuples returned by # by _score_name. if not all(parent.path in self for parent in parents): # rewind @@ -596,13 +596,34 @@ def from_path_tree(cls, path_tree: PathTree, score_name_fn) -> 'ApiTree': active_nodes.appendleft(parent) continue # If we've made it here, the immediate parents of each of the paths have - # been processed, so now we can choose its master name. + # been processed, so now we can choose its priority name. aliases = [node.path for node in duplicate_nodes] - master_path = min(aliases, key=score_name_fn) + priority_path = self._choose_priority_path(aliases, score_name_fn) - self.insert(master_path, current_node.py_object, aliases) + if priority_path is None: + # How did this happen? + # No parents in the public api -> you are not in the public API. + continue + + self.insert(priority_path, current_node.py_object, aliases) active_nodes.extend(current_node.children.values()) return self + + def _choose_priority_path(self, aliases: Sequence[ApiPath], + score_name_fn) -> Optional[ApiPath]: + # Only consider a path an option for the priority_path if its parent-path + # is the priority_path for that object. + priority_path_options = [] + for alias in aliases: + parent_path = alias[:-1] + + if self[parent_path].path == parent_path: + priority_path_options.append(alias) + + try: + return min(priority_path_options, key=score_name_fn) + except ValueError: + return None diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py index c6900ec77a8..460fefa352d 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py @@ -14,7 +14,6 @@ # ============================================================================== """Tests for tools.docs.doc_generator_visitor.""" -import argparse import inspect import io import os @@ -262,8 +261,8 @@ class Parent(object): class PathTreeTest(absltest.TestCase): def test_contains(self): - tf = argparse.Namespace() - tf.sub = argparse.Namespace() + tf = types.ModuleType('tf') + tf.sub = types.ModuleType('sub') tree = doc_generator_visitor.PathTree() tree[('tf',)] = tf @@ -273,8 +272,8 @@ def test_contains(self): self.assertIn(('tf', 'sub'), tree) def test_node_insertion(self): - tf = argparse.Namespace() - tf.sub = argparse.Namespace() + tf = types.ModuleType('tf') + tf.sub = types.ModuleType('sub') tf.sub.object = object() tree = doc_generator_visitor.PathTree() @@ -290,10 +289,10 @@ def test_node_insertion(self): self.assertIs(node.children['thing'], tree[('tf', 'sub', 'thing')]) def test_duplicate(self): - tf = argparse.Namespace() - tf.sub = argparse.Namespace() + tf = types.ModuleType('tf') + tf.sub = types.ModuleType('sub') tf.sub.thing = object() - tf.sub2 = argparse.Namespace() + tf.sub2 = types.ModuleType('sub2') tf.sub2.thing = tf.sub.thing tree = doc_generator_visitor.PathTree() @@ -308,10 +307,10 @@ def test_duplicate(self): [tree[('tf', 'sub', 'thing')], tree[('tf', 'sub2', 'thing')]]) def test_duplicate_singleton(self): - tf = argparse.Namespace() - tf.sub = argparse.Namespace() + tf = types.ModuleType('tf') + tf.sub = types.ModuleType('sub') tf.sub.thing = 999 - tf.sub2 = argparse.Namespace() + tf.sub2 = types.ModuleType('sub2') tf.sub2.thing = tf.sub.thing tree = doc_generator_visitor.PathTree() @@ -322,8 +321,7 @@ def test_duplicate_singleton(self): tree[('tf', 'sub2', 'thing')] = tf.sub2.thing found = tree.nodes_for_obj(tf.sub.thing) - self.assertIsNotNone(found) - self.assertEmpty(found) + self.assertEqual([], found) class ApiTreeTest(absltest.TestCase): @@ -469,6 +467,42 @@ def test_api_tree_toc_integration(self): self.assertEqual(expected, stream.getvalue()) + def test_non_priority_name(self): + + class Class1: + pass + + mod = types.ModuleType('mod') + mod.a = types.ModuleType('sub') + mod.a.Class1 = Class1 + mod.b = mod.a + + path_tree = doc_generator_visitor.PathTree() + path_tree[('mod',)] = mod + path_tree[('mod', 'a')] = mod.a + path_tree[('mod', 'a', 'Class1')] = mod.a.Class1 + path_tree[('mod', 'b')] = mod.b + path_tree[('mod', 'b', 'Class1')] = mod.b.Class1 + + def inconsistent_name_score(path): + # `mod.a` is prefered over `mod.b`, but `b.Class1` is prefered over + # `a.Class1`! + scores = { + ('mod',): 0, + ('mod', 'a'): 0, # prefer 'a' + ('mod', 'b'): 1, + ('mod', 'a', 'Class1'): 1, + ('mod', 'b', 'Class1'): 0, # prefer 'b.Class1' + } + return scores[path] + + api_tree = doc_generator_visitor.ApiTree.from_path_tree( + path_tree, inconsistent_name_score) + node = api_tree.node_for_object(Class1) + + # `Class1` can't choose `b.Class1` as its priority_path because + # `a` is the priority_path for `sub`. + self.assertEqual('mod.a.Class1', node.full_name) if __name__ == '__main__': absltest.main() From a0378e4981a33c48fb27a1085d86a8c4814bd37f Mon Sep 17 00:00:00 2001 From: Malcolm Slaney Date: Tue, 19 Apr 2022 05:03:33 -0700 Subject: [PATCH 069/872] Update distributed_training.ipynb --- site/en/guide/distributed_training.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb index 129e3f155cf..f9e2c98ceeb 100644 --- a/site/en/guide/distributed_training.ipynb +++ b/site/en/guide/distributed_training.ipynb @@ -508,7 +508,7 @@ "Here's what you need to change in your code:\n", "\n", "1. Create an instance of the appropriate `tf.distribute.Strategy`.\n", - "2. Move the creation of Keras model, optimizer and metrics inside `strategy.scope`. Thus the code in the model's `run()`, `train_step()`, and `test_step()` methods will all be distributed and executed on the accelerator(s).\n", + "2. Move the creation of Keras model, optimizer and metrics inside `strategy.scope`. Thus the code in the model's `call()`, `train_step()`, and `test_step()` methods will all be distributed and executed on the accelerator(s).\n", "\n", "TensorFlow distribution strategies support all types of Keras models—[Sequential](/keras/sequential_model.ipynb), [Functional](/keras/functional.ipynb), and [subclassed](/keras/custom_layers_and_models.ipynb).\n", "\n", From 1489ce0c9b8b72a6559e59238c41c315e0529ed4 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Tue, 19 Apr 2022 05:05:47 -0700 Subject: [PATCH 070/872] Protect against traverse returning different sets of children. Fixes some strange behavior around classmethods. Cache children for each object. If you see the same object, return the same set of children. PiperOrigin-RevId: 442781685 --- .../doc_generator_visitor_test.py | 47 ++++++- .../api_generator/generate_lib.py | 27 ++-- .../tensorflow_docs/api_generator/traverse.py | 121 +++++++++++------- .../api_generator/traverse_test.py | 6 +- 4 files changed, 136 insertions(+), 65 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py index 460fefa352d..b8f2c872146 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py @@ -14,7 +14,6 @@ # ============================================================================== """Tests for tools.docs.doc_generator_visitor.""" -import inspect import io import os import textwrap @@ -257,6 +256,52 @@ class Parent(object): id(Parent.obj1): 'tf.submodule.Parent.obj1', }, visitor.reverse_index) + def test_handles_duplicate_classmethods(self): + + class MyClass: + + @classmethod + def from_value(cls, value): + pass + + tf = types.ModuleType('fake_tf') + tf.__file__ = '/tmp/tf/__init__.py' + tf.MyClass = MyClass + tf.sub = types.ModuleType('sub') + tf.sub.MyClass = MyClass + + visitor = generate_lib.extract([('tf', tf)], + base_dir=os.path.dirname(tf.__file__), + private_map={}, + visitor_cls=NoDunderVisitor) + + paths = ['.'.join(p) for p in visitor.path_tree.keys()] + + expected = [ + '', + 'tf', + 'tf.MyClass', + 'tf.MyClass.from_value', + 'tf.sub', + 'tf.sub.MyClass', + 'tf.sub.MyClass.from_value', + ] + self.assertCountEqual(expected, paths) + + apis = [node.full_name for node in visitor.api_tree.iter_nodes()] + expected = [ + 'tf', + 'tf.sub', + 'tf.sub.MyClass', + 'tf.sub.MyClass.from_value', + ] + self.assertCountEqual(expected, apis) + + self.assertIs(visitor.api_tree[('tf', 'MyClass')], + visitor.api_tree[('tf', 'sub', 'MyClass')]) + self.assertIs(visitor.api_tree[('tf', 'MyClass', 'from_value')], + visitor.api_tree[('tf', 'sub', 'MyClass', 'from_value')]) + class PathTreeTest(absltest.TestCase): diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index e378bb2f9c6..737bccf190f 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -211,12 +211,15 @@ def add_dict_to_dict(add_from, add_to): add_to[key] = add_from[key] -def extract(py_modules, - base_dir, - private_map, - visitor_cls=doc_generator_visitor.DocGeneratorVisitor, - callbacks=None, - include_default_callbacks=True): +def extract( + py_modules, + base_dir, + private_map: Dict[str, Any], + visitor_cls: Type[ + doc_generator_visitor.DocGeneratorVisitor] = doc_generator_visitor + .DocGeneratorVisitor, + callbacks: Optional[public_api.ApiFilter] = None, + include_default_callbacks=True): """Walks the module contents, returns an index of all visited objects. The return value is an instance of `self._visitor_cls`, usually: @@ -250,12 +253,11 @@ def extract(py_modules, raise ValueError("only pass one [('name',module)] pair in py_modules") short_name, py_module = py_modules[0] - # The objects found during traversal, and their children are passed to each - # of these visitors in sequence. Each visitor returns the list of children + # of these filters in sequence. Each visitor returns the list of children # to be passed to the next visitor. if include_default_callbacks: - visitors = [ + filters = [ # filter the api. public_api.FailIfNestedTooDeep(10), public_api.filter_module_all, @@ -268,12 +270,11 @@ def extract(py_modules, public_api.ignore_typing ] else: - visitors = [] + filters = [] accumulator = visitor_cls() - visitors = visitors + callbacks + [accumulator] - - traverse.traverse(py_module, visitors, short_name) + traverse.traverse( + py_module, filters + callbacks, accumulator, root_name=short_name) accumulator.build() return accumulator diff --git a/tools/tensorflow_docs/api_generator/traverse.py b/tools/tensorflow_docs/api_generator/traverse.py index 5c461756e2b..e472f9291cd 100644 --- a/tools/tensorflow_docs/api_generator/traverse.py +++ b/tools/tensorflow_docs/api_generator/traverse.py @@ -14,60 +14,84 @@ # ============================================================================== """Traversing Python modules and classes.""" import inspect -import sys - import logging -# To see the logs pass: --logger_levels=tensorflow_docs:DEBUG --alsologtostderr -_LOGGER = logging.getLogger(__name__) - -__all__ = ['traverse'] - - +from typing import Any, Dict, List, Sequence, Tuple -def _traverse_internal(root, visitors, stack, path): - """Internal helper for traverse.""" - new_stack = stack + [root] +from tensorflow_docs.api_generator import doc_generator_visitor +from tensorflow_docs.api_generator import public_api - # Only traverse modules and classes - if not inspect.isclass(root) and not inspect.ismodule(root): - return - try: - children = inspect.getmembers(root) - except ImportError: - # On some Python installations, some modules do not support enumerating - # members (six in particular), leading to import errors. - children = [] - - # Break cycles. - filtered_children = [] - for name, child in children: - if any(child is item for item in new_stack): # `in`, but using `is` - continue - filtered_children.append((name, child)) - children = filtered_children - - _LOGGER.debug('path: %s', path) - _LOGGER.debug('children: %s', [n for n, c in children]) - # Apply all callbacks, allowing each to filter the children - for visitor in visitors: - old_names = [n for n, c in children] - children = visitor(path, root, children) - children = list(children) - new_names = [n for n, c in children] - - if old_names != new_names: - _LOGGER.debug('filter: %s', visitor) - _LOGGER.debug('children: %s', new_names) +# To see the logs pass: --logger_levels=tensorflow_docs:DEBUG --alsologtostderr +_LOGGER = logging.getLogger(__name__) - for name, child in children: - # Break cycles - child_path = path + (name,) - _traverse_internal(child, visitors, new_stack, child_path) +__all__ = ['traverse'] -def traverse(root, visitors, root_name): +class _Traverser: + """Crawls the public API.""" + + def __init__(self, filters: Sequence[public_api.ApiFilter], + accumulator: doc_generator_visitor.DocGeneratorVisitor): + self.filters = list(filters) + self.accumulator = accumulator + self.children_cache: Dict[int, List[Tuple[str, Any]]] = {} + + def traverse(self, root, stack, path): + """Execute the traversal.""" + new_stack = stack + [root] + + # Only traverse modules and classes + if not inspect.isclass(root) and not inspect.ismodule(root): + return + + _LOGGER.debug('path: %s', path) + children = self.children_cache.get(id(root), None) + if children is None: + children = self.get_children(root, new_stack, path) + self.children_cache[id(root)] = children + else: + _LOGGER.debug(' children (cached): %s', [n for n, c in children]) + + self.accumulator(path, root, children) + + for name, child in children: + child_path = path + (name,) + self.traverse(child, new_stack, child_path) + + def get_children(self, root, new_stack, path) -> public_api.Children: + """Return the children for an object.""" + try: + children = inspect.getmembers(root) + except ImportError: + # On some Python installations, some modules do not support enumerating + # members (six in particular), leading to import errors. + children = [] + + # Break cycles. + filtered_children = [] + for name, child in children: + if any(child is item for item in new_stack): # `in`, but using `is` + continue + filtered_children.append((name, child)) + children = filtered_children + + _LOGGER.debug(' children: %s', [n for n, c in children]) + # Apply all callbacks, allowing each to filter the children + for fil in self.filters: + old_names = [n for n, c in children] + children = fil(path, root, children) + children = list(children) + new_names = [n for n, c in children] + + if old_names != new_names: + _LOGGER.debug(' filter: %s', filter) + _LOGGER.debug(' children: %s', new_names) + + return children + + +def traverse(root, filters, accumulator, root_name) -> None: """Recursively enumerate all members of `root`. Similar to the Python library function `os.path.walk`. @@ -98,8 +122,9 @@ def traverse(root, visitors, root_name): Args: root: A python object with which to start the traversal. - visitors: A list of callables. Each taking `(path, parent, children)` as + filters: A list of callables. Each taking `(path, parent, children)` as arguments, and returns a list of accepted children. + accumulator: a DocGenerator to accumulate the results. root_name: The short-name of the root module. """ - _traverse_internal(root, visitors, [], (root_name,)) + _Traverser(filters, accumulator).traverse(root, [], (root_name,)) diff --git a/tools/tensorflow_docs/api_generator/traverse_test.py b/tools/tensorflow_docs/api_generator/traverse_test.py index 4bf3cf9454e..7ea780b10f8 100644 --- a/tools/tensorflow_docs/api_generator/traverse_test.py +++ b/tools/tensorflow_docs/api_generator/traverse_test.py @@ -38,12 +38,12 @@ class Cyclist(object): Cyclist.cycle = Cyclist visitor = TestVisitor() - traverse.traverse(Cyclist, [visitor], root_name='root_name') + traverse.traverse(Cyclist, [], visitor, root_name='root_name') # We simply want to make sure we terminate. def test_class(self): visitor = TestVisitor() - traverse.traverse(TestVisitor, [visitor], root_name='root_name') + traverse.traverse(TestVisitor, [], visitor, root_name='root_name') self.assertEqual(TestVisitor, visitor.call_log[0][1]) # There are a bunch of other members, but make sure that the ones we know @@ -58,7 +58,7 @@ def test_class(self): def test_non_class(self): integer = 5 visitor = TestVisitor() - traverse.traverse(integer, [visitor], root_name='root_name') + traverse.traverse(integer, [], visitor, root_name='root_name') self.assertEqual([], visitor.call_log) From 8299f810249be52dc7da2157a67206d1685fa9bd Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Tue, 19 Apr 2022 05:13:42 -0700 Subject: [PATCH 071/872] Try harder to collect duplicated singletons. PiperOrigin-RevId: 442782877 --- .../api_generator/doc_generator_visitor.py | 21 +++++++-- .../doc_generator_visitor_test.py | 47 +++++++++++++++++++ .../api_generator/parser_test.py | 39 ++++++++------- 3 files changed, 84 insertions(+), 23 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index 8beeb3eefca..a835ad7a836 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -18,12 +18,17 @@ import dataclasses import enum import inspect +import logging + from typing import Any, Dict, List, Optional, NamedTuple, Sequence, Tuple from tensorflow_docs.api_generator import obj_type as obj_type_lib +# To see the logs pass: --logger_levels=tensorflow_docs:DEBUG --alsologtostderr +_LOGGER = logging.getLogger(__name__) + ApiPath = Tuple[str, ...] @@ -537,6 +542,10 @@ def __setitem__(self, *args, **kwargs): def insert(self, path: ApiPath, py_object: Any, aliases: List[ApiPath]): """Add an object to the index.""" + _LOGGER.debug('ApiTree.insert') + _LOGGER.debug(' path: %s', path) + _LOGGER.debug(' py_object: %s', py_object) + _LOGGER.debug(' aliases: %s', aliases) assert path not in self, 'A path was inserted twice.' parent_path = path[:-1] @@ -577,10 +586,16 @@ def from_path_tree(cls, path_tree: PathTree, score_name_fn) -> 'ApiTree': continue duplicate_nodes = set( - # Singelton objects will return []. path_tree.nodes_for_obj(current_node.py_object)) - # Add the current node in case it's a singelton. - duplicate_nodes.add(current_node) + + if not duplicate_nodes: + # Singleton objects will return `[]`. So look up the parent object's + # duplicate nodes and collect their children. + parent_nodes = path_tree.nodes_for_obj(current_node.parent.py_object) + duplicate_nodes = [ + parent_node.children[current_node.short_name] + for parent_node in parent_nodes + ] parents = [node.parent for node in duplicate_nodes] diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py index b8f2c872146..510e9cec89e 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py @@ -303,6 +303,52 @@ def from_value(cls, value): visitor.api_tree[('tf', 'sub', 'MyClass', 'from_value')]) + def test_handles_duplicate_singleton_attributes(self): + + class MyClass: + simple = 1 + + tf = types.ModuleType('fake_tf') + tf.__file__ = '/tmp/tf/__init__.py' + tf.MyClass = MyClass + tf.sub = types.ModuleType('sub') + tf.sub.MyClass = MyClass + + visitor = generate_lib.extract([('tf', tf)], + base_dir=os.path.dirname(tf.__file__), + private_map={}, + visitor_cls=NoDunderVisitor) + + paths = ['.'.join(p) for p in visitor.path_tree.keys()] + + expected = [ + '', + 'tf', + 'tf.MyClass', + 'tf.MyClass.simple', + 'tf.sub', + 'tf.sub.MyClass', + 'tf.sub.MyClass.simple', + ] + self.assertCountEqual(expected, paths) + + apis = ['.'.join(p) for p in visitor.api_tree.keys()] + expected = [ + '', + 'tf', + 'tf.MyClass', + 'tf.MyClass.simple', + 'tf.sub', + 'tf.sub.MyClass', + 'tf.sub.MyClass.simple', + ] + self.assertCountEqual(expected, apis) + + self.assertIs(visitor.api_tree[('tf', 'MyClass')], + visitor.api_tree[('tf', 'sub', 'MyClass')]) + self.assertIs(visitor.api_tree[('tf', 'MyClass', 'simple')], + visitor.api_tree[('tf', 'sub', 'MyClass', 'simple')]) + class PathTreeTest(absltest.TestCase): def test_contains(self): @@ -369,6 +415,7 @@ def test_duplicate_singleton(self): self.assertEqual([], found) + class ApiTreeTest(absltest.TestCase): def _make_fake_module(self) -> types.ModuleType: diff --git a/tools/tensorflow_docs/api_generator/parser_test.py b/tools/tensorflow_docs/api_generator/parser_test.py index 0e11291f22b..4322aadb76f 100644 --- a/tools/tensorflow_docs/api_generator/parser_test.py +++ b/tools/tensorflow_docs/api_generator/parser_test.py @@ -95,15 +95,8 @@ def class_method(cls): CLASS_MEMBER = 'a class member' -class DummyVisitor(object): - - def __init__(self, index, duplicate_of): - self.index = index - self.duplicate_of = duplicate_of - - class ConcreteMutableMapping(collections.MutableMapping): - """MutableMapping subclass to repro inspect.getsource() IndexError.""" + """MutableMapping subclass to repro getsource() IndexError.""" def __init__(self): self._map = {} @@ -159,24 +152,30 @@ class HasOneMember(object): def foo(self): pass + class Other: + pass + + tf = types.ModuleType('tf') + tf.__file__ = __file__ + tf.reference = HasOneMember + tf.third = Other + tf.fourth = Other + string = ('A `@tf.reference`, a member `tf.reference.foo`, and a ' '`tf.third(what)`. ' 'This is `not a symbol`, and this is `tf.not.a.real.symbol`') - duplicate_of = {'tf.third': 'tf.fourth'} - index = { - 'tf.reference': HasOneMember, - 'tf.reference.foo': HasOneMember.foo, - 'tf.third': HasOneMember, - 'tf.fourth': HasOneMember - } + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('tf', tf)], + code_url_prefix='https://tensorflow.org') - visitor = DummyVisitor(index, duplicate_of) + parser_config = generator.run_extraction() - reference_resolver = reference_resolver_lib.ReferenceResolver.from_visitor( - visitor=visitor, py_module_names=['tf'], link_prefix='../..') + result = ( + parser_config.reference_resolver.with_prefix( + '../..').replace_references(string)) - result = reference_resolver.replace_references(string) self.assertEqual( 'A ' '@tf.reference, ' @@ -580,7 +579,7 @@ def test_generate_index(self): def test_getsource_indexerror_resilience(self): """Validates that parser gracefully handles IndexErrors. - inspect.getsource() can raise an IndexError in some cases. It's unclear + getsource() can raise an IndexError in some cases. It's unclear why this happens, but it consistently repros on the `get` method of collections.MutableMapping subclasses. """ From 643f895c86496121aa2bfa75bf2044774e57a1e0 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Tue, 19 Apr 2022 18:50:34 -0700 Subject: [PATCH 072/872] Fix output toc and report order. PiperOrigin-RevId: 442966981 --- .../tensorflow_docs/api_generator/generate_lib.py | 14 +++++++------- .../tensorflow_docs/api_generator/report/utils.py | 9 +++++++++ tools/tensorflow_docs/api_generator/toc.py | 15 +++++++++++---- 3 files changed, 27 insertions(+), 11 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index 737bccf190f..de6ddcbb76d 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -114,8 +114,9 @@ def write_docs( # Collect redirects for an api _redirects.yaml file. redirects = [] + api_report = None if gen_report: - api_report_obj = utils.ApiReport() + api_report = utils.ApiReport() # Parse and write Markdown pages, resolving cross-links (`tf.symbol`). num_docs_output = 0 @@ -136,10 +137,10 @@ def write_docs( search_hints=search_hints, page_builder_classes=page_builder_classes) - if gen_report and not full_name.startswith( + if api_report is not None and not full_name.startswith( ('tf.compat.v', 'tf.keras.backend', 'tf.numpy', 'tf.experimental.numpy')): - api_report_obj.fill_metrics(page_info) + api_report.fill_metrics(page_info) except Exception as e: raise ValueError( f'Failed to generate docs for symbol: `{full_name}`') from e @@ -166,10 +167,9 @@ def write_docs( to_path = site_path / full_name.replace('.', '/') redirects.append({'from': str(from_path), 'to': str(to_path)}) - if gen_report: - serialized_proto = api_report_obj.api_report.SerializeToString() - raw_proto = output_dir / root_module_name / 'api_report.pb' - raw_proto.write_bytes(serialized_proto) + if api_report is not None: + api_report.write(output_dir / root_module_name / 'api_report.pb') + if num_docs_output <= 1: raise ValueError('The `DocGenerator` failed to generate any docs. Verify ' diff --git a/tools/tensorflow_docs/api_generator/report/utils.py b/tools/tensorflow_docs/api_generator/report/utils.py index 97063b3be2e..3166cd994f4 100644 --- a/tools/tensorflow_docs/api_generator/report/utils.py +++ b/tools/tensorflow_docs/api_generator/report/utils.py @@ -35,6 +35,15 @@ def __init__(self): self.api_report.timestamp.CopyFrom(invocation_timestamp) self.api_report.date = invocation_timestamp.ToJsonString() + def write(self, path): + api_report = api_report_pb2.ApiReport( + timestamp=self.api_report.timestamp, + date=self.api_report.date, + symbol_metric=sorted( + self.api_report.symbol_metric, key=lambda sm: sm.symbol_name)) + + path.write_bytes(api_report.SerializeToString()) + def _lint( self, *, diff --git a/tools/tensorflow_docs/api_generator/toc.py b/tools/tensorflow_docs/api_generator/toc.py index 65572bfabba..1e72bcda75c 100644 --- a/tools/tensorflow_docs/api_generator/toc.py +++ b/tools/tensorflow_docs/api_generator/toc.py @@ -196,11 +196,15 @@ def _entries_from_api_node( else: return [] + def _get_docpath(self, api_path) -> pathlib.Path: + api_path = (p.replace('.', '/') for p in api_path) + return pathlib.Path(self.site_path, *api_path) + def _make_link(self, api_node: doc_generator_visitor.ApiTreeNode, title: Optional[str] = None) -> Link: - docpath = pathlib.Path(self.site_path, *api_node.path) + docpath = self._get_docpath(api_path=api_node.path) title = title or api_node.short_name return Link( title=title, path=str(docpath), status=self._make_status(api_node)) @@ -221,7 +225,7 @@ def _make_section(self, title=title or api_node.short_name, section=entries, status=status) def _make_overview(self, api_node: doc_generator_visitor.ApiTreeNode): - docpath = pathlib.Path(self.site_path, *api_node.path) + docpath = self._get_docpath(api_path=api_node.path) return Link(title='Overview', path=str(docpath)) def _section_order_key(self, entry: Entry) -> Tuple[bool, str]: @@ -311,8 +315,11 @@ class FlatModulesTocBuilder(TocBuilder): def build(self, api_tree: doc_generator_visitor.ApiTree) -> Toc: entries = [] for module_node in api_tree.root.children.values(): - assert module_node.obj_type is obj_type_lib.ObjType.MODULE - entries.extend(self._flat_module_entries(module_node)) + if '.' in module_node.short_name: + entries.extend(self._entries_from_api_node(module_node)) + else: + assert module_node.obj_type is obj_type_lib.ObjType.MODULE + entries.extend(self._flat_module_entries(module_node)) return Toc(toc=entries) From e118d1dce7a3293ea9411e64ead5f0f29068d92e Mon Sep 17 00:00:00 2001 From: "A. Unique TensorFlower" Date: Thu, 21 Apr 2022 07:30:54 -0700 Subject: [PATCH 073/872] Updated the paths for Tensorflow Python wheels for macOS Fixes : https://github.com/tensorflow/tensorflow/issues/55581 PiperOrigin-RevId: 443372970 --- site/en/install/pip.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/site/en/install/pip.html b/site/en/install/pip.html index 9a7265725a9..66a8874d9f1 100644 --- a/site/en/install/pip.html +++ b/site/en/install/pip.html @@ -354,19 +354,19 @@

Package location

macOS (CPU-only) Python 3.7 - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp37-cp37m-macosx_10_11_x86_64.whl + https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp37-cp37m-macosx_10_14_x86_64.whl Python 3.8 - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp38-cp38-macosx_10_11_x86_64.whl + https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp38-cp38-macosx_10_14_x86_64.whl Python 3.9 - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp39-cp39-macosx_10_11_x86_64.whl + https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp39-cp39-macosx_10_14_x86_64.whl Python 3.10 - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp310-cp310-macosx_10_11_x86_64.whl + https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp310-cp310-macosx_10_14_x86_64.whl Windows From 794ba098a21fae3cab82ee2ab91bed602766f9e8 Mon Sep 17 00:00:00 2001 From: Olzhas Akpambetov Date: Thu, 21 Apr 2022 10:26:59 -0700 Subject: [PATCH 074/872] Add a Caution to docs that load models PiperOrigin-RevId: 443414328 --- site/en/guide/saved_model.ipynb | 5 ++++- site/en/tutorials/distribute/save_and_load.ipynb | 6 +++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/site/en/guide/saved_model.ipynb b/site/en/guide/saved_model.ipynb index 2e5351b49c3..068d6d91de9 100644 --- a/site/en/guide/saved_model.ipynb +++ b/site/en/guide/saved_model.ipynb @@ -75,7 +75,10 @@ " - Save: `tf.saved_model.save(model, path_to_dir)`\n", " - Load: `model = tf.saved_model.load(path_to_dir)`\n", "- High-level `tf.keras.Model` API. Refer to [the keras save and serialize guide](https://www.tensorflow.org/guide/keras/save_and_serialize).\n", - "- If you just want to save/load weights during training, refer to [the checkpoints guide](./checkpoint.ipynb).\n" + "- If you just want to save/load weights during training, refer to [the checkpoints guide](./checkpoint.ipynb).\n", + "\n", + "Caution: TensorFlow models are code and it is important to be careful with untrusted code. Learn more in [Using TensorFlow securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md).\n", + "\n" ] }, { diff --git a/site/en/tutorials/distribute/save_and_load.ipynb b/site/en/tutorials/distribute/save_and_load.ipynb index 79e09b961b6..517c43e5969 100644 --- a/site/en/tutorials/distribute/save_and_load.ipynb +++ b/site/en/tutorials/distribute/save_and_load.ipynb @@ -73,7 +73,11 @@ "\n", "This tutorial demonstrates how you can save and load models in a SavedModel format with `tf.distribute.Strategy` during or after training. There are two kinds of APIs for saving and loading a Keras model: high-level (`tf.keras.Model.save` and `tf.keras.models.load_model`) and low-level (`tf.saved_model.save` and `tf.saved_model.load`).\n", "\n", - "To learn about SavedModel and serialization in general, please read the [saved model guide](../../guide/saved_model.ipynb), and the [Keras model serialization guide](https://www.tensorflow.org/guide/keras/save_and_serialize). Let's start with a simple example: " + "To learn about SavedModel and serialization in general, please read the [saved model guide](../../guide/saved_model.ipynb), and the [Keras model serialization guide](https://www.tensorflow.org/guide/keras/save_and_serialize). Let's start with a simple example.\n", + "\n", + "Caution: TensorFlow models are code and it is important to be careful with untrusted code. Learn more in [Using TensorFlow securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md).\n", + "\n" + ] }, { From 07d7aae845800adf200fbc6035efa582301e71ff Mon Sep 17 00:00:00 2001 From: tilakrayal <81610181+tilakrayal@users.noreply.github.com> Date: Fri, 22 Apr 2022 16:24:58 +0530 Subject: [PATCH 075/872] Updated the naming from 'tf.argmax' to 'tf.math.argmax' --- site/en/guide/migrate/metrics_optimizers.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/guide/migrate/metrics_optimizers.ipynb b/site/en/guide/migrate/metrics_optimizers.ipynb index dea0d11dc3a..b19d9f020b6 100644 --- a/site/en/guide/migrate/metrics_optimizers.ipynb +++ b/site/en/guide/migrate/metrics_optimizers.ipynb @@ -144,7 +144,7 @@ "\n", "def _model_fn(features, labels, mode):\n", " logits = tf1.layers.Dense(2)(features)\n", - " predictions = tf.argmax(input=logits, axis=1)\n", + " predictions = tf.math.argmax(input=logits, axis=1)\n", " loss = tf1.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits)\n", " optimizer = tf1.train.AdagradOptimizer(0.05)\n", " train_op = optimizer.minimize(loss, global_step=tf1.train.get_global_step())\n", @@ -223,7 +223,7 @@ "\n", "inputs = tf.keras.Input((2,))\n", "logits = tf.keras.layers.Dense(2)(inputs)\n", - "predictions = tf.argmax(input=logits, axis=1)\n", + "predictions = tf.math.argmax(input=logits, axis=1)\n", "model = tf.keras.models.Model(inputs, predictions)\n", "optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.05)\n", "\n", From f4b9a61f81e7808a3a25e0b594bf0ec566168b0e Mon Sep 17 00:00:00 2001 From: tfdocsbot Date: Fri, 22 Apr 2022 10:57:35 +0000 Subject: [PATCH 076/872] nbfmt --- site/en/guide/migrate/metrics_optimizers.ipynb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/site/en/guide/migrate/metrics_optimizers.ipynb b/site/en/guide/migrate/metrics_optimizers.ipynb index b19d9f020b6..61afb35aea6 100644 --- a/site/en/guide/migrate/metrics_optimizers.ipynb +++ b/site/en/guide/migrate/metrics_optimizers.ipynb @@ -370,8 +370,7 @@ "metadata": { "colab": { "collapsed_sections": [], - "name": "metrics.ipynb", - "provenance": [], + "name": "metrics_optimizers.ipynb", "toc_visible": true }, "kernelspec": { From 3152503f704befb728a7acbc2ae01748d0281d9e Mon Sep 17 00:00:00 2001 From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com> Date: Sat, 23 Apr 2022 22:22:56 +0530 Subject: [PATCH 077/872] Updated tf.keras.models.Sequential name with tf.keras.Sequential As now the Sequential API is directly inherited from Keras, So I have modified the "tf.keras.models.Sequential" API name with the new updated name as "tf.keras.Sequential" and ran the entire code successfully after modification. --- site/en/tutorials/keras/save_and_load.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/keras/save_and_load.ipynb b/site/en/tutorials/keras/save_and_load.ipynb index 47cc3af4d16..4d7ea074161 100644 --- a/site/en/tutorials/keras/save_and_load.ipynb +++ b/site/en/tutorials/keras/save_and_load.ipynb @@ -217,7 +217,7 @@ "source": [ "# Define a simple sequential model\n", "def create_model():\n", - " model = tf.keras.models.Sequential([\n", + " model = tf.keras.Sequential([\n", " keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n", " keras.layers.Dropout(0.2),\n", " keras.layers.Dense(10)\n", From 3dc12361bd8afebd52fc27264817cd22ee6ac4e1 Mon Sep 17 00:00:00 2001 From: mohantym <86464649+mohantym@users.noreply.github.com> Date: Mon, 25 Apr 2022 14:33:00 +0530 Subject: [PATCH 078/872] Updated build command for saved_model_cli Below command for building saved_model_cli is not working and throwing "Build file not found " error bazel build tensorflow/python/tools:saved_model_cli replaced with bazel build //tensorflow/python/tools:saved_model_cli . Attaching Gist for reference. https://colab.sandbox.google.com/gist/mohantym/5f2785f4c21c6b1e3fe9c5958a62418d/tensorflow-ranking.ipynb#scrollTo=Gr24fRLOQVRZ --- site/en/guide/saved_model.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/saved_model.ipynb b/site/en/guide/saved_model.ipynb index 068d6d91de9..fab111b25a9 100644 --- a/site/en/guide/saved_model.ipynb +++ b/site/en/guide/saved_model.ipynb @@ -761,7 +761,7 @@ "additional command to build `saved_model_cli`:\n", "\n", "```\n", - "$ bazel build tensorflow/python/tools:saved_model_cli\n", + "$ bazel build //tensorflow/python/tools:saved_model_cli\n", "```\n", "\n", "### Overview of commands\n", From 61d8f35eb41280a04999e2f5a4d5db522bd425d6 Mon Sep 17 00:00:00 2001 From: tfdocsbot Date: Mon, 25 Apr 2022 09:04:16 +0000 Subject: [PATCH 079/872] nbfmt --- site/en/guide/saved_model.ipynb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/site/en/guide/saved_model.ipynb b/site/en/guide/saved_model.ipynb index fab111b25a9..da9652c6554 100644 --- a/site/en/guide/saved_model.ipynb +++ b/site/en/guide/saved_model.ipynb @@ -77,8 +77,7 @@ "- High-level `tf.keras.Model` API. Refer to [the keras save and serialize guide](https://www.tensorflow.org/guide/keras/save_and_serialize).\n", "- If you just want to save/load weights during training, refer to [the checkpoints guide](./checkpoint.ipynb).\n", "\n", - "Caution: TensorFlow models are code and it is important to be careful with untrusted code. Learn more in [Using TensorFlow securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md).\n", - "\n" + "Caution: TensorFlow models are code and it is important to be careful with untrusted code. Learn more in [Using TensorFlow securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md).\n" ] }, { From de82dacb1ea221b2e821ac1b64ada336536bf87d Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Mon, 25 Apr 2022 14:45:29 -0700 Subject: [PATCH 080/872] Fix name prioritization for references into classes. Instead of "if a class holds a non-inherited reference prefer that" it's now "prefer non-class references, then non-inherited class references, then inherited class references". PiperOrigin-RevId: 444375831 --- .../api_generator/doc_generator_visitor.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index a835ad7a836..f03f64608d3 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -335,11 +335,15 @@ def _score_name(self, path: ApiPath) -> NameScore: short_name = path[-1] container = self.path_tree[path[:-1]].py_object - defining_class_score = 1 - if inspect.isclass(container): + # Prefer the reference that is not in a class. + defining_class_score = -1 + container_type = obj_type_lib.ObjType.get(container) + if container_type is obj_type_lib.ObjType.CLASS: if short_name in container.__dict__: - # prefer the defining class - defining_class_score = -1 + # If a alias points into a class, prefer the defining class + defining_class_score = 0 + else: + defining_class_score = 1 experimental_score = -1 if 'contrib' in path or any('experimental' in part for part in path): From b65661c0339453e2dd37f56b9d0a53385d6f5b2b Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Mon, 25 Apr 2022 15:21:42 -0700 Subject: [PATCH 081/872] Keep a reference to the physical path for each object. This is not used for anything yet. It will be useful for writing better inheritance trees to the docs. PiperOrigin-RevId: 444384868 --- .../api_generator/doc_generator_visitor.py | 22 +++++++- .../doc_generator_visitor_test.py | 36 ++++++++++--- .../api_generator/reference_resolver.py | 50 +++++++++++-------- .../api_generator/reference_resolver_test.py | 17 +++++-- 4 files changed, 93 insertions(+), 32 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index f03f64608d3..5ec20d87565 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -461,6 +461,7 @@ def build(self): class ApiTreeNode(PathTreeNode): """A node in the ApiTree.""" aliases: List[ApiPath] = dataclasses.field(default_factory=list) + physical_path: Optional[ApiPath] = None @property def obj_type(self) -> obj_type_lib.ObjType: @@ -556,7 +557,11 @@ def insert(self, path: ApiPath, py_object: Any, aliases: List[ApiPath]): parent = self[parent_path] node = ApiTreeNode( - path=path, py_object=py_object, aliases=aliases, parent=parent) + path=path, + py_object=py_object, + aliases=aliases, + parent=parent, + physical_path=self._get_physical_path(py_object)) super().__setitem__(path, node) self._nodes.append(node) @@ -570,6 +575,21 @@ def insert(self, path: ApiPath, py_object: Any, aliases: List[ApiPath]): parent.children[node.short_name] = node + def _get_physical_path(self, py_object): + physical_path = None + obj_type = obj_type_lib.ObjType.get(py_object) + if obj_type in [obj_type.CLASS, obj_type.CALLABLE]: + try: + physical_path = tuple( + py_object.__module__.split('.') + py_object.__qualname__.split('.')) + except AttributeError: + pass + elif obj_type is obj_type.MODULE: + physical_path = tuple(py_object.__name__.split('.')) + + return physical_path + + @classmethod def from_path_tree(cls, path_tree: PathTree, score_name_fn) -> 'ApiTree': """Create an ApiTree from an PathTree. diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py index 510e9cec89e..f26d954d94b 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py @@ -302,7 +302,6 @@ def from_value(cls, value): self.assertIs(visitor.api_tree[('tf', 'MyClass', 'from_value')], visitor.api_tree[('tf', 'sub', 'MyClass', 'from_value')]) - def test_handles_duplicate_singleton_attributes(self): class MyClass: @@ -349,6 +348,7 @@ class MyClass: self.assertIs(visitor.api_tree[('tf', 'MyClass', 'simple')], visitor.api_tree[('tf', 'sub', 'MyClass', 'simple')]) + class PathTreeTest(absltest.TestCase): def test_contains(self): @@ -415,7 +415,6 @@ def test_duplicate_singleton(self): self.assertEqual([], found) - class ApiTreeTest(absltest.TestCase): def _make_fake_module(self) -> types.ModuleType: @@ -447,6 +446,7 @@ class Nested(object): tf = types.ModuleType('tf') tf.__file__ = __file__ + tf.seven = 7 tf.Parent = Parent tf.Outer = Outer tf.fun1 = fun1 @@ -459,8 +459,32 @@ class Nested(object): return tf + def test_physical_path(self): + tf = self._make_fake_module() + + api_tree = doc_generator_visitor.ApiTree() + api_tree.insert(path=('tf',), py_object=tf, aliases=[('tf',)]) + api_tree.insert( + path=('tf', 'sub2'), py_object=tf.sub2, aliases=[('tf', 'sub2')]) + api_tree.insert( + path=('tf', 'seven'), py_object=tf.seven, aliases=[('tf', 'seven')]) + api_tree.insert( + path=('tf', 'fun1'), py_object=tf.fun1, aliases=[('tf', 'fun1')]) + api_tree.insert( + path=('tf', 'sub2', 'Child'), + py_object=tf.sub2.Child, + aliases=[('tf', 'sub2', 'Child')]) + + self.assertEqual(('sub2',), api_tree[('tf', 'sub2')].physical_path) + self.assertIsNone(api_tree[('tf', 'seven')].physical_path) + self.assertEqual(('__main__', 'ApiTreeTest', '_make_fake_module', + '', ''), + api_tree[('tf', 'fun1')].physical_path) + self.assertEqual( + ('__main__', 'ApiTreeTest', '_make_fake_module', '', 'Child'), + api_tree[('tf', 'sub2', 'Child')].physical_path) + def test_api_tree(self): - seven = 7 tf = self._make_fake_module() api_tree = doc_generator_visitor.ApiTree() @@ -470,7 +494,7 @@ def test_api_tree(self): py_object=tf.Parent, aliases=[('tf', 'Parent'), ('tf', 'Parent2')]) api_tree.insert( - path=('tf', 'seven'), py_object=seven, aliases=[('tf', 'seven')]) + path=('tf', 'seven'), py_object=tf.seven, aliases=[('tf', 'seven')]) # A node can be looked up by any alias self.assertIs(api_tree[('tf', 'Parent')], api_tree[('tf', 'Parent2')]) @@ -482,8 +506,8 @@ def test_api_tree(self): self.assertIs(api_tree[('tf', 'Parent')], api_tree.node_for_object(tf.Parent)) # You can't lookup things that maybe singeltons. - self.assertIs(api_tree[('tf', 'seven')].py_object, seven) - self.assertIsNone(api_tree.node_for_object(seven)) + self.assertIs(api_tree[('tf', 'seven')].py_object, tf.seven) + self.assertIsNone(api_tree.node_for_object(tf.seven)) def test_from_path_tree(self): tf = self._make_fake_module() diff --git a/tools/tensorflow_docs/api_generator/reference_resolver.py b/tools/tensorflow_docs/api_generator/reference_resolver.py index e7cb889f16c..c124fff9c5a 100644 --- a/tools/tensorflow_docs/api_generator/reference_resolver.py +++ b/tools/tensorflow_docs/api_generator/reference_resolver.py @@ -25,7 +25,7 @@ from typing import Dict, List, Optional from tensorflow_docs.api_generator import parser -from tensorflow_docs.api_generator import obj_type as obj_type_lib + class TFDocsError(Exception): pass @@ -78,11 +78,15 @@ class ReferenceResolver: """, flags=re.VERBOSE) - def __init__(self, - duplicate_of: Dict[str, str], - is_fragment: Dict[str, bool], - py_module_names: List[str], - link_prefix: Optional[str] = None): + def __init__( + self, + *, + duplicate_of: Dict[str, str], + is_fragment: Dict[str, bool], + py_module_names: List[str], + link_prefix: Optional[str] = None, + physical_path: Optional[Dict[str, str]] = None, + ): """Initializes a Reference Resolver. Args: @@ -95,9 +99,12 @@ def __init__(self, link_prefix: The website to which these symbols should link to. A prefix is added before the links to enable cross-site linking if `link_prefix` is not None. + physical_path: A mapping from the preferred full_name to the object's + physical path. """ self._duplicate_of = duplicate_of self._is_fragment = is_fragment + self._physical_path = physical_path self._py_module_names = py_module_names self._link_prefix = link_prefix @@ -123,22 +130,25 @@ def from_visitor(cls, visitor, **kwargs): Returns: an instance of `ReferenceResolver` () """ - is_fragment = {} - for full_name, obj in visitor.index.items(): - obj_type = obj_type_lib.ObjType.get(obj) - if obj_type in (obj_type_lib.ObjType.CLASS, obj_type_lib.ObjType.MODULE): - is_fragment[full_name] = False - elif obj_type in (obj_type_lib.ObjType.CALLABLE, - obj_type_lib.ObjType.TYPE_ALIAS): - if parser.is_class_attr(full_name, visitor.index): - is_fragment[full_name] = True - else: - is_fragment[full_name] = False - else: - is_fragment[full_name] = True + api_tree = visitor.api_tree + all_is_fragment = {} + duplicate_of = {} + physical_path = {} + for node in api_tree.iter_nodes(): + full_name = node.full_name + is_fragment = node.output_type() is node.OutputType.FRAGMENT + if node.physical_path: + physical_path[node.full_name] = '.'.join(node.physical_path) + for alias in node.aliases: + alias_name = '.'.join(alias) + duplicate_of[alias_name] = full_name + all_is_fragment[alias_name] = is_fragment return cls( - duplicate_of=visitor.duplicate_of, is_fragment=is_fragment, **kwargs) + duplicate_of=visitor.duplicate_of, + is_fragment=all_is_fragment, + physical_path=physical_path, + **kwargs) def with_prefix(self, prefix): return type(self)( diff --git a/tools/tensorflow_docs/api_generator/reference_resolver_test.py b/tools/tensorflow_docs/api_generator/reference_resolver_test.py index db938ecf457..fc31602eae0 100644 --- a/tools/tensorflow_docs/api_generator/reference_resolver_test.py +++ b/tools/tensorflow_docs/api_generator/reference_resolver_test.py @@ -49,9 +49,10 @@ def testSaveReferenceResolver(self): } py_module_names = ['tf', 'tfdbg'] - resolver = reference_resolver_lib.ReferenceResolver(duplicate_of, - is_fragment, - py_module_names) + resolver = reference_resolver_lib.ReferenceResolver( + duplicate_of=duplicate_of, + is_fragment=is_fragment, + py_module_names=py_module_names) outdir = self.workdir @@ -81,7 +82,10 @@ def test_duplicate_fragment(self): py_module_names = ['tf'] reference_resolver = reference_resolver_lib.ReferenceResolver( - duplicate_of, is_fragment, py_module_names, link_prefix='') + duplicate_of=duplicate_of, + is_fragment=is_fragment, + py_module_names=py_module_names, + link_prefix='') # Method references point to the method, in the canonical class alias. result = reference_resolver.reference_to_url('tf.Class1.method') @@ -132,7 +136,10 @@ def test_partial_symbol_references(self, string, link): py_module_names = ['tf'] resolver = reference_resolver_lib.ReferenceResolver( - duplicate_of, is_fragment, py_module_names, link_prefix='..') + duplicate_of=duplicate_of, + is_fragment=is_fragment, + py_module_names=py_module_names, + link_prefix='..') input_string = string.join('``') ref_string = resolver.replace_references(input_string) From a69f07323a9f1acc7e553d083a811905ae73509c Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:01:43 +0100 Subject: [PATCH 082/872] Update tf.math.argmax name in Introduction to Tensors --- site/en/guide/tensor.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/tensor.ipynb b/site/en/guide/tensor.ipynb index b24ee3efef9..fe2cae2a9b1 100644 --- a/site/en/guide/tensor.ipynb +++ b/site/en/guide/tensor.ipynb @@ -368,7 +368,7 @@ "# Find the largest value\n", "print(tf.reduce_max(c))\n", "# Find the index of the largest value\n", - "print(tf.argmax(c))\n", + "print(tf.math.argmax(c))\n", "# Compute the softmax\n", "print(tf.nn.softmax(c))" ] From 87ba0704fd3d637fab84552507affab3dd9e3ce6 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:03:30 +0100 Subject: [PATCH 083/872] Update tf.math.argmax name in Automatically rewrite TF 1.x and compat.v1 API symbols --- site/en/guide/migrate/upgrade.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/migrate/upgrade.ipynb b/site/en/guide/migrate/upgrade.ipynb index b59f5da3f5c..a1bb3e72b23 100644 --- a/site/en/guide/migrate/upgrade.ipynb +++ b/site/en/guide/migrate/upgrade.ipynb @@ -574,7 +574,7 @@ "source": [ "## Caveats\n", "\n", - "- Do not update parts of your code manually before running this script. In particular, functions that have had reordered arguments like `tf.argmax` or `tf.batch_to_space` cause the script to incorrectly add keyword arguments that mismap your existing code.\n", + "- Do not update parts of your code manually before running this script. In particular, functions that have had reordered arguments like `tf.math.argmax` or `tf.batch_to_space` cause the script to incorrectly add keyword arguments that mismap your existing code.\n", "\n", "- The script assumes that `tensorflow` is imported using `import tensorflow as tf`, or `import tensorflow.compat.v1 as tf`.\n", "\n", From e94e004cd2203dd8c50901fc935ed50855f93eb4 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:05:42 +0100 Subject: [PATCH 084/872] Update tf.math.argmax name in Custom training walkthrough --- .../customization/custom_training_walkthrough.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/site/en/tutorials/customization/custom_training_walkthrough.ipynb b/site/en/tutorials/customization/custom_training_walkthrough.ipynb index 513f3427f51..eb68ab2dc9f 100644 --- a/site/en/tutorials/customization/custom_training_walkthrough.ipynb +++ b/site/en/tutorials/customization/custom_training_walkthrough.ipynb @@ -475,7 +475,7 @@ }, "outputs": [], "source": [ - "print(\"Prediction: {}\".format(tf.argmax(predictions, axis=1)))\n", + "print(\"Prediction: {}\".format(tf.math.argmax(predictions, axis=1)))\n", "print(\" Labels: {}\".format(labels))" ] }, @@ -824,7 +824,7 @@ " # training=False is needed only if there are layers with different\n", " # behavior during training versus inference (e.g. Dropout).\n", " logits = model(x, training=False)\n", - " prediction = tf.argmax(logits, axis=1, output_type=tf.int64)\n", + " prediction = tf.math.argmax(logits, axis=1, output_type=tf.int64)\n", " test_accuracy(prediction, y)\n", "\n", "print(\"Test set accuracy: {:.3%}\".format(test_accuracy.result()))" @@ -895,7 +895,7 @@ "predictions = model(predict_dataset, training=False)\n", "\n", "for i, logits in enumerate(predictions):\n", - " class_idx = tf.argmax(logits).numpy()\n", + " class_idx = tf.math.argmax(logits).numpy()\n", " p = tf.nn.softmax(logits)[class_idx]\n", " name = class_names[class_idx]\n", " print(\"Example {} prediction: {} ({:4.1f}%)\".format(i, name, 100*p))" From 66f41adbfaf30c93687664ba3eaf5d06406fcf92 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:07:38 +0100 Subject: [PATCH 085/872] Update tf.math.argmax name in Transfer learning with YAMNet --- site/en/tutorials/audio/transfer_learning_audio.ipynb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/site/en/tutorials/audio/transfer_learning_audio.ipynb b/site/en/tutorials/audio/transfer_learning_audio.ipynb index 16c679aed61..4a24598e991 100644 --- a/site/en/tutorials/audio/transfer_learning_audio.ipynb +++ b/site/en/tutorials/audio/transfer_learning_audio.ipynb @@ -286,7 +286,7 @@ "source": [ "scores, embeddings, spectrogram = yamnet_model(testing_wav_data)\n", "class_scores = tf.reduce_mean(scores, axis=0)\n", - "top_class = tf.argmax(class_scores)\n", + "top_class = tf.math.argmax(class_scores)\n", "inferred_class = class_names[top_class]\n", "\n", "print(f'The main sound is: {inferred_class}')\n", @@ -736,7 +736,7 @@ "outputs": [], "source": [ "reloaded_results = reloaded_model(testing_wav_data)\n", - "cat_or_dog = my_classes[tf.argmax(reloaded_results)]\n", + "cat_or_dog = my_classes[tf.math.argmax(reloaded_results)]\n", "print(f'The main sound is: {cat_or_dog}')" ] }, @@ -758,7 +758,7 @@ "outputs": [], "source": [ "serving_results = reloaded_model.signatures['serving_default'](testing_wav_data)\n", - "cat_or_dog = my_classes[tf.argmax(serving_results['classifier'])]\n", + "cat_or_dog = my_classes[tf.math.argmax(serving_results['classifier'])]\n", "print(f'The main sound is: {cat_or_dog}')\n" ] }, @@ -805,13 +805,13 @@ "# Run the model, check the output.\n", "scores, embeddings, spectrogram = yamnet_model(waveform)\n", "class_scores = tf.reduce_mean(scores, axis=0)\n", - "top_class = tf.argmax(class_scores)\n", + "top_class = tf.math.argmax(class_scores)\n", "inferred_class = class_names[top_class]\n", "top_score = class_scores[top_class]\n", "print(f'[YAMNet] The main sound is: {inferred_class} ({top_score})')\n", "\n", "reloaded_results = reloaded_model(waveform)\n", - "your_top_class = tf.argmax(reloaded_results)\n", + "your_top_class = tf.math.argmax(reloaded_results)\n", "your_inferred_class = my_classes[your_top_class]\n", "class_probabilities = tf.nn.softmax(reloaded_results, axis=-1)\n", "your_top_score = class_probabilities[your_top_class]\n", From ed2cf5e651c88d97127346ebe462819f3db12d16 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:09:10 +0100 Subject: [PATCH 086/872] Update tf.math.argmax name in Load text --- site/en/tutorials/load_data/text.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/load_data/text.ipynb b/site/en/tutorials/load_data/text.ipynb index 8b8b5ab0768..c45342f332c 100644 --- a/site/en/tutorials/load_data/text.ipynb +++ b/site/en/tutorials/load_data/text.ipynb @@ -817,7 +817,7 @@ "outputs": [], "source": [ "def get_string_labels(predicted_scores_batch):\n", - " predicted_int_labels = tf.argmax(predicted_scores_batch, axis=1)\n", + " predicted_int_labels = tf.math.argmax(predicted_scores_batch, axis=1)\n", " predicted_labels = tf.gather(raw_train_ds.class_names, predicted_int_labels)\n", " return predicted_labels" ] @@ -1476,7 +1476,7 @@ "]\n", "\n", "predicted_scores = export_model.predict(inputs)\n", - "predicted_labels = tf.argmax(predicted_scores, axis=1)\n", + "predicted_labels = tf.math.argmax(predicted_scores, axis=1)\n", "\n", "for input, label in zip(inputs, predicted_labels):\n", " print(\"Question: \", input)\n", From 678c2b6e6480c60385f1ca6b4cea365ac32c930b Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:12:14 +0100 Subject: [PATCH 087/872] Update tf.math.argmax name in Introduction to Variables --- site/en/guide/variable.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/variable.ipynb b/site/en/guide/variable.ipynb index 0502c324a35..868ee9119e2 100644 --- a/site/en/guide/variable.ipynb +++ b/site/en/guide/variable.ipynb @@ -166,7 +166,7 @@ "source": [ "print(\"A variable:\", my_variable)\n", "print(\"\\nViewed as a tensor:\", tf.convert_to_tensor(my_variable))\n", - "print(\"\\nIndex of highest value:\", tf.argmax(my_variable))\n", + "print(\"\\nIndex of highest value:\", tf.math.argmax(my_variable))\n", "\n", "# This creates a new tensor; it does not reshape the variable.\n", "print(\"\\nCopying and reshaping: \", tf.reshape(my_variable, [1,4]))" From 0cfeffc30fb5b1140cc8d0845048ae31ac96ccef Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:51:45 +0100 Subject: [PATCH 088/872] Update tf.math.argmax namespace in Image segmentation --- site/en/tutorials/images/segmentation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/images/segmentation.ipynb b/site/en/tutorials/images/segmentation.ipynb index 1c6f94fa6ac..348d84e0c15 100644 --- a/site/en/tutorials/images/segmentation.ipynb +++ b/site/en/tutorials/images/segmentation.ipynb @@ -507,7 +507,7 @@ "outputs": [], "source": [ "def create_mask(pred_mask):\n", - " pred_mask = tf.argmax(pred_mask, axis=-1)\n", + " pred_mask = tf.math.argmax(pred_mask, axis=-1)\n", " pred_mask = pred_mask[..., tf.newaxis]\n", " return pred_mask[0]" ] From 563659de0a6fd338395035d207ca4eb9b8544834 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:52:40 +0100 Subject: [PATCH 089/872] Update tf.math.argmax namespace in Simple audio recognition --- site/en/tutorials/audio/simple_audio.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/audio/simple_audio.ipynb b/site/en/tutorials/audio/simple_audio.ipynb index 2bf92c54cb7..a4d4c2fa634 100644 --- a/site/en/tutorials/audio/simple_audio.ipynb +++ b/site/en/tutorials/audio/simple_audio.ipynb @@ -563,7 +563,7 @@ "source": [ "def get_spectrogram_and_label_id(audio, label):\n", " spectrogram = get_spectrogram(audio)\n", - " label_id = tf.argmax(label == commands)\n", + " label_id = tf.math.argmax(label == commands)\n", " return spectrogram, label_id" ] }, From 1e97c82a9bd9428dcc33a1e861690bf138ac3105 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 07:58:00 +0100 Subject: [PATCH 090/872] Update tf.math.argmax namespace in Integrated gradients --- site/en/tutorials/interpretability/integrated_gradients.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/interpretability/integrated_gradients.ipynb b/site/en/tutorials/interpretability/integrated_gradients.ipynb index 6cf1e09b32a..2b7965db805 100644 --- a/site/en/tutorials/interpretability/integrated_gradients.ipynb +++ b/site/en/tutorials/interpretability/integrated_gradients.ipynb @@ -151,7 +151,7 @@ "\n", "**Inputs**: The expected input shape for the model is `(None, 224, 224, 3)`. This is a dense 4D tensor of dtype float32 and shape `(batch_size, height, width, RGB channels)` whose elements are RGB color values of pixels normalized to the range [0, 1]. The first element is `None` to indicate that the model can take any integer batch size.\n", "\n", - "**Outputs**: A `tf.Tensor` of logits in the shape of `(batch_size, 1001)`. Each row represents the model's predicted score for each of 1,001 classes from ImageNet. For the model's top predicted class index you can use `tf.argmax(predictions, axis=-1)`. Furthermore, you can also convert the model's logit output to predicted probabilities across all classes using `tf.nn.softmax(predictions, axis=-1)` to quantify the model's uncertainty as well as explore similar predicted classes for debugging." + "**Outputs**: A `tf.Tensor` of logits in the shape of `(batch_size, 1001)`. Each row represents the model's predicted score for each of 1,001 classes from ImageNet. For the model's top predicted class index you can use `tf.math.argmax(predictions, axis=-1)`. Furthermore, you can also convert the model's logit output to predicted probabilities across all classes using `tf.nn.softmax(predictions, axis=-1)` to quantify the model's uncertainty as well as explore similar predicted classes for debugging." ] }, { From 705162180634458398e145adf9003481ecb7e658 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Tue, 26 Apr 2022 08:03:26 +0100 Subject: [PATCH 091/872] Use second person in Automatically rewrite TF 1.x and compat.v1 API symbols --- site/en/guide/migrate/upgrade.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/migrate/upgrade.ipynb b/site/en/guide/migrate/upgrade.ipynb index a1bb3e72b23..34d00aaf5a7 100644 --- a/site/en/guide/migrate/upgrade.ipynb +++ b/site/en/guide/migrate/upgrade.ipynb @@ -95,7 +95,7 @@ "source": [ "## Compatibility modules\n", "\n", - "Certain API symbols can not be upgraded simply by using a string replacement. Those that cannot be automatically upgraded will be mapped to their locations in the `compat.v1` module. This module replaces TF 1.x symbols like `tf.foo` with the equivalent `tf.compat.v1.foo` reference. If you are already using `compat.v1` APIs by importing TF via `import tensorflow.compat.v1 as tf`, the `tf_upgrade_v2` script will attempt to convert these usages to the non-compat APIs where possible. Note that while some `compat.v1` APIs are compatible with TF2.x behaviors, many are not. So, we recommend that you manually proofread replacements and migrate them to new APIs in the `tf.*` namespace instead of `tf.compat.v1` namespace as quickly as possible.\n", + "Certain API symbols can not be upgraded simply by using a string replacement. Those that cannot be automatically upgraded will be mapped to their locations in the `compat.v1` module. This module replaces TF 1.x symbols like `tf.foo` with the equivalent `tf.compat.v1.foo` reference. If you are already using `compat.v1` APIs by importing TF via `import tensorflow.compat.v1 as tf`, the `tf_upgrade_v2` script will attempt to convert these usages to the non-compat APIs where possible. Note that while some `compat.v1` APIs are compatible with TF2.x behaviors, many are not. Therefore, it's recommended to manually proofread replacements and migrate them to new APIs in the `tf.*` namespace instead of `tf.compat.v1` namespace as quickly as possible.\n", "\n", "Because of TensorFlow 2.x module deprecations (for example, `tf.flags` and `tf.contrib`), some changes can not be worked around by switching to `compat.v1`. Upgrading this code may require using an additional library (for example, [`absl.flags`](https://github.com/abseil/abseil-py)) or switching to a package in [tensorflow/addons](http://www.github.com/tensorflow/addons).\n" ] From 744a2b26454619ff3187560e1056163e9cbb0348 Mon Sep 17 00:00:00 2001 From: sushreebarsa <84765720+sushreebarsa@users.noreply.github.com> Date: Tue, 26 Apr 2022 14:28:55 +0530 Subject: [PATCH 092/872] Updated tf.add naming in TF documentation Updated the naming from 'tf.add' to 'tf.math.add' in the documentation. --- site/en/tutorials/customization/basics.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb index fa13409feaa..d23b7ca2477 100644 --- a/site/en/tutorials/customization/basics.ipynb +++ b/site/en/tutorials/customization/basics.ipynb @@ -106,7 +106,7 @@ "source": [ "## Tensors\n", "\n", - "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations ([tf.add](https://www.tensorflow.org/api_docs/python/tf/add), [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/matmul), [tf.linalg.inv](https://www.tensorflow.org/api_docs/python/tf/linalg/inv) etc.) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types, for example:\n" + "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations ([tf.math.add](https://www.tensorflow.org/api_docs/python/tf/add), [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/matmul), [tf.linalg.inv](https://www.tensorflow.org/api_docs/python/tf/linalg/inv) etc.) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types, for example:\n" ] }, { From 5ae8c625d036ba431cf66541d45da5c5d4989a61 Mon Sep 17 00:00:00 2001 From: taoxu Date: Wed, 27 Apr 2022 10:04:34 +0800 Subject: [PATCH 093/872] Bump libtensorflow version to 2.8.0 --- site/en/install/lang_c.ipynb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb index cdc62873df3..123ec8b82ef 100644 --- a/site/en/install/lang_c.ipynb +++ b/site/en/install/lang_c.ipynb @@ -133,25 +133,25 @@ " Linux\n", " \n", " Linux CPU only\n", - " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-2.7.0.tar.gz\n", + " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-2.8.0.tar.gz\n", " \n", " \n", " Linux GPU support\n", - " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-2.7.0.tar.gz\n", + " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-2.8.0.tar.gz\n", " \n", " macOS\n", " \n", " macOS CPU only\n", - " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-2.7.0.tar.gz\n", + " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-2.8.0.tar.gz\n", " \n", " Windows\n", " \n", " Windows CPU only\n", - " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-2.7.0.zip\n", + " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-2.8.0.zip\n", " \n", " \n", " Windows GPU only\n", - " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-windows-x86_64-2.7.0.zip\n", + " https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-windows-x86_64-2.8.0.zip\n", " \n", "" ] @@ -177,7 +177,7 @@ "outputs": [], "source": [ "%%bash\n", - "FILENAME=libtensorflow-cpu-linux-x86_64-2.7.0.tar.gz\n", + "FILENAME=libtensorflow-cpu-linux-x86_64-2.8.0.tar.gz\n", "wget -q --no-check-certificate https://storage.googleapis.com/tensorflow/libtensorflow/${FILENAME}\n", "sudo tar -C /usr/local -xzf ${FILENAME}" ] From d3a20809d6b887f41db56522d14bd121e84bdadc Mon Sep 17 00:00:00 2001 From: tfdocsbot Date: Wed, 27 Apr 2022 02:10:51 +0000 Subject: [PATCH 094/872] nbfmt --- site/en/install/lang_c.ipynb | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb index 123ec8b82ef..df8e3c950c2 100644 --- a/site/en/install/lang_c.ipynb +++ b/site/en/install/lang_c.ipynb @@ -99,8 +99,6 @@ "id": "qowtdsijFMYZ" }, "source": [ - "\n", - "\n", "## Supported Platforms\n", "\n", "TensorFlow for C is supported on the following systems:\n", @@ -125,7 +123,6 @@ "id": "y50y01XUFVb2" }, "source": [ - "\n", "### Download & extract\n", "\n", "\n", @@ -188,7 +185,6 @@ "id": "fcBJDdojJDyk" }, "source": [ - "\n", "### Linker\n", "\n", "On Linux/macOS, if you extract the TensorFlow C library to a system directory,\n", @@ -266,12 +262,6 @@ "cell_type": "code", "execution_count": null, "metadata": { - "attributes": { - "classes": [ - "c" - ], - "id": "" - }, "id": "b5851f1b" }, "outputs": [], @@ -351,9 +341,6 @@ "id": "ea5fd208" }, "source": [ - "\n", - "\n", - "\n", "## Build from source\n", "\n", "TensorFlow is open source. Read\n", @@ -365,9 +352,7 @@ "metadata": { "colab": { "collapsed_sections": [], - "name": "Copy of lang_c.ipynb", - "private_outputs": true, - "provenance": [], + "name": "lang_c.ipynb", "toc_visible": true }, "kernelspec": { From be2487f9f4d5c49965bf8b61c3bc4a0c0d79f5c3 Mon Sep 17 00:00:00 2001 From: gadagashwini <99852755+gadagashwini@users.noreply.github.com> Date: Wed, 27 Apr 2022 10:22:49 +0530 Subject: [PATCH 095/872] Fix typo error tf.strings.substr Replaced tf.substr with proper API name tf.strings.substr --- site/en/guide/ragged_tensor.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/guide/ragged_tensor.ipynb b/site/en/guide/ragged_tensor.ipynb index 4bc0d679499..ba31138a3cf 100644 --- a/site/en/guide/ragged_tensor.ipynb +++ b/site/en/guide/ragged_tensor.ipynb @@ -109,7 +109,7 @@ "source": [ "### What you can do with a ragged tensor\n", "\n", - "Ragged tensors are supported by more than a hundred TensorFlow operations, including math operations (such as `tf.add` and `tf.reduce_mean`), array operations (such as `tf.concat` and `tf.tile`), string manipulation ops (such as `tf.substr`), control flow operations (such as `tf.while_loop` and `tf.map_fn`), and many others:" + "Ragged tensors are supported by more than a hundred TensorFlow operations, including math operations (such as `tf.add` and `tf.reduce_mean`), array operations (such as `tf.concat` and `tf.tile`), string manipulation ops (such as `tf.strings.substr`), control flow operations (such as `tf.while_loop` and `tf.map_fn`), and many others:" ] }, { From b46ed6386a79073dea94d9e9e3a37c2465b6f760 Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 27 Apr 2022 03:57:17 -0700 Subject: [PATCH 096/872] Add class="external" to the small_source_link. + actually fix api-gen test (that's howe the golden api_cache file got out of sync). PiperOrigin-RevId: 444814024 --- .../api_generator/pretty_docs/base_page.py | 11 +++++------ .../api_generator/pretty_docs/pretty_docs_test.py | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py index 66e2bcd9569..059c13ec3ea 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py @@ -486,15 +486,14 @@ def top_source_link(location): return table -def small_source_link(location): +def small_source_link(location, text='View source'): """Returns a small source link.""" - template = 'View source\n\n' - - if not location.url: + if location.url: + return ('{text}\n\n') + else: return '' - return template.format(url=location.url) - def build_collapsable_aliases(aliases: List[str]) -> str: """Returns the top "Aliases" line.""" diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py b/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py index 0c9514cbb44..583d9bb7c27 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py @@ -54,7 +54,7 @@ def test_other_source_link_after_table(self):
- View source + View source """) self.assertEqual(expected, table) From 5b02a843d9352d444e0a1db80ee2a74a9c22edfd Mon Sep 17 00:00:00 2001 From: Mark Daoust Date: Wed, 27 Apr 2022 04:14:23 -0700 Subject: [PATCH 097/872] Pure refactor: Give the PageInfo access to the parser_config and api_node. I prefer api_nodes compared to passing around (py_object, full_name) everywhere. PiperOrigin-RevId: 444817172 --- .../api_generator/doc_generator_visitor.py | 2 +- .../api_generator/generate_lib.py | 11 ++- tools/tensorflow_docs/api_generator/parser.py | 37 +++++---- .../api_generator/parser_test.py | 68 ++++++++++------- .../api_generator/pretty_docs/base_page.py | 32 ++++---- .../api_generator/pretty_docs/class_page.py | 76 +++++++++---------- .../pretty_docs/docs_for_object.py | 24 +++--- .../pretty_docs/function_page.py | 15 ++-- .../api_generator/pretty_docs/module_page.py | 47 ++++-------- .../pretty_docs/pretty_docs_test.py | 6 +- .../pretty_docs/type_alias_page.py | 14 ++-- .../api_generator/report/linter_test.py | 14 +++- .../api_generator/report/utils.py | 4 +- .../api_generator/signature_test.py | 66 +++++++++------- 14 files changed, 211 insertions(+), 205 deletions(-) diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py index 5ec20d87565..fec792095bd 100644 --- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py +++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py @@ -73,7 +73,7 @@ class PathTreeNode(object): """ path: ApiPath py_object: Any - parent: Optional['PathTreeNode'] + parent: Optional['PathTreeNode'] = None children: Dict[str, 'PathTreeNode'] = dataclasses.field(default_factory=dict) def __hash__(self): diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py index de6ddcbb76d..c00ab44a7cf 100644 --- a/tools/tensorflow_docs/api_generator/generate_lib.py +++ b/tools/tensorflow_docs/api_generator/generate_lib.py @@ -120,18 +120,17 @@ def write_docs( # Parse and write Markdown pages, resolving cross-links (`tf.symbol`). num_docs_output = 0 - for node in parser_config.api_tree.iter_nodes(): - full_name = node.full_name - py_object = node.py_object + for api_node in parser_config.api_tree.iter_nodes(): + full_name = api_node.full_name + py_object = api_node.py_object - if node.output_type() is node.OutputType.FRAGMENT: + if api_node.output_type() is api_node.OutputType.FRAGMENT: continue # Generate docs for `py_object`, resolving references. try: page_info = docs_for_object.docs_for_object( - full_name=full_name, - py_object=py_object, + api_node=api_node, parser_config=parser_config, extra_docs=extra_docs, search_hints=search_hints, diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py index 5dd25b8f90d..6ceeb6118be 100644 --- a/tools/tensorflow_docs/api_generator/parser.py +++ b/tools/tensorflow_docs/api_generator/parser.py @@ -33,25 +33,6 @@ from tensorflow_docs.api_generator import signature as signature_lib -@dataclasses.dataclass -class FileLocation(object): - """This class indicates that the object is defined in a regular file. - - This can be used for the `defined_in` slot of the `PageInfo` objects. - """ - - base_url: Optional[str] = None - start_line: Optional[int] = None - end_line: Optional[int] = None - - @property - def url(self) -> Optional[str]: - if self.start_line and self.end_line: - if 'github.com' in self.base_url: - return f'{self.base_url}#L{self.start_line}-L{self.end_line}' - return self.base_url - - def is_class_attr(full_name, index): """Check if the object's parent is a class. @@ -614,6 +595,24 @@ def _unwrap_obj(obj): return obj +@dataclasses.dataclass +class FileLocation(object): + """This class indicates that the object is defined in a regular file. + + This can be used for the `defined_in` slot of the `PageInfo` objects. + """ + + base_url: Optional[str] = None + start_line: Optional[int] = None + end_line: Optional[int] = None + + @property + def url(self) -> Optional[str]: + if self.start_line and self.end_line: + if 'github.com' in self.base_url: + return f'{self.base_url}#L{self.start_line}-L{self.end_line}' + return self.base_url + def get_defined_in( py_object: Any, parser_config: config.ParserConfig) -> Optional[FileLocation]: diff --git a/tools/tensorflow_docs/api_generator/parser_test.py b/tools/tensorflow_docs/api_generator/parser_test.py index 4322aadb76f..b53978b239a 100644 --- a/tools/tensorflow_docs/api_generator/parser_test.py +++ b/tools/tensorflow_docs/api_generator/parser_test.py @@ -28,6 +28,7 @@ from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_controls +from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib @@ -198,10 +199,13 @@ def test_docs_for_class(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=( + 'm', + 'TestClass', + ), py_object=TestClass) page_info = docs_for_object.docs_for_object( - full_name='m.TestClass', - py_object=TestClass, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) # Make sure the brief docstring is present self.assertEqual( @@ -241,10 +245,10 @@ def test_dataclass_attributes_table(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('m', 'ExampleDataclass'), py_object=ExampleDataclass) page_info = docs_for_object.docs_for_object( - full_name='m.ExampleDataclass', - py_object=ExampleDataclass, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) self.assertCountEqual(['a', 'b', 'c', 'x', 'y', 'z'], [name for name, value in page_info.attr_block.items]) @@ -268,10 +272,10 @@ def hide(path, parent, children): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('m', 'namedtupleclass'), py_object=namedtupleclass) page_info = docs_for_object.docs_for_object( - full_name='m.namedtupleclass', - py_object=namedtupleclass, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) self.assertIsNone(page_info._namedtuplefields['hidden']) @@ -308,8 +312,13 @@ def a_method(self, arg='default'): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=( + 'm', + 'Child', + ), py_object=Child) page_info = docs_for_object.docs_for_object( - full_name='m.Child', py_object=Child, parser_config=parser_config) + api_node=api_node, parser_config=parser_config) # Make sure the `a_method` is not present self.assertEmpty(page_info.methods) @@ -347,10 +356,10 @@ def my_method(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('m', 'ChildMessage'), py_object=ChildMessage) page_info = docs_for_object.docs_for_object( - full_name='m.ChildMessage', - py_object=ChildMessage, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) self.assertLen(page_info.methods, 1) self.assertEqual('my_method', page_info.methods[0].short_name) @@ -369,8 +378,10 @@ def test_docs_for_module(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('m',), py_object=test_module) page_info = docs_for_object.docs_for_object( - full_name='m', py_object=test_module, parser_config=parser_config) + api_node=api_node, parser_config=parser_config) # Make sure the brief docstring is present self.assertEqual( @@ -395,10 +406,10 @@ def test_docs_for_function(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('test_function',), py_object=test_function) page_info = docs_for_object.docs_for_object( - full_name='test_function', - py_object=test_function, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) # Make sure the brief docstring is present self.assertEqual( @@ -420,10 +431,11 @@ def test_docs_for_function_with_kwargs(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('test_function_with_args_kwargs',), + py_object=test_function_with_args_kwargs) page_info = docs_for_object.docs_for_object( - full_name='test_function_with_args_kwargs', - py_object=test_function_with_args_kwargs, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) # Make sure the brief docstring is present self.assertEqual( @@ -594,10 +606,10 @@ def test_getsource_indexerror_resilience(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('m', 'ConcreteMutableMapping'), py_object=ConcreteMutableMapping) page_info = docs_for_object.docs_for_object( - full_name='m.ConcreteMutableMapping', - py_object=ConcreteMutableMapping, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) self.assertIn(ConcreteMutableMapping.get, [m.py_object for m in page_info.methods]) @@ -621,8 +633,10 @@ def test_strips_default_arg_memory_address(self): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=('m', 'fun'), py_object=m.fun) page_info = docs_for_object.docs_for_object( - full_name='m.fun', py_object=m.fun, parser_config=parser_config) + api_node=api_node, parser_config=parser_config) output = str(page_info.signature) self.assertNotIn('object at 0x', output) @@ -669,10 +683,10 @@ def test_empty_defined_in(self, cls, method, py_object): parser_config = generator.run_extraction() + api_node = doc_generator_visitor.ApiTreeNode( + path=(cls, method), py_object=py_object) function_info = docs_for_object.docs_for_object( - full_name='%s.%s' % (cls, method), - py_object=py_object, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) self.assertIsNone(function_info.defined_in) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py index 059c13ec3ea..8db7b624113 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py @@ -110,10 +110,10 @@ class PageInfo: def __init__( self, - full_name: str, - py_object: Any, + api_node, extra_docs: Optional[Dict[int, str]] = None, search_hints: bool = True, + parser_config=None, ): """Initialize a PageInfo. @@ -126,25 +126,23 @@ def __init__( "robots: noindex" """ - self.full_name = full_name - self.py_object = py_object + self.api_node = api_node + self.full_name = api_node.full_name + self.py_object = api_node.py_object self._extra_docs = extra_docs self.search_hints = search_hints + self.parser_config = parser_config self._defined_in = None self._aliases = None self._doc = None self._page_text = None - def collect_docs(self, parser_config: config.ParserConfig): + def collect_docs(self): """Collects additional information from the `config.ParserConfig`.""" pass - def docs_for_object(self, parser_config): - duplicate_names = parser_config.duplicates.get(self.full_name, []) - if self.full_name in duplicate_names: - duplicate_names.remove(self.full_name) - + def docs_for_object(self): relative_path = os.path.relpath( path='.', start=os.path.dirname(parser.documentation_path(self.full_name)) or '.') @@ -152,20 +150,24 @@ def docs_for_object(self, parser_config): # Convert from OS-specific path to URL/POSIX path. relative_path = posixpath.join(*relative_path.split(os.path.sep)) - with parser_config.reference_resolver.temp_prefix(relative_path): + with self.parser_config.reference_resolver.temp_prefix(relative_path): self.set_doc( parser.parse_md_docstring( self.py_object, self.full_name, - parser_config, + self.parser_config, self._extra_docs, )) - self.collect_docs(parser_config) + self.collect_docs() - self.set_aliases(duplicate_names) + aliases = ['.'.join(alias) for alias in self.api_node.aliases] + if self.full_name in aliases: + aliases.remove(self.full_name) + self.set_aliases(aliases) - self.set_defined_in(parser.get_defined_in(self.py_object, parser_config)) + self.set_defined_in( + parser.get_defined_in(self.py_object, self.parser_config)) self._page_text = self.build() diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py index 525b986cc91..909e026dba5 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py @@ -127,7 +127,7 @@ class ClassPageInfo(base_page.PageInfo): """ DEFAULT_BUILDER_CLASS = ClassPageBuilder - def __init__(self, *, full_name, py_object, **kwargs): + def __init__(self, *, api_node, **kwargs): """Initialize a ClassPageInfo. Args: @@ -135,13 +135,13 @@ def __init__(self, *, full_name, py_object, **kwargs): py_object: The object being documented. **kwargs: Extra arguments. """ - super().__init__(full_name, py_object, **kwargs) + super().__init__(api_node, **kwargs) self._namedtuplefields = collections.OrderedDict() - if issubclass(py_object, tuple): + if issubclass(api_node.py_object, tuple): namedtuple_attrs = ('_asdict', '_fields', '_make', '_replace') - if all(hasattr(py_object, attr) for attr in namedtuple_attrs): - for name in py_object._fields: + if all(hasattr(api_node.py_object, attr) for attr in namedtuple_attrs): + for name in api_node.py_object._fields: self._namedtuplefields[name] = None self._properties = collections.OrderedDict() @@ -167,24 +167,22 @@ def set_attr_block(self, attr_block): assert self.attr_block is None self.attr_block = attr_block - def _set_bases(self, parser_config): + def _set_bases(self): """Builds the `bases` attribute, to document this class' parent-classes. This method sets the `bases` to a list of `base_page.MemberInfo` objects point to the doc pages for the class' parents. - - Args: - parser_config: An instance of `config.ParserConfig`. """ bases = [] for base in self.py_object.__mro__[1:]: - base_full_name = parser_config.reverse_index.get(id(base), None) - if base_full_name is None: + base_api_node = self.parser_config.api_tree.node_for_object(base) + if base_api_node is None: continue - base_doc = parser.parse_md_docstring(base, self.full_name, parser_config, - self._extra_docs) - base_url = parser_config.reference_resolver.reference_to_url( + base_full_name = base_api_node.full_name + base_doc = parser.parse_md_docstring(base, base_full_name, + self.parser_config, self._extra_docs) + base_url = self.parser_config.reference_resolver.reference_to_url( base_full_name) link_info = base_page.MemberInfo( @@ -233,13 +231,12 @@ def _add_method( self, member_info: base_page.MemberInfo, defining_class: Optional[type], # pylint: disable=g-bare-generic - parser_config: config.ParserConfig) -> None: + ) -> None: """Adds a `MethodInfo` entry to the `methods` list. Args: member_info: a `base_page.MemberInfo` describing the method. defining_class: The `type` object where this method is defined. - parser_config: A `config.ParserConfig`. """ if defining_class is None: return @@ -260,8 +257,9 @@ def _add_method( # If the current class py_object is a dataclass then use the class object # instead of the __init__ method object because __init__ is a # generated method on dataclasses (unless the definition used init=False) - # and `inspect.getsource` doesn't work on generated methods (as the source - # file doesn't exist) which is required for signature generation. + # and `api_generator.get_source.get_source` doesn't work on generated + # methods (as the source file doesn't exist) which is required for + # signature generation. if (dataclasses.is_dataclass(self.py_object) and member_info.short_name == '__init__' and self.py_object.__dataclass_params__.init): @@ -275,11 +273,12 @@ def _add_method( member_info.short_name, is_dataclass) signature = signature_lib.generate_signature( - py_obj, parser_config, func_type=func_type) + py_obj, self.parser_config, func_type=func_type) decorators = signature_lib.extract_decorators(member_info.py_object) - defined_in = parser.get_defined_in(member_info.py_object, parser_config) + defined_in = parser.get_defined_in(member_info.py_object, + self.parser_config) method_info = MethodInfo.from_member_info(member_info, signature, decorators, defined_in) @@ -322,7 +321,6 @@ def _add_member( self, member_info: base_page.MemberInfo, defining_class: Optional[type], # pylint: disable=g-bare-generic - parser_config: config.ParserConfig, ) -> None: """Adds a member to the class page.""" obj_type = obj_type_lib.ObjType.get(member_info.py_object) @@ -334,7 +332,7 @@ def _add_member( return self._add_class(member_info) elif obj_type is obj_type_lib.ObjType.CALLABLE: - self._add_method(member_info, defining_class, parser_config) + self._add_method(member_info, defining_class) elif obj_type is obj_type_lib.ObjType.OTHER: # Exclude members defined by protobuf that are useless if issubclass(self.py_object, ProtoMessage): @@ -344,24 +342,19 @@ def _add_member( self._add_other_member(member_info) - def collect_docs(self, parser_config): + def collect_docs(self): """Collects information necessary specifically for a class's doc page. Mainly, this is details about the class's members. - - Args: - parser_config: An instance of config.ParserConfig. """ py_class = self.py_object - self._set_bases(parser_config) - - for child_short_name in parser_config.tree[self.full_name]: - child_full_name = '.'.join([self.full_name, child_short_name]) - child = parser_config.py_name_to_object(child_full_name) + self._set_bases() + class_path_node = self.parser_config.path_tree[self.api_node.path] + for _, path_node in sorted(class_path_node.children.items()): # Don't document anything that is defined in object or by protobuf. - defining_class = parser.get_defining_class(py_class, child_short_name) + defining_class = parser.get_defining_class(py_class, path_node.short_name) if defining_class in [object, type, tuple, BaseException, Exception]: continue @@ -370,18 +363,21 @@ def collect_docs(self, parser_config): defining_class.__name__ in ['CMessage', 'Message', 'MessageMeta']): continue - if doc_controls.should_skip_class_attr(py_class, child_short_name): + if doc_controls.should_skip_class_attr(py_class, path_node.short_name): continue - child_doc = parser.parse_md_docstring(child, self.full_name, - parser_config, self._extra_docs) + child_doc = parser.parse_md_docstring(path_node.py_object, self.full_name, + self.parser_config, + self._extra_docs) - child_url = parser_config.reference_resolver.reference_to_url( - child_full_name) + child_url = self.parser_config.reference_resolver.reference_to_url( + path_node.full_name) - member_info = base_page.MemberInfo(child_short_name, child_full_name, - child, child_doc, child_url) - self._add_member(member_info, defining_class, parser_config) + member_info = base_page.MemberInfo(path_node.short_name, + path_node.full_name, + path_node.py_object, child_doc, + child_url) + self._add_member(member_info, defining_class) self.set_attr_block(self._augment_attributes(self.doc.docstring_parts)) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py index ae1cb5ec6b7..91f5af47292 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/docs_for_object.py @@ -17,6 +17,7 @@ from tensorflow_docs.api_generator import config from tensorflow_docs.api_generator import doc_controls +from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import obj_type as obj_type_lib from tensorflow_docs.api_generator.pretty_docs import base_page from tensorflow_docs.api_generator.pretty_docs import class_page @@ -36,8 +37,7 @@ def docs_for_object( *, - full_name: str, - py_object: Any, + api_node: doc_generator_visitor.ApiTreeNode, parser_config: config.ParserConfig, extra_docs: Optional[Dict[int, str]] = None, search_hints: bool = True, @@ -49,9 +49,7 @@ def docs_for_object( to the appropriate location. Args: - full_name: The fully qualified name of the symbol to be documented. - py_object: The Python object to be documented. Its documentation is sourced - from `py_object`'s docstring. + api_node: The ApiTreeNode for the object. parser_config: A `config.ParserConfig` object. extra_docs: Extra docs for symbols like public constants(list, tuple, etc) that need to be added to the markdown pages created. @@ -68,24 +66,20 @@ def docs_for_object( RuntimeError: If an object is encountered for which we don't know how to make docs. """ - - # Which other aliases exist for the object referenced by full_name? - main_name = parser_config.reference_resolver.py_main_name(full_name) - if page_builder_classes is None: page_builder_classes = _DEFAULT_PAGE_BUILDER_CLASSES - page_info_class = doc_controls.get_custom_page_builder_cls(py_object) + page_info_class = doc_controls.get_custom_page_builder_cls(api_node.py_object) if page_info_class is None: - obj_type = obj_type_lib.ObjType.get(py_object) + obj_type = obj_type_lib.ObjType.get(api_node.py_object) page_info_class = page_builder_classes[obj_type] page_info = page_info_class( - full_name=main_name, - py_object=py_object, + api_node=api_node, search_hints=search_hints, - extra_docs=extra_docs) + extra_docs=extra_docs, + parser_config=parser_config) - page_info.docs_for_object(parser_config) + page_info.docs_for_object() return page_info diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py index 73d596cbc5c..9d445961c0e 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/function_page.py @@ -50,15 +50,14 @@ class FunctionPageInfo(base_page.PageInfo): """ DEFAULT_BUILDER_CLASS = FunctionPageBuilder - def __init__(self, *, full_name: str, py_object: Any, **kwargs): + def __init__(self, *, api_node, **kwargs): """Initialize a FunctionPageInfo. Args: - full_name: The full, main name, of the object being documented. - py_object: The object being documented. + api_node: the api tree node. **kwargs: Extra arguments. """ - super().__init__(full_name, py_object, **kwargs) + super().__init__(api_node, **kwargs) self._signature = None self._decorators = [] @@ -67,19 +66,15 @@ def __init__(self, *, full_name: str, py_object: Any, **kwargs): def signature(self): return self._signature - def collect_docs(self, parser_config): + def collect_docs(self): """Collect all information necessary to genertate the function page. Mainly this is details about the function signature. - - Args: - parser_config: The config.ParserConfig for the module being documented. """ - assert self.signature is None self._signature = signature_lib.generate_signature( self.py_object, - parser_config, + self.parser_config, func_type=signature_lib.FuncType.FUNCTION, ) self._decorators = signature_lib.extract_decorators(self.py_object) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py index d9d40bb9d89..9141827e1e6 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py @@ -62,7 +62,7 @@ class ModulePageInfo(base_page.PageInfo): """ DEFAULT_BUILDER_CLASS = ModulePageBuilder - def __init__(self, *, full_name, py_object, **kwargs): + def __init__(self, *, api_node, **kwargs): """Initialize a `ModulePageInfo`. Args: @@ -70,7 +70,7 @@ def __init__(self, *, full_name, py_object, **kwargs): py_object: The object being documented. **kwargs: Extra arguments. """ - super().__init__(full_name, py_object, **kwargs) + super().__init__(api_node, **kwargs) self._modules = [] self._classes = [] @@ -137,39 +137,24 @@ def _add_member(self, member_info: base_page.MemberInfo) -> None: elif obj_type is obj_type_lib.ObjType.OTHER: self._add_other_member(member_info) - def collect_docs(self, parser_config): + def collect_docs(self): """Collect information necessary specifically for a module's doc page. Mainly this is information about the members of the module. - - Args: - parser_config: An instance of config.ParserConfig. """ - - member_names = parser_config.tree.get(self.full_name, []) - for member_short_name in member_names: - - if member_short_name in [ - '__builtins__', '__doc__', '__file__', '__name__', '__path__', - '__package__', '__cached__', '__loader__', '__spec__', - 'absolute_import', 'division', 'print_function', 'unicode_literals' - ]: - continue - - if self.full_name: - member_full_name = self.full_name + '.' + member_short_name - else: - member_full_name = member_short_name - - member = parser_config.py_name_to_object(member_full_name) - - member_doc = parser.parse_md_docstring(member, self.full_name, - parser_config, self._extra_docs) - - url = parser_config.reference_resolver.reference_to_url(member_full_name) - - member_info = base_page.MemberInfo(member_short_name, member_full_name, - member, member_doc, url) + # the path_tree has nodes for all api-paths, not just the prefered paths. + module_path_node = self.parser_config.path_tree[self.api_node.path] + for (_, path_node) in sorted(module_path_node.children.items()): + member_doc = parser.parse_md_docstring(path_node.py_object, + self.full_name, self.parser_config, + self._extra_docs) + + url = self.parser_config.reference_resolver.reference_to_url( + path_node.full_name) + + member_info = base_page.MemberInfo(path_node.short_name, + path_node.full_name, + path_node.py_object, member_doc, url) self._add_member(member_info) diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py b/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py index 583d9bb7c27..27292a0306a 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/pretty_docs_test.py @@ -20,6 +20,8 @@ from tensorflow_docs.api_generator import doc_controls from tensorflow_docs.api_generator import parser +from tensorflow_docs.api_generator import doc_generator_visitor + from tensorflow_docs.api_generator.pretty_docs import base_page from tensorflow_docs.api_generator.pretty_docs import function_page @@ -76,8 +78,10 @@ def _get_test_page_builder(self, search_hints): def test_function(): pass + api_node = doc_generator_visitor.ApiTreeNode( + path=('abc',), py_object=test_function, children={}) page_info = function_page.FunctionPageInfo( - full_name='abc', py_object=test_function, search_hints=search_hints) + api_node=api_node, search_hints=search_hints) docstring_info = parser.DocstringInfo( brief='hello `tensorflow`', docstring_parts=['line1', 'line2'], diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py index daef3c77a61..0fdc6dd0f28 100644 --- a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py +++ b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py @@ -50,7 +50,7 @@ class TypeAliasPageInfo(base_page.PageInfo): """ DEFAULT_BUILDER_CLASS = TypeAliasPageBuilder - def __init__(self, *, full_name: str, py_object: Any, **kwargs) -> None: + def __init__(self, *, api_node, **kwargs) -> None: """Initialize a `TypeAliasPageInfo`. Args: @@ -59,7 +59,7 @@ def __init__(self, *, full_name: str, py_object: Any, **kwargs) -> None: **kwargs: Extra arguments. """ - super().__init__(full_name, py_object, **kwargs) + super().__init__(api_node, **kwargs) self._signature = None @property @@ -101,7 +101,7 @@ def _link_type_args(self, obj: Any, reverse_index: Dict[int, str], else: return typing._type_repr(obj) # pylint: disable=protected-access # pytype: disable=module-attr - def collect_docs(self, parser_config) -> None: + def collect_docs(self) -> None: """Collect all information necessary to genertate the function page. Mainly this is details about the function signature. @@ -125,19 +125,17 @@ def collect_docs(self, parser_config) -> None: ``` X = Union[int, str, bool, tf.Tensor, np.ndarray] ``` - - Args: - parser_config: The config.ParserConfig for the module being documented. """ assert self.signature is None - linker = signature_lib.FormatArguments(parser_config=parser_config) + linker = signature_lib.FormatArguments(parser_config=self.parser_config) sig_args = [] if self.py_object.__origin__: for arg_obj in self.py_object.__args__: sig_args.append( - self._link_type_args(arg_obj, parser_config.reverse_index, linker)) + self._link_type_args(arg_obj, self.parser_config.reverse_index, + linker)) sig_args_str = textwrap.indent(',\n'.join(sig_args), ' ') if self.py_object.__origin__: diff --git a/tools/tensorflow_docs/api_generator/report/linter_test.py b/tools/tensorflow_docs/api_generator/report/linter_test.py index c06d262bf07..5ae283a5d8a 100644 --- a/tools/tensorflow_docs/api_generator/report/linter_test.py +++ b/tools/tensorflow_docs/api_generator/report/linter_test.py @@ -22,6 +22,7 @@ from absl.testing import absltest from tensorflow_docs.api_generator import config +from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib @@ -126,10 +127,11 @@ def _build_page_info(self): code_url_prefix='https://tensorflow.org') parser_config = generator.run_extraction() + + api_node = doc_generator_visitor.ApiTreeNode( + path=('m', 'TestClass'), py_object=TestClass) return docs_for_object.docs_for_object( - full_name='m.TestClass', - py_object=TestClass, - parser_config=parser_config) + api_node=api_node, parser_config=parser_config) def _make_report(self): page_info = self._build_page_info() @@ -145,6 +147,12 @@ def test_fill_report_doesnt_edit_page(self): test_api_report = utils.ApiReport() test_api_report.fill_metrics(page2) + page1.api_node = None + page1.parser_config = None + + page2.api_node = None + page2.parser_config = None + self.assertEqual(page1, page2) def test_class_raises_lint(self): diff --git a/tools/tensorflow_docs/api_generator/report/utils.py b/tools/tensorflow_docs/api_generator/report/utils.py index 3166cd994f4..821cc14bbad 100644 --- a/tools/tensorflow_docs/api_generator/report/utils.py +++ b/tools/tensorflow_docs/api_generator/report/utils.py @@ -77,9 +77,7 @@ def _make_constructor_info( """Convert a class description into a description of the constructor.""" methods = class_page.split_methods(class_page_info.methods) - constructor_info = base_page.PageInfo( - full_name=class_page_info.full_name, - py_object=class_page_info.py_object) + constructor_info = base_page.PageInfo(api_node=class_page_info.api_node) # Replace the class py_object with constructors py_object. This is done # because each method is linted separately and class py_object contains the diff --git a/tools/tensorflow_docs/api_generator/signature_test.py b/tools/tensorflow_docs/api_generator/signature_test.py index a0eddca2f82..e988da28cc0 100644 --- a/tools/tensorflow_docs/api_generator/signature_test.py +++ b/tools/tensorflow_docs/api_generator/signature_test.py @@ -24,6 +24,7 @@ from absl.testing import parameterized from tensorflow_docs.api_generator import config +from tensorflow_docs.api_generator import doc_generator_visitor from tensorflow_docs.api_generator import generate_lib from tensorflow_docs.api_generator import parser from tensorflow_docs.api_generator import reference_resolver as reference_resolver_lib @@ -264,39 +265,49 @@ def test_dataclasses_type_annotations(self): ]""")), ) # pyformat: disable def test_type_alias_signature(self, alias, expected_sig): - info_obj = type_alias_page.TypeAliasPageInfo( - full_name='tfdocs.api_generator.generate_lib.DocGenerator', + api_node = doc_generator_visitor.ApiTreeNode( + path=tuple('tfdocs.api_generator.generate_lib.DocGenerator'.split('.')), py_object=alias) + info_obj = type_alias_page.TypeAliasPageInfo( + api_node=api_node, parser_config=self.parser_config) with self.parser_config.reference_resolver.temp_prefix('../../..'): - info_obj.collect_docs(self.parser_config) + info_obj.collect_docs() self.assertEqual(info_obj.signature, expected_sig) - def _setup_class_info(self, cls, method_name): - pc = self.parser_config - pc.tree['x.Cls'] = [method_name] - full_name = f'x.Cls.{method_name}' - pc.index[full_name] = getattr(cls, method_name) - pc.reference_resolver._duplicate_of[full_name] = full_name - pc.reference_resolver._is_fragment[full_name] = True - pc.reference_resolver._all_names.add(full_name) - pc.reference_resolver._link_prefix = '../..' - - info = class_page.ClassPageInfo(full_name='x.Cls', py_object=cls) + def _setup_class_info(self, cls): + self.known_object = object() + + x = types.ModuleType('x') + x.__file__ = __file__ + x.Cls = cls + + generator = generate_lib.DocGenerator( + root_title='test', + py_modules=[('x', x)], + code_url_prefix='https://tensorflow.org') + + parser_config = generator.run_extraction() + parser_config.reference_resolver = ( + parser_config.reference_resolver.with_prefix('/')) + + api_node = parser_config.api_tree['x', 'Cls'] + info = class_page.ClassPageInfo( + api_node=api_node, parser_config=parser_config) info._doc = parser.DocstringInfo('doc', ['doc'], {}) - info.collect_docs(self.parser_config) + info.collect_docs() return info def test_signature_method_wrong_self_name(self): - # Calling these classes all `Cls` confuses inspect.getsource. - # Use unique names. + # Calling these classes all `Cls` confuses get_source, you need to + # use unique names. class Cls1: def method(x): # pylint: disable=no-self-argument pass - info = self._setup_class_info(Cls1, 'method') + info = self._setup_class_info(Cls1) self.assertEqual('()', str(info.methods[0].signature)) def test_signature_method_star_args(self): @@ -306,7 +317,7 @@ class Cls2: def method(*args): # pylint: disable=no-method-argument pass - info = self._setup_class_info(Cls2, 'method') + info = self._setup_class_info(Cls2) self.assertEqual('(\n *args\n)', str(info.methods[0].signature)) def test_signature_classmethod_wrong_cls_name(self): @@ -317,7 +328,7 @@ class Cls3: def method(x): # pylint: disable=bad-classmethod-argument pass - info = self._setup_class_info(Cls3, 'method') + info = self._setup_class_info(Cls3) self.assertEqual('()', str(info.methods[0].signature)) def test_signature_staticmethod(self): @@ -328,7 +339,7 @@ class Cls4: def method(x): pass - info = self._setup_class_info(Cls4, 'method') + info = self._setup_class_info(Cls4) self.assertEqual('(\n x\n)', str(info.methods[0].signature)) def test_signature_new(self): @@ -338,7 +349,7 @@ class Cls5: def __new__(x): # pylint: disable=bad-classmethod-argument pass - info = self._setup_class_info(Cls5, '__new__') + info = self._setup_class_info(Cls5) self.assertEqual('()', str(info.methods[0].signature)) def test_signature_dataclass_auto_init(self): @@ -348,9 +359,11 @@ class Cls6: a: Optional[int] b: Optional[str] - info = self._setup_class_info(Cls6, '__init__') + info = self._setup_class_info(Cls6) + builder = info.DEFAULT_BUILDER_CLASS(info) + self.assertEqual('(\n a: Optional[int], b: Optional[str]\n)', - str(info.methods[0].signature)) + str(builder.methods.constructor.signature)) def test_signature_dataclass_custom_init(self): @@ -363,9 +376,10 @@ def __init__(self, x: Optional[Union[int, str]]): self.a = int(x) self.b = str(x) - info = self._setup_class_info(Cls7, '__init__') + info = self._setup_class_info(Cls7) + builder = info.DEFAULT_BUILDER_CLASS(info) self.assertEqual('(\n x: Optional[Union[int, str]]\n)', - str(info.methods[0].signature)) + str(builder.methods.constructor.signature)) def test_dataclass_default_uses_ast_repr(self): From dec90d127636386544b8faa28cfcea8581f99da5 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Thu, 28 Apr 2022 06:57:04 +0100 Subject: [PATCH 098/872] Lint the Install TensorFlow for C guide --- site/en/install/lang_c.ipynb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb index df8e3c950c2..7d431c74126 100644 --- a/site/en/install/lang_c.ipynb +++ b/site/en/install/lang_c.ipynb @@ -81,15 +81,15 @@ "id": "Vk--31hqIwSV" }, "source": [ - "## Nightly Libtensorflow C packages\n", + "## Nightly libtensorflow C packages\n", "\n", - "Libtensorflow packages are built nightly and uploaded to GCS for all supported\n", + "libtensorflow packages are built nightly and uploaded to GCS for all supported\n", "platforms. They are uploaded to the\n", "[libtensorflow-nightly GCS bucket](https://storage.googleapis.com/libtensorflow-nightly)\n", "and are indexed by operating system and date built. For MacOS and Linux shared\n", - "objects, we have a\n", + "objects, there is a\n", "[script](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/builds/libtensorflow_nightly_symlink.sh)\n", - "that renames the .so files versioned to the current date copied into the\n", + "that renames the `.so` files versioned to the current date copied into the\n", "directory with the artifacts." ] }, @@ -123,7 +123,7 @@ "id": "y50y01XUFVb2" }, "source": [ - "### Download & extract\n", + "### Download and extract\n", "\n", "\n", " \n", From 304103ff5270931ad052d60086587f8ffbf9a331 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Thu, 28 Apr 2022 07:58:38 +0100 Subject: [PATCH 099/872] Lint the Customization basics tensors and operations tutorial --- site/en/tutorials/customization/basics.ipynb | 28 +++++++++++--------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb index d23b7ca2477..ae9f6c9a6a3 100644 --- a/site/en/tutorials/customization/basics.ipynb +++ b/site/en/tutorials/customization/basics.ipynb @@ -70,10 +70,10 @@ "source": [ "This is an introductory TensorFlow tutorial that shows how to:\n", "\n", - "* Import the required package\n", - "* Create and use tensors\n", - "* Use GPU acceleration\n", - "* Demonstrate `tf.data.Dataset`" + "* Import the required package.\n", + "* Create and use tensors.\n", + "* Use GPU acceleration.\n", + "* Build a data pipeline with `tf.data.Dataset`." ] }, { @@ -106,7 +106,7 @@ "source": [ "## Tensors\n", "\n", - "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations ([tf.math.add](https://www.tensorflow.org/api_docs/python/tf/add), [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/matmul), [tf.linalg.inv](https://www.tensorflow.org/api_docs/python/tf/linalg/inv) etc.) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types, for example:\n" + "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types. For example:\n" ] }, { @@ -168,7 +168,7 @@ "id": "Dwi1tdW3JBw6" }, "source": [ - "### NumPy Compatibility\n", + "### NumPy compatibility\n", "\n", "Converting between a TensorFlow `tf.Tensor`s and a NumPy `ndarray` is easy:\n", "\n", @@ -210,7 +210,7 @@ "source": [ "## GPU acceleration\n", "\n", - "Many TensorFlow operations are accelerated using the GPU for computation. Without any annotations, TensorFlow automatically decides whether to use the GPU or CPU for an operation—copying the tensor between CPU and GPU memory, if necessary. Tensors produced by an operation are typically backed by the memory of the device on which the operation executed, for example:" + "Many TensorFlow operations are accelerated using the GPU for computation. Without any annotations, TensorFlow automatically decides whether to use the GPU or CPU for an operation—copying the tensor between CPU and GPU memory, if necessary. Tensors produced by an operation are typically backed by the memory of the device on which the operation executed. For example:" ] }, { @@ -237,7 +237,7 @@ "id": "vpgYzgVXW2Ud" }, "source": [ - "### Device Names\n", + "### Device names\n", "\n", "The `Tensor.device` property provides a fully qualified string name of the device hosting the contents of the tensor. This name encodes many details, such as an identifier of the network address of the host on which this program is executing and the device within that host. This is required for distributed execution of a TensorFlow program. The string ends with `GPU:` if the tensor is placed on the `N`-th GPU on the host." ] @@ -248,9 +248,11 @@ "id": "ZWZQCimzuqyP" }, "source": [ - "### Explicit Device Placement\n", + "### Explicit device placement\n", "\n", - "In TensorFlow, *placement* refers to how individual operations are assigned (placed on) a device for execution. As mentioned, when there is no explicit guidance provided, TensorFlow automatically decides which device to execute an operation and copies tensors to that device, if needed. However, TensorFlow operations can be explicitly placed on specific devices using the `tf.device` context manager, for example:" + "In TensorFlow, *placement* refers to how individual operations are assigned (placed on) a device for execution. As mentioned, when there is no explicit guidance provided, TensorFlow automatically decides which device to execute an operation and copies tensors to that device, if needed.\n", + "\n", + "However, TensorFlow operations can be explicitly placed on specific devices using the `tf.device` context manager. For example:" ] }, { @@ -296,7 +298,7 @@ "source": [ "## Datasets\n", "\n", - "This section uses the [`tf.data.Dataset` API](https://www.tensorflow.org/guide/datasets) to build a pipeline for feeding data to your model. The `tf.data.Dataset` API is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops." + "This section uses the [`tf.data.Dataset` API](https://www.tensorflow.org/guide/datasets) to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops." ] }, { @@ -307,7 +309,7 @@ "source": [ "### Create a source `Dataset`\n", "\n", - "Create a *source* dataset using one of the factory functions like [`Dataset.from_tensors`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensors), [`Dataset.from_tensor_slices`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensor_slices), or using objects that read from files like [`TextLineDataset`](https://www.tensorflow.org/api_docs/python/tf/data/TextLineDataset) or [`TFRecordDataset`](https://www.tensorflow.org/api_docs/python/tf/data/TFRecordDataset). See the [TensorFlow Dataset guide](https://www.tensorflow.org/guide/datasets#reading_input_data) for more information." + "Create a *source* dataset using one of the factory functions like `tf.data.Dataset.from_tensors`, `tf.data.Dataset.from_tensor_slices`, or using objects that read from files like `tf.data.TextLineDataset` or `tf.data.TFRecordDataset`. Refer to the _Reading input data_ section of the [tf.data: Build TensorFlow input pipelines](https://www.tensorflow.org/guide/datasets) guide for more information." ] }, { @@ -341,7 +343,7 @@ "source": [ "### Apply transformations\n", "\n", - "Use the transformations functions like [`map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map), [`batch`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#batch), and [`shuffle`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle) to apply transformations to dataset records." + "Use the transformations functions like `tf.data.Dataset.map`, `tf.data.Dataset.batch`, and `tf.data.Dataset.shuffle` to apply transformations to dataset records." ] }, { From f1408b6d084f6836705d05801746ae8cc821c26d Mon Sep 17 00:00:00 2001 From: tfdocsbot Date: Thu, 28 Apr 2022 06:59:15 +0000 Subject: [PATCH 100/872] nbfmt --- site/en/guide/migrate/metrics_optimizers.ipynb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/site/en/guide/migrate/metrics_optimizers.ipynb b/site/en/guide/migrate/metrics_optimizers.ipynb index dea0d11dc3a..1a5f09cf7d0 100644 --- a/site/en/guide/migrate/metrics_optimizers.ipynb +++ b/site/en/guide/migrate/metrics_optimizers.ipynb @@ -370,8 +370,7 @@ "metadata": { "colab": { "collapsed_sections": [], - "name": "metrics.ipynb", - "provenance": [], + "name": "metrics_optimizers.ipynb", "toc_visible": true }, "kernelspec": { From 8cd8f7b1ea275cac76f527e096f2edb15fad2a72 Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Thu, 28 Apr 2022 22:01:27 +0100 Subject: [PATCH 101/872] Update tf.math and tf.linalg op names in Customization basics tensors and operations tutorial --- site/en/tutorials/customization/basics.ipynb | 22 ++++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb index ae9f6c9a6a3..a541062cb23 100644 --- a/site/en/tutorials/customization/basics.ipynb +++ b/site/en/tutorials/customization/basics.ipynb @@ -106,7 +106,7 @@ "source": [ "## Tensors\n", "\n", - "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types. For example:\n" + "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.linalg.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types. For example:\n" ] }, { @@ -118,13 +118,13 @@ }, "outputs": [], "source": [ - "print(tf.add(1, 2))\n", - "print(tf.add([1, 2], [3, 4]))\n", - "print(tf.square(5))\n", - "print(tf.reduce_sum([1, 2, 3]))\n", + "print(tf.math.add(1, 2))\n", + "print(tf.math.add([1, 2], [3, 4]))\n", + "print(tf.math.square(5))\n", + "print(tf.math.reduce_sum([1, 2, 3]))\n", "\n", "# Operator overloading is also supported\n", - "print(tf.square(2) + tf.square(3))" + "print(tf.math.square(2) + tf.math.square(3))" ] }, { @@ -144,7 +144,7 @@ }, "outputs": [], "source": [ - "x = tf.matmul([[1]], [[2, 3]])\n", + "x = tf.linalg.matmul([[1]], [[2, 3]])\n", "print(x)\n", "print(x.shape)\n", "print(x.dtype)" @@ -191,11 +191,11 @@ "ndarray = np.ones([3, 3])\n", "\n", "print(\"TensorFlow operations convert numpy arrays to Tensors automatically\")\n", - "tensor = tf.multiply(ndarray, 42)\n", + "tensor = tf.math.multiply(ndarray, 42)\n", "print(tensor)\n", "\n", "\n", - "print(\"And NumPy operations convert Tensors to numpy arrays automatically\")\n", + "print(\"And NumPy operations convert Tensors to NumPy arrays automatically\")\n", "print(np.add(tensor, 1))\n", "\n", "print(\"The .numpy() method explicitly converts a Tensor to a numpy array\")\n", @@ -268,7 +268,7 @@ "def time_matmul(x):\n", " start = time.time()\n", " for loop in range(10):\n", - " tf.matmul(x, x)\n", + " tf.linalg.matmul(x, x)\n", "\n", " result = time.time()-start\n", "\n", @@ -354,7 +354,7 @@ }, "outputs": [], "source": [ - "ds_tensors = ds_tensors.map(tf.square).shuffle(2).batch(2)\n", + "ds_tensors = ds_tensors.map(tf.math.square).shuffle(2).batch(2)\n", "\n", "ds_file = ds_file.batch(2)" ] From afba8cd7e87089b24e3a735d7578279835e90b07 Mon Sep 17 00:00:00 2001 From: tfdocsbot Date: Thu, 28 Apr 2022 21:02:07 +0000 Subject: [PATCH 102/872] nbfmt --- site/en/install/lang_c.ipynb | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb index cdc62873df3..f88772e4052 100644 --- a/site/en/install/lang_c.ipynb +++ b/site/en/install/lang_c.ipynb @@ -99,8 +99,6 @@ "id": "qowtdsijFMYZ" }, "source": [ - "\n", - "\n", "## Supported Platforms\n", "\n", "TensorFlow for C is supported on the following systems:\n", @@ -125,7 +123,6 @@ "id": "y50y01XUFVb2" }, "source": [ - "\n", "### Download & extract\n", "\n", "
TensorFlow C libraryURL
\n", @@ -188,7 +185,6 @@ "id": "fcBJDdojJDyk" }, "source": [ - "\n", "### Linker\n", "\n", "On Linux/macOS, if you extract the TensorFlow C library to a system directory,\n", @@ -266,12 +262,6 @@ "cell_type": "code", "execution_count": null, "metadata": { - "attributes": { - "classes": [ - "c" - ], - "id": "" - }, "id": "b5851f1b" }, "outputs": [], @@ -351,9 +341,6 @@ "id": "ea5fd208" }, "source": [ - "\n", - "\n", - "\n", "## Build from source\n", "\n", "TensorFlow is open source. Read\n", @@ -365,9 +352,7 @@ "metadata": { "colab": { "collapsed_sections": [], - "name": "Copy of lang_c.ipynb", - "private_outputs": true, - "provenance": [], + "name": "lang_c.ipynb", "toc_visible": true }, "kernelspec": { From f2997c538414531a04ce1c730e705158bc98e93a Mon Sep 17 00:00:00 2001 From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com> Date: Thu, 28 Apr 2022 22:04:59 +0100 Subject: [PATCH 103/872] Update link to tf.data Build TensorFlow input pipelines guide --- site/en/tutorials/customization/basics.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb index a541062cb23..de651d2ac50 100644 --- a/site/en/tutorials/customization/basics.ipynb +++ b/site/en/tutorials/customization/basics.ipynb @@ -298,7 +298,7 @@ "source": [ "## Datasets\n", "\n", - "This section uses the [`tf.data.Dataset` API](https://www.tensorflow.org/guide/datasets) to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops." + "This section uses the [`tf.data.Dataset` API](../../guide/data.ipynb) to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops." ] }, { @@ -309,7 +309,7 @@ "source": [ "### Create a source `Dataset`\n", "\n", - "Create a *source* dataset using one of the factory functions like `tf.data.Dataset.from_tensors`, `tf.data.Dataset.from_tensor_slices`, or using objects that read from files like `tf.data.TextLineDataset` or `tf.data.TFRecordDataset`. Refer to the _Reading input data_ section of the [tf.data: Build TensorFlow input pipelines](https://www.tensorflow.org/guide/datasets) guide for more information." + "Create a *source* dataset using one of the factory functions like `tf.data.Dataset.from_tensors`, `tf.data.Dataset.from_tensor_slices`, or using objects that read from files like `tf.data.TextLineDataset` or `tf.data.TFRecordDataset`. Refer to the _Reading input data_ section of the [tf.data: Build TensorFlow input pipelines](../../guide/data.ipynb) guide for more information." ] }, { From 4be05178be80b4a0e41042186bdab3b263895a8a Mon Sep 17 00:00:00 2001 From: Soo Sung Date: Thu, 28 Apr 2022 17:06:05 -0700 Subject: [PATCH 104/872] DTensor guide and tutorials (ml and keras) PiperOrigin-RevId: 445285166 --- site/en/guide/_toc.yaml | 4 +- site/en/guide/dtensor_overview.ipynb | 1063 ++++++++++++++++ site/en/tutorials/_toc.yaml | 6 + .../distribute/dtensor_keras_tutorial.ipynb | 772 ++++++++++++ .../distribute/dtensor_ml_tutorial.ipynb | 1100 +++++++++++++++++ 5 files changed, 2944 insertions(+), 1 deletion(-) create mode 100644 site/en/guide/dtensor_overview.ipynb create mode 100644 site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb create mode 100644 site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml index 85ea358bc28..3cf9913c115 100644 --- a/site/en/guide/_toc.yaml +++ b/site/en/guide/_toc.yaml @@ -30,12 +30,14 @@ toc: path: /guide/ragged_tensor - title: "Sparse tensor" path: /guide/sparse_tensor - - title: "Random number generation" path: /guide/random_numbers - title: "NumPy API" status: experimental path: /guide/tf_numpy +- title: "DTensor concepts" + path: /guide/dtensor_overview + status: nightly - title: "Thinking in TensorFlow 2" path: /guide/effective_tf2 diff --git a/site/en/guide/dtensor_overview.ipynb b/site/en/guide/dtensor_overview.ipynb new file mode 100644 index 00000000000..3c88bb70d2d --- /dev/null +++ b/site/en/guide/dtensor_overview.ipynb @@ -0,0 +1,1063 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "1ljvLya59ep5" + }, + "source": [ + "##### Copyright 2019 The TensorFlow Authors.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tuOe1ymfHZPu" + }, + "outputs": [], + "source": [ + "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VcQIa1uG86Wh" + }, + "source": [ + "# DTensor Concepts" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6dWNQEum9AfY" + }, + "source": [ + "
\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " View on TensorFlow.org\n", + " \n", + " Run in Google Colab\n", + " \n", + " View source on GitHub\n", + " \n", + " Download notebook\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MGZuakHVlVQf" + }, + "source": [ + "\n", + "## Overview\n", + "\n", + "This colab introduces DTensor, an extension to TensorFlow for synchronous distributed computing.\n", + "\n", + "DTensor provides a global programming model that allows developers to compose applications that operate on Tensors globally while managing the distribution across devices internally. DTensor distributes the program and tensors according to the sharding directives through a procedure called *[Single program, multiple data (SPMD)](https://en.wikipedia.org/wiki/SPMD) expansion*.\n", + "\n", + "By decoupling the application from sharding directives, DTensor enables running the same application on a single device, multiple devices, or even multiple clients, while preserving its global semantics. \n", + "\n", + "This guide introduces DTensor concepts for distributed computing, and how DTensor integrates with TensorFlow. To see a demo of using DTensor in model training, see [Distributed training with DTensor](https://www.tensorflow.org/tutorials/distribute/dtensor_ml_tutorial.ipynb) tutorial." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "h7ZTDq7KngwA" + }, + "source": [ + "## Setup\n", + "\n", + "DTensor is part of TensorFlow 2.9.0 release, and also included in the TensorFlow nightly builds since 04/09/2022." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OKaPw8vwwZAC" + }, + "outputs": [], + "source": [ + "!pip install --quiet --upgrade --pre tensorflow" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O3pG29uZIWYO" + }, + "source": [ + "Once installed, import `tensorflow` and `tf.experimental.dtensor`. Then configure TensorFlow to use 6 virtual CPUs.\n", + "\n", + "Even though this example uses vCPUs, DTensor works the same way on CPU, GPU or TPU devices." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Q92lo0zjwej8" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "from tensorflow.experimental import dtensor\n", + "\n", + "print('TensorFlow version:', tf.__version__)\n", + "\n", + "def configure_virtual_cpus(ncpu):\n", + " phy_devices = tf.config.list_physical_devices('CPU')\n", + " tf.config.set_logical_device_configuration(phy_devices[0], [\n", + " tf.config.LogicalDeviceConfiguration(),\n", + " ] * ncpu)\n", + " \n", + "configure_virtual_cpus(6)\n", + "DEVICES = [f'CPU:{i}' for i in range(6)]\n", + "\n", + "tf.config.list_logical_devices('CPU')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O-lsrxUnlsCC" + }, + "source": [ + "## DTensor's model of distributed tensors\n", + "\n", + "DTensor introduces two concepts: `dtensor.Mesh` and `dtensor.Layout`. They are abstractions to model the sharding of tensors across topologically related devices.\n", + "\n", + "- `Mesh` defines the device list for computation. \n", + "- `Layout` defines how to shard the Tensor dimension on a `Mesh`. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JjiHaH0ql9yo" + }, + "source": [ + "\n", + "### Mesh\n", + "\n", + "`Mesh` represents a logical Cartisian topology of a set of devices. Each dimension of the Cartisian grid is called a **Mesh dimension**, and referred to with a name. Names of mesh dimension within the same `Mesh` must be unique.\n", + "\n", + "Names of mesh dimensions are referenced by `Layout` to describe the sharding behavior of a `tf.Tensor` along each of its axes. This is described in more detail later in the section on `Layout`.\n", + "\n", + "`Mesh` can be thought of as a multi-dimensional array of devices." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_J6cOieEbaUw" + }, + "source": [ + "\n", + "In a 1 dimensional `Mesh`, all devices form a list in a single mesh dimension. The following example uses `dtensor.create_mesh` to create a mesh from 6 CPU devices along a mesh dimension `'x'` with a size of 6 devices:\n", + "\n", + "\"A\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QLH5fgdBmA58" + }, + "outputs": [], + "source": [ + "mesh_1d = dtensor.create_mesh([('x', 6)], devices=DEVICES)\n", + "print(mesh_1d)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hSZwaUwnEgXB" + }, + "source": [ + "A `Mesh` can be multi dimensional as well. In the following example, 6 CPU devices form a `3x2` mesh, where the `'x'` mesh dimension has a size of 3 devices, and the `'y'` mesh dimension has a size of 2 devices:\n", + "\n", + "\"A" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "op6TmKUQE-sZ" + }, + "outputs": [], + "source": [ + "mesh_2d = dtensor.create_mesh([('x', 3), ('y', 2)], devices=DEVICES)\n", + "print(mesh_2d)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "deAqdrDPFn2f" + }, + "source": [ + "### Layout\n", + "\n", + "**`Layout`** specifies how a tensor is distributed, or sharded, on a `Mesh`. In DTensor, the placement specification is on a per-axis bases. An axis of a `Layout` can be either `sharded` or `unsharded` (replicated) along a mesh dimension.\n", + "\n", + "Note: In order to avoid confusions between `Mesh` and `Layout`, the term *dimension* is always associated with `Mesh`, and the term *axis* with `Tensor` and `Layout` in this guide. \n", + "\n", + "The rank of a `Layout` and the number of dimensions of a `Mesh` do not need to match. The `unsharded` axes of a `Layout` do not need to be associated to a mesh dimension, and `unsharded` mesh dimensions do not need to be associated with a `layout` axis.\n", + "\n", + "On the other hand, the rank of `Layout` should be the same as the rank of the `Tensor` where the `Layout` is applied.\n", + "\n", + "\"Diagram" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Px_bF1c-bQ7e" + }, + "source": [ + "Let's analyze a few examples of `Layout` for the `Mesh`'s created in the previous section." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fqzCNlWAbm-c" + }, + "source": [ + "\n", + "On a 1-dimensional mesh such as `[(\"x\", 6)]` (`mesh_1d` in the previous section), `Layout([\"unsharded\"], mesh_1d)` is a layout for a rank-1 tensor replicated on 6 devices.\n", + "\n", + "\"Layout" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-a3EnmZag6x1" + }, + "outputs": [], + "source": [ + "layout_rank_1_1d = dtensor.Layout([dtensor.UNSHARDED], mesh_1d)\n", + "print(layout_rank_1_1d)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1Kyg0V3ehMNJ" + }, + "source": [ + "Given a 2-dimensional 3x2 mesh such as `[(\"x\", 3), (\"y\", 2)]`, (`mesh_2d` from the previous section), `Layout([\"x\", dtensor.UNSHARDED], mesh_2d)` is a layout for a rank-2 `Tensor`, whose first axis is sharded on mesh dimension `x`, and second axis is replicated (`UNSHARDED`).\n", + "\n", + "\"Layout\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "p8OrehEuhPbS" + }, + "outputs": [], + "source": [ + "layout_rank_2_2d = dtensor.Layout(['x', dtensor.UNSHARDED], mesh_2d)\n", + "print(layout_rank_2_2d)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TTalu6M-ISYb" + }, + "source": [ + "### Single-Client and Multi-Client Applications\n", + "\n", + "DTensor supports both single-client and multi-client applications. The colab Python kernel is an example of a single client DTensor application, where there is a single Python process.\n", + "\n", + "In a multi-client DTensor application, multiple Python processes collectively perform as a coherent application. The Cartisian grid of a `Mesh` in a multi-client DTensor application can span across devices regardless of whether they are attached locally to the current client or attached remotely to another client. The set of all devices used by a `Mesh` are called the *global device list*. \n", + "\n", + "The creation of a `Mesh` in a multi-client DTensor application is a collective operation where the *global device list* is identicial for all of the participating clients, and the creation of the `Mesh` serves as a global barrier. \n", + "\n", + "During `Mesh` creation, each client provides its *local device list* together with the expected *global device list*. DTensor validates that both lists are consistent. Please refer to the API documentation for `dtensor.create_mesh` and `dtensor.create_distributed_mesh`\n", + " for more information on multi-client mesh creation and the *global device list*.\n", + "\n", + "Single-client can be thought of as a special case of multi-client, with 1 client. In a single-client application, the *global device list* is identical to the *local device list*.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "P_F7DWkXkB4w" + }, + "source": [ + "## DTensor as a sharded tensor\n", + "\n", + "Now let's start coding with `DTensor`. The helper function, `dtensor_from_array`, demonstrates creating DTensors from something that looks like a `tf.Tensor`. The function performs 2 steps:\n", + " - Replicates the tensor to every device on the mesh.\n", + " - Shards the copy according to the layout requested in its arguments." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "s6aws-b8dN9L" + }, + "outputs": [], + "source": [ + "def dtensor_from_array(arr, layout, shape=None, dtype=None):\n", + " \"\"\"Convert a DTensor from something that looks like an array or Tensor.\n", + " \n", + " This function is convenient for quick doodling DTensors from a known,\n", + " unsharded data object in a single-client environment. This is not the\n", + " most efficient way of creating a DTensor, but it will do for this \n", + " tutorial.\n", + " \"\"\"\n", + " if shape is not None or dtype is not None:\n", + " arr = tf.constant(arr, shape=shape, dtype=dtype)\n", + " \n", + " # replicate the input to the mesh\n", + " a = dtensor.copy_to_mesh(arr, \n", + " layout=dtensor.Layout.replicated(layout.mesh, rank=layout.rank))\n", + " # shard the copy to the desirable layout\n", + " return dtensor.relayout(a, layout=layout)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r3o6IysrlGMu" + }, + "source": [ + "### Anatomy of a DTensor\n", + "\n", + "A DTensor is a `tf.Tensor` object, but augumented with the `Layout` annotation that defines its sharding behavior. A DTensor consists of the following:\n", + " \n", + " - Global tensor meta-data, including the global shape and dtype of the tensor.\n", + " - A `Layout`, which defines the `Mesh` the `Tensor` belongs to, and how the `Tensor` is sharded onto the `Mesh`.\n", + " - A list of **component tensors**, one item per local device in the `Mesh`. \n", + " \n", + "With `dtensor_from_array`, you can create your first DTensor, `my_first_dtensor`, and examine its contents." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mQu_nScGUvYH" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 6)], devices=DEVICES)\n", + "layout = dtensor.Layout([dtensor.UNSHARDED], mesh)\n", + "\n", + "my_first_dtensor = dtensor_from_array([0, 1], layout)\n", + "\n", + "# Examine the dtensor content\n", + "print(my_first_dtensor)\n", + "print(\"global shape:\", my_first_dtensor.shape)\n", + "print(\"dtype:\", my_first_dtensor.dtype)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r8LQy1nqmvFy" + }, + "source": [ + "#### Layout and `fetch_layout`\n", + "\n", + "The layout of a DTensor is not a regular attribute of `tf.Tensor`. Instead, DTensor provides a function, `dtensor.fetch_layout` to access the layout of a DTensor." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dCSFyaAjmzGu" + }, + "outputs": [], + "source": [ + "print(dtensor.fetch_layout(my_first_dtensor))\n", + "assert layout == dtensor.fetch_layout(my_first_dtensor)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ed7i3l2lmatm" + }, + "source": [ + "#### Component tensors, `pack` and `unpack`\n", + "\n", + "A DTensor consists of a list of **component tensors**. The component tensor for a device in the `Mesh` is the `Tensor` object representing the piece of the global DTensor that is stored on this device.\n", + "\n", + "A DTensor can be unpacked into component tensors through `dtensor.unpack`. You can make use of `dtensor.unpack` to inspect the components of the DTensor, and confirm they are on all devices of the `Mesh`.\n", + "\n", + "Note that the positions of component tensors in the global view may overlap each other. For example, in the case of a fully replicated layout, all components are identical replicas of the global tensor." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BGbjqVAOnXMk" + }, + "outputs": [], + "source": [ + "for component_tensor in dtensor.unpack(my_first_dtensor):\n", + " print(\"Device:\", component_tensor.device, \",\", component_tensor)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-tqIQM52k788" + }, + "source": [ + "As shown, `my_first_dtensor` is a tensor of `[0, 1]` replicated to all 6 devices." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6By3k-CGn3yv" + }, + "source": [ + "The inverse operation of `dtensor.unpack` is `dtensor.pack`. Component tensors can be packed back into a DTensor.\n", + "\n", + "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices. \n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9lT-6qQwxOgf" + }, + "outputs": [], + "source": [ + "packed_dtensor = dtensor.pack(\n", + " [[0, 1], [0, 1], [0, 1],\n", + " [0, 1], [0, 1], [0, 1]],\n", + " layout=layout\n", + ")\n", + "print(packed_dtensor)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zvS3autrpK2U" + }, + "source": [ + "### Sharding a DTensor to a Mesh\n", + "\n", + "So far you've worked with the `my_first_dtensor`, which is a rank-1 DTensor fully replicated across a dim-1 `Mesh`.\n", + "\n", + "Next create and inspect DTensors that are sharded across a dim-2 `Mesh`. The next example does this with a 3x2 `Mesh` on 6 CPU devices, where size of mesh dimension `'x'` is 3 devices, and and size of mesh dimension`'y'` is 2 devices." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KWb9Ae0VJ-Rc" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 3), (\"y\", 2)], devices=DEVICES)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ndSeQSFWKQk9" + }, + "source": [ + "#### Fully sharded rank-2 Tensor on a dim-2 Mesh\n", + "\n", + "Create a 3x2 rank-2 DTensor, sharding its first axis along the `'x'` mesh dimension, and its second axis along the `'y'` mesh dimension.\n", + "\n", + "- Because the tensor shape equals to the mesh dimension along all of the sharded axes, each device receives a single element of the DTensor.\n", + "- The rank of the component tensor is always the same as the rank of the global shape. DTensor adopts this convention as a simple way to preserve information for locating the relation between a component tensor and the global DTensor. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ax_ZHouJp1MX" + }, + "outputs": [], + "source": [ + "fully_sharded_dtensor = dtensor_from_array(\n", + " tf.reshape(tf.range(6), (3, 2)),\n", + " layout=dtensor.Layout([\"x\", \"y\"], mesh))\n", + "\n", + "for raw_component in dtensor.unpack(fully_sharded_dtensor):\n", + " print(\"Device:\", raw_component.device, \",\", raw_component)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zhsLC-NgrC2p" + }, + "source": [ + "#### Fully replicated rank-2 Tensor on a dim-2 Mesh\n", + "\n", + "For comparison, create a 3x2 rank-2 DTensor, fully replicated to the same dim-2 Mesh.\n", + "\n", + " - Because the DTensor is fully replicated, each device receives a full replica of the 3x2 DTensor.\n", + " - The rank of the component tensors are the same as the rank of the global shape -- this fact is trivial, because in this case, the shape of the component tensors are the same as the global shape anyway." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xmyC6H6Ec90P" + }, + "outputs": [], + "source": [ + "fully_replicated_dtensor = dtensor_from_array(\n", + " tf.reshape(tf.range(6), (3, 2)),\n", + " layout=dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh))\n", + "# Or, layout=tensor.Layout.fully_replicated(mesh, rank=2)\n", + "\n", + "for component_tensor in dtensor.unpack(fully_replicated_dtensor):\n", + " print(\"Device:\", component_tensor.device, \",\", component_tensor)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KWoyv_oHMzk1" + }, + "source": [ + "#### Hybrid rank-2 Tensor on a dim-2 Mesh\n", + "\n", + "What about somewhere between fully sharded and fully replicated?\n", + "\n", + "DTensor allows a `Layout` to be a hybrid, sharded along some axes, but replicated along others.\n", + "\n", + "For example, you can shard the same 3x2 rank-2 DTensor in the following way:\n", + " \n", + " - 1st axis sharded along the `'x'` mesh dimension.\n", + " - 2nd axis replicated along the `'y'` mesh dimension.\n", + " \n", + "To achieve this sharding scheme, you just need to replace the sharding spec of the 2nd axis from `'y'` to `dtensor.UNSHARDED`, to indicate your intention of replicating along the 2nd axis. The layout object will look like `Layout(['x', dtensor.UNSHARDED], mesh)`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DygnbkQ1Lu42" + }, + "outputs": [], + "source": [ + "hybrid_sharded_dtensor = dtensor_from_array(\n", + " tf.reshape(tf.range(6), (3, 2)),\n", + " layout=dtensor.Layout(['x', dtensor.UNSHARDED], mesh))\n", + "\n", + "for component_tensor in dtensor.unpack(hybrid_sharded_dtensor):\n", + " print(\"Device:\", component_tensor.device, \",\", component_tensor)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "T7FtZ9kQRZgE" + }, + "source": [ + "\n", + "You can inspect the component tensors of the created DTensor and verify they are indeed sharded according to your scheme. It may be helpful to illustrate the situation with a chart:\n", + "\n", + " \"A\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "auAkA38XjL-q" + }, + "source": [ + "#### Tensor.numpy() and sharded DTensor\n", + "\n", + "Be aware that calling the `.numpy()` method on a sharded DTensor raises an error. The rationale for erroring is to protect against unintended gathering of data from multiple computing devices to the host CPU device backing the returned numpy array. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "hNdwmnL0jAXS" + }, + "outputs": [], + "source": [ + "print(fully_replicated_dtensor.numpy())\n", + "\n", + "try:\n", + " fully_sharded_dtensor.numpy() \n", + "except tf.errors.UnimplementedError:\n", + " print(\"got an error as expected for fully_sharded_dtensor\")\n", + "\n", + "try:\n", + " hybrid_sharded_dtensor.numpy() \n", + "except tf.errors.UnimplementedError:\n", + " print(\"got an error as expected for hybrid_sharded_dtensor\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8WcMkiagPF_6" + }, + "source": [ + "## TensorFlow API on DTensor\n", + "\n", + "DTensor strives to be a drop-in replacement for tensor in your program. The TensorFlow Python API that consume `tf.Tensor`, such as the Ops library functions, `tf.function`, `tf.GradientTape`, also work with DTensor.\n", + "\n", + "To accomplish this, for each [TensorFlow Graph](https://www.tensorflow.org/guide/intro_to_graphs), DTensor produces and executes an equivalent [SPMD](https://en.wikipedia.org/wiki/SPMD) graph in a procedure called *SPMD expansion*. A few critical steps in DTensor SPMD expansion are:\n", + "\n", + " - Propagating the sharding `Layout` of DTensor in the TensorFlow graph\n", + " - Rewriting TensorFlow Ops on the global DTensor with equivalent TensorFlow Ops on the componenent tensors, inserting collective and communication Ops when necessary\n", + " - Lowering backend neutral TensorFlow Ops to backend specific TensorFlow Ops.\n", + "\n", + "The final result is that **DTensor is a drop-in replacement for Tensor**. \n", + "\n", + "Note: DTensor is still an experimental API which means you will be exploring and pushing the boundaries and limits of the DTensor programming model.\n", + "\n", + "There are 2 ways of triggering DTensor execution: \n", + " - DTensor as operands of a Python function, e.g. `tf.matmul(a, b)` will run through DTensor if `a`, `b`, or both are DTensors.\n", + " - Requesting the result of a Python function to be a DTensor, e.g. `dtensor.call_with_layout(tf.ones, layout, shape=(3, 2))` will run through DTensor because we requested the output of tf.ones to be sharded according to a `layout`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "urKzmqAoPssT" + }, + "source": [ + "### DTensor as Operands\n", + "\n", + "Many TensorFlow API functions take `tf.Tensor` as their operands, and returns `tf.Tensor` as their results. For these functions, you can express intention to run a function through DTensor by passing in DTensor as operands. This section uses `tf.matmul(a, b)` as an example." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7LO8ZT7iWVga" + }, + "source": [ + "#### Fully replicated input and output\n", + "\n", + "In this case, the DTensors are fully replicated. On each of the devices of the `Mesh`, \n", + " - the component tensor for operand `a` is `[[1, 2, 3], [4, 5, 6]]` (2x3)\n", + " - the component tensor for operand `b` is `[[6, 5], [4, 3], [2, 1]]` (3x2)\n", + " - the computation consists of a single `MatMul` of `(2x3, 3x2) -> 2x2`, \n", + " - the component tensor for result `c` is `[[20, 14], [56,41]]` (2x2)\n", + "\n", + "Total number of floating point mul operations is `6 device * 4 result * 3 mul = 72`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "TiZf2J9JNd2D" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 6)], devices=DEVICES)\n", + "layout = dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh)\n", + "a = dtensor_from_array([[1, 2, 3], [4, 5, 6]], layout=layout)\n", + "b = dtensor_from_array([[6, 5], [4, 3], [2, 1]], layout=layout)\n", + "\n", + "c = tf.matmul(a, b) # runs 6 identical matmuls in parallel on 6 devices\n", + "\n", + "# `c` is a DTensor replicated on all devices (same as `a` and `b`)\n", + "print('Sharding spec:', dtensor.fetch_layout(c).sharding_specs)\n", + "print(\"components:\")\n", + "for component_tensor in dtensor.unpack(c):\n", + " print(component_tensor.device, component_tensor.numpy())\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QXtR9qgKWgWV" + }, + "source": [ + "#### Sharding operands along the contracted axis\n", + "\n", + "You can reduce the amount of computation per device by sharding the operands `a` and `b`. A popular sharding scheme for `tf.matmul` is to shard the operands along the axis of the contraction, which means sharding `a` along the second axis, and `b` along the first axis.\n", + "\n", + "The global matrix product sharded under this scheme can be performed efficiently, by local matmuls that runs concurrently, followed by a collective reduction to aggregate the local results. This is also the [canonical way](https://github.com/open-mpi/ompi/blob/ee87ec391f48512d3718fc7c8b13596403a09056/docs/man-openmpi/man3/MPI_Reduce.3.rst?plain=1#L265) of implementing a distributed matrix dot product.\n", + "\n", + "Total number of floating point mul operations is `6 devices * 4 result * 1 = 24`, a factor of 3 reduction compared to the fully replicated case (72) above. The factor of 3 is due to the sharing along `x` mesh dimension with a size of `3` devices.\n", + "\n", + "The reduction of the number of operations run sequentially is the main mechansism with which synchronuous model parallelism accelerates training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "EyVAUvMePbms" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 3), (\"y\", 2)], devices=DEVICES)\n", + "a_layout = dtensor.Layout([dtensor.UNSHARDED, 'x'], mesh)\n", + "a = dtensor_from_array([[1, 2, 3], [4, 5, 6]], layout=a_layout)\n", + "b_layout = dtensor.Layout(['x', dtensor.UNSHARDED], mesh)\n", + "b = dtensor_from_array([[6, 5], [4, 3], [2, 1]], layout=b_layout)\n", + "\n", + "c = tf.matmul(a, b)\n", + "# `c` is a DTensor replicated on all devices (same as `a` and `b`)\n", + "print('Sharding spec:', dtensor.fetch_layout(c).sharding_specs)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IhD8yYgJiCEh" + }, + "source": [ + "#### Additional Sharding\n", + "\n", + "You can perform additional sharding on the inputs, and they are appropriately carried over to the results. For example, you can apply additional sharding of operand `a` along its first axis to the `'y'` mesh dimension. The additional sharding will be carried over to the first axis of the result `c`.\n", + "\n", + "\n", + "Total number of floating point mul operations is `6 devices * 2 result * 1 = 12`, an additional factor of 2 reduction compared to the case (24) above. The factor of 2 is due to the sharing along `y` mesh dimension with a size of `2` devices." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0PYqe0neiOpR" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 3), (\"y\", 2)], devices=DEVICES)\n", + "\n", + "a_layout = dtensor.Layout(['y', 'x'], mesh)\n", + "a = dtensor_from_array([[1, 2, 3], [4, 5, 6]], layout=a_layout)\n", + "b_layout = dtensor.Layout(['x', dtensor.UNSHARDED], mesh)\n", + "b = dtensor_from_array([[6, 5], [4, 3], [2, 1]], layout=b_layout)\n", + "\n", + "c = tf.matmul(a, b)\n", + "# The sharding of `a` on the first axis is carried to `c'\n", + "print('Sharding spec:', dtensor.fetch_layout(c).sharding_specs)\n", + "print(\"components:\")\n", + "for component_tensor in dtensor.unpack(c):\n", + " print(component_tensor.device, component_tensor.numpy())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c-1NazCVmLWZ" + }, + "source": [ + "### DTensor as Output\n", + "\n", + "What about Python functions that do not take operands, but returns a Tensor result that can be sharded? Examples of such functions are\n", + "\n", + " - `tf.ones`, `tf.zeros`, `tf.random.stateless_normal`, \n", + "\n", + "For these Python functions, DTensor provides `dtensor.call_with_layout` which eagelry executes a Python function with DTensor, and ensures that the returned Tensor is a DTensor with the requested `Layout`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "J0jo_8NPtJiO" + }, + "outputs": [], + "source": [ + "help(dtensor.call_with_layout)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V-YdLvfytM7g" + }, + "source": [ + "The eagerly executed Python function usually only contain a single non-trivial TensorFlow Op.\n", + "\n", + "To use a Python function that emits multiple TensorFlow Ops with `dtensor.call_with_layout`, the function should be converted to a `tf.function`. Calling a `tf.function` is a single TensorFlow Op. When the `tf.function` is called, DTensor can perform layout propagation when it analyzes the computing graph of the `tf.function`, before any of the intermediate tensors are materialized." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DLrksgFjqRLS" + }, + "source": [ + "#### APIs that emit a single TensorFlow Op\n", + "\n", + "If a function emits a single TensorFlow Op, you can directly apply `dtensor.call_with_layout` to the function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "G1CuKYSFtFeM" + }, + "outputs": [], + "source": [ + "help(tf.ones)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2m_EAwy-ozOh" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 3), (\"y\", 2)], devices=DEVICES)\n", + "ones = dtensor.call_with_layout(tf.ones, dtensor.Layout(['x', 'y'], mesh), shape=(6, 4))\n", + "print(ones)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bx-7Xo8Cpb8S" + }, + "source": [ + "#### APIs that emit multiple TensorFlow Ops\n", + "\n", + "If the API emits multiple TensorFlow Ops, convert the function into a single Op through `tf.function`. For example `tf.random.stateleess_normal`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "H8BQSTRFtCih" + }, + "outputs": [], + "source": [ + "help(tf.random.stateless_normal)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "TvP81eYopSPm" + }, + "outputs": [], + "source": [ + "ones = dtensor.call_with_layout(\n", + " tf.function(tf.random.stateless_normal), \n", + " dtensor.Layout(['x', 'y'], mesh), \n", + " shape=(6, 4), \n", + " seed=(1, 1))\n", + "print(ones)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qKoojp9ZyWzW" + }, + "source": [ + "Wrapping a Python function that emits a single TensorFlow Op with `tf.function` is allowed. The only caveat is paying the associated cost and complexity of creating a `tf.function` from a Python function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LbAtKrSkpOaq" + }, + "outputs": [], + "source": [ + "ones = dtensor.call_with_layout(\n", + " tf.function(tf.ones), \n", + " dtensor.Layout(['x', 'y'], mesh), \n", + " shape=(6, 4))\n", + "print(ones)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "D-m1816JP3CE" + }, + "source": [ + "### From `tf.Variable` to `dtensor.DVariable`\n", + "\n", + "In Tensorflow, `tf.Variable` is the holder for a mutable `Tensor` value.\n", + "With DTensor, the corresponding variable semantics is provided by `dtensor.DVariable`. \n", + "\n", + "The reason a new type `DVariable` was introduced for DTensor variable is because DVariables have an additional requirement that the layout cannot change from its initial value." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "awRPuR26P0Sc" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"x\", 6)], devices=DEVICES)\n", + "layout = dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh)\n", + "\n", + "v = dtensor.DVariable(\n", + " initial_value=dtensor.call_with_layout(\n", + " tf.function(tf.random.stateless_normal),\n", + " layout=layout,\n", + " shape=tf.TensorShape([64, 32]), \n", + " seed=[1, 1],\n", + " dtype=tf.float32))\n", + "\n", + "print(v.handle)\n", + "assert layout == dtensor.fetch_layout(v)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pb9jn473prC_" + }, + "source": [ + "Other than the requirement on matching the `layout`, a `DVariable` behaves the same as a `tf.Variable`. For example, you can add a DVariable to a DTensor,\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "adxFw9wJpqQQ" + }, + "outputs": [], + "source": [ + "a = dtensor.call_with_layout(tf.ones, layout=layout, shape=(64, 32))\n", + "b = v + a # add DVariable and DTensor\n", + "print(b)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QxBdNHWSu-kV" + }, + "source": [ + "You can also assign a DTensor to a DVariable.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oYwfiyw5P94U" + }, + "outputs": [], + "source": [ + "v.assign(a) # assign a DTensor to a DVariable\n", + "print(a)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4fvSk_VUvGnj" + }, + "source": [ + "Attempting to mutate the layout of a `DVariable`, by assigning a DTensor with an incompatible layout produces an error." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3pckUugYP_r-" + }, + "outputs": [], + "source": [ + "# variable's layout is immutable.\n", + "another_mesh = dtensor.create_mesh([(\"x\", 3), (\"y\", 2)], devices=DEVICES)\n", + "b = dtensor.call_with_layout(tf.ones,\n", + " layout=dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], another_mesh),\n", + " shape=(64, 32))\n", + "try:\n", + " v.assign(b)\n", + "except:\n", + " print(\"exception raised\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3LadIcwRvR6f" + }, + "source": [ + "## What's next?\n", + "\n", + "In this colab, you learned about DTensor, an extension to TensorFlow for distributed computing. To try out these concepts in a tutorial, see [Distributed training with DTensor](https://www.tensorflow.org/tutorials/distribute/dtensor_ml_tutorial.ipynb)." + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "dtensor_overview.ipynb", + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml index fc46bbfaa30..233316fb22b 100644 --- a/site/en/tutorials/_toc.yaml +++ b/site/en/tutorials/_toc.yaml @@ -74,6 +74,12 @@ toc: section: - title: "Distributed training with Keras" path: /tutorials/distribute/keras + - title: "Distributed training with DTensors" + path: /tutorials/distribute/dtensor_ml_tutorial + status: experimental + - title: "Using DTensors with Keras" + path: /tutorials/distribute/dtensor_keras_tutorial + status: experimental - title: "Custom training loops" path: /tutorials/distribute/custom_training - title: "Multi-worker training with Keras" diff --git a/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb new file mode 100644 index 00000000000..7241f5fb2b8 --- /dev/null +++ b/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb @@ -0,0 +1,772 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "Tce3stUlHN0L" + }, + "source": [ + "##### Copyright 2019 The TensorFlow Authors.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tuOe1ymfHZPu" + }, + "outputs": [], + "source": [ + "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MT-LkFOl2axM" + }, + "source": [ + "# DTensor Integration with Keras" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r6P32iYYV27b" + }, + "source": [ + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " View on TensorFlow.org\n", + " \n", + " Run in Google Colab\n", + " \n", + " View source on GitHub\n", + " \n", + " Download notebook\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vTe9dcbUAwqx" + }, + "source": [ + "## Overview\n", + "In this tutoral, you will learn how to use DTensor with Keras.\n", + "\n", + "Through DTensor integration with Keras, you can reuse your existing Keras layers and models to build and train distributed machine learning models.\n", + "\n", + "You will train a multi-layer classification model with the MNIST data. Setting the layout for subclassing model, Sequential model, and functional model will be demonstrated.\n", + "\n", + "This tutoral assumes that you have already read the [DTensor programing guide](/guide/dtensor_overview), and are familiar with basic DTensor concepts like `Mesh` and `Layout`.\n", + "\n", + "This tutoral is based on https://www.tensorflow.org/datasets/keras_example." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "keIyP3IoA1o4" + }, + "source": [ + "## Setup\n", + "\n", + "DTensor is part of TensorFlow 2.9.0 release." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4dHik7NYA5vm" + }, + "outputs": [], + "source": [ + "!pip install --quiet --upgrade --pre tensorflow tensorflow-datasets" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VttBMZngDx8x" + }, + "source": [ + "Next, import `tensorflow` and `tensorflow.experimental.dtensor`, and configure TensorFlow to use 8 virtual CPUs.\n", + "\n", + "Even though this example uses CPUs, DTensor works the same way on CPU, GPU or TPU devices." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CodX6idGBGSm" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "import tensorflow_datasets as tfds\n", + "from tensorflow.experimental import dtensor" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aAtvrpasDpDD" + }, + "outputs": [], + "source": [ + "def configure_virtual_cpus(ncpu):\n", + " phy_devices = tf.config.list_physical_devices('CPU')\n", + " tf.config.set_logical_device_configuration(\n", + " phy_devices[0], \n", + " [tf.config.LogicalDeviceConfiguration()] * ncpu)\n", + " \n", + "configure_virtual_cpus(8)\n", + "tf.config.list_logical_devices('CPU')\n", + "\n", + "devices = [f'CPU:{i}' for i in range(8)]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ogULE1OHtyd9" + }, + "source": [ + "## Deterministic pseudo-random number generators\n", + "One thing you should note is that DTensor API requires each of the running client to have the same random seeds, so that it could have deterministic behavior for initializing the weights. You can achieve this by setting the global seeds in keras via `tf.keras.utils.set_random_seed()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9u85YypguL8N" + }, + "outputs": [], + "source": [ + "tf.keras.backend.experimental.enable_tf_random_generator()\n", + "tf.keras.utils.set_random_seed(1337)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tO11XvPDAu3_" + }, + "source": [ + "## Creating a Data Parallel Mesh\n", + "\n", + "This tutorial demonstrates Data Parallel training. Adapting to Model Parallel training and Spatial Parallel training can be as simple as switching to a different set of `Layout` objects. Refer to [DTensor in-depth ML Tutorial](https://www.tensorflow.org/tutorials/distribute/dtensor_ml_tutorial) for more information on distributed training beyond Data Parallel.\n", + "\n", + "Data Parallel training is a commonly used parallel training scheme, also used by for example `tf.distribute.MirroredStrategy`.\n", + "\n", + "With DTensor, a Data Parallel training loop uses a `Mesh` that consists of a single 'batch' dimension, where each device runs a replica of the model that recieves a shard from the global batch.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6sT6s6z4j9H-" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"batch\", 8)], devices=devices)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rouFcF6FE0aF" + }, + "source": [ + "As each device runs a full replica of the model, the model variables shall be fully replicated across the mesh (unsharded). As an example, a fully replicated Layout for a rank-2 weight on this `Mesh` would be as follows:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "U8OxvkDKE1Nu" + }, + "outputs": [], + "source": [ + "example_weight_layout = dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh) # or\n", + "example_weight_layout = dtensor.Layout.replicated(mesh, rank=2)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6Bnic98RE0xi" + }, + "source": [ + "A layout for a rank-2 data tensor on this `Mesh` would be sharded along the first dimension (sometimes known as `batch_sharded`)," + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PhYp0EKBFfxt" + }, + "outputs": [], + "source": [ + "example_data_layout = dtensor.Layout(['batch', dtensor.UNSHARDED], mesh) # or\n", + "example_data_layout = dtensor.Layout.batch_sharded(mesh, 'batch', rank=2)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4U-6n0DericV" + }, + "source": [ + "## Create Keras layers with layout\n", + "\n", + "In the data parallel scheme, you usually create your model weights with a fully replicated layout, so that each replica of the model can do calculations with the sharded input data. \n", + "\n", + "In order to configure the layout information for your layers' weights, Keras has exposed an extra parameter in the layer constructor for most of the built-in layers.\n", + "\n", + "The following example builds a small image classification model with fully replicated weight layout. You can specify layout information `kernel` and `bias` in `tf.keras.layers.Dense` via argument `kernel_layout` and `bias_layout`. Most of the built-in keras layers are ready for explicitly specifying the `Layout` for the layer weights." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Koc5GlA1tFXY" + }, + "outputs": [], + "source": [ + "unsharded_layout_2d = dtensor.Layout.replicated(mesh, 2)\n", + "unsharded_layout_1d = dtensor.Layout.replicated(mesh, 1)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GfOGTIxGs5Ql" + }, + "outputs": [], + "source": [ + "model = tf.keras.models.Sequential([\n", + " tf.keras.layers.Flatten(input_shape=(28, 28)),\n", + " tf.keras.layers.Dense(128, \n", + " activation='relu',\n", + " name='d1',\n", + " kernel_layout=unsharded_layout_2d, \n", + " bias_layout=unsharded_layout_1d),\n", + " tf.keras.layers.Dense(10,\n", + " name='d2',\n", + " kernel_layout=unsharded_layout_2d, \n", + " bias_layout=unsharded_layout_1d)\n", + "])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0frf3jsVtx_n" + }, + "source": [ + "You can check the layout information by examining the `layout` property on the weights." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Z_nqv_VdwcXo" + }, + "outputs": [], + "source": [ + "for weight in model.weights:\n", + " print(f'Weight name: {weight.name} with layout: {weight.layout}')\n", + " break" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6FMGB-QsxPtU" + }, + "source": [ + "## Load a dataset and build input pipeline\n", + "\n", + "Load a MNIST dataset and configure some pre-processing input pipeline for it. The dataset itself is not associated with any DTensor layout information. There are plans to improve DTensor Keras integration with `tf.data` in future TensorFlow releases.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zGt4kwltxOt4" + }, + "outputs": [], + "source": [ + "(ds_train, ds_test), ds_info = tfds.load(\n", + " 'mnist',\n", + " split=['train', 'test'],\n", + " shuffle_files=True,\n", + " as_supervised=True,\n", + " with_info=True,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HkUaOB_ryaLH" + }, + "outputs": [], + "source": [ + "def normalize_img(image, label):\n", + " \"\"\"Normalizes images: `uint8` -> `float32`.\"\"\"\n", + " return tf.cast(image, tf.float32) / 255., label" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Efm2H1iqydan" + }, + "outputs": [], + "source": [ + "batch_size = 128\n", + "\n", + "ds_train = ds_train.map(\n", + " normalize_img, num_parallel_calls=tf.data.AUTOTUNE)\n", + "ds_train = ds_train.cache()\n", + "ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)\n", + "ds_train = ds_train.batch(batch_size)\n", + "ds_train = ds_train.prefetch(tf.data.AUTOTUNE)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Lcrg6QAtyis4" + }, + "outputs": [], + "source": [ + "ds_test = ds_test.map(\n", + " normalize_img, num_parallel_calls=tf.data.AUTOTUNE)\n", + "ds_test = ds_test.batch(batch_size)\n", + "ds_test = ds_test.cache()\n", + "ds_test = ds_test.prefetch(tf.data.AUTOTUNE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fHEZwib7lhqn" + }, + "source": [ + "## Define the training logic for the model\n", + "\n", + "Next define the training and evalution logic for the model. \n", + "\n", + "As of TensorFlow 2.9, you have to write a custom-training-loop for a DTensor enabled Keras model. This is to pack the input data with proper layout information, which is not integrated with the standard `tf.keras.Model.fit()` or `tf.keras.Model.eval()` functions from Keras. you will get more `tf.data` support in the upcoming release. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CAx11gMjzzjs" + }, + "outputs": [], + "source": [ + "@tf.function\n", + "def train_step(model, x, y, optimizer, metrics):\n", + " with tf.GradientTape() as tape:\n", + " logits = model(x, training=True)\n", + " # tf.reduce_sum sums the batch sharded per-example loss to a replicated\n", + " # global loss (scalar).\n", + " loss = tf.reduce_sum(tf.keras.losses.sparse_categorical_crossentropy(\n", + " y, logits, from_logits=True))\n", + " \n", + " gradients = tape.gradient(loss, model.trainable_variables)\n", + " optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n", + "\n", + " for metric in metrics.values():\n", + " metric.update_state(y_true=y, y_pred=logits)\n", + "\n", + " loss_per_sample = loss / len(x)\n", + " results = {'loss': loss_per_sample}\n", + " return results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "maSTWeRemO0P" + }, + "outputs": [], + "source": [ + "@tf.function\n", + "def eval_step(model, x, y, metrics):\n", + " logits = model(x, training=False)\n", + " loss = tf.reduce_sum(tf.keras.losses.sparse_categorical_crossentropy(\n", + " y, logits, from_logits=True))\n", + "\n", + " for metric in metrics.values():\n", + " metric.update_state(y_true=y, y_pred=logits)\n", + "\n", + " loss_per_sample = loss / len(x)\n", + " results = {'eval_loss': loss_per_sample}\n", + " return results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dt00axcLmvLr" + }, + "outputs": [], + "source": [ + "def pack_dtensor_inputs(images, labels, image_layout, label_layout):\n", + " num_local_devices = image_layout.mesh.num_local_devices()\n", + " images = tf.split(images, num_local_devices)\n", + " labels = tf.split(labels, num_local_devices)\n", + " images = dtensor.pack(images, image_layout)\n", + " labels = dtensor.pack(labels, label_layout)\n", + " return images, labels" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9Eb-qIJGrxB9" + }, + "source": [ + "## Metrics and Optimizers\n", + "\n", + "When using DTensor API with Keras `Metric` and `Optimizer`, you will need to provide the extra mesh information, so that any internal state variables and tensors can work with variables in the model.\n", + "\n", + "- For an optimizer, DTensor introduces a new experimental namespace `keras.dtensor.experimental.optimizers`, where many existing Keras Optimizers are extended to receive an additional `mesh` argument. In future releases, it may be merged with Keras core optimizers.\n", + "\n", + "- For metrics, you can directly specify the `mesh` to the constructor as an argument to make it a DTensor compatible `Metric`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1lu_0mz1sxrl" + }, + "outputs": [], + "source": [ + "optimizer = tf.keras.dtensor.experimental.optimizers.Adam(0.01, mesh=mesh)\n", + "metrics = {'accuracy': tf.keras.metrics.SparseCategoricalAccuracy(mesh=mesh)}\n", + "eval_metrics = {'eval_accuracy': tf.keras.metrics.SparseCategoricalAccuracy(mesh=mesh)}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QzufrkistELx" + }, + "source": [ + "## Train the model\n", + "\n", + "The following example shards the data from input pipeline on the batch dimension, and train with the model, which has fully replicated weights. \n", + "\n", + "With 3 epochs, the model should achieve about 97% of accuracy." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "kZW568Dk0vvL" + }, + "outputs": [], + "source": [ + "num_epochs = 3\n", + "\n", + "image_layout = dtensor.Layout.batch_sharded(mesh, 'batch', rank=4)\n", + "label_layout = dtensor.Layout.batch_sharded(mesh, 'batch', rank=1)\n", + "\n", + "for epoch in range(num_epochs):\n", + " print(\"============================\") \n", + " print(\"Epoch: \", epoch)\n", + " for metric in metrics.values():\n", + " metric.reset_state()\n", + " step = 0\n", + " results = {}\n", + " pbar = tf.keras.utils.Progbar(target=None, stateful_metrics=[])\n", + " for input in ds_train:\n", + " images, labels = input[0], input[1]\n", + " images, labels = pack_dtensor_inputs(\n", + " images, labels, image_layout, label_layout)\n", + "\n", + " results.update(train_step(model, images, labels, optimizer, metrics))\n", + " for metric_name, metric in metrics.items():\n", + " results[metric_name] = metric.result()\n", + "\n", + " pbar.update(step, values=results.items(), finalize=False)\n", + " step += 1\n", + " pbar.update(step, values=results.items(), finalize=True)\n", + "\n", + " for metric in eval_metrics.values():\n", + " metric.reset_state()\n", + " for input in ds_test:\n", + " images, labels = input[0], input[1]\n", + " images, labels = pack_dtensor_inputs(\n", + " images, labels, image_layout, label_layout)\n", + " results.update(eval_step(model, images, labels, eval_metrics))\n", + "\n", + " for metric_name, metric in eval_metrics.items():\n", + " results[metric_name] = metric.result()\n", + " \n", + " for metric_name, metric in results.items():\n", + " print(f\"{metric_name}: {metric.numpy()}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HYEXF6qCuoSr" + }, + "source": [ + "## Specify Layout for existing model code\n", + "\n", + "Often you have models that work well for your use case. Specifying `Layout` information to each individual layer within the model will be a large amount of work requiring a lot of edits.\n", + "\n", + "To help you easily convert your existing Keras model to work with DTensor API you can use the new `dtensor.LayoutMap` API that allow you to specify the `Layout` from a global point of view.\n", + "\n", + "First, you need to create a `LayoutMap` instance, which is a dictionary-like object that contains all the `Layout` you would like to specify for your model weights.\n", + "\n", + "`LayoutMap` needs a `Mesh` instance at init, which can be used to provide default replicated `Layout` for any weights that doesn't have Layout configured. In case you would like all your model weights to be just fully replicated, you can provide empty `LayoutMap`, and the default mesh will be used to create replicated `Layout`.\n", + "\n", + "`LayoutMap` uses a string as key and a `Layout` as value. There is a behavior difference between a normal Python dict and this class. The string key will be treated as a regex when retrieving the value" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SCq5Nl-UP_dS" + }, + "source": [ + "### Subclassed Model\n", + "\n", + "Consider the following model defined using the Keras subclassing Model syntax." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LZ0hRFs8unu0" + }, + "outputs": [], + "source": [ + "class SubclassedModel(tf.keras.Model):\n", + "\n", + " def __init__(self, name=None):\n", + " super().__init__(name=name)\n", + " self.feature = tf.keras.layers.Dense(16)\n", + " self.feature_2 = tf.keras.layers.Dense(24)\n", + " self.dropout = tf.keras.layers.Dropout(0.1)\n", + "\n", + " def call(self, inputs, training=None):\n", + " x = self.feature(inputs)\n", + " x = self.dropout(x, training=training)\n", + " return self.feature_2(x)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1njxqPB-yS97" + }, + "source": [ + "There are 4 weights in this model, which are `kernel` and `bias` for two `Dense` layers. Each of them are mapped based on the object path:\n", + "\n", + "* `model.feature.kernel`\n", + "* `model.feature.bias`\n", + "* `model.feature_2.kernel`\n", + "* `model.feature_2.bias`\n", + "\n", + "Note: For Subclassed Models, the attribute name, rather than the `.name` attribute of layer are used as the key to retrieve the Layout from the mapping. This is consistent with the convention followed by `tf.Module` checkpointing. For complex models with more than a few layers, you can [manually inspect checkpoints](https://www.tensorflow.org/guide/checkpoint#manually_inspecting_checkpoints) to see the attribute mappings. \n", + "\n", + "Now define the following `LayoutMap` and apply it to the model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "goVX6iIZw468" + }, + "outputs": [], + "source": [ + "layout_map = tf.keras.dtensor.experimental.LayoutMap(mesh=mesh)\n", + "\n", + "layout_map['feature.*kernel'] = dtensor.Layout.batch_sharded(mesh, 'batch', rank=2)\n", + "layout_map['feature.*bias'] = dtensor.Layout.batch_sharded(mesh, 'batch', rank=1)\n", + "\n", + "with tf.keras.dtensor.experimental.layout_map_scope(layout_map):\n", + " subclassed_model = SubclassedModel()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "M32HcSp_PyWs" + }, + "source": [ + "The model weights are created on the first call, so call the model with a DTensor input and confirm the weights have the expected layouts." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c3CbD9l7qUNq" + }, + "outputs": [], + "source": [ + "dtensor_input = dtensor.copy_to_mesh(tf.zeros((16, 16)), layout=unsharded_layout_2d)\n", + "# Trigger the weights creation for subclass model\n", + "subclassed_model(dtensor_input)\n", + "\n", + "print(subclassed_model.feature.kernel.layout)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZyCnfd-4Q2jk" + }, + "source": [ + "With this, you can quickly map the `Layout` to your models without updating any of your existing code. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6GliUdWTQnKC" + }, + "source": [ + "### Sequential and Functional Models" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6zzvTqAR2Teu" + }, + "source": [ + "For keras functional and sequential models, you can use `LayoutMap` as well.\n", + "\n", + "Note: For functional and sequential models, the mappings are slightly different. The layers in the model don't have a public attribute attached to the model (though you can access them via `model.layers` as a list). Use the string name as the key in this case. The string name is guaranteed to be unique within a model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gXK2EquIRJCC" + }, + "outputs": [], + "source": [ + "layout_map = tf.keras.dtensor.experimental.LayoutMap(mesh=mesh)\n", + "\n", + "layout_map['feature.*kernel'] = dtensor.Layout.batch_sharded(mesh, 'batch', rank=2)\n", + "layout_map['feature.*bias'] = dtensor.Layout.batch_sharded(mesh, 'batch', rank=1)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cBzwJqrg2TH3" + }, + "outputs": [], + "source": [ + "with tf.keras.dtensor.experimental.layout_map_scope(layout_map):\n", + " inputs = tf.keras.Input((16,), batch_size=16)\n", + " x = tf.keras.layers.Dense(16, name='feature')(inputs)\n", + " x = tf.keras.layers.Dropout(0.1)(x)\n", + " output = tf.keras.layers.Dense(32, name='feature_2')(x)\n", + " model = tf.keras.Model(inputs, output)\n", + "\n", + "print(model.layers[1].kernel.layout)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pPuh1NlE3-wO" + }, + "outputs": [], + "source": [ + "with tf.keras.dtensor.experimental.layout_map_scope(layout_map):\n", + " model = tf.keras.Sequential([\n", + " tf.keras.layers.Dense(16, name='feature', input_shape=(16,)),\n", + " tf.keras.layers.Dropout(0.1),\n", + " tf.keras.layers.Dense(32, name='feature_2')\n", + " ])\n", + "\n", + "print(model.layers[2].kernel.layout)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "00dPVoSlRLFA" + }, + "outputs": [], + "source": [ + "" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "dtensor_keras_tutorial.ipynb", + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb new file mode 100644 index 00000000000..59c9cc9643d --- /dev/null +++ b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb @@ -0,0 +1,1100 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "Tce3stUlHN0L" + }, + "source": [ + "##### Copyright 2019 The TensorFlow Authors.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tuOe1ymfHZPu" + }, + "outputs": [], + "source": [ + "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MfBg1C5NB3X0" + }, + "source": [ + "# DTensor Maching Learning Tutorial\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r6P32iYYV27b" + }, + "source": [ + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " View on TensorFlow.org\n", + " \n", + " Run in Google Colab\n", + " \n", + " View source on GitHub\n", + " \n", + " Download notebook\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kiF4jjX4O1mF" + }, + "source": [ + "## Overview\n", + "\n", + "DTensor provides a way for you to distribute the training of your model across devices to improve efficiency, reliability and scalability. For more details on DTensor concepts, see [The DTensor Programming Guide](https://www.tensorflow.org/guide/dtensor).\n", + "\n", + "In this tutorial, you will train a Sentiment Analysis model with DTensor. Three distributed training schemes are demonstrated with this example:\n", + " \n", + " - Data Parallel training, where the training samples are sharded (partitioned) to devices.\n", + " - Model Parallel training, where the model variables are sharded to devices. \n", + " - Spatial Parallel training, where the features of input data are sharded to devices.\n", + "\n", + "The training portion of this tutorial is inspired [A Kaggle guide on Sentiment Analysis](https://www.kaggle.com/code/anasofiauzsoy/yelp-review-sentiment-analysis-tensorflow-tfds/notebook) notebook. To learn about the complete training and evaluation workflow (without DTensor), refer to that notebook. \n", + "\n", + "This tutorial will walk through the following steps:\n", + "\n", + "- First start with some data cleaning to obtain a `tf.data.Dataset` of tokenized sentences and their polarity.\n", + "\n", + "- Next build an MLP model with custom Dense and BatchNorm layers. Use a `tf.Module` to track the inference variables. The model constructor takes additional `Layout` arguments to control the sharding of variables.\n", + "\n", + "- For training, you will first use data parallel training together with `tf.experimental.dtensor`'s checkpoint feature. Then continue with Model Parallel Training and Spatial Parallel Training.\n", + "\n", + "- The final section briefly describes the interaction between `tf.saved_model` and `tf.experimental.dtensor` as of TensorFlow 2.9.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YD80veeg7QtW" + }, + "source": [ + "## Setup\n", + "\n", + "DTensor is part of TensorFlow 2.9.0 release." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-RKXLJN-7Yyb" + }, + "outputs": [], + "source": [ + "!pip install --quiet --upgrade --pre tensorflow tensorflow-datasets" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tcxP4_Zu7ciQ" + }, + "source": [ + "Next, import `tensorflow` and `tensorflow.experimental.dtensor`. Then configure TensorFlow to use 8 virtual CPUs.\n", + "\n", + "Even though this example uses CPUs, DTensor works the same way on CPU, GPU or TPU devices." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dXcB26oP7dUd" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "from tensorflow.experimental import dtensor\n", + "print('TensorFlow version:', tf.__version__)\n", + "\n", + "def configure_virtual_cpus(ncpu):\n", + " phy_devices = tf.config.list_physical_devices('CPU')\n", + " tf.config.set_logical_device_configuration(phy_devices[0], [\n", + " tf.config.LogicalDeviceConfiguration(),\n", + " ] * ncpu)\n", + " \n", + "configure_virtual_cpus(8)\n", + "DEVICES = [f'CPU:{i}' for i in range(8)]\n", + "\n", + "tf.config.list_logical_devices('CPU')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "omYd4jbF7j_I" + }, + "source": [ + "## Download the dataset\n", + "\n", + "Download the IMDB reviews data set to train the sentiment analysis model. For the sake of running speed, cap the training set to the first 10K samples." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HG_ASSzR4IWW" + }, + "outputs": [], + "source": [ + "import tensorflow_datasets as tfds\n", + "\n", + "data = tfds.load('imdb_reviews', split='train', shuffle_files=True)\n", + "sample = data.take(1024*10)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gXSIcdeTZ7jL" + }, + "source": [ + "Verify that the samples are not biased by checking that there are comparable numbers of positive and negative labels." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8rG3rz8cANmB" + }, + "outputs": [], + "source": [ + "tf.math.bincount(tf.cast(tf.reshape(list(sample.batch(1024).map(lambda x: x['label'])), -1), tf.int32))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ki3mpfi4aZH8" + }, + "source": [ + "## Prepare the data\n", + "\n", + "First tokenize the text. Here use an extension of one-hot encoding, the `'tf_idf'` mode of `tf.keras.layers.TextVectorization`.\n", + "\n", + "- For the sake of speed, limit the number of tokens to 1200.\n", + "- To keep the `tf.Module` simple, run `TextVectorization` as a preprocessing step before the training. \n", + "\n", + "The final result of the data cleaning section is a `Dataset` with the tokenized text as `x` and label as `y`.\n", + "\n", + "**Note**: Running `TextVectorization` as a preprocessing step is **neither a usual practice nor a recommended one** as doing so assumes the training data fits into the client memory, which is not always the case.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zNpxjku_57Lg" + }, + "outputs": [], + "source": [ + "tvl = tf.keras.layers.TextVectorization(output_mode='tf_idf', max_tokens=1200, output_sequence_length=None)\n", + "tvl.adapt(data=sample.batch(1024).map(lambda x: x['text']))\n", + "dataset_x = tf.keras.models.Sequential([tvl]).predict(sample.batch(1024).map(lambda x: x['text']))\n", + "dataset_y = tf.keras.models.Sequential([]).predict(sample.batch(1024).map(lambda x: x['label']))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bEtzRtXXBeBB" + }, + "outputs": [], + "source": [ + "dataset = tf.data.Dataset.from_tensor_slices({\n", + " 'x': dataset_x,\n", + " 'y': dataset_y,\n", + "})\n", + "\n", + "dataset.take(1).get_single_element()\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zPtep4hpQqJ9" + }, + "source": [ + "### Mock dataset\n", + "To get better and faster convergence, you can train with a mock dataset that has the feature channels fully correlated with the label." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "W_DKa6wkFVHe" + }, + "outputs": [], + "source": [ + "USE_MOCK_DATA = False # Change to True to use mock data\n", + "\n", + "def processor(data):\n", + " # Rewrite the data, such that \n", + " # - data channel 0 triggers label 0, and \n", + " # - channel -1 triggers label 1\n", + " x = tf.where(data['y'] != 0, \n", + " tf.range(len(data['x']), dtype=tf.float32),\n", + " len(data['x']) * 1.0 - tf.range(len(data['x']), dtype=tf.float32))\n", + " return {'x': x, 'y' : data['y']}\n", + "\n", + "if USE_MOCK_DATA: \n", + " dataset = dataset.map(processor)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "atTqL9kE5wz4" + }, + "source": [ + "## Build a neural network with DTensor\n", + "\n", + "Now build a Multi-Layer Perceptron (MLP) network with `DTensor`. The network will use fully connected Dense and BatchNorm layers.\n", + "\n", + "`DTensor` expands TensorFlow through single-program multi-data (SPMD) expansion of regular TensorFlow Ops according to the `dtensor.Layout` attributes of their input `Tensor` and variables. \n", + "\n", + "Variables of `DTensor` aware layers are `dtensor.DVariable`, and the constructors of `DTensor` aware layer objects take additional `Layout` inputs in addition to the usual layer parameters.\n", + "\n", + "Note: As of TensorFlow 2.9, Keras layers such as `tf.keras.layer.Dense`, and `tf.keras.layer.BatchNormalization` accepts `dtensor.Layout` arguments. Refer to the [DTensor Keras Integration Tutorial](/tutorials/distribute/dtensor_keras_tutorial) for more information using Keras with DTensor." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PMCt-Gj3b3Jy" + }, + "source": [ + "\n", + "### Dense Layer\n", + "\n", + "The following custom Dense layer defines 2 layer variables: $W_{ij}$ is the variable for weights, and $b_i$ is the variable for the biases.\n", + "\n", + "$$\n", + "y_j = \\sigma(\\sum_i x_i W_{ij} + b_j)\n", + "$$\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nYlFUJWNjl4N" + }, + "source": [ + "### Layout deduction\n", + "\n", + "This result comes from the following observations:\n", + "\n", + "- The preferred DTensor sharding for operands to a matrix dot product $t_j = \\sum_i x_i W_{ij}$ is to shard $\\mathbf{W}$ and $\\mathbf{x}$ the same way along the $i$-axis.\n", + "\n", + "- The preferred DTensor sharding for operands to a matrix sum $t_j + b_j$, is to shard $\\mathbf{t}$ and $\\mathbf{b}$ the same way along the $j$-axis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "VpKblz7Yb16G" + }, + "outputs": [], + "source": [ + "class Dense(tf.Module):\n", + "\n", + " def __init__(self, input_size, output_size, init_seed, weight_layout):\n", + " super().__init__()\n", + "\n", + " random_normal_initializer = tf.function(tf.random.stateless_normal)\n", + " \n", + " self.weight = dtensor.DVariable(\n", + " dtensor.call_with_layout(\n", + " random_normal_initializer, weight_layout,\n", + " shape=[input_size, output_size],\n", + " seed=init_seed\n", + " ))\n", + " \n", + " # bias is sharded the same way as the last axis of weight.\n", + " bias_layout = weight_layout.delete([0])\n", + "\n", + " self.bias = dtensor.DVariable(\n", + " dtensor.call_with_layout(tf.zeros, bias_layout, [output_size]))\n", + "\n", + " def __call__(self, x):\n", + " y = tf.matmul(x, self.weight) + self.bias\n", + " y = tf.nn.relu(y)\n", + " \n", + " return y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tfVY_vAKbxM0" + }, + "source": [ + "### BatchNorm\n", + "\n", + "A batch normalization layer helps avoid collapsing modes while training. In this case, adding batch normalization layers helps model training avoid producing a model that only produces zeros.\n", + "\n", + "The constructor of the custom `BatchNorm` layer below does not take a `Layout` argument. This is because `BatchNorm` has no layer variables. This still works with DTensor because 'x', the only input to the layer, is already a DTensor that represents the global batch.\n", + "\n", + "Note: With DTensor, the input Tensor 'x' always represents the global batch. Therefore `tf.nn.batch_normalization` is applied to the global batch. This differs from training with `tf.distribute.MirroredStrategy`, where Tensor 'x' only represents the per-replica shard of the batch (the local batch)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "riBA9pfhlPFq" + }, + "outputs": [], + "source": [ + "class BatchNorm(tf.Module):\n", + "\n", + " def __init__(self):\n", + " super().__init__()\n", + "\n", + " def __call__(self, x, training=True):\n", + " if not training: \n", + " # This branch is not used in the Tutorial.\n", + " pass\n", + " mean, variance = tf.nn.moments(x, axes=[0])\n", + " return tf.nn.batch_normalization(x, mean, variance, 0.0, 1.0, 1e-5)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "q4R4MPz5prh4" + }, + "source": [ + "A full featured batch normalization layer (such as `tf.keras.layers.BatchNormalization`) will need Layout arguments for its variables. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "unFcP99zprJj" + }, + "outputs": [], + "source": [ + "def make_keras_bn(bn_layout):\n", + " return tf.keras.layers.BatchNormalization(gamma_layout=bn_layout,\n", + " beta_layout=bn_layout,\n", + " moving_mean_layout=bn_layout,\n", + " moving_variance_layout=bn_layout,\n", + " fused=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "v8Dj7AJ_lPs0" + }, + "source": [ + "### Putting Layers Together\n", + "\n", + "Next, build a Multi-layer perceptron (MLP) network with the building blocks above.\n", + "\n", + "The output of the first `Dense` layer is passed into the input of the second `Dense` layer (after the `BatchNorm`). Therefore, the preferred DTensor sharding for the output of first `Dense` layer ($\\mathbf{W_1}$) and the input of second `Dense` layer ($\\mathbf{W_2}$) is to shard $\\mathbf{W_1}$ and $\\mathbf{W_2}$ the same way along the common axis $\\hat{j}$,\n", + "\n", + "$$\n", + "\\mathsf{Layout}[{W_{1,ij}}; i, j] = \\left[\\hat{i}, \\hat{j}\\right] \\\\\n", + "\\mathsf{Layout}[{W_{2,jk}}; j, k] = \\left[\\hat{j}, \\hat{k} \\right]\n", + "$$\n", + "\n", + "Even though the layout deduction shows that the 2 layouts are not independent, for the sake of simplicity of the model interface, `MLP` will take 2 `Layout` arguments, one per Dense layer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "junyS-965opl" + }, + "outputs": [], + "source": [ + "from typing import Tuple\n", + "\n", + "class MLP(tf.Module):\n", + "\n", + " def __init__(self, dense_layouts: Tuple[dtensor.Layout, dtensor.Layout]):\n", + " super().__init__()\n", + " \n", + " self.dense1 = Dense(1200, 48, (1, 2), dense_layouts[0])\n", + " self.bn1 = BatchNorm()\n", + " self.dense2 = Dense(48, 2, (3, 4), dense_layouts[1])\n", + " self.bn2 = BatchNorm()\n", + " \n", + " def __call__(self, x):\n", + " y = x\n", + " y = self.dense1(y)\n", + " y = self.bn1(y)\n", + " y = self.dense2(y)\n", + " y = self.bn2(y)\n", + " return y\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9dgLmebHhr7h" + }, + "source": [ + "The trade-off between correctness in layout deduction constraints and simplicity of API is a common design point of APIs that uses DTensor.\n", + "It is also possible to capture the dependency between `Layout`'s with a different API. For example, the `MLPStricter` class creates the `Layout` objects in the constructor." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wEZR7UlihsYX" + }, + "outputs": [], + "source": [ + "class MLPStricter(tf.Module):\n", + "\n", + " def __init__(self, mesh, input_mesh_dim, inner_mesh_dim1, output_mesh_dim):\n", + " super().__init__()\n", + " \n", + " self.dense1 = Dense(1200, 48, (1, 2), dtensor.Layout([input_mesh_dim, inner_mesh_dim1], mesh))\n", + " self.bn1 = BatchNorm()\n", + " self.dense2 = Dense(48, 2, (3, 4), dtensor.Layout([inner_mesh_dim1, output_mesh_dim], mesh))\n", + " \n", + "\n", + " def __call__(self, x):\n", + " y = x\n", + " y = self.dense1(y)\n", + " y = self.bn1(y)\n", + " y = self.dense2(y)\n", + " y = self.bn2(y)\n", + " return y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GcQi7D5mal2L" + }, + "source": [ + "To make sure the model runs, probe your model with fully replicated layouts and a fully replicated batch of `'x'` input." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zOPuYeQwallh" + }, + "outputs": [], + "source": [ + "WORLD = dtensor.create_mesh([(\"world\", 8)], devices=DEVICES)\n", + "\n", + "model = MLP([dtensor.Layout.replicated(WORLD, rank=2),\n", + " dtensor.Layout.replicated(WORLD, rank=2)])\n", + "x = dataset.take(5).batch(5).get_single_element()['x']\n", + "y = dataset.take(5).batch(5).get_single_element()['y']\n", + "x = dtensor.copy_to_mesh(x, dtensor.Layout.replicated(WORLD, rank=2))\n", + "print(model(x))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "akrjDstEpDv9" + }, + "source": [ + "## Moving data to the device\n", + "\n", + "Usually, `tf.data` iterators (and other data fetching methods) yield tensor objects backed by the local host device memory. This data must be transfered to the accelerator device memory that backs DTensor's component tensors.\n", + "\n", + "`dtensor.copy_to_mesh` is unsuitable for this situation because it replicates input tensors to all devices due to DTensor's global perspective. So in this tutorial, you will use a helper function `repack_local_tensor`, to facilitate the transfer of data. This helper function uses `dtensor.pack` to send (and only send) the shard of the global batch that is intended for a replica to the device backing the replica.\n", + "\n", + "This simplified function assumes single-client. Determining the correct way to split the local tensor and the mapping between the pieces of the split and the local devices can be laboring in a multi-client application.\n", + "\n", + "Additional DTensor API to simplify `tf.data` integration is planned, supporting both single-client and multi-client applications. Please stay tuned." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3t5WvQR4Hvo4" + }, + "outputs": [], + "source": [ + "def repack_local_tensor(x, layout):\n", + " \"\"\"Repacks a local Tensor-like to a DTensor with layout.\n", + " \n", + " This function assumes a single-client application.\n", + " \"\"\"\n", + " x = tf.convert_to_tensor(x)\n", + " sharded_dims = []\n", + " \n", + " # For every sharded dimension, use tf.split to split the along the dimension.\n", + " # The result is a nested list of split-tensors in queue[0].\n", + " queue = [x]\n", + " for axis, dim in enumerate(layout.sharding_specs):\n", + " if dim == dtensor.UNSHARDED:\n", + " continue \n", + " num_splits = layout.shape[axis]\n", + " queue = tf.nest.map_structure(lambda x: tf.split(x, num_splits, axis=axis), queue)\n", + " sharded_dims.append(dim)\n", + " \n", + " # Now we can build the list of component tensors by looking up the location in\n", + " # the nested list of split-tensors created in queue[0].\n", + " components = []\n", + " for locations in layout.mesh.local_device_locations():\n", + " t = queue[0]\n", + " for dim in sharded_dims:\n", + " split_index = locations[dim] # Only valid on single-client mesh.\n", + " t = t[split_index]\n", + " components.append(t) \n", + " \n", + " return dtensor.pack(components, layout)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2KKCDcjG7zj2" + }, + "source": [ + "## Data parallel training\n", + "\n", + "In this section, you will train your MLP model with data parallel training. The following sections will demonstrate model parallel training and spatial parallel training.\n", + "\n", + "Data parallel training is a commonly used scheme for distributed machine learning:\n", + "\n", + " - Model variables are replicated on N devices each.\n", + " - A global batch is split into N per-replica batches.\n", + " - Each per-replica batch is trained on the replica device.\n", + " - The gradient is reduced before weight up data is collectively performed on all replicas.\n", + " \n", + "Data parallel training provides nearly linear speedup regarding the number of devices." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UMsLUyTGq3oL" + }, + "source": [ + "### Creating a data parallel mesh\n", + "\n", + "A typical data parallelism training loop uses a DTensor `Mesh` that consists of a single `batch` dimension, where each device becomes a replica that receives a shard from the global batch.\n", + "\n", + "\"Data\n", + "\n", + "\n", + "The replicated model runs on the replica, therefore the model variables are fully replicated (unsharded)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "C0IyOlxmeu4I" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"batch\", 8)], devices=DEVICES)\n", + "\n", + "model = MLP([\n", + " dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n", + " dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n", + "])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OREKwBybo1gZ" + }, + "source": [ + "### Packing training data to DTensors\n", + "\n", + "The training data batch should be packed into DTensors sharded along the `'batch'`(first) axis, such that DTensor will evenly distribute the training data to the `'batch'` mesh dimension.\n", + "\n", + "**Note**: In DTensor, the `batch size` always refers to the global batch size. The batch size should be chosen such that it can be divided evenly by the size of the `batch` mesh dimension. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8xMYkTpGocY8" + }, + "outputs": [], + "source": [ + "batch_size = 48\n", + "\n", + "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n", + "\n", + "def repack_batch(x, y, mesh):\n", + " x = repack_local_tensor(x, layout=dtensor.Layout(['batch', dtensor.UNSHARDED], mesh))\n", + " y = repack_local_tensor(y, layout=dtensor.Layout(['batch'], mesh))\n", + " return x, y\n", + "\n", + "input = batched_dataset.take(1).get_single_element()\n", + "x, y = repack_batch(input['x'], input['y'], mesh)\n", + "\n", + "print('x', x[:, 0])\n", + "print('y', y)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uONSiqOIkFL1" + }, + "source": [ + "### Training step\n", + "\n", + "This example uses a Stochastic Gradient Descent optimizer with the Custom Training Loop (CTL). Consult the [Custom Training Loop guide](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) and [Walk through](https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough) for more information on those topics.\n", + "\n", + "The `train_step` is encapsulated as a `tf.funtion` to indicate this body is to be traced as a TensorFlow Graph. The body of `train_step` consists of a forward inference pass, a backward gradient pass, and the variable update.\n", + "\n", + "Note that the body of `train_step` does not contain any special DTensor annotations. Instead, `train_step` only contains high-level TensorFlow operations that process the input `x` and `y` from the global view of the input batch and the model. All of the DTensor annotations (`Mesh`, `Layout`) are factored out of the train step." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BwUFzLGDtQT6" + }, + "outputs": [], + "source": [ + "# Refer to the CTL (custom training loop guide)\n", + "@tf.function\n", + "def train_step(model, x, y, learning_rate=tf.constant(1e-4)):\n", + " with tf.GradientTape() as tape:\n", + " logits = model(x)\n", + " # tf.reduce_sum sums the batch sharded per-example loss to a replicated\n", + " # global loss (scalar).\n", + " loss = tf.reduce_sum(\n", + " tf.nn.sparse_softmax_cross_entropy_with_logits(\n", + " logits=logits, labels=y))\n", + " parameters = model.trainable_variables\n", + " gradients = tape.gradient(loss, parameters)\n", + " for parameter, parameter_gradient in zip(parameters, gradients):\n", + " parameter.assign_sub(learning_rate * parameter_gradient)\n", + "\n", + " # Define some metrics \n", + " accuracy = 1.0 - tf.reduce_sum(tf.cast(tf.argmax(logits, axis=-1, output_type=tf.int64) != y, tf.float32)) / x.shape[0]\n", + " loss_per_sample = loss / len(x)\n", + " return {'loss': loss_per_sample, 'accuracy': accuracy}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0OYTu4j0evWT" + }, + "source": [ + "### Checkpointing\n", + "\n", + "You can checkpoint a DTensor model using `dtensor.DTensorCheckpoint`. The format of a DTensor checkpoint is fully compatible with a Standard TensorFlow Checkpoint. There is ongoing work to consolidate `dtensor.DTensorCheckpoint` into `tf.train.Checkpoint`. \n", + "\n", + "When a DTensor checkpoint is restored, `Layout`s of variables can be different from when the checkpoint is saved. This tutorial makes use of this feature to continue the training in the Model Parallel training and Spatial Parallel training sections.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rsInFFJg7x9t" + }, + "outputs": [], + "source": [ + "def start_checkpoint_manager(mesh, model):\n", + " ckpt = dtensor.DTensorCheckpoint(mesh, root=model)\n", + " manager = tf.train.CheckpointManager(ckpt, '/tmp/checkpoints/data-and-model/cp', max_to_keep=3)\n", + "\n", + " if manager.latest_checkpoint:\n", + " print(\"Restoring a checkpoint\")\n", + " ckpt.restore(manager.latest_checkpoint).assert_consumed()\n", + " else:\n", + " print(\"new training\")\n", + " return manager\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9r77ky5Jgp1j" + }, + "source": [ + "### Training loop\n", + "\n", + "For the data parallel training scheme, train for epochs and report the progress. 3 epochs is insufficient for training the model -- an accuracy of 50% is as good as randomly guessing.\n", + "\n", + "Enable checkpointing so that you can pick up the training later. In the following section, you will load the checkpoint and train with a different parallel scheme." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "UaLn-vGZgqbS" + }, + "outputs": [], + "source": [ + "num_epochs = 3\n", + "manager = start_checkpoint_manager(mesh, model)\n", + "\n", + "for epoch in range(num_epochs):\n", + " step = 0\n", + " pbar = tf.keras.utils.Progbar(target=None, stateful_metrics=[])\n", + " metrics = {'epoch': epoch}\n", + " for input in batched_dataset:\n", + "\n", + " x, y = repack_batch(input['x'], input['y'], mesh)\n", + " \n", + " metrics.update(train_step(model, x, y, 1e-2))\n", + "\n", + " pbar.update(step, values=metrics.items(), finalize=False)\n", + " step += 1\n", + " manager.save()\n", + " pbar.update(step, values=metrics.items(), finalize=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YRFJEhum7EGD" + }, + "source": [ + "## Model Parallel Training\n", + "\n", + "If you switch to a 2 dimensional `Mesh`, and shard the model variables, along the second mesh dimension, then the training becomes Model Parallel.\n", + "\n", + "In Model Parallel training, each model replica spans multiple devices (2 in this case):\n", + "\n", + "- There are 4 model replicas, and the training data batch is distributed to the 4 replicas. \n", + "- The 2 devices within a single model replica receive replicated training data.\n", + "\n", + "\n", + "\"Model\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5gZE9IT5Dzwl" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"batch\", 4), (\"model\", 2)], devices=DEVICES)\n", + "model = MLP([\n", + " dtensor.Layout([dtensor.UNSHARDED, \"model\"], mesh), \n", + " dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)]\n", + " )\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ihof3DkMFKnf" + }, + "source": [ + "As the training data is still sharded along the batch dimension, you can reuse the same `repack_batch` function as the Data Parallel training case. DTensor will automatically replicate the per-replica batch to all devices inside the replica along the `\"model\"` mesh dimension." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dZf56ynbE_p1" + }, + "outputs": [], + "source": [ + "batch_size = 48\n", + "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n", + "\n", + "def repack_batch(x, y, mesh):\n", + " x = repack_local_tensor(x, layout=dtensor.Layout(['batch', dtensor.UNSHARDED], mesh))\n", + " y = repack_local_tensor(y, layout=dtensor.Layout(['batch'], mesh))\n", + " return x, y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UW3OXdhNFfpv" + }, + "source": [ + "Next run the training loop. The training loop reuses the same checkpoint manager as the Data Parallel training example, and the code looks identical.\n", + "\n", + "You can continue training the data parallel trained model under model parallel training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LLC0wgii7EgA" + }, + "outputs": [], + "source": [ + "num_epochs = 2\n", + "manager = start_checkpoint_manager(mesh, model)\n", + "\n", + "for epoch in range(num_epochs):\n", + " step = 0\n", + " pbar = tf.keras.utils.Progbar(target=int(batched_dataset.cardinality()))\n", + " metrics = {'epoch': epoch}\n", + " for input in batched_dataset:\n", + " x, y = input['x'], input['y']\n", + " x, y = repack_batch(x, y, mesh)\n", + " metrics.update(train_step(model, x, y, 1e-2))\n", + " pbar.update(step, values=metrics.items(), finalize=False)\n", + " step += 1\n", + " manager.save()\n", + " pbar.update(step, values=metrics.items(), finalize=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BZH-aMrVzi2L" + }, + "source": [ + "## Spatial Parallel Training" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "u-bK6IZ9GCS9" + }, + "source": [ + "When training data of very high dimensionality (e.g. a very large image or a video), it may be desirable to shard along the feature dimension. This is called Spatial Parallel training.\n", + "\n", + "\"Spatial\n", + "\n", + "DTensor also supports this case. The only change you need to do is to create a Mesh that includes a `feature` dimension, and apply the corresponding `Layout`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "jpc9mqURGpmK" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"batch\", 2), (\"feature\", 2), (\"model\", 2)], devices=DEVICES)\n", + "model = MLP([\n", + " dtensor.Layout([\"feature\", \"model\"], mesh), \n", + " dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i07Wrv-jHBc1" + }, + "source": [ + "Shard the input data along the `feature` dimension when packing the input tensors to DTensors. You do this with a slightly different repack function, `repack_batch_for_spt`, where `spt` stands for Spatial Parallel Training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DWR8qF6BGtFL" + }, + "outputs": [], + "source": [ + "batch_size = 48\n", + "\n", + "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n", + "\n", + "def repack_batch_for_spt(x, y, mesh): \n", + " # Shard data on feature dimension, too\n", + " x = repack_local_tensor(x, layout=dtensor.Layout([\"batch\", 'feature'], mesh))\n", + " y = repack_local_tensor(y, layout=dtensor.Layout([\"batch\"], mesh))\n", + " return x, y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ygl9dqMUHTVN" + }, + "source": [ + "The Spatial parallel training can also continue from a checkpoint created with other parallell training schemes.\n", + "\n", + "Let this train with a few epochs to get an accuracy above 0.5, slightly better than random guessing before saving the model in the next section." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "p3NnpHSKo-hx" + }, + "outputs": [], + "source": [ + "num_epochs = 30\n", + "\n", + "manager = start_checkpoint_manager(mesh, model)\n", + "for epoch in range(num_epochs):\n", + " step = 0\n", + " metrics = {'epoch': epoch}\n", + " pbar = tf.keras.utils.Progbar(target=int(batched_dataset.cardinality()))\n", + " \n", + " for input in batched_dataset:\n", + " x, y = input['x'], input['y']\n", + " x, y = repack_batch_for_spt(x, y, mesh)\n", + " metrics.update(train_step(model, x, y, 1e-2))\n", + " \n", + " pbar.update(step, values=metrics.items(), finalize=False)\n", + " step += 1\n", + " manager.save()\n", + " pbar.update(step, values=metrics.items(), finalize=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vp4L59CpJjYr" + }, + "source": [ + "## SavedModel and DTensor\n", + "\n", + "The integration of DTensor and SavedModel is still under development. This section only describes the current status quo for TensorFlow 2.9.0.\n", + "\n", + "As of TensorFlow 2.9.0, `tf.saved_model` only accepts DTensor models with fully replicated variables. \n", + "\n", + "As a workaround, you can convert a DTensor model to a fully replicated one by reloading a checkpoint. However, after a model is saved, all DTensor annotations are lost and the saved signatures can only be used with regular Tensors, not DTensors." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "49HfIq_SJZoj" + }, + "outputs": [], + "source": [ + "mesh = dtensor.create_mesh([(\"world\", 1)], devices=DEVICES[:1])\n", + "model_for_saving = MLP([\n", + " dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh), \n", + " dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh)])\n", + "\n", + "manager = start_checkpoint_manager(mesh, model_for_saving)\n", + "tf.saved_model.save(model_for_saving, \"/tmp/saved_model\",\n", + " signatures=tf.function(model_for_saving.__call__).get_concrete_function(tf.TensorSpec([None, 1200], tf.float32))\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "h6Csim_VMGxQ" + }, + "source": [ + "As of TensorFlow 2.9.0, you can only call a loaded signature with a regular Tensor, or a fully replicated DTensor (which will be converted to a regular Tensor)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qW8yKPrhKQ5b" + }, + "outputs": [], + "source": [ + "loaded = tf.saved_model.load(\"/tmp/saved_model\")\n", + "\n", + "batch = dataset.batch(4, drop_remainder=True).take(1).get_single_element()\n", + "\n", + "print(loaded.signatures[\"serving_default\"](batch['x']))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ks-Vs9qsH6jO" + }, + "source": [ + "## What's next? \n", + "\n", + "This tutorial demonstrated building and training an MLP sentiment analysis model with DTensor.\n", + "\n", + "Through `Mesh` and `Layout` primitives, DTensor can transform a TensorFlow `tf.function` to a distributed program suitable for a variety of training schemes. \n", + "\n", + "In a real-world machine learning application, evaluation and cross-validation should be applied to avoid producing an over-fitted model. The techniques introduced in this tutorial can also be applied to introduce parallelism to evaluation.\n", + "\n", + "Composing a model with `tf.Module` from scratch is a lot of work, and reusing existing building blocks such as layers and helper functions can drastically speed up model development.\n", + "As of TensorFlow 2.9, all Keras Layers under `tf.keras.layers` accepts DTensor layouts as their arguments, and can be used to build DTensor models. You can even directly reuse a Keras model with DTensor without modifying the model implementation. Refer to the [DTensor Keras Integration Tutorial](link) (TODO: add link) for information on using DTensor Keras. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "A-YWPfJyHPcX" + }, + "outputs": [], + "source": [ + "" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "dtensor_ml_tutorial.ipynb", + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} From 35deba90c8fd6a7ece8049c6c47f2bb56f7817e6 Mon Sep 17 00:00:00 2001 From: Nathan Luehr Date: Fri, 29 Apr 2022 14:01:01 -0500 Subject: [PATCH 105/872] Update instructions with new apt key. See https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/ for details. --- site/en/install/gpu.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/site/en/install/gpu.md b/site/en/install/gpu.md index 041241f3bed..b34ad5ea258 100644 --- a/site/en/install/gpu.md +++ b/site/en/install/gpu.md @@ -93,15 +93,14 @@ This section shows how to install CUDA® 11 (TensorFlow >= 2.4.0) on Ubuntu Caution: [Secure Boot](https://wiki.ubuntu.com/UEFI/SecureBoot){:.external} complicates installation of the NVIDIA driver and is beyond the scope of these instructions. - -#### Ubuntu 18.04 (CUDA 11.0) +#### Ubuntu 18.04 (CUDA 11.2) 
 # Add NVIDIA package repositories
-wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
-sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
-sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
-sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
+# Note: For Ubuntu version other than 18.04 or CPU arch other than x86,
+# replace 'ubuntu1804' and/or 'x86_64' as needed in the following URL.
+wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
+sudo dpkg -i cuda-keyring_1.0-1_all.deb
 sudo apt-get update
 
 # Install development and runtime libraries (~4GB)
@@ -114,19 +113,18 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i
     libnvinfer-plugin8=8.2.4-1+cuda11.4 \
     libnvinfer-plugin-dev=8.2.4-1+cuda11.4
 
+
 # Reboot. Check that GPUs are visible using the command: nvidia-smi
-#### Ubuntu 16.04 (CUDA 11.0) +#### Ubuntu 16.04 (CUDA 11.2) 
 # Add NVIDIA package repositories
-# Add HTTPS support for apt-key
-sudo apt-get install gnupg-curl
-wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-ubuntu1604.pin
-sudo mv cuda-ubuntu1604.pin /etc/apt/preferences.d/cuda-repository-pin-600
-sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
-sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/ /"
+sudo apt-get update
+sudo apt-get install -y 
+wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-keyring_1.0-1_all.deb
+sudo dpkg -i cuda-keyring_1.0-1_all.deb
 sudo apt-get update
 
 # Install development and runtime libraries (~4GB)

From dc63f4e28b445fbf1bd2e834ee885c938831aa7d Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 2 May 2022 10:47:00 -0700
Subject: [PATCH 106/872] Add a TensorFlow-models tutorial.

PiperOrigin-RevId: 445973755
---
 site/en/tutorials/images/models_vision.ipynb | 595 +++++++++++++++++++
 1 file changed, 595 insertions(+)
 create mode 100644 site/en/tutorials/images/models_vision.ipynb

diff --git a/site/en/tutorials/images/models_vision.ipynb b/site/en/tutorials/images/models_vision.ipynb
new file mode 100644
index 00000000000..03bd72d2b1b
--- /dev/null
+++ b/site/en/tutorials/images/models_vision.ipynb
@@ -0,0 +1,595 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Tce3stUlHN0L"
+      },
+      "source": [
+        "##### Copyright 2020 The TensorFlow Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "tuOe1ymfHZPu"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qFdPvlXBOdUN"
+      },
+      "source": [
+        "# Use TensorFlow Models: Fine tune a ResNet"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MfBg1C5NB3X0"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    View on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Ta_nFXaVAqLD"
+      },
+      "source": [
+        "This tutorial uses the TensorFlow Models package to fine-tune a ResNet."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "G2FlaQcEPOER"
+      },
+      "source": [
+        "## Setup\n",
+        "\n",
+        "Install and import the necessary modules"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "XvWfdCrvrV5W"
+      },
+      "outputs": [],
+      "source": [
+        "!pip uninstall -y opencv-python\n",
+        "!pip install -q tf-models-nightly"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Wlon1uoIowmZ"
+      },
+      "outputs": [],
+      "source": [
+        "# Import helper libraries\n",
+        "import pprint\n",
+        "import tempfile\n",
+        "\n",
+        "from IPython import display\n",
+        "import matplotlib.pyplot as plt\n",
+        "\n",
+        "import tensorflow as tf\n",
+        "import tensorflow_datasets as tfds"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NHT1iiIiBzlC"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow_models as tfm\n",
+        "\n",
+        "# Not in the tfm public API for v2.9. Will be available as `vision.serving` in v2.10\n",
+        "from official.vision.serving import export_saved_model_lib  "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aKv3wdqkQ8FU"
+      },
+      "source": [
+        "## Cifar-10 with ResNet-18 Backbone"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5iN8mHEJjKYE"
+      },
+      "source": [
+        "Base the experiment on `\"resnet_imagenet\"` configuration (defined by `tfm.vision.configs.image_classification.image_classification_imagenet`)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1M77f88Dj2Td"
+      },
+      "outputs": [],
+      "source": [
+        "exp_config = tfm.core.exp_factory.get_exp_config('resnet_imagenet')\n",
+        "tfds_name = 'cifar10'\n",
+        "ds_info = tfds.builder(tfds_name ).info\n",
+        "ds_info"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "U6PVwXA-j3E7"
+      },
+      "source": [
+        "Next adjust the configuration so that it works with `cifar10`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YWI7faVStQaV"
+      },
+      "outputs": [],
+      "source": [
+        "# Change model\n",
+        "exp_config.task.model.num_classes = 10\n",
+        "exp_config.task.model.input_size = list(ds_info.features[\"image\"].shape)\n",
+        "exp_config.task.model.backbone.resnet.model_id = 18\n",
+        "\n",
+        "# Change train, eval data\n",
+        "batch_size = 128\n",
+        "\n",
+        "exp_config.task.train_data.input_path = ''\n",
+        "exp_config.task.train_data.tfds_name = tfds_name\n",
+        "exp_config.task.train_data.tfds_split = 'train'\n",
+        "exp_config.task.train_data.global_batch_size = batch_size\n",
+        "\n",
+        "exp_config.task.validation_data.input_path = ''\n",
+        "exp_config.task.validation_data.tfds_name = tfds_name\n",
+        "exp_config.task.validation_data.tfds_split = 'test'\n",
+        "exp_config.task.validation_data.global_batch_size = batch_size\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DE3ggKzzTD56"
+      },
+      "source": [
+        "Adjust the trainer configuration:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "inE_-4UGkLud"
+      },
+      "outputs": [],
+      "source": [
+        "# Change trainer config\n",
+        "train_steps = 5000\n",
+        "\n",
+        "exp_config.trainer.steps_per_loop = 100\n",
+        "exp_config.trainer.summary_interval = 100\n",
+        "exp_config.trainer.checkpoint_interval = train_steps\n",
+        "exp_config.trainer.validation_interval = 1000\n",
+        "exp_config.trainer.validation_steps =  ds_info.splits['test'].num_examples // batch_size\n",
+        "exp_config.trainer.train_steps = train_steps\n",
+        "exp_config.trainer.optimizer_config.learning_rate.type = 'cosine'\n",
+        "exp_config.trainer.optimizer_config.learning_rate.cosine.decay_steps = train_steps\n",
+        "exp_config.trainer.optimizer_config.learning_rate.cosine.initial_learning_rate = 0.1\n",
+        "exp_config.trainer.optimizer_config.warmup.linear.warmup_steps = 100"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5mTcDnBiTOYD"
+      },
+      "source": [
+        "And set the runtime configuration."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "tuVfxSBCTK-y"
+      },
+      "outputs": [],
+      "source": [
+        "pprint.pprint(exp_config.as_dict())\n",
+        "\n",
+        "display.Javascript(\"google.colab.output.setIframeHeight('300px');\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "w7_X0UHaRF2m"
+      },
+      "source": [
+        "Set up the distribution strategy:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ykL14FIbTaSt"
+      },
+      "outputs": [],
+      "source": [
+        "logical_device_names = [logical_device.name for logical_device in tf.config.list_logical_devices()]\n",
+        "\n",
+        "if exp_config.runtime.mixed_precision_dtype == tf.float16:\n",
+        "    tf.keras.mixed_precision.set_global_policy('mixed_float16')\n",
+        "\n",
+        "if 'GPU' in ''.join(logical_device_names):\n",
+        "  distribution_strategy = tf.distribute.MirroredStrategy()\n",
+        "elif 'TPU' in ''.join(logical_device_names):\n",
+        "  tf.tpu.experimental.initialize_tpu_system()\n",
+        "  tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='/device:TPU_SYSTEM:0')\n",
+        "  distribution_strategy = tf.distribute.experimental.TPUStrategy(tpu)\n",
+        "else:\n",
+        "  print('Warning: this will be really slow.')\n",
+        "  distribution_strategy = tf.distribute.OneDeviceStrategy(logical_device_names[0])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "W4k5YH5pTjaK"
+      },
+      "source": [
+        "Create the `Task` object (ref: `tfm.core.base_task.Task`) form the `config_definitions.TaskConfig`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6MgYSH0PtUaW"
+      },
+      "outputs": [],
+      "source": [
+        "with distribution_strategy.scope():\n",
+        "  model_dir = tempfile.mkdtemp()\n",
+        "  task = tfm.core.task_factory.get_task(exp_config.task, logging_dir=model_dir)\n",
+        "\n",
+        "tf.keras.utils.plot_model(task.build_model(), show_shapes=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IFXEZYdzBKoX"
+      },
+      "outputs": [],
+      "source": [
+        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
+        "  print()\n",
+        "  print(f'images.shape: {str(images.shape):16}  images.dtype: {images.dtype!r}')\n",
+        "  print(f'labels.shape: {str(labels.shape):16}  labels.dtype: {labels.dtype!r}')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yrwxnGDaRU0U"
+      },
+      "source": [
+        "## Visualize Training Dataloader"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "683c255c6c52"
+      },
+      "source": [
+        "The data-loader applies a z-score normalization using \n",
+        "`preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`, so the images returned by the dataset can't be directly displayed by standard tools, so rescale the minimum to 0.0 and the maximum to 1.0: "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PdmOz2EC0Nx2"
+      },
+      "outputs": [],
+      "source": [
+        "plt.hist(images.numpy().flatten());"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7a8582ebde7b"
+      },
+      "source": [
+        "You can use the `tfds.core.DatasetInfo` (`ds_info` from earlier) to lookup the text descriptions of each class ID. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Wq4Wq_CuDG3Q"
+      },
+      "outputs": [],
+      "source": [
+        "label_info = ds_info.features['label']\n",
+        "label_info.int2str(1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8c652a6fdbcf"
+      },
+      "source": [
+        "Use these to disualize a batch of the data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZKfTxytf1l0d"
+      },
+      "outputs": [],
+      "source": [
+        "def show_batch(images, labels, predictions=None):\n",
+        "  plt.figure(figsize=(10, 10))\n",
+        "  min = images.numpy().min()\n",
+        "  max = images.numpy().max()\n",
+        "  delta = max - min\n",
+        "\n",
+        "  for i in range(12):\n",
+        "    plt.subplot(6, 6, i + 1)\n",
+        "    plt.imshow((images[i]-min) / delta)\n",
+        "    if predictions is None:\n",
+        "      plt.title(label_info.int2str(labels[i]))\n",
+        "    else:\n",
+        "      if labels[i] == predictions[i]:\n",
+        "        color = 'g'\n",
+        "      else:\n",
+        "        color = 'r'\n",
+        "      plt.title(label_info.int2str(predictions[i]), color=color)\n",
+        "    plt.axis(\"off\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xkA5h_RBtYYU"
+      },
+      "outputs": [],
+      "source": [
+        "plt.figure(figsize=(10, 10))\n",
+        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
+        "  show_batch(images, labels)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "v_A9VnL2RbXP"
+      },
+      "source": [
+        "## Visualize Evaluation Dataloader"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Ma-_Eb-nte9A"
+      },
+      "outputs": [],
+      "source": [
+        "plt.figure(figsize=(10, 10));\n",
+        "for images, labels in task.build_inputs(exp_config.task.validation_data).take(1):\n",
+        "  show_batch(images, labels)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ihKJt2FHRi2N"
+      },
+      "source": [
+        "## Train and Evaluate"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0AFMNvYxtjXx"
+      },
+      "outputs": [],
+      "source": [
+        "model, eval_logs = tfm.core.train_lib.run_experiment(\n",
+        "    distribution_strategy=distribution_strategy,\n",
+        "    task=task,\n",
+        "    mode='train_and_eval',\n",
+        "    params=exp_config,\n",
+        "    model_dir=model_dir,\n",
+        "    run_post_eval=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gCcHMQYhozmA"
+      },
+      "outputs": [],
+      "source": [
+        "tf.keras.utils.plot_model(model, show_shapes=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0124f938a1b9"
+      },
+      "outputs": [],
+      "source": [
+        "for key, value in eval_logs.items():\n",
+        "    print(f'{key:20}: {value.numpy():.3f}')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fkE9locGTBgt"
+      },
+      "source": [
+        "## Export a SavedModel"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9669d08c91af"
+      },
+      "source": [
+        "The `keras.Model` object returned by `train_lib.run_experiment` expects the data to be normalized by the dataset loader using the same mean and variance statiscics in `preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`. This export function handles those details so you can pass `tf.uint8` images and get correct result.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AQCFa7BvtmDg"
+      },
+      "outputs": [],
+      "source": [
+        "# Saving and exporting the trained model\n",
+        "export_saved_model_lib.export_inference_graph(\n",
+        "    input_type='image_tensor',\n",
+        "    batch_size=1,\n",
+        "    input_image_size=[32, 32],\n",
+        "    params=exp_config,\n",
+        "    checkpoint_path=tf.train.latest_checkpoint(model_dir),\n",
+        "    export_dir='./export/')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vVr6DxNqTyLZ"
+      },
+      "source": [
+        "Test the exported model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gP7nOvrftsB0"
+      },
+      "outputs": [],
+      "source": [
+        "# Importing SavedModel\n",
+        "imported = tf.saved_model.load('./export/')\n",
+        "model_fn = imported.signatures['serving_default']"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "GiOp2WVIUNUZ"
+      },
+      "source": [
+        "Visualize the predictions"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "BTRMrZQAN4mk"
+      },
+      "outputs": [],
+      "source": [
+        "plt.figure(figsize=(10, 10))\n",
+        "for data in tfds.load('cifar10', split='test').batch(12).take(1):\n",
+        "  predictions = []\n",
+        "  for image in data['image']:\n",
+        "    index = tf.argmax(model_fn(image[tf.newaxis, ...])['logits'], axis=1)[0]\n",
+        "    predictions.append(index)\n",
+        "  show_batch(data['image'], data['label'], predictions)"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "collapsed_sections": [],
+      "name": "models_vision.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From cf80305cda5bb919ca897c3e6d57e7c418265602 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Mon, 2 May 2022 12:49:41 -0700
Subject: [PATCH 107/872] Minor changes to improve terminology consistency
 (DTensor colab notebooks)

PiperOrigin-RevId: 446005959
---
 .../distribute/dtensor_keras_tutorial.ipynb   | 18 ++----------
 .../distribute/dtensor_ml_tutorial.ipynb      | 29 ++++---------------
 2 files changed, 8 insertions(+), 39 deletions(-)

diff --git a/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb
index 7241f5fb2b8..d6bfa26544e 100644
--- a/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb
+++ b/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb
@@ -13,6 +13,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
+        "cellView": "form",
         "id": "tuOe1ymfHZPu"
       },
       "outputs": [],
@@ -36,7 +37,7 @@
         "id": "MT-LkFOl2axM"
       },
       "source": [
-        "# DTensor Integration with Keras"
+        "# Using DTensors with Keras"
       ]
     },
     {
@@ -739,32 +740,17 @@
         "\n",
         "print(model.layers[2].kernel.layout)"
       ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "00dPVoSlRLFA"
-      },
-      "outputs": [],
-      "source": [
-        ""
-      ]
     }
   ],
   "metadata": {
     "colab": {
       "collapsed_sections": [],
       "name": "dtensor_keras_tutorial.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {
       "display_name": "Python 3",
       "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
     }
   },
   "nbformat": 4,
diff --git a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
index 59c9cc9643d..8220b5ce358 100644
--- a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
+++ b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
@@ -13,6 +13,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
+        "cellView": "form",
         "id": "tuOe1ymfHZPu"
       },
       "outputs": [],
@@ -36,7 +37,7 @@
         "id": "MfBg1C5NB3X0"
       },
       "source": [
-        "# DTensor Maching Learning Tutorial\n"
+        "# Distributed Training with DTensors\n"
       ]
     },
     {
@@ -75,7 +76,7 @@
         " \n",
         " - Data Parallel training, where the training samples are sharded (partitioned) to devices.\n",
         " - Model Parallel training, where the model variables are sharded to devices. \n",
-        " - Spatial Parallel training, where the features of input data are  sharded to devices.\n",
+        " - Spatial Parallel training, where the features of input data are  sharded to devices. (Also known as [Spatial Partitioning](https://cloud.google.com/blog/products/ai-machine-learning/train-ml-models-on-large-images-and-3d-volumes-with-spatial-partitioning-on-cloud-tpus))\n",
         "\n",
         "The training portion of this tutorial is inspired [A Kaggle guide on Sentiment Analysis](https://www.kaggle.com/code/anasofiauzsoy/yelp-review-sentiment-analysis-tensorflow-tfds/notebook) notebook. To learn about the complete training and evaluation workflow (without DTensor), refer to that notebook. \n",
         "\n",
@@ -237,8 +238,7 @@
         "    'y': dataset_y,\n",
         "})\n",
         "\n",
-        "dataset.take(1).get_single_element()\n",
-        "\n"
+        "dataset.take(1).get_single_element()\n"
       ]
     },
     {
@@ -297,7 +297,6 @@
         "id": "PMCt-Gj3b3Jy"
       },
       "source": [
-        "\n",
         "### Dense Layer\n",
         "\n",
         "The following custom Dense layer defines 2 layer variables: $W_{ij}$ is the variable for weights, and $b_i$ is the variable for the biases.\n",
@@ -809,8 +808,7 @@
         "- The 2 devices within a single model replica receive replicated training data.\n",
         "\n",
         "\n",
-        "\"Model\n",
-        "\n"
+        "\"Model\n"
       ]
     },
     {
@@ -905,7 +903,7 @@
         "id": "u-bK6IZ9GCS9"
       },
       "source": [
-        "When training data of very high dimensionality (e.g. a very large image or a video), it may be desirable to shard along the feature dimension. This is called Spatial Parallel training.\n",
+        "When training data of very high dimensionality (e.g. a very large image or a video), it may be desirable to shard along the feature dimension. This is called [Spatial Partitioning](https://cloud.google.com/blog/products/ai-machine-learning/train-ml-models-on-large-images-and-3d-volumes-with-spatial-partitioning-on-cloud-tpus), which was first introduced into TensorFlow for training models with large 3-d input samples.\n",
         "\n",
         "\"Spatial\n",
         "\n",
@@ -1067,32 +1065,17 @@
         "Composing a model with `tf.Module` from scratch is a lot of work, and reusing existing building blocks such as layers and helper functions can drastically speed up model development.\n",
         "As of TensorFlow 2.9, all Keras Layers under `tf.keras.layers` accepts DTensor layouts as their arguments, and can be used to build DTensor models. You can even directly reuse a Keras model with DTensor without modifying the model implementation. Refer to the [DTensor Keras Integration Tutorial](link) (TODO: add link) for information on using DTensor Keras. "
       ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "A-YWPfJyHPcX"
-      },
-      "outputs": [],
-      "source": [
-        ""
-      ]
     }
   ],
   "metadata": {
     "colab": {
       "collapsed_sections": [],
       "name": "dtensor_ml_tutorial.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {
       "display_name": "Python 3",
       "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
     }
   },
   "nbformat": 4,

From 21b2a4abfcee536a4affcded363c4831faa36aa6 Mon Sep 17 00:00:00 2001
From: Xinyi Wang 
Date: Mon, 2 May 2022 15:11:26 -0700
Subject: [PATCH 108/872] Keep tutorial up-to-date. (custom training loop with
 MWMS)

PiperOrigin-RevId: 446040406
---
 .../distribute/multi_worker_with_ctl.ipynb    | 128 ++++++++----------
 1 file changed, 53 insertions(+), 75 deletions(-)

diff --git a/site/en/tutorials/distribute/multi_worker_with_ctl.ipynb b/site/en/tutorials/distribute/multi_worker_with_ctl.ipynb
index 859a117d2f2..aa53a89bb12 100644
--- a/site/en/tutorials/distribute/multi_worker_with_ctl.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_ctl.ipynb
@@ -63,11 +63,9 @@
       "source": [
         "## Overview\n",
         "\n",
-        "This tutorial demonstrates multi-worker training with custom training loop API, distributed via MultiWorkerMirroredStrategy, so a Keras model designed to run on [single-worker](https://www.tensorflow.org/tutorials/distribute/custom_training) can seamlessly work on multiple workers with minimal code change.\n",
+        "This tutorial demonstrates how to perform multi-worker distributed training with a Keras model and with [custom training loops](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) using the `tf.distribute.Strategy` API. The training loop is distributed via `tf.distribute.MultiWorkerMirroredStrategy`, such that a `tf.keras` model—designed to run on [single-worker](custom_training.ipynb)—can seamlessly work on multiple workers with minimal code changes. Custom training loops provide flexibility and a greater control on training, while also making it easier to debug the model. Learn more about [writing a basic training loop](../../guide/basic_training_loops.ipynb), [writing a training loop from scratch](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) and [custom training](../customization/custom_training_walkthrough.ipynb).\n",
         "\n",
-        "We are using custom training loops to train our model because they give us flexibility and a greater control on training. Moreover, it is easier to debug the model and the training loop. More detailed information is available in [Writing a training loop from scratch](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch).\n",
-        "\n",
-        "If you are looking for how to use `MultiWorkerMirroredStrategy` with keras `model.fit`, refer to this [tutorial](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras) instead.\n",
+        "If you are looking for how to use `MultiWorkerMirroredStrategy` with `tf.keras.Model.fit`, refer to this [tutorial](multi_worker_with_keras.ipynb) instead.\n",
         "\n",
         "[Distributed Training in TensorFlow](../../guide/distributed_training.ipynb) guide is available for an overview of the distribution strategies TensorFlow supports for those interested in a deeper understanding of `tf.distribute.Strategy` APIs."
       ]
@@ -102,9 +100,8 @@
         "id": "Zz0EY91y3mxy"
       },
       "source": [
-        "Before importing TensorFlow, make a few changes to the environment.\n",
-        "\n",
-        "Disable all GPUs. This prevents errors caused by the workers all trying to use the same GPU. For a real application each worker would be on a different machine."
+        "Before importing TensorFlow, make a few changes to the environment:\n",
+        "* Disable all GPUs. This prevents errors caused by all workers trying to use the same GPU. In a real-world application, each worker would be on a different machine."
       ]
     },
     {
@@ -124,7 +121,7 @@
         "id": "7X1MS6385BWi"
       },
       "source": [
-        "Reset the `TF_CONFIG` environment variable, you'll see more about this later."
+        "* Reset the `'TF_CONFIG'` environment variable (you'll see more about this later)."
       ]
     },
     {
@@ -144,7 +141,7 @@
         "id": "Rd4L9Ii77SS8"
       },
       "source": [
-        "Be sure that the current directory is on python's path. This allows the notebook to import the files written by `%%writefile` later.\n"
+        "* Make sure that the current directory is on Python's path. This allows the notebook to import the files written by `%%writefile` later.\n"
       ]
     },
     {
@@ -194,7 +191,7 @@
         "id": "fLW6D2TzvC-4"
       },
       "source": [
-        "Next create an `mnist.py` file with a simple model and dataset setup. This python file will be used by the worker-processes in this tutorial:"
+        "Next, create an `mnist.py` file with a simple model and dataset setup. This Python file will be used by the worker-processes in this tutorial:"
       ]
     },
     {
@@ -246,9 +243,9 @@
         "id": "JmgZwwymxqt5"
       },
       "source": [
-        "## Multi-worker Configuration\n",
+        "## Multi-worker configuration\n",
         "\n",
-        "Now let's enter the world of multi-worker training. In TensorFlow, the `TF_CONFIG` environment variable is required for training on multiple machines, each of which possibly has a different role. `TF_CONFIG` used below, is a JSON string used to specify the cluster configuration on each worker that is part of the cluster. This is the default method for specifying a cluster, using `cluster_resolver.TFConfigClusterResolver`,  but there are other options available in the `distribute.cluster_resolver` module."
+        "Now let's enter the world of multi-worker training. In TensorFlow, the `'TF_CONFIG'` environment variable is required for training on multiple machines. Each machine may have a different role. The `'TF_CONFIG'` variable used below is a JSON string that specifies the cluster configuration on each worker that is part of the cluster. This is the default method for specifying a cluster, using `cluster_resolver.TFConfigClusterResolver`,  but there are other options available in the `distribute.cluster_resolver` module. Learn more about setting up the `'TF_CONFIG'` variable in the [Distributed training guide](../../guide/distributed_training.ipynb)."
       ]
     },
     {
@@ -283,7 +280,7 @@
         "id": "JjgwJbPKZkJL"
       },
       "source": [
-        "Here is the same `TF_CONFIG` serialized as a JSON string:"
+        "Note that `tf_config` is just a local variable in Python. To use it for training configuration, serialize it as a JSON and place it in a `'TF_CONFIG'` environment variable. Here is the same `'TF_CONFIG'` serialized as a JSON string:"
       ]
     },
     {
@@ -303,11 +300,11 @@
         "id": "AUBmYRZqxthH"
       },
       "source": [
-        "There are two components of `TF_CONFIG`: `cluster` and `task`.\n",
+        "There are two components of `'TF_CONFIG'`: `'cluster'` and `'task'`.\n",
         "\n",
-        "* `cluster` is the same for all workers and provides information about the training cluster, which is a dict consisting of different types of jobs such as `worker`. In multi-worker training with `MultiWorkerMirroredStrategy`, there is usually one `worker` that takes on a little more responsibility like saving checkpoint and writing summary file for TensorBoard in addition to what a regular `worker` does. Such a worker is referred to as the `chief` worker, and it is customary that the `worker` with `index` 0 is appointed as the chief `worker` (in fact this is how `tf.distribute.Strategy` is implemented).\n",
+        "* `'cluster'` is the same for all workers and provides information about the training cluster, which is a dict consisting of different types of jobs such as `'worker'`. In multi-worker training with `MultiWorkerMirroredStrategy`, there is usually one `'worker'` that takes on a little more responsibility like saving checkpoints and writing summary files for TensorBoard in addition to what a regular `'worker'` does. Such a worker is referred to as the `'chief'` worker, and it is customary that the `'worker'` with `'index'` 0 is appointed as the chief `worker`.\n",
         "\n",
-        "* `task` provides information of the current task and is different on each worker. It specifies the `type` and `index` of that worker."
+        "* `'task'` provides information of the current task and is different on each worker. It specifies the `'type'` and `'index'` of that worker."
       ]
     },
     {
@@ -316,7 +313,7 @@
         "id": "8YFpxrcsZ2xG"
       },
       "source": [
-        "In this example, you set the task `type` to `\"worker\"` and the task `index` to `0`. This machine is the first worker and will be appointed as the chief worker and do more work than the others. Note that other machines will need to have the `TF_CONFIG` environment variable set as well, and it should have the same `cluster` dict, but different task `type` or task `index` depending on what the roles of those machines are.\n"
+        "In this example, you set the task `'type'` to `'worker'` and the task `'index'` to `0`. This machine is the first worker and will be appointed as the chief worker and do more work than the others. Note that other machines will need to have the `'TF_CONFIG'` environment variable set as well, and it should have the same `'cluster'` dict, but different task `'type'` or task `'index'` depending on what the roles of those machines are.\n"
       ]
     },
     {
@@ -325,18 +322,9 @@
         "id": "aogb74kHxynz"
       },
       "source": [
-        "For illustration purposes, this tutorial shows how one may set a `TF_CONFIG` with 2 workers on `localhost`.  In practice, users would create multiple workers on external IP addresses/ports, and set `TF_CONFIG` on each worker appropriately.\n",
+        "For illustration purposes, this tutorial shows how one may set a `'TF_CONFIG'` with two workers on `'localhost'`.  In practice, users would create multiple workers on external IP addresses/ports, and set `'TF_CONFIG'` on each worker appropriately.\n",
         "\n",
-        "In this example you will use 2 workers, the first worker's `TF_CONFIG` is shown above. For the second worker you would set `tf_config['task']['index']=1`"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "f83FVYqDX3aX"
-      },
-      "source": [
-        "Above, `tf_config` is just a local variable in python. To actually use it to configure training, this dictionary needs to be serialized as JSON, and placed in the `TF_CONFIG` environment variable."
+        "This example uses two workers. The first worker's `'TF_CONFIG'` is shown above. For the second worker, set `tf_config['task']['index']=1`."
       ]
     },
     {
@@ -354,7 +342,7 @@
         "id": "FcjAbuGY1ACJ"
       },
       "source": [
-        "Subprocesses inherit environment variables from their parent. So if you set an environment variable in this `jupyter notebook` process:"
+        "Subprocesses inherit environment variables from their parent. So if you set an environment variable in this Jupyter Notebook process:"
       ]
     },
     {
@@ -374,7 +362,7 @@
         "id": "gQkIX-cg18md"
       },
       "source": [
-        "You can access the environment variable from a subprocesses:"
+        "you can then access the environment variable from a subprocess:"
       ]
     },
     {
@@ -395,7 +383,7 @@
         "id": "af6BCA-Y2fpz"
       },
       "source": [
-        "In the next section, you'll use this to pass the `TF_CONFIG` to the worker subprocesses. You would never really launch your jobs this way, but it's sufficient for the purposes of this tutorial: To demonstrate a minimal multi-worker example."
+        "In the next section, you'll use this to pass the `'TF_CONFIG'` to the worker subprocesses. You would never really launch your jobs this way, but it's sufficient for the purposes of this tutorial: To demonstrate a minimal multi-worker example."
       ]
     },
     {
@@ -406,7 +394,7 @@
       "source": [
         "## MultiWorkerMirroredStrategy\n",
         "\n",
-        "To train the model, use an instance of `tf.distribute.MultiWorkerMirroredStrategy`, which creates copies of all variables in the model's layers on each device across all workers. The [`tf.distribute.Strategy` guide](../../guide/distributed_training.ipynb) has more details about this strategy."
+        "Before training the model, first create an instance of `tf.distribute.MultiWorkerMirroredStrategy`:"
       ]
     },
     {
@@ -426,7 +414,7 @@
         "id": "N0iv7SyyAohc"
       },
       "source": [
-        "Note: `TF_CONFIG` is parsed and TensorFlow's GRPC servers are started at the time `MultiWorkerMirroredStrategy()` is called, so the `TF_CONFIG` environment variable must be set before a `tf.distribute.Strategy` instance is created. To save time in this illustrative example we have not done this so that servers do not need to start. A full example is found in the last section of this tutorial."
+        "Note: `'TF_CONFIG'` is parsed and TensorFlow's GRPC servers are started at the time you call `tf.distribute.MultiWorkerMirroredStrategy.` Therefore, you must set the `'TF_CONFIG'` environment variable before you instantiate a `tf.distribute.Strategy`. To save time in this illustrative example, this is not demonstrated in this tutorial, so that servers do not need to start. You can find a full example in the last section of this tutorial."
       ]
     },
     {
@@ -435,7 +423,7 @@
         "id": "TS4S-faBHHam"
       },
       "source": [
-        "Use `tf.distribute.Strategy.scope` to specify that a strategy should be used when building your model. This puts you in the \"[cross-replica context](https://www.tensorflow.org/guide/distributed_training?hl=en#mirroredstrategy)\" for this strategy, which means the strategy is put in control of things like variable placement."
+        "Use `tf.distribute.Strategy.scope` to specify that a strategy should be used when building your model. This allows the strategy to control things like variable placement—it will create copies of all variables in the model's layers on each device across all workers."
       ]
     },
     {
@@ -459,9 +447,8 @@
       },
       "source": [
         "## Auto-shard your data across workers\n",
-        "In multi-worker training, dataset sharding is not necessarily needed, however it gives you exactly-once semantics which makes more training more reproducible, i.e. training on multiple workers should be the same as training on one worker. Note: performance can be affected in some cases.\n",
         "\n",
-        "See: [`distribute_datasets_from_function`](https://www.tensorflow.org/api_docs/python/tf/distribute/Strategy?version=nightly#distribute_datasets_from_function)"
+        "In multi-worker training, _dataset sharding_ is needed to ensure convergence and reproducibility. Sharding means handing each worker a subset of the entire dataset—it helps create the experience similar to training on a single worker. In the example below, you're relying on the default autosharding policy of `tf.distribute`. You can also customize it by setting the `tf.data.experimental.AutoShardPolicy` of the `tf.data.experimental.DistributeOptions`. To learn more, refer to the _Sharding_ section of the [Distributed input tutorial](input.ipynb)."
       ]
     },
     {
@@ -487,8 +474,8 @@
         "id": "rkNzSR3g60iP"
       },
       "source": [
-        "## Define Custom Training Loop and Train the model\n",
-        "Specify an optimizer"
+        "## Define a custom training loop and train the model\n",
+        "Specify an optimizer:"
       ]
     },
     {
@@ -500,7 +487,7 @@
       "outputs": [],
       "source": [
         "with strategy.scope():\n",
-        "  # The creation of optimizer and train_accuracy will need to be in\n",
+        "  # The creation of optimizer and train_accuracy needs to be in\n",
         "  # `strategy.scope()` as well, since they create variables.\n",
         "  optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)\n",
         "  train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(\n",
@@ -513,7 +500,7 @@
         "id": "RmrDcAii4B5O"
       },
       "source": [
-        "Define a training step with `tf.function`\n"
+        "Define a training step with `tf.function`:\n"
       ]
     },
     {
@@ -558,7 +545,7 @@
       "source": [
         "### Checkpoint saving and restoring\n",
         "\n",
-        "Checkpointing implementation in a Custom Training Loop requires the user to handle it instead of using a keras callback. It allows you to save model's weights and restore them without having to save the whole model."
+        "As you write a custom training loop, you need to handle [checkpoint saving](../../guide/checkpoint.ipynb) manually instead of relying on a Keras callback. Note that for `MultiWorkerMirroredStrategy`, saving a checkpoint or a complete model requires the participation of all workers, because attempting to save only on the chief worker could lead to a deadlock. Workers also need to write to different paths to avoid overwriting each other. Here's an example of how to configure the directories:"
       ]
     },
     {
@@ -593,23 +580,13 @@
         "  return os.path.join(dirpath, base)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "P7fabUIEW7-M"
-      },
-      "source": [
-        "Note: Checkpointing and Saving need to happen on each worker and they need to write to different paths as they would override each others.\n",
-        "If you chose to only checkpoint/save on the chief, this can lead to deadlock and is not recommended."
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "nrcdPHtG4ObO"
       },
       "source": [
-        " Here, you'll create one `tf.train.Checkpoint` that tracks the model, which is managed by a `tf.train.CheckpointManager` so that only the latest checkpoint is preserved."
+        "Create one `tf.train.Checkpoint` that tracks the model, which is managed by a `tf.train.CheckpointManager`, so that only the latest checkpoints are preserved:"
       ]
     },
     {
@@ -627,9 +604,9 @@
         "    name='step_in_epoch')\n",
         "task_type, task_id = (strategy.cluster_resolver.task_type,\n",
         "                      strategy.cluster_resolver.task_id)\n",
-        "# We normally don't need to manually instantiate a ClusterSpec, but in this \n",
-        "# illustrative example we did not set TF_CONFIG before initializing the\n",
-        "# strategy. See the next section for \"real-world\" usage.\n",
+        "# Normally, you don't need to manually instantiate a `ClusterSpec`, but in this \n",
+        "# illustrative example you did not set `'TF_CONFIG'` before initializing the\n",
+        "# strategy. Check out the next section for \"real-world\" usage.\n",
         "cluster_spec = tf.train.ClusterSpec(tf_config['cluster'])\n",
         "\n",
         "checkpoint = tf.train.Checkpoint(\n",
@@ -647,7 +624,7 @@
         "id": "RO7cbN40XD5v"
       },
       "source": [
-        "Now, when you need to restore, you can find the latest checkpoint saved using the convenient `tf.train.latest_checkpoint` function."
+        "Now, when you need to restore a checkpoint, you can find the latest checkpoint saved using the convenient `tf.train.latest_checkpoint` function (or by calling `tf.train.CheckpointManager.restore_or_initialize`)."
       ]
     },
     {
@@ -715,7 +692,7 @@
         "id": "0W1Osks466DE"
       },
       "source": [
-        "## Full code setup on workers"
+        "## Complete code at a glance"
       ]
     },
     {
@@ -724,10 +701,11 @@
         "id": "jfYpmIxO6Jck"
       },
       "source": [
-        "To actually run with `MultiWorkerMirroredStrategy` you'll need to run worker processes and pass a `TF_CONFIG` to them.\n",
+        "To sum up all the procedures discussed so far:\n",
         "\n",
-        "Like the `mnist.py` file written earlier, here is the `main.py` that \n",
-        "contain the same code we walked through step by step previously in this colab, we're just writing it to a file so each of the workers will run it:"
+        "1. You create worker processes.\n",
+        "2. Pass `'TF_CONFIG'`s to the worker processes.\n",
+        "3. Let each work process run the script below that contains the training code."
       ]
     },
     {
@@ -781,7 +759,7 @@
         "strategy = tf.distribute.MultiWorkerMirroredStrategy()\n",
         "\n",
         "with strategy.scope():\n",
-        "  # Model building/compiling need to be within `strategy.scope()`.\n",
+        "  # Model building/compiling need to be within `tf.distribute.Strategy.scope`.\n",
         "  multi_worker_model = mnist.build_cnn_model()\n",
         "\n",
         "  multi_worker_dataset = strategy.distribute_datasets_from_function(\n",
@@ -870,7 +848,6 @@
         "id": "ItVOvPN1qnZ6"
       },
       "source": [
-        "## Train and Evaluate\n",
         "The current directory now contains both Python files:"
       ]
     },
@@ -892,7 +869,7 @@
         "id": "qmEEStPS6vR_"
       },
       "source": [
-        "So json-serialize the `TF_CONFIG` and add it to the environment variables:"
+        "So JSON-serialize the `'TF_CONFIG'` and add it to the environment variables:"
       ]
     },
     {
@@ -912,7 +889,7 @@
         "id": "MsY3dQLK7jdf"
       },
       "source": [
-        "Now, you can launch a worker process that will run the `main.py` and use the `TF_CONFIG`:"
+        "Now, you can launch a worker process that will run the `main.py` and use the `'TF_CONFIG'`:"
       ]
     },
     {
@@ -950,9 +927,9 @@
         "1. It uses the `%%bash` which is a [notebook \"magic\"](https://ipython.readthedocs.io/en/stable/interactive/magics.html) to run some bash commands.\n",
         "2. It uses the `--bg` flag to run the `bash` process in the background, because this worker will not terminate. It waits for all the workers before it starts.\n",
         "\n",
-        "The backgrounded worker process won't print output to this notebook, so the `&>` redirects its output to a file, so you can see what happened.\n",
+        "The backgrounded worker process won't print the output to this notebook. The `&>` redirects its output to a file, so that you can inspect what happened.\n",
         "\n",
-        "So, wait a few seconds for the process to start up:"
+        "Wait a few seconds for the process to start up:"
       ]
     },
     {
@@ -973,7 +950,7 @@
         "id": "ZFPoNxg_9_Mx"
       },
       "source": [
-        "Now look what's been output to the worker's logfile so far:"
+        "Now, check the output to the worker's log file so far:"
       ]
     },
     {
@@ -1003,7 +980,7 @@
         "id": "Pi8vPNNA_l4a"
       },
       "source": [
-        "So update the `tf_config` for the second worker's process to pick up:"
+        "Update the `tf_config` for the second worker's process to pick up:"
       ]
     },
     {
@@ -1045,7 +1022,7 @@
         "id": "hX4FA2O2AuAn"
       },
       "source": [
-        "Now if you recheck the logs written by the first worker you'll see that it participated in training that model:"
+        "If you recheck the logs written by the first worker, notice that it participated in training that model:"
       ]
     },
     {
@@ -1068,7 +1045,7 @@
       },
       "outputs": [],
       "source": [
-        "# Delete the `TF_CONFIG`, and kill any background tasks so they don't affect the next section.\n",
+        "# Delete the `'TF_CONFIG'`, and kill any background tasks so they don't affect the next section.\n",
         "os.environ.pop('TF_CONFIG', None)\n",
         "%killbgscripts"
       ]
@@ -1079,9 +1056,9 @@
         "id": "bhxMXa0AaZkK"
       },
       "source": [
-        "## Multi worker training in depth\n",
+        "## Multi-worker training in depth\n",
         "\n",
-        "This tutorial has demonstrated a `Custom Training Loop` workflow of the multi-worker setup. A detailed description of other topics is available in the [`model.fit's guide`](https://colab.sandbox.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/distribute/multi_worker_with_keras.ipynb) of the multi-worker setup and applicable to CTLs."
+        "This tutorial has demonstrated a custom training loop workflow of the multi-worker setup. Detailed descriptions of other topics is available in the [Multi-worker training with Keras (`tf.keras.Model.fit`)](multi_worker_with_keras.ipynb) tutorial applicable to custom training loops."
       ]
     },
     {
@@ -1090,10 +1067,11 @@
         "id": "ega2hdOQEmy_"
       },
       "source": [
-        "## See also\n",
-        "1. [Distributed Training in TensorFlow](https://www.tensorflow.org/guide/distributed_training) guide provides an overview of the available distribution strategies.\n",
+        "## Learn more\n",
+        "\n",
+        "1. The [Distributed training in TensorFlow](../../guide/distributed_training.ipynb) guide provides an overview of the available distribution strategies.\n",
         "2. [Official models](https://github.com/tensorflow/models/tree/master/official), many of which can be configured to run multiple distribution strategies.\n",
-        "3. The [Performance section](../../guide/function.ipynb) in the guide provides information about other strategies and [tools](../../guide/profiler.md) you can use to optimize the performance of your TensorFlow models.\n"
+        "3. The [Performance section](../../guide/function.ipynb) in the `tf.function` guide provides information about other strategies and [tools](../../guide/profiler.md) you can use to optimize the performance of your TensorFlow models.\n"
       ]
     }
   ],

From 6d60a98ea14f30f482c7bfc5ce0e878b99813627 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Mon, 2 May 2022 16:23:18 -0700
Subject: [PATCH 109/872] Fix broken links

PiperOrigin-RevId: 446057312
---
 site/en/guide/dtensor_overview.ipynb          | 23 +++++--------------
 .../distribute/dtensor_ml_tutorial.ipynb      |  2 +-
 2 files changed, 7 insertions(+), 18 deletions(-)

diff --git a/site/en/guide/dtensor_overview.ipynb b/site/en/guide/dtensor_overview.ipynb
index 3c88bb70d2d..4dab68a3dd8 100644
--- a/site/en/guide/dtensor_overview.ipynb
+++ b/site/en/guide/dtensor_overview.ipynb
@@ -13,6 +13,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
+        "cellView": "form",
         "id": "tuOe1ymfHZPu"
       },
       "outputs": [],
@@ -67,7 +68,6 @@
         "id": "MGZuakHVlVQf"
       },
       "source": [
-        "\n",
         "## Overview\n",
         "\n",
         "This colab introduces DTensor, an extension to TensorFlow for synchronous distributed computing.\n",
@@ -76,7 +76,7 @@
         "\n",
         "By decoupling the application from sharding directives, DTensor enables running the same application on a single device, multiple devices, or even multiple clients, while preserving its global semantics. \n",
         "\n",
-        "This guide introduces DTensor concepts for distributed computing, and how DTensor integrates with TensorFlow. To see a demo of using DTensor in model training, see [Distributed training with DTensor](https://www.tensorflow.org/tutorials/distribute/dtensor_ml_tutorial.ipynb) tutorial."
+        "This guide introduces DTensor concepts for distributed computing, and how DTensor integrates with TensorFlow. To see a demo of using DTensor in model training, see [Distributed training with DTensor](https://www.tensorflow.org/tutorials/distribute/dtensor_ml_tutorial) tutorial."
       ]
     },
     {
@@ -157,7 +157,6 @@
         "id": "JjiHaH0ql9yo"
       },
       "source": [
-        "\n",
         "### Mesh\n",
         "\n",
         "`Mesh` represents a logical Cartisian topology of a set of devices. Each dimension of the Cartisian grid is called a **Mesh dimension**, and referred to with a name. Names of mesh dimension within the same `Mesh` must be unique.\n",
@@ -173,7 +172,6 @@
         "id": "_J6cOieEbaUw"
       },
       "source": [
-        "\n",
         "In a 1 dimensional `Mesh`, all devices form a list in a single mesh dimension. The following example uses `dtensor.create_mesh` to create a mesh from 6 CPU devices along a mesh dimension `'x'` with a size of 6 devices:\n",
         "\n",
         "\"A\n"
@@ -250,7 +248,6 @@
         "id": "fqzCNlWAbm-c"
       },
       "source": [
-        "\n",
         "On a 1-dimensional mesh such as `[(\"x\", 6)]` (`mesh_1d` in the previous section), `Layout([\"unsharded\"], mesh_1d)` is a layout for a rank-1 tensor replicated on 6 devices.\n",
         "\n",
         "\"Layout"
@@ -308,8 +305,7 @@
         "During `Mesh` creation, each client provides its *local device list* together with the expected *global device list*. DTensor validates that both lists are consistent. Please refer to the API documentation for `dtensor.create_mesh` and `dtensor.create_distributed_mesh`\n",
         " for more information on multi-client mesh creation and the *global device list*.\n",
         "\n",
-        "Single-client can be thought of as a special case of multi-client, with 1 client. In a single-client application, the *global device list* is identical to the *local device list*.\n",
-        "\n"
+        "Single-client can be thought of as a special case of multi-client, with 1 client. In a single-client application, the *global device list* is identical to the *local device list*.\n"
       ]
     },
     {
@@ -454,8 +450,7 @@
       "source": [
         "The inverse operation of `dtensor.unpack` is `dtensor.pack`. Component tensors can be packed back into a DTensor.\n",
         "\n",
-        "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices. \n",
-        "\n"
+        "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices. \n"
       ]
     },
     {
@@ -601,7 +596,6 @@
         "id": "T7FtZ9kQRZgE"
       },
       "source": [
-        "\n",
         "You can inspect the component tensors of the created DTensor and verify they are indeed sharded according to your scheme. It may be helpful to illustrate the situation with a chart:\n",
         "\n",
         "  \"A
Date: Mon, 2 May 2022 16:33:30 -0700
Subject: [PATCH 110/872] Update tutorial for Multi-worker training with Keras.

PiperOrigin-RevId: 446059541
---
 .../distribute/multi_worker_with_keras.ipynb  | 178 +++++++++---------
 1 file changed, 94 insertions(+), 84 deletions(-)

diff --git a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
index 0699e1e451d..4e8cc21f70f 100644
--- a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
@@ -63,13 +63,36 @@
       "source": [
         "## Overview\n",
         "\n",
-        "This tutorial demonstrates how to perform multi-worker distributed training with a Keras model and the `Model.fit` API using the `tf.distribute.Strategy` API—specifically the `tf.distribute.MultiWorkerMirroredStrategy` class. With the help of this strategy, a Keras model that was designed to run on a single-worker can seamlessly work on multiple workers with minimal code changes.\n",
-        "\n",
-        "For those interested in a deeper understanding of `tf.distribute.Strategy` APIs, the [Distributed training in TensorFlow](../../guide/distributed_training.ipynb) guide is available for an overview of the distribution strategies TensorFlow supports.\n",
+        "This tutorial demonstrates how to perform multi-worker distributed training with a Keras model and the `Model.fit` API using the `tf.distribute.MultiWorkerMirroredStrategy` API. With the help of this strategy, a Keras model that was designed to run on a single-worker can seamlessly work on multiple workers with minimal code changes.\n",
         "\n",
         "To learn how to use the `MultiWorkerMirroredStrategy` with Keras and a custom training loop, refer to [Custom training loop with Keras and MultiWorkerMirroredStrategy](multi_worker_with_ctl.ipynb).\n",
         "\n",
-        "Note that the purpose of this tutorial is to demonstrate a minimal multi-worker example with two workers."
+        "This tutorial contains a minimal multi-worker example with two workers for demonstration purposes."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JUdRerXg6yz3"
+      },
+      "source": [
+        "### Choose the right strategy"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "YAiCV_oL63GM"
+      },
+      "source": [
+        "Before you dive in, make sure that `tf.distribute.MultiWorkerMirroredStrategy` is the right choice for your accelerator(s) and training. These are two common ways of distributing training with data parallelism:\n",
+        "\n",
+        "* _Synchronous training_, where the steps of training are synced across the workers and replicas, such as `tf.distribute.MirroredStrategy`, `tf.distribute.TPUStrategy`, and `tf.distribute.MultiWorkerMirroredStrategy`. All workers train over different slices of input data in sync, and aggregating gradients at each step.\n",
+        "* _Asynchronous training_, where the training steps are not strictly synced, such as `tf.distribute.experimental.ParameterServerStrategy`. All workers are independently training over the input data and updating variables asynchronously.\n",
+        "\n",
+        "If you are looking for multi-worker synchronous training without TPU, then `tf.distribute.MultiWorkerMirroredStrategy` is your choice. It creates copies of all variables in the model's layers on each device across all workers. It uses `CollectiveOps`, a TensorFlow op for collective communication, to aggregate gradients and keeps the variables in sync. For those interested, check out the `tf.distribute.experimental.CommunicationOptions` parameter for the collective implementation options we are providing.\n",
+        "\n",
+        "For an overview of `tf.distribute.Strategy` APIs, refer to [Distributed training in TensorFlow](../../guide/distributed_training.ipynb)."
       ]
     },
     {
@@ -104,14 +127,14 @@
       "source": [
         "Before importing TensorFlow, make a few changes to the environment:\n",
         "\n",
-        "1. Disable all GPUs. This prevents errors caused by the workers all trying to use the same GPU. In a real-world application, each worker would be on a different machine."
+        "* In a real-world application, each worker would be on a different machine. For the purposes of this tutorial, all the workers will run on the **this** machine. So disable all GPUs to prevents errors caused by all workers trying to use the same GPU."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "685pbYEY3jGC"
+        "id": "rpEIVI5upIzM"
       },
       "outputs": [],
       "source": [
@@ -124,7 +147,7 @@
         "id": "7X1MS6385BWi"
       },
       "source": [
-        "2. Reset the `TF_CONFIG` environment variable (you'll learn more about this later):"
+        "* Reset the `TF_CONFIG` environment variable (you'll learn more about this later):"
       ]
     },
     {
@@ -144,7 +167,7 @@
         "id": "Rd4L9Ii77SS8"
       },
       "source": [
-        "3. Make sure that the current directory is on Python's path—this allows the notebook to import the files written by `%%writefile` later:\n"
+        "* Make sure that the current directory is on Python's path—this allows the notebook to import the files written by `%%writefile` later:\n"
       ]
     },
     {
@@ -165,7 +188,7 @@
         "id": "pDhHuMjb7bfU"
       },
       "source": [
-        "Now import TensorFlow:"
+        "Finally, import TensorFlow:"
       ]
     },
     {
@@ -276,7 +299,7 @@
         "\n",
         "### A cluster with jobs and tasks\n",
         "\n",
-        "In TensorFlow, distributed training involves: a `'cluster'`\n",
+        "In TensorFlow, distributed training involves a `'cluster'`\n",
         "with several jobs, and each of the jobs may have one or more `'task'`s.\n",
         "\n",
         "You will need the `TF_CONFIG` configuration environment variable for training on multiple machines, each of which possibly has a different role. `TF_CONFIG` is a JSON string used to specify the cluster configuration for each worker that is part of the cluster.\n",
@@ -284,10 +307,10 @@
         "There are two components of a `TF_CONFIG` variable: `'cluster'` and `'task'`.\n",
         "\n",
         "* A `'cluster'` is the same for all workers and provides information about the training cluster, which is a dict consisting of different types of jobs, such as `'worker'` or `'chief'`.\n",
-        "    - In multi-worker training with `tf.distribute.MultiWorkerMirroredStrategy`, there is usually one `'worker'` that takes on responsibilities, such as saving a checkpoint and writing a summary file for TensorBoard, in addition to what a regular `'worker'` does. Such `'worker'` is referred to as the chief worker (with a job name `'chief'`).\n",
-        "    - It is customary for the `'chief'` to have `'index'` `0` be appointed to (in fact, this is how `tf.distribute.Strategy` is implemented).\n",
+        "    - In multi-worker training with `tf.distribute.MultiWorkerMirroredStrategy`, there is usually one `'worker'` that takes on more responsibilities, such as saving a checkpoint and writing a summary file for TensorBoard, in addition to what a regular `'worker'` does. Such `'worker'` is referred to as the chief worker (with a job name `'chief'`).\n",
+        "    - It is customary for the worker with `'index'` `0` to be the `'chief'`.\n",
         "\n",
-        "* A `'task'` provides information of the current task and is different for each worker. It specifies the `'type'` and `'index'` of that worker.\n",
+        "* A `'task'` provides information on the current task and is different for each worker. It specifies the `'type'` and `'index'` of that worker.\n",
         "\n",
         "Below is an example configuration:"
       ]
@@ -314,7 +337,7 @@
         "id": "JjgwJbPKZkJL"
       },
       "source": [
-        "Here is the same `TF_CONFIG` serialized as a JSON string:"
+        "Note that `tf_config` is just a local variable in Python. To use it for training configuration, serialize it as a JSON and place it in a `TF_CONFIG` environment variable."
       ]
     },
     {
@@ -328,22 +351,13 @@
         "json.dumps(tf_config)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "f83FVYqDX3aX"
-      },
-      "source": [
-        "Note that`tf_config` is just a local variable in Python. To be able to use it for a training configuration, this dict needs to be serialized as a JSON and placed in a `TF_CONFIG` environment variable."
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "8YFpxrcsZ2xG"
       },
       "source": [
-        "In the example configuration above, you set the task `'type'` to `'worker'` and the task `'index'` to `0`. Therefore, this machine is the _first_ worker. It will be appointed as the `'chief'` worker and do more work than the others.\n",
+        "In the example configuration above, you set the task `'type'` to `'worker'` and the task `'index'` to `0`. Therefore, this machine is the _first_ worker. It will be appointed as the `'chief'` worker.\n",
         "\n",
         "Note: Other machines will need to have the `TF_CONFIG` environment variable set as well, and it should have the same `'cluster'` dict, but different task `'type'`s or task `'index'`es, depending on the roles of those machines."
       ]
@@ -354,12 +368,8 @@
         "id": "aogb74kHxynz"
       },
       "source": [
-        "For illustration purposes, this tutorial shows how you may set up a `TF_CONFIG` variable with two workers on a `localhost`.\n",
-        "\n",
-        "In practice, you would create multiple workers on external IP addresses/ports and set a `TF_CONFIG` variable on each worker accordingly.\n",
-        "\n",
-        "In this tutorial, you will use two workers:\n",
-        "- The first (`'chief'`) worker's `TF_CONFIG` is shown above.\n",
+        "In practice, you would create multiple workers on external IP addresses/ports and set a `TF_CONFIG` variable on each worker accordingly. For illustration purposes, this tutorial shows how you may set up a `TF_CONFIG` variable with two workers on a `localhost`:\n",
+        "- The first (`'chief'`) worker's `TF_CONFIG` as shown above.\n",
         "- For the second worker, you will set `tf_config['task']['index']=1`"
       ]
     },
@@ -378,9 +388,7 @@
         "id": "FcjAbuGY1ACJ"
       },
       "source": [
-        "Subprocesses inherit environment variables from their parent.\n",
-        "\n",
-        "For example, you can set an environment variable in this Jupyter Notebook process as follows:"
+        "Subprocesses inherit environment variables from their parent. So if you set an environment variable in this Jupyter Notebook process:"
       ]
     },
     {
@@ -400,7 +408,7 @@
         "id": "gQkIX-cg18md"
       },
       "source": [
-        "Then, you can access the environment variable from a subprocesses:"
+        "... then you can access the environment variable from the subprocesses:"
       ]
     },
     {
@@ -421,7 +429,16 @@
         "id": "af6BCA-Y2fpz"
       },
       "source": [
-        "In the next section, you'll use a similar method to pass the `TF_CONFIG` to the worker subprocesses. In a real-world scenario, you wouldn't launch your jobs this way, but it's sufficient in this example."
+        "In the next section, you'll use this method to pass the `TF_CONFIG` to the worker subprocesses. You would never really launch your jobs this way in a real-world scenario—this tutorial is just showing how to do it with a minimal multi-worker example."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dnDJmaRA9qnf"
+      },
+      "source": [
+        "## Train the model"
       ]
     },
     {
@@ -430,16 +447,7 @@
         "id": "UhNtHfuxCGVy"
       },
       "source": [
-        "## Choose the right strategy\n",
-        "\n",
-        "In TensorFlow, there are two main forms of distributed training:\n",
-        "\n",
-        "* _Synchronous training_, where the steps of training are synced across the workers and replicas, and\n",
-        "* _Asynchronous training_, where the training steps are not strictly synced (for example, [parameter server training](parameter_server_training.ipynb)).\n",
-        "\n",
-        "This tutorial demonstrates how to perform synchronous multi-worker training using an instance of `tf.distribute.MultiWorkerMirroredStrategy`.\n",
-        "\n",
-        "`MultiWorkerMirroredStrategy` creates copies of all variables in the model's layers on each device across all workers. It uses `CollectiveOps`, a TensorFlow op for collective communication, to aggregate gradients and keep the variables in sync.  The `tf.distribute.Strategy` [guide](../../guide/distributed_training.ipynb) has more details about this strategy."
+        "To train the model, firstly create an instance of the `tf.distribute.MultiWorkerMirroredStrategy`:"
       ]
     },
     {
@@ -462,23 +470,12 @@
         "Note: `TF_CONFIG` is parsed and TensorFlow's GRPC servers are started at the time `MultiWorkerMirroredStrategy` is called, so the `TF_CONFIG` environment variable must be set before a `tf.distribute.Strategy` instance is created. Since `TF_CONFIG` is not set yet, the above strategy is effectively single-worker training."
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "FMy2VM4Akzpr"
-      },
-      "source": [
-        "`MultiWorkerMirroredStrategy` provides multiple implementations via the `tf.distribute.experimental.CommunicationOptions` parameter: 1) `RING` implements ring-based collectives using gRPC as the cross-host communication layer; 2) `NCCL` uses the [NVIDIA Collective Communication Library](https://developer.nvidia.com/nccl) to implement collectives; and 3) `AUTO` defers the choice to the runtime. The best choice of collective implementation depends upon the number and kind of GPUs, and the network interconnect in the cluster."
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "H47DDcOgfzm7"
       },
       "source": [
-        "## Train the model\n",
-        "\n",
         "With the integration of `tf.distribute.Strategy` API into `tf.keras`, the only change you will make to distribute the training to multiple-workers is enclosing the model building and `model.compile()` call inside `strategy.scope()`. The distribution strategy's scope dictates how and where the variables are created, and in the case of `MultiWorkerMirroredStrategy`, the variables created are `MirroredVariable`s, and they are replicated on each of the workers.\n"
       ]
     },
@@ -584,7 +581,7 @@
         "id": "qmEEStPS6vR_"
       },
       "source": [
-        "So json-serialize the `TF_CONFIG` and add it to the environment variables:"
+        "Serialize the `TF_CONFIG` to JSON and add it to the environment variables:"
       ]
     },
     {
@@ -686,7 +683,7 @@
         "id": "RqZhVF7L_KOy"
       },
       "source": [
-        "The last line of the log file should say: `Started server with target: grpc://localhost:12345`. The first worker is now ready, and is waiting for all the other worker(s) to be ready to proceed."
+        "The last line of the log file should say: `Started server with target: grpc://localhost:12345`. The first worker is now ready and is waiting for all the other worker(s) to be ready to proceed."
       ]
     },
     {
@@ -758,11 +755,7 @@
         "id": "zL79ak5PMzEg"
       },
       "source": [
-        "Unsurprisingly, this ran _slower_ than the test run at the beginning of this tutorial.\n",
-        "\n",
-        "Running multiple workers on a single machine only adds overhead.\n",
-        "\n",
-        "The goal here was not to improve the training time, but only to give an example of multi-worker training."
+        "Note: This may run slower than the test run at the beginning of this tutorial because running multiple workers on a single machine only adds overhead. The goal here is not to improve the training time but to give an example of multi-worker training.\n"
       ]
     },
     {
@@ -784,11 +777,16 @@
         "id": "9j2FJVHoUIrE"
       },
       "source": [
-        "## Multi-worker training in depth\n",
-        "\n",
-        "So far, you have learned how to perform a basic multi-worker setup.\n",
-        "\n",
-        "During the rest of the tutorial, you will learn about other factors, which may be useful or important for real use cases, in detail."
+        "## Multi-worker training in depth\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "C1hBks_dAZmT"
+      },
+      "source": [
+        "So far, you have learned how to perform a basic multi-worker setup. The rest of the tutorial goes over other factors, which may be useful or important for real use cases, in detail."
       ]
     },
     {
@@ -824,48 +822,60 @@
         "dataset_no_auto_shard = multi_worker_dataset.with_options(options)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "z85hElxsBQsT"
+      },
+      "source": [
+        "### Evaluation"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "gmqvlh5LhAoU"
       },
       "source": [
-        "### Evaluation\n",
-        "\n",
-        "If you pass the `validation_data` into `Model.fit`, it will alternate between training and evaluation for each epoch. The evaluation taking the `validation_data` is distributed across the same set of workers and the evaluation results are aggregated and available for all workers.\n",
+        "If you pass the `validation_data` into `Model.fit` as well, it will alternate between training and evaluation for each epoch. The evaluation work is distributed across the same set of workers, and its results are aggregated and available to all workers.\n",
         "\n",
         "Similar to training, the validation dataset is automatically sharded at the file level. You need to set a global batch size in the validation dataset and set the `validation_steps`.\n",
         "\n",
-        "A repeated dataset is also recommended for evaluation.\n",
+        "A repeated dataset (by calling `tf.data.Dataset.repeat`) is recommended for evaluation.\n",
         "\n",
         "Alternatively, you can also create another task that periodically reads checkpoints and runs the evaluation. This is what Estimator does. But this is not a recommended way to perform evaluation and thus its details are omitted."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FNkoxUPJBNTb"
+      },
+      "source": [
+        "### Performance"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "XVk4ftYx6JAO"
       },
       "source": [
-        "### Performance\n",
-        "\n",
-        "You now have a Keras model that is all set up to run in multiple workers with the `MultiWorkerMirroredStrategy`.\n",
-        "\n",
-        "To tweak performance of multi-worker training, you can try the following:\n",
+        "To tweak the performance of multi-worker training, you can try the following:\n",
         "\n",
         "- `tf.distribute.MultiWorkerMirroredStrategy` provides multiple [collective communication implementations](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CommunicationImplementation):\n",
         "    - `RING` implements ring-based collectives using gRPC as the cross-host communication layer.\n",
         "    - `NCCL` uses the [NVIDIA Collective Communication Library](https://developer.nvidia.com/nccl) to implement collectives.\n",
         "    -  `AUTO` defers the choice to the runtime.\n",
         "    \n",
-        "    The best choice of collective implementation depends upon the number of GPUs, the type of GPUs, and the network interconnect in the cluster. To override the automatic choice, specify the `communication_options` parameter of `MultiWorkerMirroredStrategy`'s constructor. For example:\n",
+        "    The best choice of collective implementation depends upon the number of GPUs, the type of GPUs, and the network interconnects in the cluster. To override the automatic choice, specify the `communication_options` parameter of `MultiWorkerMirroredStrategy`'s constructor. For example:\n",
         "    \n",
         "    ```python\n",
-        "    communication_options=tf.distribute.experimental.CommunicationOptions(implementation=tf.distribute.experimental.CollectiveCommunication.NCCL)\n",
+        "    communication_options=tf.distribute.experimental.CommunicationOptions(implementation=tf.distribute.experimental.CommunicationImplementation.NCCL)\n",
         "    ```\n",
         "\n",
         "- Cast the variables to `tf.float` if possible:\n",
-        "    - The official ResNet model includes [an example](https://github.com/tensorflow/models/blob/8367cf6dabe11adf7628541706b660821f397dce/official/resnet/resnet_model.py#L466) of how this can be done."
+        "    - The official ResNet model includes [an example](https://github.com/tensorflow/models/blob/8367cf6dabe11adf7628541706b660821f397dce/official/resnet/resnet_model.py#L466) of how to do this."
       ]
     },
     {
@@ -926,7 +936,7 @@
         "- `task_id` tells you the identifier of the worker.\n",
         "- The worker with `task_id == 0` is designated as the chief worker.\n",
         "\n",
-        "In the code snippet below, the `write_filepath` function provides the file path to write, which depends on the the worker's `task_id`:\n",
+        "In the code snippet below, the `write_filepath` function provides the file path to write, which depends on the worker's `task_id`:\n",
         "\n",
         "- For the chief worker (with `task_id == 0`), it writes to the original file path. \n",
         "- For other workers, it creates a temporary directory—`temp_dir`—with the `task_id` in the directory path to write in:"
@@ -1125,7 +1135,7 @@
         "\n",
         "To use it, provide an instance of `tf.keras.callbacks.BackupAndRestore` at the `Model.fit` call.\n",
         "\n",
-        "With `MultiWorkerMirroredStrategy`, if a worker gets interrupted, the whole cluster pauses until the interrupted worker is restarted. Other workers will also restart, and the interrupted worker rejoins the cluster. Then, every worker reads the checkpoint file that was previously saved and picks up its former state, thereby allowing the cluster to get back in sync. Then, the training continues.\n",
+        "With `MultiWorkerMirroredStrategy`, if a worker gets interrupted, the whole cluster will pause until the interrupted worker is restarted. Other workers will also restart, and the interrupted worker will rejoin the cluster. Then, every worker will read the checkpoint file that was previously saved and pick up its former state, thereby allowing the cluster to get back in sync. Then, the training will continue.\n",
         "\n",
         "The `BackupAndRestore` callback uses the `CheckpointManager` to save and restore the training state, which generates a file called checkpoint that tracks existing checkpoints together with the latest one. For this reason, `backup_dir` should not be re-used to store other checkpoints in order to avoid name collision.\n",
         "\n",
@@ -1162,7 +1172,7 @@
       "source": [
         "If you inspect the directory of `backup_dir` you specified in `BackupAndRestore`, you may notice some temporarily generated checkpoint files. Those files are needed for recovering the previously lost instances, and they will be removed by the library at the end of `Model.fit` upon successful exiting of your training.\n",
         "\n",
-        "Note: Currently the `BackupAndRestore` callback only supports eager mode. In graph mode, consider using [Save/Restore Model](#model_saving_and_loading) mentioned above, and by providing `initial_epoch` in `Model.fit`."
+        "Note: Currently the `BackupAndRestore` callback only supports eager mode. In graph mode, consider using `Model.save`/`tf.saved_model.save` and `tf.keras.models.load_model` for saving and restoring models, respectively, as described in the _Model saving and loading_ section above, and by providing `initial_epoch` in `Model.fit` during training."
       ]
     },
     {
@@ -1173,7 +1183,7 @@
       "source": [
         "## Additional resources\n",
         "\n",
-        "1. The [Distributed training in TensorFlow](https://www.tensorflow.org/guide/distributed_training) guide provides an overview of the available distribution strategies.\n",
+        "1. The [Distributed training in TensorFlow](../../guide/distributed_training.ipynb) guide provides an overview of the available distribution strategies.\n",
         "1. The [Custom training loop with Keras and MultiWorkerMirroredStrategy](multi_worker_with_ctl.ipynb) tutorial shows how to use the `MultiWorkerMirroredStrategy` with Keras and a custom training loop.\n",
         "1. Check out the [official models](https://github.com/tensorflow/models/tree/master/official), many of which can be configured to run multiple distribution strategies.\n",
         "1. The [Better performance with tf.function](../../guide/function.ipynb) guide provides information about other strategies and tools, such as the [TensorFlow Profiler](../../guide/profiler.md) you can use to optimize the performance of your TensorFlow models."

From 80704900e59b183c6d3792e60bd09456fddea133 Mon Sep 17 00:00:00 2001
From: Srujun Thanmay Gupta 
Date: Tue, 3 May 2022 10:56:26 -0700
Subject: [PATCH 111/872] Fix links in DTensor Overview guide and ML Tutorial.

The `.ipynb` suffix triggers a download of the notebook instead of opening it on the Tensorflow.org site. Also fixed a broken link in the ML Tutorial and remove extraneous whitespace.

PiperOrigin-RevId: 446235732
---
 site/en/guide/dtensor_overview.ipynb          | 67 +++++++--------
 .../distribute/dtensor_ml_tutorial.ipynb      | 82 +++++++++----------
 2 files changed, 75 insertions(+), 74 deletions(-)

diff --git a/site/en/guide/dtensor_overview.ipynb b/site/en/guide/dtensor_overview.ipynb
index 4dab68a3dd8..880f2bcf422 100644
--- a/site/en/guide/dtensor_overview.ipynb
+++ b/site/en/guide/dtensor_overview.ipynb
@@ -74,7 +74,7 @@
         "\n",
         "DTensor provides a global programming model that allows developers to compose applications that operate on Tensors globally while managing the distribution across devices internally. DTensor distributes the program and tensors according to the sharding directives through a procedure called *[Single program, multiple data (SPMD)](https://en.wikipedia.org/wiki/SPMD) expansion*.\n",
         "\n",
-        "By decoupling the application from sharding directives, DTensor enables running the same application on a single device, multiple devices, or even multiple clients, while preserving its global semantics. \n",
+        "By decoupling the application from sharding directives, DTensor enables running the same application on a single device, multiple devices, or even multiple clients, while preserving its global semantics.\n",
         "\n",
         "This guide introduces DTensor concepts for distributed computing, and how DTensor integrates with TensorFlow. To see a demo of using DTensor in model training, see [Distributed training with DTensor](https://www.tensorflow.org/tutorials/distribute/dtensor_ml_tutorial) tutorial."
       ]
@@ -130,7 +130,7 @@
         "  tf.config.set_logical_device_configuration(phy_devices[0], [\n",
         "        tf.config.LogicalDeviceConfiguration(),\n",
         "    ] * ncpu)\n",
-        "  \n",
+        "\n",
         "configure_virtual_cpus(6)\n",
         "DEVICES = [f'CPU:{i}' for i in range(6)]\n",
         "\n",
@@ -147,8 +147,8 @@
         "\n",
         "DTensor introduces two concepts: `dtensor.Mesh` and `dtensor.Layout`. They are abstractions to model the sharding of tensors across topologically related devices.\n",
         "\n",
-        "- `Mesh` defines the device list for computation. \n",
-        "- `Layout` defines how to shard the Tensor dimension on a `Mesh`. "
+        "- `Mesh` defines the device list for computation.\n",
+        "- `Layout` defines how to shard the Tensor dimension on a `Mesh`."
       ]
     },
     {
@@ -223,7 +223,7 @@
         "\n",
         "**`Layout`** specifies how a tensor is distributed, or sharded, on a `Mesh`. In DTensor, the placement specification is on a per-axis bases. An axis of a `Layout` can be either `sharded` or `unsharded` (replicated) along a mesh dimension.\n",
         "\n",
-        "Note: In order to avoid confusions between `Mesh` and `Layout`, the term *dimension* is always associated with `Mesh`, and the term *axis* with `Tensor` and `Layout` in this guide. \n",
+        "Note: In order to avoid confusions between `Mesh` and `Layout`, the term *dimension* is always associated with `Mesh`, and the term *axis* with `Tensor` and `Layout` in this guide.\n",
         "\n",
         "The rank of a `Layout` and the number of dimensions of a `Mesh` do not need to match. The `unsharded` axes of a `Layout` do not need to be associated to a mesh dimension, and `unsharded` mesh dimensions do not need to be associated with a `layout` axis.\n",
         "\n",
@@ -298,9 +298,9 @@
         "\n",
         "DTensor supports both single-client and multi-client applications. The colab Python kernel is an example of a single client DTensor application, where there is a single Python process.\n",
         "\n",
-        "In a multi-client DTensor application, multiple Python processes collectively perform as a coherent application. The Cartisian grid of a `Mesh` in a multi-client DTensor application can span across devices regardless of whether they are attached locally to the current client or attached remotely to another client. The set of all devices used by a `Mesh` are called the *global device list*. \n",
+        "In a multi-client DTensor application, multiple Python processes collectively perform as a coherent application. The Cartisian grid of a `Mesh` in a multi-client DTensor application can span across devices regardless of whether they are attached locally to the current client or attached remotely to another client. The set of all devices used by a `Mesh` are called the *global device list*.\n",
         "\n",
-        "The creation of a `Mesh` in a multi-client DTensor application is a collective operation where the *global device list* is identicial for all of the participating clients, and the creation of the `Mesh` serves as a global barrier. \n",
+        "The creation of a `Mesh` in a multi-client DTensor application is a collective operation where the *global device list* is identicial for all of the participating clients, and the creation of the `Mesh` serves as a global barrier.\n",
         "\n",
         "During `Mesh` creation, each client provides its *local device list* together with the expected *global device list*. DTensor validates that both lists are consistent. Please refer to the API documentation for `dtensor.create_mesh` and `dtensor.create_distributed_mesh`\n",
         " for more information on multi-client mesh creation and the *global device list*.\n",
@@ -331,17 +331,17 @@
       "source": [
         "def dtensor_from_array(arr, layout, shape=None, dtype=None):\n",
         "  \"\"\"Convert a DTensor from something that looks like an array or Tensor.\n",
-        "  \n",
+        "\n",
         "  This function is convenient for quick doodling DTensors from a known,\n",
         "  unsharded data object in a single-client environment. This is not the\n",
-        "  most efficient way of creating a DTensor, but it will do for this \n",
+        "  most efficient way of creating a DTensor, but it will do for this\n",
         "  tutorial.\n",
         "  \"\"\"\n",
         "  if shape is not None or dtype is not None:\n",
         "    arr = tf.constant(arr, shape=shape, dtype=dtype)\n",
-        "  \n",
+        "\n",
         "  # replicate the input to the mesh\n",
-        "  a = dtensor.copy_to_mesh(arr, \n",
+        "  a = dtensor.copy_to_mesh(arr,\n",
         "          layout=dtensor.Layout.replicated(layout.mesh, rank=layout.rank))\n",
         "  # shard the copy to the desirable layout\n",
         "  return dtensor.relayout(a, layout=layout)"
@@ -356,11 +356,11 @@
         "### Anatomy of a DTensor\n",
         "\n",
         "A DTensor is a `tf.Tensor` object, but augumented with the `Layout` annotation that defines its sharding behavior. A DTensor consists of the following:\n",
-        "  \n",
+        "\n",
         "  - Global tensor meta-data, including the global shape and dtype of the tensor.\n",
         "  - A `Layout`, which defines the `Mesh` the `Tensor` belongs to, and how the `Tensor` is sharded onto the `Mesh`.\n",
-        "  - A list of **component tensors**, one item per local device in the `Mesh`. \n",
-        "  \n",
+        "  - A list of **component tensors**, one item per local device in the `Mesh`.\n",
+        "\n",
         "With `dtensor_from_array`, you can create your first DTensor, `my_first_dtensor`, and examine its contents."
       ]
     },
@@ -450,7 +450,8 @@
       "source": [
         "The inverse operation of `dtensor.unpack` is `dtensor.pack`. Component tensors can be packed back into a DTensor.\n",
         "\n",
-        "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices. \n"
+        "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices.\n",
+        "\n"
       ]
     },
     {
@@ -504,7 +505,7 @@
         "Create a 3x2 rank-2 DTensor, sharding its first axis along the `'x'` mesh dimension, and its second axis along the `'y'` mesh dimension.\n",
         "\n",
         "- Because the tensor shape equals to the mesh dimension along all of the sharded axes, each device receives a single element of the DTensor.\n",
-        "- The rank of the component tensor is always the same as the rank of the global shape. DTensor adopts this convention as a simple way to preserve information for locating the relation between a component tensor and the global DTensor. "
+        "- The rank of the component tensor is always the same as the rank of the global shape. DTensor adopts this convention as a simple way to preserve information for locating the relation between a component tensor and the global DTensor."
       ]
     },
     {
@@ -567,10 +568,10 @@
         "DTensor allows a `Layout` to be a hybrid, sharded along some axes, but replicated along others.\n",
         "\n",
         "For example, you can shard the same 3x2 rank-2 DTensor in the following way:\n",
-        "  \n",
+        "\n",
         "  - 1st axis sharded along the `'x'` mesh dimension.\n",
         "  - 2nd axis replicated along the `'y'` mesh dimension.\n",
-        "  \n",
+        "\n",
         "To achieve this sharding scheme, you just need to replace the sharding spec of the 2nd axis from `'y'` to `dtensor.UNSHARDED`, to indicate your intention of replicating along the 2nd axis. The layout object will look like `Layout(['x', dtensor.UNSHARDED], mesh)`."
       ]
     },
@@ -610,7 +611,7 @@
       "source": [
         "#### Tensor.numpy() and sharded DTensor\n",
         "\n",
-        "Be aware that calling the `.numpy()` method on a sharded DTensor raises an error. The rationale for erroring is to protect against unintended gathering of data from multiple computing devices to the host CPU device backing the returned numpy array. "
+        "Be aware that calling the `.numpy()` method on a sharded DTensor raises an error. The rationale for erroring is to protect against unintended gathering of data from multiple computing devices to the host CPU device backing the returned numpy array."
       ]
     },
     {
@@ -624,12 +625,12 @@
         "print(fully_replicated_dtensor.numpy())\n",
         "\n",
         "try:\n",
-        "  fully_sharded_dtensor.numpy() \n",
+        "  fully_sharded_dtensor.numpy()\n",
         "except tf.errors.UnimplementedError:\n",
         "  print(\"got an error as expected for fully_sharded_dtensor\")\n",
         "\n",
         "try:\n",
-        "  hybrid_sharded_dtensor.numpy() \n",
+        "  hybrid_sharded_dtensor.numpy()\n",
         "except tf.errors.UnimplementedError:\n",
         "  print(\"got an error as expected for hybrid_sharded_dtensor\")"
       ]
@@ -650,11 +651,11 @@
         "  - Rewriting TensorFlow Ops on the global DTensor with equivalent TensorFlow Ops on the componenent tensors, inserting collective and communication Ops when necessary\n",
         "  - Lowering backend neutral TensorFlow Ops to backend specific TensorFlow Ops.\n",
         "\n",
-        "The final result is that **DTensor is a drop-in replacement for Tensor**. \n",
+        "The final result is that **DTensor is a drop-in replacement for Tensor**.\n",
         "\n",
         "Note: DTensor is still an experimental API which means you will be exploring and pushing the boundaries and limits of the DTensor programming model.\n",
         "\n",
-        "There are 2 ways of triggering DTensor execution: \n",
+        "There are 2 ways of triggering DTensor execution:\n",
         "  - DTensor as operands of a Python function, e.g. `tf.matmul(a, b)` will run through DTensor if `a`, `b`, or both are DTensors.\n",
         "  - Requesting the result of a Python function to be a DTensor, e.g. `dtensor.call_with_layout(tf.ones, layout, shape=(3, 2))` will run through DTensor because we requested the output of tf.ones to be sharded according to a `layout`."
       ]
@@ -678,10 +679,10 @@
       "source": [
         "#### Fully replicated input and output\n",
         "\n",
-        "In this case, the DTensors are fully replicated. On each of the devices of the `Mesh`, \n",
+        "In this case, the DTensors are fully replicated. On each of the devices of the `Mesh`,\n",
         "  - the component tensor for operand `a` is `[[1, 2, 3], [4, 5, 6]]` (2x3)\n",
         "  - the component tensor for operand `b` is `[[6, 5], [4, 3], [2, 1]]` (3x2)\n",
-        "  - the computation consists of a single `MatMul` of `(2x3, 3x2) -> 2x2`, \n",
+        "  - the computation consists of a single `MatMul` of `(2x3, 3x2) -> 2x2`,\n",
         "  - the component tensor for result `c` is `[[20, 14], [56,41]]` (2x2)\n",
         "\n",
         "Total number of floating point mul operations is `6 device * 4 result * 3 mul = 72`."
@@ -792,7 +793,7 @@
         "\n",
         "What about Python functions that do not take operands, but returns a Tensor result that can be sharded? Examples of such functions are\n",
         "\n",
-        "  - `tf.ones`, `tf.zeros`, `tf.random.stateless_normal`, \n",
+        "  - `tf.ones`, `tf.zeros`, `tf.random.stateless_normal`,\n",
         "\n",
         "For these Python functions, DTensor provides `dtensor.call_with_layout` which eagelry executes a Python function with DTensor, and ensures that the returned Tensor is a DTensor with the requested `Layout`."
       ]
@@ -885,9 +886,9 @@
       "outputs": [],
       "source": [
         "ones = dtensor.call_with_layout(\n",
-        "    tf.function(tf.random.stateless_normal), \n",
-        "    dtensor.Layout(['x', 'y'], mesh), \n",
-        "    shape=(6, 4), \n",
+        "    tf.function(tf.random.stateless_normal),\n",
+        "    dtensor.Layout(['x', 'y'], mesh),\n",
+        "    shape=(6, 4),\n",
         "    seed=(1, 1))\n",
         "print(ones)"
       ]
@@ -910,8 +911,8 @@
       "outputs": [],
       "source": [
         "ones = dtensor.call_with_layout(\n",
-        "    tf.function(tf.ones), \n",
-        "    dtensor.Layout(['x', 'y'], mesh), \n",
+        "    tf.function(tf.ones),\n",
+        "    dtensor.Layout(['x', 'y'], mesh),\n",
         "    shape=(6, 4))\n",
         "print(ones)"
       ]
@@ -925,7 +926,7 @@
         "### From `tf.Variable` to `dtensor.DVariable`\n",
         "\n",
         "In Tensorflow, `tf.Variable` is the holder for a mutable `Tensor` value.\n",
-        "With DTensor, the corresponding variable semantics is provided by `dtensor.DVariable`. \n",
+        "With DTensor, the corresponding variable semantics is provided by `dtensor.DVariable`.\n",
         "\n",
         "The reason a new type `DVariable` was introduced for DTensor variable is because DVariables have an additional requirement that the layout cannot change from its initial value."
       ]
@@ -945,7 +946,7 @@
         "    initial_value=dtensor.call_with_layout(\n",
         "        tf.function(tf.random.stateless_normal),\n",
         "        layout=layout,\n",
-        "        shape=tf.TensorShape([64, 32]), \n",
+        "        shape=tf.TensorShape([64, 32]),\n",
         "        seed=[1, 1],\n",
         "        dtype=tf.float32))\n",
         "\n",
diff --git a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
index b69551bbfbb..0b6b3241e37 100644
--- a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
+++ b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
@@ -70,15 +70,15 @@
       "source": [
         "## Overview\n",
         "\n",
-        "DTensor provides a way for you to distribute the training of your model across devices to improve efficiency, reliability and scalability. For more details on DTensor concepts, see [The DTensor Programming Guide](https://www.tensorflow.org/guide/dtensor).\n",
+        "DTensor provides a way for you to distribute the training of your model across devices to improve efficiency, reliability and scalability. For more details on DTensor concepts, see [The DTensor Programming Guide](https://www.tensorflow.org/guide/dtensor_overview).\n",
         "\n",
         "In this tutorial, you will train a Sentiment Analysis model with DTensor. Three distributed training schemes are demonstrated with this example:\n",
-        " \n",
+        "\n",
         " - Data Parallel training, where the training samples are sharded (partitioned) to devices.\n",
-        " - Model Parallel training, where the model variables are sharded to devices. \n",
+        " - Model Parallel training, where the model variables are sharded to devices.\n",
         " - Spatial Parallel training, where the features of input data are  sharded to devices. (Also known as [Spatial Partitioning](https://cloud.google.com/blog/products/ai-machine-learning/train-ml-models-on-large-images-and-3d-volumes-with-spatial-partitioning-on-cloud-tpus))\n",
         "\n",
-        "The training portion of this tutorial is inspired [A Kaggle guide on Sentiment Analysis](https://www.kaggle.com/code/anasofiauzsoy/yelp-review-sentiment-analysis-tensorflow-tfds/notebook) notebook. To learn about the complete training and evaluation workflow (without DTensor), refer to that notebook. \n",
+        "The training portion of this tutorial is inspired [A Kaggle guide on Sentiment Analysis](https://www.kaggle.com/code/anasofiauzsoy/yelp-review-sentiment-analysis-tensorflow-tfds/notebook) notebook. To learn about the complete training and evaluation workflow (without DTensor), refer to that notebook.\n",
         "\n",
         "This tutorial will walk through the following steps:\n",
         "\n",
@@ -141,7 +141,7 @@
         "  tf.config.set_logical_device_configuration(phy_devices[0], [\n",
         "        tf.config.LogicalDeviceConfiguration(),\n",
         "    ] * ncpu)\n",
-        "  \n",
+        "\n",
         "configure_virtual_cpus(8)\n",
         "DEVICES = [f'CPU:{i}' for i in range(8)]\n",
         "\n",
@@ -204,7 +204,7 @@
         "First tokenize the text. Here use an extension of one-hot encoding, the `'tf_idf'` mode of `tf.keras.layers.TextVectorization`.\n",
         "\n",
         "- For the sake of speed, limit the number of tokens to 1200.\n",
-        "- To keep the `tf.Module` simple, run `TextVectorization` as a preprocessing step before the training. \n",
+        "- To keep the `tf.Module` simple, run `TextVectorization` as a preprocessing step before the training.\n",
         "\n",
         "The final result of the data cleaning section is a `Dataset` with the tokenized text as `x` and label as `y`.\n",
         "\n",
@@ -262,15 +262,15 @@
         "USE_MOCK_DATA = False  # Change to True to use mock data\n",
         "\n",
         "def processor(data):\n",
-        "  # Rewrite the data, such that \n",
-        "  #  - data channel 0 triggers label 0, and \n",
+        "  # Rewrite the data, such that\n",
+        "  #  - data channel 0 triggers label 0, and\n",
         "  #  - channel -1 triggers label 1\n",
-        "  x = tf.where(data['y'] != 0, \n",
+        "  x = tf.where(data['y'] != 0,\n",
         "                  tf.range(len(data['x']), dtype=tf.float32),\n",
         "                  len(data['x']) * 1.0 - tf.range(len(data['x']), dtype=tf.float32))\n",
         "  return {'x': x, 'y' : data['y']}\n",
         "\n",
-        "if USE_MOCK_DATA:   \n",
+        "if USE_MOCK_DATA:\n",
         "  dataset = dataset.map(processor)"
       ]
     },
@@ -284,7 +284,7 @@
         "\n",
         "Now build a Multi-Layer Perceptron (MLP) network with `DTensor`. The network will use fully connected Dense and BatchNorm layers.\n",
         "\n",
-        "`DTensor` expands TensorFlow through single-program multi-data (SPMD) expansion of regular TensorFlow Ops according to the `dtensor.Layout` attributes of their input `Tensor` and variables. \n",
+        "`DTensor` expands TensorFlow through single-program multi-data (SPMD) expansion of regular TensorFlow Ops according to the `dtensor.Layout` attributes of their input `Tensor` and variables.\n",
         "\n",
         "Variables of `DTensor` aware layers are `dtensor.DVariable`, and the constructors of `DTensor` aware layer objects take additional `Layout` inputs in addition to the usual layer parameters.\n",
         "\n",
@@ -335,14 +335,14 @@
         "    super().__init__()\n",
         "\n",
         "    random_normal_initializer = tf.function(tf.random.stateless_normal)\n",
-        "    \n",
+        "\n",
         "    self.weight = dtensor.DVariable(\n",
         "        dtensor.call_with_layout(\n",
         "            random_normal_initializer, weight_layout,\n",
         "            shape=[input_size, output_size],\n",
         "            seed=init_seed\n",
         "            ))\n",
-        "    \n",
+        "\n",
         "    # bias is sharded the same way as the last axis of weight.\n",
         "    bias_layout = weight_layout.delete([0])\n",
         "\n",
@@ -352,7 +352,7 @@
         "  def __call__(self, x):\n",
         "    y = tf.matmul(x, self.weight) + self.bias\n",
         "    y = tf.nn.relu(y)\n",
-        "    \n",
+        "\n",
         "    return y"
       ]
     },
@@ -385,7 +385,7 @@
         "    super().__init__()\n",
         "\n",
         "  def __call__(self, x, training=True):\n",
-        "    if not training: \n",
+        "    if not training:\n",
         "      # This branch is not used in the Tutorial.\n",
         "      pass\n",
         "    mean, variance = tf.nn.moments(x, axes=[0])\n",
@@ -398,7 +398,7 @@
         "id": "q4R4MPz5prh4"
       },
       "source": [
-        "A full featured batch normalization layer (such as `tf.keras.layers.BatchNormalization`) will need Layout arguments for its variables. "
+        "A full featured batch normalization layer (such as `tf.keras.layers.BatchNormalization`) will need Layout arguments for its variables."
       ]
     },
     {
@@ -451,12 +451,12 @@
         "\n",
         "  def __init__(self, dense_layouts: Tuple[dtensor.Layout, dtensor.Layout]):\n",
         "    super().__init__()\n",
-        "        \n",
+        "\n",
         "    self.dense1 = Dense(1200, 48, (1, 2), dense_layouts[0])\n",
         "    self.bn1 = BatchNorm()\n",
         "    self.dense2 = Dense(48, 2, (3, 4), dense_layouts[1])\n",
         "    self.bn2 = BatchNorm()\n",
-        "    \n",
+        "\n",
         "  def __call__(self, x):\n",
         "    y = x\n",
         "    y = self.dense1(y)\n",
@@ -488,11 +488,11 @@
         "\n",
         "  def __init__(self, mesh, input_mesh_dim, inner_mesh_dim1, output_mesh_dim):\n",
         "    super().__init__()\n",
-        "      \n",
+        "\n",
         "    self.dense1 = Dense(1200, 48, (1, 2), dtensor.Layout([input_mesh_dim, inner_mesh_dim1], mesh))\n",
         "    self.bn1 = BatchNorm()\n",
         "    self.dense2 = Dense(48, 2, (3, 4), dtensor.Layout([inner_mesh_dim1, output_mesh_dim], mesh))\n",
-        "    \n",
+        "\n",
         "\n",
         "  def __call__(self, x):\n",
         "    y = x\n",
@@ -557,22 +557,22 @@
       "source": [
         "def repack_local_tensor(x, layout):\n",
         "  \"\"\"Repacks a local Tensor-like to a DTensor with layout.\n",
-        "  \n",
+        "\n",
         "  This function assumes a single-client application.\n",
         "  \"\"\"\n",
         "  x = tf.convert_to_tensor(x)\n",
         "  sharded_dims = []\n",
-        "  \n",
+        "\n",
         "  # For every sharded dimension, use tf.split to split the along the dimension.\n",
         "  # The result is a nested list of split-tensors in queue[0].\n",
         "  queue = [x]\n",
         "  for axis, dim in enumerate(layout.sharding_specs):\n",
         "    if dim == dtensor.UNSHARDED:\n",
-        "      continue  \n",
+        "      continue\n",
         "    num_splits = layout.shape[axis]\n",
         "    queue = tf.nest.map_structure(lambda x: tf.split(x, num_splits, axis=axis), queue)\n",
         "    sharded_dims.append(dim)\n",
-        "  \n",
+        "\n",
         "  # Now we can build the list of component tensors by looking up the location in\n",
         "  # the nested list of split-tensors created in queue[0].\n",
         "  components = []\n",
@@ -581,8 +581,8 @@
         "    for dim in sharded_dims:\n",
         "      split_index = locations[dim]  # Only valid on single-client mesh.\n",
         "      t = t[split_index]\n",
-        "    components.append(t)  \n",
-        "  \n",
+        "    components.append(t)\n",
+        "\n",
         "  return dtensor.pack(components, layout)"
       ]
     },
@@ -602,7 +602,7 @@
         " - A global batch is split into N per-replica batches.\n",
         " - Each per-replica batch is trained on the replica device.\n",
         " - The gradient is reduced before weight up data is collectively performed on all replicas.\n",
-        " \n",
+        "\n",
         "Data parallel training provides nearly linear speedup regarding the number of devices."
       ]
     },
@@ -648,7 +648,7 @@
         "\n",
         "The training data batch should be packed into DTensors sharded along the `'batch'`(first) axis, such that DTensor will evenly distribute the training data to the `'batch'` mesh dimension.\n",
         "\n",
-        "**Note**: In DTensor, the `batch size` always refers to the global batch size. The batch size should be chosen such that it can be divided evenly by the size of the `batch` mesh dimension. "
+        "**Note**: In DTensor, the `batch size` always refers to the global batch size. The batch size should be chosen such that it can be divided evenly by the size of the `batch` mesh dimension."
       ]
     },
     {
@@ -713,7 +713,7 @@
         "  for parameter, parameter_gradient in zip(parameters, gradients):\n",
         "    parameter.assign_sub(learning_rate * parameter_gradient)\n",
         "\n",
-        "  # Define some metrics  \n",
+        "  # Define some metrics\n",
         "  accuracy = 1.0 - tf.reduce_sum(tf.cast(tf.argmax(logits, axis=-1, output_type=tf.int64) != y, tf.float32)) / x.shape[0]\n",
         "  loss_per_sample = loss / len(x)\n",
         "  return {'loss': loss_per_sample, 'accuracy': accuracy}"
@@ -727,7 +727,7 @@
       "source": [
         "### Checkpointing\n",
         "\n",
-        "You can checkpoint a DTensor model using `dtensor.DTensorCheckpoint`. The format of a DTensor checkpoint is fully compatible with a Standard TensorFlow Checkpoint. There is ongoing work to consolidate `dtensor.DTensorCheckpoint` into `tf.train.Checkpoint`. \n",
+        "You can checkpoint a DTensor model using `dtensor.DTensorCheckpoint`. The format of a DTensor checkpoint is fully compatible with a Standard TensorFlow Checkpoint. There is ongoing work to consolidate `dtensor.DTensorCheckpoint` into `tf.train.Checkpoint`.\n",
         "\n",
         "When a DTensor checkpoint is restored, `Layout`s of variables can be different from when the checkpoint is saved. This tutorial makes use of this feature to continue the training in the Model Parallel training and Spatial Parallel training sections.\n"
       ]
@@ -783,7 +783,7 @@
         "  for input in batched_dataset:\n",
         "\n",
         "    x, y = repack_batch(input['x'], input['y'], mesh)\n",
-        "    \n",
+        "\n",
         "    metrics.update(train_step(model, x, y, 1e-2))\n",
         "\n",
         "    pbar.update(step, values=metrics.items(), finalize=False)\n",
@@ -804,7 +804,7 @@
         "\n",
         "In Model Parallel training, each model replica spans multiple devices (2 in this case):\n",
         "\n",
-        "- There are 4 model replicas, and the training data batch is distributed to the 4 replicas. \n",
+        "- There are 4 model replicas, and the training data batch is distributed to the 4 replicas.\n",
         "- The 2 devices within a single model replica receive replicated training data.\n",
         "\n",
         "\n",
@@ -821,7 +821,7 @@
       "source": [
         "mesh = dtensor.create_mesh([(\"batch\", 4), (\"model\", 2)], devices=DEVICES)\n",
         "model = MLP([\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, \"model\"], mesh), \n",
+        "            dtensor.Layout([dtensor.UNSHARDED, \"model\"], mesh),\n",
         "            dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)]\n",
         "            )\n"
       ]
@@ -920,7 +920,7 @@
       "source": [
         "mesh = dtensor.create_mesh([(\"batch\", 2), (\"feature\", 2), (\"model\", 2)], devices=DEVICES)\n",
         "model = MLP([\n",
-        "            dtensor.Layout([\"feature\", \"model\"], mesh), \n",
+        "            dtensor.Layout([\"feature\", \"model\"], mesh),\n",
         "            dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)])\n"
       ]
     },
@@ -945,7 +945,7 @@
         "\n",
         "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n",
         "\n",
-        "def repack_batch_for_spt(x, y, mesh): \n",
+        "def repack_batch_for_spt(x, y, mesh):\n",
         "    # Shard data on feature dimension, too\n",
         "    x = repack_local_tensor(x, layout=dtensor.Layout([\"batch\", 'feature'], mesh))\n",
         "    y = repack_local_tensor(y, layout=dtensor.Layout([\"batch\"], mesh))\n",
@@ -978,12 +978,12 @@
         "  step = 0\n",
         "  metrics = {'epoch': epoch}\n",
         "  pbar = tf.keras.utils.Progbar(target=int(batched_dataset.cardinality()))\n",
-        "  \n",
+        "\n",
         "  for input in batched_dataset:\n",
         "    x, y = input['x'], input['y']\n",
         "    x, y = repack_batch_for_spt(x, y, mesh)\n",
         "    metrics.update(train_step(model, x, y, 1e-2))\n",
-        "    \n",
+        "\n",
         "    pbar.update(step, values=metrics.items(), finalize=False)\n",
         "    step += 1\n",
         "  manager.save()\n",
@@ -1000,7 +1000,7 @@
         "\n",
         "The integration of DTensor and SavedModel is still under development. This section only describes the current status quo for TensorFlow 2.9.0.\n",
         "\n",
-        "As of TensorFlow 2.9.0, `tf.saved_model` only accepts DTensor models with fully replicated variables. \n",
+        "As of TensorFlow 2.9.0, `tf.saved_model` only accepts DTensor models with fully replicated variables.\n",
         "\n",
         "As a workaround, you can convert a DTensor model to a fully replicated one by reloading a checkpoint. However, after a model is saved, all DTensor annotations are lost and the saved signatures can only be used with regular Tensors, not DTensors."
       ]
@@ -1015,7 +1015,7 @@
       "source": [
         "mesh = dtensor.create_mesh([(\"world\", 1)], devices=DEVICES[:1])\n",
         "model_for_saving = MLP([\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh), \n",
+        "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n",
         "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh)])\n",
         "\n",
         "manager = start_checkpoint_manager(mesh, model_for_saving)\n",
@@ -1054,11 +1054,11 @@
         "id": "Ks-Vs9qsH6jO"
       },
       "source": [
-        "## What's next? \n",
+        "## What's next?\n",
         "\n",
         "This tutorial demonstrated building and training an MLP sentiment analysis model with DTensor.\n",
         "\n",
-        "Through `Mesh` and `Layout` primitives, DTensor can transform a TensorFlow `tf.function` to a distributed program suitable for a variety of training schemes. \n",
+        "Through `Mesh` and `Layout` primitives, DTensor can transform a TensorFlow `tf.function` to a distributed program suitable for a variety of training schemes.\n",
         "\n",
         "In a real-world machine learning application, evaluation and cross-validation should be applied to avoid producing an over-fitted model. The techniques introduced in this tutorial can also be applied to introduce parallelism to evaluation.\n",
         "\n",

From 5ae1433562692778e2966a0f04ecc1a335d341bb Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Tue, 3 May 2022 17:44:04 -0700
Subject: [PATCH 112/872] Teeny cleanup

 * `py_object` was unused, so removed
 * `logger.debug(..., filter)` logs the reserved function, not the local variable
 * formatter fix

PiperOrigin-RevId: 446332127
---
 tools/tensorflow_docs/api_generator/generate_lib.py | 6 ++----
 tools/tensorflow_docs/api_generator/traverse.py     | 2 +-
 2 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py
index c00ab44a7cf..21bf617ceff 100644
--- a/tools/tensorflow_docs/api_generator/generate_lib.py
+++ b/tools/tensorflow_docs/api_generator/generate_lib.py
@@ -122,7 +122,6 @@ def write_docs(
   num_docs_output = 0
   for api_node in parser_config.api_tree.iter_nodes():
     full_name = api_node.full_name
-    py_object = api_node.py_object
 
     if api_node.output_type() is api_node.OutputType.FRAGMENT:
       continue
@@ -294,9 +293,8 @@ def __init__(
       search_hints: bool = True,
       site_path: str = 'api_docs/python',
       private_map: Optional[Dict[str, str]] = None,
-      visitor_cls: Type[
-          doc_generator_visitor.DocGeneratorVisitor] = doc_generator_visitor
-      .DocGeneratorVisitor,
+      visitor_cls: Type[doc_generator_visitor.DocGeneratorVisitor] = (
+          doc_generator_visitor.DocGeneratorVisitor),
       api_cache: bool = True,
       callbacks: Optional[List[public_api.ApiFilter]] = None,
       yaml_toc: Union[bool, Type[toc_lib.TocBuilder]] = True,
diff --git a/tools/tensorflow_docs/api_generator/traverse.py b/tools/tensorflow_docs/api_generator/traverse.py
index e472f9291cd..3d76422735d 100644
--- a/tools/tensorflow_docs/api_generator/traverse.py
+++ b/tools/tensorflow_docs/api_generator/traverse.py
@@ -85,7 +85,7 @@ def get_children(self, root, new_stack, path) -> public_api.Children:
       new_names = [n for n, c in children]
 
       if old_names != new_names:
-        _LOGGER.debug('  filter: %s', filter)
+        _LOGGER.debug('  filter: %s', fil)
         _LOGGER.debug('    children: %s', new_names)
 
     return children

From 5926cdb5a44a082c12d81bc0455b939274c996a3 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Tue, 3 May 2022 18:01:35 -0700
Subject: [PATCH 113/872] Add support for using the module dir as `base_dir`.

Currently, if we try to document `/path/module/__init__.py` using `base_dir=['/path/module']`, everything under `/path/module` is trimmed for being outside of the base dir.

This is in prep for adding more modules to the TF Lite Support library, where submodules come from disparate parts of the repo.

PiperOrigin-RevId: 446334715
---
 tools/tensorflow_docs/api_generator/public_api.py |  5 +++--
 .../api_generator/public_api_test.py              | 15 +++++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/public_api.py b/tools/tensorflow_docs/api_generator/public_api.py
index ee327ee5c4f..217793f63dd 100644
--- a/tools/tensorflow_docs/api_generator/public_api.py
+++ b/tools/tensorflow_docs/api_generator/public_api.py
@@ -331,8 +331,9 @@ def __call__(self, path: Sequence[str], parent: Any,
       # contents will get filtered when the submodules are checked.
       if len(mod_base_dirs) == 1:
         mod_base_dir = mod_base_dirs[0]
-        # Check that module is in one of the `self._base_dir`s
-        if not any(base in mod_base_dir.parents for base in self.base_dirs):
+        # Check that module is, or is in one of the `self._base_dir`s
+        if not (any(base in mod_base_dir.parents for base in self.base_dirs) or
+                mod_base_dir in self.base_dirs):
           continue
       yield name, child
 
diff --git a/tools/tensorflow_docs/api_generator/public_api_test.py b/tools/tensorflow_docs/api_generator/public_api_test.py
index 2d785ef41e3..4130099d0b8 100644
--- a/tools/tensorflow_docs/api_generator/public_api_test.py
+++ b/tools/tensorflow_docs/api_generator/public_api_test.py
@@ -210,6 +210,21 @@ def test_filter_base_dirs(self):
                   ('sub2', module.sub2)])
     self.assertEqual([('a', module.a), ('sub1', module.sub1)], list(result))
 
+  def test_filter_base_dir_pointing_to_submodule_dir(self):
+    module = types.ModuleType('module')
+    module.__file__ = '/1/2/3/module'
+    module.submodule = types.ModuleType('submodule')
+    module.submodule.__file__ = '/1/2/3/submodule/__init__.py'
+
+    test_filter = public_api.FilterBaseDirs(
+        base_dirs=[pathlib.Path('/1/2/3/submodule')])
+    result = test_filter(
+        path=('module',),
+        parent=module,
+        children=[('submodule', module.submodule)])
+
+    self.assertEqual([('submodule', module.submodule)], list(result))
+
 
 if __name__ == '__main__':
   absltest.main()

From 279f9df45099ec01026ba76f6b21029b53d9d272 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 4 May 2022 09:46:16 -0700
Subject: [PATCH 114/872] Add github autoassignment for the install directory.

+ remove the keras auto-assignment since the directory is now empty.

PiperOrigin-RevId: 446481545
---
 CODEOWNERS | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/CODEOWNERS b/CODEOWNERS
index 965b3a081ca..db670ef4093 100644
--- a/CODEOWNERS
+++ b/CODEOWNERS
@@ -4,8 +4,8 @@
 # Default owners for everything in repo.
 *  @MarkDaoust @8bitmp3
 
-# Docs
-/site/en/guide/keras/ @fchollet @MarkDaoust @8bitmp3
+# Install
+/site/en/install/ @mihaimaruseac @haifeng-jin @MarkDaoust @8bitmp3
 
 # Community
 /site/en/community/ @ewilderj @theadactyl @joanafilipa

From 0da869362f7a16a91e6eb6f955b5931d2917d4f4 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 4 May 2022 10:07:12 -0700
Subject: [PATCH 115/872] DTensor docs update.

* Simplify data-handling in the keras-tutorial.
* Add a couple more figures.

PiperOrigin-RevId: 446486657
---
 site/en/guide/dtensor_overview.ipynb          |  86 ++++--
 .../distribute/dtensor_ml_tutorial.ipynb      | 278 +++++++++---------
 2 files changed, 201 insertions(+), 163 deletions(-)

diff --git a/site/en/guide/dtensor_overview.ipynb b/site/en/guide/dtensor_overview.ipynb
index 880f2bcf422..39ca1025134 100644
--- a/site/en/guide/dtensor_overview.ipynb
+++ b/site/en/guide/dtensor_overview.ipynb
@@ -174,7 +174,7 @@
       "source": [
         "In a 1 dimensional `Mesh`, all devices form a list in a single mesh dimension. The following example uses `dtensor.create_mesh` to create a mesh from 6 CPU devices along a mesh dimension `'x'` with a size of 6 devices:\n",
         "\n",
-        "\"A\n"
+        "\"A\n"
       ]
     },
     {
@@ -197,7 +197,7 @@
       "source": [
         "A `Mesh` can be multi dimensional as well. In the following example, 6 CPU devices form a `3x2` mesh, where the `'x'` mesh dimension has a size of 3 devices, and the `'y'` mesh dimension has a size of 2 devices:\n",
         "\n",
-        "\"A"
       ]
     },
@@ -221,15 +221,16 @@
       "source": [
         "### Layout\n",
         "\n",
-        "**`Layout`** specifies how a tensor is distributed, or sharded, on a `Mesh`. In DTensor, the placement specification is on a per-axis bases. An axis of a `Layout` can be either `sharded` or `unsharded` (replicated) along a mesh dimension.\n",
+        "**`Layout`** specifies how a tensor is distributed, or sharded, on a `Mesh`.\n",
         "\n",
         "Note: In order to avoid confusions between `Mesh` and `Layout`, the term *dimension* is always associated with `Mesh`, and the term *axis* with `Tensor` and `Layout` in this guide.\n",
         "\n",
-        "The rank of a `Layout` and the number of dimensions of a `Mesh` do not need to match. The `unsharded` axes of a `Layout` do not need to be associated to a mesh dimension, and `unsharded` mesh dimensions do not need to be associated with a `layout` axis.\n",
+        "The rank of `Layout` should be the same as the rank of the `Tensor` where the `Layout` is applied. For each of the `Tensor`'s axes the `Layout` may specifiy a mesh dimension to shard the tensor across, or specify the axis as \"unsharded\".\n",
+        "The tensor is replicated across any mesh dimensions that it is not sharded across.\n",
         "\n",
-        "On the other hand, the rank of `Layout` should be the same as the rank of the `Tensor` where the `Layout` is applied.\n",
+        "The rank of a `Layout` and the number of dimensions of a `Mesh` do not need to match. The `unsharded` axes of a `Layout` do not need to be associated to a mesh dimension, and `unsharded` mesh dimensions do not need to be associated with a `layout` axis.\n",
         "\n",
-        "\"Diagram"
       ]
     },
@@ -248,9 +249,8 @@
         "id": "fqzCNlWAbm-c"
       },
       "source": [
-        "On a 1-dimensional mesh such as `[(\"x\", 6)]` (`mesh_1d` in the previous section), `Layout([\"unsharded\"], mesh_1d)` is a layout for a rank-1 tensor replicated on 6 devices.\n",
-        "\n",
-        "\"Layout"
+        "On a 1-dimensional mesh such as `[(\"x\", 6)]` (`mesh_1d` in the previous section), `Layout([\"unsharded\", \"unsharded\"], mesh_1d)` is a layout for a rank-2 tensor replicated across 6 devices.\n",
+        "\"A"
       ]
     },
     {
@@ -261,19 +261,47 @@
       },
       "outputs": [],
       "source": [
-        "layout_rank_1_1d = dtensor.Layout([dtensor.UNSHARDED], mesh_1d)\n",
-        "print(layout_rank_1_1d)"
+        "layout = dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh_1d)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "1Kyg0V3ehMNJ"
+        "id": "ywRJwuLDt2Qq"
       },
       "source": [
-        "Given a 2-dimensional 3x2 mesh such as `[(\"x\", 3), (\"y\", 2)]`, (`mesh_2d` from the previous section), `Layout([\"x\", dtensor.UNSHARDED], mesh_2d)` is a layout for a rank-2 `Tensor`, whose first axis is sharded on mesh dimension `x`, and second axis is replicated (`UNSHARDED`).\n",
+        "Using the same tensor and mesh the layout `Layout(['unsharded', 'x'])` would shard the second axis of the tensor across the 6 devices.\n",
         "\n",
-        "\"Layout\n"
+        "\"A"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7BgqL0jUvV5a"
+      },
+      "outputs": [],
+      "source": [
+        "layout = dtensor.Layout([dtensor.UNSHARDED, 'x'], mesh_1d)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DgciDNmK76l9"
+      },
+      "source": [
+        "Given a 2-dimensional 3x2 mesh such as `[(\"x\", 3), (\"y\", 2)]`, (`mesh_2d` from the previous section), `Layout([\"y\", \"x\"], mesh_2d)` is a layout for a rank-2 `Tensor` whose first axis is sharded across across mesh dimension `\"y\"`, and whose second axis is sharded across mesh dimension `\"x\"`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Eyp_qOSyvieo"
+      },
+      "source": [
+        "\"A\n"
       ]
     },
     {
@@ -284,8 +312,29 @@
       },
       "outputs": [],
       "source": [
-        "layout_rank_2_2d = dtensor.Layout(['x', dtensor.UNSHARDED], mesh_2d)\n",
-        "print(layout_rank_2_2d)"
+        "layout = dtensor.Layout(['y', 'x'], mesh_2d)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1Kyg0V3ehMNJ"
+      },
+      "source": [
+        "For the same `mesh_2d`, the layout `Layout([\"x\", dtensor.UNSHARDED], mesh_2d)` is a layout for a rank-2 `Tensor` that is replicated across `\"y\"`, and whose first axis is sharded on mesh dimension `x`.\n",
+        "\n",
+        "\"A\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IkWe6mVl7uRb"
+      },
+      "outputs": [],
+      "source": [
+        "layout = dtensor.Layout([\"x\", dtensor.UNSHARDED], mesh_2d)"
       ]
     },
     {
@@ -450,8 +499,7 @@
       "source": [
         "The inverse operation of `dtensor.unpack` is `dtensor.pack`. Component tensors can be packed back into a DTensor.\n",
         "\n",
-        "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices.\n",
-        "\n"
+        "The components must have the same rank and dtype, which will be the rank and dtype of the returned DTensor. However there is no strict requirement on the device placement of component tensors as inputs of `dtensor.unpack`: the function will automatically copy the component tensors to their respective corresponding devices.\n"
       ]
     },
     {
@@ -599,7 +647,7 @@
       "source": [
         "You can inspect the component tensors of the created DTensor and verify they are indeed sharded according to your scheme. It may be helpful to illustrate the situation with a chart:\n",
         "\n",
-        "  \"A\n"
       ]
     },
diff --git a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
index 0b6b3241e37..4fdb826afab 100644
--- a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
+++ b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
@@ -132,10 +132,24 @@
       },
       "outputs": [],
       "source": [
+        "import tempfile\n",
+        "import numpy as np\n",
+        "import tensorflow_datasets as tfds\n",
+        "\n",
         "import tensorflow as tf\n",
-        "from tensorflow.experimental import dtensor\n",
-        "print('TensorFlow version:', tf.__version__)\n",
         "\n",
+        "from tensorflow.experimental import dtensor\n",
+        "print('TensorFlow version:', tf.__version__)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oHtO6MJLUXlz"
+      },
+      "outputs": [],
+      "source": [
         "def configure_virtual_cpus(ncpu):\n",
         "  phy_devices = tf.config.list_physical_devices('CPU')\n",
         "  tf.config.set_logical_device_configuration(phy_devices[0], [\n",
@@ -156,41 +170,19 @@
       "source": [
         "## Download the dataset\n",
         "\n",
-        "Download the IMDB reviews data set to train the sentiment analysis model. For the sake of running speed, cap the training set to the first 10K samples."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "HG_ASSzR4IWW"
-      },
-      "outputs": [],
-      "source": [
-        "import tensorflow_datasets as tfds\n",
-        "\n",
-        "data = tfds.load('imdb_reviews', split='train', shuffle_files=True)\n",
-        "sample = data.take(1024*10)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "gXSIcdeTZ7jL"
-      },
-      "source": [
-        "Verify that the samples are not biased by checking that there are comparable numbers of positive and negative labels."
+        "Download the IMDB reviews data set to train the sentiment analysis model."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "8rG3rz8cANmB"
+        "id": "fW4w4QlFVHhx"
       },
       "outputs": [],
       "source": [
-        "tf.math.bincount(tf.cast(tf.reshape(list(sample.batch(1024).map(lambda x: x['label'])), -1), tf.int32))"
+        "train_data = tfds.load('imdb_reviews', split='train', shuffle_files=True, batch_size=64)\n",
+        "train_data"
       ]
     },
     {
@@ -219,59 +211,23 @@
       },
       "outputs": [],
       "source": [
-        "tvl = tf.keras.layers.TextVectorization(output_mode='tf_idf', max_tokens=1200, output_sequence_length=None)\n",
-        "tvl.adapt(data=sample.batch(1024).map(lambda x: x['text']))\n",
-        "dataset_x = tf.keras.models.Sequential([tvl]).predict(sample.batch(1024).map(lambda x: x['text']))\n",
-        "dataset_y = tf.keras.models.Sequential([]).predict(sample.batch(1024).map(lambda x: x['label']))"
+        "text_vectorization = tf.keras.layers.TextVectorization(output_mode='tf_idf', max_tokens=1200, output_sequence_length=None)\n",
+        "text_vectorization.adapt(data=train_data.map(lambda x: x['text']))"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "bEtzRtXXBeBB"
+        "id": "q16bjngoVwQp"
       },
       "outputs": [],
       "source": [
-        "dataset = tf.data.Dataset.from_tensor_slices({\n",
-        "    'x': dataset_x,\n",
-        "    'y': dataset_y,\n",
-        "})\n",
-        "\n",
-        "dataset.take(1).get_single_element()\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "zPtep4hpQqJ9"
-      },
-      "source": [
-        "### Mock dataset\n",
-        "To get better and faster convergence, you can train with a mock dataset that has the feature channels fully correlated with the label."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "W_DKa6wkFVHe"
-      },
-      "outputs": [],
-      "source": [
-        "USE_MOCK_DATA = False  # Change to True to use mock data\n",
-        "\n",
-        "def processor(data):\n",
-        "  # Rewrite the data, such that\n",
-        "  #  - data channel 0 triggers label 0, and\n",
-        "  #  - channel -1 triggers label 1\n",
-        "  x = tf.where(data['y'] != 0,\n",
-        "                  tf.range(len(data['x']), dtype=tf.float32),\n",
-        "                  len(data['x']) * 1.0 - tf.range(len(data['x']), dtype=tf.float32))\n",
-        "  return {'x': x, 'y' : data['y']}\n",
+        "def vectorize(features):\n",
+        "  return text_vectorization(features['text']), features['label']\n",
         "\n",
-        "if USE_MOCK_DATA:\n",
-        "  dataset = dataset.map(processor)"
+        "train_data_vec = train_data.map(vectorize)\n",
+        "train_data_vec"
       ]
     },
     {
@@ -331,7 +287,8 @@
       "source": [
         "class Dense(tf.Module):\n",
         "\n",
-        "  def __init__(self, input_size, output_size, init_seed, weight_layout):\n",
+        "  def __init__(self, input_size, output_size,\n",
+        "               init_seed, weight_layout, activation=None):\n",
         "    super().__init__()\n",
         "\n",
         "    random_normal_initializer = tf.function(tf.random.stateless_normal)\n",
@@ -342,7 +299,10 @@
         "            shape=[input_size, output_size],\n",
         "            seed=init_seed\n",
         "            ))\n",
-        "\n",
+        "    if activation is None:\n",
+        "      activation = lambda x:x\n",
+        "    self.activation = activation\n",
+        "    \n",
         "    # bias is sharded the same way as the last axis of weight.\n",
         "    bias_layout = weight_layout.delete([0])\n",
         "\n",
@@ -351,7 +311,7 @@
         "\n",
         "  def __call__(self, x):\n",
         "    y = tf.matmul(x, self.weight) + self.bias\n",
-        "    y = tf.nn.relu(y)\n",
+        "    y = self.activation(y)\n",
         "\n",
         "    return y"
       ]
@@ -425,8 +385,24 @@
       "source": [
         "### Putting Layers Together\n",
         "\n",
-        "Next, build a Multi-layer perceptron (MLP) network with the building blocks above.\n",
-        "\n",
+        "Next, build a Multi-layer perceptron (MLP) network with the building blocks above.  The diagram below shows the axis relationships between the input `x` and the weight matrices for the two `Dense` layers without any DTensor sharding or replication applied."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "udFGAO-NrZw6"
+      },
+      "source": [
+        "\"The\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8DCQ0aQ5rQtB"
+      },
+      "source": [
         "The output of the first `Dense` layer is passed into the input of the second `Dense` layer (after the `BatchNorm`). Therefore, the preferred DTensor sharding for the output of first `Dense` layer ($\\mathbf{W_1}$) and the input of second `Dense` layer ($\\mathbf{W_2}$) is to shard $\\mathbf{W_1}$ and $\\mathbf{W_2}$ the same way along the common axis $\\hat{j}$,\n",
         "\n",
         "$$\n",
@@ -452,17 +428,16 @@
         "  def __init__(self, dense_layouts: Tuple[dtensor.Layout, dtensor.Layout]):\n",
         "    super().__init__()\n",
         "\n",
-        "    self.dense1 = Dense(1200, 48, (1, 2), dense_layouts[0])\n",
-        "    self.bn1 = BatchNorm()\n",
+        "    self.dense1 = Dense(\n",
+        "        1200, 48, (1, 2), dense_layouts[0], activation=tf.nn.relu)\n",
+        "    self.bn = BatchNorm()\n",
         "    self.dense2 = Dense(48, 2, (3, 4), dense_layouts[1])\n",
-        "    self.bn2 = BatchNorm()\n",
         "\n",
         "  def __call__(self, x):\n",
         "    y = x\n",
         "    y = self.dense1(y)\n",
-        "    y = self.bn1(y)\n",
+        "    y = self.bn(y)\n",
         "    y = self.dense2(y)\n",
-        "    y = self.bn2(y)\n",
         "    return y\n"
       ]
     },
@@ -489,17 +464,18 @@
         "  def __init__(self, mesh, input_mesh_dim, inner_mesh_dim1, output_mesh_dim):\n",
         "    super().__init__()\n",
         "\n",
-        "    self.dense1 = Dense(1200, 48, (1, 2), dtensor.Layout([input_mesh_dim, inner_mesh_dim1], mesh))\n",
-        "    self.bn1 = BatchNorm()\n",
+        "    self.dense1 = Dense(\n",
+        "        1200, 48, (1, 2), dtensor.Layout([input_mesh_dim, inner_mesh_dim1], mesh),\n",
+        "        activation=tf.nn.relu)\n",
+        "    self.bn = BatchNorm()\n",
         "    self.dense2 = Dense(48, 2, (3, 4), dtensor.Layout([inner_mesh_dim1, output_mesh_dim], mesh))\n",
         "\n",
         "\n",
         "  def __call__(self, x):\n",
         "    y = x\n",
         "    y = self.dense1(y)\n",
-        "    y = self.bn1(y)\n",
+        "    y = self.bn(y)\n",
         "    y = self.dense2(y)\n",
-        "    y = self.bn2(y)\n",
         "    return y"
       ]
     },
@@ -524,10 +500,10 @@
         "\n",
         "model = MLP([dtensor.Layout.replicated(WORLD, rank=2),\n",
         "             dtensor.Layout.replicated(WORLD, rank=2)])\n",
-        "x = dataset.take(5).batch(5).get_single_element()['x']\n",
-        "y = dataset.take(5).batch(5).get_single_element()['y']\n",
-        "x = dtensor.copy_to_mesh(x, dtensor.Layout.replicated(WORLD, rank=2))\n",
-        "print(model(x))"
+        "\n",
+        "sample_x, sample_y = train_data_vec.take(1).get_single_element()\n",
+        "sample_x = dtensor.copy_to_mesh(sample_x, dtensor.Layout.replicated(WORLD, rank=2))\n",
+        "print(model(sample_x))"
       ]
     },
     {
@@ -616,7 +592,7 @@
         "\n",
         "A typical data parallelism training loop uses a DTensor `Mesh` that consists of a single `batch` dimension, where each device becomes a replica that receives a shard from the global batch.\n",
         "\n",
-        "\"Data\n",
+        "\"Data\n",
         "\n",
         "\n",
         "The replicated model runs on the replica, therefore the model variables are fully replicated (unsharded)."
@@ -632,10 +608,8 @@
       "source": [
         "mesh = dtensor.create_mesh([(\"batch\", 8)], devices=DEVICES)\n",
         "\n",
-        "model = MLP([\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n",
-        "])\n"
+        "model = MLP([dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n",
+        "             dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),])\n"
       ]
     },
     {
@@ -659,20 +633,16 @@
       },
       "outputs": [],
       "source": [
-        "batch_size = 48\n",
-        "\n",
-        "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n",
-        "\n",
         "def repack_batch(x, y, mesh):\n",
         "  x = repack_local_tensor(x, layout=dtensor.Layout(['batch', dtensor.UNSHARDED], mesh))\n",
         "  y = repack_local_tensor(y, layout=dtensor.Layout(['batch'], mesh))\n",
         "  return x, y\n",
         "\n",
-        "input = batched_dataset.take(1).get_single_element()\n",
-        "x, y = repack_batch(input['x'], input['y'], mesh)\n",
+        "sample_x, sample_y = train_data_vec.take(1).get_single_element()\n",
+        "sample_x, sample_y = repack_batch(sample_x, sample_y, mesh)\n",
         "\n",
-        "print('x', x[:, 0])\n",
-        "print('y', y)"
+        "print('x', sample_x[:, 0])\n",
+        "print('y', sample_y)"
       ]
     },
     {
@@ -740,9 +710,11 @@
       },
       "outputs": [],
       "source": [
+        "CHECKPOINT_DIR = tempfile.mkdtemp()\n",
+        "\n",
         "def start_checkpoint_manager(mesh, model):\n",
         "  ckpt = dtensor.DTensorCheckpoint(mesh, root=model)\n",
-        "  manager = tf.train.CheckpointManager(ckpt, '/tmp/checkpoints/data-and-model/cp', max_to_keep=3)\n",
+        "  manager = tf.train.CheckpointManager(ckpt, CHECKPOINT_DIR, max_to_keep=3)\n",
         "\n",
         "  if manager.latest_checkpoint:\n",
         "    print(\"Restoring a checkpoint\")\n",
@@ -773,16 +745,16 @@
       },
       "outputs": [],
       "source": [
-        "num_epochs = 3\n",
+        "num_epochs = 2\n",
         "manager = start_checkpoint_manager(mesh, model)\n",
         "\n",
         "for epoch in range(num_epochs):\n",
         "  step = 0\n",
-        "  pbar = tf.keras.utils.Progbar(target=None, stateful_metrics=[])\n",
+        "  pbar = tf.keras.utils.Progbar(target=int(train_data_vec.cardinality()), stateful_metrics=[])\n",
         "  metrics = {'epoch': epoch}\n",
-        "  for input in batched_dataset:\n",
+        "  for x,y in train_data_vec:\n",
         "\n",
-        "    x, y = repack_batch(input['x'], input['y'], mesh)\n",
+        "    x, y = repack_batch(x, y, mesh)\n",
         "\n",
         "    metrics.update(train_step(model, x, y, 1e-2))\n",
         "\n",
@@ -800,7 +772,7 @@
       "source": [
         "## Model Parallel Training\n",
         "\n",
-        "If you switch to a 2 dimensional `Mesh`, and shard the model variables, along the second mesh dimension, then the training becomes Model Parallel.\n",
+        "If you switch to a 2 dimensional `Mesh`, and shard the model variables along the second mesh dimension, then the training becomes Model Parallel.\n",
         "\n",
         "In Model Parallel training, each model replica spans multiple devices (2 in this case):\n",
         "\n",
@@ -808,7 +780,7 @@
         "- The 2 devices within a single model replica receive replicated training data.\n",
         "\n",
         "\n",
-        "\"Model\n"
+        "\"Model\n"
       ]
     },
     {
@@ -820,10 +792,8 @@
       "outputs": [],
       "source": [
         "mesh = dtensor.create_mesh([(\"batch\", 4), (\"model\", 2)], devices=DEVICES)\n",
-        "model = MLP([\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, \"model\"], mesh),\n",
-        "            dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)]\n",
-        "            )\n"
+        "model = MLP([dtensor.Layout([dtensor.UNSHARDED, \"model\"], mesh), \n",
+        "             dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)])"
       ]
     },
     {
@@ -843,9 +813,6 @@
       },
       "outputs": [],
       "source": [
-        "batch_size = 48\n",
-        "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n",
-        "\n",
         "def repack_batch(x, y, mesh):\n",
         "  x = repack_local_tensor(x, layout=dtensor.Layout(['batch', dtensor.UNSHARDED], mesh))\n",
         "  y = repack_local_tensor(y, layout=dtensor.Layout(['batch'], mesh))\n",
@@ -876,10 +843,9 @@
         "\n",
         "for epoch in range(num_epochs):\n",
         "  step = 0\n",
-        "  pbar = tf.keras.utils.Progbar(target=int(batched_dataset.cardinality()))\n",
+        "  pbar = tf.keras.utils.Progbar(target=int(train_data_vec.cardinality()))\n",
         "  metrics = {'epoch': epoch}\n",
-        "  for input in batched_dataset:\n",
-        "    x, y = input['x'], input['y']\n",
+        "  for x,y in train_data_vec:\n",
         "    x, y = repack_batch(x, y, mesh)\n",
         "    metrics.update(train_step(model, x, y, 1e-2))\n",
         "    pbar.update(step, values=metrics.items(), finalize=False)\n",
@@ -905,7 +871,7 @@
       "source": [
         "When training data of very high dimensionality (e.g. a very large image or a video), it may be desirable to shard along the feature dimension. This is called [Spatial Partitioning](https://cloud.google.com/blog/products/ai-machine-learning/train-ml-models-on-large-images-and-3d-volumes-with-spatial-partitioning-on-cloud-tpus), which was first introduced into TensorFlow for training models with large 3-d input samples.\n",
         "\n",
-        "\"Spatial\n",
+        "\"Spatial\n",
         "\n",
         "DTensor also supports this case. The only change you need to do is to create a Mesh that includes a `feature` dimension, and apply the corresponding `Layout`.\n"
       ]
@@ -919,9 +885,8 @@
       "outputs": [],
       "source": [
         "mesh = dtensor.create_mesh([(\"batch\", 2), (\"feature\", 2), (\"model\", 2)], devices=DEVICES)\n",
-        "model = MLP([\n",
-        "            dtensor.Layout([\"feature\", \"model\"], mesh),\n",
-        "            dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)])\n"
+        "model = MLP([dtensor.Layout([\"feature\", \"model\"], mesh), \n",
+        "             dtensor.Layout([\"model\", dtensor.UNSHARDED], mesh)])\n"
       ]
     },
     {
@@ -941,10 +906,6 @@
       },
       "outputs": [],
       "source": [
-        "batch_size = 48\n",
-        "\n",
-        "batched_dataset = dataset.batch(batch_size, drop_remainder=True)\n",
-        "\n",
         "def repack_batch_for_spt(x, y, mesh):\n",
         "    # Shard data on feature dimension, too\n",
         "    x = repack_local_tensor(x, layout=dtensor.Layout([\"batch\", 'feature'], mesh))\n",
@@ -958,9 +919,7 @@
         "id": "Ygl9dqMUHTVN"
       },
       "source": [
-        "The Spatial parallel training can also continue from a checkpoint created with other parallell training schemes.\n",
-        "\n",
-        "Let this train with a few epochs to get an accuracy above 0.5, slightly better than random guessing before saving the model in the next section."
+        "The Spatial parallel training can also continue from a checkpoint created with other parallell training schemes."
       ]
     },
     {
@@ -971,16 +930,15 @@
       },
       "outputs": [],
       "source": [
-        "num_epochs = 30\n",
+        "num_epochs = 2\n",
         "\n",
         "manager = start_checkpoint_manager(mesh, model)\n",
         "for epoch in range(num_epochs):\n",
         "  step = 0\n",
         "  metrics = {'epoch': epoch}\n",
-        "  pbar = tf.keras.utils.Progbar(target=int(batched_dataset.cardinality()))\n",
+        "  pbar = tf.keras.utils.Progbar(target=int(train_data_vec.cardinality()))\n",
         "\n",
-        "  for input in batched_dataset:\n",
-        "    x, y = input['x'], input['y']\n",
+        "  for x, y in train_data_vec:\n",
         "    x, y = repack_batch_for_spt(x, y, mesh)\n",
         "    metrics.update(train_step(model, x, y, 1e-2))\n",
         "\n",
@@ -1014,14 +972,23 @@
       "outputs": [],
       "source": [
         "mesh = dtensor.create_mesh([(\"world\", 1)], devices=DEVICES[:1])\n",
-        "model_for_saving = MLP([\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh),\n",
-        "            dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh)])\n",
+        "mlp = MLP([dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh), \n",
+        "           dtensor.Layout([dtensor.UNSHARDED, dtensor.UNSHARDED], mesh)])\n",
+        "\n",
+        "manager = start_checkpoint_manager(mesh, mlp)\n",
+        "\n",
+        "model_for_saving = tf.keras.Sequential([\n",
+        "  text_vectorization,\n",
+        "  mlp\n",
+        "])\n",
         "\n",
-        "manager = start_checkpoint_manager(mesh, model_for_saving)\n",
-        "tf.saved_model.save(model_for_saving, \"/tmp/saved_model\",\n",
-        "                    signatures=tf.function(model_for_saving.__call__).get_concrete_function(tf.TensorSpec([None, 1200], tf.float32))\n",
-        "                    )"
+        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
+        "def run(inputs):\n",
+        "  return {'result': model_for_saving(inputs)}\n",
+        "\n",
+        "tf.saved_model.save(\n",
+        "    model_for_saving, \"/tmp/saved_model\",\n",
+        "    signatures=run)"
       ]
     },
     {
@@ -1033,6 +1000,18 @@
         "As of TensorFlow 2.9.0, you can only call a loaded signature with a regular Tensor, or a fully replicated DTensor (which will be converted to a regular Tensor)."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "HG_ASSzR4IWW"
+      },
+      "outputs": [],
+      "source": [
+        "sample_batch = train_data.take(1).get_single_element()\n",
+        "sample_batch"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -1043,9 +1022,19 @@
       "source": [
         "loaded = tf.saved_model.load(\"/tmp/saved_model\")\n",
         "\n",
-        "batch = dataset.batch(4, drop_remainder=True).take(1).get_single_element()\n",
-        "\n",
-        "print(loaded.signatures[\"serving_default\"](batch['x']))"
+        "run_sig = loaded.signatures[\"serving_default\"]\n",
+        "result = run_sig(sample_batch['text'])['result']"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GahGbv0ZmkJb"
+      },
+      "outputs": [],
+      "source": [
+        "np.mean(tf.argmax(result, axis=-1) == sample_batch['label'])"
       ]
     },
     {
@@ -1071,6 +1060,7 @@
     "colab": {
       "collapsed_sections": [],
       "name": "dtensor_ml_tutorial.ipynb",
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From b123c0a57b7e75646a091e958ee28ab5833194ce Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 4 May 2022 10:44:34 -0700
Subject: [PATCH 116/872] Fix broken quotes.

PiperOrigin-RevId: 446496649
---
 site/en/community/contribute/docs_ref.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/community/contribute/docs_ref.md b/site/en/community/contribute/docs_ref.md
index 12d43792ebf..fbf207a47f1 100644
--- a/site/en/community/contribute/docs_ref.md
+++ b/site/en/community/contribute/docs_ref.md
@@ -100,7 +100,7 @@ TensorFlow uses a few customizations to the builtin doctest logic:
 
     ```
     def NewLayer():
-      “””This layer does cool stuff.
+      """This layer does cool stuff.
 
       Example usage:
 
@@ -108,7 +108,7 @@ TensorFlow uses a few customizations to the builtin doctest logic:
       >>> new_layer = NewLayer(x)
       >>> new_layer
       
-      “””
+      """
     ```
 
 *   *Floating point values*: The TensorFlow doctest extracts float values from

From d28cb65a39ace6d661bde83e4c74fcebca7d0ef0 Mon Sep 17 00:00:00 2001
From: Nathan Luehr 
Date: Wed, 4 May 2022 14:27:23 -0500
Subject: [PATCH 117/872] Add note about libnvinfer packages being optional.

---
 site/en/install/gpu.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/site/en/install/gpu.md b/site/en/install/gpu.md
index b34ad5ea258..21cd3494cc4 100644
--- a/site/en/install/gpu.md
+++ b/site/en/install/gpu.md
@@ -67,7 +67,7 @@ The following NVIDIA® software must be installed on your system:
 *   [cuDNN SDK 8.1.0](https://developer.nvidia.com/cudnn){:.external}
     [cuDNN versions](https://developer.nvidia.com/rdp/cudnn-archive){:.external}).
 *   *(Optional)*
-    [TensorRT 7](https://docs.nvidia.com/deeplearning/tensorrt/archives/index.html#trt_7){:.external}
+    [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/archives/index.html#trt_7){:.external}
     to improve latency and throughput for inference on some models.
 
 ## Linux setup
@@ -104,6 +104,7 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i
 sudo apt-get update
 
 # Install development and runtime libraries (~4GB)
+# libnvinfer packages are optional, needed to support TensorRT inference.
 sudo apt-get install --no-install-recommends \
     cuda-11-2 \
     libcudnn8=8.1.0.77-1+cuda11.2  \
@@ -128,6 +129,7 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i
 sudo apt-get update
 
 # Install development and runtime libraries (~4GB)
+# libnvinfer packages are optional, needed to support TensorRT inference.
 sudo apt-get install --no-install-recommends \
     cuda-11-2 \
     libcudnn8=8.1.0.77-1+cuda11.2  \

From 5bf40425cb31e061d90c491401e22921ae326714 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 5 May 2022 07:15:02 +0100
Subject: [PATCH 118/872] Update site/en/install/gpu.md

---
 site/en/install/gpu.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/install/gpu.md b/site/en/install/gpu.md
index 21cd3494cc4..629a0d2182e 100644
--- a/site/en/install/gpu.md
+++ b/site/en/install/gpu.md
@@ -97,8 +97,8 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i
 
 
 # Add NVIDIA package repositories
-# Note: For Ubuntu version other than 18.04 or CPU arch other than x86,
-# replace 'ubuntu1804' and/or 'x86_64' as needed in the following URL.
+# Note: For the Ubuntu version other than 18.04 or CPU architecture other than x86,
+# replace `ubuntu1804` and/or `x86_64` as needed in the following URL.
 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
 sudo dpkg -i cuda-keyring_1.0-1_all.deb
 sudo apt-get update

From 7c660f603802d4c4c72048937462348a9e37c5b6 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 5 May 2022 07:32:29 +0100
Subject: [PATCH 119/872] Fix /code in GPU installation instructions gpu.md

---
 site/en/install/gpu.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/install/gpu.md b/site/en/install/gpu.md
index 629a0d2182e..9662cb6f5b4 100644
--- a/site/en/install/gpu.md
+++ b/site/en/install/gpu.md
@@ -123,7 +123,7 @@ complicates installation of the NVIDIA driver and is beyond the scope of these i
 
 # Add NVIDIA package repositories
 sudo apt-get update
-sudo apt-get install -y 
+sudo apt-get install -y apt-transport-https
 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-keyring_1.0-1_all.deb
 sudo dpkg -i cuda-keyring_1.0-1_all.deb
 sudo apt-get update

From 3bc689180d713ce22df07aa7b41e1ab283dee03e Mon Sep 17 00:00:00 2001
From: sushreebarsa <84765720+sushreebarsa@users.noreply.github.com>
Date: Thu, 5 May 2022 13:38:00 +0530
Subject: [PATCH 120/872] Changed the name of the API in documentation

'tf.metrics.Mean' is redirected to tf.keras.metrics.Mean so changed the name in the TF documentation accordingly.
---
 site/en/tutorials/distribute/custom_training.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index 9dbb2d3fad5..c81c3b5c8ec 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -649,7 +649,7 @@
         "\n",
         "Note: As a general rule, you should use `tf.keras.Metrics` to track per-sample values and avoid values that have been aggregated within a replica.\n",
         "\n",
-        "Because of the loss scaling computation that is carried out, it's not recommended to use `tf.metrics.Mean` to track the training loss across different replicas.\n",
+        "Because of the loss scaling computation that is carried out, it's not recommended to use `tf.keras.metrics.Mean` to track the training loss across different replicas.\n",
         "\n",
         "For example, if you run a training job with the following characteristics:\n",
         "* Two replicas\n",
@@ -659,7 +659,7 @@
         "\n",
         "With loss scaling, you calculate the per-sample value of loss on each replica by adding the loss values, and then dividing by the global batch size. In this case: `(2 + 3) / 4 = 1.25` and `(4 + 5) / 4 = 2.25`. \n",
         "\n",
-        "If you use `tf.metrics.Mean` to track loss across the two replicas, the result is different. In this example, you end up with a `total` of 3.50 and `count` of 2, which results in `total`/`count` = 1.75  when `result()` is called on the metric. Loss calculated with `tf.keras.Metrics` is scaled by an additional factor that is equal to the number of replicas in sync."
+        "If you use `tf.keras.metrics.Mean` to track loss across the two replicas, the result is different. In this example, you end up with a `total` of 3.50 and `count` of 2, which results in `total`/`count` = 1.75  when `result()` is called on the metric. Loss calculated with `tf.keras.Metrics` is scaled by an additional factor that is equal to the number of replicas in sync."
       ]
     },
     {

From fed6ce5469285e16ba7a5cd4524c667e8728912e Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Thu, 5 May 2022 08:08:55 +0000
Subject: [PATCH 121/872] nbfmt

---
 site/en/tutorials/distribute/custom_training.ipynb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index c81c3b5c8ec..4ea0d1ef424 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -699,7 +699,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "custom_training.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 7d5ea2e986a4eae7573be3face00b3cccd4b8b8b Mon Sep 17 00:00:00 2001
From: Edward Loper 
Date: Thu, 5 May 2022 06:00:38 -0700
Subject: [PATCH 122/872] RaggedTensor guide: Add a section on ragged shapes.

PiperOrigin-RevId: 446696647
---
 site/en/guide/ragged_tensor.ipynb | 259 +++++++++++++++++++++++++++++-
 1 file changed, 257 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/ragged_tensor.ipynb b/site/en/guide/ragged_tensor.ipynb
index ba31138a3cf..aa4774d4e10 100644
--- a/site/en/guide/ragged_tensor.ipynb
+++ b/site/en/guide/ragged_tensor.ipynb
@@ -81,6 +81,7 @@
       },
       "outputs": [],
       "source": [
+        "!pip install --pre -U tensorflow\n",
         "import math\n",
         "import tensorflow as tf"
       ]
@@ -1459,13 +1460,267 @@
         "print(\"Indexed value:\", rt[1].numpy())"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "J87jMZa0M_YW"
+      },
+      "source": [
+        "## Ragged Shapes\n",
+        "\n",
+        "The shape of a tensor specifies the size of each axis.  For example, the shape of `[[1, 2], [3, 4], [5, 6]]` is `[3, 2]`, since there are 3 rows and 2 columns.  TensorFlow has two separate but related ways to describe shapes:\n",
+        "\n",
+        "* ***static shape***: Information about axis sizes that is known statically (e.g., while tracing a `tf.function`).  May be partially specified.\n",
+        "\n",
+        "* ***dynamic shape***: Runtime information about the axis sizes."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "IOETE_OLPLZo"
+      },
+      "source": [
+        "### Static shape\n",
+        "\n",
+        "A Tensor's static shape contains information about its axis sizes that is known at graph-construction time.  For both `tf.Tensor` and `tf.RaggedTensor`, it is available using the `.shape` property, and is encoded using `tf.TensorShape`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "btGDjT4uNgQy"
+      },
+      "outputs": [],
+      "source": [
+        "x = tf.constant([[1, 2], [3, 4], [5, 6]])\n",
+        "x.shape  # shape of a tf.tensor"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "__OgvmrGPEjq"
+      },
+      "outputs": [],
+      "source": [
+        "rt = tf.ragged.constant([[1], [2, 3], [], [4]])\n",
+        "rt.shape  # shape of a tf.RaggedTensor"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9EWnQd3qPWaw"
+      },
+      "source": [
+        "The static shape of a ragged dimension is always `None` (i.e., unspecified).  However, the inverse is not true -- if a `TensorShape` dimension is `None`, then that could indicate that the dimension is ragged, *or* it could indicate that the dimension is uniform but that its size is not statically known."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "75E9YXYMNfne"
+      },
+      "source": [
+        "### Dynamic shape\n",
+        "\n",
+        "A tensor's dynamic shape contains information about its axis sizes that is known when the graph is run.  It is constructed using the `tf.shape` operation.  For `tf.Tensor`, `tf.shape` returns the shape as a 1D integer `Tensor`, where `tf.shape(x)[i]` is the size of axis `i`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "kWJ7Cn1EQTD_"
+      },
+      "outputs": [],
+      "source": [
+        "x = tf.constant([['a', 'b'], ['c', 'd'], ['e', 'f']])\n",
+        "tf.shape(x)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BeZEfxwmRcSv"
+      },
+      "source": [
+        "However, a 1D `Tensor` is not expressive enough to describe the shape of a `tf.RaggedTensor`.  Instead, the dynamic shape for ragged tensors is encoded using a dedicated type, `tf.experimental.DynamicRaggedShape`.  In the following example, the `DynamicRaggedShape` returned by `tf.shape(rt)` indicates that the ragged tensor has 4 rows, with lengths 1, 3, 0, and 2:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "nZc2wqgQQUFU"
+      },
+      "outputs": [],
+      "source": [
+        "rt = tf.ragged.constant([[1], [2, 3, 4], [], [5, 6]])\n",
+        "rt_shape = tf.shape(rt)\n",
+        "print(rt_shape)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EphU60YvTf98"
+      },
+      "source": [
+        "#### Dynamic shape: operations\n",
+        "\n",
+        "`DynamicRaggedShape`s can be used with most TensorFlow ops that expect shapes, including `tf.reshape`, `tf.zeros`, `tf.ones`. `tf.fill`, `tf.broadcast_dynamic_shape`, and `tf.broadcast_to`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pclAODLXT6Gr"
+      },
+      "outputs": [],
+      "source": [
+        "print(f\"tf.reshape(x, rt_shape) = {tf.reshape(x, rt_shape)}\")\n",
+        "print(f\"tf.zeros(rt_shape) = {tf.zeros(rt_shape)}\")\n",
+        "print(f\"tf.ones(rt_shape) = {tf.ones(rt_shape)}\")\n",
+        "print(f\"tf.fill(rt_shape, 9) = {tf.fill(rt_shape, 'x')}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rNP_3_btRAHj"
+      },
+      "source": [
+        "#### Dynamic shape: indexing and slicing\n",
+        "\n",
+        "`DynamicRaggedShape` can be also be indexed to get the sizes of uniform dimensions.  For example, we can find the number of rows in a raggedtensor using `tf.shape(rt)[0]` (just as we would for a non-ragged tensor):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "MzQvPhsxS6HN"
+      },
+      "outputs": [],
+      "source": [
+        "rt_shape[0]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "wvr2iT6zS_e8"
+      },
+      "source": [
+        "However, it is an error to use indexing to try to retrieve the size of a ragged dimension, since it doesn't have a single size.  (Since `RaggedTensor` keeps track of which axes are ragged, this error is only thrown during eager execution or when tracing a `tf.function`; it will never be thrown when executing a concrete function.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "HgGMk0LeTGik"
+      },
+      "outputs": [],
+      "source": [
+        "try:\n",
+        "  rt_shape[1]\n",
+        "except ValueError as e:\n",
+        "  print(\"Got expected ValueError:\", e)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5QUsdawGU0SM"
+      },
+      "source": [
+        "`DynamicRaggedShape`s can also be sliced, as long as the slice either begins with axis `0`, or contains only dense dimensions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "APT72EaBU70t"
+      },
+      "outputs": [],
+      "source": [
+        "rt_shape[:1]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "a-Wl9IrQXcdY"
+      },
+      "source": [
+        "#### Dynamic shape: encoding\n",
+        "\n",
+        "`DynamicRaggedShape` is encoded using two fields:\n",
+        "\n",
+        "* `inner_shape`: An integer vector giving the shape of a dense `tf.Tensor`.\n",
+        "* `row_partitions`: A list of `tf.experimental.RowPartition` objects, describing how the outermost dimension of that inner shape should be partitioned to add ragged axes.\n",
+        "\n",
+        "For more information about row partitions, see the \"RaggedTensor encoding\" section below, and the API docs for `tf.experimental.RowPartition`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "jfeY9tTcV_zL"
+      },
+      "source": [
+        "#### Dynamic shape: construction\n",
+        "\n",
+        "`DynamicRaggedShape` is most often constructed by applying `tf.shape` to a `RaggedTensor`, but it can also be constructed directly:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NSRgD667WwIZ"
+      },
+      "outputs": [],
+      "source": [
+        "tf.experimental.DynamicRaggedShape(\n",
+        "    row_partitions=[tf.experimental.RowPartition.from_row_lengths([5, 3, 2])],\n",
+        "    inner_shape=[10, 8])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EjzVjs9MXIIA"
+      },
+      "source": [
+        "If the lengths of all rows are known statically, `DynamicRaggedShape.from_lengths` can also be used to construct a dynamic ragged shape.  (This is mostly useful for testing and demonstration code, since it's rare for the lengths of ragged dimensions to be known statically).\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gMxCzADUYIjY"
+      },
+      "outputs": [],
+      "source": [
+        "tf.experimental.DynamicRaggedShape.from_lengths([4, (2, 1, 0, 8), 12])"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "EdljbNPq-PWS"
       },
       "source": [
-        "## Broadcasting\n",
+        "### Broadcasting\n",
         "\n",
         "Broadcasting is the process of making tensors with different shapes have compatible shapes for elementwise operations. For more background on broadcasting, refer to:\n",
         "\n",
@@ -1491,7 +1746,7 @@
         "id": "-S2hOUWx-PWU"
       },
       "source": [
-        "### Broadcasting examples"
+        "#### Broadcasting examples"
       ]
     },
     {

From 9f3296b616620590ac27c69084c895b63b03172f Mon Sep 17 00:00:00 2001
From: Tomohiro Endo 
Date: Thu, 5 May 2022 23:14:07 +0900
Subject: [PATCH 123/872] Update bazel reference link

---
 site/en/install/source.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/site/en/install/source.md b/site/en/install/source.md
index 67db7c7ca96..7124f0b4ddf 100644
--- a/site/en/install/source.md
+++ b/site/en/install/source.md
@@ -54,7 +54,7 @@ Bazel and automatically downloads the correct Bazel version for TensorFlow. For
 ease of use, add Bazelisk as the `bazel` executable in your `PATH`.
 
 If Bazelisk is not available, you can manually
-[install Bazel](https://docs.bazel.build/versions/master/install.html). Make
+[install Bazel](https://bazel.build/install). Make
 sure to install a supported Bazel version: any version between
 `_TF_MIN_BAZEL_VERSION` and `_TF_MAX_BAZEL_VERSION` as specified in
 `tensorflow/configure.py`.
@@ -240,9 +240,9 @@ bazel build --config=v1 [--config=option] //tensorflow/tools/pip_package:build_p
 
 ### Bazel build options
 
-See the Bazel [command-line reference](https://docs.bazel.build/versions/master/command-line-reference.html)
+See the Bazel [command-line reference](https://bazel.build/reference/command-line-reference)
 for
-[build options](https://docs.bazel.build/versions/master/command-line-reference.html#build-options).
+[build options](https://bazel.build/reference/command-line-reference#build-options).
 
 Building TensorFlow from source can use a lot of RAM. If your system is
 memory-constrained, limit Bazel's RAM usage with: `--local_ram_resources=2048`.

From b56f626114e28e5c471112bde7fc88eeaa290fbe Mon Sep 17 00:00:00 2001
From: chunduriv <74177924+chunduriv@users.noreply.github.com>
Date: Fri, 6 May 2022 13:38:29 +0530
Subject: [PATCH 124/872] `tf.SparseTensor` to `tf.sparse.SparseTensor` in
 sparse_tensor guide

`tf.SparseTensor` is currently an alias for `tf.sparse.SparseTensor`. Hence replaced with the more consistent alias.
---
 site/en/guide/sparse_tensor.ipynb | 38 +++++++++++++++----------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/site/en/guide/sparse_tensor.ipynb b/site/en/guide/sparse_tensor.ipynb
index 2395c6e6365..37bad964320 100644
--- a/site/en/guide/sparse_tensor.ipynb
+++ b/site/en/guide/sparse_tensor.ipynb
@@ -79,7 +79,7 @@
       "source": [
         "## Sparse tensors in TensorFlow\n",
         "\n",
-        "TensorFlow represents sparse tensors through the `tf.SparseTensor` object. Currently, sparse tensors in TensorFlow are encoded using the coordinate list (COO) format. This encoding format is optimized for hyper-sparse matrices such as embeddings.\n",
+        "TensorFlow represents sparse tensors through the `tf.sparse.SparseTensor` object. Currently, sparse tensors in TensorFlow are encoded using the coordinate list (COO) format. This encoding format is optimized for hyper-sparse matrices such as embeddings.\n",
         "\n",
         "The COO encoding for sparse tensors is comprised of:\n",
         "\n",
@@ -87,9 +87,9 @@
         "  * `indices`: A 2D tensor with shape `[N, rank]`, containing the indices of the nonzero values.\n",
         "  * `dense_shape`: A 1D tensor with shape `[rank]`, specifying the shape of the tensor.\n",
         "\n",
-        "A ***nonzero*** value in the context of a `tf.SparseTensor` is a value that's not explicitly encoded. It is possible to explicitly include zero values in the `values` of a COO sparse matrix, but these \"explicit zeros\" are generally not included when referring to nonzero values in a sparse tensor.\n",
+        "A ***nonzero*** value in the context of a `tf.sparse.SparseTensor` is a value that's not explicitly encoded. It is possible to explicitly include zero values in the `values` of a COO sparse matrix, but these \"explicit zeros\" are generally not included when referring to nonzero values in a sparse tensor.\n",
         "\n",
-        "Note: `tf.SparseTensor` does not require that indices/values be in any particular order, but several ops assume that they're in row-major order. Use `tf.sparse.reorder` to create a copy of the sparse tensor that is sorted in the canonical row-major order. "
+        "Note: `tf.sparse.SparseTensor` does not require that indices/values be in any particular order, but several ops assume that they're in row-major order. Use `tf.sparse.reorder` to create a copy of the sparse tensor that is sorted in the canonical row-major order. "
       ]
     },
     {
@@ -98,7 +98,7 @@
         "id": "6Aq7ruwlyz79"
       },
       "source": [
-        "## Creating a `tf.SparseTensor`\n",
+        "## Creating a `tf.sparse.SparseTensor`\n",
         "\n",
         "Construct sparse tensors by directly specifying their `values`, `indices`, and `dense_shape`."
       ]
@@ -122,7 +122,7 @@
       },
       "outputs": [],
       "source": [
-        "st1 = tf.SparseTensor(indices=[[0, 3], [2, 4]],\n",
+        "st1 = tf.sparse.SparseTensor(indices=[[0, 3], [2, 4]],\n",
         "                      values=[10, 20],\n",
         "                      dense_shape=[3, 10])"
       ]
@@ -252,11 +252,11 @@
       },
       "outputs": [],
       "source": [
-        "st_a = tf.SparseTensor(indices=[[0, 2], [3, 4]],\n",
+        "st_a = tf.sparse.SparseTensor(indices=[[0, 2], [3, 4]],\n",
         "                       values=[31, 2], \n",
         "                       dense_shape=[4, 10])\n",
         "\n",
-        "st_b = tf.SparseTensor(indices=[[0, 2], [7, 0]],\n",
+        "st_b = tf.sparse.SparseTensor(indices=[[0, 2], [7, 0]],\n",
         "                       values=[56, 38],\n",
         "                       dense_shape=[4, 10])\n",
         "\n",
@@ -282,7 +282,7 @@
       },
       "outputs": [],
       "source": [
-        "st_c = tf.SparseTensor(indices=([0, 1], [1, 0], [1, 1]),\n",
+        "st_c = tf.sparse.SparseTensor(indices=([0, 1], [1, 0], [1, 1]),\n",
         "                       values=[13, 15, 17],\n",
         "                       dense_shape=(2,2))\n",
         "\n",
@@ -309,14 +309,14 @@
       },
       "outputs": [],
       "source": [
-        "sparse_pattern_A = tf.SparseTensor(indices = [[2,4], [3,3], [3,4], [4,3], [4,4], [5,4]],\n",
+        "sparse_pattern_A = tf.sparse.SparseTensor(indices = [[2,4], [3,3], [3,4], [4,3], [4,4], [5,4]],\n",
         "                         values = [1,1,1,1,1,1],\n",
         "                         dense_shape = [8,5])\n",
-        "sparse_pattern_B = tf.SparseTensor(indices = [[0,2], [1,1], [1,3], [2,0], [2,4], [2,5], [3,5], \n",
+        "sparse_pattern_B = tf.sparse.SparseTensor(indices = [[0,2], [1,1], [1,3], [2,0], [2,4], [2,5], [3,5], \n",
         "                                              [4,5], [5,0], [5,4], [5,5], [6,1], [6,3], [7,2]],\n",
         "                         values = [1,1,1,1,1,1,1,1,1,1,1,1,1,1],\n",
         "                         dense_shape = [8,6])\n",
-        "sparse_pattern_C = tf.SparseTensor(indices = [[3,0], [4,0]],\n",
+        "sparse_pattern_C = tf.sparse.SparseTensor(indices = [[3,0], [4,0]],\n",
         "                         values = [1,1],\n",
         "                         dense_shape = [8,6])\n",
         "\n",
@@ -381,7 +381,7 @@
       },
       "outputs": [],
       "source": [
-        "st2_plus_5 = tf.SparseTensor(\n",
+        "st2_plus_5 = tf.sparse.SparseTensor(\n",
         "    st2.indices,\n",
         "    st2.values + 5,\n",
         "    st2.dense_shape)\n",
@@ -394,7 +394,7 @@
         "id": "GFhO2ZZ53ga1"
       },
       "source": [
-        "## Using `tf.SparseTensor` with other TensorFlow APIs\n",
+        "## Using `tf.sparse.SparseTensor` with other TensorFlow APIs\n",
         "\n",
         "Sparse tensors work transparently with these TensorFlow APIs:\n",
         "\n",
@@ -449,7 +449,7 @@
         "y = tf.keras.layers.Dense(4)(x)\n",
         "model = tf.keras.Model(x, y)\n",
         "\n",
-        "sparse_data = tf.SparseTensor(\n",
+        "sparse_data = tf.sparse.SparseTensor(\n",
         "    indices = [(0,0),(0,1),(0,2),\n",
         "               (4,3),(5,0),(5,1)],\n",
         "    values = [1,1,1,1,1,1],\n",
@@ -569,9 +569,9 @@
         "\n",
         "`tf.train.Example` is a standard protobuf encoding for TensorFlow data. When using sparse tensors with `tf.train.Example`, you can:\n",
         "\n",
-        "* Read variable-length data into a `tf.SparseTensor` using `tf.io.VarLenFeature`. However, you should consider using `tf.io.RaggedFeature` instead.\n",
+        "* Read variable-length data into a `tf.sparse.SparseTensor` using `tf.io.VarLenFeature`. However, you should consider using `tf.io.RaggedFeature` instead.\n",
         "\n",
-        "* Read arbitrary sparse data into a `tf.SparseTensor` using `tf.io.SparseFeature`, which uses three separate feature keys to store the `indices`, `values`, and `dense_shape`."
+        "* Read arbitrary sparse data into a `tf.sparse.SparseTensor` using `tf.io.SparseFeature`, which uses three separate feature keys to store the `indices`, `values`, and `dense_shape`."
       ]
     },
     {
@@ -597,7 +597,7 @@
         "def f(x,y):\n",
         "  return tf.sparse.sparse_dense_matmul(x,y)\n",
         "\n",
-        "a = tf.SparseTensor(indices=[[0, 3], [2, 4]],\n",
+        "a = tf.sparse.SparseTensor(indices=[[0, 3], [2, 4]],\n",
         "                    values=[15, 25],\n",
         "                    dense_shape=[3, 10])\n",
         "\n",
@@ -616,11 +616,11 @@
       "source": [
         "## Distinguishing missing values from zero values\n",
         "\n",
-        "Most ops on `tf.SparseTensor`s treat missing values and explicit zero values identically. This is by design — a `tf.SparseTensor` is supposed to act just like a dense tensor.\n",
+        "Most ops on `tf.sparse.SparseTensor`s treat missing values and explicit zero values identically. This is by design — a `tf.sparse.SparseTensor` is supposed to act just like a dense tensor.\n",
         "\n",
         "However, there are a few cases where it can be useful to distinguish zero values from missing values. In particular, this allows for one way to encode missing/unknown data in your training data. For example, consider a use case where you have a tensor of scores (that can have any floating point value from -Inf to +Inf), with some missing scores. You can encode this tensor using a sparse tensor where the explicit zeros are known zero scores but the implicit zero values actually represent missing data and not zero. \n",
         "\n",
-        "Note: This is generally not the intended usage of `tf.SparseTensor`s; and you might want to also consier other techniques for encoding this such as for example using a separate mask tensor that identifies the locations of known/unknown values. However, exercise caution while using this approach, since most sparse operations will treat explicit and implicit zero values identically."
+        "Note: This is generally not the intended usage of `tf.sparse.SparseTensor`s; and you might want to also consier other techniques for encoding this such as for example using a separate mask tensor that identifies the locations of known/unknown values. However, exercise caution while using this approach, since most sparse operations will treat explicit and implicit zero values identically."
       ]
     },
     {

From a2d07f37aee39d76ccb9d2ecc7cdc0ca4e2fb89d Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Fri, 6 May 2022 08:09:48 +0000
Subject: [PATCH 125/872] nbfmt

---
 site/en/guide/sparse_tensor.ipynb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/site/en/guide/sparse_tensor.ipynb b/site/en/guide/sparse_tensor.ipynb
index 37bad964320..cd38fdf55ab 100644
--- a/site/en/guide/sparse_tensor.ipynb
+++ b/site/en/guide/sparse_tensor.ipynb
@@ -680,8 +680,7 @@
   "metadata": {
     "colab": {
       "collapsed_sections": [],
-      "name": "sparse_tensor_guide.ipynb",
-      "provenance": [],
+      "name": "sparse_tensor.ipynb",
       "toc_visible": true
     },
     "kernelspec": {

From fe4ffd46804a8ea58e11bd8f22119dffcc83b789 Mon Sep 17 00:00:00 2001
From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com>
Date: Sat, 7 May 2022 23:37:21 +0530
Subject: [PATCH 126/872] Provided direct path for HyperModel classes

Modified the URLs by providing the exact direct path for `Hypermodels classes` - HyperXception and HyperResNet.
---
 site/en/tutorials/keras/keras_tuner.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/keras/keras_tuner.ipynb b/site/en/tutorials/keras/keras_tuner.ipynb
index c7a6bc1dbe7..7e20fefb869 100644
--- a/site/en/tutorials/keras/keras_tuner.ipynb
+++ b/site/en/tutorials/keras/keras_tuner.ipynb
@@ -190,7 +190,7 @@
         "* By using a model builder function\n",
         "* By subclassing the `HyperModel` class of the Keras Tuner API\n",
         "\n",
-        "You can also use two pre-defined `HyperModel` classes - [HyperXception](https://keras-team.github.io/keras-tuner/documentation/hypermodels/#hyperxception-class) and [HyperResNet](https://keras-team.github.io/keras-tuner/documentation/hypermodels/#hyperresnet-class) for computer vision applications.\n",
+        "You can also use two pre-defined [HyperModel](https://keras.io/api/keras_tuner/hypermodels/) classes - [HyperXception](https://keras.io/api/keras_tuner/hypermodels/hyper_xception/) and [HyperResNet](https://keras.io/api/keras_tuner/hypermodels/hyper_resnet/) for computer vision applications.\n",
         "\n",
         "In this tutorial, you use a model builder function to define the image classification model. The model builder function returns a compiled model and uses hyperparameters you define inline to hypertune the model."
       ]

From 98c0ff090e31ca724bb256eedac9679721d8f5d7 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 9 May 2022 13:19:22 +0100
Subject: [PATCH 127/872] Lint Build from source guide

---
 site/en/install/source.md | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/site/en/install/source.md b/site/en/install/source.md
index 7124f0b4ddf..e817491bdff 100644
--- a/site/en/install/source.md
+++ b/site/en/install/source.md
@@ -188,7 +188,7 @@ building.
 
 For compilation optimization flags, the default (`-march=native`) optimizes the
 generated code for your machine's CPU type. However, if building TensorFlow for
-a different CPU type, consider a more specific optimization flag. See the
+a different CPU type, consider a more specific optimization flag. Check the
 [GCC manual](https://gcc.gnu.org/onlinedocs/gcc-4.5.3/gcc/i386-and-x86_002d64-Options.html){:.external}
 for examples.
 
@@ -240,7 +240,8 @@ bazel build --config=v1 [--config=option] //tensorflow/tools/pip_package:build_p
 
 ### Bazel build options
 
-See the Bazel [command-line reference](https://bazel.build/reference/command-line-reference)
+Refer to the Bazel
+[command-line reference](https://bazel.build/reference/command-line-reference)
 for
 [build options](https://bazel.build/reference/command-line-reference#build-options).
 
@@ -293,17 +294,17 @@ Success: TensorFlow is now installed.
 
 TensorFlow's Docker development images are an easy way to set up an environment
 to build Linux packages from source. These images already contain the source
-code and dependencies required to build TensorFlow. See the TensorFlow
-[Docker guide](./docker.md) for installation and the
+code and dependencies required to build TensorFlow. Go to the TensorFlow
+[Docker guide](./docker.md) for installation instructions and the
 [list of available image tags](https://hub.docker.com/r/tensorflow/tensorflow/tags/){:.external}.
 
 ### CPU-only
 
 The following example uses the `:devel` image to build a CPU-only package from
-the latest TensorFlow source code. See the [Docker guide](./docker.md) for
+the latest TensorFlow source code. Check the [Docker guide](./docker.md) for
 available TensorFlow `-devel` tags.
 
-Download the latest development image and start a Docker container that we'll
+Download the latest development image and start a Docker container that you'll
 use to build the *pip* package:
 
 
@@ -368,7 +369,7 @@ On your host machine, the TensorFlow *pip* package is in the current directory
 Docker is the easiest way to build GPU support for TensorFlow since the *host*
 machine only requires the
 [NVIDIA® driver](https://github.com/NVIDIA/nvidia-docker/wiki/Frequently-Asked-Questions#how-do-i-install-the-nvidia-driver){:.external}
-(the *NVIDIA® CUDA® Toolkit* doesn't have to be installed). See the
+(the *NVIDIA® CUDA® Toolkit* doesn't have to be installed). Refer to the
 [GPU support guide](./gpu.md) and the TensorFlow [Docker guide](./docker.md) to
 set up [nvidia-docker](https://github.com/NVIDIA/nvidia-docker){:.external}
 (Linux only).

From e51ebb43b99bcf5a747ee3ed7af4e2fafa405073 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 9 May 2022 14:24:52 +0100
Subject: [PATCH 128/872] Change loss, metric, optimizer aliases to Keras in
 Time series forecasting tutorial

---
 .../structured_data/time_series.ipynb         | 22 +++++++++----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/site/en/tutorials/structured_data/time_series.ipynb b/site/en/tutorials/structured_data/time_series.ipynb
index 9cd405c3c50..1536a1d012d 100644
--- a/site/en/tutorials/structured_data/time_series.ipynb
+++ b/site/en/tutorials/structured_data/time_series.ipynb
@@ -1166,8 +1166,8 @@
       "source": [
         "baseline = Baseline(label_index=column_indices['T (degC)'])\n",
         "\n",
-        "baseline.compile(loss=tf.losses.MeanSquaredError(),\n",
-        "                 metrics=[tf.metrics.MeanAbsoluteError()])\n",
+        "baseline.compile(loss=tf.keras.losses.MeanSquaredError(),\n",
+        "                 metrics=[tf.keras.metrics.MeanAbsoluteError()])\n",
         "\n",
         "val_performance = {}\n",
         "performance = {}\n",
@@ -1323,9 +1323,9 @@
         "                                                    patience=patience,\n",
         "                                                    mode='min')\n",
         "\n",
-        "  model.compile(loss=tf.losses.MeanSquaredError(),\n",
-        "                optimizer=tf.optimizers.Adam(),\n",
-        "                metrics=[tf.metrics.MeanAbsoluteError()])\n",
+        "  model.compile(loss=tf.keras.losses.MeanSquaredError(),\n",
+        "                optimizer=tf.keras.optimizers.Adam(),\n",
+        "                metrics=[tf.keras.metrics.MeanAbsoluteError()])\n",
         "\n",
         "  history = model.fit(window.train, epochs=MAX_EPOCHS,\n",
         "                      validation_data=window.val,\n",
@@ -2020,8 +2020,8 @@
       "outputs": [],
       "source": [
         "baseline = Baseline()\n",
-        "baseline.compile(loss=tf.losses.MeanSquaredError(),\n",
-        "                 metrics=[tf.metrics.MeanAbsoluteError()])"
+        "baseline.compile(loss=tf.keras.losses.MeanSquaredError(),\n",
+        "                 metrics=[tf.keras.metrics.MeanAbsoluteError()])"
       ]
     },
     {
@@ -2343,8 +2343,8 @@
         "    return tf.tile(inputs[:, -1:, :], [1, OUT_STEPS, 1])\n",
         "\n",
         "last_baseline = MultiStepLastBaseline()\n",
-        "last_baseline.compile(loss=tf.losses.MeanSquaredError(),\n",
-        "                      metrics=[tf.metrics.MeanAbsoluteError()])\n",
+        "last_baseline.compile(loss=tf.keras.losses.MeanSquaredError(),\n",
+        "                      metrics=[tf.keras.metrics.MeanAbsoluteError()])\n",
         "\n",
         "multi_val_performance = {}\n",
         "multi_performance = {}\n",
@@ -2378,8 +2378,8 @@
         "    return inputs\n",
         "\n",
         "repeat_baseline = RepeatBaseline()\n",
-        "repeat_baseline.compile(loss=tf.losses.MeanSquaredError(),\n",
-        "                        metrics=[tf.metrics.MeanAbsoluteError()])\n",
+        "repeat_baseline.compile(loss=tf.keras.losses.MeanSquaredError(),\n",
+        "                        metrics=[tf.keras.metrics.MeanAbsoluteError()])\n",
         "\n",
         "multi_val_performance['Repeat'] = repeat_baseline.evaluate(multi_window.val)\n",
         "multi_performance['Repeat'] = repeat_baseline.evaluate(multi_window.test, verbose=0)\n",

From 520c3561175a234154c83b7a5df429c5893035d1 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 9 May 2022 15:14:29 +0100
Subject: [PATCH 129/872] Update aliases to Keras in the Save and load models
 tutorial, lint the tutorial

---
 site/en/tutorials/keras/save_and_load.ipynb | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/site/en/tutorials/keras/save_and_load.ipynb b/site/en/tutorials/keras/save_and_load.ipynb
index 4d7ea074161..3a32f6439de 100644
--- a/site/en/tutorials/keras/save_and_load.ipynb
+++ b/site/en/tutorials/keras/save_and_load.ipynb
@@ -111,7 +111,7 @@
         "\n",
         "### Options\n",
         "\n",
-        "There are different ways to save TensorFlow models depending on the API you're using. This guide uses [tf.keras](https://www.tensorflow.org/guide/keras), a high-level API to build and train models in TensorFlow. For other approaches see the TensorFlow [Save and Restore](https://www.tensorflow.org/guide/saved_model) guide or [Saving in eager](https://www.tensorflow.org/guide/eager#object-based_saving)."
+        "There are different ways to save TensorFlow models depending on the API you're using. This guide uses [tf.keras](https://www.tensorflow.org/guide/keras)—a high-level API to build and train models in TensorFlow. For other approaches, refer to the [Using the SavedModel format guide](../../guide/saved_model.ipynb) and the [Save and load Keras models guide](https://www.tensorflow.org/guide/keras/save_and_serialize)."
       ]
     },
     {
@@ -224,8 +224,8 @@
         "  ])\n",
         "\n",
         "  model.compile(optimizer='adam',\n",
-        "                loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
-        "                metrics=[tf.metrics.SparseCategoricalAccuracy()])\n",
+        "                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
+        "                metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n",
         "\n",
         "  return model\n",
         "\n",
@@ -414,7 +414,7 @@
         "id": "1zFrKTjjavWI"
       },
       "source": [
-        "Now, look at the resulting checkpoints and choose the latest one:"
+        "Now, review the resulting checkpoints and choose the latest one:"
       ]
     },
     {
@@ -446,9 +446,9 @@
         "id": "Zk2ciGbKg561"
       },
       "source": [
-        "Note: the default TensorFlow format only saves the 5 most recent checkpoints.\n",
+        "Note: The default TensorFlow format only saves the 5 most recent checkpoints.\n",
         "\n",
-        "To test, reset the model and load the latest checkpoint:"
+        "To test, reset the model, and load the latest checkpoint:"
       ]
     },
     {
@@ -533,13 +533,13 @@
       "source": [
         "## Save the entire model\n",
         "\n",
-        "Call [`model.save`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#save) to save a model's architecture, weights, and training configuration in a single file/folder. This allows you to export a model so it can be used without access to the original Python code*. Since the optimizer-state is recovered, you can resume training from exactly where you left off.\n",
+        "Call `tf.keras.Model.save` to save a model's architecture, weights, and training configuration in a single `file/folder`. This allows you to export a model so it can be used without access to the original Python code*. Since the optimizer-state is recovered, you can resume training from exactly where you left off.\n",
         "\n",
         "An entire model can be saved in two different file formats (`SavedModel` and `HDF5`). The TensorFlow `SavedModel` format is the default file format in TF2.x. However, models can be saved in `HDF5` format. More details on saving entire models in the two file formats is described below.\n",
         "\n",
         "Saving a fully-functional model is very useful—you can load them in TensorFlow.js ([Saved Model](https://www.tensorflow.org/js/tutorials/conversion/import_saved_model), [HDF5](https://www.tensorflow.org/js/tutorials/conversion/import_keras)) and then train and run them in web browsers, or convert them to run on mobile devices using TensorFlow Lite ([Saved Model](https://www.tensorflow.org/lite/convert/python_api#converting_a_savedmodel_), [HDF5](https://www.tensorflow.org/lite/convert/python_api#converting_a_keras_model_))\n",
         "\n",
-        "\\*Custom objects (e.g. subclassed models or layers) require special attention when saving and loading. See the **Saving custom objects** section below "
+        "\\*Custom objects (for example, subclassed models or layers) require special attention when saving and loading. Refer to the **Saving custom objects** section below."
       ]
     },
     {
@@ -557,7 +557,7 @@
         "id": "LtcN4VIb7JkK"
       },
       "source": [
-        "The SavedModel format is another way to serialize models. Models saved in this format can be restored using `tf.keras.models.load_model` and are compatible with TensorFlow Serving. The [SavedModel guide](https://www.tensorflow.org/guide/saved_model) goes into detail about how to serve/inspect the SavedModel. The section below illustrates the steps to save and restore the model."
+        "The SavedModel format is another way to serialize models. Models saved in this format can be restored using `tf.keras.models.load_model` and are compatible with TensorFlow Serving. The [SavedModel guide](../../guide/saved_model.ipynb) goes into detail about how to `serve/inspect` the SavedModel. The section below illustrates the steps to save and restore the model."
       ]
     },
     {
@@ -754,7 +754,7 @@
         "  * `from_config(cls, config)` uses the returned config from `get_config` to create a new object. By default, this function will use the config as initialization kwargs (`return cls(**config)`).\n",
         "2. Pass the object to the `custom_objects` argument when loading the model. The argument must be a dictionary mapping the string class name to the Python class. E.g. `tf.keras.models.load_model(path, custom_objects={'CustomLayer': CustomLayer})`\n",
         "\n",
-        "See the [Writing layers and models from scratch](https://www.tensorflow.org/guide/keras/custom_layers_and_models) tutorial for examples of custom objects and `get_config`.\n"
+        "Refer to the [Writing layers and models from scratch](https://www.tensorflow.org/guide/keras/custom_layers_and_models) tutorial for examples of custom objects and `get_config`.\n"
       ]
     }
   ],

From 6375ea2d5eb7bc5e6a884a26c4fcd41afa2c8a16 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 9 May 2022 15:33:34 +0100
Subject: [PATCH 130/872] Update the tf.keras.losses alias in the image
 segmentation tutorial, lint the tutorial

---
 site/en/tutorials/images/segmentation.ipynb | 35 +++++++++++----------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/site/en/tutorials/images/segmentation.ipynb b/site/en/tutorials/images/segmentation.ipynb
index 348d84e0c15..cd4e583aafc 100644
--- a/site/en/tutorials/images/segmentation.ipynb
+++ b/site/en/tutorials/images/segmentation.ipynb
@@ -80,7 +80,7 @@
         "\n",
         "## What is image segmentation?\n",
         "\n",
-        "In an image classification task the network assigns a label (or class) to each input image. However, suppose you want to know the shape of that object, which pixel belongs to which object, etc. In this case you will want to assign a class to each pixel of the image. This task is known as segmentation. A segmentation model returns much more detailed information about the image. Image segmentation has many applications in medical imaging, self-driving cars and satellite imaging to name a few.\n",
+        "In an image classification task, the network assigns a label (or class) to each input image. However, suppose you want to know the shape of that object, which pixel belongs to which object, etc. In this case, you need to assign a class to each pixel of the image—this task is known as segmentation. A segmentation model returns much more detailed information about the image. Image segmentation has many applications in medical imaging, self-driving cars and satellite imaging, just to name a few.\n",
         "\n",
         "This tutorial uses the [Oxford-IIIT Pet Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/) ([Parkhi et al, 2012](https://www.robots.ox.ac.uk/~vgg/publications/2012/parkhi12a/parkhi12a.pdf)). The dataset consists of images of 37 pet breeds, with 200 images per breed (~100 each in the training and test splits). Each image includes the corresponding labels, and pixel-wise masks. The masks are class-labels for each pixel. Each pixel is given one of three categories:\n",
         "\n",
@@ -155,7 +155,7 @@
         "id": "rJcVdj_U4vzf"
       },
       "source": [
-        " In addition, the image color values are normalized to the `[0,1]` range. Finally, as mentioned above the pixels in the segmentation mask are labeled either {1, 2, 3}. For the sake of convenience, subtract 1 from the segmentation mask, resulting in labels that are : {0, 1, 2}."
+        " In addition, the image color values are normalized to the `[0, 1]` range. Finally, as mentioned above the pixels in the segmentation mask are labeled either {1, 2, 3}. For the sake of convenience, subtract 1 from the segmentation mask, resulting in labels that are : {0, 1, 2}."
       ]
     },
     {
@@ -195,7 +195,7 @@
         "id": "65-qHTjX5VZh"
       },
       "source": [
-        "The dataset already contains the required training and test splits, so continue to use the same splits."
+        "The dataset already contains the required training and test splits, so continue to use the same splits:"
       ]
     },
     {
@@ -261,7 +261,7 @@
         "id": "xTIbNIBdcgL3"
       },
       "source": [
-        "Build the input pipeline, applying the Augmentation after batching the inputs."
+        "Build the input pipeline, applying the augmentation after batching the inputs:"
       ]
     },
     {
@@ -290,7 +290,7 @@
         "id": "Xa3gMAE_9qNa"
       },
       "source": [
-        "Visualize an image example and its corresponding mask from the dataset."
+        "Visualize an image example and its corresponding mask from the dataset:"
       ]
     },
     {
@@ -334,7 +334,7 @@
       },
       "source": [
         "## Define the model\n",
-        "The model being used here is a modified [U-Net](https://arxiv.org/abs/1505.04597). A U-Net consists of an encoder (downsampler) and decoder (upsampler). In-order to learn robust features and reduce the number of trainable parameters, you will use a pretrained model - MobileNetV2 - as the encoder. For the decoder, you will use the upsample block, which is already implemented in the [pix2pix](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/pix2pix/pix2pix.py) example in the TensorFlow Examples repo. (Check out the [pix2pix: Image-to-image translation with a conditional GAN](../generative/pix2pix.ipynb) tutorial in a notebook.)\n"
+        "The model being used here is a modified [U-Net](https://arxiv.org/abs/1505.04597). A U-Net consists of an encoder (downsampler) and decoder (upsampler). To learn robust features and reduce the number of trainable parameters, use a pretrained model—[MobileNetV2](https://arxiv.org/abs/1801.04381)—as the encoder. For the decoder, you will use the upsample block, which is already implemented in the [pix2pix](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/pix2pix/pix2pix.py) example in the TensorFlow Examples repo. (Check out the [pix2pix: Image-to-image translation with a conditional GAN](../generative/pix2pix.ipynb) tutorial in a notebook.)\n"
       ]
     },
     {
@@ -343,7 +343,7 @@
         "id": "W4mQle3lthit"
       },
       "source": [
-        "As mentioned, the encoder will be a pretrained MobileNetV2 model which is prepared and ready to use in `tf.keras.applications`. The encoder consists of specific outputs from intermediate layers in the model. Note that the encoder will not be trained during the training process."
+        "As mentioned, the encoder is a pretrained MobileNetV2 model. You will use the model from `tf.keras.applications`. The encoder consists of specific outputs from intermediate layers in the model. Note that the encoder will not be trained during the training process."
       ]
     },
     {
@@ -378,7 +378,7 @@
         "id": "KPw8Lzra5_T9"
       },
       "source": [
-        "The decoder/upsampler is simply a series of upsample blocks implemented in TensorFlow examples."
+        "The decoder/upsampler is simply a series of upsample blocks implemented in TensorFlow examples:"
       ]
     },
     {
@@ -448,7 +448,7 @@
         "\n",
         "Now, all that is left to do is to compile and train the model. \n",
         "\n",
-        "Since this is a multiclass classification problem, use the `tf.keras.losses.CategoricalCrossentropy` loss function with the `from_logits` argument set to `True`, since the labels are scalar integers instead of vectors of scores for each pixel of every class. \n",
+        "Since this is a multiclass classification problem, use the `tf.keras.losses.CategoricalCrossentropy` loss function with the `from_logits` argument set to `True`, since the labels are scalar integers instead of vectors of scores for each pixel of every class.\n",
         "\n",
         "When running inference, the label assigned to the pixel is the channel with the highest value. This is what the `create_mask` function is doing."
       ]
@@ -475,7 +475,7 @@
         "id": "xVMzbIZLcyEF"
       },
       "source": [
-        "Have a quick look at the resulting model architecture:"
+        "Plot the resulting model architecture:"
       ]
     },
     {
@@ -495,7 +495,7 @@
         "id": "Tc3MiEO2twLS"
       },
       "source": [
-        "Try out the model to check what it predicts before training."
+        "Try out the model to check what it predicts before training:"
       ]
     },
     {
@@ -547,7 +547,7 @@
         "id": "22AyVYWQdkgk"
       },
       "source": [
-        "The callback defined below is used to observe how the model improves while it is training."
+        "The callback defined below is used to observe how the model improves while it is training:"
       ]
     },
     {
@@ -680,7 +680,7 @@
       "source": [
         "So, in this case you need to implement the weighting yourself. You'll do this using sample weights: In addition to `(data, label)` pairs, `Model.fit` also accepts `(data, label, sample_weight)` triples.\n",
         "\n",
-        "`Model.fit` propagates the `sample_weight` to the losses and metrics, which also accept a `sample_weight` argument. The sample weight is multiplied by the sample's value before the reduction step. For example:"
+        "Keras `Model.fit` propagates the `sample_weight` to the losses and metrics, which also accept a `sample_weight` argument. The sample weight is multiplied by the sample's value before the reduction step. For example:"
       ]
     },
     {
@@ -695,8 +695,8 @@
         "prediction = [[-3., 0], [-3, 0]] \n",
         "sample_weight = [1, 10] \n",
         "\n",
-        "loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True,\n",
-        "                                               reduction=tf.losses.Reduction.NONE)\n",
+        "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True,\n",
+        "                                               reduction=tf.keras.losses.Reduction.NONE)\n",
         "loss(label, prediction, sample_weight).numpy()"
       ]
     },
@@ -706,7 +706,7 @@
         "id": "Gbwo3DZ-9TxM"
       },
       "source": [
-        "So to make sample weights for this tutorial you need a function that takes a `(data, label)` pair and returns a `(data, label, sample_weight)` triple. Where the `sample_weight` is a 1-channel image containing the class weight for each pixel. \n",
+        "So, to make sample weights for this tutorial, you need a function that takes a `(data, label)` pair and returns a `(data, label, sample_weight)` triple. Where the `sample_weight` is a 1-channel image containing the class weight for each pixel.\n",
         "\n",
         "The simplest possible implementation is to use the label as an index into a `class_weight` list:"
       ]
@@ -758,7 +758,7 @@
         "id": "Yc-EpIzaRbSL"
       },
       "source": [
-        "Now you can train a model on this weighted dataset:"
+        "Now, you can train a model on this weighted dataset:"
       ]
     },
     {
@@ -797,6 +797,7 @@
       },
       "source": [
         "## Next steps\n",
+        "\n",
         "Now that you have an understanding of what image segmentation is and how it works, you can try this tutorial out with different intermediate layer outputs, or even different pretrained models. You may also challenge yourself by trying out the [Carvana](https://www.kaggle.com/c/carvana-image-masking-challenge/overview) image masking challenge hosted on Kaggle.\n",
         "\n",
         "You may also want to see the [Tensorflow Object Detection API](https://github.com/tensorflow/models/blob/master/research/object_detection/README.md) for another model you can retrain on your own data. Pretrained models are available on [TensorFlow Hub](https://www.tensorflow.org/hub/tutorials/tf2_object_detection#optional)"

From 27821b6b275e976ef18a479997573c7bf8a62161 Mon Sep 17 00:00:00 2001
From: Kris Tonthat 
Date: Mon, 9 May 2022 13:59:30 -0700
Subject: [PATCH 131/872] Notebook documentation for
 classification_with_model_garden

PiperOrigin-RevId: 447557463
---
 site/en/tutorials/_toc.yaml                   |   3 +
 ...=> classification_with_model_garden.ipynb} | 158 ++++++++++++++----
 2 files changed, 130 insertions(+), 31 deletions(-)
 rename site/en/tutorials/images/{models_vision.ipynb => classification_with_model_garden.ipynb} (70%)

diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml
index 233316fb22b..ee408fff0a5 100644
--- a/site/en/tutorials/_toc.yaml
+++ b/site/en/tutorials/_toc.yaml
@@ -109,6 +109,9 @@ toc:
     path: /tutorials/images/data_augmentation
   - title: "Image segmentation"
     path: /tutorials/images/segmentation
+  - title: "Image classification with Model Garden"
+    path: /tutorials/images/classification_with_model_garden
+    status: new
   - title: "Object detection with TF Hub"
     path: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_object_detection.ipynb
     status: external
diff --git a/site/en/tutorials/images/models_vision.ipynb b/site/en/tutorials/images/classification_with_model_garden.ipynb
similarity index 70%
rename from site/en/tutorials/images/models_vision.ipynb
rename to site/en/tutorials/images/classification_with_model_garden.ipynb
index 03bd72d2b1b..36b94bc1a00 100644
--- a/site/en/tutorials/images/models_vision.ipynb
+++ b/site/en/tutorials/images/classification_with_model_garden.ipynb
@@ -37,7 +37,7 @@
         "id": "qFdPvlXBOdUN"
       },
       "source": [
-        "# Use TensorFlow Models: Fine tune a ResNet"
+        "# Image classification with Model Garden"
       ]
     },
     {
@@ -48,16 +48,16 @@
       "source": [
         "\n",
         "  \n",
         "  \n",
         "  \n",
         "  \n",
         "
\n",
-        "    View on TensorFlow.org\n",
+        "    View on TensorFlow.org\n",
         "  \n",
-        "    Run in Google Colab\n",
+        "    Run in Google Colab\n",
         "  \n",
-        "    View on GitHub\n",
+        "    View on GitHub\n",
         "  \n",
-        "    Download notebook\n",
+        "    Download notebook\n",
         "  
"
       ]
@@ -68,7 +68,16 @@
         "id": "Ta_nFXaVAqLD"
       },
       "source": [
-        "This tutorial uses the TensorFlow Models package to fine-tune a ResNet."
+        "This tutorial fine-tunes a Residual Network (ResNet) from the TensorFlow [Model Garden](https://github.com/tensorflow/models) package (`tensorflow-models`) to classify images in the [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html) dataset.\n",
+        "\n",
+        "Model Garden contains a collection of state-of-the-art vision models, implemented with TensorFlow's high-level APIs. The implementations demonstrate the best practices for modeling, letting users to take full advantage of TensorFlow for their research and product development.\n",
+        "\n",
+        "This tutorial uses a [ResNet](https://arxiv.org/pdf/1512.03385.pdf) model, a state-of-the-art image classifier. This tutorial uses the ResNet-18 model, a convolutional neural network with 18 layers.\n",
+        "\n",
+        "This tutorial demonstrates how to:\n",
+        "1. Use models from the TensorFlow Models package.\n",
+        "2. Fine-tune a pre-built ResNet for image classification.\n",
+        "3. Export the tuned ResNet model."
       ]
     },
     {
@@ -79,7 +88,7 @@
       "source": [
         "## Setup\n",
         "\n",
-        "Install and import the necessary modules"
+        "Install and import the necessary modules. This tutorial uses the `tf-models-nightly` version of Model Garden."
       ]
     },
     {
@@ -94,6 +103,15 @@
         "!pip install -q tf-models-nightly"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CKYMTPjOE400"
+      },
+      "source": [
+        "Import TensorFlow, TensorFlow Datasets, and a few helper libraries."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -102,7 +120,6 @@
       },
       "outputs": [],
       "source": [
-        "# Import helper libraries\n",
         "import pprint\n",
         "import tempfile\n",
         "\n",
@@ -113,6 +130,15 @@
         "import tensorflow_datasets as tfds"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AVTs0jDd1b24"
+      },
+      "source": [
+        "The `tensorflow_models` package contains the ResNet vision model, and the `official.vision.serving` model contains the function to save and export the tuned model."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -124,7 +150,7 @@
         "import tensorflow_models as tfm\n",
         "\n",
         "# Not in the tfm public API for v2.9. Will be available as `vision.serving` in v2.10\n",
-        "from official.vision.serving import export_saved_model_lib  "
+        "from official.vision.serving import export_saved_model_lib"
       ]
     },
     {
@@ -133,7 +159,7 @@
         "id": "aKv3wdqkQ8FU"
       },
       "source": [
-        "## Cifar-10 with ResNet-18 Backbone"
+        "## Configure the ResNet-18 model for the Cifar-10 dataset"
       ]
     },
     {
@@ -142,7 +168,11 @@
         "id": "5iN8mHEJjKYE"
       },
       "source": [
-        "Base the experiment on `\"resnet_imagenet\"` configuration (defined by `tfm.vision.configs.image_classification.image_classification_imagenet`)."
+        "The CIFAR10 dataset contains 60,000 color images in mutually exclusive 10 classes, with 6,000 images in each class.\n",
+        "\n",
+        "In Model Garden, the collections of parameters that define a model are called *configs*. Model Garden can create a config based on a known set of parameters via a [factory](https://en.wikipedia.org/wiki/Factory_method_pattern).\n",
+        "\n",
+        "Use the `resnet_imagenet` factory configuration, as defined by `tfm.vision.configs.image_classification.image_classification_imagenet`. The configuration is set up to train ResNet to converge on [ImageNet](https://www.image-net.org/)."
       ]
     },
     {
@@ -165,7 +195,7 @@
         "id": "U6PVwXA-j3E7"
       },
       "source": [
-        "Next adjust the configuration so that it works with `cifar10`."
+        "Adjust the model and dataset configurations so that it works with Cifar-10 (`cifar10`)."
       ]
     },
     {
@@ -176,12 +206,12 @@
       },
       "outputs": [],
       "source": [
-        "# Change model\n",
+        "# Configure model\n",
         "exp_config.task.model.num_classes = 10\n",
         "exp_config.task.model.input_size = list(ds_info.features[\"image\"].shape)\n",
         "exp_config.task.model.backbone.resnet.model_id = 18\n",
         "\n",
-        "# Change train, eval data\n",
+        "# Configure training and testing data\n",
         "batch_size = 128\n",
         "\n",
         "exp_config.task.train_data.input_path = ''\n",
@@ -201,7 +231,7 @@
         "id": "DE3ggKzzTD56"
       },
       "source": [
-        "Adjust the trainer configuration:"
+        "Adjust the trainer configuration."
       ]
     },
     {
@@ -212,8 +242,24 @@
       },
       "outputs": [],
       "source": [
-        "# Change trainer config\n",
-        "train_steps = 5000\n",
+        "logical_device_names = [logical_device.name for logical_device in tf.config.list_logical_devices()]\n",
+        "\n",
+        "if 'GPU' in ''.join(logical_device_names):\n",
+        "  print('This may be broken in Colab.')\n",
+        "  device = 'GPU'\n",
+        "elif 'TPU' in ''.join(logical_device_names):\n",
+        "  print('This may be broken in Colab.')\n",
+        "  device = 'TPU'\n",
+        "else:\n",
+        "  print('This is slow, and doesn\\'t train to convergence.')\n",
+        "  device = 'CPU'\n",
+        "\n",
+        "if device=='CPU':\n",
+        "  train_steps = 20\n",
+        "  exp_config.trainer.steps_per_loop = 5\n",
+        "else:\n",
+        "  train_steps=5000\n",
+        "  exp_config.trainer.steps_per_loop = 100\n",
         "\n",
         "exp_config.trainer.steps_per_loop = 100\n",
         "exp_config.trainer.summary_interval = 100\n",
@@ -233,7 +279,7 @@
         "id": "5mTcDnBiTOYD"
       },
       "source": [
-        "And set the runtime configuration."
+        "Print the modified configuration."
       ]
     },
     {
@@ -255,7 +301,7 @@
         "id": "w7_X0UHaRF2m"
       },
       "source": [
-        "Set up the distribution strategy:"
+        "Set up the distribution strategy."
       ]
     },
     {
@@ -288,7 +334,9 @@
         "id": "W4k5YH5pTjaK"
       },
       "source": [
-        "Create the `Task` object (ref: `tfm.core.base_task.Task`) form the `config_definitions.TaskConfig`:"
+        "Create the `Task` object (`tfm.core.base_task.Task`) from the `config_definitions.TaskConfig`.\n",
+        "\n",
+        "The `Task` object has all the methods necessary for building the dataset, building the model, and running training & evaluation. These methods are driven by `tfm.core.train_lib.run_experiment`."
       ]
     },
     {
@@ -326,7 +374,7 @@
         "id": "yrwxnGDaRU0U"
       },
       "source": [
-        "## Visualize Training Dataloader"
+        "## Visualize the training data"
       ]
     },
     {
@@ -335,8 +383,8 @@
         "id": "683c255c6c52"
       },
       "source": [
-        "The data-loader applies a z-score normalization using \n",
-        "`preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`, so the images returned by the dataset can't be directly displayed by standard tools, so rescale the minimum to 0.0 and the maximum to 1.0: "
+        "The dataloader applies a z-score normalization using \n",
+        "`preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`, so the images returned by the dataset can't be directly displayed by standard tools. The visualization code needs to rescale the data into the [0,1] range."
       ]
     },
     {
@@ -356,7 +404,7 @@
         "id": "7a8582ebde7b"
       },
       "source": [
-        "You can use the `tfds.core.DatasetInfo` (`ds_info` from earlier) to lookup the text descriptions of each class ID. "
+        "Use `ds_info` (which is an instance of `tfds.core.DatasetInfo`) to lookup the text descriptions of each class ID."
       ]
     },
     {
@@ -377,7 +425,7 @@
         "id": "8c652a6fdbcf"
       },
       "source": [
-        "Use these to disualize a batch of the data:"
+        "Visualize a batch of the data."
       ]
     },
     {
@@ -427,7 +475,16 @@
         "id": "v_A9VnL2RbXP"
       },
       "source": [
-        "## Visualize Evaluation Dataloader"
+        "## Visualize the testing data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AXovuumW_I2z"
+      },
+      "source": [
+        "Visualize a batch of images from the validation dataset."
       ]
     },
     {
@@ -449,7 +506,7 @@
         "id": "ihKJt2FHRi2N"
       },
       "source": [
-        "## Train and Evaluate"
+        "## Train and evaluate"
       ]
     },
     {
@@ -480,6 +537,15 @@
         "tf.keras.utils.plot_model(model, show_shapes=True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "L7nVfxlBA8Gb"
+      },
+      "source": [
+        "Print the `accuracy`, `top_5_accuracy`, and `validation_loss` evaluation metrics."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -492,6 +558,33 @@
         "    print(f'{key:20}: {value.numpy():.3f}')"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TDys5bZ1zsml"
+      },
+      "source": [
+        "Run a batch of the processed training data through the model, and view the results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GhI7zR-Uz1JT"
+      },
+      "outputs": [],
+      "source": [
+        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
+        "  predictions = model.predict(images)\n",
+        "  predictions = tf.argmax(predictions, axis=-1)\n",
+        "\n",
+        "show_batch(images, labels, tf.cast(predictions, tf.int32))\n",
+        "\n",
+        "if device=='CPU':\n",
+        "  plt.title('The model was only trained for a few steps, so it is not expected to do well.')"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -507,7 +600,7 @@
         "id": "9669d08c91af"
       },
       "source": [
-        "The `keras.Model` object returned by `train_lib.run_experiment` expects the data to be normalized by the dataset loader using the same mean and variance statiscics in `preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`. This export function handles those details so you can pass `tf.uint8` images and get correct result.\n"
+        "The `keras.Model` object returned by `train_lib.run_experiment` expects the data to be normalized by the dataset loader using the same mean and variance statiscics in `preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`. This export function handles those details, so you can pass `tf.uint8` images and get the correct results.\n"
       ]
     },
     {
@@ -534,7 +627,7 @@
         "id": "vVr6DxNqTyLZ"
       },
       "source": [
-        "Test the exported model"
+        "Test the exported model."
       ]
     },
     {
@@ -556,7 +649,7 @@
         "id": "GiOp2WVIUNUZ"
       },
       "source": [
-        "Visualize the predictions"
+        "Visualize the predictions."
       ]
     },
     {
@@ -573,7 +666,10 @@
         "  for image in data['image']:\n",
         "    index = tf.argmax(model_fn(image[tf.newaxis, ...])['logits'], axis=1)[0]\n",
         "    predictions.append(index)\n",
-        "  show_batch(data['image'], data['label'], predictions)"
+        "  show_batch(data['image'], data['label'], predictions)\n",
+        "\n",
+        "  if device=='CPU':\n",
+        "    plt.title('The model was only trained for a few steps, it is not expected to do well.')"
       ]
     }
   ],

From a8c319008a706f147bfa6cbe63e0a6dbc0e330da Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Tue, 10 May 2022 15:26:59 -0700
Subject: [PATCH 132/872] Fix suptitles.

PiperOrigin-RevId: 447842364
---
 .../images/classification_with_model_garden.ipynb          | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/site/en/tutorials/images/classification_with_model_garden.ipynb b/site/en/tutorials/images/classification_with_model_garden.ipynb
index 36b94bc1a00..70f18e4c894 100644
--- a/site/en/tutorials/images/classification_with_model_garden.ipynb
+++ b/site/en/tutorials/images/classification_with_model_garden.ipynb
@@ -251,7 +251,7 @@
         "  print('This may be broken in Colab.')\n",
         "  device = 'TPU'\n",
         "else:\n",
-        "  print('This is slow, and doesn\\'t train to convergence.')\n",
+        "  print('Running on CPU is slow, so only train for a few steps.')\n",
         "  device = 'CPU'\n",
         "\n",
         "if device=='CPU':\n",
@@ -261,7 +261,6 @@
         "  train_steps=5000\n",
         "  exp_config.trainer.steps_per_loop = 100\n",
         "\n",
-        "exp_config.trainer.steps_per_loop = 100\n",
         "exp_config.trainer.summary_interval = 100\n",
         "exp_config.trainer.checkpoint_interval = train_steps\n",
         "exp_config.trainer.validation_interval = 1000\n",
@@ -582,7 +581,7 @@
         "show_batch(images, labels, tf.cast(predictions, tf.int32))\n",
         "\n",
         "if device=='CPU':\n",
-        "  plt.title('The model was only trained for a few steps, so it is not expected to do well.')"
+        "  plt.suptitle('The model was only trained for a few steps, it is not expected to do well.')"
       ]
     },
     {
@@ -669,7 +668,7 @@
         "  show_batch(data['image'], data['label'], predictions)\n",
         "\n",
         "  if device=='CPU':\n",
-        "    plt.title('The model was only trained for a few steps, it is not expected to do well.')"
+        "    plt.suptitle('The model was only trained for a few steps, it is not expected to do well.')"
       ]
     }
   ],

From 97019f03550fb19d91fce4367f1043b462abff03 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 11 May 2022 14:30:53 -0700
Subject: [PATCH 133/872] test

PiperOrigin-RevId: 448085748
---
 site/en/tutorials/images/classification_with_model_garden.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/images/classification_with_model_garden.ipynb b/site/en/tutorials/images/classification_with_model_garden.ipynb
index 70f18e4c894..9e23168d04a 100644
--- a/site/en/tutorials/images/classification_with_model_garden.ipynb
+++ b/site/en/tutorials/images/classification_with_model_garden.ipynb
@@ -1,5 +1,5 @@
 {
-  "cells": [
+  "cells": [ 
     {
       "cell_type": "markdown",
       "metadata": {

From fb0365faf31342cd90e2bacfad54f3522877decb Mon Sep 17 00:00:00 2001
From: Joe Fernandez 
Date: Fri, 13 May 2022 09:36:42 -0700
Subject: [PATCH 134/872] Add an overview page for Model Garden

PiperOrigin-RevId: 448514167
---
 site/en/guide/_toc.yaml             |   4 +
 site/en/guide/model_garden/index.md | 135 ++++++++++++++++++++++++++++
 2 files changed, 139 insertions(+)
 create mode 100644 site/en/guide/model_garden/index.md

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index 3cf9913c115..1665073dc7c 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -82,6 +82,10 @@ toc:
 - title: "Mixed precision"
   path: /guide/mixed_precision
 
+- heading: "Model Garden"
+- title: "Overview"
+  path: /guide/model_garden
+  status: new
 - heading: "Estimators"
 - title: "Estimator overview"
   path: /guide/estimator
diff --git a/site/en/guide/model_garden/index.md b/site/en/guide/model_garden/index.md
new file mode 100644
index 00000000000..c08ec950a95
--- /dev/null
+++ b/site/en/guide/model_garden/index.md
@@ -0,0 +1,135 @@
+# Model Garden overview
+
+The TensorFlow Model Garden provides implementations of many state-of-the-art
+machine learning (ML) models for vision and natural language processing (NLP),
+as well as workflow tools to let you quickly configure and run those models on
+standard datasets. Whether you are looking to benchmark performance for a
+well-known model, verify the results of recently released research, or extend
+existing models, the Model Garden can help you drive your ML research and
+applications forward.
+
+The Model Garden includes the following resources for machine learning
+developers:
+
+-   [**Official models**](#official) for vision and NLP, maintained by Google
+    engineers
+-   [**Research models**](#research) published as part of ML research papers
+-   [**Training experiment framework**](#training_framework) for fast,
+    declarative training configuration of official models
+-   [**Specialized ML operations**](#ops) for vision and natural language
+    processing (NLP)
+-   [**Model training loop**](#orbit) management with Orbit
+
+These resources are built to be used with the TensorFlow Core framework and
+integrate with your existing TensorFlow development projects. Model
+Garden resources are also provided under an [open
+source](https://github.com/tensorflow/models/blob/master/LICENSE) license, so
+you can freely extend and distribute the models and tools.
+
+Practical ML models are computationally intensive to train and run, and may
+require accelerators such as Graphical Processing Units (GPUs) and Tensor
+Processing Units (TPUs). Most of the models in Model Garden were trained on
+large datasets using TPUs. However, you can also train and run these models on
+GPU and CPU processors.
+
+## Model Garden models
+
+The machine learning models in the Model Garden include full code so you can
+test, train, or re-train them for research and experimentation. The Model Garden
+includes two primary categories of models: *official models* and *research
+models*.
+
+### Official models {:#official}
+
+The [Official Models](https://github.com/tensorflow/models/tree/master/official)
+repository is a collection of state-of-the-art models, with a focus on
+vision and natural language processing (NLP).
+These models are implemented using current TensorFlow 2.x high-level
+APIs. Model libraries in this repository are optimized for fast performance and
+actively maintained by Google engineers. The official models include additional
+metadata you can use to quickly configure experiments using the Model Garden
+[training experiment framework](#training_framework).
+
+### Research models {:#research}
+
+The [Research Models](https://github.com/tensorflow/models/tree/master/research)
+repository is a collection of models published as code resources for research
+papers. These models are implemented using both TensorFlow 1.x and 2.x. Model
+libraries in the research folder are supported by the code owners and the
+research community.
+
+## Training experiment framework {:#training_framework}
+
+The Model Garden training experiment framework lets you quickly assemble and
+run training experiments using its official models and standard datasets. The
+training framework uses additional metadata included with the Model Garden's
+official models to allow you to configure models quickly using a declarative
+programming model. You can define a training experiment using Python commands in
+the [TensorFlow Model library](../../api_docs/python/tfm/core)
+or configure training using a YAML configuration file, like this
+[example](https://github.com/tensorflow/models/blob/master/official/vision/configs/experiments/image_classification/imagenet_resnet50_tpu.yaml).
+
+The training framework uses
+[`tfm.core.base_trainer.ExperimentConfig`](../../api_docs/python/tfm/core/base_trainer/ExperimentConfig)
+as the configuration object, which contains the following top-level
+configuration objects:
+
+-   [`runtime`](https://www.tensorflow.org/api_docs/python/tfm/core/base_task/RuntimeConfig):
+    Defines the processing hardware, distribution strategy, and other
+    performance optimizations
+-   [`task`](https://www.tensorflow.org/api_docs/python/tfm/core/config_definitions/TaskConfig):
+    Defines the model, training data, losses, and initialization
+-   [`trainer`](https://www.tensorflow.org/api_docs/python/tfm/core/base_trainer/TrainerConfig):
+    Defines the optimizer, training loops, evaluation loops, summaries, and
+    checkpoints
+
+For a complete example using the Model Garden training experiment framework,
+see the
+[Image classification with Model Garden](../../tutorials/images/classification_with_model_garden)
+tutorial. For information on the training experiment framework, check out the
+[TensorFlow Models API documentation](../../api_docs/python/tfm/core).
+If you are looking for a solution to manage training loops for your model
+training experiments, check out [Orbit](#orbit).
+
+## Specialized ML operations {:#ops}
+
+The Model Garden contains many vision and NLP operations specifically designed
+to execute state-of-the-art models that run efficiently on GPUs and TPUs. Review
+the TensorFlow Models Vision library API docs for a list of specialized [vision
+operations](../../api_docs/python/tfm/vision). Review the
+TensorFlow Models NLP Library API docs for a list of [NLP
+operations](../../api_docs/python/tfm/nlp). These libraries
+also include additional utility functions used for vision and NLP data
+processing, training, and model execution.
+
+## Training loops with Orbit {:#orbit}
+
+The Orbit tool is a flexible, lightweight library designed to make it easier to
+write custom training loops in TensorFlow 2.x, and works well with the Model
+Garden [training experiment framework](#training_framework). Orbit handles
+common model training tasks such as saving checkpoints, running model
+evaluations, and setting up summary writing. It seamlessly integrates with
+`tf.distribute` and supports running on different device types, including CPU,
+GPU, and TPU hardware. The Orbit tool is also [open
+source](https://github.com/tensorflow/models/blob/master/orbit/LICENSE), so you
+can extend and adapt to your model training needs.
+
+You generally train TensorFlow models by writing a
+[custom training loop](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch),
+or using the high-level Keras
+[Model.fit](../../api_docs/python/tf/keras/Model#fit)
+function. For simple models, you can define and manage a custom training loop
+with low-level TensorFlow methods such as `tf.GradientTape` or `tf.function`.
+Alternatively, you can use the high-level Keras `Model.fit`.
+
+However, if your model is complex and your training loop requires more flexible
+control or customization, then you should use Orbit. You can define most of your
+training loop by extending Orbit's `AbstractTrainerclass`. Learn more about the
+Orbit tool in the [Orbit API documentation](../../api_docs/python/orbit).
+
+Note: You can use the Keras API to do what Orbit does, but you must override
+the TensorFlow `train_step` function or use callbacks like ModelCheckpoint or
+TensorBoard. For more information about modifying the behavior of `train_step`,
+check out the
+[Customize what happens in Model.fit](https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit)
+page.

From 19e52253a4dd710a976f7a4ca114303ea19a2a80 Mon Sep 17 00:00:00 2001
From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com>
Date: Sat, 14 May 2022 00:55:39 +0530
Subject: [PATCH 135/872] Fixed some typo errors along and one link update

Fixed typo error by removing additional 's' in the name `tf.tensor`s to `tf.tensor`. Also updated the correct link for `tf.data.dataset` API.
---
 site/en/tutorials/customization/basics.ipynb | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb
index de651d2ac50..da317fa9c41 100644
--- a/site/en/tutorials/customization/basics.ipynb
+++ b/site/en/tutorials/customization/basics.ipynb
@@ -106,7 +106,7 @@
       "source": [
         "## Tensors\n",
         "\n",
-        "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.linalg.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types. For example:\n"
+        "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor` can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.linalg.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`. These operations automatically convert built-in Python types. For example:\n"
       ]
     },
     {
@@ -156,7 +156,7 @@
         "id": "eBPw8e8vrsom"
       },
       "source": [
-        "The most obvious differences between NumPy arrays and `tf.Tensor`s are:\n",
+        "The most obvious differences between NumPy arrays and `tf.Tensor` are:\n",
         "\n",
         "1. Tensors can be backed by accelerator memory (like GPU, TPU).\n",
         "2. Tensors are immutable."
@@ -170,7 +170,7 @@
       "source": [
         "### NumPy compatibility\n",
         "\n",
-        "Converting between a TensorFlow `tf.Tensor`s and a NumPy `ndarray` is easy:\n",
+        "Converting between a TensorFlow `tf.Tensor` and a NumPy `ndarray` is easy:\n",
         "\n",
         "* TensorFlow operations automatically convert NumPy ndarrays to Tensors.\n",
         "* NumPy operations automatically convert Tensors to NumPy ndarrays.\n",
@@ -298,7 +298,7 @@
       "source": [
         "## Datasets\n",
         "\n",
-        "This section uses the [`tf.data.Dataset` API](../../guide/data.ipynb) to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops."
+        "This section uses the [`tf.data.Dataset` API](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops."
       ]
     },
     {

From 68b572f4e7357e626f969d95d00a1f2dc43b9244 Mon Sep 17 00:00:00 2001
From: gowthamkpr <47574994+gowthamkpr@users.noreply.github.com>
Date: Fri, 13 May 2022 12:50:14 -0700
Subject: [PATCH 136/872] Remove Broken link

Removed a link that was broken
---
 site/en/guide/graph_optimization.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/graph_optimization.ipynb b/site/en/guide/graph_optimization.ipynb
index 50eedda621c..51de1efda93 100644
--- a/site/en/guide/graph_optimization.ipynb
+++ b/site/en/guide/graph_optimization.ipynb
@@ -166,7 +166,7 @@
       "source": [
         "## Compare execution performance with and without Grappler\n",
         "\n",
-        "TensorFlow 2 and beyond executes [eagerly](../eager.md) by default. Use `tf.function` to switch the default execution to Graph mode. Grappler runs automatically in the background to apply the graph optimizations above and improve execution performance. \n"
+        "TensorFlow 2 and beyond executes eagerly by default. Use `tf.function` to switch the default execution to Graph mode. Grappler runs automatically in the background to apply the graph optimizations above and improve execution performance. \n"
       ]
     },
     {

From bd31a3a1f7417568667a07c86d36a3d8035db3dd Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 16 May 2022 14:16:42 +0100
Subject: [PATCH 137/872] Update tf.data guide link in Customization basics
 tutorial.ipynb

---
 site/en/tutorials/customization/basics.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb
index da317fa9c41..e864cb534d3 100644
--- a/site/en/tutorials/customization/basics.ipynb
+++ b/site/en/tutorials/customization/basics.ipynb
@@ -298,7 +298,7 @@
       "source": [
         "## Datasets\n",
         "\n",
-        "This section uses the [`tf.data.Dataset` API](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops."
+       "This section uses the `tf.data.Dataset` API to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops. (Refer to the [tf.data: Build TensorFlow input pipelines](../../guide/data.ipynb) guide to learn more.)"
       ]
     },
     {

From 71017525529e89acdf446d81c2bf22c39acaade2 Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Mon, 16 May 2022 13:17:18 +0000
Subject: [PATCH 138/872] nbfmt

---
 site/en/tutorials/customization/basics.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb
index e864cb534d3..304debdedb0 100644
--- a/site/en/tutorials/customization/basics.ipynb
+++ b/site/en/tutorials/customization/basics.ipynb
@@ -298,7 +298,7 @@
       "source": [
         "## Datasets\n",
         "\n",
-       "This section uses the `tf.data.Dataset` API to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops. (Refer to the [tf.data: Build TensorFlow input pipelines](../../guide/data.ipynb) guide to learn more.)"
+        "This section uses the `tf.data.Dataset` API to build a pipeline for feeding data to your model. `tf.data.Dataset` is used to build performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops. (Refer to the [tf.data: Build TensorFlow input pipelines](../../guide/data.ipynb) guide to learn more.)"
       ]
     },
     {

From 6cc2e627ebe212ad2d19604ec4e6e5588221aef0 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 16 May 2022 14:20:55 +0100
Subject: [PATCH 139/872] Update tf.Tensors in Customization basics tutorial

---
 site/en/tutorials/customization/basics.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/customization/basics.ipynb b/site/en/tutorials/customization/basics.ipynb
index 304debdedb0..2df0840ad5e 100644
--- a/site/en/tutorials/customization/basics.ipynb
+++ b/site/en/tutorials/customization/basics.ipynb
@@ -106,7 +106,7 @@
       "source": [
         "## Tensors\n",
         "\n",
-        "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor` can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.linalg.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`. These operations automatically convert built-in Python types. For example:\n"
+        "A Tensor is a multi-dimensional array. Similar to NumPy `ndarray` objects, `tf.Tensor` objects have a data type and a shape. Additionally, `tf.Tensor`s can reside in accelerator memory (like a GPU). TensorFlow offers a rich library of operations (for example, `tf.math.add`, `tf.linalg.matmul`, and `tf.linalg.inv`) that consume and produce `tf.Tensor`s. These operations automatically convert built-in Python types. For example:\n"
       ]
     },
     {
@@ -156,7 +156,7 @@
         "id": "eBPw8e8vrsom"
       },
       "source": [
-        "The most obvious differences between NumPy arrays and `tf.Tensor` are:\n",
+        "The most obvious differences between NumPy arrays and `tf.Tensor`s are:\n",
         "\n",
         "1. Tensors can be backed by accelerator memory (like GPU, TPU).\n",
         "2. Tensors are immutable."

From 651037811d536e5755323c778d94dd13cdf5775b Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Mon, 16 May 2022 20:14:48 -0700
Subject: [PATCH 140/872] Update the Tensorflow document wheels after 2.9.0
 release

PiperOrigin-RevId: 449117632
---
 site/en/install/pip.html          | 40 +++++++++++++++----------------
 site/en/install/source.md         |  3 +++
 site/en/install/source_windows.md |  2 ++
 3 files changed, 25 insertions(+), 20 deletions(-)

diff --git a/site/en/install/pip.html b/site/en/install/pip.html
index 66a8874d9f1..bbb670b0af8 100644
--- a/site/en/install/pip.html
+++ b/site/en/install/pip.html
@@ -320,87 +320,87 @@ 
Package location

   Linux
   
     Python 3.7 GPU support
-    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.8.0-cp37-cp37m-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-manylinux2014.whl
   
   
     Python 3.7 CPU-only
-    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.8.0-cp37-cp37m-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-manylinux2014.whl
   
   
     Python 3.8 GPU support
-    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.8.0-cp38-cp38-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp38-cp38-manylinux2014.whl
   
   
     Python 3.8 CPU-only
-    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.8.0-cp38-cp38-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp38-cp38-manylinux2014.whl
   
   
     Python 3.9 GPU support
-    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.8.0-cp39-cp39-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp39-cp39-manylinux2014.whl
   
   
     Python 3.9 CPU-only
-    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.8.0-cp39-cp39-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp39-cp39-manylinux2014.whl
   
   
     Python 3.10 GPU support
-    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.8.0-cp310-cp310-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp310-cp310-manylinux2014.whl
   
   
     Python 3.10 CPU-only
-    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.8.0-cp310-cp310-manylinux2010_x86_64.whl
+    https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp310-cp310-manylinux2014.whl
   
 
   macOS (CPU-only)
   
     Python 3.7
-    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp37-cp37m-macosx_10_14_x86_64.whl
+    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp37-cp37m-macosx_10_14_x86_64.whl
   
   
     Python 3.8
-    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp38-cp38-macosx_10_14_x86_64.whl
+    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp38-cp38-macosx_10_14_x86_64.whl
   
   
     Python 3.9
-    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp39-cp39-macosx_10_14_x86_64.whl
+    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp39-cp39-macosx_10_14_x86_64.whl
   
    
     Python 3.10
-    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.8.0-cp310-cp310-macosx_10_14_x86_64.whl
+    https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp310-cp310-macosx_10_14_x86_64.whl
   
 
   Windows
   
     Python 3.7 GPU support
-    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.8.0-cp37-cp37m-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-win_amd64.whl
   
   
     Python 3.7 CPU-only
-    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.8.0-cp37-cp37m-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-win_amd64.whl
   
   
     Python 3.8 GPU support
-    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.8.0-cp38-cp38-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp38-cp38-win_amd64.whl
   
   
     Python 3.8 CPU-only
-    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.8.0-cp38-cp38-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp38-cp38-win_amd64.whl
   
   
     Python 3.9 GPU support
-    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.8.0-cp39-cp39-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp39-cp39-win_amd64.whl
   
   
     Python 3.9 CPU-only
-    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.8.0-cp39-cp39-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp39-cp39-win_amd64.whl
   
     
     Python 3.10 GPU support
-    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.8.0-cp310-cp310-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp310-cp310-win_amd64.whl
   
   
     Python 3.10 CPU-only
-    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.8.0-cp310-cp310-win_amd64.whl
+    https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp310-cp310-win_amd64.whl
   
 
 
diff --git a/site/en/install/source.md b/site/en/install/source.md
index e817491bdff..88d8cc5221d 100644
--- a/site/en/install/source.md
+++ b/site/en/install/source.md
@@ -420,6 +420,7 @@ Success: TensorFlow is now installed.
 
 
+
@@ -451,6 +452,7 @@ Success: TensorFlow is now installed.
 
 

 
VersionPython versionCompilerBuild tools
tensorflow-2.9.03.7-3.10GCC 7.3.1Bazel 5.0.0

 
tensorflow-2.8.03.7-3.10GCC 7.3.1Bazel 4.2.1

 
tensorflow-2.7.03.7-3.9GCC 7.3.1Bazel 3.7.2

 
tensorflow-2.6.03.6-3.9GCC 7.3.1Bazel 3.7.2

+
@@ -484,6 +486,7 @@ Success: TensorFlow is now installed.
 
 

 
VersionPython versionCompilerBuild toolscuDNNCUDA
tensorflow-2.9.03.7-3.10GCC 7.3.1Bazel 5.0.08.111.2

 
tensorflow-2.8.03.7-3.10GCC 7.3.1Bazel 4.2.18.111.2

 
tensorflow-2.7.03.7-3.9GCC 7.3.1Bazel 3.7.28.111.2

 
tensorflow-2.6.03.6-3.9GCC 7.3.1Bazel 3.7.28.111.2

+
diff --git a/site/en/install/source_windows.md b/site/en/install/source_windows.md
index bdded635a33..7d999f54594 100644
--- a/site/en/install/source_windows.md
+++ b/site/en/install/source_windows.md
@@ -284,6 +284,7 @@ For GPU support, add the CUDA and cuDNN bin directories to your `$PATH`:
 
 

 
VersionPython versionCompilerBuild tools
tensorflow-2.9.03.7-3.10Clang from xcode 10.14Bazel 5.0.0

 
tensorflow-2.8.03.7-3.10Clang from xcode 10.14Bazel 4.2.1

 
tensorflow-2.7.03.7-3.9Clang from xcode 10.11Bazel 3.7.2

 
tensorflow-2.6.03.6-3.9Clang from xcode 10.11Bazel 3.7.2

+
@@ -315,6 +316,7 @@ For GPU support, add the CUDA and cuDNN bin directories to your `$PATH`:
 
 

 
VersionPython versionCompilerBuild tools
tensorflow-2.9.03.7-3.10MSVC 2019Bazel 5.0.0

 
tensorflow-2.8.03.7-3.10MSVC 2019Bazel 4.2.1

 
tensorflow-2.7.03.7-3.9MSVC 2019Bazel 3.7.2

 
tensorflow-2.6.03.6-3.9MSVC 2019Bazel 3.7.2

+

From be62ee25b6c9934fe648ff0537b3ed6ef2cc860f Mon Sep 17 00:00:00 2001
From: Haifeng Jin 
Date: Tue, 17 May 2022 10:16:52 -0700
Subject: [PATCH 141/872] Update the installation guide.

PiperOrigin-RevId: 449252327
---
 site/en/install/_index.yaml       |   4 +-
 site/en/install/_toc.yaml         |   2 -
 site/en/install/gpu.md            | 176 ----------
 site/en/install/pip.html          | 409 ------------------------
 site/en/install/pip.md            | 514 ++++++++++++++++++++++++++++++
 site/en/install/source.md         |   6 +-
 site/en/install/source_windows.md |   2 +-
 7 files changed, 519 insertions(+), 594 deletions(-)
 delete mode 100644 site/en/install/gpu.md
 delete mode 100644 site/en/install/pip.html
 create mode 100644 site/en/install/pip.md

diff --git a/site/en/install/_index.yaml b/site/en/install/_index.yaml
index 906537fc062..50a875b4006 100644
--- a/site/en/install/_index.yaml
+++ b/site/en/install/_index.yaml
@@ -41,7 +41,6 @@ landing_page:
         
Install TensorFlow with Python's pip package manager.

         
         
Official packages available for Ubuntu, Windows, and macOS.

-        
See the GPU guide for CUDA®-enabled cards.

 
       buttons:
       - label: Read the pip install guide
@@ -67,8 +66,7 @@ landing_page:
           The TensorFlow
           Docker images are already configured to run TensorFlow. A
           Docker container runs in a
-          virtual environment and is the easiest way to set up GPU
-          support.
+          virtual environment and is the easiest way to set up GPU support.
         

         
         docker pull tensorflow/tensorflow:latest  # Download latest stable image

diff --git a/site/en/install/_toc.yaml b/site/en/install/_toc.yaml
index c8f60bde852..26cdb270bb8 100644
--- a/site/en/install/_toc.yaml
+++ b/site/en/install/_toc.yaml
@@ -7,8 +7,6 @@ toc:
 - title: Docker
   path: /install/docker
 - heading: Additional setup
-- title: GPU support
-  path: /install/gpu
 - title: GPU device plugins
   path: /install/gpu_plugins
 - title: Problems
diff --git a/site/en/install/gpu.md b/site/en/install/gpu.md
deleted file mode 100644
index 9662cb6f5b4..00000000000
--- a/site/en/install/gpu.md
+++ /dev/null
@@ -1,176 +0,0 @@
-# GPU support
-
-Note: GPU support is available for Ubuntu and Windows with CUDA®-enabled cards.
-
-TensorFlow GPU support requires an assortment of drivers and libraries. To
-simplify installation and avoid library conflicts, we recommend using a
-[TensorFlow Docker image with GPU support](./docker.md) (Linux only). This setup
-only requires the [NVIDIA® GPU drivers](https://www.nvidia.com/drivers){:.external}.
-
-These install instructions are for the latest release of TensorFlow. See the
-[tested build configurations](./source.md#gpu) for CUDA® and cuDNN versions to
-use with older TensorFlow releases.
-
-## Pip package
-
-See the [pip install guide](./pip) for available packages, systems requirements,
-and instructions. The TensorFlow `pip` package includes GPU support for
-CUDA®-enabled cards:
-
-
-pip install tensorflow
-

-
-This guide covers GPU support and installation steps for the latest *stable*
-TensorFlow release.
-
-### Older versions of TensorFlow
-
-For releases 1.15 and older, CPU and GPU packages are separate:
-
-
-pip install tensorflow==1.15      # CPU
-pip install tensorflow-gpu==1.15  # GPU
-

-
-## Hardware requirements
-
-The following GPU-enabled devices are supported:
-
-*   NVIDIA® GPU card with CUDA® architectures 3.5, 5.0, 6.0, 7.0, 7.5, 8.0 and
-    higher than 8.0. See the list of
-    CUDA®-enabled
-    GPU cards.
-*   For GPUs with unsupported CUDA® architectures, or to avoid JIT compilation
-    from PTX, or to use different versions of the NVIDIA® libraries, see the
-    [Linux build from source](./source.md) guide.
-*   Packages do not contain PTX code except for the latest supported CUDA®
-    architecture; therefore, TensorFlow fails to load on older GPUs when
-    `CUDA_FORCE_PTX_JIT=1` is set. (See
-    Application
-    Compatibility for details.)
-
-Note: The error message "Status: device kernel image is invalid" indicates that
-the TensorFlow package does not contain PTX for your architecture. You can
-enable compute capabilities by [building TensorFlow from source](./source.md).
-
-## Software requirements
-
-The following NVIDIA® software must be installed on your system:
-
-*   [NVIDIA® GPU drivers](https://www.nvidia.com/drivers){:.external} —CUDA®
-    11.2 requires 450.80.02 or higher.
-*   [CUDA® Toolkit](https://developer.nvidia.com/cuda-toolkit-archive){:.external}
-    —TensorFlow supports CUDA® 11.2 (TensorFlow >= 2.5.0)
-*   [CUPTI](http://docs.nvidia.com/cuda/cupti/){:.external} ships with the CUDA®
-    Toolkit.
-*   [cuDNN SDK 8.1.0](https://developer.nvidia.com/cudnn){:.external}
-    [cuDNN versions](https://developer.nvidia.com/rdp/cudnn-archive){:.external}).
-*   *(Optional)*
-    [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/archives/index.html#trt_7){:.external}
-    to improve latency and throughput for inference on some models.
-
-## Linux setup
-
-The `apt` instructions below are the easiest way to install the required NVIDIA
-software on Ubuntu. However, if [building TensorFlow from source](./source.md),
-manually install the software requirements listed above, and consider using a
-`-devel` [TensorFlow Docker image](./docker.md) as a base.
-
-Install [CUPTI](http://docs.nvidia.com/cuda/cupti/){:.external} which ships with
-the CUDA® Toolkit. Append its installation directory to the `$LD_LIBRARY_PATH`
-environmental variable:
-
-
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64
-

-
-### Install CUDA with apt
-
-This section shows how to install CUDA® 11 (TensorFlow >= 2.4.0) on Ubuntu
-16.04 and 18.04. These instructions may work for other Debian-based distros.
-
-Caution: [Secure Boot](https://wiki.ubuntu.com/UEFI/SecureBoot){:.external}
-complicates installation of the NVIDIA driver and is beyond the scope of these instructions.
-
-#### Ubuntu 18.04 (CUDA 11.2)
-
-
-# Add NVIDIA package repositories
-# Note: For the Ubuntu version other than 18.04 or CPU architecture other than x86,
-# replace `ubuntu1804` and/or `x86_64` as needed in the following URL.
-wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
-sudo dpkg -i cuda-keyring_1.0-1_all.deb
-sudo apt-get update
-
-# Install development and runtime libraries (~4GB)
-# libnvinfer packages are optional, needed to support TensorRT inference.
-sudo apt-get install --no-install-recommends \
-    cuda-11-2 \
-    libcudnn8=8.1.0.77-1+cuda11.2  \
-    libcudnn8-dev=8.1.0.77-1+cuda11.2 \
-    libnvinfer8=8.2.4-1+cuda11.4 \
-    libnvinfer-dev=8.2.4-1+cuda11.4 \
-    libnvinfer-plugin8=8.2.4-1+cuda11.4 \
-    libnvinfer-plugin-dev=8.2.4-1+cuda11.4
-
-
-# Reboot. Check that GPUs are visible using the command: nvidia-smi
-

-
-#### Ubuntu 16.04 (CUDA 11.2)
-
-
-# Add NVIDIA package repositories
-sudo apt-get update
-sudo apt-get install -y apt-transport-https
-wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-keyring_1.0-1_all.deb
-sudo dpkg -i cuda-keyring_1.0-1_all.deb
-sudo apt-get update
-
-# Install development and runtime libraries (~4GB)
-# libnvinfer packages are optional, needed to support TensorRT inference.
-sudo apt-get install --no-install-recommends \
-    cuda-11-2 \
-    libcudnn8=8.1.0.77-1+cuda11.2  \
-    libcudnn8-dev=8.1.0.77-1+cuda11.2 \
-    libnvinfer8=8.0.1-1+cuda11.3 \
-    libnvinfer-dev=8.0.1-1+cuda11.3 \
-    libnvinfer-plugin8=8.0.1-1+cuda11.3 \
-    libnvinfer-plugin-dev=8.0.1-1+cuda11.3
-
-
-# Reboot. Check that GPUs are visible using the command: nvidia-smi
-

-
-
-## Windows setup
-
-See the [hardware requirements](#hardware_requirements) and
-[software requirements](#software_requirements) listed above. Read the
-[CUDA® install guide for Windows](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/){:.external}.
-
-Make sure the installed NVIDIA software packages match the versions listed above. In
-particular, TensorFlow will not load without the `cuDNN64_8.dll` file. To use a
-different version, see the [Windows build from source](./source_windows.md) guide.
-
-Add the CUDA®, CUPTI, and cuDNN installation directories to the `%PATH%`
-environmental variable. For example, if the CUDA® Toolkit is installed to
-`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0` and cuDNN to
-`C:\tools\cuda`, update your `%PATH%` to match:
-
-
-SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;%PATH%
-SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\extras\CUPTI\lib64;%PATH%
-SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include;%PATH%
-SET PATH=C:\tools\cuda\bin;%PATH%
-

-
-## WSL2 setup
-
-Experimental support for WSL2 on Windows 10 19044 or higher with GPU access is now available. This corresponds to the most recent update of Windows 10 (aka version 21H2/November 2021 Update). You can get the latest update from here: [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10).
-
-For instructions, please see [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html) for CUDA in WSL.
-
-
-
diff --git a/site/en/install/pip.html b/site/en/install/pip.html
deleted file mode 100644
index bbb670b0af8..00000000000
--- a/site/en/install/pip.html
+++ /dev/null
@@ -1,409 +0,0 @@
-
-  
-    Install TensorFlow with pip
-    
-    
-    
-  
-  
-
-
TensorFlow 2 packages are available

-
    
-  
  • tensorflow —Latest stable release with CPU and GPU support (Ubuntu and Windows)
    • 
-  
  • tf-nightly —Preview build (unstable). Ubuntu and Windows include GPU support. 
    • 
-

-
-
-
Older versions of TensorFlow

-
-
For TensorFlow 1.x, CPU and GPU packages are separate:

-
-
    
-  
  • tensorflow==1.15 —Release for CPU-only
    • 
-  
  • tensorflow-gpu==1.15 —Release with GPU support (Ubuntu and Windows)
    • 
-

-
-
-
System requirements

-
    
-  
  • Python 3.7–3.10
-    
      
-      
    • Python 3.10 support requires TensorFlow 2.8 or later.
      • 
-      
    • Python 3.9 support requires TensorFlow 2.5 or later.
      • 
-      
    • Python 3.8 support requires TensorFlow 2.2 or later.
      • 
-    
    
-  
    • 
-  
  • pip 19.0 or later (requires manylinux2010 support)
    • 
-  
  • Ubuntu 16.04 or later (64-bit)
    • 
-  
  • macOS 10.12.6 (Sierra) or later (64-bit) (no GPU support)
-    
      
-      
    • macOS requires pip 20.3 or later
      • 
-    
    
-  
    • 
-  
  • Windows 7 or later (64-bit)
-    
-  
    • 
-  
  • GPU support requires a CUDA®-enabled card (Ubuntu and Windows)
    • 
-

-
-
-
-
Hardware requirements

-
    
-  
  • Starting with TensorFlow 1.6, binaries use AVX instructions which may not run on older CPUs.
    • 
-  
  • Read the GPU support guide to set up a CUDA®-enabled GPU card on Ubuntu or Windows.
    • 
-

-
-
-
1. Install the Python development environment on your system

-
-

-  Check if your Python environment is already configured:
-

-
-
-
-
-python3 --version
-pip3 --version
-

-
-

-  If these packages are already installed, skip to the next step.

-  Otherwise, install Python, the
-  pip package manager,
-  and venv:
-

-
-

-

-
Ubuntu

-
-sudo apt update
-sudo apt install python3-dev python3-pip python3-venv
-

-

-
-

-
macOS

-
-
Install using the Homebrew package manager:

-
-/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
-export PATH="/usr/local/opt/python/libexec/bin:$PATH"
-# if you are on macOS 10.12 (Sierra) use `export PATH="/usr/local/bin:/usr/local/sbin:$PATH"`
-brew update
-brew install python  # Python 3
-

-

-
-

-
Windows

-

-  Install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017,
-  and 2019. Starting with the TensorFlow 2.1.0 version, the msvcp140_1.dll
-  file is required from this package (which may not be provided from older redistributable packages).
-  The redistributable comes with Visual Studio 2019 but can be installed separately:
-

-
    
-
  1. Go to the Microsoft Visual C++ downloads,
    2. 
-
  2. Scroll down the page to the Visual Studio 2015, 2017 and 2019 section.
    3. 
-
  3. Download and install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019 for your platform.
    4. 
-

-
Make sure long paths are enabled on Windows.

-
Install the 64-bit Python 3 release for Windows (select pip as an optional feature).

-

-
-

-
Other

-
-curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
-python get-pip.py
-

-

-

-
-
-
-
-
2. Create a virtual environment (recommended)

-
-

-  Python virtual environments are used to isolate package installation from the system.
-

-
-

-

-
Ubuntu

-

-  Create a new virtual environment by choosing a Python interpreter and making a
-  ./venv directory to hold it:
-

-
python3 -m venv --system-site-packages ./venv

-

-  Activate the virtual environment using a shell-specific command:
-

-
source ./venv/bin/activate  # sh, bash, or zsh

-
. ./venv/bin/activate.fish  # fish

-
source ./venv/bin/activate.csh  # csh or tcsh

-
-

-  When the virtual environment is active, your shell prompt is prefixed with (venv).
-

-

-  Install packages within a virtual environment without affecting the host system
-  setup. Start by upgrading pip:
-

-
-pip install --upgrade pip
-
-pip list  # show packages installed within the virtual environment
-

-

-  And to exit the virtual environment later:
-

-
deactivate  # don't exit until you're done using TensorFlow

-

-
-

-
macOS

-
-

-  Create a new virtual environment by choosing a Python interpreter and making a
-  ./venv directory to hold it:
-

-
python3 -m venv --system-site-packages ./venv

-

-  Activate the virtual environment using a shell-specific command:
-

-
source ./venv/bin/activate  # sh, bash, or zsh

-
. ./venv/bin/activate.fish  # fish

-
source ./venv/bin/activate.csh  # csh or tcsh

-
-

-  When the virtual environment is active, your shell prompt is prefixed with (venv).
-

-

-  Install packages within a virtual environment without affecting the host system
-  setup. Start by upgrading pip:
-

-
-pip install --upgrade pip
-
-pip list  # show packages installed within the virtual environment
-

-

-  And to exit the virtual environment later:
-

-
deactivate  # don't exit until you're done using TensorFlow

-

-
-
-

-  
Windows

-  

-  Create a new virtual environment by choosing a Python interpreter and making a
-  .\venv directory to hold it:
-

-
python -m venv --system-site-packages .\venv

-

-  Activate the virtual environment:
-

-
.\venv\Scripts\activate

-

-  Install packages within a virtual environment without affecting the host system
-  setup. Start by upgrading pip:
-

-
-pip install --upgrade pip
-
-pip list  # show packages installed within the virtual environment
-

-

-  And to exit the virtual environment later:
-

-
deactivate  # don't exit until you're done using TensorFlow

-

-
-
-

-
Conda

-

-While the TensorFlow provided pip package is recommended, a
-community-supported Anaconda package
-is available. To install, read the Anaconda TensorFlow guide.
-

-

-

-
-
-
3. Install the TensorFlow pip package

-
-

-  Choose one of the following TensorFlow packages to install from PyPI:
-

-
-
    
-  
  • tensorflow —Latest stable release with CPU and GPU support (Ubuntu and Windows).
    • 
-  
  • tf-nightly —Preview build (unstable). Ubuntu and Windows include GPU support.
    • 
-  
  • tensorflow==1.15 —The final version of TensorFlow 1.x.
    • 
-

-
-
-
-

-

-
Virtual environment install

-
pip install --upgrade tensorflow

-
Verify the install:

-
python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

-

-
-

-
System install

-
pip3 install --user --upgrade tensorflow  # install in $HOME

-
Verify the install:

-
python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

-

-

-
-
-
-
Package location

-
-

-  A few installation mechanisms require the URL of the TensorFlow Python package.
-  The value you specify depends on your Python version.
-

-
-
-

 
VersionPython versionCompilerBuild toolscuDNNCUDA
tensorflow_gpu-2.9.03.7-3.10MSVC 2019Bazel 5.0.08.111.2

 
tensorflow_gpu-2.8.03.7-3.10MSVC 2019Bazel 4.2.18.111.2

 
tensorflow_gpu-2.7.03.7-3.9MSVC 2019Bazel 3.7.28.111.2

 
tensorflow_gpu-2.6.03.6-3.9MSVC 2019Bazel 3.7.28.111.2

-  
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-   
-    
-    
-  
-
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-  
-    
-    
-  
-    
-    
-    
-  
-  
-    
-    
-  
-
-
VersionURL
Linux
Python 3.7 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-manylinux2014.whl
Python 3.7 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-manylinux2014.whl
Python 3.8 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp38-cp38-manylinux2014.whl
Python 3.8 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp38-cp38-manylinux2014.whl
Python 3.9 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp39-cp39-manylinux2014.whl
Python 3.9 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp39-cp39-manylinux2014.whl
Python 3.10 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp310-cp310-manylinux2014.whl
Python 3.10 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp310-cp310-manylinux2014.whl
macOS (CPU-only)
Python 3.7https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp37-cp37m-macosx_10_14_x86_64.whl
Python 3.8https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp38-cp38-macosx_10_14_x86_64.whl
Python 3.9https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp39-cp39-macosx_10_14_x86_64.whl
Python 3.10https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp310-cp310-macosx_10_14_x86_64.whl
Windows
Python 3.7 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-win_amd64.whl
Python 3.7 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-win_amd64.whl
Python 3.8 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp38-cp38-win_amd64.whl
Python 3.8 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp38-cp38-win_amd64.whl
Python 3.9 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp39-cp39-win_amd64.whl
Python 3.9 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp39-cp39-win_amd64.whl
Python 3.10 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp310-cp310-win_amd64.whl
Python 3.10 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp310-cp310-win_amd64.whl

-
-
-
diff --git a/site/en/install/pip.md b/site/en/install/pip.md
new file mode 100644
index 00000000000..b52765c3e66
--- /dev/null
+++ b/site/en/install/pip.md
@@ -0,0 +1,514 @@
+# Install TensorFlow with pip
+
+This guide is for the latest stable version of TensorFlow. For the
+preview build *(nightly)*, please use the pip package named
+`tf-nightly`. Refer to [these tables](./source#tested_build_configurations) for
+older TensorFlow version requirements. For TensorFlow 1.x users, please refer to
+the [migration guide](../guide/migrate) to upgrade to TensorFlow 2.
+
+Here is a lookup table for the install commands. Scroll down for the
+step-by-step instructions.
+
+* {Linux}
+
+   ```bash
+   conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
+   python3 -m pip install tensorflow
+   # Verify install:
+   python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+   ```
+
+* {MacOS}
+
+   ```bash
+   # Currently, we do not have official GPU support for MacOS.
+   python3 -m pip install tensorflow
+   # Verify install:
+   python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+   ```
+
+* {Windows}
+
+   ```bash
+   conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+   python3 -m pip install tensorflow
+   # Verify install:
+   python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+   ```
+
+* {CPU}
+
+   ```bash
+   python3 -m pip install tensorflow
+   # Verify install:
+   python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+   ```
+
+* {Nightly}
+
+   ```bash
+   python3 -m pip install tf-nightly
+   # Verify install:
+   python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+   ```
+
+## Hardware requirements
+
+Note: TensorFlow binaries use
+[AVX instructions](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX){:.external}
+which may not run on older CPUs.
+
+The following GPU-enabled devices are supported:
+
+*   NVIDIA® GPU card with CUDA® architectures 3.5, 5.0, 6.0, 7.0, 7.5, 8.0 and
+    higher. See the list of
+    [CUDA®-enabled GPU cards](https://developer.nvidia.com/cuda-gpus){:.external}.
+*   For GPUs with unsupported CUDA® architectures, or to avoid JIT compilation
+    from PTX, or to use different versions of the NVIDIA® libraries, see the
+    [Linux build from source](./source.md) guide.
+*   Packages do not contain PTX code except for the latest supported CUDA®
+    architecture; therefore, TensorFlow fails to load on older GPUs when
+    `CUDA_FORCE_PTX_JIT=1` is set. (See
+    [Application Compatibility](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#application-compatibility)
+    for details.) {:.external}
+
+Note: The error message "Status: device kernel image is invalid" indicates that
+the TensorFlow package does not contain PTX for your architecture. You can
+enable compute capabilities by [building TensorFlow from source](./source.md).
+
+## System requirements
+
+*   Ubuntu 16.04 or higher (64-bit)
+*   macOS 10.12.6 (Sierra) or higher (64-bit) *(no GPU support)*
+*   Windows 7 or higher (64-bit)
+
+Note: GPU support is available for Ubuntu and Windows with CUDA®-enabled cards.
+
+## Software requirements
+
+*   Python 3.7–3.10
+*   pip version 19.0 or higher for Linux (requires `manylinux2010` support) and
+    Windows, version 20.3 or higher for macOS
+*   Windows Requires
+    [Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads){:.external}
+
+The following NVIDIA® software are only required for GPU support.
+
+*   [NVIDIA® GPU drivers](https://www.nvidia.com/drivers){:.external}
+    version 450.80.02 or higher.
+*   [CUDA® Toolkit 11.2](https://developer.nvidia.com/cuda-toolkit-archive){:.external}.
+*   [cuDNN SDK 8.1.0](https://developer.nvidia.com/cudnn){:.external}.
+*   *(Optional)*
+    [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/archives/index.html#trt_7){:.external}
+    to improve latency and throughput for inference.
+
+## Step-by-step instructions
+
+
+*   {Linux}
+
+    We only officially support Ubuntu. However, the following instructions may
+    also work for other Linux distros.
+
+    We recommend using
+    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
+    create a separate environment to avoid changing any installed software in
+    your system. This is also the easiest way to install the required software,
+    especially for the GPU setup.
+
+    ### 1. Install Miniconda
+
+    You can use the following command to install Miniconda. During installation,
+    you may need to press enter and type "yes".
+
+    ```bash
+    curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
+    bash Miniconda3-latest-Linux-x86_64.sh
+    ```
+
+    You may need to restart your terminal or `source ~/.bashrc` to enable the
+    `conda` command. Use `conda -V` to test if it is installed successfully.
+
+    ### 2. Create a conda environment
+
+    Create a new conda environment named `tf` with the following command.
+
+    ```bash
+    conda create --name tf python=3.9
+    ```
+
+    You can deactivate and activate it with the following commands.
+
+    ```bash
+    conda deactivate
+    conda activate tf
+    ```
+
+    Please make sure it is activated for the rest of the installation.
+
+    ### 3. GPU setup
+
+    You can skip this section if you only run TensorFlow on CPU.
+
+    First, we need to install
+    [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
+    if you have not. You can use the following command to verify it is
+    installed.
+
+    ```bash
+    nvidia-smi
+    ```
+
+    Then, we install the CUDA, cuDNN with conda.
+
+    ```bash
+    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+    ```
+
+    Configure the system paths. You can do it with following command everytime
+    your start a new terminal after activating your conda environment.
+
+    ```bash
+    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
+    ```
+
+    However, for your convenience, we recommend automating it with the following
+    commands. The system paths will be automatically configured when you
+    activate this conda environment.
+
+    ```bash
+    mkdir -p $CONDA_PREFIX/etc/conda/activate.d
+    echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
+    ```
+
+    ### 4. Install TensorFlow
+
+    TensorFlow requires a recent version of pip, so upgrade your pip
+    installation to be sure you're running the latest version.
+
+    ```bash
+    pip install --upgrade pip
+    ```
+
+    Then, install TensorFlow with pip.
+
+    Note: Do not install with conda. It may not have the latest stable
+    version. We recommend using pip since TensorFlow is only
+    officially released to PyPI.
+
+    ```bash
+    pip install tensorflow
+    ```
+
+    ### 5. Verify install
+
+    Verify the CPU setup:
+
+    ```bash
+    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+    ```
+
+    If a tensor is returned, you've installed TensorFlow successfully.
+
+    Verify the GPU setup:
+
+    ```bash
+    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+    ```
+
+    If a list of GPU devices is returned, you've installed TensorFlow
+    successfully.
+
+*   {MacOS}
+
+    Note: For users of Apple M1 computers, to get native performance, you'll
+    want to follow the instructions found
+    [here](https://developer.apple.com/metal/tensorflow-plugin/){:.external}.
+    Conda has shown to have the smoothest install. Some TensorFlow binaries
+    (specifically, ones with custom C++ extensions like TensorFlow Decision
+    Forests, object detection, and TFX) are not compiled against M1 targets. If
+    you need those libraries, you will have to use TensorFlow with x86 emulation
+    and Rosetta.
+
+    Currently, we do not have official GPU support for running TensorFlow on
+    MacOS. The following is instructions are for running on CPU.
+
+    ### 1. Check Python version
+
+    Check if your Python environment is already configured:
+
+    Note: Requires Python 3.7–3.10, and pip >= 20.3 for MacOS.
+
+    ```bash
+    python3 --version
+    python3 -m pip --version
+    ```
+
+    If you have the correct version of Python and pip, you may skip the next two
+    steps and go to "4. Install TensorFlow". However, we still recommend not
+    skipping the steps. Use
+    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
+    install Python and pip. It create a separate environment to avoid
+    changing any installed software in your system.
+
+    ### 2. Install Miniconda
+
+    You can use the following command to install Miniconda. During installation,
+    you may need to press enter and type "yes".
+
+    ```bash
+    curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o Miniconda3-latest-MacOSX-x86_64.sh
+    bash Miniconda3-latest-MacOSX-x86_64.sh
+    ```
+
+    You may need to restart your terminal or `source ~/.bashrc` to enable the
+    `conda` command. Use `conda -V` to test if it is installed successfully.
+
+    ### 3. Create a conda environment
+
+    Create a new conda environment named `tf` with the following command.
+
+    ```bash
+    conda create --name tf python=3.9
+    ```
+
+    You can deactivate and activate it with the following commands.
+
+    ```bash
+    conda deactivate
+    conda activate tf
+    ```
+
+    Please make sure it is activated for the rest of the installation.
+
+    ### 4. Install TensorFlow
+
+    TensorFlow requires a recent version of pip, so upgrade your pip
+    installation to be sure you're running the latest version.
+
+    ```bash
+    pip install --upgrade pip
+    ```
+
+    Then, install TensorFlow with pip.
+
+    Note: Do not install with conda. It may not have the latest stable
+    version. We recommend using pip since TensorFlow is only
+    officially released to PyPI.
+
+    ```bash
+    pip install tensorflow
+    ```
+
+    ### 5. Verify install
+
+    ```bash
+    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+    ```
+
+    If a tensor is returned, you've installed TensorFlow successfully.
+
+*   {Windows}
+
+    Note: Experimental support for WSL2 on Windows 10 19044 or higher with GPU
+    access is now available. This corresponds to Windows 10 version
+    21H2, the November 2021 update. You can get the latest update from here:
+    [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
+    For instructions, please see
+    [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
+    for CUDA in WSL.
+
+    ### 1. Install Microsoft Visual C++ Redistributable
+
+    Install the *Microsoft Visual C++ Redistributable for Visual Studio 2015,
+    2017, and 2019*. Starting with the TensorFlow 2.1.0 version, the
+    `msvcp140_1.dll` file is required from this package (which may not be
+    provided from older redistributable packages). The redistributable comes
+    with *Visual Studio 2019* but can be installed separately:
+
+    1.  Go to the
+        [Microsoft Visual C++ downloads](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads){:.external}.
+    2.  Scroll down the page to the *Visual Studio 2015, 2017 and 2019* section.
+    3.  Download and install the *Microsoft Visual C++ Redistributable for
+        Visual Studio 2015, 2017 and 2019* for your platform.
+
+    Make sure
+    [long paths are enabled](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing){:.external}
+    on Windows.
+
+    ### 2. Install Miniconda
+
+    We recommend using
+    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
+    create a separate environment to avoid changing any installed software in
+    your system. This is also the easiest way to install the required software,
+    especially for the GPU setup.
+
+    Download the
+    [Miniconda Windows Installer](https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe){:.external}.
+    Double-click the downloaded file and follow the instructions on the screen.
+
+    ### 3. Create a conda environment
+
+    Create a new conda environment named `tf` with the following command.
+
+    ```bash
+    conda create --name tf python=3.9
+    ```
+
+    You can deactivate and activate it with the following commands.
+
+    ```bash
+    conda deactivate
+    conda activate tf
+    ```
+
+    Please make sure it is activated for the rest of the installation.
+
+    ### 4. GPU setup
+
+    You can skip this section if you only run TensorFlow on CPU.
+
+    First, we need to install
+    [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
+    if you have not.
+
+    Then, we install the CUDA, cuDNN with conda.
+
+    ```bash
+    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+    ```
+
+    ### 5. Install TensorFlow
+
+    TensorFlow requires a recent version of pip, so upgrade your pip
+    installation to be sure you're running the latest version.
+
+    ```bash
+    pip install --upgrade pip
+    ```
+
+    Then, install TensorFlow with pip.
+
+    Note: Do not install with conda. It may not have the latest stable
+    version. We recommend using pip since TensorFlow is only
+    officially released to PyPI.
+
+    ```bash
+    pip install tensorflow
+    ```
+
+    ### 6. Verify install
+
+    Verify the CPU setup:
+
+    ```bash
+    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+    ```
+
+    If a tensor is returned, you've installed TensorFlow successfully.
+
+    Verify the GPU setup:
+
+    ```bash
+    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+    ```
+
+    If a list of GPU devices is returned, you've installed TensorFlow
+    successfully.
+
+## Package location
+
+A few installation mechanisms require the URL of the TensorFlow Python package.
+The value you specify depends on your Python version.
+
+
+  
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+   
+    
+    
+  
+
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+  
+    
+    
+  
+    
+    
+    
+  
+  
+    
+    
+  
+
+
VersionURL
Linux
Python 3.7 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-manylinux2014.whl
Python 3.7 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-manylinux2014.whl
Python 3.8 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp38-cp38-manylinux2014.whl
Python 3.8 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp38-cp38-manylinux2014.whl
Python 3.9 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp39-cp39-manylinux2014.whl
Python 3.9 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp39-cp39-manylinux2014.whl
Python 3.10 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp310-cp310-manylinux2014.whl
Python 3.10 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp310-cp310-manylinux2014.whl
macOS (CPU-only)
Python 3.7https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp37-cp37m-macosx_10_14_x86_64.whl
Python 3.8https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp38-cp38-macosx_10_14_x86_64.whl
Python 3.9https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp39-cp39-macosx_10_14_x86_64.whl
Python 3.10https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.9.0-cp310-cp310-macosx_10_14_x86_64.whl
Windows
Python 3.7 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-win_amd64.whl
Python 3.7 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-win_amd64.whl
Python 3.8 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp38-cp38-win_amd64.whl
Python 3.8 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp38-cp38-win_amd64.whl
Python 3.9 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp39-cp39-win_amd64.whl
Python 3.9 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp39-cp39-win_amd64.whl
Python 3.10 GPU supporthttps://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.9.0-cp310-cp310-win_amd64.whl
Python 3.10 CPU-onlyhttps://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.9.0-cp310-cp310-win_amd64.whl

diff --git a/site/en/install/source.md b/site/en/install/source.md
index 88d8cc5221d..f6581453894 100644
--- a/site/en/install/source.md
+++ b/site/en/install/source.md
@@ -4,8 +4,8 @@ Build a TensorFlow *pip* package from source and install it on Ubuntu Linux and
 macOS. While the instructions might work for other systems, it is only tested
 and supported for Ubuntu and macOS.
 
-Note: Well-tested, pre-built
-[TensorFlow packages](./pip.html) for Linux and macOS systems are already provided.
+Note: Well-tested, pre-built [TensorFlow packages](./pip.md) for Linux and macOS
+systems are already provided.
 
 ## Setup for Linux and macOS
 
@@ -248,7 +248,7 @@ for
 Building TensorFlow from source can use a lot of RAM. If your system is
 memory-constrained, limit Bazel's RAM usage with: `--local_ram_resources=2048`.
 
-The [official TensorFlow packages](./pip.html) are built with a GCC 7.3
+The [official TensorFlow packages](./pip.md) are built with a GCC 7.3
 toolchain that complies with the manylinux2010 package standard.
 
 For GCC 5 and later, compatibility with the older ABI can be built using:
diff --git a/site/en/install/source_windows.md b/site/en/install/source_windows.md
index 7d999f54594..e4259229691 100644
--- a/site/en/install/source_windows.md
+++ b/site/en/install/source_windows.md
@@ -3,7 +3,7 @@
 Build a TensorFlow *pip* package from source and install it on Windows.
 
 Note: We already provide well-tested, pre-built
-[TensorFlow packages](./pip.html) for Windows systems.
+[TensorFlow packages](./pip.md) for Windows systems.
 
 ## Setup for Windows
 

From 1d826a49a37f931af0d72e001a95e3e5f770c0e9 Mon Sep 17 00:00:00 2001
From: Mihai Maruseac 
Date: Tue, 17 May 2022 11:40:27 -0700
Subject: [PATCH 142/872] Update TF documentation pages

Need to update C++ compilers and a few missing pages

PiperOrigin-RevId: 449275849
---
 site/en/install/source.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/install/source.md b/site/en/install/source.md
index f6581453894..9c4d518aeb5 100644
--- a/site/en/install/source.md
+++ b/site/en/install/source.md
@@ -420,7 +420,7 @@ Success: TensorFlow is now installed.
 
 
-
+
@@ -452,7 +452,7 @@ Success: TensorFlow is now installed.
 
 

 
VersionPython versionCompilerBuild tools
tensorflow-2.9.03.7-3.10GCC 7.3.1Bazel 5.0.0
tensorflow-2.9.03.7-3.10GCC 9.3.1Bazel 5.0.0

 
tensorflow-2.8.03.7-3.10GCC 7.3.1Bazel 4.2.1

 
tensorflow-2.7.03.7-3.9GCC 7.3.1Bazel 3.7.2

 
tensorflow-2.6.03.6-3.9GCC 7.3.1Bazel 3.7.2

-
+

From a71f8a8b40b428308cea65c2618acaf4be79c8f6 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 18 May 2022 07:16:29 -0700
Subject: [PATCH 143/872] Clear asserts, the devices are swapped in some cases.
 I don't think it matters.

PiperOrigin-RevId: 449475153
---
 site/en/tutorials/distribute/parameter_server_training.ipynb | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/distribute/parameter_server_training.ipynb b/site/en/tutorials/distribute/parameter_server_training.ipynb
index 88c21c58601..f997cc4e159 100644
--- a/site/en/tutorials/distribute/parameter_server_training.ipynb
+++ b/site/en/tutorials/distribute/parameter_server_training.ipynb
@@ -684,8 +684,9 @@
         "assert len(emb_layer.weights) == 2\n",
         "assert emb_layer.weights[0].shape == (4, 16384)\n",
         "assert emb_layer.weights[1].shape == (4, 16384)\n",
-        "assert emb_layer.weights[0].device == \"/job:ps/replica:0/task:0/device:CPU:0\"\n",
-        "assert emb_layer.weights[1].device == \"/job:ps/replica:0/task:1/device:CPU:0\""
+        "\n",
+        "print(emb_layer.weights[0].device)\n",
+        "print(emb_layer.weights[1].device)\n"
       ]
     },
     {

From eecf755cfa66a651390e6a18c2ce930a66fc78d3 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 18 May 2022 19:13:17 -0700
Subject: [PATCH 144/872] Update the dogs and cats notebook.

Now with more docs and cats.

PiperOrigin-RevId: 449633097
---
 site/en/tutorials/images/transfer_learning.ipynb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/site/en/tutorials/images/transfer_learning.ipynb b/site/en/tutorials/images/transfer_learning.ipynb
index c51993985b6..7174ae31693 100644
--- a/site/en/tutorials/images/transfer_learning.ipynb
+++ b/site/en/tutorials/images/transfer_learning.ipynb
@@ -83,7 +83,7 @@
         "    View on TensorFlow.org\n",
         "  \n",
         "  \n",
         "  
-    
+    
-    
+    
-    
+    
-    
+    
-    
+    
-    
+    
-    
+    
-    
+    

From 117593dc878f0ccd2b36cd2d1f6ea4b7442fa74c Mon Sep 17 00:00:00 2001
From: mohantym <86464649+mohantym@users.noreply.github.com>
Date: Thu, 2 Jun 2022 15:53:35 +0530
Subject: [PATCH 160/872] Fixed some grammar mistakes

Fixed some grammar mistakes at line
---
 site/en/tutorials/distribute/multi_worker_with_keras.ipynb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
index 4e8cc21f70f..d49f784fa2c 100644
--- a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
@@ -843,7 +843,7 @@
         "\n",
         "A repeated dataset (by calling `tf.data.Dataset.repeat`) is recommended for evaluation.\n",
         "\n",
-        "Alternatively, you can also create another task that periodically reads checkpoints and runs the evaluation. This is what Estimator does. But this is not a recommended way to perform evaluation and thus its details are omitted."
+        "Alternatively, you can also create another task that periodically reads checkpoints and runs the evaluation. This is what an Estimator does. But this is not a recommended way to perform evaluation and thus its details are omitted."
       ]
     },
     {
@@ -892,7 +892,7 @@
         "\n",
         "When a worker becomes unavailable, other workers will fail (possibly after a timeout). In such cases, the unavailable worker needs to be restarted, as well as other workers that have failed.\n",
         "\n",
-        "Note: Previously, the `ModelCheckpoint` callback provided a mechanism to restore the training state upon a restart from a job failure for multi-worker training. The TensorFlow team are introducing a new [`BackupAndRestore`](#scrollTo=kmH8uCUhfn4w) callback, which also adds the support to single-worker training for a consistent experience, and removed the fault tolerance functionality from existing `ModelCheckpoint` callback. From now on, applications that rely on this behavior should migrate to the new `BackupAndRestore` callback."
+        "Note: Previously, the `ModelCheckpoint` callback provided a mechanism to restore the training state upon a restart from a job failure for multi-worker training. The TensorFlow team is introducing a new [`BackupAndRestore`](#scrollTo=kmH8uCUhfn4w) callback, which also adds the support to single-worker training for a consistent experience, and removed the fault tolerance functionality from existing `ModelCheckpoint` callback. From now on, applications that rely on this behavior should migrate to the new `BackupAndRestore` callback."
       ]
     },
     {
@@ -907,7 +907,7 @@
         "\n",
         "The `ModelCheckpoint` callback can still be used to save checkpoints. But with this, if training was interrupted or successfully finished, in order to continue training from the checkpoint, the user is responsible to load the model manually.\n",
         "\n",
-        "Optionally the user can choose to save and restore model/weights outside `ModelCheckpoint` callback."
+        "Optionally, Users can choose to save and restore model/weights outside `ModelCheckpoint` callback."
       ]
     },
     {

From 3d1b9f0d66a0900fbc8a97a262dc0bc395b8cc19 Mon Sep 17 00:00:00 2001
From: mohantym <86464649+mohantym@users.noreply.github.com>
Date: Thu, 2 Jun 2022 22:21:43 +0530
Subject: [PATCH 161/872] fixed minor grammar mistake

fixed minor grammar mistake at line 910
---
 site/en/tutorials/distribute/multi_worker_with_keras.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
index d49f784fa2c..74955925f0b 100644
--- a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
@@ -907,7 +907,7 @@
         "\n",
         "The `ModelCheckpoint` callback can still be used to save checkpoints. But with this, if training was interrupted or successfully finished, in order to continue training from the checkpoint, the user is responsible to load the model manually.\n",
         "\n",
-        "Optionally, Users can choose to save and restore model/weights outside `ModelCheckpoint` callback."
+        "Optionally, users can choose to save and restore model/weights outside `ModelCheckpoint` callback."
       ]
     },
     {

From f89288ec4c535eba916438c81aeddcdca6e9184f Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Thu, 2 Jun 2022 14:10:42 -0700
Subject: [PATCH 162/872] Fix install.

PiperOrigin-RevId: 452618544
---
 site/en/tutorials/reinforcement_learning/actor_critic.ipynb | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
index ea644d46342..ef09ea310a1 100644
--- a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
+++ b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
@@ -111,7 +111,7 @@
         "\n",
         "
\n",
         "  
\n",
-        "    \n",
+        "    \n",
         "    
\n",
         "      Trained actor-critic model in Cartpole-v0 environment\n",
         "    
\n",
@@ -147,7 +147,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install gym\n",
+        "!pip install gym[classic_control]\n",
         "!pip install pyglet"
       ]
     },
@@ -738,6 +738,7 @@
         "_jQ1tEQCxwRx"
       ],
       "name": "actor_critic.ipynb",
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 27d363e58ed78544a9eee98ac2d2ba4e3518278c Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Fri, 3 Jun 2022 00:05:10 +0100
Subject: [PATCH 163/872] Lint Multi-worker training with Keras tutorial

---
 site/en/tutorials/distribute/multi_worker_with_keras.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
index 74955925f0b..c98b3287c8a 100644
--- a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
@@ -90,7 +90,7 @@
         "* _Synchronous training_, where the steps of training are synced across the workers and replicas, such as `tf.distribute.MirroredStrategy`, `tf.distribute.TPUStrategy`, and `tf.distribute.MultiWorkerMirroredStrategy`. All workers train over different slices of input data in sync, and aggregating gradients at each step.\n",
         "* _Asynchronous training_, where the training steps are not strictly synced, such as `tf.distribute.experimental.ParameterServerStrategy`. All workers are independently training over the input data and updating variables asynchronously.\n",
         "\n",
-        "If you are looking for multi-worker synchronous training without TPU, then `tf.distribute.MultiWorkerMirroredStrategy` is your choice. It creates copies of all variables in the model's layers on each device across all workers. It uses `CollectiveOps`, a TensorFlow op for collective communication, to aggregate gradients and keeps the variables in sync. For those interested, check out the `tf.distribute.experimental.CommunicationOptions` parameter for the collective implementation options we are providing.\n",
+        "If you are looking for multi-worker synchronous training without TPU, then `tf.distribute.MultiWorkerMirroredStrategy` is your choice. It creates copies of all variables in the model's layers on each device across all workers. It uses `CollectiveOps`, a TensorFlow op for collective communication, to aggregate gradients and keeps the variables in sync. For those interested, check out the `tf.distribute.experimental.CommunicationOptions` parameter for the collective implementation options.\n",
         "\n",
         "For an overview of `tf.distribute.Strategy` APIs, refer to [Distributed training in TensorFlow](../../guide/distributed_training.ipynb)."
       ]

From 6ddc5346a35f5db919b07988de5fd3cc2881d346 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Thu, 2 Jun 2022 16:26:26 -0700
Subject: [PATCH 164/872] Fix distributed_training guide.

PiperOrigin-RevId: 452647033
---
 site/en/guide/distributed_training.ipynb | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb
index f9e2c98ceeb..96c98edc82a 100644
--- a/site/en/guide/distributed_training.ipynb
+++ b/site/en/guide/distributed_training.ipynb
@@ -585,6 +585,17 @@
         "In both cases—with `Dataset` or NumPy—each batch of the given input is divided equally among the multiple replicas. For instance, if you are using the `MirroredStrategy` with 2 GPUs, each batch of size 10 will be divided among the 2 GPUs, with each receiving 5 input examples in each step. Each epoch will then train faster as you add more GPUs. Typically, you would want to increase your batch size as you add more accelerators, so as to make effective use of the extra computing power. You will also need to re-tune your learning rate, depending on the model. You can use `strategy.num_replicas_in_sync` to get the number of replicas."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8ZmJqErtS4A1"
+      },
+      "outputs": [],
+      "source": [
+        "mirrored_strategy.num_replicas_in_sync"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -600,7 +611,7 @@
         "dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(100)\n",
         "dataset = dataset.batch(global_batch_size)\n",
         "\n",
-        "LEARNING_RATES_BY_BATCH_SIZE = {5: 0.1, 10: 0.15}\n",
+        "LEARNING_RATES_BY_BATCH_SIZE = {5: 0.1, 10: 0.15, 20:0.175}\n",
         "learning_rate = LEARNING_RATES_BY_BATCH_SIZE[global_batch_size]"
       ]
     },
@@ -684,7 +695,7 @@
       },
       "outputs": [],
       "source": [
-        "dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(100).batch(\n",
+        "dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(1000).batch(\n",
         "    global_batch_size)\n",
         "dist_dataset = mirrored_strategy.experimental_distribute_dataset(dataset)"
       ]
@@ -889,6 +900,8 @@
         "Tce3stUlHN0L"
       ],
       "name": "distributed_training.ipynb",
+      "private_outputs": true,
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 0a1387ba09f917de0ba8b2cb1a526cb892353557 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Fri, 3 Jun 2022 05:43:58 -0700
Subject: [PATCH 165/872] Restrict protobuf dependency to match generated code.

Our generated schemas currently include references to `_internal_create_key`, which doesn't exist prior to 3.12, so we can't claim to support it.

PiperOrigin-RevId: 452748647
---
 setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/setup.py b/setup.py
index 3e44fb34e16..2311b26fa7f 100644
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@
     # See also: https://github.com/protocolbuffers/protobuf/issues/9954
     # See also: https://github.com/tensorflow/tensorflow/issues/56077
     # This is a temporary patch for now, to patch previous TF releases.
-    'protobuf >= 3.1.4, < 3.20',
+    'protobuf >= 3.12.0, < 3.20',
     'pyyaml',
 ]
 

From dfbed06c3f5f82431f6bec8e1b2fe2c7b33b92a2 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Fri, 3 Jun 2022 12:07:16 -0700
Subject: [PATCH 166/872] Fix references to `AbstractTrainer`.

PiperOrigin-RevId: 452815564
---
 site/en/guide/model_garden/index.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/model_garden/index.md b/site/en/guide/model_garden/index.md
index c08ec950a95..57b022a62d1 100644
--- a/site/en/guide/model_garden/index.md
+++ b/site/en/guide/model_garden/index.md
@@ -124,8 +124,9 @@ Alternatively, you can use the high-level Keras `Model.fit`.
 
 However, if your model is complex and your training loop requires more flexible
 control or customization, then you should use Orbit. You can define most of your
-training loop by extending Orbit's `AbstractTrainerclass`. Learn more about the
-Orbit tool in the [Orbit API documentation](../../api_docs/python/orbit).
+training loop by the `orbit.AbstractTrainer` or `orbit.StandardTrainer` class.
+Learn more about the Orbit tool in the
+[Orbit API documentation](../../api_docs/python/orbit).
 
 Note: You can use the Keras API to do what Orbit does, but you must override
 the TensorFlow `train_step` function or use callbacks like ModelCheckpoint or

From 649de77178d26571db53d3f1ea88af7431cb18c7 Mon Sep 17 00:00:00 2001
From: Olzhas Akpambetov 
Date: Sat, 4 Jun 2022 05:03:46 -0700
Subject: [PATCH 167/872] Update the Use TPUs guide

PiperOrigin-RevId: 452941060
---
 site/en/guide/tpu.ipynb | 68 +++++++++++++++++++++++++++--------------
 1 file changed, 45 insertions(+), 23 deletions(-)

diff --git a/site/en/guide/tpu.ipynb b/site/en/guide/tpu.ipynb
index f64450ba04c..c17af68516e 100644
--- a/site/en/guide/tpu.ipynb
+++ b/site/en/guide/tpu.ipynb
@@ -61,7 +61,9 @@
         "id": "Ys81cOhXOWUP"
       },
       "source": [
-        "Before you run this Colab notebook, make sure that your hardware accelerator is a TPU by checking your notebook settings: **Runtime** > **Change runtime type** > **Hardware accelerator** > **TPU**."
+        "This guide demonstrates how to perform basic training on [Tensor Processing Units (TPUs)](https://cloud.google.com/tpu/) and TPU Pods, a collection of TPU devices connected by dedicated high-speed network interfaces, with `tf.keras` and custom training loops.\n",
+        "\n",
+        "TPUs are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads. They are available through [Google Colab](https://colab.research.google.com/), the [TPU Research Cloud](https://sites.research.google/trc/), and [Cloud TPU](https://cloud.google.com/tpu)."
       ]
     },
     {
@@ -73,6 +75,17 @@
         "## Setup"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ebf7f8489bb7"
+      },
+      "source": [
+        "Before you run this Colab notebook, make sure that your hardware accelerator is a TPU by checking your notebook settings: **Runtime** > **Change runtime type** > **Hardware accelerator** > **TPU**.\n",
+        "\n",
+        "Import some necessary libraries, including TensorFlow Datasets:"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -95,7 +108,7 @@
       "source": [
         "## TPU initialization\n",
         "\n",
-        "TPUs are typically Cloud TPU workers, which are different from the local process running the user's Python program. Thus, you need to do some initialization work to connect to the remote cluster and initialize the TPUs. Note that the `tpu` argument to `tf.distribute.cluster_resolver.TPUClusterResolver` is a special address just for Colab. If you are running your code on Google Compute Engine (GCE), you should instead pass in the name of your Cloud TPU."
+        "TPUs are typically [Cloud TPU](https://cloud.google.com/tpu/docs/) workers, which are different from the local process running the user's Python program. Thus, you need to do some initialization work to connect to the remote cluster and initialize the TPUs. Note that the `tpu` argument to `tf.distribute.cluster_resolver.TPUClusterResolver` is a special address just for Colab. If you are running your code on Google Compute Engine (GCE), you should instead pass in the name of your Cloud TPU."
       ]
     },
     {
@@ -159,7 +172,7 @@
       "source": [
         "## Distribution strategies\n",
         "\n",
-        "Usually you run your model on multiple TPUs in a data-parallel way. To distribute your model on multiple TPUs (or other accelerators), TensorFlow offers several distribution strategies. You can replace your distribution strategy and the model will run on any given (TPU) device. Check the [distribution strategy guide](./distributed_training.ipynb) for more information."
+        "Usually, you run your model on multiple TPUs in a data-parallel way. To distribute your model on multiple TPUs (as well as multiple GPUs or multiple machines), TensorFlow offers the `tf.distribute.Strategy` API. You can replace your distribution strategy and the model will run on any given (TPU) device. Learn more in the [Distributed training with TensorFlow](./distributed_training.ipynb) guide."
       ]
     },
     {
@@ -168,6 +181,8 @@
         "id": "DcDPMZs-9uLJ"
       },
       "source": [
+        "Using the `tf.distribute.TPUStrategy` option implements synchronous distributed training. TPUs provide their own implementation of efficient all-reduce and other collective operations across multiple TPU cores, which are used in `TPUStrategy`.\n",
+        "\n",
         "To demonstrate this, create a `tf.distribute.TPUStrategy` object:"
       ]
     },
@@ -188,7 +203,7 @@
         "id": "JlaAmswWPsU6"
       },
       "source": [
-        "To replicate a computation so it can run in all TPU cores, you can pass it into the `strategy.run` API. Below is an example that shows all cores receiving the same inputs `(a, b)` and performing matrix multiplication on each core independently. The outputs will be the values from all the replicas."
+        "To replicate a computation so it can run in all TPU cores, you can pass it into the `Strategy.run` API. Below is an example that shows all cores receiving the same inputs `(a, b)` and performing matrix multiplication on each core independently. The outputs will be the values from all the replicas."
       ]
     },
     {
@@ -216,7 +231,7 @@
       "source": [
         "## Classification on TPUs\n",
         "\n",
-        "Having covered the basic concepts, consider a more concrete example. This section demonstrates how to use the distribution strategy—`tf.distribute.TPUStrategy`—to train a Keras model on a Cloud TPU.\n"
+        "Having covered the basic concepts, consider a more concrete example. This section demonstrates how to use the distribution strategy—`tf.distribute.TPUStrategy`—to train a Keras model on a Cloud TPU."
       ]
     },
     {
@@ -227,7 +242,7 @@
       "source": [
         "### Define a Keras model\n",
         "\n",
-        "Start with a definition of a `Sequential` Keras model for image classification on the MNIST dataset using Keras. It's no different than what you would use if you were training on CPUs or GPUs. Note that Keras model creation needs to be inside `strategy.scope`, so the variables can be created on each TPU device. Other parts of the code are not necessary to be inside the strategy scope."
+        "Start with a definition of a [`Sequential` Keras model](./sequential_model.ipynb) for image classification on the MNIST dataset. It's no different than what you would use if you were training on CPUs or GPUs. Note that Keras model creation needs to be inside the `Strategy.scope`, so the variables can be created on each TPU device. Other parts of the code are not necessary to be inside the `Strategy` scope."
       ]
     },
     {
@@ -256,9 +271,9 @@
       "source": [
         "### Load the dataset\n",
         "\n",
-        "Efficient use of the `tf.data.Dataset` API is critical when using a Cloud TPU, as it is impossible to use the Cloud TPUs unless you can feed them data quickly enough. You can learn more about dataset performance in the [Input pipeline performance guide](./data_performance.ipynb).\n",
+        "Efficient use of the `tf.data.Dataset` API is critical when using a Cloud TPU. You can learn more about dataset performance in the [Input pipeline performance guide](./data_performance.ipynb).\n",
         "\n",
-        "For all but the simplest experiments (using `tf.data.Dataset.from_tensor_slices` or other in-graph data), you need to store all data files read by the Dataset in Google Cloud Storage (GCS) buckets.\n",
+        "If you are using [TPU Nodes](https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm), you need to store all data files read by the TensorFlow `Dataset` in [Google Cloud Storage (GCS) buckets](https://cloud.google.com/tpu/docs/storage-buckets). If you are using [TPU VMs](https://cloud.google.com/tpu/docs/users-guide-tpu-vm), you can store data wherever you like. For more information on TPU Nodes and TPU VMs, refer to the [TPU System Architecture](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm) documentation.\n",
         "\n",
         "For most use cases, it is recommended to convert your data into the `TFRecord` format and use a `tf.data.TFRecordDataset` to read it. Check the [TFRecord and tf.Example tutorial](../tutorials/load_data/tfrecord.ipynb) for details on how to do this. It is not a hard requirement and you can use other dataset readers, such as `tf.data.FixedLengthRecordDataset` or `tf.data.TextLineDataset`.\n",
         "\n",
@@ -266,7 +281,7 @@
         "\n",
         "Regardless of the data format used, it is strongly recommended that you use large files on the order of 100MB. This is especially important in this networked setting, as the overhead of opening a file is significantly higher.\n",
         "\n",
-        "As shown in the code below, you should use the `tensorflow_datasets` module to get a copy of the MNIST training and test data. Note that `try_gcs` is specified to use a copy that is available in a public GCS bucket. If you don't specify this, the TPU will not be able to access the downloaded data. "
+        "As shown in the code below, you should use the Tensorflow Datasets `tfds.load` module to get a copy of the MNIST training and test data. Note that `try_gcs` is specified to use a copy that is available in a public GCS bucket. If you don't specify this, the TPU will not be able to access the downloaded data."
       ]
     },
     {
@@ -311,7 +326,7 @@
       "source": [
         "### Train the model using Keras high-level APIs\n",
         "\n",
-        "You can train your model with Keras `fit` and `compile` APIs. There is nothing TPU-specific in this step—you write the code as if you were using mutliple GPUs and a `MirroredStrategy` instead of the `TPUStrategy`. You can learn more in the [Distributed training with Keras](https://www.tensorflow.org/tutorials/distribute/keras) tutorial."
+        "You can train your model with Keras `Model.fit` and `Model.compile` APIs. There is nothing TPU-specific in this step—you write the code as if you were using multiple GPUs and a `MirroredStrategy` instead of the `TPUStrategy`. You can learn more in the [Distributed training with Keras](../tutorials/distribute/keras.ipynb) tutorial."
       ]
     },
     {
@@ -338,7 +353,7 @@
         "model.fit(train_dataset,\n",
         "          epochs=5,\n",
         "          steps_per_epoch=steps_per_epoch,\n",
-        "          validation_data=test_dataset, \n",
+        "          validation_data=test_dataset,\n",
         "          validation_steps=validation_steps)"
       ]
     },
@@ -348,7 +363,7 @@
         "id": "8hSGBIYtUugJ"
       },
       "source": [
-        "To reduce Python overhead and maximize the performance of your TPU, pass in the argument—`steps_per_execution`—to `Model.compile`. In this example, it increases throughput by about 50%:"
+        "To reduce Python overhead and maximize the performance of your TPU, pass in the `steps_per_execution` argument to Keras `Model.compile`. In this example, it increases throughput by about 50%:"
       ]
     },
     {
@@ -382,7 +397,7 @@
       "source": [
         "### Train the model using a custom training loop\n",
         "\n",
-        "You can also create and train your model using `tf.function` and `tf.distribute` APIs directly. You can use the `strategy.experimental_distribute_datasets_from_function` API to distribute the dataset given a dataset function. Note that in the example below the batch size passed into the dataset is the per-replica batch size instead of the global batch size. To learn more, check out the [Custom training with tf.distribute.Strategy](https://www.tensorflow.org/tutorials/distribute/custom_training) tutorial.\n"
+        "You can also create and train your model using `tf.function` and `tf.distribute` APIs directly. You can use the `Strategy.experimental_distribute_datasets_from_function` API to distribute the `tf.data.Dataset` given a dataset function. Note that in the example below the batch size passed into the `Dataset` is the per-replica batch size instead of the global batch size. To learn more, check out the [Custom training with `tf.distribute.Strategy`](../tutorials/distribute/custom_training.ipynb) tutorial.\n"
       ]
     },
     {
@@ -391,7 +406,7 @@
         "id": "DxdgXPAL6iFE"
       },
       "source": [
-        "First, create the model, datasets and tf.functions:"
+        "First, create the model, datasets and `tf.function`s:"
       ]
     },
     {
@@ -402,8 +417,8 @@
       },
       "outputs": [],
       "source": [
-        "# Create the model, optimizer and metrics inside the strategy scope, so that the\n",
-        "# variables can be mirrored on each device.\n",
+        "# Create the model, optimizer and metrics inside the `tf.distribute.Strategy`\n",
+        "# scope, so that the variables can be mirrored on each device.\n",
         "with strategy.scope():\n",
         "  model = create_model()\n",
         "  optimizer = tf.keras.optimizers.Adam()\n",
@@ -411,8 +426,8 @@
         "  training_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(\n",
         "      'training_accuracy', dtype=tf.float32)\n",
         "\n",
-        "# Calculate per replica batch size, and distribute the datasets on each TPU\n",
-        "# worker.\n",
+        "# Calculate per replica batch size, and distribute the `tf.data.Dataset`s\n",
+        "# on each TPU worker.\n",
         "per_replica_batch_size = batch_size // strategy.num_replicas_in_sync\n",
         "\n",
         "train_dataset = strategy.experimental_distribute_datasets_from_function(\n",
@@ -479,9 +494,9 @@
       "source": [
         "### Improving performance with multiple steps inside `tf.function`\n",
         "\n",
-        "You can improve the performance by running multiple steps within a `tf.function`. This is achieved by wrapping the `strategy.run` call with a `tf.range` inside `tf.function`, and AutoGraph will convert it to a `tf.while_loop` on the TPU worker.\n",
+        "You can improve the performance by running multiple steps within a `tf.function`. This is achieved by wrapping the `Strategy.run` call with a `tf.range` inside `tf.function`, and AutoGraph will convert it to a `tf.while_loop` on the TPU worker. You can learn more about `tf.function`s in the [Better performance with `tf.function`](./function.ipynb) guide.\n",
         "\n",
-        "Despite the improved performance, there are tradeoffs with this method compared to running a single step inside `tf.function`. Running multiple steps in a `tf.function` is less flexible—you cannot run things eagerly or arbitrary Python code within the steps.\n"
+        "Despite the improved performance, there are tradeoffs with this method compared to running a single step inside a `tf.function`. Running multiple steps in a `tf.function` is less flexible—you cannot run things eagerly or arbitrary Python code within the steps.\n"
       ]
     },
     {
@@ -512,7 +527,7 @@
         "  for _ in tf.range(steps):\n",
         "    strategy.run(step_fn, args=(next(iterator),))\n",
         "\n",
-        "# Convert `steps_per_epoch` to `tf.Tensor` so the `tf.function` won't get \n",
+        "# Convert `steps_per_epoch` to `tf.Tensor` so the `tf.function` won't get\n",
         "# retraced if the value changes.\n",
         "train_multiple_steps(train_iterator, tf.convert_to_tensor(steps_per_epoch))\n",
         "\n",
@@ -530,10 +545,17 @@
       "source": [
         "## Next steps\n",
         "\n",
-        "- [Google Cloud TPU documentation](https://cloud.google.com/tpu/docs/): How to set up and run a Google Cloud TPU.\n",
+        "To learn more about Cloud TPUs and how to use them:\n",
+        "\n",
+        "- [Google Cloud TPU](https://cloud.google.com/tpu): The Google Cloud TPU homepage.\n",
+        "- [Google Cloud TPU documentation](https://cloud.google.com/tpu/docs/): Google Cloud TPU documentation, which includes:\n",
+        "  - [Introduction to Cloud TPU](https://cloud.google.com/tpu/docs/intro-to-tpu): An overview of working with Cloud TPUs.\n",
+        "  - [Cloud TPU quickstarts](https://cloud.google.com/tpu/docs/quick-starts): Quickstart introductions to working with Cloud TPU VMs using TensorFlow and other main machine learning frameworks.\n",
         "- [Google Cloud TPU Colab notebooks](https://cloud.google.com/tpu/docs/colabs): End-to-end training examples.\n",
         "- [Google Cloud TPU performance guide](https://cloud.google.com/tpu/docs/performance-guide): Enhance Cloud TPU performance further by adjusting Cloud TPU configuration parameters for your application\n",
-        "- [Distributed training with TensorFlow](./distributed_training.ipynb): How to use distribution strategies—including `tf.distribute.TPUStrategy`—with examples showing best practices."
+        "- [Distributed training with TensorFlow](./distributed_training.ipynb): How to use distribution strategies—including `tf.distribute.TPUStrategy`—with examples showing best practices.\n",
+        "- TPU embeddings: TensorFlow includes specialized support for training embeddings on TPUs via `tf.tpu.experimental.embedding`. In addition, [TensorFlow Recommenders](https://www.tensorflow.org/recommenders) has `tfrs.layers.embedding.TPUEmbedding`. Embeddings provide efficient and dense representations, capturing complex similarities and relationships between features. TensorFlow's TPU-specific embedding support allows you to train embeddings that are larger than the memory of a single TPU device, and to use sparse and ragged inputs on TPUs.\n",
+        "- [TPU Research Cloud (TRC)] https://sites.research.google/trc/about/: TRC enables researchers to apply for access to a cluster of more than 1,000 Cloud TPU devices.\n"
       ]
     }
   ],

From 95850072117d6dc2aa8518a5c829f1b35f8575f9 Mon Sep 17 00:00:00 2001
From: Boyd Kane <33420535+beyarkay@users.noreply.github.com>
Date: Tue, 7 Jun 2022 18:46:20 +0200
Subject: [PATCH 168/872] Fix dead link

---
 site/en/guide/data.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb
index 6421484aafd..bf1fc11eb93 100644
--- a/site/en/guide/data.ipynb
+++ b/site/en/guide/data.ipynb
@@ -2422,7 +2422,7 @@
         "\n",
         "When working with a dataset that is very class-imbalanced, you may want to resample the dataset. `tf.data` provides two methods to do this. The credit card fraud dataset is a good example of this sort of problem.\n",
         "\n",
-        "Note: See [Imbalanced Data](../tutorials/keras/imbalanced_data.ipynb) for a full tutorial.\n"
+        "Note: See [Classification on imbalanced data](../tutorials/structured_data/imbalanced_data) for a full tutorial.\n"
       ]
     },
     {

From 3e2105bb4dee89137a85794f0de99e51aac56344 Mon Sep 17 00:00:00 2001
From: Boyd Kane <33420535+beyarkay@users.noreply.github.com>
Date: Tue, 7 Jun 2022 18:48:27 +0200
Subject: [PATCH 169/872] Fix typo

---
 site/en/guide/data.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb
index bf1fc11eb93..96a056ce48e 100644
--- a/site/en/guide/data.ipynb
+++ b/site/en/guide/data.ipynb
@@ -2422,7 +2422,7 @@
         "\n",
         "When working with a dataset that is very class-imbalanced, you may want to resample the dataset. `tf.data` provides two methods to do this. The credit card fraud dataset is a good example of this sort of problem.\n",
         "\n",
-        "Note: See [Classification on imbalanced data](../tutorials/structured_data/imbalanced_data) for a full tutorial.\n"
+        "Note: See [Classification on imbalanced data](../tutorials/structured_data/imbalanced_data.ipynb) for a full tutorial.\n"
       ]
     },
     {

From 72732a61d59666f33e6858b9730813b86b483fee Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Tue, 7 Jun 2022 16:58:03 -0700
Subject: [PATCH 170/872] Automated typo cleanup.

Now we have a pre-submission check for these, clean up some old typos.

PiperOrigin-RevId: 453557531
---
 setup.py                                                    | 4 ++--
 site/en/guide/advanced_autodiff.ipynb                       | 2 +-
 site/en/guide/basic_training_loops.ipynb                    | 2 +-
 site/en/guide/data.ipynb                                    | 4 ++--
 site/en/guide/dtensor_overview.ipynb                        | 6 +++---
 site/en/guide/graph_optimization.ipynb                      | 2 +-
 site/en/guide/intro_to_graphs.ipynb                         | 2 +-
 site/en/guide/migrate/canned_estimators.ipynb               | 2 +-
 site/en/guide/migrate/evaluator.ipynb                       | 2 +-
 site/en/guide/migrate/migrating_feature_columns.ipynb       | 2 +-
 site/en/guide/migrate/migration_debugging.ipynb             | 2 +-
 site/en/guide/migrate/tf1_vs_tf2.ipynb                      | 2 +-
 site/en/guide/migrate/upgrade.ipynb                         | 2 +-
 site/en/guide/ragged_tensor.ipynb                           | 2 +-
 site/en/guide/random_numbers.ipynb                          | 4 ++--
 site/en/r1/guide/autograph.ipynb                            | 2 +-
 site/en/r1/guide/distribute_strategy.ipynb                  | 4 ++--
 site/en/r1/guide/eager.ipynb                                | 2 +-
 site/en/r1/guide/keras.ipynb                                | 2 +-
 site/en/r1/guide/ragged_tensors.ipynb                       | 4 ++--
 site/en/r1/tutorials/_index.ipynb                           | 2 +-
 site/en/r1/tutorials/distribute/keras.ipynb                 | 4 ++--
 site/en/r1/tutorials/distribute/tpu_custom_training.ipynb   | 2 +-
 site/en/r1/tutorials/distribute/training_loops.ipynb        | 2 +-
 site/en/r1/tutorials/eager/automatic_differentiation.ipynb  | 2 +-
 site/en/r1/tutorials/eager/custom_layers.ipynb              | 2 +-
 site/en/r1/tutorials/eager/custom_training.ipynb            | 2 +-
 .../en/r1/tutorials/eager/custom_training_walkthrough.ipynb | 2 +-
 site/en/r1/tutorials/eager/eager_basics.ipynb               | 2 +-
 site/en/r1/tutorials/images/hub_with_keras.ipynb            | 2 +-
 site/en/r1/tutorials/images/transfer_learning.ipynb         | 2 +-
 site/en/r1/tutorials/keras/basic_classification.ipynb       | 2 +-
 site/en/r1/tutorials/keras/basic_regression.ipynb           | 2 +-
 site/en/r1/tutorials/keras/basic_text_classification.ipynb  | 2 +-
 site/en/r1/tutorials/keras/overfit_and_underfit.ipynb       | 2 +-
 site/en/r1/tutorials/keras/save_and_restore_models.ipynb    | 4 ++--
 site/en/r1/tutorials/load_data/images.ipynb                 | 2 +-
 site/en/r1/tutorials/load_data/tf_records.ipynb             | 2 +-
 site/en/r1/tutorials/non-ml/mandelbrot.ipynb                | 2 +-
 site/en/r1/tutorials/non-ml/pdes.ipynb                      | 2 +-
 site/en/r1/tutorials/representation/unicode.ipynb           | 2 +-
 site/en/r1/tutorials/sequences/text_generation.ipynb        | 2 +-
 site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb   | 2 +-
 site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb      | 4 ++--
 site/en/tutorials/generative/cyclegan.ipynb                 | 2 +-
 site/en/tutorials/generative/data_compression.ipynb         | 2 +-
 site/en/tutorials/images/transfer_learning.ipynb            | 2 +-
 site/en/tutorials/load_data/tfrecord.ipynb                  | 2 +-
 site/en/tutorials/structured_data/imbalanced_data.ipynb     | 2 +-
 site/en/tutorials/text/image_captioning.ipynb               | 4 ++--
 site/en/tutorials/text/word2vec.ipynb                       | 4 ++--
 site/en/tutorials/understanding/sngp.ipynb                  | 6 +++---
 tools/tensorflow_docs/api_generator/doc_controls.py         | 4 ++--
 .../tensorflow_docs/api_generator/doc_generator_visitor.py  | 2 +-
 .../api_generator/doc_generator_visitor_test.py             | 4 ++--
 .../api_generator/pretty_docs/module_page.py                | 2 +-
 tools/tensorflow_docs/api_generator/reference_resolver.py   | 2 +-
 tools/tensorflow_docs/tools/nbfmt/__main__.py               | 4 ++--
 tools/tensorflow_docs/tools/nblint/__main__.py              | 2 +-
 tools/tensorflow_docs/tools/nblint/linter.py                | 4 ++--
 tools/tensorflow_docs/tools/nblint/style/google.py          | 4 ++--
 tools/tensorflow_docs/tools/nblint/style/tensorflow.py      | 2 +-
 62 files changed, 81 insertions(+), 81 deletions(-)

diff --git a/setup.py b/setup.py
index 2311b26fa7f..3a726be637b 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,7 @@
 if (sys.version_info.major, sys.version_info.minor) == (3, 6):
   REQUIRED_PKGS.append('dataclasses')
 
-VIS_REQURE = [
+VIS_REQUIRE = [
     'numpy',
     'PILLOW',
     'webp',
@@ -65,7 +65,7 @@
     package_dir={'': 'tools'},
     scripts=[],
     install_requires=REQUIRED_PKGS,
-    extras_require={'vis': VIS_REQURE},
+    extras_require={'vis': VIS_REQUIRE},
     classifiers=[
         'Development Status :: 4 - Beta',
         'Intended Audience :: Developers',
diff --git a/site/en/guide/advanced_autodiff.ipynb b/site/en/guide/advanced_autodiff.ipynb
index 7da53d8ee48..e04b9db4d77 100644
--- a/site/en/guide/advanced_autodiff.ipynb
+++ b/site/en/guide/advanced_autodiff.ipynb
@@ -1101,7 +1101,7 @@
         "id": "M_x7ih5sarvG"
       },
       "source": [
-        "In this case, `batch_jacobian` still runs and returns _something_ with the expected shape, but it's contents have an unclear meaning:"
+        "In this case, `batch_jacobian` still runs and returns _something_ with the expected shape, but its contents have an unclear meaning:"
       ]
     },
     {
diff --git a/site/en/guide/basic_training_loops.ipynb b/site/en/guide/basic_training_loops.ipynb
index 74284442cde..a1558b1903e 100644
--- a/site/en/guide/basic_training_loops.ipynb
+++ b/site/en/guide/basic_training_loops.ipynb
@@ -233,7 +233,7 @@
         "id": "rdpN_3ssG9D5"
       },
       "source": [
-        "The initial variables are set here in a fixed way, but Keras comes with any of a number of [initalizers](https://www.tensorflow.org/api_docs/python/tf/keras/initializers) you could use, with or without the rest of Keras."
+        "The initial variables are set here in a fixed way, but Keras comes with any of a number of [initializers](https://www.tensorflow.org/api_docs/python/tf/keras/initializers) you could use, with or without the rest of Keras."
       ]
     },
     {
diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb
index 6421484aafd..e2384ea9989 100644
--- a/site/en/guide/data.ipynb
+++ b/site/en/guide/data.ipynb
@@ -472,7 +472,7 @@
         "\n",
         "Another common data source that can easily be ingested as a `tf.data.Dataset` is the python generator.\n",
         "\n",
-        "Caution: While this is a convienient approach it has limited portability and scalibility. It must run in the same python process that created the generator, and is still subject to the Python [GIL](https://en.wikipedia.org/wiki/Global_interpreter_lock)."
+        "Caution: While this is a convenient approach it has limited portability and scalability. It must run in the same python process that created the generator, and is still subject to the Python [GIL](https://en.wikipedia.org/wiki/Global_interpreter_lock)."
       ]
     },
     {
@@ -2631,7 +2631,7 @@
         "\n",
         "`data.Dataset.rejection_resample` takes a `class_func` argument. This `class_func` is applied to each dataset element, and is used to determine which class an example belongs to for the purposes of balancing.\n",
         "\n",
-        "The goal here is to balance the lable distribution, and the elements of `creditcard_ds` are already `(features, label)` pairs. So the `class_func` just needs to return those labels:"
+        "The goal here is to balance the label distribution, and the elements of `creditcard_ds` are already `(features, label)` pairs. So the `class_func` just needs to return those labels:"
       ]
     },
     {
diff --git a/site/en/guide/dtensor_overview.ipynb b/site/en/guide/dtensor_overview.ipynb
index 39ca1025134..3dcbe06698a 100644
--- a/site/en/guide/dtensor_overview.ipynb
+++ b/site/en/guide/dtensor_overview.ipynb
@@ -225,7 +225,7 @@
         "\n",
         "Note: In order to avoid confusions between `Mesh` and `Layout`, the term *dimension* is always associated with `Mesh`, and the term *axis* with `Tensor` and `Layout` in this guide.\n",
         "\n",
-        "The rank of `Layout` should be the same as the rank of the `Tensor` where the `Layout` is applied. For each of the `Tensor`'s axes the `Layout` may specifiy a mesh dimension to shard the tensor across, or specify the axis as \"unsharded\".\n",
+        "The rank of `Layout` should be the same as the rank of the `Tensor` where the `Layout` is applied. For each of the `Tensor`'s axes the `Layout` may specify a mesh dimension to shard the tensor across, or specify the axis as \"unsharded\".\n",
         "The tensor is replicated across any mesh dimensions that it is not sharded across.\n",
         "\n",
         "The rank of a `Layout` and the number of dimensions of a `Mesh` do not need to match. The `unsharded` axes of a `Layout` do not need to be associated to a mesh dimension, and `unsharded` mesh dimensions do not need to be associated with a `layout` axis.\n",
@@ -528,7 +528,7 @@
         "\n",
         "So far you've worked with the `my_first_dtensor`, which is a rank-1 DTensor fully replicated across a dim-1 `Mesh`.\n",
         "\n",
-        "Next create and inspect DTensors that are sharded across a dim-2 `Mesh`. The next example does this with a 3x2 `Mesh` on 6 CPU devices, where size of mesh dimension `'x'` is 3 devices, and and size of mesh dimension`'y'` is 2 devices."
+        "Next create and inspect DTensors that are sharded across a dim-2 `Mesh`. The next example does this with a 3x2 `Mesh` on 6 CPU devices, where size of mesh dimension `'x'` is 3 devices, and size of mesh dimension`'y'` is 2 devices."
       ]
     },
     {
@@ -696,7 +696,7 @@
         "To accomplish this, for each [TensorFlow Graph](https://www.tensorflow.org/guide/intro_to_graphs), DTensor produces and executes an equivalent [SPMD](https://en.wikipedia.org/wiki/SPMD) graph in a procedure called *SPMD expansion*. A few critical steps in DTensor SPMD expansion are:\n",
         "\n",
         "  - Propagating the sharding `Layout` of DTensor in the TensorFlow graph\n",
-        "  - Rewriting TensorFlow Ops on the global DTensor with equivalent TensorFlow Ops on the componenent tensors, inserting collective and communication Ops when necessary\n",
+        "  - Rewriting TensorFlow Ops on the global DTensor with equivalent TensorFlow Ops on the component tensors, inserting collective and communication Ops when necessary\n",
         "  - Lowering backend neutral TensorFlow Ops to backend specific TensorFlow Ops.\n",
         "\n",
         "The final result is that **DTensor is a drop-in replacement for Tensor**.\n",
diff --git a/site/en/guide/graph_optimization.ipynb b/site/en/guide/graph_optimization.ipynb
index 51de1efda93..fa573e67800 100644
--- a/site/en/guide/graph_optimization.ipynb
+++ b/site/en/guide/graph_optimization.ipynb
@@ -90,7 +90,7 @@
         "* *Constant folding optimizer -* Statically infers the value of tensors when possible by folding constant nodes in the graph and materializes the result using constants.\n",
         "* *Arithmetic optimizer -* Simplifies arithmetic operations by eliminating common subexpressions and simplifying arithmetic statements.  \n",
         "* *Layout optimizer -* Optimizes tensor layouts to execute data format dependent operations such as convolutions more efficiently.\n",
-        "* *Remapper optimizer -* Remaps subgraphs onto more efficient implementations by replacing commonly occuring subgraphs with optimized fused monolithic kernels.\n",
+        "* *Remapper optimizer -* Remaps subgraphs onto more efficient implementations by replacing commonly occurring subgraphs with optimized fused monolithic kernels.\n",
         "* *Memory optimizer -* Analyzes the graph to inspect the peak memory usage for each operation and inserts CPU-GPU memory copy operations for swapping GPU memory to CPU to reduce the peak memory usage.\n",
         "* *Dependency optimizer -* Removes or rearranges control dependencies to shorten the critical path for a model step or enables other\n",
         "optimizations. Also removes nodes that are effectively no-ops such as Identity.\n",
diff --git a/site/en/guide/intro_to_graphs.ipynb b/site/en/guide/intro_to_graphs.ipynb
index 19b5c5f432e..bcad05b5a33 100644
--- a/site/en/guide/intro_to_graphs.ipynb
+++ b/site/en/guide/intro_to_graphs.ipynb
@@ -676,7 +676,7 @@
         "  tf.gather(x, [1]) # unused\n",
         "  return x\n",
         "\n",
-        "# Only needed operations are run during graph exection. The error is not raised.\n",
+        "# Only needed operations are run during graph execution. The error is not raised.\n",
         "print(unused_return_graph(tf.constant([0.0])))"
       ]
     },
diff --git a/site/en/guide/migrate/canned_estimators.ipynb b/site/en/guide/migrate/canned_estimators.ipynb
index 66d688b7676..37e9dbe8512 100644
--- a/site/en/guide/migrate/canned_estimators.ipynb
+++ b/site/en/guide/migrate/canned_estimators.ipynb
@@ -718,7 +718,7 @@
         "id": "Z0QYolhoZb_k"
       },
       "source": [
-        "Finaly, in the next example, we train and evaluate a CART model."
+        "Finally, in the next example, we train and evaluate a CART model."
       ]
     },
     {
diff --git a/site/en/guide/migrate/evaluator.ipynb b/site/en/guide/migrate/evaluator.ipynb
index 3588838467c..e81863edb81 100644
--- a/site/en/guide/migrate/evaluator.ipynb
+++ b/site/en/guide/migrate/evaluator.ipynb
@@ -193,7 +193,7 @@
       "source": [
         "## TensorFlow 2: Evaluating a Keras model\n",
         "\n",
-        "In TensorFlow 2, if you use the Keras `Model.fit` API for training, you can evaluate the model with `tf.keras.utils.SidecarEvaluator`. You can also visualize the evaluation metrics in Tensorboard which is not shown in this guide.\n",
+        "In TensorFlow 2, if you use the Keras `Model.fit` API for training, you can evaluate the model with `tf.keras.utils.SidecarEvaluator`. You can also visualize the evaluation metrics in TensorBoard which is not shown in this guide.\n",
         "\n",
         "To help demonstrate this, let's first start by defining and training the model:\n"
       ]
diff --git a/site/en/guide/migrate/migrating_feature_columns.ipynb b/site/en/guide/migrate/migrating_feature_columns.ipynb
index 65aa1a786b8..62f91932935 100644
--- a/site/en/guide/migrate/migrating_feature_columns.ipynb
+++ b/site/en/guide/migrate/migrating_feature_columns.ipynb
@@ -961,7 +961,7 @@
         "\n",
         "\\* `output_mode` can be passed to `layers.CategoryEncoding`, `layers.StringLookup`, `layers.IntegerLookup`, and `layers.TextVectorization`.\n",
         "\n",
-        "† `layers.TextVectorization` can handle freeform text input directly (e.g. entire sentences or paragraphs). This is not one-to-one replacement for categorical sequence handling in TF1, but may offer a convinient replacement for ad-hoc text preprocessing.\n",
+        "† `layers.TextVectorization` can handle freeform text input directly (e.g. entire sentences or paragraphs). This is not one-to-one replacement for categorical sequence handling in TF1, but may offer a convenient replacement for ad-hoc text preprocessing.\n",
         "\n",
         "Note: Linear estimators, such as `tf.estimator.LinearClassifier`, can handle direct categorical input (integer indices) without an `embedding_column` or `indicator_column`. However, integer indices cannot be passed directly to `tf.keras.layers.Dense` or `tf.keras.experimental.LinearModel`. These inputs should be first encoded with `tf.layers.CategoryEncoding` with `output_mode='count'` (and `sparse=True` if the category sizes are large) before calling into `Dense` or `LinearModel`."
       ]
diff --git a/site/en/guide/migrate/migration_debugging.ipynb b/site/en/guide/migrate/migration_debugging.ipynb
index 774fca2160b..6a2afa27425 100644
--- a/site/en/guide/migrate/migration_debugging.ipynb
+++ b/site/en/guide/migrate/migration_debugging.ipynb
@@ -486,7 +486,7 @@
       "source": [
         "Compare numerical equivalence for first few training steps.\n",
         "\n",
-        "You can also check the [Validating correctness & numerical equivalence notebook](./validate_correctness.ipynb) for additonal advice for numerical equivalence."
+        "You can also check the [Validating correctness & numerical equivalence notebook](./validate_correctness.ipynb) for additional advice for numerical equivalence."
       ]
     },
     {
diff --git a/site/en/guide/migrate/tf1_vs_tf2.ipynb b/site/en/guide/migrate/tf1_vs_tf2.ipynb
index 80bb11bd530..f4b2cd85b62 100644
--- a/site/en/guide/migrate/tf1_vs_tf2.ipynb
+++ b/site/en/guide/migrate/tf1_vs_tf2.ipynb
@@ -340,7 +340,7 @@
         "  print(e) #  is out of scope and cannot be used here.\n",
         "```\n",
         "\n",
-        "The most straightfoward solution is ensuring that the variable creation and dataset creation are both outside of the `tf.funciton` call. For example:\n",
+        "The most straightfoward solution is ensuring that the variable creation and dataset creation are both outside of the `tf.function` call. For example:\n",
         "\n",
         "```python\n",
         "class Model(tf.Module):\n",
diff --git a/site/en/guide/migrate/upgrade.ipynb b/site/en/guide/migrate/upgrade.ipynb
index 34d00aaf5a7..7223a8c8c81 100644
--- a/site/en/guide/migrate/upgrade.ipynb
+++ b/site/en/guide/migrate/upgrade.ipynb
@@ -108,7 +108,7 @@
       "source": [
         "## Recommended upgrade process\n",
         "\n",
-        "The rest of this guide demonstrates how to use the symbol-rewriting script. While the script is easy to use, it is strongly recomended that you use the script as part of the following process: \n",
+        "The rest of this guide demonstrates how to use the symbol-rewriting script. While the script is easy to use, it is strongly recommended that you use the script as part of the following process: \n",
         "\n",
         "1. **Unit Test**: Ensure that the code you’re upgrading has a unit test suite with reasonable coverage. This is Python code, so the language won’t protect you from many classes of mistakes. Also ensure that any dependency you have has already been upgraded to be compatible with TensorFlow 2.x.\n",
         "\n",
diff --git a/site/en/guide/ragged_tensor.ipynb b/site/en/guide/ragged_tensor.ipynb
index aa4774d4e10..d36010699db 100644
--- a/site/en/guide/ragged_tensor.ipynb
+++ b/site/en/guide/ragged_tensor.ipynb
@@ -1440,7 +1440,7 @@
         "\n",
         "1.  Use `tf.RaggedTensor.to_list` to convert the ragged tensor to a nested Python list.\n",
         "2.  Use `tf.RaggedTensor.numpy` to convert the ragged tensor to a NumPy array whose values are nested NumPy arrays.\n",
-        "3.  Decompose the ragged tensor into its components, using the `tf.RaggedTensor.values` and `tf.RaggedTensor.row_splits` properties, or row-paritioning methods such as `tf.RaggedTensor.row_lengths` and `tf.RaggedTensor.value_rowids`.\n",
+        "3.  Decompose the ragged tensor into its components, using the `tf.RaggedTensor.values` and `tf.RaggedTensor.row_splits` properties, or row-partitioning methods such as `tf.RaggedTensor.row_lengths` and `tf.RaggedTensor.value_rowids`.\n",
         "4.  Use Python indexing to select values from the ragged tensor.\n"
       ]
     },
diff --git a/site/en/guide/random_numbers.ipynb b/site/en/guide/random_numbers.ipynb
index 37c83ae76a0..5212a10a49a 100644
--- a/site/en/guide/random_numbers.ipynb
+++ b/site/en/guide/random_numbers.ipynb
@@ -268,7 +268,7 @@
       "source": [
         "Note: In theory, you can use constructors such as `from_seed` instead of `split` here to obtain a new generator, but by doing so you lose the guarantee that the new generator is independent of the global generator. You will also run the risk that you may accidentally create two generators with the same seed or with seeds that lead to overlapping random-number streams.\n",
         "\n",
-        "You can do splitting recursively, calling `split` on splitted generators. There are no limits (barring integer overflow) on the depth of recursions."
+        "You can do splitting recursively, calling `split` on split generators. There are no limits (barring integer overflow) on the depth of recursions."
       ]
     },
     {
@@ -325,7 +325,7 @@
       "source": [
         "#### Creating generators inside `tf.function` \n",
         "\n",
-        "Creation of generators inside a `tf.function` can only happend during the first run of the function. "
+        "Creation of generators inside a `tf.function` can only happened during the first run of the function. "
       ]
     },
     {
diff --git a/site/en/r1/guide/autograph.ipynb b/site/en/r1/guide/autograph.ipynb
index 5d8d7c97999..f028b33ce9f 100644
--- a/site/en/r1/guide/autograph.ipynb
+++ b/site/en/r1/guide/autograph.ipynb
@@ -66,7 +66,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/guide/distribute_strategy.ipynb b/site/en/r1/guide/distribute_strategy.ipynb
index f6d85912e16..79d6293eba7 100644
--- a/site/en/r1/guide/distribute_strategy.ipynb
+++ b/site/en/r1/guide/distribute_strategy.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
@@ -223,7 +223,7 @@
         "id": "KY1nJHNkMl7b"
       },
       "source": [
-        "This will create a `CentralStorageStrategy` instance which will use all visible GPUs and CPU. Update to variables on replicas will be aggragated before being applied to variables."
+        "This will create a `CentralStorageStrategy` instance which will use all visible GPUs and CPU. Update to variables on replicas will be aggregated before being applied to variables."
       ]
     },
     {
diff --git a/site/en/r1/guide/eager.ipynb b/site/en/r1/guide/eager.ipynb
index 230974ab5a4..6a0a78c2443 100644
--- a/site/en/r1/guide/eager.ipynb
+++ b/site/en/r1/guide/eager.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/guide/keras.ipynb b/site/en/r1/guide/keras.ipynb
index 08a778b60a5..d722660a59c 100644
--- a/site/en/r1/guide/keras.ipynb
+++ b/site/en/r1/guide/keras.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/guide/ragged_tensors.ipynb b/site/en/r1/guide/ragged_tensors.ipynb
index 61bce66ecfb..289d29ce82e 100644
--- a/site/en/r1/guide/ragged_tensors.ipynb
+++ b/site/en/r1/guide/ragged_tensors.ipynb
@@ -57,7 +57,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
@@ -1010,7 +1010,7 @@
         "    `tf.RaggedTensor.values`\n",
         "    and\n",
         "    `tf.RaggedTensor.row_splits`\n",
-        "    properties, or row-paritioning methods such as `tf.RaggedTensor.row_lengths()`\n",
+        "    properties, or row-partitioning methods such as `tf.RaggedTensor.row_lengths()`\n",
         "    and `tf.RaggedTensor.value_rowids()`."
       ]
     },
diff --git a/site/en/r1/tutorials/_index.ipynb b/site/en/r1/tutorials/_index.ipynb
index e2fe960d125..eca1450964f 100644
--- a/site/en/r1/tutorials/_index.ipynb
+++ b/site/en/r1/tutorials/_index.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/distribute/keras.ipynb b/site/en/r1/tutorials/distribute/keras.ipynb
index b8d3c87bfab..6b5258ee5a0 100644
--- a/site/en/r1/tutorials/distribute/keras.ipynb
+++ b/site/en/r1/tutorials/distribute/keras.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
@@ -345,7 +345,7 @@
       "source": [
         "The callbacks used here are:\n",
         "\n",
-        "*   *Tensorboard*: This callback writes a log for Tensorboard which allows you to visualize the graphs.\n",
+        "*   *TensorBoard*: This callback writes a log for TensorBoard which allows you to visualize the graphs.\n",
         "*   *Model Checkpoint*: This callback saves the model after every epoch.\n",
         "*   *Learning Rate Scheduler*: Using this callback, you can schedule the learning rate to change after every epoch/batch.\n",
         "\n",
diff --git a/site/en/r1/tutorials/distribute/tpu_custom_training.ipynb b/site/en/r1/tutorials/distribute/tpu_custom_training.ipynb
index 6d09d2623de..c61f893ca4c 100644
--- a/site/en/r1/tutorials/distribute/tpu_custom_training.ipynb
+++ b/site/en/r1/tutorials/distribute/tpu_custom_training.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/distribute/training_loops.ipynb b/site/en/r1/tutorials/distribute/training_loops.ipynb
index 1343e8c8b6b..8eb72c13030 100644
--- a/site/en/r1/tutorials/distribute/training_loops.ipynb
+++ b/site/en/r1/tutorials/distribute/training_loops.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/eager/automatic_differentiation.ipynb b/site/en/r1/tutorials/eager/automatic_differentiation.ipynb
index bbbb689a617..df843bac3b8 100644
--- a/site/en/r1/tutorials/eager/automatic_differentiation.ipynb
+++ b/site/en/r1/tutorials/eager/automatic_differentiation.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/eager/custom_layers.ipynb b/site/en/r1/tutorials/eager/custom_layers.ipynb
index c82458cb857..954601e371b 100644
--- a/site/en/r1/tutorials/eager/custom_layers.ipynb
+++ b/site/en/r1/tutorials/eager/custom_layers.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/eager/custom_training.ipynb b/site/en/r1/tutorials/eager/custom_training.ipynb
index 72beefe89ad..f0f7faffa7f 100644
--- a/site/en/r1/tutorials/eager/custom_training.ipynb
+++ b/site/en/r1/tutorials/eager/custom_training.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/eager/custom_training_walkthrough.ipynb b/site/en/r1/tutorials/eager/custom_training_walkthrough.ipynb
index a4839429827..3989f3e44bc 100644
--- a/site/en/r1/tutorials/eager/custom_training_walkthrough.ipynb
+++ b/site/en/r1/tutorials/eager/custom_training_walkthrough.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/eager/eager_basics.ipynb b/site/en/r1/tutorials/eager/eager_basics.ipynb
index 90d7c02f18d..acd00ec2e20 100644
--- a/site/en/r1/tutorials/eager/eager_basics.ipynb
+++ b/site/en/r1/tutorials/eager/eager_basics.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/images/hub_with_keras.ipynb b/site/en/r1/tutorials/images/hub_with_keras.ipynb
index ece9c0fa4a9..5572fd237d9 100644
--- a/site/en/r1/tutorials/images/hub_with_keras.ipynb
+++ b/site/en/r1/tutorials/images/hub_with_keras.ipynb
@@ -60,7 +60,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/images/transfer_learning.ipynb b/site/en/r1/tutorials/images/transfer_learning.ipynb
index c695da4ebb7..25779babd17 100644
--- a/site/en/r1/tutorials/images/transfer_learning.ipynb
+++ b/site/en/r1/tutorials/images/transfer_learning.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/keras/basic_classification.ipynb b/site/en/r1/tutorials/keras/basic_classification.ipynb
index be7f5e9e8b1..14950538ce4 100644
--- a/site/en/r1/tutorials/keras/basic_classification.ipynb
+++ b/site/en/r1/tutorials/keras/basic_classification.ipynb
@@ -96,7 +96,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/keras/basic_regression.ipynb b/site/en/r1/tutorials/keras/basic_regression.ipynb
index 7d9cb711efa..4bffd62f982 100644
--- a/site/en/r1/tutorials/keras/basic_regression.ipynb
+++ b/site/en/r1/tutorials/keras/basic_regression.ipynb
@@ -96,7 +96,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/keras/basic_text_classification.ipynb b/site/en/r1/tutorials/keras/basic_text_classification.ipynb
index 0303d54d973..5424185bcbd 100644
--- a/site/en/r1/tutorials/keras/basic_text_classification.ipynb
+++ b/site/en/r1/tutorials/keras/basic_text_classification.ipynb
@@ -96,7 +96,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/keras/overfit_and_underfit.ipynb b/site/en/r1/tutorials/keras/overfit_and_underfit.ipynb
index a8f266f9869..8e35b06e556 100644
--- a/site/en/r1/tutorials/keras/overfit_and_underfit.ipynb
+++ b/site/en/r1/tutorials/keras/overfit_and_underfit.ipynb
@@ -96,7 +96,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/keras/save_and_restore_models.ipynb b/site/en/r1/tutorials/keras/save_and_restore_models.ipynb
index 7911e37e139..e9d112bd3f3 100644
--- a/site/en/r1/tutorials/keras/save_and_restore_models.ipynb
+++ b/site/en/r1/tutorials/keras/save_and_restore_models.ipynb
@@ -96,7 +96,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
@@ -698,7 +698,7 @@
         "id": "B7qfpvpY9HCe"
       },
       "source": [
-        "Load the the saved model."
+        "Load the saved model."
       ]
     },
     {
diff --git a/site/en/r1/tutorials/load_data/images.ipynb b/site/en/r1/tutorials/load_data/images.ipynb
index dbee204323b..923b95130d1 100644
--- a/site/en/r1/tutorials/load_data/images.ipynb
+++ b/site/en/r1/tutorials/load_data/images.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/load_data/tf_records.ipynb b/site/en/r1/tutorials/load_data/tf_records.ipynb
index 8b57d3f2f1e..64631cf0eab 100644
--- a/site/en/r1/tutorials/load_data/tf_records.ipynb
+++ b/site/en/r1/tutorials/load_data/tf_records.ipynb
@@ -57,7 +57,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/non-ml/mandelbrot.ipynb b/site/en/r1/tutorials/non-ml/mandelbrot.ipynb
index 88177211896..bca8a142be4 100644
--- a/site/en/r1/tutorials/non-ml/mandelbrot.ipynb
+++ b/site/en/r1/tutorials/non-ml/mandelbrot.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/non-ml/pdes.ipynb b/site/en/r1/tutorials/non-ml/pdes.ipynb
index d2646daa8da..832fa450523 100644
--- a/site/en/r1/tutorials/non-ml/pdes.ipynb
+++ b/site/en/r1/tutorials/non-ml/pdes.ipynb
@@ -64,7 +64,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/representation/unicode.ipynb b/site/en/r1/tutorials/representation/unicode.ipynb
index 6762a483a42..98aaacff5b9 100644
--- a/site/en/r1/tutorials/representation/unicode.ipynb
+++ b/site/en/r1/tutorials/representation/unicode.ipynb
@@ -57,7 +57,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/r1/tutorials/sequences/text_generation.ipynb b/site/en/r1/tutorials/sequences/text_generation.ipynb
index 423a7f659f0..1350df2a486 100644
--- a/site/en/r1/tutorials/sequences/text_generation.ipynb
+++ b/site/en/r1/tutorials/sequences/text_generation.ipynb
@@ -65,7 +65,7 @@
       "source": [
         "> Note: This is an archived TF1 notebook. These are configured\n",
         "to run in TF2's \n",
-        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
+        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
         "but will run in TF1 as well. To use TF1 in Colab, use the\n",
         "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
         "magic."
diff --git a/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb
index d6bfa26544e..b09348d9264 100644
--- a/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb
+++ b/site/en/tutorials/distribute/dtensor_keras_tutorial.ipynb
@@ -180,7 +180,7 @@
         "\n",
         "Data Parallel training is a commonly used parallel training scheme, also used by for example `tf.distribute.MirroredStrategy`.\n",
         "\n",
-        "With DTensor, a Data Parallel training loop uses a `Mesh` that consists of a single 'batch' dimension, where each device runs a replica of the model that recieves a shard from the global batch.\n"
+        "With DTensor, a Data Parallel training loop uses a `Mesh` that consists of a single 'batch' dimension, where each device runs a replica of the model that receives a shard from the global batch.\n"
       ]
     },
     {
diff --git a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
index 4fdb826afab..4a0f7bd60f6 100644
--- a/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
+++ b/site/en/tutorials/distribute/dtensor_ml_tutorial.ipynb
@@ -514,7 +514,7 @@
       "source": [
         "## Moving data to the device\n",
         "\n",
-        "Usually, `tf.data` iterators (and other data fetching methods) yield tensor objects backed by the local host device memory. This data must be transfered to the accelerator device memory that backs DTensor's component tensors.\n",
+        "Usually, `tf.data` iterators (and other data fetching methods) yield tensor objects backed by the local host device memory. This data must be transferred to the accelerator device memory that backs DTensor's component tensors.\n",
         "\n",
         "`dtensor.copy_to_mesh` is unsuitable for this situation because it replicates input tensors to all devices due to DTensor's global perspective. So in this tutorial, you will use a helper function `repack_local_tensor`, to facilitate the transfer of data. This helper function uses `dtensor.pack` to send (and only send) the shard of the global batch that is intended for a replica to the device backing the replica.\n",
         "\n",
@@ -655,7 +655,7 @@
         "\n",
         "This example uses a Stochastic Gradient Descent optimizer with the Custom Training Loop (CTL). Consult the [Custom Training Loop guide](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) and [Walk through](https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough) for more information on those topics.\n",
         "\n",
-        "The `train_step` is encapsulated as a `tf.funtion` to indicate this body is to be traced as a TensorFlow Graph. The body of `train_step` consists of a forward inference pass, a backward gradient pass, and the variable update.\n",
+        "The `train_step` is encapsulated as a `tf.function` to indicate this body is to be traced as a TensorFlow Graph. The body of `train_step` consists of a forward inference pass, a backward gradient pass, and the variable update.\n",
         "\n",
         "Note that the body of `train_step` does not contain any special DTensor annotations. Instead, `train_step` only contains high-level TensorFlow operations that process the input `x` and `y` from the global view of the input batch and the model. All of the DTensor annotations (`Mesh`, `Layout`) are factored out of the train step."
       ]
diff --git a/site/en/tutorials/generative/cyclegan.ipynb b/site/en/tutorials/generative/cyclegan.ipynb
index 7136dd143ef..4dae5b77791 100644
--- a/site/en/tutorials/generative/cyclegan.ipynb
+++ b/site/en/tutorials/generative/cyclegan.ipynb
@@ -154,7 +154,7 @@
         "This is similar to what was done in [pix2pix](https://www.tensorflow.org/tutorials/generative/pix2pix#load_the_dataset)\n",
         "\n",
         "* In random jittering, the image is resized to `286 x 286` and then randomly cropped to `256 x 256`.\n",
-        "* In random mirroring, the image is randomly flipped horizontally i.e left to right."
+        "* In random mirroring, the image is randomly flipped horizontally i.e. left to right."
       ]
     },
     {
diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
index bd467381e26..974d3f5dc01 100644
--- a/site/en/tutorials/generative/data_compression.ipynb
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -314,7 +314,7 @@
         "id": "V8IvuFkrRJIa"
       },
       "source": [
-        "To get the latent representation $y$, we need to cast it to `float32`, add a batch dimension, and pass it throught the analysis transform."
+        "To get the latent representation $y$, we need to cast it to `float32`, add a batch dimension, and pass it through the analysis transform."
       ]
     },
     {
diff --git a/site/en/tutorials/images/transfer_learning.ipynb b/site/en/tutorials/images/transfer_learning.ipynb
index 7174ae31693..09f9705a038 100644
--- a/site/en/tutorials/images/transfer_learning.ipynb
+++ b/site/en/tutorials/images/transfer_learning.ipynb
@@ -1003,7 +1003,7 @@
         "id": "PSXH7PRMxOi5"
       },
       "source": [
-        "Finaly you can verify the performance of the model on new data using test set."
+        "Finally you can verify the performance of the model on new data using test set."
       ]
     },
     {
diff --git a/site/en/tutorials/load_data/tfrecord.ipynb b/site/en/tutorials/load_data/tfrecord.ipynb
index 27c3c372b35..f5d2ffad878 100644
--- a/site/en/tutorials/load_data/tfrecord.ipynb
+++ b/site/en/tutorials/load_data/tfrecord.ipynb
@@ -884,7 +884,7 @@
         "           List[str]]]\n",
         "```\n",
         "\n",
-        "The following code manually converts the `Example` to a dictionary of NumPy arrays, without using TensorFlow Ops. Refer to [the PROTO file](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/feature.proto) for detials."
+        "The following code manually converts the `Example` to a dictionary of NumPy arrays, without using TensorFlow Ops. Refer to [the PROTO file](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/feature.proto) for details."
       ]
     },
     {
diff --git a/site/en/tutorials/structured_data/imbalanced_data.ipynb b/site/en/tutorials/structured_data/imbalanced_data.ipynb
index ffce3927e0b..4d7e0f22cab 100644
--- a/site/en/tutorials/structured_data/imbalanced_data.ipynb
+++ b/site/en/tutorials/structured_data/imbalanced_data.ipynb
@@ -234,7 +234,7 @@
         "\n",
         "# The `Amount` column covers a huge range. Convert to log-space.\n",
         "eps = 0.001 # 0 => 0.1¢\n",
-        "cleaned_df['Log Ammount'] = np.log(cleaned_df.pop('Amount')+eps)"
+        "cleaned_df['Log Amount'] = np.log(cleaned_df.pop('Amount')+eps)"
       ]
     },
     {
diff --git a/site/en/tutorials/text/image_captioning.ipynb b/site/en/tutorials/text/image_captioning.ipynb
index 8613be7150a..df54e172796 100644
--- a/site/en/tutorials/text/image_captioning.ipynb
+++ b/site/en/tutorials/text/image_captioning.ipynb
@@ -364,7 +364,7 @@
         "You will transform the text captions into integer sequences using the [TextVectorization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization) layer, with the following steps:\n",
         "\n",
         "* Use [adapt](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization#adapt) to iterate over all captions, split the captions into words, and compute a vocabulary of the top 5,000 words (to save memory).\n",
-        "* Tokenize all captions by mapping each word to it's index in the vocabulary. All output sequences will be padded to length 50.\n",
+        "* Tokenize all captions by mapping each word to its index in the vocabulary. All output sequences will be padded to length 50.\n",
         "* Create word-to-index and index-to-word mappings to display results."
       ]
     },
@@ -417,7 +417,7 @@
       },
       "outputs": [],
       "source": [
-        "# Create mappings for words to indices and indicies to words.\n",
+        "# Create mappings for words to indices and indices to words.\n",
         "word_to_index = tf.keras.layers.StringLookup(\n",
         "    mask_token=\"\",\n",
         "    vocabulary=tokenizer.get_vocabulary())\n",
diff --git a/site/en/tutorials/text/word2vec.ipynb b/site/en/tutorials/text/word2vec.ipynb
index e3de292d653..113b9419662 100644
--- a/site/en/tutorials/text/word2vec.ipynb
+++ b/site/en/tutorials/text/word2vec.ipynb
@@ -1249,7 +1249,7 @@
         "id": "P3MUMrluqNX2"
       },
       "source": [
-        "Also define a callback to log training statistics for Tensorboard:"
+        "Also define a callback to log training statistics for TensorBoard:"
       ]
     },
     {
@@ -1289,7 +1289,7 @@
         "id": "wze38jG57XvZ"
       },
       "source": [
-        "Tensorboard now shows the word2vec model's accuracy and loss:"
+        "TensorBoard now shows the word2vec model's accuracy and loss:"
       ]
     },
     {
diff --git a/site/en/tutorials/understanding/sngp.ipynb b/site/en/tutorials/understanding/sngp.ipynb
index 7675f69e27f..3df14d852ab 100644
--- a/site/en/tutorials/understanding/sngp.ipynb
+++ b/site/en/tutorials/understanding/sngp.ipynb
@@ -123,7 +123,7 @@
         "\n",
         "* The predictive uncertainty of a SNGP is computed using the [Laplace approximation](http://www.gaussianprocess.org/gpml/chapters/RW3.pdf). Therefore theoretically, the posterior uncertainty of SNGP is different from that of an exact Gaussian process.\n",
         "\n",
-        "* SNGP training needs a covariance reset step at the begining of a new epoch. This can add a tiny amount of extra complexity to a training pipeline. This tutorial shows a simple way to implement this using Keras callbacks."
+        "* SNGP training needs a covariance reset step at the beginning of a new epoch. This can add a tiny amount of extra complexity to a training pipeline. This tutorial shows a simple way to implement this using Keras callbacks."
       ]
     },
     {
@@ -315,7 +315,7 @@
         "plt.scatter(neg_examples[:, 0], neg_examples[:, 1], c=\"#ff7f00\", alpha=0.5)\n",
         "plt.scatter(ood_examples[:, 0], ood_examples[:, 1], c=\"red\", alpha=0.1)\n",
         "\n",
-        "plt.legend([\"Postive\", \"Negative\", \"Out-of-Domain\"])\n",
+        "plt.legend([\"Positive\", \"Negative\", \"Out-of-Domain\"])\n",
         "\n",
         "plt.ylim(DEFAULT_Y_RANGE)\n",
         "plt.xlim(DEFAULT_X_RANGE)\n",
@@ -1008,7 +1008,7 @@
         "class ResetCovarianceCallback(tf.keras.callbacks.Callback):\n",
         "\n",
         "  def on_epoch_begin(self, epoch, logs=None):\n",
-        "    \"\"\"Resets covariance matrix at the begining of the epoch.\"\"\"\n",
+        "    \"\"\"Resets covariance matrix at the beginning of the epoch.\"\"\"\n",
         "    if epoch > 0:\n",
         "      self.model.classifier.reset_covariance_matrix()"
       ]
diff --git a/tools/tensorflow_docs/api_generator/doc_controls.py b/tools/tensorflow_docs/api_generator/doc_controls.py
index 5cdeaf6eafc..ead9019553f 100644
--- a/tools/tensorflow_docs/api_generator/doc_controls.py
+++ b/tools/tensorflow_docs/api_generator/doc_controls.py
@@ -352,8 +352,8 @@ def doc_in_current_and_subclasses(obj: T) -> T:
   """Overrides `do_not_doc_in_subclasses` decorator.
 
   If this decorator is set on a child class's method whose parent's method
-  contains `do_not_doc_in_subclasses`, then that will be overriden and the
-  child method will get documented. All classes inherting from the child will
+  contains `do_not_doc_in_subclasses`, then that will be overridden and the
+  child method will get documented. All classes inheriting from the child will
   also document that method.
 
   For example:
diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
index fec792095bd..fa7a23c33a8 100644
--- a/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
+++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor.py
@@ -504,7 +504,7 @@ class ApiTree(Dict[ApiPath, ApiTreeNode]):
   node = index.node_from_obj(obj)
   ```
 
-  Remember that `maybe_singelton` (numbers, strings, tuples) classes can't be
+  Remember that `maybe_singleton` (numbers, strings, tuples) classes can't be
   looked up this way.
 
   To build a tree, nodes must be inserted in tree order starting from the root.
diff --git a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py
index f26d954d94b..62b6a25de01 100644
--- a/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py
+++ b/tools/tensorflow_docs/api_generator/doc_generator_visitor_test.py
@@ -505,7 +505,7 @@ def test_api_tree(self):
     # Test lookup by object.
     self.assertIs(api_tree[('tf', 'Parent')],
                   api_tree.node_for_object(tf.Parent))
-    # You can't lookup things that maybe singeltons.
+    # You can't lookup things that maybe singletons.
     self.assertIs(api_tree[('tf', 'seven')].py_object, tf.seven)
     self.assertIsNone(api_tree.node_for_object(tf.seven))
 
@@ -601,7 +601,7 @@ class Class1:
     path_tree[('mod', 'b', 'Class1')] = mod.b.Class1
 
     def inconsistent_name_score(path):
-      # `mod.a` is prefered over `mod.b`, but `b.Class1` is prefered over
+      # `mod.a` is preferred over `mod.b`, but `b.Class1` is preferred over
       # `a.Class1`!
       scores = {
           ('mod',): 0,
diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py
index 9141827e1e6..4154eb7a456 100644
--- a/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py
+++ b/tools/tensorflow_docs/api_generator/pretty_docs/module_page.py
@@ -142,7 +142,7 @@ def collect_docs(self):
 
     Mainly this is information about the members of the module.
     """
-    # the path_tree has nodes for all api-paths, not just the prefered paths.
+    # the path_tree has nodes for all api-paths, not just the preferred paths.
     module_path_node = self.parser_config.path_tree[self.api_node.path]
     for (_, path_node) in sorted(module_path_node.children.items()):
       member_doc = parser.parse_md_docstring(path_node.py_object,
diff --git a/tools/tensorflow_docs/api_generator/reference_resolver.py b/tools/tensorflow_docs/api_generator/reference_resolver.py
index c124fff9c5a..268d23c97f7 100644
--- a/tools/tensorflow_docs/api_generator/reference_resolver.py
+++ b/tools/tensorflow_docs/api_generator/reference_resolver.py
@@ -238,7 +238,7 @@ def _create_partial_symbols_dict(self):
     return new_partial_dict
 
   def to_json_file(self, filepath):
-    """Converts the RefenceResolver to json and writes it to the specified file.
+    """Converts the ReferenceResolver to json and writes it to the specified file.
 
     Args:
       filepath: The file path to write the json to.
diff --git a/tools/tensorflow_docs/tools/nbfmt/__main__.py b/tools/tensorflow_docs/tools/nbfmt/__main__.py
index 9fd6b04e6b6..9426e6fd690 100644
--- a/tools/tensorflow_docs/tools/nbfmt/__main__.py
+++ b/tools/tensorflow_docs/tools/nbfmt/__main__.py
@@ -76,7 +76,7 @@ def clean_notebook(data: Dict[str, Any], nb_source: str, filepath: pathlib.Path,
   nbjson = json.dumps(data, sort_keys=True, ensure_ascii=False, indent=indent)
 
   if not OSS:
-    # Serialization differences in enviroments.
+    # Serialization differences in environments.
     str_replaces = {"<": r"\u003c", ">": r"\u003e", "&": r"\u0026"}
     for str_from, str_to in str_replaces.items():
       nbjson = nbjson.replace(str_from, str_to)
@@ -157,7 +157,7 @@ def _clean_metadata_colab(cell_metadata: Dict[str, Any],
 
   Remove all `metadata.colab` contents except for `metadata.colab.resources`, if
   present. The Colab resources are used to embed data within the notebook and
-  can be treated like output cells (kept unless explictly removed).
+  can be treated like output cells (kept unless explicitly removed).
 
   Args:
     cell_metadata: object representing the parsed JSON metadata from a cell.
diff --git a/tools/tensorflow_docs/tools/nblint/__main__.py b/tools/tensorflow_docs/tools/nblint/__main__.py
index 0fbac6f4caa..1411fbf4c7b 100644
--- a/tools/tensorflow_docs/tools/nblint/__main__.py
+++ b/tools/tensorflow_docs/tools/nblint/__main__.py
@@ -132,7 +132,7 @@ def add_styles(styles, excluded_lints, verbose):
         getattr(mem[1], "_lint") for mem in inspect.getmembers(mod, is_lint)
     ]
 
-    # Remove lints that have been explictly excluded at the command-line.
+    # Remove lints that have been explicitly excluded at the command-line.
     lints = [
         lint for lint in lints if f"{style}::{lint.name}" not in excluded_lints
     ]
diff --git a/tools/tensorflow_docs/tools/nblint/linter.py b/tools/tensorflow_docs/tools/nblint/linter.py
index b3df30af09a..68be9d7b549 100644
--- a/tools/tensorflow_docs/tools/nblint/linter.py
+++ b/tools/tensorflow_docs/tools/nblint/linter.py
@@ -94,7 +94,7 @@ def _run_lint(self, lint: decorator.Lint, lint_args: Dict[str, Any],
     """Run lint and capture any stderr output for the status display.
 
     Args:
-      lint: `decorator.Lint` containg the assertion, scope, and condition.
+      lint: `decorator.Lint` containing the assertion, scope, and condition.
       lint_args: Nested dictionary of args to pass the lint callback function.
       status: The `LinterStatus` to add individual entries for group members.
 
@@ -131,7 +131,7 @@ def _run_lint_group(self, lint: decorator.Lint, lint_args: Dict[str, Any],
     """Run lint over all cells with scope and return cumulative pass/fail.
 
     Args:
-      lint: `decorator.Lint` containg the assertion, scope, and condition.
+      lint: `decorator.Lint` containing the assertion, scope, and condition.
       lint_args: Nested dictionary of args to pass the lint callback function.
       data: `dict` containing data of entire parse notebook.
       status: The `LinterStatus` to add individual entries for group members.
diff --git a/tools/tensorflow_docs/tools/nblint/style/google.py b/tools/tensorflow_docs/tools/nblint/style/google.py
index ef2e56a43ca..7619267c1a0 100644
--- a/tools/tensorflow_docs/tools/nblint/style/google.py
+++ b/tools/tensorflow_docs/tools/nblint/style/google.py
@@ -14,7 +14,7 @@
 # ==============================================================================
 r"""Lint assertions that adhere to the Google dev docs style guide.
 
-This style module is a non-exhaustive implemention of style rules found in the
+This style module is a non-exhaustive implementation of style rules found in the
 Google developer documentation style guide: https://developers.google.com/style
 
 When adding lints, please link to the URL of the relevant style rule.
@@ -38,7 +38,7 @@ def search_wordlist(wordlist, src_str):
   """
   found_words = {}
   for word in wordlist:
-    # Word-boundary and ignore between path seperator '/'.
+    # Word-boundary and ignore between path separator '/'.
     if re.search(rf"[^/]\b{word}\b[^/]", src_str, re.IGNORECASE):
       alt_word = wordlist[word]
       if not alt_word:
diff --git a/tools/tensorflow_docs/tools/nblint/style/tensorflow.py b/tools/tensorflow_docs/tools/nblint/style/tensorflow.py
index c192deabce2..5dd591ac0b1 100644
--- a/tools/tensorflow_docs/tools/nblint/style/tensorflow.py
+++ b/tools/tensorflow_docs/tools/nblint/style/tensorflow.py
@@ -14,7 +14,7 @@
 # ==============================================================================
 r"""Lint assertions for notebooks published on tensorflow.org.
 
-These lints are a non-exhaustive implemention of style rules found in the
+These lints are a non-exhaustive implementation of style rules found in the
 TensorFlow documentation and style guides. See:
 
 - https://www.tensorflow.org/community/contribute/docs

From 4ca683088ab1fe0959b34ccbefcf2863ac2b24ea Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Wed, 8 Jun 2022 10:57:47 +0100
Subject: [PATCH 171/872] Lint tf.data (build TF input pipelines) guide, fix
 typos

---
 site/en/guide/data.ipynb | 82 ++++++++++++++++++++--------------------
 1 file changed, 41 insertions(+), 41 deletions(-)

diff --git a/site/en/guide/data.ipynb b/site/en/guide/data.ipynb
index 96a056ce48e..d3dde22fd43 100644
--- a/site/en/guide/data.ipynb
+++ b/site/en/guide/data.ipynb
@@ -139,8 +139,8 @@
         "\n",
         "Once you have a `Dataset` object, you can *transform* it into a new `Dataset` by\n",
         "chaining method calls on the `tf.data.Dataset` object. For example, you can\n",
-        "apply per-element transformations such as `Dataset.map()`, and multi-element\n",
-        "transformations such as `Dataset.batch()`. See the documentation for\n",
+        "apply per-element transformations such as `Dataset.map`, and multi-element\n",
+        "transformations such as `Dataset.batch`. Refer to the documentation for\n",
         "`tf.data.Dataset` for a complete list of transformations.\n",
         "\n",
         "The `Dataset` object is a Python iterable. This makes it possible to consume its\n",
@@ -238,9 +238,9 @@
         "structure of elements include `tuple`, `dict`, `NamedTuple`, and\n",
         "`OrderedDict`. In particular, `list` is not a valid construct for\n",
         "expressing the structure of dataset elements. This is because\n",
-        "early tf.data users felt strongly about `list` inputs (e.g. passed\n",
+        "early `tf.data` users felt strongly about `list` inputs (for example, when passed\n",
         "to `tf.data.Dataset.from_tensors`) being automatically packed as\n",
-        "tensors and `list` outputs (e.g. return values of user-defined\n",
+        "tensors and `list` outputs (for example, return values of user-defined\n",
         "functions) being coerced into a `tuple`. As a consequence, if you\n",
         "would like a `list` input to be treated as a structure, you need\n",
         "to convert it into `tuple` and if you would like a `list` output\n",
@@ -328,7 +328,7 @@
       },
       "source": [
         "The `Dataset` transformations support datasets of any structure. When using the\n",
-        "`Dataset.map()`, and `Dataset.filter()` transformations,\n",
+        "`Dataset.map`, and `Dataset.filter` transformations,\n",
         "which apply a function to each element, the element structure determines the\n",
         "arguments of the function:"
       ]
@@ -416,11 +416,11 @@
       "source": [
         "### Consuming NumPy arrays\n",
         "\n",
-        "See [Loading NumPy arrays](../tutorials/load_data/numpy.ipynb) for more examples.\n",
+        "Refer to the [Loading NumPy arrays](../tutorials/load_data/numpy.ipynb) tutorial for more examples.\n",
         "\n",
         "If all of your input data fits in memory, the simplest way to create a `Dataset`\n",
         "from them is to convert them to `tf.Tensor` objects and use\n",
-        "`Dataset.from_tensor_slices()`."
+        "`Dataset.from_tensor_slices`."
       ]
     },
     {
@@ -472,7 +472,7 @@
         "\n",
         "Another common data source that can easily be ingested as a `tf.data.Dataset` is the python generator.\n",
         "\n",
-        "Caution: While this is a convienient approach it has limited portability and scalibility. It must run in the same python process that created the generator, and is still subject to the Python [GIL](https://en.wikipedia.org/wiki/Global_interpreter_lock)."
+        "Caution: While this is a convenient approach it has limited portability and scalability. It must run in the same python process that created the generator, and is still subject to the Python [GIL](https://en.wikipedia.org/wiki/Global_interpreter_lock)."
       ]
     },
     {
@@ -548,7 +548,7 @@
         "\n",
         "It's also important to note that the `output_shapes` and `output_types` follow the same nesting rules as other dataset methods.\n",
         "\n",
-        "Here is an example generator that demonstrates both aspects, it returns tuples of arrays, where the second array is a vector with unknown length."
+        "Here is an example generator that demonstrates both aspects: it returns tuples of arrays, where the second array is a vector with unknown length."
       ]
     },
     {
@@ -739,7 +739,7 @@
       "source": [
         "### Consuming TFRecord data\n",
         "\n",
-        "See [Loading TFRecords](../tutorials/load_data/tfrecord.ipynb) for an end-to-end example.\n",
+        "Refer to the [Loading TFRecords](../tutorials/load_data/tfrecord.ipynb) tutorial for an end-to-end example.\n",
         "\n",
         "The `tf.data` API supports a variety of file formats so that you can process\n",
         "large datasets that do not fit in memory. For example, the TFRecord file format\n",
@@ -825,7 +825,7 @@
       "source": [
         "### Consuming text data\n",
         "\n",
-        "See [Loading Text](../tutorials/load_data/text.ipynb) for an end to end example.\n",
+        "Refer to the [Load text](../tutorials/load_data/text.ipynb) tutorial for an end-to-end example.\n",
         "\n",
         "Many datasets are distributed as one or more text files. The\n",
         "`tf.data.TextLineDataset` provides an easy way to extract lines from one or more\n",
@@ -916,7 +916,7 @@
       "source": [
         "By default, a `TextLineDataset` yields *every* line of each file, which may\n",
         "not be desirable, for example, if the file starts with a header line, or contains comments. These lines can be removed using the `Dataset.skip()` or\n",
-        "`Dataset.filter()` transformations. Here, you skip the first line, then filter to\n",
+        "`Dataset.filter` transformations. Here, you skip the first line, then filter to\n",
         "find only survivors."
       ]
     },
@@ -985,7 +985,7 @@
         "id": "ChDHNi3qbDch"
       },
       "source": [
-        "See [Loading CSV Files](../tutorials/load_data/csv.ipynb), and [Loading Pandas DataFrames](../tutorials/load_data/pandas_dataframe.ipynb) for more examples. \n",
+        "Refer to the [Loading CSV Files](../tutorials/load_data/csv.ipynb) and [Loading Pandas DataFrames](../tutorials/load_data/pandas_dataframe.ipynb) tutorials for more examples.\n",
         "\n",
         "The CSV file format is a popular format for storing tabular data in plain text.\n",
         "\n",
@@ -1045,11 +1045,11 @@
         "id": "47yippqaHFk6"
       },
       "source": [
-        "A more scalable approach is to load from disk as necessary. \n",
+        "A more scalable approach is to load from disk as necessary.\n",
         "\n",
         "The `tf.data` module provides methods to extract records from one or more CSV files that comply with [RFC 4180](https://tools.ietf.org/html/rfc4180).\n",
         "\n",
-        "The `experimental.make_csv_dataset` function is the high level interface for reading sets of csv files. It supports column type inference and many other features, like batching and shuffling, to make usage simple."
+        "The `tf.data.experimental.make_csv_dataset` function is the high-level interface for reading sets of CSV files. It supports column type inference and many other features, like batching and shuffling, to make usage simple."
       ]
     },
     {
@@ -1122,7 +1122,7 @@
         "id": "TSVgJJ1HJD6M"
       },
       "source": [
-        "There is also a lower-level `experimental.CsvDataset` class which provides finer grained control. It does not support column type inference. Instead you must specify the type of each column. "
+        "There is also a lower-level `experimental.CsvDataset` class which provides finer grained control. It does not support column type inference. Instead you must specify the type of each column."
       ]
     },
     {
@@ -1133,7 +1133,7 @@
       },
       "outputs": [],
       "source": [
-        "titanic_types  = [tf.int32, tf.string, tf.float32, tf.int32, tf.int32, tf.float32, tf.string, tf.string, tf.string, tf.string] \n",
+        "titanic_types  = [tf.int32, tf.string, tf.float32, tf.int32, tf.int32, tf.float32, tf.string, tf.string, tf.string, tf.string]\n",
         "dataset = tf.data.experimental.CsvDataset(titanic_file, titanic_types , header=True)\n",
         "\n",
         "for line in dataset.take(10):\n",
@@ -1457,10 +1457,10 @@
         "### Batching tensors with padding\n",
         "\n",
         "The above recipe works for tensors that all have the same size. However, many\n",
-        "models (e.g. sequence models) work with input data that can have varying size\n",
-        "(e.g. sequences of different lengths). To handle this case, the\n",
+        "models (including sequence models) work with input data that can have varying size\n",
+        "(for example, sequences of different lengths). To handle this case, the\n",
         "`Dataset.padded_batch` transformation enables you to batch tensors of\n",
-        "different shape by specifying one or more dimensions in which they may be\n",
+        "different shapes by specifying one or more dimensions in which they may be\n",
         "padded."
       ]
     },
@@ -1604,7 +1604,7 @@
         "id": "DlEM5f9loSHR"
       },
       "source": [
-        "If you would like to perform a custom computation (e.g. to collect statistics) at the end of each epoch then it's simplest to restart the dataset iteration on each epoch:"
+        "If you would like to perform a custom computation (for example, to collect statistics) at the end of each epoch then it's simplest to restart the dataset iteration on each epoch:"
       ]
     },
     {
@@ -1693,7 +1693,7 @@
       "source": [
         "As with `Dataset.batch` the order relative to `Dataset.repeat` matters.\n",
         "\n",
-        "`Dataset.shuffle` doesn't signal the end of an epoch until the shuffle buffer is empty. So a shuffle placed before a repeat will show every element of one epoch before moving to the next: "
+        "`Dataset.shuffle` doesn't signal the end of an epoch until the shuffle buffer is empty. So a shuffle placed before a repeat will show every element of one epoch before moving to the next:"
       ]
     },
     {
@@ -1906,7 +1906,7 @@
         "\n",
         "For performance reasons, use TensorFlow operations for\n",
         "preprocessing your data whenever possible. However, it is sometimes useful to\n",
-        "call external Python libraries when parsing your input data. You can use the `tf.py_function()` operation in a `Dataset.map()` transformation."
+        "call external Python libraries when parsing your input data. You can use the `tf.py_function` operation in a `Dataset.map` transformation."
       ]
     },
     {
@@ -1915,7 +1915,7 @@
         "id": "R2u7CeA67DU8"
       },
       "source": [
-        "For example, if you want to apply a random rotation, the `tf.image` module only has `tf.image.rot90`, which is not very useful for image augmentation. \n",
+        "For example, if you want to apply a random rotation, the `tf.image` module only has `tf.image.rot90`, which is not very useful for image augmentation.\n",
         "\n",
         "Note: `tensorflow_addons` has a TensorFlow compatible `rotate` in `tensorflow_addons.image.rotate`.\n",
         "\n",
@@ -2124,7 +2124,7 @@
         "id": "t0JMgvXEz9y1"
       },
       "source": [
-        "For an end to end time series example see: [Time series forecasting](../../tutorials/structured_data/time_series.ipynb)."
+        "For an end-to-end time series example see: [Time series forecasting](../../tutorials/structured_data/time_series.ipynb)."
       ]
     },
     {
@@ -2155,7 +2155,7 @@
         "id": "o6GLGhxgpazJ"
       },
       "source": [
-        "Typically, models based on this sort of data will want a contiguous time slice. \n",
+        "Typically, models based on this sort of data will want a contiguous time slice.\n",
         "\n",
         "The simplest approach would be to batch the data:"
       ]
@@ -2283,7 +2283,7 @@
         "id": "fF6pEdlduq8E"
       },
       "source": [
-        "While using `Dataset.batch` works, there are situations where you may need finer control. The `Dataset.window` method gives you complete control, but requires some care: it returns a `Dataset` of `Datasets`. See [Dataset structure](#dataset_structure) for details."
+        "While using `Dataset.batch` works, there are situations where you may need finer control. The `Dataset.window` method gives you complete control, but requires some care: it returns a `Dataset` of `Datasets`. Go to the [Dataset structure](#dataset_structure) section for details."
       ]
     },
     {
@@ -2328,7 +2328,7 @@
         "id": "sgLIwq9Anc34"
       },
       "source": [
-        "In nearly all cases, you will want to `.batch` the dataset first:"
+        "In nearly all cases, you will want to `Dataset.batch` the dataset first:"
       ]
     },
     {
@@ -2422,7 +2422,7 @@
         "\n",
         "When working with a dataset that is very class-imbalanced, you may want to resample the dataset. `tf.data` provides two methods to do this. The credit card fraud dataset is a good example of this sort of problem.\n",
         "\n",
-        "Note: See [Classification on imbalanced data](../tutorials/structured_data/imbalanced_data.ipynb) for a full tutorial.\n"
+        "Note: Go to [Classification on imbalanced data](../tutorials/structured_data/imbalanced_data.ipynb) for a full tutorial.\n"
       ]
     },
     {
@@ -2529,7 +2529,7 @@
         "id": "ov14SRrQyQE3"
       },
       "source": [
-        "One approach to resampling a dataset is to use `sample_from_datasets`. This is more applicable when you have a separate `data.Dataset` for each class.\n",
+        "One approach to resampling a dataset is to use `sample_from_datasets`. This is more applicable when you have a separate `tf.data.Dataset` for each class.\n",
         "\n",
         "Here, just use filter to generate them from the credit card fraud data:"
       ]
@@ -2593,7 +2593,7 @@
         "id": "2K4ObOms082B"
       },
       "source": [
-        "Now the dataset produces examples of each class with 50/50 probability:"
+        "Now the dataset produces examples of each class with a 50/50 probability:"
       ]
     },
     {
@@ -2627,11 +2627,11 @@
         "it needs a separate `tf.data.Dataset` per class. You could use `Dataset.filter`\n",
         "to create those two datasets, but that results in all the data being loaded twice.\n",
         "\n",
-        "The `data.Dataset.rejection_resample` method can be applied to a dataset to rebalance it, while only loading it once. Elements will be dropped from the dataset to achieve balance.\n",
+        "The `tf.data.Dataset.rejection_resample` method can be applied to a dataset to rebalance it, while only loading it once. Elements will be dropped from the dataset to achieve balance.\n",
         "\n",
-        "`data.Dataset.rejection_resample` takes a `class_func` argument. This `class_func` is applied to each dataset element, and is used to determine which class an example belongs to for the purposes of balancing.\n",
+        "The `rejection_resample` method takes a `class_func` argument. This `class_func` is applied to each dataset element, and is used to determine which class an example belongs to for the purposes of balancing.\n",
         "\n",
-        "The goal here is to balance the lable distribution, and the elements of `creditcard_ds` are already `(features, label)` pairs. So the `class_func` just needs to return those labels:"
+        "The goal here is to balance the label distribution, and the elements of `creditcard_ds` are already `(features, label)` pairs. So the `class_func` just needs to return those labels:"
       ]
     },
     {
@@ -2699,7 +2699,7 @@
         "id": "j3d2jyEhx9kD"
       },
       "source": [
-        "Now the dataset produces examples of each class with 50/50 probability:"
+        "Now the dataset produces examples of each class with a 50/50 probability:"
       ]
     },
     {
@@ -2729,7 +2729,7 @@
         "id": "SOGg1UFhUE4z"
       },
       "source": [
-        "Tensorflow supports [taking checkpoints](https://www.tensorflow.org/guide/checkpoint) so that when your training process restarts it can restore the latest checkpoint to recover most of its progress. In addition to checkpointing the model variables, you can also checkpoint the progress of the dataset iterator. This could be useful if you have a large dataset and don't want to start the dataset from the beginning on each restart. Note however that iterator checkpoints may be large, since transformations such as `shuffle` and `prefetch` require buffering elements within the iterator. \n",
+        "Tensorflow supports [taking checkpoints](./checkpoint.ipynb) so that when your training process restarts it can restore the latest checkpoint to recover most of its progress. In addition to checkpointing the model variables, you can also checkpoint the progress of the dataset iterator. This could be useful if you have a large dataset and don't want to start the dataset from the beginning on each restart. Note however that iterator checkpoints may be large, since transformations such as `Dataset.shuffle` and `Dataset.prefetch` require buffering elements within the iterator.\n",
         "\n",
         "To include your iterator in a checkpoint, pass the iterator to the `tf.train.Checkpoint` constructor."
       ]
@@ -2765,7 +2765,7 @@
         "id": "gxWglTwX9Fex"
       },
       "source": [
-        "Note: It is not possible to checkpoint an iterator which relies on external state such as a `tf.py_function`. Attempting to do so will raise an exception complaining about the external state."
+        "Note: It is not possible to checkpoint an iterator which relies on an external state, such as a `tf.py_function`. Attempting to do so will raise an exception complaining about the external state."
       ]
     },
     {
@@ -2774,7 +2774,7 @@
         "id": "uLRdedPpbDdD"
       },
       "source": [
-        "## Using tf.data with tf.keras"
+        "## Using `tf.data` with `tf.keras`"
       ]
     },
     {
@@ -2784,7 +2784,7 @@
       },
       "source": [
         "The `tf.keras` API simplifies many aspects of creating and executing machine\n",
-        "learning models. Its `.fit()` and `.evaluate()` and `.predict()` APIs support datasets as inputs. Here is a quick dataset and model setup:"
+        "learning models. Its `Model.fit` and `Model.evaluate` and `Model.predict` APIs support datasets as inputs. Here is a quick dataset and model setup:"
       ]
     },
     {
@@ -2849,7 +2849,7 @@
         "id": "FzpAQfJMJF41"
       },
       "source": [
-        "If you pass an infinite dataset, for example by calling `Dataset.repeat()`, you just need to also pass the `steps_per_epoch` argument:"
+        "If you pass an infinite dataset, for example by calling `Dataset.repeat`, you just need to also pass the `steps_per_epoch` argument:"
       ]
     },
     {
@@ -2913,7 +2913,7 @@
         "id": "aZYhJ_YSIl6w"
       },
       "source": [
-        "The labels are not required in when calling `Model.predict`. "
+        "The labels are not required when calling `Model.predict`."
       ]
     },
     {

From 8b457500c77bdd092a2e1d08773339d818cc5cac Mon Sep 17 00:00:00 2001
From: tilakrayal <81610181+tilakrayal@users.noreply.github.com>
Date: Thu, 9 Jun 2022 18:04:26 +0530
Subject: [PATCH 172/872] Fixing the incorrect link in actor_critic.ipynb

---
 site/en/tutorials/reinforcement_learning/actor_critic.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
index ef09ea310a1..161ce74be1a 100644
--- a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
+++ b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
@@ -104,7 +104,7 @@
       "source": [
         "**CartPole-v0**\n",
         "\n",
-        "In the [CartPole-v0 environment](https://gym.openai.com/envs/CartPole-v0), a pole is attached to a cart moving along a frictionless track. \n",
+        "In the [CartPole-v0 environment](https://www.gymlibrary.ml/environments/classic_control/cart_pole/?highlight=cart%20pole%20v0), a pole is attached to a cart moving along a frictionless track. \n",
         "The pole starts upright and the goal of the agent is to prevent it from falling over by applying a force of -1 or +1 to the cart. \n",
         "A reward of +1 is given for every time step the pole remains upright.\n",
         "An episode ends when (1) the pole is more than 15 degrees from vertical or (2) the cart moves more than 2.4 units from the center.\n",

From cd27cd401b51e0e0b8e5da09f93427a14dbe2203 Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Thu, 9 Jun 2022 12:36:39 +0000
Subject: [PATCH 173/872] nbfmt

---
 site/en/tutorials/reinforcement_learning/actor_critic.ipynb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
index 161ce74be1a..4b832a3d40c 100644
--- a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
+++ b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
@@ -738,7 +738,6 @@
         "_jQ1tEQCxwRx"
       ],
       "name": "actor_critic.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 56b0440ab204e40106d490957ad713016229f456 Mon Sep 17 00:00:00 2001
From: tilakrayal <81610181+tilakrayal@users.noreply.github.com>
Date: Thu, 9 Jun 2022 19:40:05 +0530
Subject: [PATCH 174/872] Fixing the incorrect link in actor_critic.ipynb

---
 site/en/tutorials/reinforcement_learning/actor_critic.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
index 4b832a3d40c..75fba0fc46c 100644
--- a/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
+++ b/site/en/tutorials/reinforcement_learning/actor_critic.ipynb
@@ -104,7 +104,7 @@
       "source": [
         "**CartPole-v0**\n",
         "\n",
-        "In the [CartPole-v0 environment](https://www.gymlibrary.ml/environments/classic_control/cart_pole/?highlight=cart%20pole%20v0), a pole is attached to a cart moving along a frictionless track. \n",
+        "In the [CartPole-v0 environment](https://www.gymlibrary.ml/environments/classic_control/cart_pole/), a pole is attached to a cart moving along a frictionless track. \n",
         "The pole starts upright and the goal of the agent is to prevent it from falling over by applying a force of -1 or +1 to the cart. \n",
         "A reward of +1 is given for every time step the pole remains upright.\n",
         "An episode ends when (1) the pole is more than 15 degrees from vertical or (2) the cart moves more than 2.4 units from the center.\n",

From 63a423d1ec13147d0102765440fb949cdd6eedef Mon Sep 17 00:00:00 2001
From: synandi <98147397+synandi@users.noreply.github.com>
Date: Mon, 13 Jun 2022 12:24:50 +0530
Subject: [PATCH 175/872] Fixed spelling mistakes

---
 site/en/guide/migrate/migrating_feature_columns.ipynb | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/site/en/guide/migrate/migrating_feature_columns.ipynb b/site/en/guide/migrate/migrating_feature_columns.ipynb
index 62f91932935..44a49280341 100644
--- a/site/en/guide/migrate/migrating_feature_columns.ipynb
+++ b/site/en/guide/migrate/migrating_feature_columns.ipynb
@@ -557,7 +557,7 @@
         "id": "fd6eluARXndC"
       },
       "source": [
-        "In Keras, there is no `combiner` option to `tf.keras.layers.Embedding`, but you can acheive the same effect with `tf.keras.layers.Dense`. The `embedding_column` above is simply linearly combining embedding vectors according to category weight. Though not obvious at first, it is exactly equivalent to representing your categorical inputs as a sparse weight vector of size `(num_tokens)`, and mutiplying them by a `Dense` kernel of shape `(embedding_size, num_tokens)`."
+        "In Keras, there is no `combiner` option to `tf.keras.layers.Embedding`, but you can achieve the same effect with `tf.keras.layers.Dense`. The `embedding_column` above is simply linearly combining embedding vectors according to category weight. Though not obvious at first, it is exactly equivalent to representing your categorical inputs as a sparse weight vector of size `(num_tokens)`, and multiplying them by a `Dense` kernel of shape `(embedding_size, num_tokens)`."
       ]
     },
     {
@@ -572,7 +572,7 @@
         "weights = tf.constant([[0.5, 1.5, 0.7, 1.8, 0.2]])\n",
         "\n",
         "# For `combiner='mean'`, normalize your weights to sum to 1. Removing this line\n",
-        "# would be eqivalent to an `embedding_column` with `combiner='sum'`.\n",
+        "# would be equivalent to an `embedding_column` with `combiner='sum'`.\n",
         "weights = weights / tf.reduce_sum(weights, axis=-1, keepdims=True)\n",
         "\n",
         "count_layer = tf.keras.layers.CategoryEncoding(\n",
@@ -845,8 +845,8 @@
       "outputs": [],
       "source": [
         "inputs = preprocessing_model.input\n",
-        "outpus = training_model(preprocessing_model(inputs))\n",
-        "inference_model = tf.keras.Model(inputs, outpus)\n",
+        "outputs = training_model(preprocessing_model(inputs))\n",
+        "inference_model = tf.keras.Model(inputs, outputs)\n",
         "\n",
         "predict_dataset = tf.data.Dataset.from_tensor_slices(predict_features).batch(1)\n",
         "inference_model.predict(predict_dataset)"
@@ -880,7 +880,7 @@
         "id": "IXMBwzggwUjI"
       },
       "source": [
-        "Note: Preprocessing layers are not trainable, which allows you to apply them *asynchronously* using `tf.data`. This has performence benefits, as you can both [prefetch](https://www.tensorflow.org/guide/data_performance#prefetching) preprocessed batches, and free up any accelerators to focus on the differentiable parts of a model. As this guide shows, seperating preprocessing during training and composing it during inference is a flexible way to leverage these performance gains. However, if your model is small or preprocessing time is negligable, it may be simpler to build preprocessing into a complete model from the start. To do this you can build a single model starting with `tf.keras.Input`, followed by preprocessing layers, followed by trainable layers."
+        "Note: Preprocessing layers are not trainable, which allows you to apply them *asynchronously* using `tf.data`. This has performance benefits, as you can both [prefetch](https://www.tensorflow.org/guide/data_performance#prefetching) preprocessed batches, and free up any accelerators to focus on the differentiable parts of a model. As this guide shows, separating preprocessing during training and composing it during inference is a flexible way to leverage these performance gains. However, if your model is small or preprocessing time is negligible, it may be simpler to build preprocessing into a complete model from the start. To do this you can build a single model starting with `tf.keras.Input`, followed by preprocessing layers, followed by trainable layers."
       ]
     },
     {

From f6e494832383756e9b81987d0e8145c817c4d072 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 13:38:46 +0100
Subject: [PATCH 176/872] Update optimizer and loss aliases to Keras in the
 Load CSV data tutorial, lint the tutorial

---
 site/en/tutorials/load_data/csv.ipynb | 138 +++++++++++++-------------
 1 file changed, 69 insertions(+), 69 deletions(-)

diff --git a/site/en/tutorials/load_data/csv.ipynb b/site/en/tutorials/load_data/csv.ipynb
index 6112ce0a393..60dacdb65bf 100644
--- a/site/en/tutorials/load_data/csv.ipynb
+++ b/site/en/tutorials/load_data/csv.ipynb
@@ -75,7 +75,7 @@
         "1. **Loading the data off disk**\n",
         "2. **Pre-processing it into a form suitable for training.**\n",
         "\n",
-        "This tutorial focuses on the loading, and gives some quick examples of preprocessing. For a tutorial that focuses on the preprocessing aspect see the [preprocessing layers guide](https://www.tensorflow.org/guide/keras/preprocessing_layers#quick_recipes) and [tutorial](https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers). \n"
+        "This tutorial focuses on the loading, and gives some quick examples of preprocessing. To learn more about the preprocessing aspect, check out the [Working with preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) guide and the [Classify structured data using Keras preprocessing layers](../structured_data/preprocessing_layers.ipynb) tutorial.\n"
       ]
     },
     {
@@ -120,7 +120,7 @@
         "id": "ny5TEgcmHjVx"
       },
       "source": [
-        "For any small CSV dataset the simplest way to train a TensorFlow model on it is to load it into memory as a pandas Dataframe or a NumPy array. \n"
+        "For any small CSV dataset the simplest way to train a TensorFlow model on it is to load it into memory as a pandas Dataframe or a NumPy array.\n"
       ]
     },
     {
@@ -129,12 +129,12 @@
         "id": "LgpBOuU8PGFf"
       },
       "source": [
-        "A relatively simple example is the [abalone dataset](https://archive.ics.uci.edu/ml/datasets/abalone). \n",
+        "A relatively simple example is the [abalone dataset](https://archive.ics.uci.edu/ml/datasets/abalone).\n",
         "\n",
-        "* The dataset is small. \n",
-        "* All the input features are all limited-range floating point values. \n",
+        "* The dataset is small.\n",
+        "* All the input features are all limited-range floating point values.\n",
         "\n",
-        "Here is how to download the data into a [Pandas `DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html):"
+        "Here is how to download the data into a [pandas `DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html):"
       ]
     },
     {
@@ -159,7 +159,7 @@
         "id": "hP22mdyPQ1_t"
       },
       "source": [
-        "The dataset contains a set of measurements of [abalone](https://en.wikipedia.org/wiki/Abalone), a type of sea snail. \n",
+        "The dataset contains a set of measurements of [abalone](https://en.wikipedia.org/wiki/Abalone), a type of sea snail.\n",
         "\n",
         "![an abalone shell](https://tensorflow.org/images/abalone_shell.jpg)\n",
         "\n",
@@ -214,7 +214,7 @@
         "id": "1C1yFOxLOdxh"
       },
       "source": [
-        "Next make a regression model predict the age. Since there is only a single input tensor, a `keras.Sequential` model is sufficient here."
+        "Next make a regression model predict the age. Since there is only a single input tensor, a `tf.keras.Sequential` model is sufficient here."
       ]
     },
     {
@@ -231,7 +231,7 @@
         "])\n",
         "\n",
         "abalone_model.compile(loss = tf.keras.losses.MeanSquaredError(),\n",
-        "                      optimizer = tf.optimizers.Adam())"
+        "                      optimizer = tf.keras.optimizers.Adam())"
       ]
     },
     {
@@ -278,11 +278,11 @@
         "id": "yCrB2Jd-U0Vt"
       },
       "source": [
-        "It's good practice to normalize the inputs to your model. The Keras preprocessing layers provide a convenient way to build this normalization into your model. \n",
+        "It's good practice to normalize the inputs to your model. The Keras preprocessing layers provide a convenient way to build this normalization into your model.\n",
         "\n",
-        "The layer will precompute the mean and variance of each column, and use these to normalize the data.\n",
+        "The `tf.keras.layers.Normalization` layer precomputes the mean and variance of each column, and uses these to normalize the data.\n",
         "\n",
-        "First you create the layer:"
+        "First, create the layer:"
       ]
     },
     {
@@ -302,9 +302,9 @@
         "id": "hGgEZE-7Vpt6"
       },
       "source": [
-        "Then you use the `Normalization.adapt()` method to adapt the normalization layer to your data.\n",
+        "Then, use the `Normalization.adapt` method to adapt the normalization layer to your data.\n",
         "\n",
-        "Note: Only use your training data to `.adapt()` preprocessing layers. Do not use your validation or test data."
+        "Note: Only use your training data with the `PreprocessingLayer.adapt` method. Do not use your validation or test data."
       ]
     },
     {
@@ -324,7 +324,7 @@
         "id": "rE6vh0byV7cE"
       },
       "source": [
-        "Then use the normalization layer in your model:"
+        "Then, use the normalization layer in your model:"
       ]
     },
     {
@@ -341,8 +341,8 @@
         "  layers.Dense(1)\n",
         "])\n",
         "\n",
-        "norm_abalone_model.compile(loss = tf.losses.MeanSquaredError(),\n",
-        "                           optimizer = tf.optimizers.Adam())\n",
+        "norm_abalone_model.compile(loss = tf.keras.losses.MeanSquaredError(),\n",
+        "                           optimizer = tf.keras.optimizers.Adam())\n",
         "\n",
         "norm_abalone_model.fit(abalone_features, abalone_labels, epochs=10)"
       ]
@@ -355,13 +355,13 @@
       "source": [
         "## Mixed data types\n",
         "\n",
-        "The \"Titanic\" dataset contains information about the passengers on the Titanic. The nominal task on this dataset is to predict who survived. \n",
+        "The \"Titanic\" dataset contains information about the passengers on the Titanic. The nominal task on this dataset is to predict who survived.\n",
         "\n",
         "![The Titanic](images/csv/Titanic.jpg)\n",
         "\n",
         "Image [from Wikimedia](https://commons.wikimedia.org/wiki/File:RMS_Titanic_3.jpg)\n",
         "\n",
-        "The raw data can easily be loaded as a Pandas `DataFrame`, but is not immediately usable as input to a TensorFlow model. \n"
+        "The raw data can easily be loaded as a Pandas `DataFrame`, but is not immediately usable as input to a TensorFlow model.\n"
       ]
     },
     {
@@ -394,7 +394,7 @@
         "id": "urHOwpCDYtcI"
       },
       "source": [
-        "Because of the different data types and ranges you can't simply stack the features into  NumPy array and pass it to a `keras.Sequential` model. Each column needs to be handled individually. \n",
+        "Because of the different data types and ranges, you can't simply stack the features into a NumPy array and pass it to a `tf.keras.Sequential` model. Each column needs to be handled individually.\n",
         "\n",
         "As one option, you could preprocess your data offline (using any tool you like) to convert categorical columns to numeric columns, then pass the processed output to your TensorFlow model. The disadvantage to that approach is that if you save and export your model the preprocessing is not saved with it. The Keras preprocessing layers avoid this problem because they're part of the model.\n"
       ]
@@ -407,7 +407,7 @@
       "source": [
         "In this example, you'll build a model that implements the preprocessing logic using [Keras functional API](https://www.tensorflow.org/guide/keras/functional). You could also do it by [subclassing](https://www.tensorflow.org/guide/keras/custom_layers_and_models).\n",
         "\n",
-        "The functional API operates on \"symbolic\" tensors. Normal \"eager\" tensors have a value. In contrast these \"symbolic\" tensors do not. Instead they keep track of which operations are run on them, and build representation of the calculation, that you can run later. Here's a quick example:"
+        "The functional API operates on \"symbolic\" tensors. Normal \"eager\" tensors have a value. In contrast these \"symbolic\" tensors do not. Instead they keep track of which operations are run on them, and build a representation of the calculation, that you can run later. Here's a quick example:"
       ]
     },
     {
@@ -457,7 +457,7 @@
         "id": "rNS9lT7f6_U2"
       },
       "source": [
-        "To build the preprocessing model, start by building a set of symbolic `keras.Input` objects, matching the names and data-types of the CSV columns."
+        "To build the preprocessing model, start by building a set of symbolic `tf.keras.Input` objects, matching the names and data-types of the CSV columns."
       ]
     },
     {
@@ -516,7 +516,7 @@
         "id": "-JoR45Uj712l"
       },
       "source": [
-        "Collect all the symbolic preprocessing results, to concatenate them later."
+        "Collect all the symbolic preprocessing results, to concatenate them later:"
       ]
     },
     {
@@ -536,9 +536,9 @@
         "id": "r0Hryylyosfm"
       },
       "source": [
-        "For the string inputs use the `tf.keras.layers.StringLookup` function to map from strings to integer indices in a vocabulary. Next, use `tf.keras.layers.CategoryEncoding` to convert the indexes into `float32` data appropriate for the model. \n",
+        "For the string inputs use the `tf.keras.layers.StringLookup` function to map from strings to integer indices in a vocabulary. Next, use `tf.keras.layers.CategoryEncoding` to convert the indexes into `float32` data appropriate for the model.\n",
         "\n",
-        "The default settings for the `tf.keras.layers.CategoryEncoding` layer create a one-hot vector for each input. A `layers.Embedding` would also work. See the [preprocessing layers guide](https://www.tensorflow.org/guide/keras/preprocessing_layers#quick_recipes) and [tutorial](../structured_data/preprocessing_layers.ipynb) for more on this topic."
+        "The default settings for the `tf.keras.layers.CategoryEncoding` layer create a one-hot vector for each input. A `tf.keras.layers.Embedding` would also work. Check out the [Working with preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) guide and the [Classify structured data using Keras preprocessing layers](../structured_data/preprocessing_layers.ipynb) tutorial for more on this topic."
       ]
     },
     {
@@ -567,7 +567,7 @@
         "id": "Wnhv0T7itnc7"
       },
       "source": [
-        "With the collection of `inputs` and `processed_inputs`, you can concatenate all the preprocessed inputs together, and build a model that handles the preprocessing:"
+        "With the collection of `inputs` and `preprocessed_inputs`, you can concatenate all the preprocessed inputs together, and build a model that handles the preprocessing:"
       ]
     },
     {
@@ -591,7 +591,7 @@
         "id": "PNHxrNW8vdda"
       },
       "source": [
-        "This `model` just contains the input preprocessing. You can run it to see what it does to your data. Keras models don't automatically convert Pandas `DataFrames` because it's not clear if it should be converted to one tensor or to a dictionary of tensors. So convert it to a dictionary of tensors:"
+        "This model just contains the input preprocessing. You can run it to see what it does to your data. Keras models don't automatically convert pandas `DataFrame`s because it's not clear if it should be converted to one tensor or to a dictionary of tensors. So, convert it to a dictionary of tensors:"
       ]
     },
     {
@@ -633,7 +633,7 @@
         "id": "qkBf4LvmzMDp"
       },
       "source": [
-        "Now build the model on top of this:"
+        "Now, build the model on top of this:"
       ]
     },
     {
@@ -654,8 +654,8 @@
         "  result = body(preprocessed_inputs)\n",
         "  model = tf.keras.Model(inputs, result)\n",
         "\n",
-        "  model.compile(loss=tf.losses.BinaryCrossentropy(from_logits=True),\n",
-        "                optimizer=tf.optimizers.Adam())\n",
+        "  model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
+        "                optimizer=tf.keras.optimizers.Adam())\n",
         "  return model\n",
         "\n",
         "titanic_model = titanic_model(titanic_preprocessing, inputs)"
@@ -734,11 +734,11 @@
         "id": "NyVDCwGzR5HW"
       },
       "source": [
-        "In the previous section you relied on the model's built-in data shuffling and batching while training the model. \n",
+        "In the previous section you relied on the model's built-in data shuffling and batching while training the model.\n",
         "\n",
-        "If you need more control over the input data pipeline or need to use data that doesn't easily fit into memory: use `tf.data`. \n",
+        "If you need more control over the input data pipeline or need to use data that doesn't easily fit into memory: use `tf.data`.\n",
         "\n",
-        "For more examples see the [tf.data guide](../../guide/data.ipynb)."
+        "For more examples, refer to the [`tf.data`: Build TensorFlow input pipelines](../../guide/data.ipynb) guide."
       ]
     },
     {
@@ -749,7 +749,7 @@
       "source": [
         "### On in memory data\n",
         "\n",
-        "As a first example of applying `tf.data` to CSV data consider the following code to manually slice up the dictionary of features from the previous section. For each index, it takes that index for each feature:\n"
+        "As a first example of applying `tf.data` to CSV data, consider the following code to manually slice up the dictionary of features from the previous section. For each index, it takes that index for each feature:\n"
       ]
     },
     {
@@ -798,7 +798,7 @@
         "id": "vvp8Dct6YOIE"
       },
       "source": [
-        "The most basic `tf.data.Dataset` in memory data loader is the `Dataset.from_tensor_slices` constructor. This returns a `tf.data.Dataset` that implements a generalized version of the above `slices` function, in TensorFlow. "
+        "The most basic `tf.data.Dataset` in memory data loader is the `Dataset.from_tensor_slices` constructor. This returns a `tf.data.Dataset` that implements a generalized version of the above `slices` function, in TensorFlow."
       ]
     },
     {
@@ -923,7 +923,7 @@
         "id": "t4N-plO4tDXd"
       },
       "source": [
-        "Now read the CSV data from the file and create a `tf.data.Dataset`. \n",
+        "Now read the CSV data from the file and create a `tf.data.Dataset`.\n",
         "\n",
         "(For the full documentation, see `tf.data.experimental.make_csv_dataset`)\n"
       ]
@@ -950,7 +950,7 @@
         "id": "Sf3v3BKgy4AG"
       },
       "source": [
-        "This function includes many convenient features so the data is easy to work with. This includes:\n",
+        "This function includes many convenient features, so the data is easy to work with. This includes:\n",
         "\n",
         "* Using the column headers as dictionary keys.\n",
         "* Automatically determining the type of each column."
@@ -977,7 +977,7 @@
         "id": "k-TgA6o2Ja6U"
       },
       "source": [
-        "Note: if you run the above cell twice it will produce different results. The default settings for `make_csv_dataset` include `shuffle_buffer_size=1000`, which is more than sufficient for this small dataset, but may not be for a real-world dataset."
+        "Note: If you run the above cell twice it will produce different results. The default settings for `tf.data.experimental.make_csv_dataset` include `shuffle_buffer_size=1000`, which is more than sufficient for this small dataset, but may not be for a real-world dataset."
       ]
     },
     {
@@ -986,7 +986,7 @@
         "id": "d6uviU_KCCWD"
       },
       "source": [
-        "It can also decompress the data on the fly. Here's a gzipped CSV file containing the [metro interstate traffic dataset](https://archive.ics.uci.edu/ml/datasets/Metro+Interstate+Traffic+Volume)\n",
+        "It can also decompress the data on the fly. Here's a gzipped CSV file containing the [metro interstate traffic dataset](https://archive.ics.uci.edu/ml/datasets/Metro+Interstate+Traffic+Volume).\n",
         "\n",
         "![A traffic jam.](images/csv/traffic.jpg)\n",
         "\n",
@@ -1013,7 +1013,7 @@
         "id": "F-IOsFHbCw0i"
       },
       "source": [
-        "Set the `compression_type` argument to read directly from the compressed file: "
+        "Set the `compression_type` argument to read directly from the compressed file:"
       ]
     },
     {
@@ -1044,7 +1044,7 @@
         "id": "p12Y6tGq8D6M"
       },
       "source": [
-        "Note: If you need to parse those date-time strings in the `tf.data` pipeline you can use `tfa.text.parse_time`."
+        "Note: If you need to parse those date-time strings in the `tf.data` pipeline, you can use `tfa.text.parse_time`."
       ]
     },
     {
@@ -1062,13 +1062,13 @@
         "id": "fN2dL_LRP83r"
       },
       "source": [
-        "There is some overhead to parsing the csv data. For small models this can be the bottleneck in training.\n",
+        "There is some overhead to parsing the CSV data. For small models this can be the bottleneck in training.\n",
         "\n",
-        "Depending on your use case it may be a good idea to use `Dataset.cache` or `data.experimental.snapshot` so that the csv data is only parsed on the first epoch. \n",
+        "Depending on your use case, it may be a good idea to use `Dataset.cache` or `tf.data.experimental.snapshot`, so that the CSV data is only parsed on the first epoch.\n",
         "\n",
         "The main difference between the `cache` and `snapshot` methods is that `cache` files can only be used by the TensorFlow process that created them, but `snapshot` files can be read by other processes.\n",
         "\n",
-        "For example, iterating over the `traffic_volume_csv_gz_ds` 20 times, takes ~15 seconds without caching, or ~2s with caching."
+        "For example, iterating over the `traffic_volume_csv_gz_ds` 20 times may take around 15 seconds without caching, or about two seconds with caching."
       ]
     },
     {
@@ -1092,7 +1092,7 @@
         "id": "pN3HtDONh5TX"
       },
       "source": [
-        "Note: `Dataset.cache`  stores the data form the first epoch and replays it in order. So using `.cache` disables any shuffles earlier in the pipeline. Below the `.shuffle` is added back in after `.cache`."
+        "Note: `Dataset.cache` stores the data from the first epoch and replays it in order. So, using the `cache` method disables any shuffles earlier in the pipeline. Below, `Dataset.shuffle` is added back in after `Dataset.cache`."
       ]
     },
     {
@@ -1118,7 +1118,7 @@
         "id": "wN7uUBjmgNZ9"
       },
       "source": [
-        "Note: `snapshot` files are meant for *temporary* storage of a dataset while in use. This is *not* a format for long term storage. The file format is considered an internal detail, and not guaranteed between TensorFlow versions. "
+        "Note: The `tf.data.experimental.snapshot` files are meant for *temporary* storage of a dataset while in use. This is *not* a format for long term storage. The file format is considered an internal detail, and not guaranteed between TensorFlow versions."
       ]
     },
     {
@@ -1145,7 +1145,7 @@
         "id": "fUSSegnMCGRz"
       },
       "source": [
-        "If your data loading is slowed by loading csv files, and `cache` and `snapshot` are insufficient for your use case, consider re-encoding your data into a more streamlined format."
+        "If your data loading is slowed by loading CSV files, and `Dataset.cache` and `tf.data.experimental.snapshot` are insufficient for your use case, consider re-encoding your data into a more streamlined format."
       ]
     },
     {
@@ -1171,7 +1171,7 @@
         "\n",
         "Image by Willi Heidelbach from Pixabay\n",
         "\n",
-        "Download the dataset, and have a look at the files inside:"
+        "Download the dataset, and review the files inside:"
       ]
     },
     {
@@ -1219,7 +1219,7 @@
         "id": "19Udrw9iG-FS"
       },
       "source": [
-        "When dealing with a bunch of files you can pass a glob-style `file_pattern` to the `experimental.make_csv_dataset` function. The order of the files is shuffled each iteration.\n",
+        "When dealing with a bunch of files, you can pass a glob-style `file_pattern` to the `tf.data.experimental.make_csv_dataset` function. The order of the files is shuffled each iteration.\n",
         "\n",
         "Use the `num_parallel_reads` argument to set how many files are read in parallel and interleaved together."
       ]
@@ -1245,7 +1245,7 @@
         "id": "XMoexinLHYFa"
       },
       "source": [
-        "These csv files have the images flattened out into a single row. The column names are formatted `r{row}c{column}`. Here's the first batch:"
+        "These CSV files have the images flattened out into a single row. The column names are formatted `r{row}c{column}`. Here's the first batch:"
       ]
     },
     {
@@ -1273,7 +1273,7 @@
       "source": [
         "#### Optional: Packing fields\n",
         "\n",
-        "You probably don't want to work with each pixel in separate columns like this. Before trying to use this dataset be sure to pack the pixels into an image-tensor. \n",
+        "You probably don't want to work with each pixel in separate columns like this. Before trying to use this dataset be sure to pack the pixels into an image-tensor.\n",
         "\n",
         "Here is code that parses the column names to build images for each example:"
       ]
@@ -1372,12 +1372,12 @@
         "id": "3jiGZeUijJNd"
       },
       "source": [
-        "So far this tutorial has focused on the highest level utilities for reading csv data. There are other two APIs that may be helpful for advanced users if your use-case doesn't fit the basic patterns.\n",
+        "So far this tutorial has focused on the highest-level utilities for reading csv data. There are other two APIs that may be helpful for advanced users if your use-case doesn't fit the basic patterns.\n",
         "\n",
-        "* `tf.io.decode_csv` - a function for parsing lines of text into a list of CSV column tensors.\n",
-        "* `tf.data.experimental.CsvDataset` - a lower level csv dataset constructor.\n",
+        "* `tf.io.decode_csv`: a function for parsing lines of text into a list of CSV column tensors.\n",
+        "* `tf.data.experimental.CsvDataset`: a lower-level CSV dataset constructor.\n",
         "\n",
-        "This section recreates functionality provided by `make_csv_dataset`, to demonstrate how this lower level functionality can be used.\n"
+        "This section recreates functionality provided by `tf.data.experimental.make_csv_dataset`, to demonstrate how this lower-level functionality can be used.\n"
       ]
     },
     {
@@ -1390,9 +1390,9 @@
         "\n",
         "This function decodes a string, or list of strings into a list of columns.\n",
         "\n",
-        "Unlike `make_csv_dataset` this function does not try to guess column data-types. You specify the column types by providing a list of `record_defaults` containing a value of the correct type, for each column.\n",
+        "Unlike `tf.data.experimental.make_csv_dataset` this function does not try to guess column data-types. You specify the column types by providing a list of `record_defaults` containing a value of the correct type, for each column.\n",
         "\n",
-        "To read the Titanic data **as strings** using `decode_csv` you would say: "
+        "To read the Titanic data **as strings** using `tf.io.decode_csv` you would say:"
       ]
     },
     {
@@ -1476,7 +1476,7 @@
         "id": "m-LkTUTnpn2P"
       },
       "source": [
-        "Note: it is more efficient to call `decode_csv` on large batches of lines than on individual lines of csv text."
+        "Note: It is more efficient to call `tf.io.decode_csv` on large batches of lines than on individual lines of CSV text."
       ]
     },
     {
@@ -1487,7 +1487,7 @@
       "source": [
         "### `tf.data.experimental.CsvDataset`\n",
         "\n",
-        "The `tf.data.experimental.CsvDataset` class provides a minimal CSV `Dataset` interface without the convenience features of the `make_csv_dataset` function: column header parsing, column type-inference, automatic shuffling, file interleaving.\n",
+        "The `tf.data.experimental.CsvDataset` class provides a minimal CSV `Dataset` interface without the convenience features of the `tf.data.experimental.make_csv_dataset` function: column header parsing, column type-inference, automatic shuffling, file interleaving.\n",
         "\n",
         "This constructor follows uses `record_defaults` the same way as `io.parse_csv`:\n"
       ]
@@ -1547,7 +1547,7 @@
       "source": [
         "#### Multiple files\n",
         "\n",
-        "To parse the fonts dataset using `experimental.CsvDataset`, you first need to determine the column types for the `record_defaults`. Start by inspecting the first row of one file: "
+        "To parse the fonts dataset using `tf.data.experimental.CsvDataset`, you first need to determine the column types for the `record_defaults`. Start by inspecting the first row of one file:"
       ]
     },
     {
@@ -1568,7 +1568,7 @@
         "id": "etyGu8K_ySRz"
       },
       "source": [
-        "Only the first two fields are strings, the rest are ints or floats, and you can get the total number of features by counting the commas:"
+        "Only the first two fields are strings, the rest are integers or floats, and you can get the total number of features by counting the commas:"
       ]
     },
     {
@@ -1589,7 +1589,7 @@
         "id": "YeK2Pw540RNj"
       },
       "source": [
-        "The `CsvDatasaet` constructor can take a list of input files, but reads them sequentially. The first file in the list of CSVs is `AGENCY.csv`:"
+        "The `tf.data.experimental.CsvDataset` constructor can take a list of input files, but reads them sequentially. The first file in the list of CSVs is `AGENCY.csv`:"
       ]
     },
     {
@@ -1609,7 +1609,7 @@
         "id": "EfAX3G8Xywy6"
       },
       "source": [
-        "So when you pass the list of files to `CsvDataaset` the records from `AGENCY.csv` are read first:"
+        "So, when you pass the list of files to `CsvDataset`, the records from `AGENCY.csv` are read first:"
       ]
     },
     {
@@ -1646,7 +1646,7 @@
       "source": [
         "To interleave multiple files, use `Dataset.interleave`.\n",
         "\n",
-        "Here's an initial dataset that contains the csv file names: "
+        "Here's an initial dataset that contains the CSV file names: "
       ]
     },
     {
@@ -1695,9 +1695,9 @@
         "id": "B0QB1PtU3WAN"
       },
       "source": [
-        "The `interleave` method takes a `map_func` that creates a child-`Dataset` for each element of the parent-`Dataset`. \n",
+        "The `interleave` method takes a `map_func` that creates a child-`Dataset` for each element of the parent-`Dataset`.\n",
         "\n",
-        "Here, you want to create a `CsvDataset` from each element of the dataset of files:"
+        "Here, you want to create a `tf.data.experimental.CsvDataset` from each element of the dataset of files:"
       ]
     },
     {
@@ -1768,7 +1768,7 @@
         "id": "8BtGHraUApdJ"
       },
       "source": [
-        "Earlier, it was noted that `io.decode_csv` is more efficient when run on a batch of strings.\n",
+        "Earlier, it was noted that `tf.io.decode_csv` is more efficient when run on a batch of strings.\n",
         "\n",
         "It is possible to take advantage of this fact, when using large batch sizes, to improve CSV loading performance (but try [caching](#caching) first)."
       ]
@@ -1858,9 +1858,9 @@
         "id": "aebC1plsMeOi"
       },
       "source": [
-        "For another example of increasing csv performance by using large batches see the [overfit and underfit tutorial](../keras/overfit_and_underfit.ipynb).\n",
+        "For another example of increasing CSV performance by using large batches, refer to the [Overfit and underfit tutorial](../keras/overfit_and_underfit.ipynb).\n",
         "\n",
-        "This sort of approach may work, but consider other options like `cache` and `snapshot`, or re-encoding your data into a more streamlined format."
+        "This sort of approach may work, but consider other options like `Dataset.cache` and `tf.data.experimental.snapshot`, or re-encoding your data into a more streamlined format."
       ]
     }
   ],

From 6554297ac6c384153664cde6a959f8cac3bea815 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 14:04:49 +0100
Subject: [PATCH 177/872] Update optimizer alias to Keras in TensorFlow basics
 guide, lint the doc

---
 site/en/guide/basics.ipynb | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index e9499d3f25a..0fe53c9920a 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -637,7 +637,7 @@
         "y = f(x) + tf.random.normal(shape=[201])\n",
         "\n",
         "plt.plot(x.numpy(), y.numpy(), '.', label='Data')\n",
-        "plt.plot(x, f(x),  label='Ground truth')\n",
+        "plt.plot(x, f(x), label='Ground truth')\n",
         "plt.legend();"
       ]
     },
@@ -720,7 +720,7 @@
       "source": [
         "variables = model.variables\n",
         "\n",
-        "optimizer = tf.optimizers.SGD(learning_rate=0.01)\n",
+        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)\n",
         "\n",
         "for step in range(1000):\n",
         "  with tf.GradientTape() as tape:\n",
@@ -755,7 +755,7 @@
         "id": "hbtmFJIXb6qm"
       },
       "source": [
-        "That's working, but remember that implementations of common training utilities are available in the `tf.keras` module. So consider using those before writing your own. To start with, the `Model.compile` and  `Model.fit` methods implement a training loop for you:"
+        "That's working, but remember that implementations of common training utilities are available in the `tf.keras` module. So, consider using those before writing your own. To start with, the `Model.compile` and `Model.fit` methods implement a training loop for you:"
       ]
     },
     {
@@ -779,7 +779,7 @@
       "source": [
         "new_model.compile(\n",
         "    loss=tf.keras.losses.MSE,\n",
-        "    optimizer=tf.optimizers.SGD(learning_rate=0.01))\n",
+        "    optimizer=tf.keras.optimizers.SGD(learning_rate=0.01))\n",
         "\n",
         "history = new_model.fit(x, y,\n",
         "                        epochs=100,\n",

From a0ff1c106e7a6ddacafbba0a8a11f354977115d3 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 14:59:34 +0100
Subject: [PATCH 178/872] Update tf.keras.optimizers in Introduction to graphs
 and tf.function, lint the guide

---
 site/en/guide/intro_to_graphs.ipynb | 51 +++++++++++++++++------------
 1 file changed, 30 insertions(+), 21 deletions(-)

diff --git a/site/en/guide/intro_to_graphs.ipynb b/site/en/guide/intro_to_graphs.ipynb
index 19b5c5f432e..6ff1ad8f272 100644
--- a/site/en/guide/intro_to_graphs.ipynb
+++ b/site/en/guide/intro_to_graphs.ipynb
@@ -37,7 +37,7 @@
         "id": "6xgB0Oz5eGSQ"
       },
       "source": [
-        "# Introduction to graphs and tf.function"
+        "# Introduction to graphs and `tf.function`"
       ]
     },
     {
@@ -76,7 +76,7 @@
         "\n",
         "Note: For those of you who are only familiar with TensorFlow 1.x, this guide demonstrates a very different view of graphs.\n",
         "\n",
-        "**This is a big-picture overview that covers how `tf.function` allows you to switch from eager execution to graph execution.** For a more complete specification of `tf.function`, go to the [`tf.function` guide](function.ipynb).\n"
+        "**This is a big-picture overview that covers how `tf.function` allows you to switch from eager execution to graph execution.** For a more complete specification of `tf.function`, go to the [Better performance with `tf.function`](./function.ipynb) guide.\n"
       ]
     },
     {
@@ -93,7 +93,7 @@
         "\n",
         "**Graphs are data structures that contain a set of `tf.Operation` objects, which represent units of computation; and `tf.Tensor` objects, which represent the units of data that flow between operations.** They are defined in a `tf.Graph` context. Since these graphs are data structures, they can be saved, run, and restored all without the original Python code.\n",
         "\n",
-        "This is what a TensorFlow graph representing a two-layer neural network looks like when visualized in TensorBoard.\n"
+        "This is what a TensorFlow graph representing a two-layer neural network looks like when visualized in TensorBoard:"
       ]
     },
     {
@@ -113,7 +113,7 @@
       "source": [
         "### The benefits of graphs\n",
         "\n",
-        "With a graph, you have a great deal of flexibility.  You can use your TensorFlow graph in environments that don't have a Python interpreter, like mobile applications, embedded devices, and backend servers.  TensorFlow uses graphs as the format for [saved models](saved_model) when it exports them from Python.\n",
+        "With a graph, you have a great deal of flexibility.  You can use your TensorFlow graph in environments that don't have a Python interpreter, like mobile applications, embedded devices, and backend servers. TensorFlow uses graphs as the format for [saved models](./saved_model.ipynb) when it exports them from Python.\n",
         "\n",
         "Graphs are also easily optimized, allowing the compiler to do transformations like:\n",
         "\n",
@@ -144,6 +144,15 @@
         "## Setup"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0d1689fa928f"
+      },
+      "source": [
+        "Import some necessary libraries:"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -202,7 +211,7 @@
         "id": "PNvuAYpdrTOf"
       },
       "source": [
-        "On the outside, a `Function` looks like a regular function you write using TensorFlow operations. [Underneath](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/eager/def_function.py), however, it is *very different*. A `Function` **encapsulates [several `tf.Graph`s behind one API](#polymorphism_one_function_many_graphs).** That is how `Function` is able to give you the [benefits of graph execution](#the_benefits_of_graphs), like speed and deployability."
+        "On the outside, a `Function` looks like a regular function you write using TensorFlow operations. [Underneath](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/eager/def_function.py), however, it is *very different*. A `Function` **encapsulates several `tf.Graph`s behind one API** (learn more in the _Polymorphism_ section). That is how a `Function` is able to give you the benefits of graph execution, like speed and deployability (refer to _The benefits of graphs_ above)."
       ]
     },
     {
@@ -320,7 +329,7 @@
         "id": "GZ4Ieg6tBE6l"
       },
       "source": [
-        "Most of the time, `tf.function` will work without  special considerations.  However, there are some caveats, and the [tf.function guide](./function.ipynb) can help here, as well as the [complete AutoGraph reference](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/index.md)"
+        "Most of the time, `tf.function` will work without  special considerations. However, there are some caveats, and the [`tf.function` guide](./function.ipynb) can help here, as well as the [complete AutoGraph reference](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/index.md)."
       ]
     },
     {
@@ -333,7 +342,7 @@
         "\n",
         "A `tf.Graph` is specialized to a specific type of inputs (for example, tensors with a specific [`dtype`](https://www.tensorflow.org/api_docs/python/tf/dtypes/DType) or objects with the same [`id()`](https://docs.python.org/3/library/functions.html#id])).\n",
         "\n",
-        "Each time you invoke a `Function` with a set of arguments that can't be handled by any of its existing graphs (such as arguments with new `dtypes` or incompatible shapes), `Function` creates a new `tf.Graph` specialized to those new arguments. The type specification of a `tf.Graph`'s inputs is known as its **input signature** or just a **signature**. For more information regarding when a new `tf.Graph` is generated and how that can be controlled, see the [rules of retracing](https://www.tensorflow.org/guide/function#rules_of_tracing).\n",
+        "Each time you invoke a `Function` with a set of arguments that can't be handled by any of its existing graphs (such as arguments with new `dtypes` or incompatible shapes), `Function` creates a new `tf.Graph` specialized to those new arguments. The type specification of a `tf.Graph`'s inputs is known as its **input signature** or just a **signature**. For more information regarding when a new `tf.Graph` is generated and how that can be controlled, go to the _Rules of tracing_ section of the [Better performance with `tf.function`](./function.ipynb) guide.\n",
         "\n",
         "The `Function` stores the `tf.Graph` corresponding to that signature in a `ConcreteFunction`. **A `ConcreteFunction` is a wrapper around a `tf.Graph`.**\n"
       ]
@@ -384,7 +393,7 @@
         "id": "UohRmexhIpvQ"
       },
       "source": [
-        "Because it's backed by multiple graphs, a `Function` is **polymorphic**. That enables it to support more input types than a single `tf.Graph` could represent, as well as to optimize each `tf.Graph` for better performance."
+        "Because it's backed by multiple graphs, a `Function` is **polymorphic**. That enables it to support more input types than a single `tf.Graph` could represent, and to optimize each `tf.Graph` for better performance."
       ]
     },
     {
@@ -467,7 +476,7 @@
         "id": "cyZNCRcQorGO"
       },
       "source": [
-        "To verify that your `Function`'s graph is doing the same computation as its equivalent Python function, you can make it execute eagerly with `tf.config.run_functions_eagerly(True)`.  This is a switch that **turns off `Function`'s ability to create and run graphs**, instead executing the code normally."
+        "To verify that your `Function`'s graph is doing the same computation as its equivalent Python function, you can make it execute eagerly with `tf.config.run_functions_eagerly(True)`. This is a switch that **turns off `Function`'s ability to create and run graphs**, instead of executing the code normally."
       ]
     },
     {
@@ -510,7 +519,7 @@
         "id": "DKT3YBsqy0x4"
       },
       "source": [
-        "However, `Function` can behave differently under graph and eager execution. The Python [`print`](https://docs.python.org/3/library/functions.html#print) function is one example of how these two modes differ. Let's check out what happens when you insert a `print` statement to your function and call it repeatedly.\n"
+        "However, `Function` can behave differently under graph and eager execution. The Python [`print`](https://docs.python.org/3/library/functions.html#print) function is one example of how these two modes differ. Let's check out what happens when you insert a `print` statement to your function and call it repeatedly."
       ]
     },
     {
@@ -558,7 +567,7 @@
       "source": [
         "Is the output surprising? **`get_MSE` only printed once even though it was called *three* times.**\n",
         "\n",
-        "To explain, the `print` statement is executed when `Function` runs the original code in order to create the graph in a process known as [\"tracing\"](function.ipynb#tracing). **Tracing captures the TensorFlow operations into a graph, and `print`  is not captured in the graph.**  That graph is then executed for all three calls **without ever running the Python code again**.\n",
+        "To explain, the `print` statement is executed when `Function` runs the original code in order to create the graph in a process known as \"tracing\" (refer to the _Tracing_ section of the [`tf.function` guide](./function.ipynb). **Tracing captures the TensorFlow operations into a graph, and `print` is not captured in the graph.**  That graph is then executed for all three calls **without ever running the Python code again**.\n",
         "\n",
         "As a sanity check, let's turn off graph execution to compare:"
       ]
@@ -606,7 +615,7 @@
         "id": "PUR7qC_bquCn"
       },
       "source": [
-        "`print` is a *Python side effect*, and there are other differences that you should be aware of when converting a function into a `Function`. Learn more in the _Limitations_ section of the [Better performance with tf.function](./function.ipynb#limitations) guide."
+        "`print` is a *Python side effect*, and there are other differences that you should be aware of when converting a function into a `Function`. Learn more in the _Limitations_ section of the [Better performance with `tf.function`](./function.ipynb#limitations) guide."
       ]
     },
     {
@@ -676,7 +685,7 @@
         "  tf.gather(x, [1]) # unused\n",
         "  return x\n",
         "\n",
-        "# Only needed operations are run during graph exection. The error is not raised.\n",
+        "# Only needed operations are run during graph execution. The error is not raised.\n",
         "print(unused_return_graph(tf.constant([0.0])))"
       ]
     },
@@ -686,16 +695,16 @@
         "id": "def6MupG9R0O"
       },
       "source": [
-        "###`tf.function` best practices\n",
+        "### `tf.function` best practices\n",
         "\n",
         "It may take some time to get used to the behavior of `Function`.  To get started quickly, first-time users should play around with decorating toy functions with `@tf.function` to get experience with going from eager to graph execution.\n",
         "\n",
         "*Designing for `tf.function`* may be your best bet for writing graph-compatible TensorFlow programs. Here are some tips:\n",
         "-  Toggle between eager and graph execution early and often with `tf.config.run_functions_eagerly` to pinpoint if/ when the two modes diverge.\n",
         "- Create `tf.Variable`s\n",
-        "outside the Python function and modify them on the inside. The same goes for objects that use `tf.Variable`, like `keras.layers`, `keras.Model`s and `tf.optimizers`.\n",
-        "- Avoid writing functions that [depend on outer Python variables](function#depending_on_python_global_and_free_variables), excluding `tf.Variable`s and Keras objects.\n",
-        "- Prefer to write functions which take tensors and other TensorFlow types as input. You can pass in other object types but [be careful](function#depending_on_python_objects)!\n",
+        "outside the Python function and modify them on the inside. The same goes for objects that use `tf.Variable`, like `tf.keras.layers`, `tf.keras.Model`s and `tf.keras.optimizers`.\n",
+        "- Avoid writing functions that depend on outer Python variables, excluding `tf.Variable`s and Keras objects. Learn more in _Depending on Python global and free variables_ of the [`tf.function` guide](./function.ipynb).\n",
+        "- Prefer to write functions which take tensors and other TensorFlow types as input. You can pass in other object types but be careful! Learn more in _Depending on Python objects_ of the [`tf.function` guide](./function.ipynb).\n",
         "- Include as much computation as possible under a `tf.function` to maximize the performance gain. For example, decorate a whole training step or the entire training loop.\n"
       ]
     },
@@ -763,9 +772,9 @@
         "id": "Q1Pfo5YwwILi"
       },
       "source": [
-        "`tf.function` is commonly used to speed up training loops, and you can learn more about it in [Writing a training loop from scratch](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch#speeding-up_your_training_step_with_tffunction) with Keras.\n",
+        "`tf.function` is commonly used to speed up training loops, and you can learn more about it in the _Speeding-up your training step with `tf.function`_ section of the [Writing a training loop from scratch](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch) with Keras guide.\n",
         "\n",
-        "Note: You can also try [`tf.function(jit_compile=True)`](https://www.tensorflow.org/xla#explicit_compilation_with_tffunctionjit_compiletrue) for a more significant performance boost, especially if your code is heavy on TF control flow and uses many small tensors."
+        "Note: You can also try `tf.function(jit_compile=True)` for a more significant performance boost, especially if your code is heavy on TensorFlow control flow and uses many small tensors. Learn more in the _Explicit compilation with `tf.function(jit_compile=True)`_ section of the [XLA overview](https://www.tensorflow.org/xla)."
       ]
     },
     {
@@ -778,7 +787,7 @@
         "\n",
         "Graphs can speed up your code, but the process of creating them has some overhead. For some functions, the creation of the graph takes more time than the execution of the graph. **This investment is usually quickly paid back with the performance boost of subsequent executions, but it's important to be aware that the first few  steps of any large model training can be slower due to tracing.**\n",
         "\n",
-        "No matter how large your model, you want to avoid tracing frequently. The `tf.function` guide discusses [how to set input specifications and use tensor arguments](function#controlling_retracing) to avoid retracing.  If you find you are getting unusually poor performance, it's a good idea to check if you are retracing accidentally."
+        "No matter how large your model, you want to avoid tracing frequently. The [`tf.function` guide](./function.ipynb) discusses how to set input specifications and use tensor arguments to avoid retracing in the _Controlling retracing_ section. If you find you are getting unusually poor performance, it's a good idea to check if you are retracing accidentally."
       ]
     },
     {
@@ -843,7 +852,7 @@
       "source": [
         "## Next steps\n",
         "\n",
-        "You can learn more about `tf.function` on the API reference page and by following the [Better performance with `tf.function`](function.ipynb) guide."
+        "You can learn more about `tf.function` on the API reference page and by following the [Better performance with `tf.function`](./function.ipynb) guide."
       ]
     }
   ],

From 560589c62ad4e3bfca3b538d1cb25b0e6f18676f Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 15:17:12 +0100
Subject: [PATCH 179/872] Update Keras optimizer alias in Neural style
 transfer, lint the doc

---
 site/en/tutorials/generative/style_transfer.ipynb | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/site/en/tutorials/generative/style_transfer.ipynb b/site/en/tutorials/generative/style_transfer.ipynb
index fca7d8a40d7..ef8590b5afa 100644
--- a/site/en/tutorials/generative/style_transfer.ipynb
+++ b/site/en/tutorials/generative/style_transfer.ipynb
@@ -73,7 +73,7 @@
       "source": [
         "This tutorial uses deep learning to compose one image in the style of another image (ever wish you could paint like Picasso or Van Gogh?). This is known as *neural style transfer* and the technique is outlined in A Neural Algorithm of Artistic Style (Gatys et al.). \n",
         "\n",
-        "Note: This tutorial demonstrates the original style-transfer algorithm. It optimizes the image content to a particular style. Modern approaches train a model to generate the stylized image directly (similar to [cyclegan](cyclegan.ipynb)). This approach is much faster (up to 1000x).\n",
+        "Note: This tutorial demonstrates the original style-transfer algorithm. It optimizes the image content to a particular style. Modern approaches train a model to generate the stylized image directly (similar to [CycleGAN](./cyclegan.ipynb)). This approach is much faster (up to 1000x).\n",
         "\n",
         "For a simple application of style transfer check out this [tutorial](https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization) to learn more about how to use the pretrained [Arbitrary Image Stylization model](https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2) from [TensorFlow Hub](https://tfhub.dev) or how to use a style transfer model with [TensorFlow Lite](https://www.tensorflow.org/lite/models/style_transfer/overview). "
       ]
@@ -104,7 +104,7 @@
         "\n",
         "\n",
         "\n",
-        "Now how would it look like if Kandinsky decided to paint the picture of this Dog exclusively with this style? Something like this?\n",
+        "Now, what would it look like if Kandinsky decided to paint the picture of this Dog exclusively with this style? Something like this?\n",
         "\n",
         ""
       ]
@@ -453,8 +453,8 @@
       "outputs": [],
       "source": [
         "def vgg_layers(layer_names):\n",
-        "  \"\"\" Creates a vgg model that returns a list of intermediate output values.\"\"\"\n",
-        "  # Load our model. Load pretrained VGG, trained on imagenet data\n",
+        "  \"\"\" Creates a VGG model that returns a list of intermediate output values.\"\"\"\n",
+        "  # Load our model. Load pretrained VGG, trained on ImageNet data\n",
         "  vgg = tf.keras.applications.VGG19(include_top=False, weights='imagenet')\n",
         "  vgg.trainable = False\n",
         "  \n",
@@ -694,7 +694,7 @@
         "id": "MBU5RFpcAo7W"
       },
       "source": [
-        "Create an optimizer. The paper recommends LBFGS, but `Adam` works okay, too:"
+        "Create an optimizer. The paper recommends LBFGS, but Adam works okay, too:"
       ]
     },
     {
@@ -705,7 +705,7 @@
       },
       "outputs": [],
       "source": [
-        "opt = tf.optimizers.Adam(learning_rate=0.02, beta_1=0.99, epsilon=1e-1)"
+        "opt = tf.keras.optimizers.Adam(learning_rate=0.02, beta_1=0.99, epsilon=1e-1)"
       ]
     },
     {

From 8e589114347c957076e3bc7037f9bbaa06a5e9b2 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 15:21:24 +0100
Subject: [PATCH 180/872] Update tf.keras.optimizer alias in Basic regression
 tutorial, lint the doc

---
 site/en/tutorials/keras/regression.ipynb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/site/en/tutorials/keras/regression.ipynb b/site/en/tutorials/keras/regression.ipynb
index 3dd3a4041f6..23a8bde4271 100644
--- a/site/en/tutorials/keras/regression.ipynb
+++ b/site/en/tutorials/keras/regression.ipynb
@@ -616,7 +616,7 @@
       "outputs": [],
       "source": [
         "horsepower_model.compile(\n",
-        "    optimizer=tf.optimizers.Adam(learning_rate=0.1),\n",
+        "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.1),\n",
         "    loss='mean_absolute_error')"
       ]
     },
@@ -863,7 +863,7 @@
       "outputs": [],
       "source": [
         "linear_model.compile(\n",
-        "    optimizer=tf.optimizers.Adam(learning_rate=0.1),\n",
+        "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.1),\n",
         "    loss='mean_absolute_error')"
       ]
     },
@@ -961,7 +961,7 @@
         "* Two hidden, non-linear, `Dense` layers with the ReLU (`relu`) activation function nonlinearity.\n",
         "* A linear `Dense` single-output layer.\n",
         "\n",
-        "Both models will use the same training procedure so the `compile` method is included in the `build_and_compile_model` function below."
+        "Both models will use the same training procedure, so the `compile` method is included in the `build_and_compile_model` function below."
       ]
     },
     {

From 2ec0c137e43d55ad6bc4de01350deac77602cbcf Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 15:24:36 +0100
Subject: [PATCH 181/872] Lint tf.function guide link in Intro to graphs guide

---
 site/en/guide/intro_to_graphs.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/intro_to_graphs.ipynb b/site/en/guide/intro_to_graphs.ipynb
index 6ff1ad8f272..108e9d89c9d 100644
--- a/site/en/guide/intro_to_graphs.ipynb
+++ b/site/en/guide/intro_to_graphs.ipynb
@@ -615,7 +615,7 @@
         "id": "PUR7qC_bquCn"
       },
       "source": [
-        "`print` is a *Python side effect*, and there are other differences that you should be aware of when converting a function into a `Function`. Learn more in the _Limitations_ section of the [Better performance with `tf.function`](./function.ipynb#limitations) guide."
+        "`print` is a *Python side effect*, and there are other differences that you should be aware of when converting a function into a `Function`. Learn more in the _Limitations_ section of the [Better performance with `tf.function`](./function.ipynb) guide."
       ]
     },
     {

From de3512f9b9785f090b94e29859bc1f2eb6fa35af Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 15:34:44 +0100
Subject: [PATCH 182/872] Lint Custom training with tf.distribute.Strategy
 tutorial

---
 .../distribute/custom_training.ipynb          | 27 ++++++++++---------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index 4ea0d1ef424..0ffbe23b753 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -37,7 +37,7 @@
         "id": "jYysdyb-CaWM"
       },
       "source": [
-        "# Custom training with tf.distribute.Strategy"
+        "# Custom training with `tf.distribute.Strategy`"
       ]
     },
     {
@@ -68,9 +68,9 @@
         "id": "FbVhjPpzn6BM"
       },
       "source": [
-        "This tutorial demonstrates how to use `tf.distribute.Strategy` — a TensorFlow API that provides an abstraction for [distributing your training](../../guide/distributed_training.ipynb) across multiple processing units (GPUs, multiple machines, or TPUs) — with custom training loops. In this example, you will train a simple convolutional neural network on the [Fashion MNIST dataset](https://github.com/zalandoresearch/fashion-mnist) containing 70,000 images of size 28 x 28.\n",
+        "This tutorial demonstrates how to use `tf.distribute.Strategy`—a TensorFlow API that provides an abstraction for [distributing your training](../../guide/distributed_training.ipynb) across multiple processing units (GPUs, multiple machines, or TPUs)—with custom training loops. In this example, you will train a simple convolutional neural network on the [Fashion MNIST dataset](https://github.com/zalandoresearch/fashion-mnist) containing 70,000 images of size 28 x 28.\n",
         "\n",
-        "[Custom training loops](../customization/custom_training_walkthrough.ipynb) provide flexibility and a greater control on training. They also make it is easier to debug the model and the training loop."
+        "[Custom training loops](../customization/custom_training_walkthrough.ipynb) provide flexibility and a greater control on training. They also make it easier to debug the model and the training loop."
       ]
     },
     {
@@ -312,13 +312,13 @@
         "optional sample weights, and `GLOBAL_BATCH_SIZE` as arguments and returns the scaled loss.\n",
         "\n",
         "* If you are using regularization losses in your model then you need to scale\n",
-        "the loss value by number of replicas. You can do this by using the `tf.nn.scale_regularization_loss` function.\n",
+        "the loss value by the number of replicas. You can do this by using the `tf.nn.scale_regularization_loss` function.\n",
         "\n",
         "* Using `tf.reduce_mean` is not recommended. Doing so divides the loss by actual per replica batch size which may vary step to step.\n",
         "\n",
-        "* This reduction and scaling is done automatically in keras `model.compile` and `model.fit`\n",
+        "* This reduction and scaling is done automatically in Keras `Model.compile` and `Model.fit`\n",
         "\n",
-        "* If using `tf.keras.losses` classes (as in the example below), the loss reduction needs to be explicitly specified to be one of `NONE` or `SUM`. `AUTO` and `SUM_OVER_BATCH_SIZE`  are disallowed when used with `tf.distribute.Strategy`. `AUTO` is disallowed because the user should explicitly think about what reduction they want to make sure it is correct in the distributed case. `SUM_OVER_BATCH_SIZE` is disallowed because currently it would only divide by per replica batch size, and leave the dividing by number of replicas to the user, which might be easy to miss. So instead we ask the user do the reduction themselves explicitly.\n",
+        "* If using `tf.keras.losses` classes (as in the example below), the loss reduction needs to be explicitly specified to be one of `NONE` or `SUM`. `AUTO` and `SUM_OVER_BATCH_SIZE`  are disallowed when used with `tf.distribute.Strategy`. `AUTO` is disallowed because the user should explicitly think about what reduction they want to make sure it is correct in the distributed case. `SUM_OVER_BATCH_SIZE` is disallowed because currently it would only divide by per replica batch size, and leave the dividing by number of replicas to the user, which might be easy to miss. So, instead, you need to do the reduction yourself explicitly.\n",
         "* If `labels` is multi-dimensional, then average the `per_example_loss` across the number of elements in each sample. For example, if the shape of `predictions` is `(batch_size, H, W, n_classes)` and `labels` is `(batch_size, H, W)`, you will need to update `per_example_loss` like: `per_example_loss /= tf.cast(tf.reduce_prod(tf.shape(labels)[1:]), tf.float32)`\n",
         "\n",
         "  Caution: **Verify the shape of your loss**. \n",
@@ -397,7 +397,7 @@
       },
       "outputs": [],
       "source": [
-        "# model, optimizer, and checkpoint must be created under `strategy.scope`.\n",
+        "# A model, an optimizer, and a checkpoint must be created under `strategy.scope`.\n",
         "with strategy.scope():\n",
         "  model = create_model()\n",
         "\n",
@@ -575,7 +575,7 @@
         "\n",
         "### Using iterators\n",
         "\n",
-        "If you want to iterate over a given number of steps and not through the entire dataset you can create an iterator using the `iter` call and explicity call `next` on the iterator. You can choose to iterate over the dataset both inside and outside the tf.function. Here is a small snippet demonstrating iteration of the dataset outside the tf.function using an iterator.\n"
+        "If you want to iterate over a given number of steps and not through the entire dataset, you can create an iterator using the `iter` call and explicitly call `next` on the iterator. You can choose to iterate over the dataset both inside and outside the `tf.function`. Here is a small snippet demonstrating iteration of the dataset outside the `tf.function` using an iterator.\n"
       ]
     },
     {
@@ -607,7 +607,8 @@
         "id": "GxVp48Oy0m6y"
       },
       "source": [
-        "### Iterating inside a tf.function\n",
+        "### Iterating inside a `tf.function`\n",
+        "\n",
         "You can also iterate over the entire input `train_dist_dataset` inside a `tf.function` using the `for x in ...` construct or by creating iterators like you did above. The example below demonstrates wrapping one epoch of training with a `@tf.function` decorator and iterating over `train_dist_dataset` inside the function."
       ]
     },
@@ -652,6 +653,7 @@
         "Because of the loss scaling computation that is carried out, it's not recommended to use `tf.keras.metrics.Mean` to track the training loss across different replicas.\n",
         "\n",
         "For example, if you run a training job with the following characteristics:\n",
+        "\n",
         "* Two replicas\n",
         "* Two samples are processed on each replica\n",
         "* Resulting loss values: [2,  3] and [4,  5] on each replica\n",
@@ -669,6 +671,7 @@
       },
       "source": [
         "### Guide and examples\n",
+        "\n",
         "Here are some examples for using distribution strategy with custom training loops:\n",
         "\n",
         "1. [Distributed training guide](../../guide/distributed_training)\n",
@@ -678,7 +681,7 @@
         "2. [NCF](https://github.com/tensorflow/models/blob/master/official/recommendation/ncf_keras_main.py) example trained using `MirroredStrategy` that can be enabled using the `keras_use_ctl` flag.\n",
         "3. [NMT](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/nmt_with_attention/distributed_train.py) example trained using `MirroredStrategy`.\n",
         "\n",
-        "More examples listed in the [Distribution strategy guide](../../guide/distributed_training.ipynb#examples_and_tutorials)."
+        "You can find more examples listed under _Examples and tutorials_ in the [Distribution strategy guide](../../guide/distributed_training.ipynb)."
       ]
     },
     {
@@ -690,8 +693,8 @@
         "## Next steps\n",
         "\n",
         "*   Try out the new `tf.distribute.Strategy` API on your models.\n",
-        "*   Visit the [Better performance with tf.function](../../guide/function.ipynb) and [TensorFlow Profiler](../../guide/profiler.md) guide to learn more about tools to optimize the performance of your TensorFlow models.\n",
-        "*   The [Distributed training in TensorFlow](../../guide/distributed_training.ipynb) guide provides an overview of the available distribution strategies."
+        "*   Visit the [Better performance with `tf.function`](../../guide/function.ipynb) and [TensorFlow Profiler](../../guide/profiler.md) guides to learn more about tools to optimize the performance of your TensorFlow models.\n",
+        "*   Check out the [Distributed training in TensorFlow](../../guide/distributed_training.ipynb) guide, which provides an overview of the available distribution strategies."
       ]
     }
   ],

From c0d99a4cc8c63c845758ad5cd7585a16e8f4b315 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 15:51:49 +0100
Subject: [PATCH 183/872] Lint Migrate tf.feature_column to KPL guide

---
 .../migrate/migrating_feature_columns.ipynb   | 20 +++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/site/en/guide/migrate/migrating_feature_columns.ipynb b/site/en/guide/migrate/migrating_feature_columns.ipynb
index 44a49280341..aab2ce1d092 100644
--- a/site/en/guide/migrate/migrating_feature_columns.ipynb
+++ b/site/en/guide/migrate/migrating_feature_columns.ipynb
@@ -37,7 +37,7 @@
         "id": "77z2OchJTk0l"
       },
       "source": [
-        "# Migrating feature_columns to TF2's Keras Preprocessing Layers\n",
+        "# Migrate `tf.feature_column`s to Keras preprocessing layers\n",
         "\n",
         "

 
VersionPython versionCompilerBuild toolscuDNNCUDA
tensorflow-2.9.03.7-3.10GCC 7.3.1Bazel 5.0.08.111.2
tensorflow-2.9.03.7-3.10GCC 9.3.1Bazel 5.0.08.111.2

 
tensorflow-2.8.03.7-3.10GCC 7.3.1Bazel 4.2.18.111.2

 
tensorflow-2.7.03.7-3.9GCC 7.3.1Bazel 3.7.28.111.2

 
tensorflow-2.6.03.6-3.9GCC 7.3.1Bazel 3.7.28.111.2
\n",
-        "    Run in Google Colab\n",
+        "    Run in Google Colab\n",
         "  \n",
         "    View source on GitHub\n",
@@ -1077,6 +1077,7 @@
     "colab": {
       "collapsed_sections": [],
       "name": "transfer_learning.ipynb",
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 8a1d674aaa7b8b8f94464ec4ab50ea37b909611c Mon Sep 17 00:00:00 2001
From: Aswin Murali 
Date: Thu, 19 May 2022 21:24:39 +0530
Subject: [PATCH 145/872] Fix typo in music generation

---
 site/en/tutorials/audio/music_generation.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/audio/music_generation.ipynb b/site/en/tutorials/audio/music_generation.ipynb
index 89802d0447b..cd59b6600d8 100644
--- a/site/en/tutorials/audio/music_generation.ipynb
+++ b/site/en/tutorials/audio/music_generation.ipynb
@@ -857,7 +857,7 @@
         "id": "iGQn32q-hdK2"
       },
       "source": [
-        "The model will have three outputs, one for each note variable. For `pitch` and `duration`, you will use a custom loss function based on mean squared error that encourages the model to output non-negative values."
+        "The model will have three outputs, one for each note variable. For `step` and `duration`, you will use a custom loss function based on mean squared error that encourages the model to output non-negative values."
       ]
     },
     {

From c14a48d39bde8b271019b3488ce0069eb73df967 Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Thu, 19 May 2022 16:14:18 +0000
Subject: [PATCH 146/872] nbfmt

---
 site/en/tutorials/audio/music_generation.ipynb | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/site/en/tutorials/audio/music_generation.ipynb b/site/en/tutorials/audio/music_generation.ipynb
index cd59b6600d8..52e0b96a0b9 100644
--- a/site/en/tutorials/audio/music_generation.ipynb
+++ b/site/en/tutorials/audio/music_generation.ipynb
@@ -1230,8 +1230,7 @@
         "In the above plots, you will notice the change in distribution of the note variables.\n",
         "Since there is a feedback loop between the model's outputs and inputs, the model tends to generate similar sequences of outputs to reduce the loss. \n",
         "This is particularly relevant for `step` and `duration`, which has uses MSE loss.\n",
-        "For `pitch`, you can increase the randomness by increasing the `temperature` in `predict_next_note`.\n",
-        "\n"
+        "For `pitch`, you can increase the randomness by increasing the `temperature` in `predict_next_note`.\n"
       ]
     },
     {
@@ -1253,7 +1252,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "music_generation.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 720e555e8cd219876b2deac1d4fbd67b9fc7de61 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Thu, 19 May 2022 10:40:00 -0700
Subject: [PATCH 147/872] Switch tensorflow models docs to use
 `tf-models-official`.

This works now that 2.9 (tensorflow and tf-models) is released.

PiperOrigin-RevId: 449782178
---
 .../classification_with_model_garden.ipynb    | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/site/en/tutorials/images/classification_with_model_garden.ipynb b/site/en/tutorials/images/classification_with_model_garden.ipynb
index 9e23168d04a..0eea176f634 100644
--- a/site/en/tutorials/images/classification_with_model_garden.ipynb
+++ b/site/en/tutorials/images/classification_with_model_garden.ipynb
@@ -1,5 +1,5 @@
 {
-  "cells": [ 
+  "cells": [
     {
       "cell_type": "markdown",
       "metadata": {
@@ -88,7 +88,9 @@
       "source": [
         "## Setup\n",
         "\n",
-        "Install and import the necessary modules. This tutorial uses the `tf-models-nightly` version of Model Garden."
+        "Install and import the necessary modules. This tutorial uses the `tf-models-nightly` version of Model Garden.\n",
+        "\n",
+        "Note: Upgrading TensorFlow to 2.9 in Colab breaks GPU support, so this colab is set to run on CPU until the Colab runtimes are updated."
       ]
     },
     {
@@ -100,7 +102,7 @@
       "outputs": [],
       "source": [
         "!pip uninstall -y opencv-python\n",
-        "!pip install -q tf-models-nightly"
+        "!pip install -U -q \"tensorflow>=2.9.0\" \"tf-models-official\""
       ]
     },
     {
@@ -149,8 +151,9 @@
       "source": [
         "import tensorflow_models as tfm\n",
         "\n",
-        "# Not in the tfm public API for v2.9. Will be available as `vision.serving` in v2.10\n",
-        "from official.vision.serving import export_saved_model_lib"
+        "# These are not in the tfm public API for v2.9. They will be available in v2.10\n",
+        "from official.vision.serving import export_saved_model_lib\n",
+        "import official.core.train_lib"
       ]
     },
     {
@@ -668,16 +671,14 @@
         "  show_batch(data['image'], data['label'], predictions)\n",
         "\n",
         "  if device=='CPU':\n",
-        "    plt.suptitle('The model was only trained for a few steps, it is not expected to do well.')"
+        "    plt.suptitle('The model was only trained for a few steps, it is not expected to do better than random.')"
       ]
     }
   ],
   "metadata": {
-    "accelerator": "GPU",
     "colab": {
       "collapsed_sections": [],
-      "name": "models_vision.ipynb",
-      "provenance": [],
+      "name": "classification_with_model_garden.ipynb",
       "toc_visible": true
     },
     "kernelspec": {

From 328016a8506402365ee38472b1d179549243779c Mon Sep 17 00:00:00 2001
From: gadagashwini <99852755+gadagashwini@users.noreply.github.com>
Date: Fri, 20 May 2022 09:58:03 +0530
Subject: [PATCH 148/872] Fix invalid links

Install TensorFlow for C linking to wrong links.
---
 site/en/install/lang_c.ipynb | 17 +----------------
 1 file changed, 1 insertion(+), 16 deletions(-)

diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb
index 7d431c74126..31c5205ea3a 100644
--- a/site/en/install/lang_c.ipynb
+++ b/site/en/install/lang_c.ipynb
@@ -45,22 +45,7 @@
       "metadata": {
         "id": "Birwb-khUOIq"
       },
-      "source": [
-        "\n",
-        "  \n",
-        "  \n",
-        "  \n",
-        "  \n",
-        "
\n",
-        "    View on TensorFlow.org\n",
-        "  \n",
-        "    Run in Google Colab\n",
-        "  \n",
-        "    View source on GitHub\n",
-        "  \n",
-        "    Download notebook\n",
-        "  
"
-      ]
+      
     },
     {
       "cell_type": "markdown",

From 1054aa3c4bc8379799fe84b32f5b8ef31ddfa61f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Johannes=20Ball=C3=A9?= 
Date: Fri, 20 May 2022 12:09:53 -0700
Subject: [PATCH 149/872] Adds lossy data compression tutorial using TFC.

PiperOrigin-RevId: 450036225
---
 site/en/tutorials/_toc.yaml                   |   3 +
 .../generative/data_compression.ipynb         | 844 ++++++++++++++++++
 2 files changed, 847 insertions(+)
 create mode 100644 site/en/tutorials/generative/data_compression.ipynb

diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml
index ee408fff0a5..36ccea93cbf 100644
--- a/site/en/tutorials/_toc.yaml
+++ b/site/en/tutorials/_toc.yaml
@@ -185,6 +185,9 @@ toc:
     path: /tutorials/generative/autoencoder
   - title: "Variational Autoencoder"
     path: /tutorials/generative/cvae
+  - title: "Lossy data compression"
+    path: /tutorials/generative/data_compression
+    status: new
 
 - title: "Model Understanding"
   style: accordion
diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
new file mode 100644
index 00000000000..bd467381e26
--- /dev/null
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -0,0 +1,844 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Tce3stUlHN0L"
+      },
+      "source": [
+        "##### Copyright 2022 The TensorFlow Compression Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "tuOe1ymfHZPu"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qFdPvlXBOdUN"
+      },
+      "source": [
+        "# Learned data compression"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MfBg1C5NB3X0"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    \n",
+        "    \n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    \n",
+        "    \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    \n",
+        "    \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "xHxb-dlhMIzW"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This notebook shows how to do lossy data compression using neural networks and [TensorFlow Compression](https://github.com/tensorflow/compression).\n",
+        "\n",
+        "Lossy compression involves making a trade-off between **rate**, the expected number of bits needed to encode a sample, and **distortion**, the expected error in the reconstruction of the sample.\n",
+        "\n",
+        "The examples below use an autoencoder-like model to compress images from the MNIST dataset. The method is based on the paper [End-to-end Optimized Image Compression](https://arxiv.org/abs/1611.01704).\n",
+        "\n",
+        "More background on learned data compression can be found in [this paper](https://arxiv.org/abs/2007.03034) targeted at people familiar with classical data compression, or [this survey](https://arxiv.org/abs/2202.06533) targeted at a machine learning audience.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MUXex9ctTuDB"
+      },
+      "source": [
+        "## Setup\n",
+        "\n",
+        "Install Tensorflow Compression via `pip`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K489KsEgxuLI"
+      },
+      "outputs": [],
+      "source": [
+        "# Installs the latest version of TFC compatible with the installed TF version.\n",
+        "!pip install tensorflow-compression~=$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d\\.\\d).*/\\1.0/sg')\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WfVAmHCVxpTS"
+      },
+      "source": [
+        "Import library dependencies."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IqR2PQG4ZaZ0"
+      },
+      "outputs": [],
+      "source": [
+        "import matplotlib.pyplot as plt\n",
+        "import tensorflow as tf\n",
+        "import tensorflow_compression as tfc\n",
+        "import tensorflow_datasets as tfds\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "wsncKT2iymgQ"
+      },
+      "source": [
+        "## Define the trainer model.\n",
+        "\n",
+        "Because the model resembles an autoencoder, and we need to perform a different set of functions during training and inference, the setup is a little different from, say, a classifier.\n",
+        "\n",
+        "The training model consists of three parts:\n",
+        "- the **analysis** (or encoder) transform, converting from the image into a latent space,\n",
+        "- the **synthesis** (or decoder) transform, converting from the latent space back into image space, and\n",
+        "- a **prior** and entropy model, modeling the marginal probabilities of the latents.\n",
+        "\n",
+        "First, define the transforms:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8yZESLgW-vp1"
+      },
+      "outputs": [],
+      "source": [
+        "def make_analysis_transform(latent_dims):\n",
+        "  \"\"\"Creates the analysis (encoder) transform.\"\"\"\n",
+        "  return tf.keras.Sequential([\n",
+        "      tf.keras.layers.Conv2D(\n",
+        "          20, 5, use_bias=True, strides=2, padding=\"same\",\n",
+        "          activation=\"leaky_relu\", name=\"conv_1\"),\n",
+        "      tf.keras.layers.Conv2D(\n",
+        "          50, 5, use_bias=True, strides=2, padding=\"same\",\n",
+        "          activation=\"leaky_relu\", name=\"conv_2\"),\n",
+        "      tf.keras.layers.Flatten(),\n",
+        "      tf.keras.layers.Dense(\n",
+        "          500, use_bias=True, activation=\"leaky_relu\", name=\"fc_1\"),\n",
+        "      tf.keras.layers.Dense(\n",
+        "          latent_dims, use_bias=True, activation=None, name=\"fc_2\"),\n",
+        "  ], name=\"analysis_transform\")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2sHdYBzF2xcu"
+      },
+      "outputs": [],
+      "source": [
+        "def make_synthesis_transform():\n",
+        "  \"\"\"Creates the synthesis (decoder) transform.\"\"\"\n",
+        "  return tf.keras.Sequential([\n",
+        "      tf.keras.layers.Dense(\n",
+        "          500, use_bias=True, activation=\"leaky_relu\", name=\"fc_1\"),\n",
+        "      tf.keras.layers.Dense(\n",
+        "          2450, use_bias=True, activation=\"leaky_relu\", name=\"fc_2\"),\n",
+        "      tf.keras.layers.Reshape((7, 7, 50)),\n",
+        "      tf.keras.layers.Conv2DTranspose(\n",
+        "          20, 5, use_bias=True, strides=2, padding=\"same\",\n",
+        "          activation=\"leaky_relu\", name=\"conv_1\"),\n",
+        "      tf.keras.layers.Conv2DTranspose(\n",
+        "          1, 5, use_bias=True, strides=2, padding=\"same\",\n",
+        "          activation=\"leaky_relu\", name=\"conv_2\"),\n",
+        "  ], name=\"synthesis_transform\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "lYC8tHhkxTlK"
+      },
+      "source": [
+        "The trainer holds an instance of both transforms, as well as the parameters of the prior.\n",
+        "\n",
+        "Its `call` method is set up to compute:\n",
+        "- **rate**, an estimate of the number of bits needed to represent the batch of digits, and\n",
+        "- **distortion**, the mean absolute difference between the pixels of the original digits and their reconstructions.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ROn2DbzsBirI"
+      },
+      "outputs": [],
+      "source": [
+        "class MNISTCompressionTrainer(tf.keras.Model):\n",
+        "  \"\"\"Model that trains a compressor/decompressor for MNIST.\"\"\"\n",
+        "\n",
+        "  def __init__(self, latent_dims):\n",
+        "    super().__init__()\n",
+        "    self.analysis_transform = make_analysis_transform(latent_dims)\n",
+        "    self.synthesis_transform = make_synthesis_transform()\n",
+        "    self.prior_log_scales = tf.Variable(tf.zeros((latent_dims,)))\n",
+        "\n",
+        "  @property\n",
+        "  def prior(self):\n",
+        "    return tfc.NoisyLogistic(loc=0., scale=tf.exp(self.prior_log_scales))\n",
+        "\n",
+        "  def call(self, x, training):\n",
+        "    \"\"\"Computes rate and distortion losses.\"\"\"\n",
+        "    # Ensure inputs are floats in the range (0, 1).\n",
+        "    x = tf.cast(x, self.compute_dtype) / 255.\n",
+        "    x = tf.reshape(x, (-1, 28, 28, 1))\n",
+        "\n",
+        "    # Compute latent space representation y, perturb it and model its entropy,\n",
+        "    # then compute the reconstructed pixel-level representation x_hat.\n",
+        "    y = self.analysis_transform(x)\n",
+        "    entropy_model = tfc.ContinuousBatchedEntropyModel(\n",
+        "        self.prior, coding_rank=1, compression=False)\n",
+        "    y_tilde, rate = entropy_model(y, training=training)\n",
+        "    x_tilde = self.synthesis_transform(y_tilde)\n",
+        "\n",
+        "    # Average number of bits per MNIST digit.\n",
+        "    rate = tf.reduce_mean(rate)\n",
+        "\n",
+        "    # Mean absolute difference across pixels.\n",
+        "    distortion = tf.reduce_mean(abs(x - x_tilde))\n",
+        "\n",
+        "    return dict(rate=rate, distortion=distortion)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vEXbp9RV3kRX"
+      },
+      "source": [
+        "### Compute rate and distortion.\n",
+        "\n",
+        "Let's walk through this step by step, using one image from the training set. Load the MNIST dataset for training and validation:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7FV99WTrIBen"
+      },
+      "outputs": [],
+      "source": [
+        "training_dataset, validation_dataset = tfds.load(\n",
+        "    \"mnist\",\n",
+        "    split=[\"train\", \"test\"],\n",
+        "    shuffle_files=True,\n",
+        "    as_supervised=True,\n",
+        "    with_info=False,\n",
+        ")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SwKgNTg_QfjH"
+      },
+      "source": [
+        "And extract one image $x$:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "O-BSdeHcPBBf"
+      },
+      "outputs": [],
+      "source": [
+        "(x, _), = validation_dataset.take(1)\n",
+        "\n",
+        "plt.imshow(x)\n",
+        "print(f\"Data type: {x.dtype}\")\n",
+        "print(f\"Shape: {x.shape}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "V8IvuFkrRJIa"
+      },
+      "source": [
+        "To get the latent representation $y$, we need to cast it to `float32`, add a batch dimension, and pass it throught the analysis transform."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jA0DOWq23lEq"
+      },
+      "outputs": [],
+      "source": [
+        "x = tf.cast(x, tf.float32) / 255.\n",
+        "x = tf.reshape(x, (-1, 28, 28, 1))\n",
+        "y = make_analysis_transform(10)(x)\n",
+        "\n",
+        "print(\"y:\", y)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rTojJQvZT8SX"
+      },
+      "source": [
+        "The latents will be quantized at test time. To model this in a differentiable way during training, we add uniform noise in the interval $(-.5, .5)$ and call the result $\\tilde y$. This is the same terminology as used in the paper [End-to-end Optimized Image Compression](https://arxiv.org/abs/1611.01704)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Spr3503OUOFQ"
+      },
+      "outputs": [],
+      "source": [
+        "y_tilde = y + tf.random.uniform(y.shape, -.5, .5)\n",
+        "\n",
+        "print(\"y_tilde:\", y_tilde)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7hRN89R7SA3U"
+      },
+      "source": [
+        "The \"prior\" is a probability density that we train to model the marginal distribution of the noisy latents. For example, it could be a set of independent [logistic distributions](https://en.wikipedia.org/wiki/Logistic_distribution) with different scales for each latent dimension. `tfc.NoisyLogistic` accounts for the fact that the latents have additive noise. As the scale approaches zero, a logistic distribution approaches a dirac delta (spike), but the added noise causes the \"noisy\" distribution to approach the uniform distribution instead."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2tmA1Bw7ReMY"
+      },
+      "outputs": [],
+      "source": [
+        "prior = tfc.NoisyLogistic(loc=0., scale=tf.linspace(.01, 2., 10))\n",
+        "\n",
+        "_ = tf.linspace(-6., 6., 501)[:, None]\n",
+        "plt.plot(_, prior.prob(_));\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2NSWtBZmUvVY"
+      },
+      "source": [
+        "During training, `tfc.ContinuousBatchedEntropyModel` adds uniform noise, and uses the noise and the prior to compute a (differentiable) upper bound on the rate (the average number of bits necessary to encode the latent representation). That bound can be minimized as a loss."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hFuGlyJuThBC"
+      },
+      "outputs": [],
+      "source": [
+        "entropy_model = tfc.ContinuousBatchedEntropyModel(\n",
+        "    prior, coding_rank=1, compression=False)\n",
+        "y_tilde, rate = entropy_model(y, training=True)\n",
+        "\n",
+        "print(\"rate:\", rate)\n",
+        "print(\"y_tilde:\", y_tilde)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Cyr8DGgmWd32"
+      },
+      "source": [
+        "Lastly, the noisy latents are passed back through the synthesis transform to produce an image reconstruction $\\tilde x$. Distortion is the error between original image and reconstruction. Obviously, with the transforms untrained, the reconstruction is not very useful."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gtmI0xGEVym0"
+      },
+      "outputs": [],
+      "source": [
+        "x_tilde = make_synthesis_transform()(y_tilde)\n",
+        "\n",
+        "# Mean absolute difference across pixels.\n",
+        "distortion = tf.reduce_mean(abs(x - x_tilde))\n",
+        "print(\"distortion:\", distortion)\n",
+        "\n",
+        "x_tilde = tf.saturate_cast(x_tilde[0] * 255, tf.uint8)\n",
+        "plt.imshow(x_tilde)\n",
+        "print(f\"Data type: {x_tilde.dtype}\")\n",
+        "print(f\"Shape: {x_tilde.shape}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "UVz3I7E8ecij"
+      },
+      "source": [
+        "For every batch of digits, calling the `MNISTCompressionTrainer` produces the rate and distortion as an average over that batch:\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ICJnjj1LeB8L"
+      },
+      "outputs": [],
+      "source": [
+        "(example_batch, _), = validation_dataset.batch(32).take(1)\n",
+        "trainer = MNISTCompressionTrainer(10)\n",
+        "example_output = trainer(example_batch)\n",
+        "\n",
+        "print(\"rate: \", example_output[\"rate\"])\n",
+        "print(\"distortion: \", example_output[\"distortion\"])\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "lgdfRtmee5Mn"
+      },
+      "source": [
+        "In the next section, we set up the model to do gradient descent on these two losses."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fKGVwv5MAq6w"
+      },
+      "source": [
+        "## Train the model.\n",
+        "\n",
+        "We compile the trainer in a way that it optimizes the rate–distortion Lagrangian, that is, a sum of rate and distortion, where one of the terms is weighted by Lagrange parameter $\\lambda$.\n",
+        "\n",
+        "This loss function affects the different parts of the model differently:\n",
+        "- The analysis transform is trained to produce a latent representation that achieves the desired trade-off between rate and distortion.\n",
+        "- The synthesis transform is trained to minimize distortion, given the latent representation.\n",
+        "- The parameters of the prior are trained to minimize the rate given the latent representation. This is identical to fitting the prior to the marginal distribution of latents in a maximum likelihood sense."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "k5mm1aDkcgAf"
+      },
+      "outputs": [],
+      "source": [
+        "def pass_through_loss(_, x):\n",
+        "  # Since rate and distortion are unsupervised, the loss doesn't need a target.\n",
+        "  return x\n",
+        "\n",
+        "def make_mnist_compression_trainer(lmbda, latent_dims=50):\n",
+        "  trainer = MNISTCompressionTrainer(latent_dims)\n",
+        "  trainer.compile(\n",
+        "    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),\n",
+        "    # Just pass through rate and distortion as losses/metrics.\n",
+        "    loss=dict(rate=pass_through_loss, distortion=pass_through_loss),\n",
+        "    metrics=dict(rate=pass_through_loss, distortion=pass_through_loss),\n",
+        "    loss_weights=dict(rate=1., distortion=lmbda),\n",
+        "  )\n",
+        "  return trainer\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DPwd4DTs3Mfr"
+      },
+      "source": [
+        "Next, train the model. The human annotations are not necessary here, since we just want to compress the images, so we drop them using a `map` and instead add \"dummy\" targets for rate and distortion."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QNBpCTgzAV7M"
+      },
+      "outputs": [],
+      "source": [
+        "def add_rd_targets(image, label):\n",
+        "  # Training is unsupervised, so labels aren't necessary here. However, we\n",
+        "  # need to add \"dummy\" targets for rate and distortion.\n",
+        "  return image, dict(rate=0., distortion=0.)\n",
+        "\n",
+        "def train_mnist_model(lmbda):\n",
+        "  trainer = make_mnist_compression_trainer(lmbda)\n",
+        "  trainer.fit(\n",
+        "      training_dataset.map(add_rd_targets).batch(128).prefetch(8),\n",
+        "      epochs=15,\n",
+        "      validation_data=validation_dataset.map(add_rd_targets).batch(128).cache(),\n",
+        "      validation_freq=1,\n",
+        "      verbose=1,\n",
+        "  )\n",
+        "  return trainer\n",
+        "\n",
+        "trainer = train_mnist_model(lmbda=2000)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Td4xuttmCd7T"
+      },
+      "source": [
+        "## Compress some MNIST images.\n",
+        "\n",
+        "For compression and decompression at test time, we split the trained model in two parts:\n",
+        "\n",
+        "- The encoder side consists of the analysis transform and the entropy model.\n",
+        "- The decoder side consists of the synthesis transform and the same entropy model.\n",
+        "\n",
+        "At test time, the latents will not have additive noise, but they will be quantized and then losslessly compressed, so we give them new names. We call them and the image reconstruction $\\hat x$ and $\\hat y$, respectively (following [End-to-end Optimized Image Compression](https://arxiv.org/abs/1611.01704))."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sBRAPa5jksss"
+      },
+      "outputs": [],
+      "source": [
+        "class MNISTCompressor(tf.keras.Model):\n",
+        "  \"\"\"Compresses MNIST images to strings.\"\"\"\n",
+        "\n",
+        "  def __init__(self, analysis_transform, entropy_model):\n",
+        "    super().__init__()\n",
+        "    self.analysis_transform = analysis_transform\n",
+        "    self.entropy_model = entropy_model\n",
+        "\n",
+        "  def call(self, x):\n",
+        "    # Ensure inputs are floats in the range (0, 1).\n",
+        "    x = tf.cast(x, self.compute_dtype) / 255.\n",
+        "    y = self.analysis_transform(x)\n",
+        "    # Also return the exact information content of each digit.\n",
+        "    _, bits = self.entropy_model(y, training=False)\n",
+        "    return self.entropy_model.compress(y), bits\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sSZ0X2xPnkN-"
+      },
+      "outputs": [],
+      "source": [
+        "class MNISTDecompressor(tf.keras.Model):\n",
+        "  \"\"\"Decompresses MNIST images from strings.\"\"\"\n",
+        "\n",
+        "  def __init__(self, entropy_model, synthesis_transform):\n",
+        "    super().__init__()\n",
+        "    self.entropy_model = entropy_model\n",
+        "    self.synthesis_transform = synthesis_transform\n",
+        "\n",
+        "  def call(self, string):\n",
+        "    y_hat = self.entropy_model.decompress(string, ())\n",
+        "    x_hat = self.synthesis_transform(y_hat)\n",
+        "    # Scale and cast back to 8-bit integer.\n",
+        "    return tf.saturate_cast(tf.round(x_hat * 255.), tf.uint8)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "GI7rxeOUDnaC"
+      },
+      "source": [
+        "When instantiated with `compression=True`, the entropy model converts the learned prior into tables for a range coding algorithm. When calling `compress()`, this algorithm is invoked to convert the latent space vector into bit sequences. The length of each binary string approximates the information content of the latent (the negative log likelihood of the latent under the prior).\n",
+        "\n",
+        "The entropy model for compression and decompression must be the same instance, because the range coding tables need to be exactly identical on both sides. Otherwise, decoding errors can occur."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Dnm_p7mbnigo"
+      },
+      "outputs": [],
+      "source": [
+        "def make_mnist_codec(trainer, **kwargs):\n",
+        "  # The entropy model must be created with `compression=True` and the same\n",
+        "  # instance must be shared between compressor and decompressor.\n",
+        "  entropy_model = tfc.ContinuousBatchedEntropyModel(\n",
+        "      trainer.prior, coding_rank=1, compression=True, **kwargs)\n",
+        "  compressor = MNISTCompressor(trainer.analysis_transform, entropy_model)\n",
+        "  decompressor = MNISTDecompressor(entropy_model, trainer.synthesis_transform)\n",
+        "  return compressor, decompressor\n",
+        "\n",
+        "compressor, decompressor = make_mnist_codec(trainer)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SYu5sVVH3YMv"
+      },
+      "source": [
+        "Grab 16 images from the validation dataset. You can select a different subset by changing the argument to `skip`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qAxArlU728K5"
+      },
+      "outputs": [],
+      "source": [
+        "(originals, _), = validation_dataset.batch(16).skip(3).take(1)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CHeN_ny929YS"
+      },
+      "source": [
+        "Compress them to strings, and keep track of each of their information content in bits."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "smOk42gQ3IXv"
+      },
+      "outputs": [],
+      "source": [
+        "strings, entropies = compressor(originals)\n",
+        "\n",
+        "print(f\"String representation of first digit in hexadecimal: 0x{strings[0].numpy().hex()}\")\n",
+        "print(f\"Number of bits actually needed to represent it: {entropies[0]:0.2f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5j9R4bTT3Qhl"
+      },
+      "source": [
+        "Decompress the images back from the strings."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "yOP6pEqU3P0w"
+      },
+      "outputs": [],
+      "source": [
+        "reconstructions = decompressor(strings)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JWo0Q-vy23tt"
+      },
+      "source": [
+        "Display each of 16 original digits together with its compressed binary representation, and the reconstructed digit."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "jU5IqzZzeEpf"
+      },
+      "outputs": [],
+      "source": [
+        "#@title\n",
+        "\n",
+        "def display_digits(originals, strings, entropies, reconstructions):\n",
+        "  \"\"\"Visualizes 16 digits together with their reconstructions.\"\"\"\n",
+        "  fig, axes = plt.subplots(4, 4, sharex=True, sharey=True, figsize=(12.5, 5))\n",
+        "  axes = axes.ravel()\n",
+        "  for i in range(len(axes)):\n",
+        "    image = tf.concat([\n",
+        "        tf.squeeze(originals[i]),\n",
+        "        tf.zeros((28, 14), tf.uint8),\n",
+        "        tf.squeeze(reconstructions[i]),\n",
+        "    ], 1)\n",
+        "    axes[i].imshow(image)\n",
+        "    axes[i].text(\n",
+        "        .5, .5, f\"→ 0x{strings[i].numpy().hex()} →\\n{entropies[i]:0.2f} bits\",\n",
+        "        ha=\"center\", va=\"top\", color=\"white\", fontsize=\"small\",\n",
+        "        transform=axes[i].transAxes)\n",
+        "    axes[i].axis(\"off\")\n",
+        "  plt.subplots_adjust(wspace=0, hspace=0, left=0, right=1, bottom=0, top=1)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "km9PqVEtPJPc"
+      },
+      "outputs": [],
+      "source": [
+        "display_digits(originals, strings, entropies, reconstructions)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EzlrIOiYOzJc"
+      },
+      "source": [
+        "Note that the length of the encoded string differs from the information content of each digit.\n",
+        "\n",
+        "This is because the range coding process works with discretized probabilities, and has a small amount of overhead. So, especially for short strings, the correspondence is only approximate. However, range coding is **asymptotically optimal**: in the limit, the expected bit count will approach the cross entropy (the expected information content), for which the rate term in the training model is an upper bound."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "78qIG8t8FvJW"
+      },
+      "source": [
+        "## The rate–distortion trade-off\n",
+        "\n",
+        "Above, the model was trained for a specific trade-off (given by `lmbda=2000`) between the average number of bits used to represent each digit and the incurred error in the reconstruction.\n",
+        "\n",
+        "What happens when we repeat the experiment with different values?\n",
+        "\n",
+        "Let's start by reducing $\\lambda$ to 500."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1iFcAD0WF78p"
+      },
+      "outputs": [],
+      "source": [
+        "def train_and_visualize_model(lmbda):\n",
+        "  trainer = train_mnist_model(lmbda=lmbda)\n",
+        "  compressor, decompressor = make_mnist_codec(trainer)\n",
+        "  strings, entropies = compressor(originals)\n",
+        "  reconstructions = decompressor(strings)\n",
+        "  display_digits(originals, strings, entropies, reconstructions)\n",
+        "\n",
+        "train_and_visualize_model(lmbda=500)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Uy5OkgJMObMc"
+      },
+      "source": [
+        "The bit rate of our code goes down, as does the fidelity of the digits. However, most of the digits remain recognizable.\n",
+        "\n",
+        "Let's reduce $\\lambda$ further."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NQp9_9_5GcxH"
+      },
+      "outputs": [],
+      "source": [
+        "train_and_visualize_model(lmbda=300)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3ELLMAN1OwMQ"
+      },
+      "source": [
+        "The strings begin to get much shorter now, on the order of one byte per digit. However, this comes at a cost. More digits are becoming unrecognizable.\n",
+        "\n",
+        "This demonstrates that this model is agnostic to human perceptions of error, it just measures the absolute deviation in terms of pixel values. To achieve a better perceived image quality, we would need to replace the pixel loss with a perceptual loss."
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "collapsed_sections": [],
+      "name": "data_compression.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From 80c8fd4a84803d3b1f7bc064c52a2e772fa86faa Mon Sep 17 00:00:00 2001
From: gowthamkpr <47574994+gowthamkpr@users.noreply.github.com>
Date: Mon, 23 May 2022 09:41:57 -0700
Subject: [PATCH 150/872] Fixing Broken Link

Fixed broken link with correct link
---
 site/en/guide/function.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index e39c65234b1..a821ae9557b 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -531,7 +531,7 @@
       "source": [
         "#### Use the tracing protocol\n",
         "\n",
-        "Where possible, you should prefer converting the Python type into a [`tf.experimental.ExtensionType`](www.tensorflow.org/api_docs/python/tf/experimental/ExtensionType) instead. Moreover, the `TraceType` of an `ExtensionType` is the `tf.TypeSpec` associated with it. Therefore, if needed, you can simply [override](https://www.tensorflow.org/guide/extension_type#customizing_the_extensiontypes_typespec) the default `tf.TypeSpec` to take control of an `ExtensionType`'s `Tracing Protocol`.\n",
+        "Where possible, you should prefer converting the Python type into a [`tf.experimental.ExtensionType`](https://www.tensorflow.org/api_docs/python/tf/experimental/ExtensionType) instead. Moreover, the `TraceType` of an `ExtensionType` is the `tf.TypeSpec` associated with it. Therefore, if needed, you can simply [override](https://www.tensorflow.org/guide/extension_type#customizing_the_extensiontypes_typespec) the default `tf.TypeSpec` to take control of an `ExtensionType`'s `Tracing Protocol`.\n",
         "\n",
         "Otherwise, for direct control over when `Function` should retrace in regards to a particular Python type, you can implement the `Tracing Protocol` for it yourself."
       ]

From 87c99d097be912d854f73fd64f58c4fdd5369e40 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Tue, 24 May 2022 00:06:02 +0100
Subject: [PATCH 151/872] Update links to ExtensionType and Extension types
 guide

---
 site/en/guide/function.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index a821ae9557b..20514875cf4 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -531,7 +531,7 @@
       "source": [
         "#### Use the tracing protocol\n",
         "\n",
-        "Where possible, you should prefer converting the Python type into a [`tf.experimental.ExtensionType`](https://www.tensorflow.org/api_docs/python/tf/experimental/ExtensionType) instead. Moreover, the `TraceType` of an `ExtensionType` is the `tf.TypeSpec` associated with it. Therefore, if needed, you can simply [override](https://www.tensorflow.org/guide/extension_type#customizing_the_extensiontypes_typespec) the default `tf.TypeSpec` to take control of an `ExtensionType`'s `Tracing Protocol`.\n",
+        "Where possible, you should prefer converting the Python type into a `tf.experimental.ExtensionType` instead. Moreover, the `TraceType` of an `ExtensionType` is the `tf.TypeSpec` associated with it. Therefore, if needed, you can simply override the default `tf.TypeSpec` to take control of an `ExtensionType`'s `Tracing Protocol`. Refer to the _Customizing the ExtensionType's TypeSpec_ section in the [Extension types](extension_type.ipynb) guide for details.\n",
         "\n",
         "Otherwise, for direct control over when `Function` should retrace in regards to a particular Python type, you can implement the `Tracing Protocol` for it yourself."
       ]

From 8d7775038aa7714f0ce7c70844c7f1587e2535a3 Mon Sep 17 00:00:00 2001
From: Anas Neumann 
Date: Wed, 25 May 2022 09:41:48 +0100
Subject: [PATCH 152/872] Update text_classification.ipynb

This line was a mistake and probably comes from this page: https://www.tensorflow.org/text/guide/word_embeddings (the model is almost the same except for the Dense(16, activation='relu') layer).
---
 site/en/tutorials/keras/text_classification.ipynb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/site/en/tutorials/keras/text_classification.ipynb b/site/en/tutorials/keras/text_classification.ipynb
index 3dabeeff095..1d39a064b13 100644
--- a/site/en/tutorials/keras/text_classification.ipynb
+++ b/site/en/tutorials/keras/text_classification.ipynb
@@ -650,8 +650,7 @@
         "\n",
         "1. The first layer is an `Embedding` layer. This layer takes the integer-encoded reviews and looks up an embedding vector for each word-index. These vectors are learned as the model trains. The vectors add a dimension to the output array. The resulting dimensions are: `(batch, sequence, embedding)`.  To learn more about embeddings, check out the [Word embeddings](https://www.tensorflow.org/text/guide/word_embeddings) tutorial.\n",
         "2. Next, a `GlobalAveragePooling1D` layer returns a fixed-length output vector for each example by averaging over the sequence dimension. This allows the model to handle input of variable length, in the simplest way possible.\n",
-        "3. This fixed-length output vector is piped through a fully-connected (`Dense`) layer with 16 hidden units. \n",
-        "4. The last layer is densely connected with a single output node."
+        "3. The last layer is densely connected with a single output node."
       ]
     },
     {

From 208bea22f40e3e179e4f125e3beb8be3144895f5 Mon Sep 17 00:00:00 2001
From: Mihai Maruseac 
Date: Wed, 25 May 2022 19:06:11 -0700
Subject: [PATCH 153/872] Update install from source page.

Currently we need `requests`, it was brought up before by a transitive dep but this is no longer the case.

After cl/450786506 surfaces again, we also need `opt_einsum`

PiperOrigin-RevId: 451066739
---
 site/en/install/source.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/install/source.md b/site/en/install/source.md
index 9c4d518aeb5..0ba98b99ac5 100644
--- a/site/en/install/source.md
+++ b/site/en/install/source.md
@@ -37,7 +37,7 @@ Install the TensorFlow *pip* package dependencies (if using a virtual
 environment, omit the `--user` argument):
 
 
-pip install -U --user pip numpy wheel packaging
+pip install -U --user pip numpy wheel packaging requests opt_einsum
 pip install -U --user keras_preprocessing --no-deps
 

 

From 016db152699ea08bafa87ddbbdaaebd2f60c0611 Mon Sep 17 00:00:00 2001
From: Joe Fernandez 
Date: Thu, 26 May 2022 11:04:59 -0700
Subject: [PATCH 154/872] Link fixes for /lite/models/convert move

PiperOrigin-RevId: 451201062
---
 site/en/tutorials/images/transfer_learning_with_hub.ipynb | 2 +-
 site/en/tutorials/keras/save_and_load.ipynb               | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/images/transfer_learning_with_hub.ipynb b/site/en/tutorials/images/transfer_learning_with_hub.ipynb
index 2b04297da5e..91e3b17e28d 100644
--- a/site/en/tutorials/images/transfer_learning_with_hub.ipynb
+++ b/site/en/tutorials/images/transfer_learning_with_hub.ipynb
@@ -906,7 +906,7 @@
       "source": [
         "## Next steps\n",
         "\n",
-        "You can use the SavedModel to load for inference or convert it to a [TensorFlow Lite](https://www.tensorflow.org/lite/convert/)  model (for on-device machine learning) or a [TensorFlow.js](https://www.tensorflow.org/js/tutorials#convert_pretrained_models_to_tensorflowjs) model (for machine learning in JavaScript).\n",
+        "You can use the SavedModel to load for inference or convert it to a [TensorFlow Lite](https://www.tensorflow.org/lite/models/convert/)  model (for on-device machine learning) or a [TensorFlow.js](https://www.tensorflow.org/js/tutorials#convert_pretrained_models_to_tensorflowjs) model (for machine learning in JavaScript).\n",
         "\n",
         "Discover [more tutorials](https://www.tensorflow.org/hub/tutorials) to learn how to use pre-trained models from TensorFlow Hub on image, text, audio, and video tasks."
       ]
diff --git a/site/en/tutorials/keras/save_and_load.ipynb b/site/en/tutorials/keras/save_and_load.ipynb
index 4d7ea074161..8ac0e2f0512 100644
--- a/site/en/tutorials/keras/save_and_load.ipynb
+++ b/site/en/tutorials/keras/save_and_load.ipynb
@@ -537,7 +537,7 @@
         "\n",
         "An entire model can be saved in two different file formats (`SavedModel` and `HDF5`). The TensorFlow `SavedModel` format is the default file format in TF2.x. However, models can be saved in `HDF5` format. More details on saving entire models in the two file formats is described below.\n",
         "\n",
-        "Saving a fully-functional model is very useful—you can load them in TensorFlow.js ([Saved Model](https://www.tensorflow.org/js/tutorials/conversion/import_saved_model), [HDF5](https://www.tensorflow.org/js/tutorials/conversion/import_keras)) and then train and run them in web browsers, or convert them to run on mobile devices using TensorFlow Lite ([Saved Model](https://www.tensorflow.org/lite/convert/python_api#converting_a_savedmodel_), [HDF5](https://www.tensorflow.org/lite/convert/python_api#converting_a_keras_model_))\n",
+        "Saving a fully-functional model is very useful—you can load them in TensorFlow.js ([Saved Model](https://www.tensorflow.org/js/tutorials/conversion/import_saved_model), [HDF5](https://www.tensorflow.org/js/tutorials/conversion/import_keras)) and then train and run them in web browsers, or convert them to run on mobile devices using TensorFlow Lite ([Saved Model](https://www.tensorflow.org/lite/models/convert/#convert_a_savedmodel_recommended_), [HDF5](https://www.tensorflow.org/lite/models/convert/#convert_a_keras_model_))\n",
         "\n",
         "\\*Custom objects (e.g. subclassed models or layers) require special attention when saving and loading. See the **Saving custom objects** section below "
       ]

From f57dc21a4d253028c2be8360f7f0eaf6e682df42 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Tue, 31 May 2022 13:17:24 +0100
Subject: [PATCH 155/872] Fix notebook button links in Install TensorFlow for
 C.ipynb

---
 site/en/install/lang_c.ipynb | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb
index 31c5205ea3a..da4bbf03fc6 100644
--- a/site/en/install/lang_c.ipynb
+++ b/site/en/install/lang_c.ipynb
@@ -45,7 +45,22 @@
       "metadata": {
         "id": "Birwb-khUOIq"
       },
-      
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
     },
     {
       "cell_type": "markdown",

From 047c8904be7689ee42b63ee2d32f2ddf9d9c4ba9 Mon Sep 17 00:00:00 2001
From: Mihai Maruseac 
Date: Tue, 31 May 2022 09:37:24 -0700
Subject: [PATCH 156/872] Remove myself from CODEOWNERS

PiperOrigin-RevId: 452068564
---
 CODEOWNERS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CODEOWNERS b/CODEOWNERS
index db670ef4093..94c3b5d8d2e 100644
--- a/CODEOWNERS
+++ b/CODEOWNERS
@@ -5,7 +5,7 @@
 *  @MarkDaoust @8bitmp3
 
 # Install
-/site/en/install/ @mihaimaruseac @haifeng-jin @MarkDaoust @8bitmp3
+/site/en/install/ @haifeng-jin @MarkDaoust @8bitmp3
 
 # Community
 /site/en/community/ @ewilderj @theadactyl @joanafilipa

From 0655301d85a8d714d266a087d967feee79221389 Mon Sep 17 00:00:00 2001
From: Sean Morgan 
Date: Tue, 31 May 2022 14:02:11 -0700
Subject: [PATCH 157/872] Pin compatible protobuf version

---
 setup.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/setup.py b/setup.py
index 8c24bee6204..3e44fb34e16 100644
--- a/setup.py
+++ b/setup.py
@@ -29,7 +29,14 @@
     'astor',
     'absl-py',
     'jinja2',
-    'protobuf>=3.14',
+    # TODO(b/182876485): Protobuf 3.20 results in linker errors on Windows
+    # Protobuf 4.0 is binary incompatible with what C++ TF uses.
+    # We need ~1 quarter to update properly.
+    # See also: https://github.com/tensorflow/tensorflow/issues/53234
+    # See also: https://github.com/protocolbuffers/protobuf/issues/9954
+    # See also: https://github.com/tensorflow/tensorflow/issues/56077
+    # This is a temporary patch for now, to patch previous TF releases.
+    'protobuf >= 3.1.4, < 3.20',
     'pyyaml',
 ]
 

From fc327a7bd2997d25eeffa8372314a89cd15d5df7 Mon Sep 17 00:00:00 2001
From: Soo Sung 
Date: Wed, 1 Jun 2022 11:15:33 -0700
Subject: [PATCH 158/872] Updating dtensor to experimental in toc

PiperOrigin-RevId: 452340482
---
 site/en/guide/_toc.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index 1665073dc7c..e5c774568a8 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -37,7 +37,7 @@ toc:
   path: /guide/tf_numpy
 - title: "DTensor concepts"
   path: /guide/dtensor_overview
-  status: nightly
+  status: experimental
 - title: "Thinking in TensorFlow 2"
   path: /guide/effective_tf2
 

From e4232194702de5589194ec5f0443f59d721e3eab Mon Sep 17 00:00:00 2001
From: chunduriv <74177924+chunduriv@users.noreply.github.com>
Date: Thu, 2 Jun 2022 15:10:12 +0530
Subject: [PATCH 159/872] Corrected Python Wheels for Unix

Updated correct paths for Tensorflow Unix Python wheels
---
 site/en/install/pip.md | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index b52765c3e66..6bc4a61e5c8 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -428,35 +428,35 @@ The value you specify depends on your Python version.
   
Linux

   

     Python 3.7 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.7 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.8 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp38-cp38-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.8 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp38-cp38-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.9 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp39-cp39-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.9 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp39-cp39-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.10 GPU supporthttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp310-cp310-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

   

     Python 3.10 CPU-onlyhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp310-cp310-manylinux2014.whlhttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   

 
   
macOS (CPU-only)
\n",
         "  \n",
         "
\n",
@@ -67,11 +67,11 @@
         "id": "-5jGPDA2PDPI"
       },
       "source": [
-        "Training a model will usually come with some amount of feature preprocessing, particularly when dealing with structured data. When training a `tf.estimator.Estimator` in TF1, this feature preprocessing is usually done with the `tf.feature_column` API. In TF2, this preprocessing can be done directly with Keras layers, called _preprocessing layers_.\n",
+        "Training a model usually comes with some amount of feature preprocessing, particularly when dealing with structured data. When training a `tf.estimator.Estimator` in TensorFlow 1, you usually perform feature preprocessing with the `tf.feature_column` API. In TensorFlow 2, you can do this directly with Keras preprocessing layers.\n",
         "\n",
-        "In this migration guide, you will perform some common feature transformations using both feature columns and preprocessing layers, followed by training a complete model with both APIs.\n",
+        "This migration guide demonstrates common feature transformations using both feature columns and preprocessing layers, followed by training a complete model with both APIs.\n",
         "\n",
-        "First, start with a couple of necessary imports,"
+        "First, start with a couple of necessary imports:"
       ]
     },
     {
@@ -93,7 +93,7 @@
         "id": "NVPYTQAWtDwH"
       },
       "source": [
-        "and add a utility for calling a feature column for demonstration:"
+        "Now, add a utility function for calling a feature column for demonstration:"
       ]
     },
     {
@@ -880,7 +880,7 @@
         "id": "IXMBwzggwUjI"
       },
       "source": [
-        "Note: Preprocessing layers are not trainable, which allows you to apply them *asynchronously* using `tf.data`. This has performance benefits, as you can both [prefetch](https://www.tensorflow.org/guide/data_performance#prefetching) preprocessed batches, and free up any accelerators to focus on the differentiable parts of a model. As this guide shows, separating preprocessing during training and composing it during inference is a flexible way to leverage these performance gains. However, if your model is small or preprocessing time is negligible, it may be simpler to build preprocessing into a complete model from the start. To do this you can build a single model starting with `tf.keras.Input`, followed by preprocessing layers, followed by trainable layers."
+        "Note: Preprocessing layers are not trainable, which allows you to apply them *asynchronously* using `tf.data`. This has performance benefits, as you can both prefetch preprocessed batches, and free up any accelerators to focus on the differentiable parts of a model (learn more in the _Prefetching_ section of the [Better performance with the `tf.data` API](../data_performance.ipynb) guide). As this guide shows, separating preprocessing during training and composing it during inference is a flexible way to leverage these performance gains. However, if your model is small or preprocessing time is negligible, it may be simpler to build preprocessing into a complete model from the start. To do this you can build a single model starting with `tf.keras.Input`, followed by preprocessing layers, followed by trainable layers."
       ]
     },
     {
@@ -959,7 +959,7 @@
         "  
\n",
         "\n",
-        "\\* `output_mode` can be passed to `layers.CategoryEncoding`, `layers.StringLookup`, `layers.IntegerLookup`, and `layers.TextVectorization`.\n",
+        "\\* `output_mode` can be passed to `tf.keras.layers.CategoryEncoding`, `tf.keras.layers.StringLookup`, `tf.keras.layers.IntegerLookup`, and `tf.keras.layers.TextVectorization`.\n",
         "\n",
         "† `layers.TextVectorization` can handle freeform text input directly (e.g. entire sentences or paragraphs). This is not one-to-one replacement for categorical sequence handling in TF1, but may offer a convenient replacement for ad-hoc text preprocessing.\n",
         "\n",
@@ -972,10 +972,10 @@
         "id": "AQCJ6lM3YDq_"
       },
       "source": [
-        "## Next Steps\n",
+        "## Next steps\n",
         "\n",
-        " - For more information on keras preprocessing layers, see [the guide to preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers).\n",
-        " - For a more in-depth example of applying preprocessing layers to structured data, see [the structured data tutorial](https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers)."
+        " - For more information on Keras preprocessing layers, go to the [Working with preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) guide.\n",
+        " - For a more in-depth example of applying preprocessing layers to structured data, refer to the [Classify structured data using Keras preprocessing layers](../../tutorials/structured_data/preprocessing_layers.ipynb) tutorial."
       ]
     }
   ],

From e98febee36a8ecd9c821b87112a363549d662220 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Mon, 13 Jun 2022 16:03:57 +0100
Subject: [PATCH 184/872] Use full namespaces to discoverability, lint the doc

---
 .../migrate/migrating_feature_columns.ipynb   | 70 +++++++++----------
 1 file changed, 35 insertions(+), 35 deletions(-)

diff --git a/site/en/guide/migrate/migrating_feature_columns.ipynb b/site/en/guide/migrate/migrating_feature_columns.ipynb
index aab2ce1d092..500894efa8f 100644
--- a/site/en/guide/migrate/migrating_feature_columns.ipynb
+++ b/site/en/guide/migrate/migrating_feature_columns.ipynb
@@ -615,7 +615,7 @@
         "id": "e_4Xx2c37lqD"
       },
       "source": [
-        "Define some common constants for both TF1 and TF2 workflows:"
+        "Define some common constants for both TensorFlow 1 and TensorFlow 2 workflows:"
       ]
     },
     {
@@ -892,76 +892,76 @@
         "## Feature column equivalence table\n",
         "\n",
         "For reference, here is an approximate correspondence between feature columns and\n",
-        "preprocessing layers:\n",
+        "Keras preprocessing layers:
\n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
+        "    \n",
         "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "  \n",
-        "    \n",
-        "    \n",
+        "    \n",
+        "    \n",
         "  \n",
         "
Feature ColumnKeras LayerFeature columnKeras layer
`feature_column.bucketized_column``layers.Discretization``tf.feature_column.bucketized_column``tf.keras.layers.Discretization`
`feature_column.categorical_column_with_hash_bucket``layers.Hashing``tf.feature_column.categorical_column_with_hash_bucket``tf.keras.layers.Hashing`
`feature_column.categorical_column_with_identity``layers.CategoryEncoding``tf.feature_column.categorical_column_with_identity``tf.keras.layers.CategoryEncoding`
`feature_column.categorical_column_with_vocabulary_file``layers.StringLookup` or `layers.IntegerLookup``tf.feature_column.categorical_column_with_vocabulary_file``tf.keras.layers.StringLookup` or `tf.keras.layers.IntegerLookup`
`feature_column.categorical_column_with_vocabulary_list``layers.StringLookup` or `layers.IntegerLookup``tf.feature_column.categorical_column_with_vocabulary_list``tf.keras.layers.StringLookup` or `tf.keras.layers.IntegerLookup`
`feature_column.crossed_column``layers.experimental.preprocessing.HashedCrossing``tf.feature_column.crossed_column``tf.keras.layers.experimental.preprocessing.HashedCrossing`
`feature_column.embedding_column``layers.Embedding``tf.feature_column.embedding_column``tf.keras.layers.Embedding`
`feature_column.indicator_column``tf.feature_column.indicator_column``output_mode='one_hot'` or `output_mode='multi_hot'`*
`feature_column.numeric_column``layers.Normalization``tf.feature_column.numeric_column``tf.keras.layers.Normalization`
`feature_column.sequence_categorical_column_with_hash_bucket``layers.Hashing``tf.feature_column.sequence_categorical_column_with_hash_bucket``tf.keras.layers.Hashing`
`feature_column.sequence_categorical_column_with_identity``layers.CategoryEncoding``tf.feature_column.sequence_categorical_column_with_identity``tf.keras.layers.CategoryEncoding`
`feature_column.sequence_categorical_column_with_vocabulary_file``layers.StringLookup`, `layers.IntegerLookup`, or `layer.TextVectorization`†`tf.feature_column.sequence_categorical_column_with_vocabulary_file``tf.keras.layers.StringLookup`, `tf.keras.layers.IntegerLookup`, or `tf.keras.layer.TextVectorization`†
`feature_column.sequence_categorical_column_with_vocabulary_list``layers.StringLookup`, `layers.IntegerLookup`, or `layer.TextVectorization`†`tf.feature_column.sequence_categorical_column_with_vocabulary_list``tf.keras.layers.StringLookup`, `tf.keras.layers.IntegerLookup`, or `tf.keras.layer.TextVectorization`†
`feature_column.sequence_numeric_column``layers.Normalization``tf.feature_column.sequence_numeric_column``tf.keras.layers.Normalization`
`feature_column.weighted_categorical_column``layers.CategoryEncoding``tf.feature_column.weighted_categorical_column``tf.keras.layers.CategoryEncoding`
\n",
         "\n",
-        "\\* `output_mode` can be passed to `tf.keras.layers.CategoryEncoding`, `tf.keras.layers.StringLookup`, `tf.keras.layers.IntegerLookup`, and `tf.keras.layers.TextVectorization`.\n",
+        "\\* The `output_mode` can be passed to `tf.keras.layers.CategoryEncoding`, `tf.keras.layers.StringLookup`, `tf.keras.layers.IntegerLookup`, and `tf.keras.layers.TextVectorization`.\n",
         "\n",
-        "† `layers.TextVectorization` can handle freeform text input directly (e.g. entire sentences or paragraphs). This is not one-to-one replacement for categorical sequence handling in TF1, but may offer a convenient replacement for ad-hoc text preprocessing.\n",
+        "† `tf.keras.layers.TextVectorization` can handle freeform text input directly (for example, entire sentences or paragraphs). This is not one-to-one replacement for categorical sequence handling in TensorFlow 1, but may offer a convenient replacement for ad-hoc text preprocessing.\n",
         "\n",
         "Note: Linear estimators, such as `tf.estimator.LinearClassifier`, can handle direct categorical input (integer indices) without an `embedding_column` or `indicator_column`. However, integer indices cannot be passed directly to `tf.keras.layers.Dense` or `tf.keras.experimental.LinearModel`. These inputs should be first encoded with `tf.layers.CategoryEncoding` with `output_mode='count'` (and `sparse=True` if the category sizes are large) before calling into `Dense` or `LinearModel`."
       ]

From 1edfe155d5899ca000532da59cb71a3d6c4c6cf0 Mon Sep 17 00:00:00 2001
From: Lyndon Duong 
Date: Mon, 13 Jun 2022 08:45:08 -0700
Subject: [PATCH 185/872] Plot shape error in tf data compression

Two cells in the tensorflow data compression tutorial throw an error when run in Colab.

https://www.tensorflow.org/tutorials/generative/data_compression

To Reproduce:
Steps to reproduce the behavior:
1. Go to [the tutorial on colab](https://colab.sandbox.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/generative/data_compression.ipynb)
2. Two cells: `plt.imshow(x)` and `plt.imshow(x_tilde)`
3. They throw an Error: `Invalid shape (28, 28, 1) for image data`

Resolution:
Simply putting a `tf.squeeze()` so that e.g. `plt.imshow(x)` -> `plt.imshow(tf.squeeze(x))` will resolve the issue.
---
 site/en/tutorials/generative/data_compression.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
index 974d3f5dc01..6f47811f6a2 100644
--- a/site/en/tutorials/generative/data_compression.ipynb
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -303,7 +303,7 @@
       "source": [
         "(x, _), = validation_dataset.take(1)\n",
         "\n",
-        "plt.imshow(x)\n",
+        "plt.imshow(tf.squeeze(x))\n",
         "print(f\"Data type: {x.dtype}\")\n",
         "print(f\"Shape: {x.shape}\")\n"
       ]
@@ -426,7 +426,7 @@
         "print(\"distortion:\", distortion)\n",
         "\n",
         "x_tilde = tf.saturate_cast(x_tilde[0] * 255, tf.uint8)\n",
-        "plt.imshow(x_tilde)\n",
+        "plt.imshow(tf.squeeze(x_tilde))\n",
         "print(f\"Data type: {x_tilde.dtype}\")\n",
         "print(f\"Shape: {x_tilde.shape}\")\n"
       ]

From 6f4d56c8c12b41da8f0d8c303ca66bc877364096 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Mon, 13 Jun 2022 11:36:33 -0700
Subject: [PATCH 186/872] Updated Tensorflow docs on BackupAndRestore callback.
 Added examples to illustrate step granularity of BackupAndRestore callback.

PiperOrigin-RevId: 454660672
---
 site/en/guide/migrate/fault_tolerance.ipynb   | 150 ++++++++++++++++--
 .../distribute/multi_worker_with_keras.ipynb  |  93 ++++++++++-
 2 files changed, 223 insertions(+), 20 deletions(-)

diff --git a/site/en/guide/migrate/fault_tolerance.ipynb b/site/en/guide/migrate/fault_tolerance.ipynb
index 1109db36840..a44b113aeaa 100644
--- a/site/en/guide/migrate/fault_tolerance.ipynb
+++ b/site/en/guide/migrate/fault_tolerance.ipynb
@@ -13,6 +13,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
+        "cellView": "form",
         "id": "HMUDt0CiUJk9"
       },
       "outputs": [],
@@ -84,6 +85,26 @@
         "## Setup"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TOVQubuDzdmA"
+      },
+      "source": [
+        "Install `tf-nightly` as step Granularity for `BackupAndRestore` callback is only available in Tensorflow 2.10"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pGW0XhXkxY_q"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install tf-nightly"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -254,7 +275,7 @@
         "\n",
         "In TensorFlow 2, if you use the Keras `Model.fit` API for training, you can provide the `tf.keras.callbacks.BackupAndRestore` callback to add the fault tolerance functionality.\n",
         "\n",
-        "To help demonstrate this, let's first start by defining a callback class that artificially throws an error during the fifth checkpoint:\n"
+        "To help demonstrate this, let's first start by defining a callback class that artificially throws an error during the fourth epoch checkpoint:\n"
       ]
     },
     {
@@ -265,10 +286,13 @@
       },
       "outputs": [],
       "source": [
-        "class InterruptingCallback(tf.keras.callbacks.Callback):\n",
+        "class InterruptAtEpoch(tf.keras.callbacks.Callback):\n",
         "  # A callback for artificially interrupting training.\n",
+        "  def __init__(self, interrupting_epoch=3):\n",
+        "    self.interrupting_epoch = interrupting_epoch\n",
+        "\n",
         "  def on_epoch_end(self, epoch, log=None):\n",
-        "    if epoch == 4:\n",
+        "    if epoch == self.interrupting_epoch:\n",
         "      raise RuntimeError('Interruption')"
       ]
     },
@@ -278,7 +302,7 @@
         "id": "AhU3VTYZoDh-"
       },
       "source": [
-        "Then, define and instantiate a simple Keras model, define the loss function, call `Model.compile`, and set up a `tf.keras.callbacks.BackupAndRestore` callback that will save the checkpoints in a temporary directory:"
+        "Then, define and instantiate a simple Keras model, define the loss function, call `Model.compile`, and set up a `tf.keras.callbacks.BackupAndRestore` callback that will save the checkpoints in a temporary directory at epoch boundaries:"
       ]
     },
     {
@@ -296,20 +320,14 @@
         "    tf.keras.layers.Dropout(0.2),\n",
         "    tf.keras.layers.Dense(10)\n",
         "  ])\n",
-        "\n",
         "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
-        "\n",
         "model = create_model()\n",
         "model.compile(optimizer='adam',\n",
         "              loss=loss,\n",
-        "              metrics=['accuracy'],\n",
-        "              steps_per_execution=10)\n",
-        "\n",
+        "              metrics=['accuracy'])\n",
         "log_dir = tempfile.mkdtemp()\n",
-        "\n",
         "backup_restore_callback = tf.keras.callbacks.BackupAndRestore(\n",
-        "    backup_dir = log_dir\n",
-        ")"
+        "    backup_dir = log_dir)"
       ]
     },
     {
@@ -318,7 +336,7 @@
         "id": "LRRWmZqsvMrq"
       },
       "source": [
-        "Now, start training the model with `Model.fit`. During training, checkpoints will be saved thanks to the `backup_restore_callback` defined above, while the `InterruptingCallback` will raise an artificial exception to simulate a failure."
+        "Now, start training the model with `Model.fit`. During training, checkpoints will be saved thanks to the `backup_restore_callback` defined above, while the `InterruptingCallbackAtEpoch` will raise an artificial exception to simulate a failure after 4th epoch."
       ]
     },
     {
@@ -333,8 +351,9 @@
         "  model.fit(x=x_train,\n",
         "            y=y_train,\n",
         "            epochs=10,\n",
+        "            steps_per_epoch=100,\n",
         "            validation_data=(x_test, y_test),\n",
-        "            callbacks=[backup_restore_callback, InterruptingCallback()])\n",
+        "            callbacks=[backup_restore_callback, InterruptAtEpoch()])\n",
         "except Exception as e:\n",
         "  print(f'{type(e).__name__}:{e}')"
       ]
@@ -364,6 +383,108 @@
         "model.fit(x=x_train,\n",
         "            y=y_train,\n",
         "            epochs=10,\n",
+        "            steps_per_epoch=100,\n",
+        "            validation_data=(x_test, y_test),\n",
+        "            callbacks=[backup_restore_callback])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nP2dnpMPxtYj"
+      },
+      "source": [
+        "let's define another callback class that artificially throws an error during the 140th step."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YardkAaBxr-c"
+      },
+      "outputs": [],
+      "source": [
+        "class InterruptAtStep(tf.keras.callbacks.Callback):\n",
+        "  # A callback for artificially interrupting training.\n",
+        "  def __init__(self, interrupting_step=140):\n",
+        "    self.total_step_count = 0\n",
+        "    self.interrupting_step = interrupting_step\n",
+        "\n",
+        "  def on_batch_begin(self, batch, logs=None):\n",
+        "    self.total_step_count += 1\n",
+        "\n",
+        "  def on_batch_end(self, batch, logs=None):\n",
+        "    if self.total_step_count == self.interrupting_step:\n",
+        "      print(\"\\nInterrupting at step count\", self.total_step_count)\n",
+        "      raise RuntimeError('Interruption')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Af3VpehxyTpb"
+      },
+      "source": [
+        "Note: This section uses features that are only available in `tf-nightly`(until Tensorflow 2.10 is released).\n",
+        "\n",
+        "Now, lets set `save_freq` in `BackupAndRestore` to an integer value 30. Checkpoints will be saved every 30 steps. The `InterruptingCallbackAtSteps` will raise an artificial exception to simulate a failure at epoch 1 and step 40 (total step 140). The checkpoint would be last saved at epoch 1 step 20."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dHHCENDPyUHS"
+      },
+      "outputs": [],
+      "source": [
+        "log_dir_2 = tempfile.mkdtemp()\n",
+        "\n",
+        "backup_restore_callback = tf.keras.callbacks.BackupAndRestore(\n",
+        "    backup_dir = log_dir_2, save_freq=30\n",
+        ")\n",
+        "model = create_model()\n",
+        "model.compile(optimizer='adam',\n",
+        "              loss=loss,\n",
+        "              metrics=['accuracy'])\n",
+        "try:\n",
+        "  model.fit(x=x_train,\n",
+        "            y=y_train,\n",
+        "            epochs=10,\n",
+        "            steps_per_epoch=100,\n",
+        "            validation_data=(x_test, y_test),\n",
+        "            callbacks=[backup_restore_callback, InterruptAtStep()])\n",
+        "except Exception as e:\n",
+        "  print(f'{type(e).__name__}:{e}')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2-ggMFEHynMR"
+      },
+      "source": [
+        "Next, instantiate the Keras model, call `Model.compile`, and continue training the model with `Model.fit` from a previously saved checkpoint: notice that the training starts from epoch 2 and step 21."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "vT7Kx30NEqly"
+      },
+      "outputs": [],
+      "source": [
+        "model = create_model()\n",
+        "model.compile(optimizer='adam',\n",
+        "              loss=loss,\n",
+        "              metrics=['accuracy'],\n",
+        "              steps_per_execution=10)\n",
+        "model.fit(x=x_train,\n",
+        "            y=y_train,\n",
+        "            epochs=10,\n",
+        "            steps_per_epoch=100,\n",
         "            validation_data=(x_test, y_test),\n",
         "            callbacks=[backup_restore_callback])"
       ]
@@ -467,7 +588,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "fault_tolerance.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {
diff --git a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
index c98b3287c8a..1cbec8e2fb8 100644
--- a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
@@ -191,6 +191,26 @@
         "Finally, import TensorFlow:"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9hLpDZhAz2q-"
+      },
+      "source": [
+        "Install `tf-nightly` as step Granularity for `BackupAndRestore` callback is only available in Tensorflow 2.10"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "-XqozLfzz30N"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install tf-nightly"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -1129,13 +1149,15 @@
       "source": [
         "#### BackupAndRestore callback\n",
         "\n",
-        "The `tf.keras.callbacks.BackupAndRestore` callback provides the fault tolerance functionality by backing up the model and current epoch number in a temporary checkpoint file under `backup_dir` argument to `BackupAndRestore`. This is done at the end of each epoch.\n",
+        "The tf.keras.callbacks.BackupAndRestore callback provides the fault tolerance functionality by backing up the model and current training state in a temporary checkpoint file under `backup_dir` argument to `BackupAndRestore`. In the current stable Tensorflow 2.9 version, the current model and traning state is backed up at epoch boundaries.\n",
         "\n",
-        "Once jobs get interrupted and restart, the callback restores the last checkpoint, and training continues from the beginning of the interrupted epoch. Any partial training already done in the unfinished epoch before interruption will be thrown away, so that it doesn't affect the final model state.\n",
+        "In `tf-nightly` version, `BackupAndRestore` can back up the model and training state at epoch or step boundaries. `BackupAndRestore` accepts an optional `save_freq` argument. `save_freq` accepts either `'epoch'` or an `int` value. If `save_freq` is set to `'epoch'` the model is backed up after every epoch. If `save_freq` is set to an int value greater than 0, the model is backed up after every `save_freq` number of batches.\n",
         "\n",
-        "To use it, provide an instance of `tf.keras.callbacks.BackupAndRestore` at the `Model.fit` call.\n",
+        "Once jobs get interrupted and restarted, the callback restores the last checkpoint, and training continues from the beginning of the epoch and step at which the traning state was last saved.\n",
         "\n",
-        "With `MultiWorkerMirroredStrategy`, if a worker gets interrupted, the whole cluster will pause until the interrupted worker is restarted. Other workers will also restart, and the interrupted worker will rejoin the cluster. Then, every worker will read the checkpoint file that was previously saved and pick up its former state, thereby allowing the cluster to get back in sync. Then, the training will continue.\n",
+        "To use it, provide an instance of tf.keras.callbacks.BackupAndRestore at the Model.fit call.\n",
+        "\n",
+        "With `MultiWorkerMirroredStrategy`, if a worker gets interrupted, the whole cluster will pause until the interrupted worker is restarted. Other workers will also restart, and the interrupted worker will rejoin the cluster. Then, every worker will read the checkpoint file that was previously saved and pick up its former state, thereby allowing the cluster to get back in sync. Then, the training will continue. The distributed dataset iterator state will be re-initialized and not restored.\n",
         "\n",
         "The `BackupAndRestore` callback uses the `CheckpointManager` to save and restore the training state, which generates a file called checkpoint that tracks existing checkpoints together with the latest one. For this reason, `backup_dir` should not be re-used to store other checkpoints in order to avoid name collision.\n",
         "\n",
@@ -1153,7 +1175,37 @@
       "outputs": [],
       "source": [
         "# Multi-worker training with `MultiWorkerMirroredStrategy`\n",
-        "# and the `BackupAndRestore` callback.\n",
+        "# and the `BackupAndRestore` callback. The training state \n",
+        "# is backed up at epoch boundaries by default.\n",
+        "\n",
+        "callbacks = [tf.keras.callbacks.BackupAndRestore(backup_dir='/tmp/backup')]\n",
+        "with strategy.scope():\n",
+        "  multi_worker_model = mnist_setup.build_and_compile_cnn_model()\n",
+        "multi_worker_model.fit(multi_worker_dataset,\n",
+        "                       epochs=3,\n",
+        "                       steps_per_epoch=70,\n",
+        "                       callbacks=callbacks)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f8e86TAp0Rsl"
+      },
+      "source": [
+        "If `save_freq` is set to `'epoch'` the model is backed up after every epoch."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "rZjQGPsF0aEI"
+      },
+      "outputs": [],
+      "source": [
+        "# The training stateis backed up at epoch boundaries because `save_freq` is\n",
+        "# set to `epoch`.\n",
         "\n",
         "callbacks = [tf.keras.callbacks.BackupAndRestore(backup_dir='/tmp/backup')]\n",
         "with strategy.scope():\n",
@@ -1161,6 +1213,37 @@
         "multi_worker_model.fit(multi_worker_dataset,\n",
         "                       epochs=3,\n",
         "                       steps_per_epoch=70,\n",
+        "                       callbacks=callbacks)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "p-r44kCM0jc6"
+      },
+      "source": [
+        "Note: The next code block uses fatures that are only available in `tf-nightly`(until Tensorflow 2.10 is released)\n",
+        "\n",
+        "If `save_freq` is set to an int value greater than 0, the model is backed up after every `save_freq` number of batches."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bSJUyLSF0moC"
+      },
+      "outputs": [],
+      "source": [
+        "# The training state is backed up at every 30 steps because save_freq is set\n",
+        "# an integer value of 30.\n",
+        "\n",
+        "callbacks = [tf.keras.callbacks.BackupAndRestore(backup_dir='/tmp/backup', save_freq=30)]\n",
+        "with strategy.scope():\n",
+        "  multi_worker_model = mnist_setup.build_and_compile_cnn_model()\n",
+        "multi_worker_model.fit(multi_worker_dataset,\n",
+        "                       epochs=3,\n",
+        "                       steps_per_epoch=70,\n",
         "                       callbacks=callbacks)"
       ]
     },

From 053b5ff7701bb74fd25c9b1983b21d093a8159c7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Johannes=20Ball=C3=A9?= 
Date: Mon, 13 Jun 2022 14:25:05 -0700
Subject: [PATCH 187/872] Fixes matplotlib error when displaying digit.

PiperOrigin-RevId: 454699205
---
 site/en/tutorials/generative/data_compression.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
index 974d3f5dc01..6f47811f6a2 100644
--- a/site/en/tutorials/generative/data_compression.ipynb
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -303,7 +303,7 @@
       "source": [
         "(x, _), = validation_dataset.take(1)\n",
         "\n",
-        "plt.imshow(x)\n",
+        "plt.imshow(tf.squeeze(x))\n",
         "print(f\"Data type: {x.dtype}\")\n",
         "print(f\"Shape: {x.shape}\")\n"
       ]
@@ -426,7 +426,7 @@
         "print(\"distortion:\", distortion)\n",
         "\n",
         "x_tilde = tf.saturate_cast(x_tilde[0] * 255, tf.uint8)\n",
-        "plt.imshow(x_tilde)\n",
+        "plt.imshow(tf.squeeze(x_tilde))\n",
         "print(f\"Data type: {x_tilde.dtype}\")\n",
         "print(f\"Shape: {x_tilde.shape}\")\n"
       ]

From bfcede96b138823fec918ef739a089d872418695 Mon Sep 17 00:00:00 2001
From: Olzhas Akpambetov 
Date: Thu, 16 Jun 2022 16:01:50 -0700
Subject: [PATCH 188/872] Add Caution to make_csv_dataset, fix decode_csv usage
 in Load CSV data tutorial

PiperOrigin-RevId: 455487168
---
 site/en/tutorials/load_data/csv.ipynb | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/load_data/csv.ipynb b/site/en/tutorials/load_data/csv.ipynb
index 60dacdb65bf..2626beaf8d5 100644
--- a/site/en/tutorials/load_data/csv.ipynb
+++ b/site/en/tutorials/load_data/csv.ipynb
@@ -953,7 +953,9 @@
         "This function includes many convenient features, so the data is easy to work with. This includes:\n",
         "\n",
         "* Using the column headers as dictionary keys.\n",
-        "* Automatically determining the type of each column."
+        "* Automatically determining the type of each column.\n",
+        "\n",
+        "Caution: Make sure to set the `num_epochs` argument in `tf.data.experimental.make_csv_dataset`, otherwise the default behavior for `tf.data.Dataset` is to loop endlessly."
       ]
     },
     {
@@ -1489,7 +1491,7 @@
         "\n",
         "The `tf.data.experimental.CsvDataset` class provides a minimal CSV `Dataset` interface without the convenience features of the `tf.data.experimental.make_csv_dataset` function: column header parsing, column type-inference, automatic shuffling, file interleaving.\n",
         "\n",
-        "This constructor follows uses `record_defaults` the same way as `io.parse_csv`:\n"
+        "This constructor uses `record_defaults` the same way as `tf.io.decode_csv`:\n"
       ]
     },
     {

From d2030c2840f95929cbb0ea455351cb00540baaee Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Fri, 17 Jun 2022 09:42:43 -0700
Subject: [PATCH 189/872] Tag estimator section as deprecated.

PiperOrigin-RevId: 455637386
---
 site/en/tutorials/_toc.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml
index 36ccea93cbf..b183932ecad 100644
--- a/site/en/tutorials/_toc.yaml
+++ b/site/en/tutorials/_toc.yaml
@@ -210,6 +210,7 @@ toc:
 
 - title: "tf.Estimator"
   style: accordion
+  status: deprecated
   section:
   - title: "Premade estimator"
     path: /tutorials/estimator/premade

From 6c14c6e788198d1b4ff812d9c2e9748a39c5cfc8 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Fri, 17 Jun 2022 17:51:18 -0700
Subject: [PATCH 190/872] Add support for TOC sections with overlapping
 namespace.

e.g. `org.tf.lite.task` and `org.tf.lite.task.gms`.

PiperOrigin-RevId: 455730498
---
 .../api_generator/toc_processing.py           | 13 ++++-
 .../api_generator/toc_processing_test.py      | 52 ++++++++++++++++++-
 2 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/toc_processing.py b/tools/tensorflow_docs/api_generator/toc_processing.py
index a11ed1e9eee..fbe0c3cf842 100644
--- a/tools/tensorflow_docs/api_generator/toc_processing.py
+++ b/tools/tensorflow_docs/api_generator/toc_processing.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 # ==============================================================================
 """Tools for processing generated Java documentation."""
+import itertools
 from typing import Any, Iterable, Mapping, MutableMapping, MutableSequence
 
 # TODO(b/193033225): If possible, this should be a TypedDict. If not, using the
@@ -33,8 +34,16 @@ def add_package_headings(toc: Toc, root_pkgs: Iterable[str],
       if new_entry.get('title', '').startswith(root_pkg):
         # Strip the common root_pkg from the title.
         new_title = new_entry['title'][len(root_pkg):].lstrip('.')
-        # The section label is the first sub-package (this.notthis.orthis)
-        section, *_ = new_title.split('.')
+        # a, a.b, a.b.c from the remainder of the package name
+        all_sections = itertools.accumulate(
+            new_title.split('.'), lambda a, b: f'{a}.{b}')
+        # Filter out any packages that aren't defined as labelled sections.
+        candidate_sections = filter(lambda s: f'{root_pkg}.{s}' in labels,
+                                    all_sections)
+        # If there are more than one, pick the most specific (longest). If there
+        # are none, use the bare trailing package.
+        section = max(candidate_sections, key=len, default=new_title)
+
         if section != current_section:
           # We've hit a new section, add a label if one was supplied.
           section_pkg = f'{root_pkg}.{section}' if section else root_pkg
diff --git a/tools/tensorflow_docs/api_generator/toc_processing_test.py b/tools/tensorflow_docs/api_generator/toc_processing_test.py
index 087f69e626d..8b49c0b3728 100644
--- a/tools/tensorflow_docs/api_generator/toc_processing_test.py
+++ b/tools/tensorflow_docs/api_generator/toc_processing_test.py
@@ -125,7 +125,57 @@ def test_multiple_roots(self):
             },
         ]
     }
-    self.assertDictEqual(actual_toc, expected_toc)
+    self.assertEqual(actual_toc['toc'], expected_toc['toc'])
+
+  def test_overlapping_packages(self):
+    toc_in = {
+        'toc': [{
+            'title': 'org.tf.one',
+            'path': '/tf/one/docs.html',
+            'section': [{
+                'title': 'SymbolOne',
+                'path': '/tf/one/symbol.html',
+            }],
+        }, {
+            'title':
+                'org.tf.one.two',
+            'path':
+                '/tf/one/two/docs.html',
+            'section': [{
+                'title': 'SymbolOneTwo',
+                'path': '/tf/one/two/SymbolOneTwo.html',
+            }],
+        }]
+    }
+    labels = {
+        'org.tf.one': 'Section One',
+        'org.tf.one.two': 'Section Two',
+    }
+    actual_toc = toc_processing.add_package_headings(toc_in, ['org.tf'], labels)
+    expected_toc = {
+        'toc': [{
+            'heading': 'Section One',
+        }, {
+            'title': 'one',
+            'path': '/tf/one/docs.html',
+            'section': [{
+                'title': 'SymbolOne',
+                'path': '/tf/one/symbol.html',
+            }],
+        }, {
+            'heading': 'Section Two',
+        }, {
+            'title':
+                'one.two',
+            'path':
+                '/tf/one/two/docs.html',
+            'section': [{
+                'title': 'SymbolOneTwo',
+                'path': '/tf/one/two/SymbolOneTwo.html',
+            }],
+        }]
+    }
+    self.assertEqual(actual_toc['toc'], expected_toc['toc'])
 
   def test_nesting_toc(self):
     toc_in = {

From 5d42b865e45d76d738418b838c62659b7f88d1bd Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 22 Jun 2022 11:45:41 -0700
Subject: [PATCH 191/872] Centralize tensorflow-models docs into a top-level
 docs/ directory.

PiperOrigin-RevId: 456569753
---
 site/en/guide/_toc.yaml                       |  10 +-
 site/en/guide/model_garden/index.md           | 136 ----
 site/en/tutorials/_toc.yaml                   |   3 -
 .../classification_with_model_garden.ipynb    | 691 ------------------
 4 files changed, 8 insertions(+), 832 deletions(-)
 delete mode 100644 site/en/guide/model_garden/index.md
 delete mode 100644 site/en/tutorials/images/classification_with_model_garden.ipynb

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index e5c774568a8..1f98745faee 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -83,10 +83,16 @@ toc:
   path: /guide/mixed_precision
 
 - heading: "Model Garden"
-- title: "Overview"
-  path: /guide/model_garden
   status: new
+- title: "Overview"
+  path: /tfmodels
+- title: "Example: Image classification"
+  path: /tfmodels/vision/image_classification
+- title: "Training with Orbit"
+  path: /tfmodels/orbit
+
 - heading: "Estimators"
+  status: deprecated
 - title: "Estimator overview"
   path: /guide/estimator
 
diff --git a/site/en/guide/model_garden/index.md b/site/en/guide/model_garden/index.md
deleted file mode 100644
index 57b022a62d1..00000000000
--- a/site/en/guide/model_garden/index.md
+++ /dev/null
@@ -1,136 +0,0 @@
-# Model Garden overview
-
-The TensorFlow Model Garden provides implementations of many state-of-the-art
-machine learning (ML) models for vision and natural language processing (NLP),
-as well as workflow tools to let you quickly configure and run those models on
-standard datasets. Whether you are looking to benchmark performance for a
-well-known model, verify the results of recently released research, or extend
-existing models, the Model Garden can help you drive your ML research and
-applications forward.
-
-The Model Garden includes the following resources for machine learning
-developers:
-
--   [**Official models**](#official) for vision and NLP, maintained by Google
-    engineers
--   [**Research models**](#research) published as part of ML research papers
--   [**Training experiment framework**](#training_framework) for fast,
-    declarative training configuration of official models
--   [**Specialized ML operations**](#ops) for vision and natural language
-    processing (NLP)
--   [**Model training loop**](#orbit) management with Orbit
-
-These resources are built to be used with the TensorFlow Core framework and
-integrate with your existing TensorFlow development projects. Model
-Garden resources are also provided under an [open
-source](https://github.com/tensorflow/models/blob/master/LICENSE) license, so
-you can freely extend and distribute the models and tools.
-
-Practical ML models are computationally intensive to train and run, and may
-require accelerators such as Graphical Processing Units (GPUs) and Tensor
-Processing Units (TPUs). Most of the models in Model Garden were trained on
-large datasets using TPUs. However, you can also train and run these models on
-GPU and CPU processors.
-
-## Model Garden models
-
-The machine learning models in the Model Garden include full code so you can
-test, train, or re-train them for research and experimentation. The Model Garden
-includes two primary categories of models: *official models* and *research
-models*.
-
-### Official models {:#official}
-
-The [Official Models](https://github.com/tensorflow/models/tree/master/official)
-repository is a collection of state-of-the-art models, with a focus on
-vision and natural language processing (NLP).
-These models are implemented using current TensorFlow 2.x high-level
-APIs. Model libraries in this repository are optimized for fast performance and
-actively maintained by Google engineers. The official models include additional
-metadata you can use to quickly configure experiments using the Model Garden
-[training experiment framework](#training_framework).
-
-### Research models {:#research}
-
-The [Research Models](https://github.com/tensorflow/models/tree/master/research)
-repository is a collection of models published as code resources for research
-papers. These models are implemented using both TensorFlow 1.x and 2.x. Model
-libraries in the research folder are supported by the code owners and the
-research community.
-
-## Training experiment framework {:#training_framework}
-
-The Model Garden training experiment framework lets you quickly assemble and
-run training experiments using its official models and standard datasets. The
-training framework uses additional metadata included with the Model Garden's
-official models to allow you to configure models quickly using a declarative
-programming model. You can define a training experiment using Python commands in
-the [TensorFlow Model library](../../api_docs/python/tfm/core)
-or configure training using a YAML configuration file, like this
-[example](https://github.com/tensorflow/models/blob/master/official/vision/configs/experiments/image_classification/imagenet_resnet50_tpu.yaml).
-
-The training framework uses
-[`tfm.core.base_trainer.ExperimentConfig`](../../api_docs/python/tfm/core/base_trainer/ExperimentConfig)
-as the configuration object, which contains the following top-level
-configuration objects:
-
--   [`runtime`](https://www.tensorflow.org/api_docs/python/tfm/core/base_task/RuntimeConfig):
-    Defines the processing hardware, distribution strategy, and other
-    performance optimizations
--   [`task`](https://www.tensorflow.org/api_docs/python/tfm/core/config_definitions/TaskConfig):
-    Defines the model, training data, losses, and initialization
--   [`trainer`](https://www.tensorflow.org/api_docs/python/tfm/core/base_trainer/TrainerConfig):
-    Defines the optimizer, training loops, evaluation loops, summaries, and
-    checkpoints
-
-For a complete example using the Model Garden training experiment framework,
-see the
-[Image classification with Model Garden](../../tutorials/images/classification_with_model_garden)
-tutorial. For information on the training experiment framework, check out the
-[TensorFlow Models API documentation](../../api_docs/python/tfm/core).
-If you are looking for a solution to manage training loops for your model
-training experiments, check out [Orbit](#orbit).
-
-## Specialized ML operations {:#ops}
-
-The Model Garden contains many vision and NLP operations specifically designed
-to execute state-of-the-art models that run efficiently on GPUs and TPUs. Review
-the TensorFlow Models Vision library API docs for a list of specialized [vision
-operations](../../api_docs/python/tfm/vision). Review the
-TensorFlow Models NLP Library API docs for a list of [NLP
-operations](../../api_docs/python/tfm/nlp). These libraries
-also include additional utility functions used for vision and NLP data
-processing, training, and model execution.
-
-## Training loops with Orbit {:#orbit}
-
-The Orbit tool is a flexible, lightweight library designed to make it easier to
-write custom training loops in TensorFlow 2.x, and works well with the Model
-Garden [training experiment framework](#training_framework). Orbit handles
-common model training tasks such as saving checkpoints, running model
-evaluations, and setting up summary writing. It seamlessly integrates with
-`tf.distribute` and supports running on different device types, including CPU,
-GPU, and TPU hardware. The Orbit tool is also [open
-source](https://github.com/tensorflow/models/blob/master/orbit/LICENSE), so you
-can extend and adapt to your model training needs.
-
-You generally train TensorFlow models by writing a
-[custom training loop](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch),
-or using the high-level Keras
-[Model.fit](../../api_docs/python/tf/keras/Model#fit)
-function. For simple models, you can define and manage a custom training loop
-with low-level TensorFlow methods such as `tf.GradientTape` or `tf.function`.
-Alternatively, you can use the high-level Keras `Model.fit`.
-
-However, if your model is complex and your training loop requires more flexible
-control or customization, then you should use Orbit. You can define most of your
-training loop by the `orbit.AbstractTrainer` or `orbit.StandardTrainer` class.
-Learn more about the Orbit tool in the
-[Orbit API documentation](../../api_docs/python/orbit).
-
-Note: You can use the Keras API to do what Orbit does, but you must override
-the TensorFlow `train_step` function or use callbacks like ModelCheckpoint or
-TensorBoard. For more information about modifying the behavior of `train_step`,
-check out the
-[Customize what happens in Model.fit](https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit)
-page.
diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml
index b183932ecad..5eb4825ca41 100644
--- a/site/en/tutorials/_toc.yaml
+++ b/site/en/tutorials/_toc.yaml
@@ -109,9 +109,6 @@ toc:
     path: /tutorials/images/data_augmentation
   - title: "Image segmentation"
     path: /tutorials/images/segmentation
-  - title: "Image classification with Model Garden"
-    path: /tutorials/images/classification_with_model_garden
-    status: new
   - title: "Object detection with TF Hub"
     path: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_object_detection.ipynb
     status: external
diff --git a/site/en/tutorials/images/classification_with_model_garden.ipynb b/site/en/tutorials/images/classification_with_model_garden.ipynb
deleted file mode 100644
index 0eea176f634..00000000000
--- a/site/en/tutorials/images/classification_with_model_garden.ipynb
+++ /dev/null
@@ -1,691 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "Tce3stUlHN0L"
-      },
-      "source": [
-        "##### Copyright 2020 The TensorFlow Authors."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "cellView": "form",
-        "id": "tuOe1ymfHZPu"
-      },
-      "outputs": [],
-      "source": [
-        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
-        "# you may not use this file except in compliance with the License.\n",
-        "# You may obtain a copy of the License at\n",
-        "#\n",
-        "# https://www.apache.org/licenses/LICENSE-2.0\n",
-        "#\n",
-        "# Unless required by applicable law or agreed to in writing, software\n",
-        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
-        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
-        "# See the License for the specific language governing permissions and\n",
-        "# limitations under the License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "qFdPvlXBOdUN"
-      },
-      "source": [
-        "# Image classification with Model Garden"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "MfBg1C5NB3X0"
-      },
-      "source": [
-        "\n",
-        "  \n",
-        "  \n",
-        "  \n",
-        "  \n",
-        "
\n",
-        "    View on TensorFlow.org\n",
-        "  \n",
-        "    Run in Google Colab\n",
-        "  \n",
-        "    View on GitHub\n",
-        "  \n",
-        "    Download notebook\n",
-        "  
"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "Ta_nFXaVAqLD"
-      },
-      "source": [
-        "This tutorial fine-tunes a Residual Network (ResNet) from the TensorFlow [Model Garden](https://github.com/tensorflow/models) package (`tensorflow-models`) to classify images in the [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html) dataset.\n",
-        "\n",
-        "Model Garden contains a collection of state-of-the-art vision models, implemented with TensorFlow's high-level APIs. The implementations demonstrate the best practices for modeling, letting users to take full advantage of TensorFlow for their research and product development.\n",
-        "\n",
-        "This tutorial uses a [ResNet](https://arxiv.org/pdf/1512.03385.pdf) model, a state-of-the-art image classifier. This tutorial uses the ResNet-18 model, a convolutional neural network with 18 layers.\n",
-        "\n",
-        "This tutorial demonstrates how to:\n",
-        "1. Use models from the TensorFlow Models package.\n",
-        "2. Fine-tune a pre-built ResNet for image classification.\n",
-        "3. Export the tuned ResNet model."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "G2FlaQcEPOER"
-      },
-      "source": [
-        "## Setup\n",
-        "\n",
-        "Install and import the necessary modules. This tutorial uses the `tf-models-nightly` version of Model Garden.\n",
-        "\n",
-        "Note: Upgrading TensorFlow to 2.9 in Colab breaks GPU support, so this colab is set to run on CPU until the Colab runtimes are updated."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "XvWfdCrvrV5W"
-      },
-      "outputs": [],
-      "source": [
-        "!pip uninstall -y opencv-python\n",
-        "!pip install -U -q \"tensorflow>=2.9.0\" \"tf-models-official\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "CKYMTPjOE400"
-      },
-      "source": [
-        "Import TensorFlow, TensorFlow Datasets, and a few helper libraries."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "Wlon1uoIowmZ"
-      },
-      "outputs": [],
-      "source": [
-        "import pprint\n",
-        "import tempfile\n",
-        "\n",
-        "from IPython import display\n",
-        "import matplotlib.pyplot as plt\n",
-        "\n",
-        "import tensorflow as tf\n",
-        "import tensorflow_datasets as tfds"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "AVTs0jDd1b24"
-      },
-      "source": [
-        "The `tensorflow_models` package contains the ResNet vision model, and the `official.vision.serving` model contains the function to save and export the tuned model."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "NHT1iiIiBzlC"
-      },
-      "outputs": [],
-      "source": [
-        "import tensorflow_models as tfm\n",
-        "\n",
-        "# These are not in the tfm public API for v2.9. They will be available in v2.10\n",
-        "from official.vision.serving import export_saved_model_lib\n",
-        "import official.core.train_lib"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "aKv3wdqkQ8FU"
-      },
-      "source": [
-        "## Configure the ResNet-18 model for the Cifar-10 dataset"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "5iN8mHEJjKYE"
-      },
-      "source": [
-        "The CIFAR10 dataset contains 60,000 color images in mutually exclusive 10 classes, with 6,000 images in each class.\n",
-        "\n",
-        "In Model Garden, the collections of parameters that define a model are called *configs*. Model Garden can create a config based on a known set of parameters via a [factory](https://en.wikipedia.org/wiki/Factory_method_pattern).\n",
-        "\n",
-        "Use the `resnet_imagenet` factory configuration, as defined by `tfm.vision.configs.image_classification.image_classification_imagenet`. The configuration is set up to train ResNet to converge on [ImageNet](https://www.image-net.org/)."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "1M77f88Dj2Td"
-      },
-      "outputs": [],
-      "source": [
-        "exp_config = tfm.core.exp_factory.get_exp_config('resnet_imagenet')\n",
-        "tfds_name = 'cifar10'\n",
-        "ds_info = tfds.builder(tfds_name ).info\n",
-        "ds_info"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "U6PVwXA-j3E7"
-      },
-      "source": [
-        "Adjust the model and dataset configurations so that it works with Cifar-10 (`cifar10`)."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "YWI7faVStQaV"
-      },
-      "outputs": [],
-      "source": [
-        "# Configure model\n",
-        "exp_config.task.model.num_classes = 10\n",
-        "exp_config.task.model.input_size = list(ds_info.features[\"image\"].shape)\n",
-        "exp_config.task.model.backbone.resnet.model_id = 18\n",
-        "\n",
-        "# Configure training and testing data\n",
-        "batch_size = 128\n",
-        "\n",
-        "exp_config.task.train_data.input_path = ''\n",
-        "exp_config.task.train_data.tfds_name = tfds_name\n",
-        "exp_config.task.train_data.tfds_split = 'train'\n",
-        "exp_config.task.train_data.global_batch_size = batch_size\n",
-        "\n",
-        "exp_config.task.validation_data.input_path = ''\n",
-        "exp_config.task.validation_data.tfds_name = tfds_name\n",
-        "exp_config.task.validation_data.tfds_split = 'test'\n",
-        "exp_config.task.validation_data.global_batch_size = batch_size\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "DE3ggKzzTD56"
-      },
-      "source": [
-        "Adjust the trainer configuration."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "inE_-4UGkLud"
-      },
-      "outputs": [],
-      "source": [
-        "logical_device_names = [logical_device.name for logical_device in tf.config.list_logical_devices()]\n",
-        "\n",
-        "if 'GPU' in ''.join(logical_device_names):\n",
-        "  print('This may be broken in Colab.')\n",
-        "  device = 'GPU'\n",
-        "elif 'TPU' in ''.join(logical_device_names):\n",
-        "  print('This may be broken in Colab.')\n",
-        "  device = 'TPU'\n",
-        "else:\n",
-        "  print('Running on CPU is slow, so only train for a few steps.')\n",
-        "  device = 'CPU'\n",
-        "\n",
-        "if device=='CPU':\n",
-        "  train_steps = 20\n",
-        "  exp_config.trainer.steps_per_loop = 5\n",
-        "else:\n",
-        "  train_steps=5000\n",
-        "  exp_config.trainer.steps_per_loop = 100\n",
-        "\n",
-        "exp_config.trainer.summary_interval = 100\n",
-        "exp_config.trainer.checkpoint_interval = train_steps\n",
-        "exp_config.trainer.validation_interval = 1000\n",
-        "exp_config.trainer.validation_steps =  ds_info.splits['test'].num_examples // batch_size\n",
-        "exp_config.trainer.train_steps = train_steps\n",
-        "exp_config.trainer.optimizer_config.learning_rate.type = 'cosine'\n",
-        "exp_config.trainer.optimizer_config.learning_rate.cosine.decay_steps = train_steps\n",
-        "exp_config.trainer.optimizer_config.learning_rate.cosine.initial_learning_rate = 0.1\n",
-        "exp_config.trainer.optimizer_config.warmup.linear.warmup_steps = 100"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "5mTcDnBiTOYD"
-      },
-      "source": [
-        "Print the modified configuration."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "tuVfxSBCTK-y"
-      },
-      "outputs": [],
-      "source": [
-        "pprint.pprint(exp_config.as_dict())\n",
-        "\n",
-        "display.Javascript(\"google.colab.output.setIframeHeight('300px');\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "w7_X0UHaRF2m"
-      },
-      "source": [
-        "Set up the distribution strategy."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "ykL14FIbTaSt"
-      },
-      "outputs": [],
-      "source": [
-        "logical_device_names = [logical_device.name for logical_device in tf.config.list_logical_devices()]\n",
-        "\n",
-        "if exp_config.runtime.mixed_precision_dtype == tf.float16:\n",
-        "    tf.keras.mixed_precision.set_global_policy('mixed_float16')\n",
-        "\n",
-        "if 'GPU' in ''.join(logical_device_names):\n",
-        "  distribution_strategy = tf.distribute.MirroredStrategy()\n",
-        "elif 'TPU' in ''.join(logical_device_names):\n",
-        "  tf.tpu.experimental.initialize_tpu_system()\n",
-        "  tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='/device:TPU_SYSTEM:0')\n",
-        "  distribution_strategy = tf.distribute.experimental.TPUStrategy(tpu)\n",
-        "else:\n",
-        "  print('Warning: this will be really slow.')\n",
-        "  distribution_strategy = tf.distribute.OneDeviceStrategy(logical_device_names[0])"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "W4k5YH5pTjaK"
-      },
-      "source": [
-        "Create the `Task` object (`tfm.core.base_task.Task`) from the `config_definitions.TaskConfig`.\n",
-        "\n",
-        "The `Task` object has all the methods necessary for building the dataset, building the model, and running training & evaluation. These methods are driven by `tfm.core.train_lib.run_experiment`."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "6MgYSH0PtUaW"
-      },
-      "outputs": [],
-      "source": [
-        "with distribution_strategy.scope():\n",
-        "  model_dir = tempfile.mkdtemp()\n",
-        "  task = tfm.core.task_factory.get_task(exp_config.task, logging_dir=model_dir)\n",
-        "\n",
-        "tf.keras.utils.plot_model(task.build_model(), show_shapes=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "IFXEZYdzBKoX"
-      },
-      "outputs": [],
-      "source": [
-        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
-        "  print()\n",
-        "  print(f'images.shape: {str(images.shape):16}  images.dtype: {images.dtype!r}')\n",
-        "  print(f'labels.shape: {str(labels.shape):16}  labels.dtype: {labels.dtype!r}')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "yrwxnGDaRU0U"
-      },
-      "source": [
-        "## Visualize the training data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "683c255c6c52"
-      },
-      "source": [
-        "The dataloader applies a z-score normalization using \n",
-        "`preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`, so the images returned by the dataset can't be directly displayed by standard tools. The visualization code needs to rescale the data into the [0,1] range."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "PdmOz2EC0Nx2"
-      },
-      "outputs": [],
-      "source": [
-        "plt.hist(images.numpy().flatten());"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "7a8582ebde7b"
-      },
-      "source": [
-        "Use `ds_info` (which is an instance of `tfds.core.DatasetInfo`) to lookup the text descriptions of each class ID."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "Wq4Wq_CuDG3Q"
-      },
-      "outputs": [],
-      "source": [
-        "label_info = ds_info.features['label']\n",
-        "label_info.int2str(1)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "8c652a6fdbcf"
-      },
-      "source": [
-        "Visualize a batch of the data."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "ZKfTxytf1l0d"
-      },
-      "outputs": [],
-      "source": [
-        "def show_batch(images, labels, predictions=None):\n",
-        "  plt.figure(figsize=(10, 10))\n",
-        "  min = images.numpy().min()\n",
-        "  max = images.numpy().max()\n",
-        "  delta = max - min\n",
-        "\n",
-        "  for i in range(12):\n",
-        "    plt.subplot(6, 6, i + 1)\n",
-        "    plt.imshow((images[i]-min) / delta)\n",
-        "    if predictions is None:\n",
-        "      plt.title(label_info.int2str(labels[i]))\n",
-        "    else:\n",
-        "      if labels[i] == predictions[i]:\n",
-        "        color = 'g'\n",
-        "      else:\n",
-        "        color = 'r'\n",
-        "      plt.title(label_info.int2str(predictions[i]), color=color)\n",
-        "    plt.axis(\"off\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "xkA5h_RBtYYU"
-      },
-      "outputs": [],
-      "source": [
-        "plt.figure(figsize=(10, 10))\n",
-        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
-        "  show_batch(images, labels)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "v_A9VnL2RbXP"
-      },
-      "source": [
-        "## Visualize the testing data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "AXovuumW_I2z"
-      },
-      "source": [
-        "Visualize a batch of images from the validation dataset."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "Ma-_Eb-nte9A"
-      },
-      "outputs": [],
-      "source": [
-        "plt.figure(figsize=(10, 10));\n",
-        "for images, labels in task.build_inputs(exp_config.task.validation_data).take(1):\n",
-        "  show_batch(images, labels)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "ihKJt2FHRi2N"
-      },
-      "source": [
-        "## Train and evaluate"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "0AFMNvYxtjXx"
-      },
-      "outputs": [],
-      "source": [
-        "model, eval_logs = tfm.core.train_lib.run_experiment(\n",
-        "    distribution_strategy=distribution_strategy,\n",
-        "    task=task,\n",
-        "    mode='train_and_eval',\n",
-        "    params=exp_config,\n",
-        "    model_dir=model_dir,\n",
-        "    run_post_eval=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "gCcHMQYhozmA"
-      },
-      "outputs": [],
-      "source": [
-        "tf.keras.utils.plot_model(model, show_shapes=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "L7nVfxlBA8Gb"
-      },
-      "source": [
-        "Print the `accuracy`, `top_5_accuracy`, and `validation_loss` evaluation metrics."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "0124f938a1b9"
-      },
-      "outputs": [],
-      "source": [
-        "for key, value in eval_logs.items():\n",
-        "    print(f'{key:20}: {value.numpy():.3f}')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "TDys5bZ1zsml"
-      },
-      "source": [
-        "Run a batch of the processed training data through the model, and view the results"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "GhI7zR-Uz1JT"
-      },
-      "outputs": [],
-      "source": [
-        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
-        "  predictions = model.predict(images)\n",
-        "  predictions = tf.argmax(predictions, axis=-1)\n",
-        "\n",
-        "show_batch(images, labels, tf.cast(predictions, tf.int32))\n",
-        "\n",
-        "if device=='CPU':\n",
-        "  plt.suptitle('The model was only trained for a few steps, it is not expected to do well.')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "fkE9locGTBgt"
-      },
-      "source": [
-        "## Export a SavedModel"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "9669d08c91af"
-      },
-      "source": [
-        "The `keras.Model` object returned by `train_lib.run_experiment` expects the data to be normalized by the dataset loader using the same mean and variance statiscics in `preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`. This export function handles those details, so you can pass `tf.uint8` images and get the correct results.\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "AQCFa7BvtmDg"
-      },
-      "outputs": [],
-      "source": [
-        "# Saving and exporting the trained model\n",
-        "export_saved_model_lib.export_inference_graph(\n",
-        "    input_type='image_tensor',\n",
-        "    batch_size=1,\n",
-        "    input_image_size=[32, 32],\n",
-        "    params=exp_config,\n",
-        "    checkpoint_path=tf.train.latest_checkpoint(model_dir),\n",
-        "    export_dir='./export/')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "vVr6DxNqTyLZ"
-      },
-      "source": [
-        "Test the exported model."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "gP7nOvrftsB0"
-      },
-      "outputs": [],
-      "source": [
-        "# Importing SavedModel\n",
-        "imported = tf.saved_model.load('./export/')\n",
-        "model_fn = imported.signatures['serving_default']"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "GiOp2WVIUNUZ"
-      },
-      "source": [
-        "Visualize the predictions."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "BTRMrZQAN4mk"
-      },
-      "outputs": [],
-      "source": [
-        "plt.figure(figsize=(10, 10))\n",
-        "for data in tfds.load('cifar10', split='test').batch(12).take(1):\n",
-        "  predictions = []\n",
-        "  for image in data['image']:\n",
-        "    index = tf.argmax(model_fn(image[tf.newaxis, ...])['logits'], axis=1)[0]\n",
-        "    predictions.append(index)\n",
-        "  show_batch(data['image'], data['label'], predictions)\n",
-        "\n",
-        "  if device=='CPU':\n",
-        "    plt.suptitle('The model was only trained for a few steps, it is not expected to do better than random.')"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "collapsed_sections": [],
-      "name": "classification_with_model_garden.ipynb",
-      "toc_visible": true
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

From 92ab1d8ab4f2fd6ff00fea1cc76b6d211d94c478 Mon Sep 17 00:00:00 2001
From: chunduriv <74177924+chunduriv@users.noreply.github.com>
Date: Fri, 24 Jun 2022 16:51:51 +0530
Subject: [PATCH 192/872] Typo fix

Typo fix `overriden` to `overridden` & `he` to `the`
---
 site/en/guide/extension_type.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/extension_type.ipynb b/site/en/guide/extension_type.ipynb
index bb1f7e5b29d..123ce307c75 100644
--- a/site/en/guide/extension_type.ipynb
+++ b/site/en/guide/extension_type.ipynb
@@ -407,9 +407,9 @@
       "source": [
         "### Validation method\n",
         "\n",
-        "`ExtensionType` adds a `__validate__` method, which can be overriden to perform validation checks on fields.  It is run after the constructor is called, and after fields have been type-checked and converted to their declared types, so it can assume that all fields have their declared types.\n",
+        "`ExtensionType` adds a `__validate__` method, which can be overridden to perform validation checks on fields.  It is run after the constructor is called, and after fields have been type-checked and converted to their declared types, so it can assume that all fields have their declared types.\n",
         "\n",
-        "he following example updates `MaskedTensor` to validate the `shape`s and `dtype`s of its fields:"
+        "The following example updates `MaskedTensor` to validate the `shape`s and `dtype`s of its fields:"
       ]
     },
     {

From efc735f937ac81527174ebed14eb340377f94323 Mon Sep 17 00:00:00 2001
From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com>
Date: Fri, 24 Jun 2022 23:13:23 +0530
Subject: [PATCH 193/872] Updated correct URL link for tf.data.dataset

I have updated the tf.data.dataset link with the correct URL path(https://www.tensorflow.org/api_docs/python/tf/data/Dataset). Earlier this link was routing to tensorflow datasets page.
---
 site/en/tutorials/audio/music_generation.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/audio/music_generation.ipynb b/site/en/tutorials/audio/music_generation.ipynb
index 52e0b96a0b9..133eaa4b0b4 100644
--- a/site/en/tutorials/audio/music_generation.ipynb
+++ b/site/en/tutorials/audio/music_generation.ipynb
@@ -680,7 +680,7 @@
         "id": "xIBLvj-cODWS"
       },
       "source": [
-        "Next, create a [tf.data.Dataset](https://www.tensorflow.org/datasets) from the parsed notes."
+        "Next, create a [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) from the parsed notes."
       ]
     },
     {

From 7ca9ca90112a886f7a51581aed597e467759f6a8 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sat, 25 Jun 2022 20:06:44 +1000
Subject: [PATCH 194/872] Format tf.data.Dataset in Music generation tutorial

---
 site/en/tutorials/audio/music_generation.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/audio/music_generation.ipynb b/site/en/tutorials/audio/music_generation.ipynb
index 133eaa4b0b4..43a6e02e09a 100644
--- a/site/en/tutorials/audio/music_generation.ipynb
+++ b/site/en/tutorials/audio/music_generation.ipynb
@@ -680,7 +680,7 @@
         "id": "xIBLvj-cODWS"
       },
       "source": [
-        "Next, create a [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) from the parsed notes."
+        "Next, create a `tf.data.Dataset` from the parsed notes."
       ]
     },
     {

From 42b39eb246178ef43277f2a99edce4adfb5dd7a2 Mon Sep 17 00:00:00 2001
From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com>
Date: Sat, 25 Jun 2022 18:09:07 +0530
Subject: [PATCH 195/872] Updated broken link for TF Lite

Updated broken link for Style transfer using TF Lite.
---
 site/en/tutorials/generative/style_transfer.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/generative/style_transfer.ipynb b/site/en/tutorials/generative/style_transfer.ipynb
index ef8590b5afa..96a7005a21e 100644
--- a/site/en/tutorials/generative/style_transfer.ipynb
+++ b/site/en/tutorials/generative/style_transfer.ipynb
@@ -75,7 +75,7 @@
         "\n",
         "Note: This tutorial demonstrates the original style-transfer algorithm. It optimizes the image content to a particular style. Modern approaches train a model to generate the stylized image directly (similar to [CycleGAN](./cyclegan.ipynb)). This approach is much faster (up to 1000x).\n",
         "\n",
-        "For a simple application of style transfer check out this [tutorial](https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization) to learn more about how to use the pretrained [Arbitrary Image Stylization model](https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2) from [TensorFlow Hub](https://tfhub.dev) or how to use a style transfer model with [TensorFlow Lite](https://www.tensorflow.org/lite/models/style_transfer/overview). "
+        "For a simple application of style transfer check out this [tutorial](https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization) to learn more about how to use the pretrained [Arbitrary Image Stylization model](https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2) from [TensorFlow Hub](https://tfhub.dev) or how to use a style transfer model with [TensorFlow Lite](https://www.tensorflow.org/lite/examples/style_transfer/overview). "
       ]
     },
     {

From fb97816044bd4a26471fc0a598dfa00f4de02371 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sun, 26 Jun 2022 13:40:22 +0100
Subject: [PATCH 196/872] Split style transfer example sentence into smaller
 ones in Style transfer tutorial

---
 site/en/tutorials/generative/style_transfer.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/generative/style_transfer.ipynb b/site/en/tutorials/generative/style_transfer.ipynb
index 96a7005a21e..1bb4309e918 100644
--- a/site/en/tutorials/generative/style_transfer.ipynb
+++ b/site/en/tutorials/generative/style_transfer.ipynb
@@ -75,7 +75,7 @@
         "\n",
         "Note: This tutorial demonstrates the original style-transfer algorithm. It optimizes the image content to a particular style. Modern approaches train a model to generate the stylized image directly (similar to [CycleGAN](./cyclegan.ipynb)). This approach is much faster (up to 1000x).\n",
         "\n",
-        "For a simple application of style transfer check out this [tutorial](https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization) to learn more about how to use the pretrained [Arbitrary Image Stylization model](https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2) from [TensorFlow Hub](https://tfhub.dev) or how to use a style transfer model with [TensorFlow Lite](https://www.tensorflow.org/lite/examples/style_transfer/overview). "
+        "For a simple application of style transfer with a pretrained model from [TensorFlow Hub](https://tfhub.dev), check out the [Fast style transfer for arbitrary styles](https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization) tutorial that uses an [arbitrary image stylization model](https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2). For an example of style transfer with [TensorFlow Lite](https://www.tensorflow.org/lite), refer to [Artistic style transfer with TensorFlow Lite](https://www.tensorflow.org/lite/examples/style_transfer/overview)."
       ]
     },
     {

From 1d42f1bbd476adf2d5e3d28fd5820a3352a725f1 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sun, 26 Jun 2022 13:54:16 +0100
Subject: [PATCH 197/872] Lint the Generate music with an RNN tutorial

---
 site/en/tutorials/audio/music_generation.ipynb | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/site/en/tutorials/audio/music_generation.ipynb b/site/en/tutorials/audio/music_generation.ipynb
index 43a6e02e09a..1eee7809ca7 100644
--- a/site/en/tutorials/audio/music_generation.ipynb
+++ b/site/en/tutorials/audio/music_generation.ipynb
@@ -68,9 +68,9 @@
         "id": "hr78EkAY-FFg"
       },
       "source": [
-        "This tutorial shows you how to generate musical notes using a simple RNN. You will train a model using a collection of piano MIDI files from the [MAESTRO dataset](https://magenta.tensorflow.org/datasets/maestro). Given a sequence of notes, your model will learn to predict the next note in the sequence. You can generate a longer sequences of notes by calling the model repeatedly.\n",
+        "This tutorial shows you how to generate musical notes using a simple recurrent neural network (RNN). You will train a model using a collection of piano MIDI files from the [MAESTRO dataset](https://magenta.tensorflow.org/datasets/maestro). Given a sequence of notes, your model will learn to predict the next note in the sequence. You can generate longer sequences of notes by calling the model repeatedly.\n",
         "\n",
-        "This tutorial contains complete code to parse and create MIDI files. You can learn more about how RNNs work by visiting [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation)."
+        "This tutorial contains complete code to parse and create MIDI files. You can learn more about how RNNs work by visiting the [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation) tutorial."
       ]
     },
     {
@@ -713,7 +713,7 @@
         "id": "Sj9SXRCjt3I7"
       },
       "source": [
-        "You will train the model on batches of sequences of notes. Each example will consist of a sequence of notes as the input features, and next note as the label. In this way, the model will be trained to predict the next note in a sequence. You can find a diagram explaining this process (and more details) in [Text classification with an RNN](https://www.tensorflow.org/text/tutorials/text_generation).\n",
+        "You will train the model on batches of sequences of notes. Each example will consist of a sequence of notes as the input features, and the next note as the label. In this way, the model will be trained to predict the next note in a sequence. You can find a diagram describing this process (and more details) in [Text classification with an RNN](https://www.tensorflow.org/text/tutorials/text_generation).\n",
         "\n",
         "You can use the handy [window](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#window) function with size `seq_length` to create the features and labels in this format."
       ]
@@ -1056,7 +1056,7 @@
       "source": [
         "To use the model to generate notes, you will first need to provide a starting sequence of notes. The function below generates one note from a sequence of notes. \n",
         "\n",
-        "For note pitch, it draws a sample from softmax distribution of notes produced by the model, and does not simply pick the note with the highest probability.\n",
+        "For note pitch, it draws a sample from the softmax distribution of notes produced by the model, and does not simply pick the note with the highest probability.\n",
         "Always picking the note with the highest probability would lead to repetitive sequences of notes being generated.\n",
         "\n",
         "The `temperature` parameter can be used to control the randomness of notes generated. You can find more details on temperature in [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation)."
@@ -1229,7 +1229,7 @@
       "source": [
         "In the above plots, you will notice the change in distribution of the note variables.\n",
         "Since there is a feedback loop between the model's outputs and inputs, the model tends to generate similar sequences of outputs to reduce the loss. \n",
-        "This is particularly relevant for `step` and `duration`, which has uses MSE loss.\n",
+        "This is particularly relevant for `step` and `duration`, which uses the MSE loss.\n",
         "For `pitch`, you can increase the randomness by increasing the `temperature` in `predict_next_note`.\n"
       ]
     },
@@ -1243,7 +1243,7 @@
         "\n",
         "This tutorial demonstrated the mechanics of using an RNN to generate sequences of notes from a dataset of MIDI files. To learn more, you can visit the closely related [Text generation with an RNN](https://www.tensorflow.org/text/tutorials/text_generation) tutorial, which contains additional diagrams and explanations. \n",
         "\n",
-        "An alternative to using RNNs for music generation is using GANs. Rather than generating audio, a GAN-based approach can generate a entire sequence in parallel. The Magenta team has done impressive work on this approach with [GANSynth](https://magenta.tensorflow.org/gansynth). You can also find many wonderful music and art projects and open-source code on [Magenta project website](https://magenta.tensorflow.org/)."
+        "One of the alternatives to using RNNs for music generation is using GANs. Rather than generating audio, a GAN-based approach can generate an entire sequence in parallel. The Magenta team has done impressive work on this approach with [GANSynth](https://magenta.tensorflow.org/gansynth). You can also find many wonderful music and art projects and open-source code on [Magenta project website](https://magenta.tensorflow.org/)."
       ]
     }
   ],

From 2fe4cca1374ae2fc99648ce35244dfa750a916d7 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sun, 26 Jun 2022 14:26:38 +0100
Subject: [PATCH 198/872] Lint Extension types guide

---
 site/en/guide/extension_type.ipynb | 164 ++++++++++++++---------------
 1 file changed, 82 insertions(+), 82 deletions(-)

diff --git a/site/en/guide/extension_type.ipynb b/site/en/guide/extension_type.ipynb
index 123ce307c75..542450945d1 100644
--- a/site/en/guide/extension_type.ipynb
+++ b/site/en/guide/extension_type.ipynb
@@ -87,7 +87,7 @@
       "source": [
         "## Extension types\n",
         "\n",
-        "User-defined types can make projects more readable, modular, maintainable. However, most TensorFlow APIs have very limited support for user-defined Python types. This includes both high-level APIs (such as [Keras](https://www.tensorflow.org/guide/keras/overview), [tf.function](https://www.tensorflow.org/guide/function), [tf.SavedModel](https://www.tensorflow.org/guide/saved_model)) and lower-level APIs (such as `tf.while_loop` and `tf.concat`).  TensorFlow **extension types** can be used to create user-defined object-oriented types that work seamlessly with TensorFlow's APIs. To create an extension type, simply define a Python class with `tf.experimental.ExtensionType` as its base, and use [type annotations](https://www.python.org/dev/peps/pep-0484/) to specify the type for each field."
+        "User-defined types can make projects more readable, modular, maintainable. However, most TensorFlow APIs have very limited support for user-defined Python types. This includes both high-level APIs (such as [Keras](https://www.tensorflow.org/guide/keras/overview), [tf.function](https://www.tensorflow.org/guide/function), [`tf.SavedModel`](https://www.tensorflow.org/guide/saved_model)) and lower-level APIs (such as `tf.while_loop` and `tf.concat`). TensorFlow **extension types** can be used to create user-defined object-oriented types that work seamlessly with TensorFlow's APIs. To create an extension type, simply define a Python class with `tf.experimental.ExtensionType` as its base, and use [type annotations](https://www.python.org/dev/peps/pep-0484/) to specify the type for each field."
       ]
     },
     {
@@ -121,7 +121,7 @@
         "id": "FiaNXPa7pNK-"
       },
       "source": [
-        "The `tf.experimental.ExtensionType` base class works similarly to [`typing.NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple) and [`@dataclasses.dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass) from the standard Python library.  In particular, it automatically adds a constructor and special methods (such as `__repr__` and `__eq__`) based on the field type annotations."
+        "The `tf.experimental.ExtensionType` base class works similarly to [`typing.NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple) and [`@dataclasses.dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass) from the standard Python library. In particular, it automatically adds a constructor and special methods (such as `__repr__` and `__eq__`) based on the field type annotations."
       ]
     },
     {
@@ -132,9 +132,9 @@
       "source": [
         "Typically, extension types tend to fall into one of two categories:\n",
         "\n",
-        "* ***Data structures***, which group together a collection of related values, and can provide useful operations based on those values.  Data structures may be fairly general (such as the `TensorGraph` example above); or they may be highly customized to a specific model.\n",
+        "* ***Data structures***, which group together a collection of related values, and can provide useful operations based on those values. Data structures may be fairly general (such as the `TensorGraph` example above); or they may be highly customized to a specific model.\n",
         "\n",
-        "* ***Tensor-like types***, which specialize or extend the concept of \"Tensor.\"  Types in this category have a `rank`, a `shape`, and usually a `dtype`; and it makes sense to use them with Tensor operations (such as `tf.stack`, `tf.add`, or `tf.matmul`).  `MaskedTensor` and `CSRSparseMatrix` are examples of tensor-like types."
+        "* ***Tensor-like types***, which specialize or extend the concept of \"Tensor.\" Types in this category have a `rank`, a `shape`, and usually a `dtype`; and it makes sense to use them with Tensor operations (such as `tf.stack`, `tf.add`, or `tf.matmul`). `MaskedTensor` and `CSRSparseMatrix` are examples of tensor-like types."
       ]
     },
     {
@@ -148,15 +148,15 @@
         "Extension types are supported by the following TensorFlow APIs:\n",
         "\n",
         "* **Keras**: Extension types can be used as inputs and outputs for Keras `Models` and `Layers`.\n",
-        "* **tf.data.Dataset**: Extension types can be included in `Datasets`, and returned by dataset `Iterators`.\n",
-        "* **Tensorflow hub**: Extension types can be used as inputs and outputs for `tf.hub` modules.\n",
+        "* **`tf.data.Dataset`**: Extension types can be included in `Datasets`, and returned by dataset `Iterators`.\n",
+        "* **TensorFlow Hub**: Extension types can be used as inputs and outputs for `tf.hub` modules.\n",
         "* **SavedModel**: Extension types can be used as inputs and outputs for `SavedModel` functions.\n",
-        "* **tf.function**: Extension types can be used as arguments and return values for functions wrapped with the `@tf.function` decorator.\n",
-        "* **while loops**: Extension types can be used as loop variables in `tf.while_loop`, and can be used as arguments and return values for the while-loop's body.\n",
-        "* **conditionals**: Extension types can be conditionally selected using `tf.cond` and `tf.case`.\n",
-        "* **py_function**: Extension types can be used as arguments and return values for the `func` argument to `tf.py_function`.\n",
-        "* **Tensor ops**: Extension types can be extended to support most TensorFlow ops that accept Tensor inputs (e.g., `tf.matmul`, `tf.gather`, and `tf.reduce_sum`). See the \"*Dispatch*\" section below for more information.\n",
-        "* **distribution strategy**: Extension types can be used as per-replica values.\n",
+        "* **`tf.function`**: Extension types can be used as arguments and return values for functions wrapped with the `@tf.function` decorator.\n",
+        "* **While loops**: Extension types can be used as loop variables in `tf.while_loop`, and can be used as arguments and return values for the while-loop's body.\n",
+        "* **Conditionals**: Extension types can be conditionally selected using `tf.cond` and `tf.case`.\n",
+        "* **`tf.py_function`**: Extension types can be used as arguments and return values for the `func` argument to `tf.py_function`.\n",
+        "* **Tensor ops**: Extension types can be extended to support most TensorFlow ops that accept Tensor inputs (such as `tf.matmul`, `tf.gather`, and `tf.reduce_sum`). Go to the \"*Dispatch*\" section below for more information.\n",
+        "* **Distribution strategy**: Extension types can be used as per-replica values.\n",
         "\n",
         "For more details, see the section on \"TensorFlow APIs that support ExtensionTypes\" below.\n"
       ]
@@ -178,7 +178,7 @@
       "source": [
         "### Field types\n",
         "\n",
-        "All fields (aka instance variables) must be declared, and a type annotation must be provided for each field.  The following type annotations are supported:\n",
+        "All fields—instance variables—must be declared, and a type annotation must be provided for each field. The following type annotations are supported:\n",
         "\n",
         "Type | Example\n",
         "---- | -------\n",
@@ -186,15 +186,15 @@
         "Python floats | `f: float`\n",
         "Python strings | `s: str`\n",
         "Python booleans | `b: bool`\n",
-        "Python None | `n: None`\n",
+        "Python `None` | `n: None`\n",
         "[Tensor shapes](https://www.tensorflow.org/api_docs/python/tf/TensorShape) | `shape: tf.TensorShape`\n",
-        "[Tensor dtypes](https://www.tensorflow.org/api_docs/python/tf/dtypes/DType) | `dtype: tf.DType`\n",
+        "[Tensor `dtype`s](https://www.tensorflow.org/api_docs/python/tf/dtypes/DType) | `dtype: tf.DType`\n",
         "[Tensors](https://www.tensorflow.org/api_docs/python/tf/Tensor) | `t: tf.Tensor`\n",
         "[Extension types](https://www.tensorflow.org/api_docs/python/tf/experimental/ExtensionType) | `mt: MyMaskedTensor`\n",
-        "[Ragged Tensors](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor) | `rt: tf.RaggedTensor`\n",
-        "[Sparse Tensors](https://www.tensorflow.org/api_docs/python/tf/sparse/SparseTensor) | `st: tf.SparseTensor`\n",
-        "[Indexed Slices](https://www.tensorflow.org/api_docs/python/tf/IndexedSlices) | `s: tf.IndexedSlices`\n",
-        "[Optional Tensors](https://www.tensorflow.org/api_docs/python/tf/experimental/Optional) | `o: tf.experimental.Optional`\n",
+        "[Ragged tensors](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor) | `rt: tf.RaggedTensor`\n",
+        "[Sparse tensors](https://www.tensorflow.org/api_docs/python/tf/sparse/SparseTensor) | `st: tf.SparseTensor`\n",
+        "[Indexed slices](https://www.tensorflow.org/api_docs/python/tf/IndexedSlices) | `s: tf.IndexedSlices`\n",
+        "[Optional tensors](https://www.tensorflow.org/api_docs/python/tf/experimental/Optional) | `o: tf.experimental.Optional`\n",
         "[Type unions](https://docs.python.org/3/library/typing.html#typing.Union) | `int_or_float: typing.Union[int, float]`\n",
         "[Tuples](https://docs.python.org/3/library/typing.html#typing.Tuple) | `params: typing.Tuple[int, float, tf.Tensor, int]`\n",
         "[Var-length tuples](https://docs.python.org/3/library/typing.html#typing.Tuple) | `lengths: typing.Tuple[int, ...]`\n",
@@ -210,8 +210,8 @@
       "source": [
         "### Mutability\n",
         "\n",
-        "Extension types are required to be immutable.  This ensures that they can be properly tracked by TensorFlow's graph-tracing mechanisms.\n",
-        "If you find yourself wanting to mutate an extension type value, consider  instead defining methods that transform values.  For example, rather than defining a `set_mask` method to mutate a `MaskedTensor`, you could define a `replace_mask` method that returns a new `MaskedTensor`:"
+        "Extension types are required to be immutable. This ensures that they can be properly tracked by TensorFlow's graph-tracing mechanisms.\n",
+        "If you find yourself wanting to mutate an extension type value, consider instead defining methods that transform values. For example, rather than defining a `set_mask` method to mutate a `MaskedTensor`, you could define a `replace_mask` method that returns a new `MaskedTensor`:"
       ]
     },
     {
@@ -249,7 +249,7 @@
         "* A nested `TypeSpec`.\n",
         "* Tensor API dispatch support.\n",
         "\n",
-        "See the \"Customizing ExtensionTypes\" section below for more information on customizing this functionality."
+        "Go to the \"Customizing `ExtensionType`s\" section below for more information on customizing this functionality."
       ]
     },
     {
@@ -259,7 +259,7 @@
       },
       "source": [
         "### Constructor\n",
-        "The constructor added by `ExtensionType` takes each field as a named argument (in the order they were listed in the class definition).  This constructor will type-check each parameter, and convert them where necessary.  In particular, `Tensor` fields are converted using `tf.convert_to_tensor`; `Tuple` fields are converted to `tuple`s; and `Mapping` fields are converted to immutable dicts."
+        "The constructor added by `ExtensionType` takes each field as a named argument (in the order they were listed in the class definition). This constructor will type-check each parameter, and convert them where necessary. In particular, `Tensor` fields are converted using `tf.convert_to_tensor`; `Tuple` fields are converted to `tuple`s; and `Mapping` fields are converted to immutable dicts."
       ]
     },
     {
@@ -279,7 +279,7 @@
         "                  mask=[[True, True, False], [True, False, True]])\n",
         "\n",
         "# Fields are type-checked and converted to the declared types.\n",
-        "# E.g., mt.values is converted to a Tensor.\n",
+        "# For example, `mt.values` is converted to a Tensor.\n",
         "print(mt.values)"
       ]
     },
@@ -372,7 +372,7 @@
       "source": [
         "### Equality operators\n",
         "\n",
-        "`ExtensionType` adds default equality operators (`__eq__` and `__ne__`) that consider two values equal if they have the same type and all their fields are equal.  Tensor fields are considered equal if they have the same shape and are elementwise equal for all elements."
+        "`ExtensionType` adds default equality operators (`__eq__` and `__ne__`) that consider two values equal if they have the same type and all their fields are equal. Tensor fields are considered equal if they have the same shape and are elementwise equal for all elements."
       ]
     },
     {
@@ -407,7 +407,7 @@
       "source": [
         "### Validation method\n",
         "\n",
-        "`ExtensionType` adds a `__validate__` method, which can be overridden to perform validation checks on fields.  It is run after the constructor is called, and after fields have been type-checked and converted to their declared types, so it can assume that all fields have their declared types.\n",
+        "`ExtensionType` adds a `__validate__` method, which can be overridden to perform validation checks on fields. It is run after the constructor is called, and after fields have been type-checked and converted to their declared types, so it can assume that all fields have their declared types.\n",
         "\n",
         "The following example updates `MaskedTensor` to validate the `shape`s and `dtype`s of its fields:"
       ]
@@ -438,7 +438,7 @@
       "outputs": [],
       "source": [
         "try:\n",
-        "  MaskedTensor([1, 2, 3], [0, 1, 0])  # wrong dtype for mask.\n",
+        "  MaskedTensor([1, 2, 3], [0, 1, 0])  # Wrong `dtype` for mask.\n",
         "except AssertionError as e:\n",
         "  print(f\"Got expected AssertionError: {e}\")"
       ]
@@ -531,7 +531,7 @@
         "\n",
         "Each `ExtensionType` class has a corresponding `TypeSpec` class, which is created automatically and stored as `.Spec`.\n",
         "\n",
-        "This class captures all the information from a value *except* for the values of any nested tensors.  In particular, the `TypeSpec` for a value is created by replacing any nested Tensor, ExtensionType, or CompositeTensor with its `TypeSpec`.\n"
+        "This class captures all the information from a value *except* for the values of any nested tensors. In particular, the `TypeSpec` for a value is created by replacing any nested Tensor, ExtensionType, or CompositeTensor with its `TypeSpec`.\n"
       ]
     },
     {
@@ -548,7 +548,7 @@
         "\n",
         "anne = Player(\"Anne\", {\"height\": 8.3, \"speed\": 28.1})\n",
         "anne_spec = tf.type_spec_from_value(anne)\n",
-        "print(anne_spec.name)  # Records dtype and shape, but not the string value.\n",
+        "print(anne_spec.name)  # Records `dtype` and `shape`, but not the string value.\n",
         "print(anne_spec.attributes)  # Records keys and TensorSpecs for values."
       ]
     },
@@ -652,13 +652,13 @@
         "id": "gX613uRk0qLz"
       },
       "source": [
-        "## Customizing ExtensionTypes\n",
+        "## Customizing `ExtensionType`s\n",
         "\n",
         "In addition to simply declaring fields and their types, extension types may:\n",
         "\n",
         "* Override the default printable representation (`__repr__`).\n",
         "* Define methods.\n",
-        "* Define classmethods and staticmethods.\n",
+        "* Define `classmethod`s and `staticmethod`s.\n",
         "* Define properties.\n",
         "* Override the default constructor (`__init__`).\n",
         "* Override the default equality operator (`__eq__`).\n",
@@ -675,7 +675,7 @@
       "source": [
         "### Overriding the default printable representation\n",
         "\n",
-        "You can override this default string conversion operator for extension types.  The following example updates the `MaskedTensor` class to generate a more readable string representation when values are printed in Eager mode."
+        "You can override this default string conversion operator for extension types. The following example updates the `MaskedTensor` class to generate a more readable string representation when values are printed in Eager mode."
       ]
     },
     {
@@ -719,7 +719,7 @@
       "source": [
         "### Defining methods\n",
         "\n",
-        "Extension types may define methods, just like any normal Python class.  For example, the `MaskedTensor` type could define a `with_default` method that returns a copy of `self` with masked values replaced by a given `default` value.  Methods may optionally be annotated with the `@tf.function` decorator."
+        "Extension types may define methods, just like any normal Python class. For example, the `MaskedTensor` type could define a `with_default` method that returns a copy of `self` with masked values replaced by a given `default` value. Methods may optionally be annotated with the `@tf.function` decorator."
       ]
     },
     {
@@ -746,9 +746,9 @@
         "id": "Qwd_gGKp9RP0"
       },
       "source": [
-        "### Defining classmethods and staticmethods\n",
+        "### Defining `classmethod`s and `staticmethod`s\n",
         "\n",
-        "Extension types may define methods using the `@classmethod` and `@staticmethod` decorators.  For example, the `MaskedTensor` type could define a factory method that masks any element with a given value:"
+        "Extension types may define methods using the `@classmethod` and `@staticmethod` decorators. For example, the `MaskedTensor` type could define a factory method that masks any element with a given value:"
       ]
     },
     {
@@ -781,7 +781,7 @@
       },
       "source": [
         "### Defining properties\n",
-        "Extension types may define properties using the `@property` decorator, just like any normal Python class.  For example, the `MaskedTensor` type could define a `dtype` property that's a shorthand for the dtype of the values:"
+        "Extension types may define properties using the `@property` decorator, just like any normal Python class. For example, the `MaskedTensor` type could define a `dtype` property that's a shorthand for the `dtype` of the values:"
       ]
     },
     {
@@ -811,7 +811,7 @@
       "source": [
         "### Overriding the default constructor\n",
         "\n",
-        "You can override the default constructor for extension types.  Custom constructors must set a value for every declared field; and after the custom constructor returns, all fields will be type-checked, and values will be converted as described above."
+        "You can override the default constructor for extension types. Custom constructors must set a value for every declared field; and after the custom constructor returns, all fields will be type-checked, and values will be converted as described above."
       ]
     },
     {
@@ -838,7 +838,7 @@
         "id": "qyQxMlwLFQt7"
       },
       "source": [
-        "Alternatively, you might consider leaving the default constructor as-is, but adding one or more factory methods.  E.g.:"
+        "Alternatively, you might consider leaving the default constructor as-is, but adding one or more factory methods. For example:"
       ]
     },
     {
@@ -868,7 +868,7 @@
       "source": [
         "### Overriding the default equality operator (`__eq__`)\n",
         "\n",
-        "You can override the default `__eq__` operator for extension types.  The follow example updates `MaskedTensor` to ignore masked elements when comparing for equality."
+        "You can override the default `__eq__` operator for extension types. The following example updates `MaskedTensor` to ignore masked elements when comparing for equality."
       ]
     },
     {
@@ -913,7 +913,7 @@
       "source": [
         "### Using forward references\n",
         "\n",
-        "If the type for a field has not been defined yet, you may use a string containing the name of the type instead.  In the following example, the string `\"Node\"` is used to annotate the `children` field because the `Node` type hasn't been (fully) defined yet.\n"
+        "If the type for a field has not been defined yet, you may use a string containing the name of the type instead. In the following example, the string `\"Node\"` is used to annotate the `children` field because the `Node` type hasn't been (fully) defined yet.\n"
       ]
     },
     {
@@ -939,7 +939,7 @@
       "source": [
         "### Defining subclasses\n",
         "\n",
-        "Extension types may be subclassed using the standard Python syntax.  Extension type subclasses may add new fields, methods, and properties; and may override the constructor, the printable representation, and the equality operator.  The following example defines a basic `TensorGraph` class that uses three `Tensor` fields to encode a set of edges between nodes.  It then defines a subclass that adds a `Tensor` field to record a \"feature value\" for each node.  The subclass also defines a method to propagage the feature values along the edges."
+        "Extension types may be subclassed using the standard Python syntax. Extension type subclasses may add new fields, methods, and properties; and may override the constructor, the printable representation, and the equality operator. The following example defines a basic `TensorGraph` class that uses three `Tensor` fields to encode a set of edges between nodes. It then defines a subclass that adds a `Tensor` field to record a \"feature value\" for each node. The subclass also defines a method to propagate the feature values along the edges."
       ]
     },
     {
@@ -981,7 +981,7 @@
       "source": [
         "### Defining private fields\n",
         "\n",
-        "An extension type's fields may be marked private by prefixing them with an underscore (following standard Python conventions).  This does not impact the way that TensorFlow treats the fields in any way; but simply serves as a signal to any users of the extension type that those fields are private.\n"
+        "An extension type's fields may be marked private by prefixing them with an underscore (following standard Python conventions). This does not impact the way that TensorFlow treats the fields in any way; but simply serves as a signal to any users of the extension type that those fields are private.\n"
       ]
     },
     {
@@ -990,15 +990,15 @@
         "id": "oMdH7ORqh8Pl"
       },
       "source": [
-        "### Customizing the ExtensionType's `TypeSpec`\n",
+        "### Customizing the `ExtensionType`'s `TypeSpec`\n",
         "\n",
-        "Each `ExtensionType` class has a corresponding `TypeSpec` class, which is created automatically and stored as `.Spec`.  For more information, see the section \"Nested TypeSpec\" above.\n",
+        "Each `ExtensionType` class has a corresponding `TypeSpec` class, which is created automatically and stored as `.Spec`. For more information, see the section \"Nested TypeSpec\" above.\n",
         "\n",
-        "To customize the `TypeSpec`, simply define your own nested class named `Spec`, and `ExtensionType` will use that as the basis for the automatically constructed `TypeSpec`.  You can customize the `Spec` class by:\n",
+        "To customize the `TypeSpec`, simply define your own nested class named `Spec`, and `ExtensionType` will use that as the basis for the automatically constructed `TypeSpec`. You can customize the `Spec` class by:\n",
         "\n",
         "* Overriding the default printable representation.\n",
         "* Overriding the default constructor.\n",
-        "* Defining methods, classmethods, staticmethods, and properties.\n",
+        "* Defining methods, `classmethod`s, `staticmethod`s, and properties.\n",
         "\n",
         "The following example customizes the `MaskedTensor.Spec` class to make it easier to use:"
       ]
@@ -1053,7 +1053,7 @@
       "source": [
         "## Tensor API dispatch\n",
         "\n",
-        "Extension types can be \"tensor-like\", in the sense that they specialize or extend the interface defined by the `tf.Tensor` type.  Examples of tensor-like extension types include `RaggedTensor`, `SparseTensor`, and `MaskedTensor`.  ***Dispatch decorators*** can be used to override the default behavior of TensorFlow operations when applied to tensor-like extension types.  TensorFlow currently defines three dispatch decorators:\n",
+        "Extension types can be \"tensor-like\", in the sense that they specialize or extend the interface defined by the `tf.Tensor` type. Examples of tensor-like extension types include `RaggedTensor`, `SparseTensor`, and `MaskedTensor`. ***Dispatch decorators*** can be used to override the default behavior of TensorFlow operations when applied to tensor-like extension types. TensorFlow currently defines three dispatch decorators:\n",
         "\n",
         "* `@tf.experimental.dispatch_for_api(tf_api)`\n",
         "* `@tf.experimental.dispatch_for_unary_elementwise_api(x_type)`\n",
@@ -1068,7 +1068,7 @@
       "source": [
         "### Dispatch for a single API\n",
         "\n",
-        "The `tf.experimental.dispatch_for_api` decorator overrides the default behavior of a specified TensorFlow operation when it is called with the specified signature.  For example, you can use this decorator to specify how `tf.stack` should process `MaskedTensor` values:"
+        "The `tf.experimental.dispatch_for_api` decorator overrides the default behavior of a specified TensorFlow operation when it is called with the specified signature. For example, you can use this decorator to specify how `tf.stack` should process `MaskedTensor` values:"
       ]
     },
     {
@@ -1159,9 +1159,9 @@
       "source": [
         "### Dispatch for all unary elementwise APIs\n",
         "\n",
-        "The `tf.experimental.dispatch_for_unary_elementwise_apis` decorator overrides the default behavior of ***all*** unary elementwise ops (such as `tf.math.cos`) whenever the value for the first argument (typically named `x`) matches the type annotation `x_type`.  The decorated function should take two arguments:\n",
+        "The `tf.experimental.dispatch_for_unary_elementwise_apis` decorator overrides the default behavior of ***all*** unary elementwise ops (such as `tf.math.cos`) whenever the value for the first argument (typically named `x`) matches the type annotation `x_type`. The decorated function should take two arguments:\n",
         "\n",
-        "* `api_func`: A function that takes a single parameter and performs the elementwise operation (e.g., `tf.abs`).\n",
+        "* `api_func`: A function that takes a single parameter and performs the elementwise operation (for example, `tf.abs`).\n",
         "* `x`: The first argument to the elementwise operation.\n",
         "\n",
         "The following example updates all unary elementwise operations to handle the `MaskedTensor` type:"
@@ -1255,7 +1255,7 @@
         "id": "txTGg9pzG0Ux"
       },
       "source": [
-        "For a list of the elementwise APIs that are overridden, see the API documentation for `tf.experimental.dispatch_for_unary_elementwise_apis` and `tf.experimental.dispatch_for_binary_elementwise_apis`."
+        "For a list of the elementwise APIs that are overridden, go to the API documentation for `tf.experimental.dispatch_for_unary_elementwise_apis` and `tf.experimental.dispatch_for_binary_elementwise_apis`."
       ]
     },
     {
@@ -1264,9 +1264,9 @@
         "id": "UseRtohYKiE5"
       },
       "source": [
-        "## Batchable ExtensionTypes\n",
+        "## Batchable `ExtensionType`s\n",
         "\n",
-        "An `ExtensionType` is *batchable* if a single instance can be used to represent a batch of values.  Typically, this is accomplished by adding batch dimensions to all nested `Tensor`s.  The following TensorFlow APIs require that any extension type inputs be batchable:\n",
+        "An `ExtensionType` is *batchable* if a single instance can be used to represent a batch of values. Typically, this is accomplished by adding batch dimensions to all nested `Tensor`s. The following TensorFlow APIs require that any extension type inputs be batchable:\n",
         "\n",
         "* `tf.data.Dataset` (`batch`, `unbatch`, `from_tensor_slices`)\n",
         "* `tf.Keras` (`fit`, `evaluate`, `predict`)\n",
@@ -1279,10 +1279,10 @@
         "id": "hWPauKGj_yRz"
       },
       "source": [
-        "By default, `BatchableExtensionType` creates batched values by batching any nested `Tensor`s, `CompositeTensor`s, and `ExtensionType`s.  If this is not appropriate for your class, then you will need to use `tf.experimental.ExtensionTypeBatchEncoder` to override this default behavior.  For example, it would not be appropriate to create a batch of `tf.SparseTensor` values by simply stacking individual sparse tensors' `values`, `indices`, and `dense_shape` fields -- in most cases, you can't stack these tensors, since they have incompatible shapes; and even if you could, the result would not be a valid `SparseTensor`.\n",
+        "By default, `BatchableExtensionType` creates batched values by batching any nested `Tensor`s, `CompositeTensor`s, and `ExtensionType`s. If this is not appropriate for your class, then you will need to use `tf.experimental.ExtensionTypeBatchEncoder` to override this default behavior. For example, it would not be appropriate to create a batch of `tf.SparseTensor` values by simply stacking individual sparse tensors' `values`, `indices`, and `dense_shape` fields -- in most cases, you can't stack these tensors, since they have incompatible shapes; and even if you could, the result would not be a valid `SparseTensor`.\n",
         "\n",
         "\n",
-        "**Note**: `BatchableExtensionType`s do *not* automatically define dispatchers for `tf.stack`, `tf.concat`, `tf.slice`, etc.  If your class needs to be supported by these APIs, then use the dispatch decorators described above."
+        "**Note**: `BatchableExtensionType`s do *not* automatically define dispatchers for `tf.stack`, `tf.concat`, `tf.slice`, etc. If your class needs to be supported by these APIs, then use the dispatch decorators described above."
       ]
     },
     {
@@ -1291,7 +1291,7 @@
         "id": "xkOJ8ke8GH7s"
       },
       "source": [
-        "### BatchableExtensionType example: Network\n",
+        "### `BatchableExtensionType` example: `Network`\n",
         "As an example, consider a simple `Network` class used for load balancing, which tracks how much work is left to do at each node, and how much bandwidth is available to move work between nodes:"
       ]
     },
@@ -1317,7 +1317,7 @@
         "id": "PaOzUev6g3wT"
       },
       "source": [
-        "To make this type batchable, change the base type to `BatchableExtensionType`, and adjust the shape of each field to include optional batch dimensions.  The following example also adds a `shape` field to keept track of the batch shape.  This `shape` field is not required by `tf.data.Dataset` or `tf.map_fn`, but it *is* required by `tf.Keras`."
+        "To make this type batchable, change the base type to `BatchableExtensionType`, and adjust the shape of each field to include optional batch dimensions. The following example also adds a `shape` field to keep track of the batch shape. This `shape` field is not required by `tf.data.Dataset` or `tf.map_fn`, but it *is* required by `tf.Keras`."
       ]
     },
     {
@@ -1329,7 +1329,7 @@
       "outputs": [],
       "source": [
         "class Network(tf.experimental.BatchableExtensionType):\n",
-        "  shape: tf.TensorShape  # batch shape.  A single network has shape=[].\n",
+        "  shape: tf.TensorShape  # batch shape. A single network has shape=[].\n",
         "  work: tf.Tensor        # work[*shape, n] = work left to do at node n\n",
         "  bandwidth: tf.Tensor   # bandwidth[*shape, n1, n2] = bandwidth from n1->n2\n",
         "\n",
@@ -1426,7 +1426,7 @@
         "id": "f_HLsTT02Xul"
       },
       "source": [
-        "## TensorFlow APIs that support ExtensionTypes"
+        "## TensorFlow APIs that support `ExtensionType`s"
       ]
     },
     {
@@ -1437,7 +1437,7 @@
       "source": [
         "### @tf.function\n",
         "\n",
-        "[tf.function](https://www.tensorflow.org/guide/function) is a decorator that precomputes TensorFlow graphs for Python functions, which can substantially improve the performance of your TensorFlow code. Extension type values can be used transparently with `@tf.function`-decorated functions."
+        "[`tf.function`](https://www.tensorflow.org/guide/function) is a decorator that precomputes TensorFlow graphs for Python functions, which can substantially improve the performance of your TensorFlow code. Extension type values can be used transparently with `@tf.function`-decorated functions."
       ]
     },
     {
@@ -1532,7 +1532,7 @@
       },
       "outputs": [],
       "source": [
-        "# Example: using tf.cond to select between two MaskedTensors.  Note that the\n",
+        "# Example: using tf.cond to select between two MaskedTensors. Note that the\n",
         "# two MaskedTensors don't need to have the same shape.\n",
         "a = MaskedTensor([1., 2, 3], [True, False, True])\n",
         "b = MaskedTensor([22., 33, 108, 55], [True, True, True, False])\n",
@@ -1563,7 +1563,7 @@
       "source": [
         "### Autograph control flow\n",
         "\n",
-        "Extension types are also supported by control flow statements in tf.function (using autograph).  In the following example, the `if` statement and `for` statements are automatically converted to `tf.cond` and `tf.while_loop` operations, which support extension types."
+        "Extension types are also supported by control flow statements in `tf.function` (using autograph). In the following example, the `if` statement and `for` statements are automatically converted to `tf.cond` and `tf.while_loop` operations, which support extension types."
       ]
     },
     {
@@ -1596,10 +1596,10 @@
       "source": [
         "### Keras\n",
         "\n",
-        "[tf.keras](https://www.tensorflow.org/guide/keras) is TensorFlow's high-level API for building and training deep learning models. Extension types may be passed as inputs to a Keras model, passed between Keras layers, and returned by Keras models.  Keras currently puts two requirements on extension types:\n",
+        "[tf.keras](https://www.tensorflow.org/guide/keras) is TensorFlow's high-level API for building and training deep learning models. Extension types may be passed as inputs to a Keras model, passed between Keras layers, and returned by Keras models. Keras currently puts two requirements on extension types:\n",
         "\n",
-        "* They must be batchable (see \"Batchable ExtensionTypes\" above).\n",
-        "* The must have a field or property named `shape`. `shape[0]` is assumed to be the batch dimension.\n",
+        "* They must be batchable (go to \"Batchable `ExtensionType`s\" above).\n",
+        "* They must have a field or property named `shape`. `shape[0]` is assumed to be the batch dimension.\n",
         "\n",
         "The following two subsections give examples showing how extension types can be used with Keras.\n"
       ]
@@ -1612,7 +1612,7 @@
       "source": [
         "#### Keras example: `Network`\n",
         "\n",
-        "For the first example, consider the `Network` class defined in the \"Batchable ExtensionTypes\" section above, which can be used for load balancing work between nodes.  Its definition is repeated here:"
+        "For the first example, consider the `Network` class defined in the \"Batchable `ExtensionType`s\" section above, which can be used for load balancing work between nodes. Its definition is repeated here:"
       ]
     },
     {
@@ -1624,7 +1624,7 @@
       "outputs": [],
       "source": [
         "class Network(tf.experimental.BatchableExtensionType):\n",
-        "  shape: tf.TensorShape  # batch shape.  A single network has shape=[].\n",
+        "  shape: tf.TensorShape  # batch shape. A single network has shape=[].\n",
         "  work: tf.Tensor        # work[*shape, n] = work left to do at node n\n",
         "  bandwidth: tf.Tensor   # bandwidth[*shape, n1, n2] = bandwidth from n1->n2\n",
         "\n",
@@ -1647,7 +1647,7 @@
       },
       "outputs": [],
       "source": [
-        "single_network = Network(  # A single network w/ 4 nodes.\n",
+        "single_network = Network(  # A single network with 4 nodes.\n",
         "    work=[8.0, 5, 12, 2],\n",
         "    bandwidth=[[0.0, 1, 2, 2], [1, 0, 0, 2], [2, 0, 0, 1], [2, 2, 1, 0]])\n",
         "\n",
@@ -1679,7 +1679,7 @@
         "  Shifts work from more busy nodes to less busy nodes, constrained by bandwidth.\n",
         "  \"\"\"\n",
         "  def call(self, inputs):\n",
-        "    # This function is defined above, in \"Batchable ExtensionTypes\" section.\n",
+        "    # This function is defined above in the \"Batchable `ExtensionType`s\" section.\n",
         "    return balance_work_greedy(inputs)"
       ]
     },
@@ -1689,7 +1689,7 @@
         "id": "VWwFJNb1E03q"
       },
       "source": [
-        "You can then use this layers to create a simple model.  To feed an `ExtensionType` into a model, you can use a `tf.keras.layer.Input` layer with `type_spec` set to the extension type's `TypeSpec`.  If the Keras model will be used to process batches, then the `type_spec` must include the batch dimension."
+        "You can then use these layers to create a simple model. To feed an `ExtensionType` into a model, you can use a `tf.keras.layer.Input` layer with `type_spec` set to the extension type's `TypeSpec`. If the Keras model will be used to process batches, then the `type_spec` must include the batch dimension."
       ]
     },
     {
@@ -1748,7 +1748,7 @@
       "source": [
         "#### Keras example: MaskedTensor\n",
         "\n",
-        "In this example, `MaskedTensor` is extended to support `Keras`.  `shape` is defined as a property that is calculated from the `values` field.  Keras requires thatyou add this property to both the extension type and its `TypeSpec`.  `MaskedTensor` also defines a `__name__` variable, which will be required for `SavedModel` serialization (below)."
+        "In this example, `MaskedTensor` is extended to support `Keras`. `shape` is defined as a property that is calculated from the `values` field. Keras requires that you add this property to both the extension type and its `TypeSpec`. `MaskedTensor` also defines a `__name__` variable, which will be required for `SavedModel` serialization (below)."
       ]
     },
     {
@@ -1794,7 +1794,7 @@
         "id": "oer8BVc8H7_V"
       },
       "source": [
-        "Next, the dispatch decorators are used to override the default behavior of several TensorFlow APIs.  Since these APIs are used by standard Keras layers (such as the `Dense` layer), overriding these will allow us to use those layers with `MaskedTensor`.  For the purposes of this example, `matmul` for masked tensors is defined to treat the masked values as zeros (i.e., to not include them in the product)."
+        "Next, the dispatch decorators are used to override the default behavior of several TensorFlow APIs. Since these APIs are used by standard Keras layers (such as the `Dense` layer), overriding these will allow us to use those layers with `MaskedTensor`. For the purposes of this example, `matmul` for masked tensors is defined to treat the masked values as zeros (that is, to not include them in the product)."
       ]
     },
     {
@@ -1881,7 +1881,7 @@
         "\n",
         "A [SavedModel](https://www.tensorflow.org/guide/saved_model) is a serialized TensorFlow program, including both weights and computation. It can be built from a Keras model or from a custom model. In either case, extension types can be used transparently with the functions and methods defined by a SavedModel.\n",
         "\n",
-        "SavedModel can save models, layers, and functions that process extension types, as long as the extension types have a `__name__` field.  This name is used to register the extension type, so it can be located when the model is loaded."
+        "SavedModel can save models, layers, and functions that process extension types, as long as the extension types have a `__name__` field. This name is used to register the extension type, so it can be located when the model is loaded."
       ]
     },
     {
@@ -1954,9 +1954,9 @@
         "id": "o6beljh576ee"
       },
       "source": [
-        "#### Loading a SavedModel when the ExtensionType is unavailable\n",
+        "#### Loading a SavedModel when the `ExtensionType` is unavailable\n",
         "\n",
-        "If you load a `SavedModel` that uses an `ExtensionType`, but that `ExtensionType` is not available (i.e., has not been imported), then you will see a warning and TensorFlow will fall back to using an \"anonymous extension type\" object.  This object will have the same fields as the original type, but will lack any further customization you have added for the type, such as custom methods or properties."
+        "If you load a `SavedModel` that uses an `ExtensionType`, but that `ExtensionType` is not available (that is, it has not been imported), then you will get a warning and TensorFlow will fall back to using an \"anonymous extension type\" object. This object will have the same fields as the original type, but will lack any further customization you have added for the type, such as custom methods or properties."
       ]
     },
     {
@@ -1965,9 +1965,9 @@
         "id": "ec9PcUkJ9bFK"
       },
       "source": [
-        "#### Using ExtensionTypes with TensorFlow serving\n",
+        "#### Using `ExtensionType`s with TensorFlow Serving\n",
         "\n",
-        "Currently, [TensorFlow serving](https://www.tensorflow.org/tfx/guide/serving) (and other consumers of the SavedModel \"signatures\" dictionary) require that all inputs and outputs be raw tensors.  If you wish to use TensorFlow serving with a model that uses extension types, then you can add wrapper methods that compose or decompose extension type values from tensors.  E.g.:"
+        "Currently, [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving) (and other consumers of the SavedModel \"signatures\" dictionary) require that all inputs and outputs be raw tensors. If you wish to use TensorFlow Serving with a model that uses extension types, then you can add wrapper methods that compose or decompose extension type values from tensors. For example:"
       ]
     },
     {
@@ -2012,9 +2012,9 @@
         "id": "4dwBadWQ5G9_"
       },
       "source": [
-        "### Datasets\n",
+        "### `Dataset`s\n",
         "\n",
-        "[tf.data](https://www.tensorflow.org/guide/data) is an API that enables you to build complex input pipelines from simple, reusable pieces.  Its core data structure is `tf.data.Dataset`, which represents a sequence of elements, in which each element consists of one or more components."
+        "[`tf.data`](https://www.tensorflow.org/guide/data) is an API that enables you to build complex input pipelines from simple, reusable pieces. Its core data structure is `tf.data.Dataset`, which represents a sequence of elements, in which each element consists of one or more components."
       ]
     },
     {
@@ -2023,7 +2023,7 @@
         "id": "GcIR19FuwRJV"
       },
       "source": [
-        "#### Building Datasets with extension types\n",
+        "#### Building `Dataset`s with extension types\n",
         "\n",
         "Datasets can be built from extension type values using `Dataset.from_tensors`, `Dataset.from_tensor_slices`, or `Dataset.from_generator`:"
       ]
@@ -2078,9 +2078,9 @@
         "id": "wfEm4NInyqtj"
       },
       "source": [
-        "#### Batching and unbatching Datasets with extension types\n",
+        "#### Batching and unbatching `Dataset`s with extension types\n",
         "\n",
-        "Datasets with extension types can be batchand and unbatched using `Dataset.batch` adn `Dataset.unbatch`."
+        "Datasets with extension types can be batchand and unbatched using `Dataset.batch` and `Dataset.unbatch`."
       ]
     },
     {

From 6542fc8254f9b7791a1cf51f43ca3c0abada2dcc Mon Sep 17 00:00:00 2001
From: synandi <98147397+synandi@users.noreply.github.com>
Date: Mon, 27 Jun 2022 16:40:43 +0530
Subject: [PATCH 199/872] Fix typos

Spellings in some lines have been modified.
---
 site/en/tutorials/understanding/sngp.ipynb | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/site/en/tutorials/understanding/sngp.ipynb b/site/en/tutorials/understanding/sngp.ipynb
index 3df14d852ab..3f73a3502c9 100644
--- a/site/en/tutorials/understanding/sngp.ipynb
+++ b/site/en/tutorials/understanding/sngp.ipynb
@@ -70,7 +70,7 @@
       "source": [
         "In AI applications that are safety-critical (e.g., medical decision making and autonomous driving) or where the data is inherently noisy (e.g., natural language understanding), it is important for a deep classifier to reliably quantify its uncertainty. The deep classifier should be able to be aware of its own limitations and when it should hand control over to the human experts. This tutorial shows how to improve a deep classifier's ability in quantifying uncertainty using a technique called **Spectral-normalized Neural Gaussian Process ([SNGP](https://arxiv.org/abs/2006.10108))**.\n",
         "\n",
-        "The core idea of SNGP is to improve a deep classifier's _**distance awareness**_ by applying simple modifications to the network. A model's _distance awareness_ is a measure of how its predictive probability reflects the distance between the test example and the training data. This is a desirable property that is common for gold-standard probablistic models (e.g., the [Gaussian process](https://en.wikipedia.org/wiki/Gaussian_process) with RBF kernels) but is lacking in models with deep neural networks. SNGP provides a simple way to inject this Gaussian-process behavior into a deep classifier while maintaining its predictive accuracy.\n",
+        "The core idea of SNGP is to improve a deep classifier's _**distance awareness**_ by applying simple modifications to the network. A model's _distance awareness_ is a measure of how its predictive probability reflects the distance between the test example and the training data. This is a desirable property that is common for gold-standard probabilistic models (e.g., the [Gaussian process](https://en.wikipedia.org/wiki/Gaussian_process) with RBF kernels) but is lacking in models with deep neural networks. SNGP provides a simple way to inject this Gaussian-process behavior into a deep classifier while maintaining its predictive accuracy.\n",
         "\n",
         "This tutorial implements a deep residual network (ResNet)-based SNGP model on the [two moons](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html) dataset, and compares its uncertainty surface with that of two other popular uncertainty approaches - [Monte Carlo dropout](https://arxiv.org/abs/1506.02142) and [Deep ensemble](https://arxiv.org/abs/1612.01474)).\n",
         "\n",
@@ -220,7 +220,7 @@
         "id": "nqazrSzhd24R"
       },
       "source": [
-        "Create the trainining and evaluation datasets from the [two moon dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html)."
+        "Create the training and evaluation datasets from the [two moon dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html)."
       ]
     },
     {
@@ -834,7 +834,7 @@
         "id": "xgzw09gS03ae"
       },
       "source": [
-        "Note: For a deep neural network that is sensitive to the learning rate (e.g., ResNet-50 and ResNet-110), it is generally recommended to set `normalize_input=True` to stablize training, and set `scale_random_features=False` to avoid the learning rate from being modified in unexpected ways when passing through the GP layer."
+        "Note: For a deep neural network that is sensitive to the learning rate (e.g., ResNet-50 and ResNet-110), it is generally recommended to set `normalize_input=True` to stabilize training, and set `scale_random_features=False` to avoid the learning rate from being modified in unexpected ways when passing through the GP layer."
       ]
     },
     {
@@ -843,7 +843,7 @@
         "id": "pZkcKw-u7XRp"
       },
       "source": [
-        "* `gp_cov_momentum` controls how the model covariance is computed. If set to a positive value (e.g., 0.999), the covariance matrix is computed using the momentum-based moving average update (similar to batch normalization). If set to -1, the the covariance matrix is updated without momentum."
+        "* `gp_cov_momentum` controls how the model covariance is computed. If set to a positive value (e.g., 0.999), the covariance matrix is computed using the momentum-based moving average update (similar to batch normalization). If set to -1, the covariance matrix is updated without momentum."
       ]
     },
     {
@@ -852,7 +852,7 @@
         "id": "P13X7Adt-c2d"
       },
       "source": [
-        "Note: The momentum-based update method can be sensitive to batch size. Therefore it is generally recommended to set  `gp_cov_momentum=-1` to compute the covariance exactly. For this to work properly, the covariance matrix estimator needs to be reset at the begining of a new epoch in order to avoid counting the same data twice. For `RandomFeatureGaussianProcess`, this is can be done by calling its `reset_covariance_matrix()`. The next section shows an easy implementation of this using Keras' built-in API.\n"
+        "Note: The momentum-based update method can be sensitive to batch size. Therefore it is generally recommended to set  `gp_cov_momentum=-1` to compute the covariance exactly. For this to work properly, the covariance matrix estimator needs to be reset at the beginning of a new epoch in order to avoid counting the same data twice. For `RandomFeatureGaussianProcess`, this can be done by calling its `reset_covariance_matrix()`. The next section shows an easy implementation of this using Keras' built-in API.\n"
       ]
     },
     {

From 0acdd9a06e0fd5bd201017e30e5a1b16fb0aae29 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 27 Jun 2022 09:24:18 -0700
Subject: [PATCH 200/872] Add #anchors to more table entries.

I don't add anchors in methods because some class have lots of methods -> duplicate anchors. The method anchors are sufficient.

Also: Prefer tables. This catches a lot of blocks in the source that look like they should be formatted like Args/Returns/Raises but currently aren't. "Call Args" "References" "Input Shape" "Properties". One interesting side-efffect is that these TitleBlocks are basically an alternative Callout syntax:

```
title:

  indented whatever
```

PiperOrigin-RevId: 457497252
---
 tools/tensorflow_docs/api_generator/parser.py   | 17 +++++++++++------
 .../api_generator/pretty_docs/base_page.py      | 14 +++++++++-----
 .../api_generator/pretty_docs/class_page.py     |  4 +++-
 3 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/parser.py b/tools/tensorflow_docs/api_generator/parser.py
index 6ceeb6118be..159d962227f 100644
--- a/tools/tensorflow_docs/api_generator/parser.py
+++ b/tools/tensorflow_docs/api_generator/parser.py
@@ -307,11 +307,10 @@ def _dedent_after_first_line(self, text):
     result = '\n'.join([first, remainder])
     return result
 
-  def table_view(self, title_template: Optional[str] = None) -> str:
-    """Returns a tabular markdown version of the TitleBlock.
-
-    Tabular view is only for `Args`, `Returns`, `Raises` and `Attributes`. If
-    anything else is encountered, redirect to list view.
+  def table_view(self,
+                 title_template: Optional[str] = None,
+                 anchors: bool = True) -> str:
+    """Returns the TitleBlock as an HTML table.
 
     Args:
       title_template: Template for title detailing how to display it.
@@ -337,8 +336,14 @@ def table_view(self, title_template: Optional[str] = None) -> str:
       else:
         description = description.strip('\n')
         description = self._dedent_after_first_line(description)
+
+      if anchors:
+        anchor = f''
+      else:
+        anchor = ''
+
       item_table = ITEMS_TEMPLATE.format(
-          name=f'`{name}`', anchor='', description=description)
+          name=f'`{name}`', anchor=anchor, description=description)
       items.append(item_table)
 
     return '\n' + TABLE_TEMPLATE.format(
diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py
index 8db7b624113..5d051e0fcfb 100644
--- a/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py
+++ b/tools/tensorflow_docs/api_generator/pretty_docs/base_page.py
@@ -243,13 +243,14 @@ class MemberInfo(NamedTuple):
   url: str
 
 
-_TABLE_ITEMS = ('arg', 'return', 'raise', 'attr', 'yield')
+_ALWAYS_TABLE_ITEMS = ('arg', 'return', 'raise', 'attr', 'yield')
 
 
 def format_docstring(item,
                      *,
-                     table_title_template: Optional[str] = None) -> str:
-  """Formats TitleBlock into a table or list or a normal string.
+                     table_title_template: Optional[str] = None,
+                     anchors: bool = True) -> str:
+  """Formats a docstring part into a string.
 
   Args:
     item: A TitleBlock instance or a normal string.
@@ -261,8 +262,11 @@ def format_docstring(item,
   """
 
   if isinstance(item, parser.TitleBlock):
-    if item.title.lower().startswith(_TABLE_ITEMS):
-      return item.table_view(title_template=table_title_template)
+    if (item.items or  # A colon-list like under args
+        item.text.strip() or  # An indented block
+        item.title.lower().startswith(_ALWAYS_TABLE_ITEMS)):
+      return item.table_view(
+          title_template=table_title_template, anchors=anchors)
     else:
       return str(item)
   else:
diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py
index 909e026dba5..f29f5216c68 100644
--- a/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py
+++ b/tools/tensorflow_docs/api_generator/pretty_docs/class_page.py
@@ -639,7 +639,9 @@ def _build_method_section(method_info, heading_level=3):
   parts.append(base_page.build_top_compat(method_info, h_level=4))
 
   for item in method_info.doc.docstring_parts:
-    parts.append(base_page.format_docstring(item, table_title_template=None))
+    parts.append(
+        base_page.format_docstring(
+            item, table_title_template=None, anchors=False))
 
   parts.append(base_page.build_bottom_compat(method_info, h_level=4))
 

From 77ec394a53d519e4c19c0cf8d07e02f49b70d0f9 Mon Sep 17 00:00:00 2001
From: Liangchen Luo 
Date: Mon, 27 Jun 2022 18:33:23 -0700
Subject: [PATCH 201/872] Fix the signature of types with __origin__ attribute.

PiperOrigin-RevId: 457615379
---
 .../api_generator/pretty_docs/type_alias_page.py               | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py
index 0fdc6dd0f28..29f620dc813 100644
--- a/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py
+++ b/tools/tensorflow_docs/api_generator/pretty_docs/type_alias_page.py
@@ -139,7 +139,8 @@ def collect_docs(self) -> None:
 
     sig_args_str = textwrap.indent(',\n'.join(sig_args), '    ')
     if self.py_object.__origin__:
-      sig = f'{self.py_object.__origin__}[\n{sig_args_str}\n]'
+      origin_str = typing._type_repr(self.py_object.__origin__)  # pylint: disable=protected-access # pytype: disable=module-attr
+      sig = f'{origin_str}[\n{sig_args_str}\n]'
     else:
       sig = repr(self.py_object)
 

From 7314c6ef494dab27f97c5fb5da2e1319a154cb27 Mon Sep 17 00:00:00 2001
From: Patrice Vignola 
Date: Wed, 29 Jun 2022 13:34:14 -0700
Subject: [PATCH 202/872] Add tensorflow-directml-plugin to the gpu_plugins.md
 page

---
 site/en/install/gpu_plugins.md | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index 358db01b312..4763d423f5c 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -60,3 +60,9 @@ Metal `PluggableDevice` for macOS GPUs:
 *   [Getting started guide](https://developer.apple.com/metal/tensorflow-plugin/){:.external}.
 *   For questions and feedback, please visit the
     [Apple Developer Forum](https://developer.apple.com/forums/tags/tensorflow-metal){:.external}.
+
+DML `PluggableDevice` for Windows and WSL (preview):
+
+*   [PyPI Wheel](https://pypi.org/project/tensorflow-directml-plugin/){:.external}.
+*   [GitHub Repo](https://github.com/microsoft/tensorflow-directml-plugin){.external}.
+*   For questions, feedback or to raise issues, please visit the [issues page of the github repo](https://github.com/microsoft/tensorflow-directml-plugin/issues){:.external}.

From 8c3f6982e8336955c553b0d2f0bdd4d0dbee7c91 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 29 Jun 2022 13:37:11 -0700
Subject: [PATCH 203/872] Update link to point to the copy on tensorflow.org

PiperOrigin-RevId: 458051857
---
 site/en/tutorials/_toc.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml
index 5eb4825ca41..7c629d69dea 100644
--- a/site/en/tutorials/_toc.yaml
+++ b/site/en/tutorials/_toc.yaml
@@ -110,7 +110,7 @@ toc:
   - title: "Image segmentation"
     path: /tutorials/images/segmentation
   - title: "Object detection with TF Hub"
-    path: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_object_detection.ipynb
+    path: /hub/tutorials/tf2_object_detection
     status: external
 
 - title: "Text"

From 965c1ab28a11e12c9b9d2ca231b9ca099dacc80f Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 29 Jun 2022 14:49:31 -0700
Subject: [PATCH 204/872] DML -> DirectML

---
 site/en/install/gpu_plugins.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index 4763d423f5c..e15c219e19c 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -61,7 +61,7 @@ Metal `PluggableDevice` for macOS GPUs:
 *   For questions and feedback, please visit the
     [Apple Developer Forum](https://developer.apple.com/forums/tags/tensorflow-metal){:.external}.
 
-DML `PluggableDevice` for Windows and WSL (preview):
+DirectML `PluggableDevice` for Windows and WSL (preview):
 
 *   [PyPI Wheel](https://pypi.org/project/tensorflow-directml-plugin/){:.external}.
 *   [GitHub Repo](https://github.com/microsoft/tensorflow-directml-plugin){.external}.

From f9deb660ae698b9fa592e0161e42d6c4e6f444d8 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Wed, 29 Jun 2022 19:45:41 -0700
Subject: [PATCH 205/872] Fix MathJax on `integrated_gradients.ipynb`.

This equation was not rendering correctly on tensorflow.org. MathJax itself renders it fine (tested locally), but the single-dollar delimiter is for in-line equations, so can't be relied upon to work for a multi-line equation.

PiperOrigin-RevId: 458119384
---
 site/en/tutorials/interpretability/integrated_gradients.ipynb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/site/en/tutorials/interpretability/integrated_gradients.ipynb b/site/en/tutorials/interpretability/integrated_gradients.ipynb
index 2b7965db805..bd96fd9202b 100644
--- a/site/en/tutorials/interpretability/integrated_gradients.ipynb
+++ b/site/en/tutorials/interpretability/integrated_gradients.ipynb
@@ -770,8 +770,7 @@
         "id": "GshPZQgROs80"
       },
       "source": [
-        "$IntegratedGrads^{approx}_{i}(x)::=(x_{i}-x'_{i})\\times \\overbrace{\\sum_{k=1}^{m}}^\\text{Sum m local gradients}\n",
-        "\\text{gradients(interpolated images)} \\times \\overbrace{\\frac{1}{m}}^\\text{Divide by m steps}$\n",
+        "$IntegratedGrads^{approx}_{i}(x)::=(x_{i}-x'_{i})\\times \\overbrace{\\sum_{k=1}^{m}}^\\text{Sum m local gradients}\\text{gradients(interpolated images)} \\times \\overbrace{\\frac{1}{m}}^\\text{Divide by m steps}$\n",
         "\n",
         "From the equation, you can see you are summing over `m` gradients and dividing by `m` steps. You can implement the two operations together for part 3 as an *average of the local gradients of `m` interpolated predictions and input images*."
       ]

From a214d352e30419380c3d13b6e7b3d7f5e202e760 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 30 Jun 2022 09:49:54 +0100
Subject: [PATCH 206/872] Update/format GPU device plugins doc

---
 site/en/install/gpu_plugins.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index e15c219e19c..7b6b9eab5f7 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -63,6 +63,6 @@ Metal `PluggableDevice` for macOS GPUs:
 
 DirectML `PluggableDevice` for Windows and WSL (preview):
 
-*   [PyPI Wheel](https://pypi.org/project/tensorflow-directml-plugin/){:.external}.
-*   [GitHub Repo](https://github.com/microsoft/tensorflow-directml-plugin){.external}.
-*   For questions, feedback or to raise issues, please visit the [issues page of the github repo](https://github.com/microsoft/tensorflow-directml-plugin/issues){:.external}.
+*   [PyPI wheel](https://pypi.org/project/tensorflow-directml-plugin/){:.external}.
+*   [GitHub repo](https://github.com/microsoft/tensorflow-directml-plugin){.external}.
+*   For questions, feedback or to raise issues, please visit the [Issues page of `tensorflow-directml-plugin` on GitHub](https://github.com/microsoft/tensorflow-directml-plugin/issues){:.external}.

From 4406487d2dead14eedef7edbe3fd4c12b9d84069 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 30 Jun 2022 09:54:06 +0100
Subject: [PATCH 207/872] Format links in GPU device plugins doc

---
 site/en/install/gpu_plugins.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index 7b6b9eab5f7..446107b4b0e 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -1,11 +1,11 @@
 # GPU device plugins
 
-Note: This page is for non-NVIDIA® GPU devices. For NVIDIA® GPU support, click
-[here](./gpu.md).
+Note: This page is for non-NVIDIA® GPU devices. For NVIDIA® GPU support, go to
+the [Install TensorFlow with pip](./gpu.md) guide.
 
 TensorFlow's
-pluggable
-device architecture adds new device support as separate plug-in packages
+[pluggable device](https://github.com/tensorflow/community/blob/master/rfcs/20200624-pluggable-device-for-tensorflow.md){.external}
+architecture adds new device support as separate plug-in packages
 that are installed alongside the official TensorFlow package.
 
 The mechanism requires no device-specific changes in the TensorFlow code. It

From f914f300e749d43a45dc79fb1b4c1f63f256e75e Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 30 Jun 2022 10:38:18 +0100
Subject: [PATCH 208/872] Lint Spectral-normalized Neural Gaussian Process
 tutorial

---
 site/en/tutorials/understanding/sngp.ipynb | 98 +++++++++++-----------
 1 file changed, 49 insertions(+), 49 deletions(-)

diff --git a/site/en/tutorials/understanding/sngp.ipynb b/site/en/tutorials/understanding/sngp.ipynb
index 3f73a3502c9..e99ffe264f9 100644
--- a/site/en/tutorials/understanding/sngp.ipynb
+++ b/site/en/tutorials/understanding/sngp.ipynb
@@ -68,13 +68,13 @@
         "id": "UvW1QEMtP7Gy"
       },
       "source": [
-        "In AI applications that are safety-critical (e.g., medical decision making and autonomous driving) or where the data is inherently noisy (e.g., natural language understanding), it is important for a deep classifier to reliably quantify its uncertainty. The deep classifier should be able to be aware of its own limitations and when it should hand control over to the human experts. This tutorial shows how to improve a deep classifier's ability in quantifying uncertainty using a technique called **Spectral-normalized Neural Gaussian Process ([SNGP](https://arxiv.org/abs/2006.10108))**.\n",
+        "In AI applications that are safety-critical, such as medical decision making and autonomous driving, or where the data is inherently noisy (for example, natural language understanding), it is important for a deep classifier to reliably quantify its uncertainty. The deep classifier should be able to be aware of its own limitations and when it should hand control over to the human experts. This tutorial shows how to improve a deep classifier's ability in quantifying uncertainty using a technique called **Spectral-normalized Neural Gaussian Process ([SNGP](https://arxiv.org/abs/2006.10108){.external})**.\n",
         "\n",
-        "The core idea of SNGP is to improve a deep classifier's _**distance awareness**_ by applying simple modifications to the network. A model's _distance awareness_ is a measure of how its predictive probability reflects the distance between the test example and the training data. This is a desirable property that is common for gold-standard probabilistic models (e.g., the [Gaussian process](https://en.wikipedia.org/wiki/Gaussian_process) with RBF kernels) but is lacking in models with deep neural networks. SNGP provides a simple way to inject this Gaussian-process behavior into a deep classifier while maintaining its predictive accuracy.\n",
+        "The core idea of SNGP is to improve a deep classifier's _**distance awareness**_ by applying simple modifications to the network. A model's _distance awareness_ is a measure of how its predictive probability reflects the distance between the test example and the training data. This is a desirable property that is common for gold-standard probabilistic models (for example, the [Gaussian process](https://en.wikipedia.org/wiki/Gaussian_process){.external} with RBF kernels) but is lacking in models with deep neural networks. SNGP provides a simple way to inject this Gaussian-process behavior into a deep classifier while maintaining its predictive accuracy.\n",
         "\n",
-        "This tutorial implements a deep residual network (ResNet)-based SNGP model on the [two moons](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html) dataset, and compares its uncertainty surface with that of two other popular uncertainty approaches - [Monte Carlo dropout](https://arxiv.org/abs/1506.02142) and [Deep ensemble](https://arxiv.org/abs/1612.01474)).\n",
+        "This tutorial implements a deep residual network (ResNet)-based SNGP model on [scikit-learn’s two moons](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html){.external} dataset, and compares its uncertainty surface with that of two other popular uncertainty approaches: [Monte Carlo dropout](https://arxiv.org/abs/1506.02142){.external} and [Deep ensemble](https://arxiv.org/abs/1612.01474){.external}.\n",
         "\n",
-        "This tutorial illustrates the SNGP model on a toy 2D dataset. For an example of applying SNGP to a real-world natural language understanding task using  BERT-base, check out the [SNGP-BERT tutorial](https://www.tensorflow.org/text/tutorials/uncertainty_quantification_with_sngp_bert). For high-quality implementations of an SNGP model (and many other uncertainty methods) on a wide variety of benchmark datasets (such as [CIFAR-100](https://www.tensorflow.org/datasets/catalog/cifar100), [ImageNet](https://www.tensorflow.org/datasets/catalog/imagenet2012), [Jigsaw toxicity detection](https://www.tensorflow.org/datasets/catalog/wikipedia_toxicity_subtypes), etc), refer to the [Uncertainty Baselines](https://github.com/google/uncertainty-baselines) benchmark."
+        "This tutorial illustrates the SNGP model on a toy 2D dataset. For an example of applying SNGP to a real-world natural language understanding task using a BERT-base, check out the [SNGP-BERT tutorial](https://www.tensorflow.org/text/tutorials/uncertainty_quantification_with_sngp_bert). For high-quality implementations of an SNGP model (and many other uncertainty methods) on a wide variety of benchmark datasets (such as [CIFAR-100](https://www.tensorflow.org/datasets/catalog/cifar100), [ImageNet](https://www.tensorflow.org/datasets/catalog/imagenet2012), [Jigsaw toxicity detection](https://www.tensorflow.org/datasets/catalog/wikipedia_toxicity_subtypes), etc), refer to the [Uncertainty Baselines](https://github.com/google/uncertainty-baselines){.external} benchmark."
       ]
     },
     {
@@ -92,7 +92,7 @@
         "id": "ysyslHCyvYi-"
       },
       "source": [
-        "[Spectral-normalized Neural Gaussian Process](https://arxiv.org/abs/2006.10108)  (SNGP) is a simple approach to improve a deep classifier's uncertainty quality while maintaining a similar level of accuracy and latency. Given a deep residual network, SNGP makes two simple changes to the model:\n",
+        "SNGP is a simple approach to improve a deep classifier's uncertainty quality while maintaining a similar level of accuracy and latency. Given a deep residual network, SNGP makes two simple changes to the model:\n",
         "\n",
         "* It applies spectral normalization to the hidden residual layers.\n",
         "* It replaces the Dense output layer with a Gaussian process layer."
@@ -113,15 +113,15 @@
         "id": "2L88PoKr6XaE"
       },
       "source": [
-        "Compared to other uncertainty approaches (e.g., Monte Carlo dropout or Deep ensemble), SNGP has several advantages:\n",
+        "Compared to other uncertainty approaches (such as Monte Carlo dropout or Deep ensemble), SNGP has several advantages:\n",
         "\n",
-        "* It works for a wide range of state-of-the-art residual-based architectures (e.g., (Wide) ResNet, DenseNet, BERT, etc).\n",
-        "* It is a single-model method (i.e., does not rely on ensemble averaging). Therefore SNGP has a similar level of latency as a single deterministic network, and can be scaled easily to large datasets like [ImageNet](https://github.com/google/uncertainty-baselines/tree/main/baselines/imagenet) and [Jigsaw Toxic Comments classification](https://github.com/google/uncertainty-baselines/tree/main/baselines/toxic_comments).\n",
+        "* It works for a wide range of state-of-the-art residual-based architectures (for example, (Wide) ResNet, DenseNet, or BERT).\n",
+        "* It is a single-model method—it does not rely on ensemble averaging). Therefore, SNGP has a similar level of latency as a single deterministic network, and can be scaled easily to large datasets like [ImageNet](https://github.com/google/uncertainty-baselines/tree/main/baselines/imagenet){.external} and [Jigsaw Toxic Comments classification](https://github.com/google/uncertainty-baselines/tree/main/baselines/toxic_comments){.external}.\n",
         "* It has strong out-of-domain detection performance due to the _distance-awareness_ property.\n",
         "\n",
         "The downsides of this method are:\n",
         "\n",
-        "* The predictive uncertainty of a SNGP is computed using the [Laplace approximation](http://www.gaussianprocess.org/gpml/chapters/RW3.pdf). Therefore theoretically, the posterior uncertainty of SNGP is different from that of an exact Gaussian process.\n",
+        "* The predictive uncertainty of SNGP is computed using the [Laplace approximation](http://www.gaussianprocess.org/gpml/chapters/RW3.pdf){.external}. Therefore, theoretically, the posterior uncertainty of SNGP is different from that of an exact Gaussian process.\n",
         "\n",
         "* SNGP training needs a covariance reset step at the beginning of a new epoch. This can add a tiny amount of extra complexity to a training pipeline. This tutorial shows a simple way to implement this using Keras callbacks."
       ]
@@ -220,7 +220,7 @@
         "id": "nqazrSzhd24R"
       },
       "source": [
-        "Create the training and evaluation datasets from the [two moon dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html)."
+        "Create the training and evaluation datasets from the [scikit-learn two moon dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html){.external}."
       ]
     },
     {
@@ -275,7 +275,7 @@
         "id": "G9BYe4yqfeFa"
       },
       "source": [
-        "To evaluate model uncertainty, add an out-of-domain (OOD) dataset that belongs to a third class. The model never sees these OOD examples during training."
+        "To evaluate model uncertainty, add an out-of-domain (OOD) dataset that belongs to a third class. The model never observes these OOD examples during training."
       ]
     },
     {
@@ -329,7 +329,7 @@
         "id": "nlzxsnBBkybB"
       },
       "source": [
-        "Here the blue and orange represent the positive and negative classes, and the red represents the OOD data. A model that quantifies the uncertainty well is expected to be confident when close to training data  (i.e., $p(x_{test})$ close to 0 or 1), and be uncertain when far away from the training data regions  (i.e., $p(x_{test})$ close to 0.5)."
+        "Here, the blue and orange represent the positive and negative classes, and the red represents the OOD data. A model that quantifies the uncertainty well is expected to be confident when close to training data  (i.e., $p(x_{test})$ close to 0 or 1), and be uncertain when far away from the training data regions  (i.e., $p(x_{test})$ close to 0.5)."
       ]
     },
     {
@@ -391,7 +391,7 @@
         "    # Projects the 2d input data to high dimension.\n",
         "    hidden = self.input_layer(inputs)\n",
         "\n",
-        "    # Computes the resnet hidden representations.\n",
+        "    # Computes the ResNet hidden representations.\n",
         "    for i in range(self.num_layers):\n",
         "      resid = self.dense_layers[i](hidden)\n",
         "      resid = tf.keras.layers.Dropout(self.dropout_rate)(resid)\n",
@@ -415,7 +415,7 @@
         "id": "4u870GAen2aO"
       },
       "source": [
-        "This tutorial uses a 6-layer ResNet with 128 hidden units."
+        "This tutorial uses a six-layer ResNet with 128 hidden units."
       ]
     },
     {
@@ -548,11 +548,11 @@
         "  Arguments:\n",
         "    test_uncertainty: Array of uncertainty scores, shape (num_test,).\n",
         "    ax: A matplotlib Axes object that specifies a matplotlib figure.\n",
-        "    cmap: A matplotlib colormap object specifying the palette of the \n",
+        "    cmap: A matplotlib colormap object specifying the palette of the\n",
         "      predictive surface.\n",
         "\n",
         "  Returns:\n",
-        "    pcm: A matplotlib PathCollection object that contains the palette \n",
+        "    pcm: A matplotlib PathCollection object that contains the palette\n",
         "      information of the uncertainty plot.\n",
         "  \"\"\"\n",
         "  # Normalize uncertainty for better visualization.\n",
@@ -564,13 +564,13 @@
         "\n",
         "  # Plot normalized uncertainty surface.\n",
         "  pcm = ax.imshow(\n",
-        "      np.reshape(test_uncertainty, [DEFAULT_N_GRID, DEFAULT_N_GRID]), \n",
+        "      np.reshape(test_uncertainty, [DEFAULT_N_GRID, DEFAULT_N_GRID]),\n",
         "      cmap=cmap,\n",
         "      origin=\"lower\",\n",
         "      extent=DEFAULT_X_RANGE + DEFAULT_Y_RANGE,\n",
         "      vmin=DEFAULT_NORM.vmin,\n",
         "      vmax=DEFAULT_NORM.vmax,\n",
-        "      interpolation='bicubic', \n",
+        "      interpolation='bicubic',\n",
         "      aspect='auto')\n",
         "\n",
         "  # Plot training data.\n",
@@ -587,7 +587,7 @@
         "id": "a1age2y0339T"
       },
       "source": [
-        "Now visualize the predictions of the deterministic model. First plot the class probability: \n",
+        "Now visualize the predictions of the deterministic model. First plot the class probability:\n",
         "$$p(x) = softmax(logit(x))$$"
       ]
     },
@@ -627,7 +627,7 @@
         "id": "7ShGAB7FNYgU"
       },
       "source": [
-        "In this plot, the yellow and purple are the predictive probabilities for the two classes. The deterministic model did a good job in classifying the two known classes (blue and orange) with a nonlinear decision boundary. However, it is not **distance-aware**, and classified the never-seen red out-of-domain (OOD) examples confidently as the orange class.\n",
+        "In this plot, the yellow and purple are the predictive probabilities for the two classes. The deterministic model did a good job in classifying the two known classes—blue and orange—with a nonlinear decision boundary. However, it is not **distance-aware**, and classified the never-observed red out-of-domain (OOD) examples confidently as the orange class.\n",
         "\n",
         "Visualize the model uncertainty by computing the [predictive variance](https://en.wikipedia.org/wiki/Bernoulli_distribution#Variance):\n",
         "$$var(x) = p(x) * (1 - p(x))$$"
@@ -713,7 +713,7 @@
         "id": "rp2O2iv8LSke"
       },
       "source": [
-        "Let's look at these two components in more detail. (You can also jump to the [The SNGP model](#full-sngp-model) section to see how the full model is implemented.)"
+        "Let's inspect these two components in more detail. (You can also jump to [the full SNGP model](#full-sngp-model) section to learn how SNGP is implemented.)"
       ]
     },
     {
@@ -722,7 +722,7 @@
         "id": "5n4NIt3QjKwl"
       },
       "source": [
-        "#### Spectral Normalization wrapper"
+        "#### `SpectralNormalization` wrapper"
       ]
     },
     {
@@ -731,7 +731,7 @@
         "id": "tE-Va7J2jR2X"
       },
       "source": [
-        "[`SpectralNormalization`](https://github.com/tensorflow/models/blob/master/official/nlp/modeling/layers/spectral_normalization.py) is a Keras layer wrapper. It can be applied to an existing Dense layer like this:"
+        "[`SpectralNormalization`](https://github.com/tensorflow/models/blob/master/official/nlp/modeling/layers/spectral_normalization.py){.external} is a Keras layer wrapper. It can be applied to an existing Dense layer like this:"
       ]
     },
     {
@@ -752,7 +752,7 @@
         "id": "E9q25_6fRJRh"
       },
       "source": [
-        "Spectral normalization regularizes the hidden weight $W$ by gradually guiding its spectral norm (i.e., the largest eigenvalue of $W$) toward the target value  `norm_multiplier`. \n"
+        "Spectral normalization regularizes the hidden weight $W$ by gradually guiding its spectral norm (that is, the largest eigenvalue of $W$) toward the target value `norm_multiplier`).\n"
       ]
     },
     {
@@ -779,11 +779,11 @@
         "id": "7rYfIgtrjHnB"
       },
       "source": [
-        "[`RandomFeatureGaussianProcess`](https://github.com/tensorflow/models/blob/master/official/nlp/modeling/layers/gaussian_process.py) implements a [random-feature based approximation](https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf) to a Gaussian process model that is end-to-end trainable with a deep neural network. Under the hood, the Gaussian process layer implements a two-layer network:\n",
+        "[`RandomFeatureGaussianProcess`](https://github.com/tensorflow/models/blob/master/official/nlp/modeling/layers/gaussian_process.py){.external} implements a [random-feature based approximation](https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf){.external} to a Gaussian process model that is end-to-end trainable with a deep neural network. Under the hood, the Gaussian process layer implements a two-layer network:\n",
         "\n",
         "$$logits(x) = \\Phi(x) \\beta, \\quad \\Phi(x)=\\sqrt{\\frac{2}{M}} * cos(Wx + b)$$\n",
         "\n",
-        "Here $x$ is the input, and $W$ and $b$ are frozen weights initialized randomly from Gaussian and uniform distributions, respectively. (Therefore $\\Phi(x)$ are called \"random features\".) $\\beta$ is the learnable kernel weight similar to that of a Dense layer. "
+        "Here, $x$ is the input, and $W$ and $b$ are frozen weights initialized randomly from Gaussian and Uniform distributions, respectively. (Therefore, $\\Phi(x)$ are called \"random features\".) $\\beta$ is the learnable kernel weight similar to that of a Dense layer. "
       ]
     },
     {
@@ -834,7 +834,7 @@
         "id": "xgzw09gS03ae"
       },
       "source": [
-        "Note: For a deep neural network that is sensitive to the learning rate (e.g., ResNet-50 and ResNet-110), it is generally recommended to set `normalize_input=True` to stabilize training, and set `scale_random_features=False` to avoid the learning rate from being modified in unexpected ways when passing through the GP layer."
+        "Note: For a deep neural network that is sensitive to the learning rate (for example, ResNet-50 and ResNet-110), it is generally recommended to set `normalize_input=True` to stabilize training, and set `scale_random_features=False` to avoid the learning rate from being modified in unexpected ways when passing through the GP layer."
       ]
     },
     {
@@ -843,7 +843,7 @@
         "id": "pZkcKw-u7XRp"
       },
       "source": [
-        "* `gp_cov_momentum` controls how the model covariance is computed. If set to a positive value (e.g., 0.999), the covariance matrix is computed using the momentum-based moving average update (similar to batch normalization). If set to -1, the covariance matrix is updated without momentum."
+        "* `gp_cov_momentum` controls how the model covariance is computed. If set to a positive value (for example, `0.999`), the covariance matrix is computed using the momentum-based moving average update (similar to batch normalization). If set to `-1`, the covariance matrix is updated without momentum."
       ]
     },
     {
@@ -883,9 +883,9 @@
         "id": "ALBqcAtwDNiO"
       },
       "source": [
-        "Note: Notice that under this implementation of the SNGP model, the predictive logits $logit(x_{test})$ for all classes share the same covariance matrix $var(x_{test})$, which describes the distance between $x_{test}$ from the training data. \n",
+        "Note: Notice that under this implementation of the SNGP model, the predictive logits $logit(x_{test})$ for all classes share the same covariance matrix $var(x_{test})$, which describes the distance between $x_{test}$ from the training data.\n",
         "\n",
-        "Theoretically, it is possible to extend the algorithm to compute different variance values for different classes (as introduced in the [original SNGP paper](https://arxiv.org/abs/2006.10108)). However, this is difficult to scale to problems with large output spaces (e.g., ImageNet or language modeling)."
+        "Theoretically, it is possible to extend the algorithm to compute different variance values for different classes (as introduced in the [original SNGP paper](https://arxiv.org/abs/2006.10108){.external}). However, this is difficult to scale to problems with large output spaces (such as classification with ImageNet or language modeling)."
       ]
     },
     {
@@ -929,12 +929,12 @@
         "  def make_output_layer(self, num_classes):\n",
         "    \"\"\"Uses Gaussian process as the output layer.\"\"\"\n",
         "    return nlp_layers.RandomFeatureGaussianProcess(\n",
-        "        num_classes, \n",
+        "        num_classes,\n",
         "        gp_cov_momentum=-1,\n",
         "        **self.classifier_kwargs)\n",
         "\n",
         "  def call(self, inputs, training=False, return_covmat=False):\n",
-        "    # Gets logits and covariance matrix from GP layer.\n",
+        "    # Gets logits and a covariance matrix from the GP layer.\n",
         "    logits, covmat = super().call(inputs)\n",
         "\n",
         "    # Returns only logits during training.\n",
@@ -1121,7 +1121,7 @@
         "\n",
         "$$E(p(x)) = \\frac{1}{M} \\sum_{m=1}^M logit_m(x), $$\n",
         "\n",
-        "where $M$ is the sample size, and $logit_m(x)$ are random samples from the SNGP posterior $MultivariateNormal$(`sngp_logits`,`sngp_covmat`). However, this approach can be slow for latency-sensitive applications such as autonomous driving or real-time bidding. Instead, can approximate $E(p(x))$ using the [mean-field method](https://arxiv.org/abs/2006.07584):\n",
+        "where $M$ is the sample size, and $logit_m(x)$ are random samples from the SNGP posterior $MultivariateNormal$(`sngp_logits`,`sngp_covmat`). However, this approach can be slow for latency-sensitive applications such as autonomous driving or real-time bidding. Instead, you can approximate $E(p(x))$ using the [mean-field method](https://arxiv.org/abs/2006.07584){.external}:\n",
         "\n",
         "$$E(p(x)) \\approx softmax(\\frac{logit(x)}{\\sqrt{1+ \\lambda * \\sigma^2(x)}})$$\n",
         "\n",
@@ -1146,7 +1146,7 @@
         "id": "bNVs_KO-5HdL"
       },
       "source": [
-        "Note: Instead of fixing $\\lambda$ to a fixed value, you can also treat it as a hyperparameter, and tune it to optimize the model's calibration performance. This is known as [temperature scaling](http://proceedings.mlr.press/v70/guo17a.html) in the deep learning uncertainty literature. "
+        "Note: Instead of fixing $\\lambda$ to a fixed value, you can also treat it as a hyperparameter, and tune it to optimize the model's calibration performance. This is known as [temperature scaling](http://proceedings.mlr.press/v70/guo17a.html){.external} in the deep learning uncertainty literature. "
       ]
     },
     {
@@ -1235,7 +1235,7 @@
         "id": "R9kY5dJg8fEi"
       },
       "source": [
-        "Put everything together. The entire procedure (training, evaluation and uncertainty computation) can be done in just five lines:"
+        "You can now put everything together. The entire procedure—training, evaluation and uncertainty computation—can be done in just five lines:"
       ]
     },
     {
@@ -1324,7 +1324,7 @@
         "id": "mao9L-LYE1Nl"
       },
       "source": [
-        "Like mentioned earlier, a deterministic model is not _distance-aware_. Its uncertainty is defined by the distance of the test example from the decision boundary. This leads the model to produce overconfident predictions for the out-of-domain examples (red)."
+        "As mentioned earlier, a deterministic model is not _distance-aware_. Its uncertainty is defined by the distance of the test example from the decision boundary. This leads the model to produce overconfident predictions for the out-of-domain examples (red)."
       ]
     },
     {
@@ -1342,9 +1342,9 @@
         "id": "S1DPELWE6LL8"
       },
       "source": [
-        "This section compares the uncertainty of SNGP with [Monte Carlo dropout](https://arxiv.org/abs/1506.02142) and [Deep ensemble](https://arxiv.org/abs/1612.01474).\n",
+        "This section compares the uncertainty of SNGP with [Monte Carlo dropout](https://arxiv.org/abs/1506.02142){.external} and [Deep ensemble](https://arxiv.org/abs/1612.01474){.external}.\n",
         "\n",
-        "Both of these methods are based on Monte Carlo averaging of multiple forward passes of deterministic models. First set the ensemble size $M$."
+        "Both of these methods are based on Monte Carlo averaging of multiple forward passes of deterministic models. First, set the ensemble size $M$."
       ]
     },
     {
@@ -1373,9 +1373,9 @@
         "id": "ZBzp2LBt7-kj"
       },
       "source": [
-        "Given a trained neural network with Dropout layers, [Monte Carlo dropout](https://arxiv.org/abs/1506.02142) computes the mean predictive probability \n",
+        "Given a trained neural network with Dropout layers, Monte Carlo dropout computes the mean predictive probability\n",
         "\n",
-        "$$E(p(x)) = \\frac{1}{M}\\sum_{m=1}^M softmax(logit_m(x))$$ \n",
+        "$$E(p(x)) = \\frac{1}{M}\\sum_{m=1}^M softmax(logit_m(x))$$\n",
         "\n",
         "by averaging over multiple Dropout-enabled forward passes $\\{logit_m(x)\\}_{m=1}^M$."
       ]
@@ -1444,7 +1444,7 @@
         "id": "L-Z2veGJ9ZgY"
       },
       "source": [
-        "[Deep ensemble](https://arxiv.org/abs/1612.01474) is a state-of-the-art (but expensive) method for deep learning uncertainty. To train a Deep ensemble, first train $M$ ensemble members."
+        "Deep ensemble is a state-of-the-art (but expensive) method for deep learning uncertainty. To train a Deep ensemble, first train $M$ ensemble members."
       ]
     },
     {
@@ -1460,7 +1460,7 @@
         "for _ in range(num_ensemble):\n",
         "  resnet_model = DeepResNet(**resnet_config)\n",
         "  resnet_model.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n",
-        "  resnet_model.fit(train_examples, train_labels, verbose=0, **fit_config)  \n",
+        "  resnet_model.fit(train_examples, train_labels, verbose=0, **fit_config)\n",
         "\n",
         "  resnet_ensemble.append(resnet_model)"
       ]
@@ -1471,7 +1471,7 @@
         "id": "Al7uM-fn_ZE1"
       },
       "source": [
-        "Collect logits and compute the mean predctive probability $E(p(x)) = \\frac{1}{M}\\sum_{m=1}^M softmax(logit_m(x))$."
+        "Collect logits and compute the mean predictive probability $E(p(x)) = \\frac{1}{M}\\sum_{m=1}^M softmax(logit_m(x))$."
       ]
     },
     {
@@ -1505,7 +1505,7 @@
         "id": "GH33oVvV5-ez"
       },
       "source": [
-        "Both MC Dropout and Deep ensemble improve a model's uncertainty ability by making the decision boundary less certain. However, they both inherit the deterministic deep network's limitation in lacking distance awareness."
+        "Both the Monte Carlo Dropout and Deep ensemble methods improve the model's uncertainty ability by making the decision boundary less certain. However, they both inherit the deterministic deep network's limitation in lacking distance awareness."
       ]
     },
     {
@@ -1524,8 +1524,8 @@
       },
       "source": [
         "In this tutorial, you have:\n",
-        "* Implemented a SNGP model on a deep classifier to improve its distance awareness.\n",
-        "* Trained the SNGP model end-to-end using Keras `model.fit()` API.\n",
+        "* Implemented the SNGP model on a deep classifier to improve its distance awareness.\n",
+        "* Trained the SNGP model end-to-end using Keras `Model.fit` API.\n",
         "* Visualized the uncertainty behavior of SNGP.\n",
         "* Compared the uncertainty behavior between SNGP, Monte Carlo dropout and deep ensemble models."
       ]
@@ -1545,9 +1545,9 @@
         "id": "HoIikRybke-b"
       },
       "source": [
-        "* See the [SNGP-BERT tutorial](https://www.tensorflow.org/text/tutorials/uncertainty_quantification_with_sngp_bert) for an example of applying SNGP on a BERT model for uncertainty-aware natural language understanding.\n",
-        "* See [Uncertainty Baselines](https://github.com/google/uncertainty-baselines)  for the implementation of SNGP model (and many other uncertainty methods) on a wide variety of benchmark datasets (e.g., [CIFAR](https://www.tensorflow.org/datasets/catalog/cifar100), [ImageNet](https://www.tensorflow.org/datasets/catalog/imagenet2012), [Jigsaw toxicity detection](https://www.tensorflow.org/datasets/catalog/wikipedia_toxicity_subtypes), etc).\n",
-        "* For a deeper understanding of the SNGP method, check out the paper [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://arxiv.org/abs/2006.10108).\n"
+        "* Check out the [SNGP-BERT tutorial](https://www.tensorflow.org/text/tutorials/uncertainty_quantification_with_sngp_bert) for an example of applying SNGP on a BERT model for uncertainty-aware natural language understanding.\n",
+        "* Go to the [Uncertainty Baselines GitHub repo](https://github.com/google/uncertainty-baselines){.external} for the implementation of SNGP model (and many other uncertainty methods) on a wide variety of benchmark datasets (for example, [CIFAR](https://www.tensorflow.org/datasets/catalog/cifar100), [ImageNet](https://www.tensorflow.org/datasets/catalog/imagenet2012), [Jigsaw toxicity detection](https://www.tensorflow.org/datasets/catalog/wikipedia_toxicity_subtypes), etc).\n",
+        "* For a deeper understanding of the SNGP method, check out the paper titled [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://arxiv.org/abs/2006.10108){.external}.\n"
       ]
     }
   ],

From a2d1763230270504f871b14b1b5c9c504c452add Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 30 Jun 2022 13:47:41 +0100
Subject: [PATCH 209/872] Change Install TensorFlow with pip link

---
 site/en/install/gpu_plugins.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index 446107b4b0e..583c4940c18 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -1,7 +1,7 @@
 # GPU device plugins
 
 Note: This page is for non-NVIDIA® GPU devices. For NVIDIA® GPU support, go to
-the [Install TensorFlow with pip](./gpu.md) guide.
+the [Install TensorFlow with pip](./pip) guide.
 
 TensorFlow's
 [pluggable device](https://github.com/tensorflow/community/blob/master/rfcs/20200624-pluggable-device-for-tensorflow.md){.external}

From 033c7a03d19aef28e810659899586eb277034372 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 30 Jun 2022 14:48:14 +0100
Subject: [PATCH 210/872] Change Install TensorFlow with pip link

---
 site/en/install/gpu_plugins.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index 583c4940c18..290cd48cf76 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -1,7 +1,7 @@
 # GPU device plugins
 
 Note: This page is for non-NVIDIA® GPU devices. For NVIDIA® GPU support, go to
-the [Install TensorFlow with pip](./pip) guide.
+the [Install TensorFlow with pip](./pip.md) guide.
 
 TensorFlow's
 [pluggable device](https://github.com/tensorflow/community/blob/master/rfcs/20200624-pluggable-device-for-tensorflow.md){.external}

From 95141f54c51f89c30a98b43dde722b2618d16a43 Mon Sep 17 00:00:00 2001
From: Ahmed Negmeldin 
Date: Sun, 3 Jul 2022 18:01:26 +0200
Subject: [PATCH 211/872] correct typical axis order visulization

fixes # 56662 on tensorflow/tensorflow
---
 site/en/guide/images/tensor/shape2.png | Bin 4608 -> 3625 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/site/en/guide/images/tensor/shape2.png b/site/en/guide/images/tensor/shape2.png
index 3609ff2c263de35b4b66bed4f652801f58daf523..a316359c8fc5a4a86bdd44dc4d4d9abcc8faf7bc 100644
GIT binary patch
literal 3625
zcmeHK`8N~{_a7m$FHeRnqlmIRC{ly5Yh)NEWNkKFt7@QVJeYyfPL_;CTRHO$uZ82q37
zdjd-X+AhbDV=%(%CO{o2eqEQ3*o}-M0Y0Q9EiMN!DAP1&fp;~V%(sg~#4_b+uMN^u-WW)Q^C-s~Ldc5&~YhpD-2}embF6=Nf
z&{&IjmN1aA6-Te?S8`}ObLgO`maZU9QBJRTJA(DR7@vThA2-rb`{;>nT=%QGr@A*E
z&&}gq{_$bE*!D9~^jop}uFv5$na(LwV-M1
zmHx-se4FYl+5_HRX&Sa17RH_LyUbrK0&(wz)2jiV^x$Oo6M_M=w>}1Xb(~#ESd8LT+UPg#C4Hw{&x->iyyeR1CZ!`O
zO}@w>3ySP!&Vr)3(Z%7{(36ful?*I<1w8M!k>*|cXwah&i5-gmQP6|d_Q^mw*m^4-ZNhrwWVbnAuhL$~j)RYIVWrHZ*lglkf@Hy!W0
zYd$U9MSz~(@@Fj
z=JTdtDSB^7cZY>|i*x4whrGhd)K2;1fcg!b=&-H|4O0l0IfUG_ZMD$*3pX9n%Ixw$e@J&b!Cb&=`
zW9c*rPQ%ek;UAMo=~0oJ2kED*ODNhEBXY&Z-aJ+
zE8n3nprhsaACetd-)HFjWW)3~S8Dq7Y2S8A2v?x!j%<3k9ED`IzLGxCCyCG0Mt}m!
zA@lzMJS9%7rlfN!ETon6O^I&FR-)H~9v26lhnB>Ia%ky+EGG9*?R91w$&d)L9TlHh
zFD?RPW+ne@e9iO6LCRP8m+vyXJ9_hn^(&Sl0qQA!
zHiaqrophQ;Oln7=Y+c
zS*(xkl7J~!hKPaRuZKowVCLrzNpEXsolWn^KtWCTDf9Cxh*u)Dl}5BI<9Nx|^x8CU
z&-Z`oe#?2P^21jx6?6a*HZ
z1BgAK^J(0D($7VcQ)31RD+=*^=Mh+Z2;uQ|Of}(_Zi5U`?V^)oQl~8J0GkzXkyj=I
zTBC(*i+LZ9Fx6Eh7n}XRPL%@DBL&GqUwzR!SQp0tFsSQt&DiZ)-x@=(iAEsdFO2vCGRw
zt90F~kwOxhw)R_NYD5E-FSQ~V8Q*-X8@QAEs$s3yWA=hPFZ~)3va2wZ?nmITRy~Ci
z*)p)#8`~1C9Af3&3NZ^OqLZzGp9X@03@*2_y)ydh{AT>h(0lW>G6DDS!eA(#eLCFU
zf2i8nV|Y_#;E5;gO|>e;ut2@)7M83c*Ew-7%Iz%~8IpeK-3tXus9PHNH%WBtk_BU6
zIxBcq7@MSUk8PTPUudR|-bi}uBq3o|cpLvOx>=Kyjpleb_@fQKUS68v12Z4tiPoah
z9QXTVu;EtL&fJc6+AJJx#rEHp=0CSdNV5hs36FSlsNH*;{F*gMC+XC!%uZ$!EG5Ar
zH@YymAQ`IuFKbFYOZfdB_y`9$Vz%W-bB%QwSg4zP*b=*1OvNqL-R@(}s>QPUR&Rt#
zRk6z1n`NlT*xv<}hkad=#Z(q^w-Kb*a_~Tz5!?M^dxvL@pX#CL5s8uYYYqS(^$i3C
z`#M#7=XN%{ID9F$&0O#*_~+Esdpt+;+23llDx%Nzx;=JT&a`g@a%r?QU330p46XpM
z96q7E>~C)|;SaE#+e+gV+8{=s!s?hJN^)7VT0nPp2o3JYKt!(sKkA4q|H4DoiyhXXct5XOh_+^5Tw@xFCC
zY0a(q#TX}RfEnSCNPI_lI7v;Xrx`|PI#}|?9%22)WqnH_J_&UN#oFO6+x>eSA2h1N
z1!36?@gVo!EtFwcrSfS|F&ChF`EEAb^asWUVBaoKdS!n9z??+swjqv?qIQG@(tgC8&YDBXW~oS>-`By7x;aK0E8t$Dchvi_>Ca7SNKM!sapWj*T;
zA_s)CnhSr0xmFSQ_6x6u1VY-HO9&Gx;<^~H0NM3kQg<}HY-tVId(I;m2UVPt}~}s;+Ywk9J;R;O|ol4jWRQ`tsOIkTyJ>xEgj^Z(b
z=)8!-N^R^Iyg{NIe^<$6#}32O$*(u1m;k3uou
zdP+55ASSbkaxpt~HT47;Ddv|9Q*ILuM2Gth7=d_q*A3vU)kGZ{(ivp09m1^uD*wCy
z_T}VA-RMER8|A{PJG9MU6aRF*QNq8Hh7&%=xVwbwIXUgC(#l0gGol~y_|t8n_5Fhd
z>G>3Y0VFx`^=8+dCw{K%Klz6%TqjZJU%2Fau~335BaVJx3}hN~kayq)QGZf}?wSBU
zl%ujbzF?x=cO$Q^NFIQUrYlnEmWGvjbYIzV#qFAA^-r`>4Aj`CR;)DNA|vD_K>8Ni
zI=EBN80m~lCOx=Pp6+vkvjM!;*G{U+Hm$nUQXfWTMn0}-l<_&ObsJ}c#73K$q5l5?
cu*zbwpL_rox`EMB`1_eLH?=aUHF8V%KX+W7@Bjb+

literal 4608
zcmZ`+c{tQv^#6t;(pZv8mKyRVOA+#7ET!ygyb_~OW66^2Gh<)I*pkX_vW+FOWY1Es
z(J0Gc3e7Mw)D#9|nHj|Vy#4<9z0dRh?(^Jp&pFRI&%Ni|dp`HxWNXVCg8ZlW0RRv*
zF}`LC00*S^`9{7&`xIU>@BsjX@g~;{ZibQQg(~I`2x9mfHTpIBG7kG)h>ypuPgItV
zhkE$QSs|4CzRVS)ufyPpJ;%4NuzGwbCM@-(-DsU44A;Wy%3XCj6s!pvi|XKk48pB!KHzJs~E
zCo$bKPnWQn)e7eE(0(7spG>&>!tD$MMYx@u;mmpTH`Q)J#}Jh2eIN?d<}7Hxfq$Ft
z?!c;Jd4d=*O?|Kt#2LupKrzoB5>Bh`q{7a1rvJ}96TG}0Yf6*xEOgWP;q>r^;5<5G
zdS+xjRF{$2HkMGt88PS0lh;S^NBCK=roAffCkivcT-FDtN}-l08S%lDtTyr4aI`+m1gI5DhhVj06$&KM6Lcoq!7XBj0Y!kkB-iG
z&@=8!#!BAQnc&?m>r(yD+aMMC5j@bvKGq~P3Ee?=3mE~`-|wZ4t$2~Q5yhNVnr@iJ
zgYGh^y%Bd-^U(=)P-(>vM;Y
zX(&`u%ckK(G9UOR7zZ1T=e15{`WaH^SRRJpHz)#7@Nx&d+KC9zsXgegb>D(`WhQm7mj~sq5a~t2u8S;rb$U*WF;TH0*0IqC`oWy{FqHcVoQeJ;UGbY9(B2HOpfKOp;HBuZAHaUaq1jikOYz8=nTKk4l5#_#jTQ
zFb}&#jO~Nl^O~04E*vcJvf>YNX+{!{0u=A1NNE*lc-%25AKp*j&>^m4BWGv>_vBe3W
z<|SVHHz;rX0K&O&@)sbvKa39fZa)c-n!o+MFT=T?Id{Xxq}xPwjMn^Z)*1A8zY-hW
z`hr}cS|7yA6i2}`r!hlu>{K{;V7Pk-Qir6#%8mAueODOsER{~PMWlvodCc6xwHxWy
zeX3wTHD?o_bng!kh)4#zGm3HEGan`D_s3#G0#9GXos#TW1-$29VBIM~KAM2?stf
zS1x?MUx&#h=1pg-MQA#B$<)81=Ee)fq991MKn9^ghwktCH@D0rc)
zOawl-%K91#ysFmJ^&tsVe+b4Fa8-Gq`N&-`PL~*t%boBL_r1NkCJXxZ5Ob@cAubJUg!XbQNb_b$A(yDF_
zIydPR6=GoZSM+cbU3oD$djl#pC~@fBn}Y?ra_*YVs|X`kJ{`pwdgH(cD4fgolZ5X2
z+B}&uN+%+xDv&-K`RT$K^Y+uLTT0CH5SH)+nGvxjxE`_L<9zVv+pXIR9KW}eLW;Y5
z>&w3wRT1=>GGEpWeH|5b5HswrBT5dHSc(%OeQ4k=4y?q65$>TE0{z|#VXPv|e?+*)
z=sZjMgYT~!h`@U@U8Vk`ylD(EZ0vJmX&|E
zU|Nr3d{^4&C$^kJ{^Spa?R32_SMW;Cc!|1@4&SVCY2r?s9rVgpe8omwN7QrP;qdP@
zN}q@oRnKiB>_h99+8%kK2q8%b%doCSExN(v#2
ze9c=wka_E9{BCKj7Qzs(Degpw|wCbD@%(w{66F
zROwq+mbvSD6KGgTj}||2njbkJ_BQdK6&JIQeMKH
zZx)1vQ&AsFu`a=6Z>+9;c|QBZP`>10@+XZju^*Z3+%_*smM^_`hOWQSP+yM}DV^a;
z_Yg#+mls$*dTigpPU|wx>Aclr$6mZoIio@0GIW)$Wjm
z&CvA-Q?&DnA#0+*-Qg1@UVN=>)F)kw(qC2NY9DhRPA@5ostamswUfutzHi=)k0ph!
z(EccG$%cFW+G+vm&-Fs~VGW7njeehm+{dUL6bMKI?Ew4n2}=X}(;X0!UIIga%aA2*
z8OY=f1mOAW|C&U%R(OzRG`6$%cI1hiwb9*4&Py2Q+`XNsM@yWi23s?KGJS5E^oy=d
zXZ7TdCq0Y2RF^Gkk~fpfGz@A!vc#b@>79+tHrV34Yv91D!Sjc=`o~D$>$ic8g=3`ch_^I4YqLKHr
z(~OJ#cqxZjS5?>4IHPs;a&G-IN(+&QlhbIXf9LKOJU+2k$?sZl({y&v#nihs
z$95~F@SswZ$!jbp_giot(5cCI2Vowl{Pj;||0gR0kJKoXUy>^RlZ
z#mlda0{W+PWgfnJZ+?MK+aeRtcXIaJwzL8C^NfW$?mG3306^8HDr&FQ5kSV?&NvbyC(Zq4WO0}`y->Z@-)^8sWRP;XHph0TNt|$D
zLk5x+(dA2oJ1?Zz9lC%5yKSiQ6d0Ki(ow
z*&)tJ!rve_uu_{v=-)0Gmp{UEL&)UTQnJyt`vr7&1!Y0q*3&{5m@gY+;gdsV#@6&J
z1_@&f;%`JaWalN_stZl}9Zl$ha>7^_@rTOb;tG9o_mQyR)G|g8Wd;7*;?Ki;Xcb5t
zU&bAGN9_X>B8&sZCWsEXlYXvhv%xLzJEbxR;k=CY^eZrJ8LqjaJGW+k!kP(aX4tDP
zxFG_tJ&v(k4|G_6(jnW@swTD`k+q;u`*I+d&Uf7yk-l>bqmB1n3=WrkNSCcwO1Wss
zJ7tFBl`&#yr3U0KjjWd@Lv0MF>ivu0_w9y+(O;CQV=3%ON=};k=GxECNW8rSYb+6r
z4v_%L6d^6y))Vd?B2dq8SM4!5-DI%uTM_-11K$Vszf{u$WJ?gsS(6cEJN5z%cz_{(
z2-%>ig1NnG?e{~zul_B8&bZeNVwv%%Dzrh>>$q2e$C+$CxMeR!0B5`)Ota9U)ym31
zmUTB4A`@i3b6@nJ+66^i46evjxuTq{=KyE-{yn->cdoqXMU`Pck=I!0+5t5ERGC5jgJ|)##ETvNFDB4l$;grRN85}5H
zb)i2~T+osq4jz9qDM(jYY>CM3EX-R)lwDC|I!XZ-c~U)B%C6b``-uOq^!Cjx=QV8j
UfziFE`TuB4j4ZFcGQ1P}Kl_v?
Date: Fri, 8 Jul 2022 10:35:39 +0530
Subject: [PATCH 212/872] Update `tf.losses.*` to `tf.keras.loss`

Update alias to standard name.
From `tf.losses.SparseCategoricalCrossentropy` to `tf.keras.losses.SparseCategoricalCrossentropy`
---
 site/en/tutorials/load_data/images.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/load_data/images.ipynb b/site/en/tutorials/load_data/images.ipynb
index 13d6682b60a..7367cda62fa 100644
--- a/site/en/tutorials/load_data/images.ipynb
+++ b/site/en/tutorials/load_data/images.ipynb
@@ -548,7 +548,7 @@
       "source": [
         "model.compile(\n",
         "  optimizer='adam',\n",
-        "  loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
+        "  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
         "  metrics=['accuracy'])"
       ]
     },

From dd0732c99de55cb1416f03e49596912c359fea05 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sun, 10 Jul 2022 13:41:29 +0100
Subject: [PATCH 213/872] Update tf.keras.optimizers.SGD alias in SavedModel
 format guide

---
 site/en/guide/saved_model.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/saved_model.ipynb b/site/en/guide/saved_model.ipynb
index da9652c6554..a6b2c7185f0 100644
--- a/site/en/guide/saved_model.ipynb
+++ b/site/en/guide/saved_model.ipynb
@@ -493,7 +493,7 @@
       },
       "outputs": [],
       "source": [
-        "optimizer = tf.optimizers.SGD(0.05)\n",
+        "optimizer = tf.keras.optimizers.SGD(0.05)\n",
         "\n",
         "def train_step():\n",
         "  with tf.GradientTape() as tape:\n",

From 29058e9ee08ff0164378b81ec60357d871e8ebe9 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sun, 10 Jul 2022 13:52:26 +0100
Subject: [PATCH 214/872] Update tf.keras.optimizers alias in Distributed
 training guide

---
 site/en/guide/distributed_training.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb
index 96c98edc82a..0614ba51e18 100644
--- a/site/en/guide/distributed_training.ipynb
+++ b/site/en/guide/distributed_training.ipynb
@@ -421,7 +421,7 @@
       "source": [
         "This strategy serves two main purposes:\n",
         "\n",
-        "* It allows writing distribution-aware library code unconditionally. For example, in `tf.optimizer`s you can use `tf.distribute.get_strategy` and use that strategy for reducing gradients—it will always return a strategy object on which you can call the `Strategy.reduce` API.\n"
+        "* It allows writing distribution-aware library code unconditionally. For example, in `tf.keras.optimizers` you can use `tf.distribute.get_strategy` and use that strategy for reducing gradients—it will always return a strategy object on which you can call the `Strategy.reduce` API.\n"
       ]
     },
     {

From 92fba4eb91c2e32cfd90c6650e346fd958b1f7a0 Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Sun, 10 Jul 2022 12:53:04 +0000
Subject: [PATCH 215/872] nbfmt

---
 site/en/guide/distributed_training.ipynb | 2 --
 1 file changed, 2 deletions(-)

diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb
index 0614ba51e18..a15be67cfec 100644
--- a/site/en/guide/distributed_training.ipynb
+++ b/site/en/guide/distributed_training.ipynb
@@ -900,8 +900,6 @@
         "Tce3stUlHN0L"
       ],
       "name": "distributed_training.ipynb",
-      "private_outputs": true,
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 645d3945665871bf4bab86ddab69ad03a5281c68 Mon Sep 17 00:00:00 2001
From: chunduriv <74177924+chunduriv@users.noreply.github.com>
Date: Mon, 11 Jul 2022 21:04:42 +0530
Subject: [PATCH 216/872] Update `Broadcasting` reference

Update correct reference for Broadcasting
---
 site/en/guide/tensor.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/tensor.ipynb b/site/en/guide/tensor.ipynb
index fe2cae2a9b1..a9eb8fe0d7e 100644
--- a/site/en/guide/tensor.ipynb
+++ b/site/en/guide/tensor.ipynb
@@ -962,7 +962,7 @@
       "source": [
         "## Broadcasting\n",
         "\n",
-        "Broadcasting is a concept borrowed from the [equivalent feature in NumPy](https://numpy.org/doc/stable/user/basics.html).  In short, under certain conditions, smaller tensors are \"stretched\" automatically to fit larger tensors when running combined operations on them.\n",
+        "Broadcasting is a concept borrowed from the [equivalent feature in NumPy](https://numpy.org/doc/stable/user/basics.broadcasting.html).  In short, under certain conditions, smaller tensors are \"stretched\" automatically to fit larger tensors when running combined operations on them.\n",
         "\n",
         "The simplest and most common case is when you attempt to multiply or add a tensor to a scalar.  In that case, the scalar is broadcast to be the same shape as the other argument. "
       ]

From 1cb0db13f36910b3c9675e537c99128463872bb9 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Tue, 12 Jul 2022 16:58:08 +0100
Subject: [PATCH 217/872] Add external link symbols to Introduction to Tensors
 guide

---
 site/en/guide/tensor.ipynb | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/site/en/guide/tensor.ipynb b/site/en/guide/tensor.ipynb
index a9eb8fe0d7e..5ac82475a9d 100644
--- a/site/en/guide/tensor.ipynb
+++ b/site/en/guide/tensor.ipynb
@@ -82,7 +82,7 @@
       "source": [
         "Tensors are multi-dimensional arrays with a uniform type (called a `dtype`).  You can see all supported `dtypes` at `tf.dtypes.DType`.\n",
         "\n",
-        "If you're familiar with [NumPy](https://numpy.org/devdocs/user/quickstart.html), tensors are (kind of) like `np.arrays`.\n",
+        "If you're familiar with [NumPy](https://numpy.org/devdocs/user/quickstart.html){:.external}, tensors are (kind of) like `np.arrays`.\n",
         "\n",
         "All tensors are immutable like Python numbers and strings: you can never update the contents of a tensor, only create a new one.\n"
       ]
@@ -498,7 +498,7 @@
       "source": [
         "### Single-axis indexing\n",
         "\n",
-        "TensorFlow follows standard Python indexing rules, similar to [indexing a list or a string in Python](https://docs.python.org/3/tutorial/introduction.html#strings), and the basic rules for NumPy indexing.\n",
+        "TensorFlow follows standard Python indexing rules, similar to [indexing a list or a string in Python](https://docs.python.org/3/tutorial/introduction.html#strings){:.external}, and the basic rules for NumPy indexing.\n",
         "\n",
         "* indexes start at `0`\n",
         "* negative indices count backwards from the end\n",
@@ -962,7 +962,7 @@
       "source": [
         "## Broadcasting\n",
         "\n",
-        "Broadcasting is a concept borrowed from the [equivalent feature in NumPy](https://numpy.org/doc/stable/user/basics.broadcasting.html).  In short, under certain conditions, smaller tensors are \"stretched\" automatically to fit larger tensors when running combined operations on them.\n",
+        "Broadcasting is a concept borrowed from the [equivalent feature in NumPy](https://numpy.org/doc/stable/user/basics.broadcasting.html){:.external}.  In short, under certain conditions, smaller tensors are \"stretched\" automatically to fit larger tensors when running combined operations on them.\n",
         "\n",
         "The simplest and most common case is when you attempt to multiply or add a tensor to a scalar.  In that case, the scalar is broadcast to be the same shape as the other argument. "
       ]
@@ -1088,7 +1088,7 @@
       "source": [
         "Unlike a mathematical op, for example, `broadcast_to` does nothing special to save memory.  Here, you are materializing the tensor.\n",
         "\n",
-        "It can get even more complicated.  [This section](https://jakevdp.github.io/PythonDataScienceHandbook/02.05-computation-on-arrays-broadcasting.html) of Jake VanderPlas's book _Python Data Science Handbook_ shows more broadcasting tricks (again in NumPy)."
+        "It can get even more complicated.  [This section](https://jakevdp.github.io/PythonDataScienceHandbook/02.05-computation-on-arrays-broadcasting.html){:.external} of Jake VanderPlas's book _Python Data Science Handbook_ shows more broadcasting tricks (again in NumPy)."
       ]
     },
     {

From df0826ae23fdc631bdbe62b7d41c4c04787e8ac8 Mon Sep 17 00:00:00 2001
From: RenuPatelGoogle <89264621+RenuPatelGoogle@users.noreply.github.com>
Date: Wed, 13 Jul 2022 21:23:44 +0530
Subject: [PATCH 218/872] Updated correct link for model subclassing API

The old link was routing to `the Sequential model` page which was not relevant, so updated the model subclassing API link with the exact page.
---
 site/en/tutorials/quickstart/advanced.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/quickstart/advanced.ipynb b/site/en/tutorials/quickstart/advanced.ipynb
index 5fcbedb3ab5..923d9dd50ad 100644
--- a/site/en/tutorials/quickstart/advanced.ipynb
+++ b/site/en/tutorials/quickstart/advanced.ipynb
@@ -165,7 +165,7 @@
         "id": "BPZ68wASog_I"
       },
       "source": [
-        "Build the `tf.keras` model using the Keras [model subclassing API](https://www.tensorflow.org/guide/keras#model_subclassing):"
+        "Build the `tf.keras` model using the Keras [model subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models#the_model_class):"
       ]
     },
     {

From 8facede0f76b58e9905df7c1d81efe7bd4c4d76b Mon Sep 17 00:00:00 2001
From: synandi <98147397+synandi@users.noreply.github.com>
Date: Thu, 14 Jul 2022 11:29:43 +0530
Subject: [PATCH 219/872] Update custom_training.ipynb

Fixed broken link at line 679
---
 site/en/tutorials/distribute/custom_training.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index 0ffbe23b753..bfdb18316f4 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -676,7 +676,7 @@
         "\n",
         "1. [Distributed training guide](../../guide/distributed_training)\n",
         "2. [DenseNet](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/densenet/distributed_train.py) example using `MirroredStrategy`.\n",
-        "1. [BERT](https://github.com/tensorflow/models/blob/master/official/nlp/bert/run_classifier.py) example trained using `MirroredStrategy` and `TPUStrategy`.\n",
+        "1. [BERT](https://github.com/tensorflow/models/blob/v2.8.0/official/nlp/bert/run_classifier.py) example trained using `MirroredStrategy` and `TPUStrategy`.\n",
         "This example is particularly helpful for understanding how to load from a checkpoint and generate periodic checkpoints during distributed training etc.\n",
         "2. [NCF](https://github.com/tensorflow/models/blob/master/official/recommendation/ncf_keras_main.py) example trained using `MirroredStrategy` that can be enabled using the `keras_use_ctl` flag.\n",
         "3. [NMT](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/nmt_with_attention/distributed_train.py) example trained using `MirroredStrategy`.\n",

From b435eca227727272d535c03e04f1a6bdadb2834e Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Fri, 15 Jul 2022 18:47:43 -0700
Subject: [PATCH 220/872] Fix title rendering on tensorflow.org for Custom
 training with tf.distribute.Strategy tutorial

---
 site/en/tutorials/distribute/custom_training.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index bfdb18316f4..513cce5d340 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -37,7 +37,7 @@
         "id": "jYysdyb-CaWM"
       },
       "source": [
-        "# Custom training with `tf.distribute.Strategy`"
+        "# Custom training with tf.distribute.Strategy"
       ]
     },
     {

From 05e43ee40b97d85d0a53bb124cca64d2fc4391e1 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Fri, 15 Jul 2022 18:58:36 -0700
Subject: [PATCH 221/872] Update BERT example link in Custom training with
 tf.distribute.Strategy tutorial

---
 site/en/tutorials/distribute/custom_training.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index 513cce5d340..4534f1a7044 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -676,7 +676,7 @@
         "\n",
         "1. [Distributed training guide](../../guide/distributed_training)\n",
         "2. [DenseNet](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/densenet/distributed_train.py) example using `MirroredStrategy`.\n",
-        "1. [BERT](https://github.com/tensorflow/models/blob/v2.8.0/official/nlp/bert/run_classifier.py) example trained using `MirroredStrategy` and `TPUStrategy`.\n",
+        "1. [BERT](https://github.com/tensorflow/models/blob/master/official/legacy/bert/run_classifier.py) example trained using `MirroredStrategy` and `TPUStrategy`.\n",
         "This example is particularly helpful for understanding how to load from a checkpoint and generate periodic checkpoints during distributed training etc.\n",
         "2. [NCF](https://github.com/tensorflow/models/blob/master/official/recommendation/ncf_keras_main.py) example trained using `MirroredStrategy` that can be enabled using the `keras_use_ctl` flag.\n",
         "3. [NMT](https://github.com/tensorflow/examples/blob/master/tensorflow_examples/models/nmt_with_attention/distributed_train.py) example trained using `MirroredStrategy`.\n",

From e85505a560b461eb3d50bcd86d5fb01852eb2dc8 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sat, 16 Jul 2022 16:05:29 -0700
Subject: [PATCH 222/872] Update link to subclassing guide in TF 2 quickstart
 for experts tutorial

---
 site/en/tutorials/quickstart/advanced.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/quickstart/advanced.ipynb b/site/en/tutorials/quickstart/advanced.ipynb
index 923d9dd50ad..62e05de61bd 100644
--- a/site/en/tutorials/quickstart/advanced.ipynb
+++ b/site/en/tutorials/quickstart/advanced.ipynb
@@ -165,7 +165,7 @@
         "id": "BPZ68wASog_I"
       },
       "source": [
-        "Build the `tf.keras` model using the Keras [model subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models#the_model_class):"
+        "Build the `tf.keras` model using the Keras [model subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models):"
       ]
     },
     {

From d6fb0a812acbee1e1849f7de076ccd8699338fb9 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Sat, 16 Jul 2022 16:19:18 -0700
Subject: [PATCH 223/872] Lint Learned data compression tutorial

---
 site/en/tutorials/generative/data_compression.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
index 6f47811f6a2..f8a5fe3d201 100644
--- a/site/en/tutorials/generative/data_compression.ipynb
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -705,7 +705,7 @@
         "id": "JWo0Q-vy23tt"
       },
       "source": [
-        "Display each of 16 original digits together with its compressed binary representation, and the reconstructed digit."
+        "Display each of the 16 original digits together with its compressed binary representation, and the reconstructed digit."
       ]
     },
     {
@@ -757,7 +757,7 @@
       "source": [
         "Note that the length of the encoded string differs from the information content of each digit.\n",
         "\n",
-        "This is because the range coding process works with discretized probabilities, and has a small amount of overhead. So, especially for short strings, the correspondence is only approximate. However, range coding is **asymptotically optimal**: in the limit, the expected bit count will approach the cross entropy (the expected information content), for which the rate term in the training model is an upper bound."
+        "This is because the range coding process works with discrete probabilities, and has a small amount of overhead. So, especially for short strings, the correspondence is only approximate. However, range coding is **asymptotically optimal**: in the limit, the expected bit count will approach the cross entropy (the expected information content), for which the rate term in the training model is an upper bound."
       ]
     },
     {

From f582cf5ac31dd1eada13b5c7252504cc9df2f70c Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Sat, 16 Jul 2022 23:19:48 +0000
Subject: [PATCH 224/872] nbfmt

---
 site/en/guide/distributed_training.ipynb           | 2 --
 site/en/guide/migrate/fault_tolerance.ipynb        | 2 +-
 site/en/tutorials/distribute/custom_training.ipynb | 1 -
 3 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/site/en/guide/distributed_training.ipynb b/site/en/guide/distributed_training.ipynb
index 96c98edc82a..851b6027ce6 100644
--- a/site/en/guide/distributed_training.ipynb
+++ b/site/en/guide/distributed_training.ipynb
@@ -900,8 +900,6 @@
         "Tce3stUlHN0L"
       ],
       "name": "distributed_training.ipynb",
-      "private_outputs": true,
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {
diff --git a/site/en/guide/migrate/fault_tolerance.ipynb b/site/en/guide/migrate/fault_tolerance.ipynb
index 1109db36840..de09eba0588 100644
--- a/site/en/guide/migrate/fault_tolerance.ipynb
+++ b/site/en/guide/migrate/fault_tolerance.ipynb
@@ -13,6 +13,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
+        "cellView": "form",
         "id": "HMUDt0CiUJk9"
       },
       "outputs": [],
@@ -467,7 +468,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "fault_tolerance.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {
diff --git a/site/en/tutorials/distribute/custom_training.ipynb b/site/en/tutorials/distribute/custom_training.ipynb
index 9dbb2d3fad5..672c06b1dbb 100644
--- a/site/en/tutorials/distribute/custom_training.ipynb
+++ b/site/en/tutorials/distribute/custom_training.ipynb
@@ -699,7 +699,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "custom_training.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 184f3cb0c081efb9dd3bb834d6917ec2722dd5ed Mon Sep 17 00:00:00 2001
From: SuperKenVery <39673849+SuperKenVery@users.noreply.github.com>
Date: Mon, 18 Jul 2022 12:38:28 +0800
Subject: [PATCH 225/872] tutorials/keras/text_classification.ipynb: Refine
 exercise instruction

Add an instruction to add a Softmax layer. Otherwise the accuracy would be around 0.3, not far away from the random-guess 0.25.
---
 .../tutorials/keras/text_classification.ipynb | 162 ++++++++++++++----
 1 file changed, 130 insertions(+), 32 deletions(-)

diff --git a/site/en/tutorials/keras/text_classification.ipynb b/site/en/tutorials/keras/text_classification.ipynb
index 3dabeeff095..04885a980a8 100644
--- a/site/en/tutorials/keras/text_classification.ipynb
+++ b/site/en/tutorials/keras/text_classification.ipynb
@@ -14,7 +14,10 @@
       "execution_count": null,
       "metadata": {
         "cellView": "form",
-        "id": "ioaprt5q5US7"
+        "id": "ioaprt5q5US7",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -36,7 +39,10 @@
       "execution_count": null,
       "metadata": {
         "cellView": "form",
-        "id": "yCl0eTNH5RS3"
+        "id": "yCl0eTNH5RS3",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -107,7 +113,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "8RZOuS9LWQvv"
+        "id": "8RZOuS9LWQvv",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -126,7 +135,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "6-tTFS04dChr"
+        "id": "6-tTFS04dChr",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -161,7 +173,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "k7ZYnuajVlFN"
+        "id": "k7ZYnuajVlFN",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -178,7 +193,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "355CfOvsV1pl"
+        "id": "355CfOvsV1pl",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -189,7 +207,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "7ASND15oXpF1"
+        "id": "7ASND15oXpF1",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -210,7 +231,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "R7g8hFvzWLIZ"
+        "id": "R7g8hFvzWLIZ",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -253,7 +277,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "VhejsClzaWfl"
+        "id": "VhejsClzaWfl",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -278,7 +305,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "nOrK-MTYaw3C"
+        "id": "nOrK-MTYaw3C",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -306,7 +336,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "51wNaPPApk1K"
+        "id": "51wNaPPApk1K",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -331,7 +364,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "MlICTG8spyO2"
+        "id": "MlICTG8spyO2",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -361,7 +397,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "JsMwwhOoqjKF"
+        "id": "JsMwwhOoqjKF",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -377,7 +416,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "rdSr0Nt3q_ns"
+        "id": "rdSr0Nt3q_ns",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -414,7 +456,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "SDRI_s_tX1Hk"
+        "id": "SDRI_s_tX1Hk",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -441,7 +486,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "-c76RvSzsMnX"
+        "id": "-c76RvSzsMnX",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -477,7 +525,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "GH4_2ZGJsa_X"
+        "id": "GH4_2ZGJsa_X",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -499,7 +550,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "SCIg_T50wOCU"
+        "id": "SCIg_T50wOCU",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -512,7 +566,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "XULcm6B3xQIO"
+        "id": "XULcm6B3xQIO",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -537,7 +594,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "kRq9hTQzhVhW"
+        "id": "kRq9hTQzhVhW",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -559,7 +619,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "2zhmpeViI1iG"
+        "id": "2zhmpeViI1iG",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -589,7 +652,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "wMcs_H7izm5m"
+        "id": "wMcs_H7izm5m",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -615,7 +681,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "dkQP6in8yUBR"
+        "id": "dkQP6in8yUBR",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -626,7 +695,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "xpKOoWgu-llD"
+        "id": "xpKOoWgu-llD",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -671,7 +743,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "Mr0GP-cQ-llN"
+        "id": "Mr0GP-cQ-llN",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -695,7 +770,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "tXSGrjWZ-llW"
+        "id": "tXSGrjWZ-llW",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -721,7 +799,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "zOMKywn4zReN"
+        "id": "zOMKywn4zReN",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -755,7 +836,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "-YcvZsdvWfDf"
+        "id": "-YcvZsdvWfDf",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -776,7 +860,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "2SEMeQ5YXs8z"
+        "id": "2SEMeQ5YXs8z",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -803,7 +890,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "Z3PJemLPXwz_"
+        "id": "Z3PJemLPXwz_",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -847,7 +937,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "FWXsMvryuZuq"
+        "id": "FWXsMvryuZuq",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -881,7 +974,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "QW355HH5L49K"
+        "id": "QW355HH5L49K",
+        "vscode": {
+          "languageId": "python"
+        }
       },
       "outputs": [],
       "source": [
@@ -947,6 +1043,8 @@
         "\n",
         "1. Modify the last layer of your model to `Dense(4)`, as there are now four output classes.\n",
         "\n",
+        "1. Add a layer `tf.keras.layers.Softmax()` to the end of the list. \n",
+        "\n",
         "1. When compiling the model, change the loss to `tf.keras.losses.SparseCategoricalCrossentropy`. This is the correct loss function to use for a multi-class classification problem, when the labels for each class are integers (in this case, they can be 0, *1*, *2*, or *3*). In addition, change the metrics to `metrics=['accuracy']`, since this is a multi-class classification problem (`tf.metrics.BinaryAccuracy` is only used for binary classifiers).\n",
         "\n",
         "1. When plotting accuracy over time, change `binary_accuracy` and `val_binary_accuracy` to `accuracy` and `val_accuracy`, respectively.\n",

From f528efa3ba306c32c0d0e88dc4c7c2439a55906c Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Mon, 18 Jul 2022 04:40:19 +0000
Subject: [PATCH 226/872] nbfmt

---
 .../tutorials/keras/text_classification.ipynb | 160 ++++--------------
 1 file changed, 32 insertions(+), 128 deletions(-)

diff --git a/site/en/tutorials/keras/text_classification.ipynb b/site/en/tutorials/keras/text_classification.ipynb
index 04885a980a8..c174d6bb47c 100644
--- a/site/en/tutorials/keras/text_classification.ipynb
+++ b/site/en/tutorials/keras/text_classification.ipynb
@@ -14,10 +14,7 @@
       "execution_count": null,
       "metadata": {
         "cellView": "form",
-        "id": "ioaprt5q5US7",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "ioaprt5q5US7"
       },
       "outputs": [],
       "source": [
@@ -39,10 +36,7 @@
       "execution_count": null,
       "metadata": {
         "cellView": "form",
-        "id": "yCl0eTNH5RS3",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "yCl0eTNH5RS3"
       },
       "outputs": [],
       "source": [
@@ -113,10 +107,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "8RZOuS9LWQvv",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "8RZOuS9LWQvv"
       },
       "outputs": [],
       "source": [
@@ -135,10 +126,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "6-tTFS04dChr",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "6-tTFS04dChr"
       },
       "outputs": [],
       "source": [
@@ -173,10 +161,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "k7ZYnuajVlFN",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "k7ZYnuajVlFN"
       },
       "outputs": [],
       "source": [
@@ -193,10 +178,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "355CfOvsV1pl",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "355CfOvsV1pl"
       },
       "outputs": [],
       "source": [
@@ -207,10 +189,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "7ASND15oXpF1",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "7ASND15oXpF1"
       },
       "outputs": [],
       "source": [
@@ -231,10 +210,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "R7g8hFvzWLIZ",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "R7g8hFvzWLIZ"
       },
       "outputs": [],
       "source": [
@@ -277,10 +253,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "VhejsClzaWfl",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "VhejsClzaWfl"
       },
       "outputs": [],
       "source": [
@@ -305,10 +278,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "nOrK-MTYaw3C",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "nOrK-MTYaw3C"
       },
       "outputs": [],
       "source": [
@@ -336,10 +306,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "51wNaPPApk1K",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "51wNaPPApk1K"
       },
       "outputs": [],
       "source": [
@@ -364,10 +331,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "MlICTG8spyO2",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "MlICTG8spyO2"
       },
       "outputs": [],
       "source": [
@@ -397,10 +361,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "JsMwwhOoqjKF",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "JsMwwhOoqjKF"
       },
       "outputs": [],
       "source": [
@@ -416,10 +377,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "rdSr0Nt3q_ns",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "rdSr0Nt3q_ns"
       },
       "outputs": [],
       "source": [
@@ -456,10 +414,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "SDRI_s_tX1Hk",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "SDRI_s_tX1Hk"
       },
       "outputs": [],
       "source": [
@@ -486,10 +441,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "-c76RvSzsMnX",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "-c76RvSzsMnX"
       },
       "outputs": [],
       "source": [
@@ -525,10 +477,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "GH4_2ZGJsa_X",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "GH4_2ZGJsa_X"
       },
       "outputs": [],
       "source": [
@@ -550,10 +499,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "SCIg_T50wOCU",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "SCIg_T50wOCU"
       },
       "outputs": [],
       "source": [
@@ -566,10 +512,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "XULcm6B3xQIO",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "XULcm6B3xQIO"
       },
       "outputs": [],
       "source": [
@@ -594,10 +537,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "kRq9hTQzhVhW",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "kRq9hTQzhVhW"
       },
       "outputs": [],
       "source": [
@@ -619,10 +559,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "2zhmpeViI1iG",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "2zhmpeViI1iG"
       },
       "outputs": [],
       "source": [
@@ -652,10 +589,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "wMcs_H7izm5m",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "wMcs_H7izm5m"
       },
       "outputs": [],
       "source": [
@@ -681,10 +615,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "dkQP6in8yUBR",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "dkQP6in8yUBR"
       },
       "outputs": [],
       "source": [
@@ -695,10 +626,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "xpKOoWgu-llD",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "xpKOoWgu-llD"
       },
       "outputs": [],
       "source": [
@@ -743,10 +671,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "Mr0GP-cQ-llN",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "Mr0GP-cQ-llN"
       },
       "outputs": [],
       "source": [
@@ -770,10 +695,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "tXSGrjWZ-llW",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "tXSGrjWZ-llW"
       },
       "outputs": [],
       "source": [
@@ -799,10 +721,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "zOMKywn4zReN",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "zOMKywn4zReN"
       },
       "outputs": [],
       "source": [
@@ -836,10 +755,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "-YcvZsdvWfDf",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "-YcvZsdvWfDf"
       },
       "outputs": [],
       "source": [
@@ -860,10 +776,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "2SEMeQ5YXs8z",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "2SEMeQ5YXs8z"
       },
       "outputs": [],
       "source": [
@@ -890,10 +803,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "Z3PJemLPXwz_",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "Z3PJemLPXwz_"
       },
       "outputs": [],
       "source": [
@@ -937,10 +847,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "FWXsMvryuZuq",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "FWXsMvryuZuq"
       },
       "outputs": [],
       "source": [
@@ -974,10 +881,7 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "QW355HH5L49K",
-        "vscode": {
-          "languageId": "python"
-        }
+        "id": "QW355HH5L49K"
       },
       "outputs": [],
       "source": [

From f3c562afd9ad21936121d88bd9f4f6390e3ab0d7 Mon Sep 17 00:00:00 2001
From: SuperKenVery <39673849+SuperKenVery@users.noreply.github.com>
Date: Mon, 18 Jul 2022 15:37:20 +0800
Subject: [PATCH 227/872] tutorials/keras/text_classification.ipynb: Fix type

without the `()` there will be some strange errors
---
 site/en/tutorials/keras/text_classification.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/keras/text_classification.ipynb b/site/en/tutorials/keras/text_classification.ipynb
index c174d6bb47c..759e59b500d 100644
--- a/site/en/tutorials/keras/text_classification.ipynb
+++ b/site/en/tutorials/keras/text_classification.ipynb
@@ -949,7 +949,7 @@
         "\n",
         "1. Add a layer `tf.keras.layers.Softmax()` to the end of the list. \n",
         "\n",
-        "1. When compiling the model, change the loss to `tf.keras.losses.SparseCategoricalCrossentropy`. This is the correct loss function to use for a multi-class classification problem, when the labels for each class are integers (in this case, they can be 0, *1*, *2*, or *3*). In addition, change the metrics to `metrics=['accuracy']`, since this is a multi-class classification problem (`tf.metrics.BinaryAccuracy` is only used for binary classifiers).\n",
+        "1. When compiling the model, change the loss to `tf.keras.losses.SparseCategoricalCrossentropy()`. This is the correct loss function to use for a multi-class classification problem, when the labels for each class are integers (in this case, they can be 0, *1*, *2*, or *3*). In addition, change the metrics to `metrics=['accuracy']`, since this is a multi-class classification problem (`tf.metrics.BinaryAccuracy` is only used for binary classifiers).\n",
         "\n",
         "1. When plotting accuracy over time, change `binary_accuracy` and `val_binary_accuracy` to `accuracy` and `val_accuracy`, respectively.\n",
         "\n",

From 74af507b8ea4bcdc8c5e24403b6920e9957f5891 Mon Sep 17 00:00:00 2001
From: Olzhas Akpambetov 
Date: Mon, 18 Jul 2022 17:34:13 -0700
Subject: [PATCH 228/872] Fix class names, lint Fault tolerance migration and
 Multi-worker training with Keras doc

PiperOrigin-RevId: 461755165
---
 site/en/guide/migrate/fault_tolerance.ipynb   | 18 +++---
 .../distribute/multi_worker_with_keras.ipynb  | 60 +++++++++----------
 2 files changed, 39 insertions(+), 39 deletions(-)

diff --git a/site/en/guide/migrate/fault_tolerance.ipynb b/site/en/guide/migrate/fault_tolerance.ipynb
index a44b113aeaa..fdbd0b972c3 100644
--- a/site/en/guide/migrate/fault_tolerance.ipynb
+++ b/site/en/guide/migrate/fault_tolerance.ipynb
@@ -91,7 +91,7 @@
         "id": "TOVQubuDzdmA"
       },
       "source": [
-        "Install `tf-nightly` as step Granularity for `BackupAndRestore` callback is only available in Tensorflow 2.10"
+        "Install `tf-nightly`, as the frequency of checkpoint saving at a particular step with the `save_freq` argument in `tf.keras.callbacks.BackupAndRestore` is introduced from TensorFlow 2.10:"
       ]
     },
     {
@@ -140,7 +140,7 @@
         "id": "TtlucRG_Uro_"
       },
       "source": [
-        "## TensorFlow 1: Save checkpoints with tf.estimator.RunConfig\n",
+        "## TensorFlow 1: Save checkpoints with `tf.estimator.RunConfig`\n",
         "\n",
         "In TensorFlow 1, you can configure a `tf.estimator` to save checkpoints every step by configuring `tf.estimator.RunConfig`.\n",
         "\n",
@@ -271,11 +271,11 @@
         "id": "T5LtVtmvYx7J"
       },
       "source": [
-        "## TensorFlow 2: Back up and restore with a callback and Model.fit\n",
+        "## TensorFlow 2: Back up and restore with a callback and `Model.fit`\n",
         "\n",
         "In TensorFlow 2, if you use the Keras `Model.fit` API for training, you can provide the `tf.keras.callbacks.BackupAndRestore` callback to add the fault tolerance functionality.\n",
         "\n",
-        "To help demonstrate this, let's first start by defining a callback class that artificially throws an error during the fourth epoch checkpoint:\n"
+        "To help demonstrate this, first start by defining a Keras `Callback` class that artificially throws an error during the fourth epoch checkpoint:\n"
       ]
     },
     {
@@ -336,7 +336,7 @@
         "id": "LRRWmZqsvMrq"
       },
       "source": [
-        "Now, start training the model with `Model.fit`. During training, checkpoints will be saved thanks to the `backup_restore_callback` defined above, while the `InterruptingCallbackAtEpoch` will raise an artificial exception to simulate a failure after 4th epoch."
+        "Start training the model with `Model.fit`. During training, checkpoints will be saved thanks to `tf.keras.callbacks.BackupAndRestore` instantiated above, while the `InterruptAtEpoch` class will raise an artificial exception to simulate a failure after the fourth epoch."
       ]
     },
     {
@@ -394,7 +394,7 @@
         "id": "nP2dnpMPxtYj"
       },
       "source": [
-        "let's define another callback class that artificially throws an error during the 140th step."
+        "Define another `Callback` class that artificially throws an error during the 140th step:"
       ]
     },
     {
@@ -426,9 +426,9 @@
         "id": "Af3VpehxyTpb"
       },
       "source": [
-        "Note: This section uses features that are only available in `tf-nightly`(until Tensorflow 2.10 is released).\n",
+        "Note: This section uses features that are only available in `tf-nightly` until Tensorflow 2.10 is released.\n",
         "\n",
-        "Now, lets set `save_freq` in `BackupAndRestore` to an integer value 30. Checkpoints will be saved every 30 steps. The `InterruptingCallbackAtSteps` will raise an artificial exception to simulate a failure at epoch 1 and step 40 (total step 140). The checkpoint would be last saved at epoch 1 step 20."
+        "To make sure the checkpoints are saved every 30 steps, set the `save_freq` in the `BackupAndRestore` callback to `30`. The `InterruptAtStep` will raise an artificial exception to simulate a failure at epoch 1 and step 40 (total step count 140). The checkpoint would be last saved at epoch 1 and step 20."
       ]
     },
     {
@@ -465,7 +465,7 @@
         "id": "2-ggMFEHynMR"
       },
       "source": [
-        "Next, instantiate the Keras model, call `Model.compile`, and continue training the model with `Model.fit` from a previously saved checkpoint: notice that the training starts from epoch 2 and step 21."
+        "Next, instantiate the Keras model, call `Model.compile`, and continue training the model with `Model.fit` from a previously saved checkpoint. Notice that the training starts from epoch 2 and step 21."
       ]
     },
     {
diff --git a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
index 1cbec8e2fb8..753bdb8afe3 100644
--- a/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
+++ b/site/en/tutorials/distribute/multi_worker_with_keras.ipynb
@@ -127,7 +127,7 @@
       "source": [
         "Before importing TensorFlow, make a few changes to the environment:\n",
         "\n",
-        "* In a real-world application, each worker would be on a different machine. For the purposes of this tutorial, all the workers will run on the **this** machine. So disable all GPUs to prevents errors caused by all workers trying to use the same GPU."
+        "* In a real-world application, each worker would be on a different machine. For the purposes of this tutorial, all the workers will run on the **this** machine. Therefore, disable all GPUs to prevent errors caused by all workers trying to use the same GPU."
       ]
     },
     {
@@ -182,22 +182,13 @@
         "  sys.path.insert(0, '.')"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "pDhHuMjb7bfU"
-      },
-      "source": [
-        "Finally, import TensorFlow:"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "9hLpDZhAz2q-"
       },
       "source": [
-        "Install `tf-nightly` as step Granularity for `BackupAndRestore` callback is only available in Tensorflow 2.10"
+        "Install `tf-nightly`, as the frequency of checkpoint saving at a particular step with the `save_freq` argument in `tf.keras.callbacks.BackupAndRestore` is introduced from TensorFlow 2.10:"
       ]
     },
     {
@@ -211,6 +202,15 @@
         "!pip install tf-nightly"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "524e38dab658"
+      },
+      "source": [
+        "Finally, import TensorFlow:"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -921,7 +921,7 @@
         "id": "KvHPjGlyyFt6"
       },
       "source": [
-        "#### ModelCheckpoint callback\n",
+        "#### The `ModelCheckpoint` callback\n",
         "\n",
         "`ModelCheckpoint` callback no longer provides fault tolerance functionality, please use [`BackupAndRestore`](#scrollTo=kmH8uCUhfn4w) callback instead.\n",
         "\n",
@@ -949,10 +949,10 @@
         "\n",
         "You should have some cleanup logic that deletes the temporary directories created by the workers once your training has completed.\n",
         "\n",
-        "The reason for saving on the chief and workers at the same time is because you might be aggregating variables during checkpointing which requires both the chief and workers to participate in the allreduce communication protocol. On the other hand, letting chief and workers save to the same model directory will result in errors due to contention.\n",
+        "The reason for saving on the chief and workers at the same time is because you might be aggregating variables during checkpointing, which requires both the chief and workers to participate in the allreduce communication protocol. On the other hand, letting chief and workers save to the same model directory will result in errors due to contention.\n",
         "\n",
-        "Using the `MultiWorkerMirroredStrategy`, the program is run on every worker, and in order to know whether the current worker is chief, it takes advantage of the cluster resolver object that has attributes `task_type` and `task_id`:\n",
-        "- `task_type` tells you what the current job is (e.g. `'worker'`).\n",
+        "Using the `MultiWorkerMirroredStrategy`, the program is run on every worker, and in order to know whether the current worker is the chief, it takes advantage of the cluster resolver object that has attributes `task_type` and `task_id`:\n",
+        "- `task_type` tells you what the current job is (for example, `'worker'`).\n",
         "- `task_id` tells you the identifier of the worker.\n",
         "- The worker with `task_id == 0` is designated as the chief worker.\n",
         "\n",
@@ -973,14 +973,14 @@
         "model_path = '/tmp/keras-model'\n",
         "\n",
         "def _is_chief(task_type, task_id):\n",
-        "  # Note: there are two possible `TF_CONFIG` configuration.\n",
+        "  # Note: there are two possible `TF_CONFIG` configurations.\n",
         "  #   1) In addition to `worker` tasks, a `chief` task type is use;\n",
         "  #      in this case, this function should be modified to\n",
         "  #      `return task_type == 'chief'`.\n",
         "  #   2) Only `worker` task type is used; in this case, worker 0 is\n",
         "  #      regarded as the chief. The implementation demonstrated here\n",
         "  #      is for this case.\n",
-        "  # For the purpose of this Colab section, the `task_type is None` case\n",
+        "  # For the purpose of this Colab section, the `task_type` is `None` case\n",
         "  # is added because it is effectively run with only a single worker.\n",
         "  return (task_type == 'worker' and task_id == 0) or task_type is None\n",
         "\n",
@@ -1028,7 +1028,7 @@
         "id": "8LXUVVl9_v5x"
       },
       "source": [
-        "As described above, later on the model should only be loaded from the path chief saved to, so let's remove the temporary ones the non-chief workers saved:"
+        "As described above, later on the model should only be loaded from the file path the chief worker saved to. Therefore, remove the temporary ones the non-chief workers have saved:"
       ]
     },
     {
@@ -1049,7 +1049,7 @@
         "id": "Nr-2PKlHAPBT"
       },
       "source": [
-        "Now, when it's time to load, let's use convenient `tf.keras.models.load_model` API, and continue with further work.\n",
+        "Now, when it's time to load, use the convenient `tf.keras.models.load_model` API, and continue with further work.\n",
         "\n",
         "Here, assume only using single worker to load and continue training, in which case you do not call `tf.keras.models.load_model` within another `strategy.scope()` (note that `strategy = tf.distribute.MultiWorkerMirroredStrategy()`, as defined earlier):"
       ]
@@ -1147,15 +1147,15 @@
         "id": "kmH8uCUhfn4w"
       },
       "source": [
-        "#### BackupAndRestore callback\n",
+        "#### The `BackupAndRestore` callback\n",
         "\n",
-        "The tf.keras.callbacks.BackupAndRestore callback provides the fault tolerance functionality by backing up the model and current training state in a temporary checkpoint file under `backup_dir` argument to `BackupAndRestore`. In the current stable Tensorflow 2.9 version, the current model and traning state is backed up at epoch boundaries.\n",
+        "The `tf.keras.callbacks.BackupAndRestore` callback provides the fault tolerance functionality by backing up the model and current training state in a temporary checkpoint file under `backup_dir` argument to `BackupAndRestore`. \n",
         "\n",
-        "In `tf-nightly` version, `BackupAndRestore` can back up the model and training state at epoch or step boundaries. `BackupAndRestore` accepts an optional `save_freq` argument. `save_freq` accepts either `'epoch'` or an `int` value. If `save_freq` is set to `'epoch'` the model is backed up after every epoch. If `save_freq` is set to an int value greater than 0, the model is backed up after every `save_freq` number of batches.\n",
+        "Note: In Tensorflow 2.9, the current model and the training state is backed up at epoch boundaries. In the `tf-nightly` version and from TensorFlow 2.10, the `BackupAndRestore` callback can back up the model and the training state at epoch or step boundaries. `BackupAndRestore` accepts an optional `save_freq` argument. `save_freq` accepts either `'epoch'` or an `int` value. If `save_freq` is set to `'epoch'` the model is backed up after every epoch. If `save_freq` is set to an integer value greater than `0`, the model is backed up after every `save_freq` number of batches.\n",
         "\n",
-        "Once jobs get interrupted and restarted, the callback restores the last checkpoint, and training continues from the beginning of the epoch and step at which the traning state was last saved.\n",
+        "Once the jobs get interrupted and restarted, the `BackupAndRestore` callback restores the last checkpoint, and you can continue training from the beginning of the epoch and step at which the training state was last saved.\n",
         "\n",
-        "To use it, provide an instance of tf.keras.callbacks.BackupAndRestore at the Model.fit call.\n",
+        "To use it, provide an instance of `tf.keras.callbacks.BackupAndRestore` at the `Model.fit` call.\n",
         "\n",
         "With `MultiWorkerMirroredStrategy`, if a worker gets interrupted, the whole cluster will pause until the interrupted worker is restarted. Other workers will also restart, and the interrupted worker will rejoin the cluster. Then, every worker will read the checkpoint file that was previously saved and pick up its former state, thereby allowing the cluster to get back in sync. Then, the training will continue. The distributed dataset iterator state will be re-initialized and not restored.\n",
         "\n",
@@ -1193,7 +1193,7 @@
         "id": "f8e86TAp0Rsl"
       },
       "source": [
-        "If `save_freq` is set to `'epoch'` the model is backed up after every epoch."
+        "If the `save_freq` argument in the `BackupAndRestore` callback is set to `'epoch'`, the model is backed up after every epoch."
       ]
     },
     {
@@ -1204,7 +1204,7 @@
       },
       "outputs": [],
       "source": [
-        "# The training stateis backed up at epoch boundaries because `save_freq` is\n",
+        "# The training state is backed up at epoch boundaries because `save_freq` is\n",
         "# set to `epoch`.\n",
         "\n",
         "callbacks = [tf.keras.callbacks.BackupAndRestore(backup_dir='/tmp/backup')]\n",
@@ -1222,9 +1222,9 @@
         "id": "p-r44kCM0jc6"
       },
       "source": [
-        "Note: The next code block uses fatures that are only available in `tf-nightly`(until Tensorflow 2.10 is released)\n",
+        "Note: The next code block uses features that are only available in `tf-nightly` until Tensorflow 2.10 is released.\n",
         "\n",
-        "If `save_freq` is set to an int value greater than 0, the model is backed up after every `save_freq` number of batches."
+        "If the `save_freq` argument in the `BackupAndRestore` callback is set to an integer value greater than `0`, the model is backed up after every `save_freq` number of batches."
       ]
     },
     {
@@ -1235,8 +1235,8 @@
       },
       "outputs": [],
       "source": [
-        "# The training state is backed up at every 30 steps because save_freq is set\n",
-        "# an integer value of 30.\n",
+        "# The training state is backed up at every 30 steps because `save_freq` is set\n",
+        "# to an integer value of `30`.\n",
         "\n",
         "callbacks = [tf.keras.callbacks.BackupAndRestore(backup_dir='/tmp/backup', save_freq=30)]\n",
         "with strategy.scope():\n",

From e08a976dc9fa3f8211a7001e007f851df519ff02 Mon Sep 17 00:00:00 2001
From: Vincent Gurgul <74213413+VincentGurgul@users.noreply.github.com>
Date: Tue, 19 Jul 2022 16:10:14 +0200
Subject: [PATCH 229/872] Fixed typo in generate_training_data

"generate_training_data" function used global variable "SEED" even
though it takes a variable "seed" as its argument
---
 site/en/tutorials/text/word2vec.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/text/word2vec.ipynb b/site/en/tutorials/text/word2vec.ipynb
index 113b9419662..fd9accdfac9 100644
--- a/site/en/tutorials/text/word2vec.ipynb
+++ b/site/en/tutorials/text/word2vec.ipynb
@@ -747,7 +747,7 @@
         "          num_sampled=num_ns,\n",
         "          unique=True,\n",
         "          range_max=vocab_size,\n",
-        "          seed=SEED,\n",
+        "          seed=seed,\n",
         "          name=\"negative_sampling\")\n",
         "\n",
         "      # Build context and label vectors (for one target word)\n",

From 65b113fc60060297a924435bd4422c90221f2679 Mon Sep 17 00:00:00 2001
From: Binjian 
Date: Wed, 20 Jul 2022 08:41:06 +0800
Subject: [PATCH 230/872] Update function.ipynb

tried on colab,
some of the features run only on tensorflow >= 2.9.1
you might need to install the nightly built version in order to try this tutorial.
---
 site/en/guide/function.ipynb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index 20514875cf4..896b7aae763 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -91,6 +91,7 @@
       },
       "outputs": [],
       "source": [
+        "!pip install -q tf_nightly\n",
         "import tensorflow as tf"
       ]
     },

From c63269bbb0f5ec921cbe577895d9d2adb832885d Mon Sep 17 00:00:00 2001
From: Binjian 
Date: Wed, 20 Jul 2022 16:04:39 +0800
Subject: [PATCH 231/872] Update extension_type.ipynb

It'll make more sense to use != than == to mask the zeros.
---
 site/en/guide/extension_type.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/extension_type.ipynb b/site/en/guide/extension_type.ipynb
index 542450945d1..6edbf1577e4 100644
--- a/site/en/guide/extension_type.ipynb
+++ b/site/en/guide/extension_type.ipynb
@@ -768,7 +768,7 @@
         "\n",
         "  @staticmethod\n",
         "  def from_tensor_and_value_to_mask(values, value_to_mask):\n",
-        "    return MaskedTensor(values, values == value_to_mask)\n",
+        "    return MaskedTensor(values, values != value_to_mask)\n",
         "\n",
         "x = tf.constant([[1, 0, 2], [3, 0, 0]])\n",
         "MaskedTensor.from_tensor_and_value_to_mask(x, 0)"

From 349bc33312bd44ccabfad3ed889ac247f4f3ffe8 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Fri, 22 Jul 2022 16:01:08 -0700
Subject: [PATCH 232/872] Adding a space between `data` and `rate`

PiperOrigin-RevId: 462723079
---
 site/en/tutorials/audio/transfer_learning_audio.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/audio/transfer_learning_audio.ipynb b/site/en/tutorials/audio/transfer_learning_audio.ipynb
index 4a24598e991..44c9c33d46b 100644
--- a/site/en/tutorials/audio/transfer_learning_audio.ipynb
+++ b/site/en/tutorials/audio/transfer_learning_audio.ipynb
@@ -235,7 +235,7 @@
         "_ = plt.plot(testing_wav_data)\n",
         "\n",
         "# Play the audio file.\n",
-        "display.Audio(testing_wav_data,rate=16000)"
+        "display.Audio(testing_wav_data, rate=16000)"
       ]
     },
     {

From 0b890f134a8ff1c80d0272f4b496a2573eafbcba Mon Sep 17 00:00:00 2001
From: Rajesh Nakka <57556866+338rajesh@users.noreply.github.com>
Date: Sun, 24 Jul 2022 11:26:26 +0530
Subject: [PATCH 233/872] Update autodiff.ipynb

Getting a gradient of `None` sounds like the section talks about the procedure for evaluating a gradient of `None`
---
 site/en/guide/autodiff.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/autodiff.ipynb b/site/en/guide/autodiff.ipynb
index 5571079fed5..c063e9e0b5b 100644
--- a/site/en/guide/autodiff.ipynb
+++ b/site/en/guide/autodiff.ipynb
@@ -746,9 +746,9 @@
         "id": "egypBxISAHhx"
       },
       "source": [
-        "## Getting a gradient of `None`\n",
+        "## Cases where gradient returns `None`\n",
         "\n",
-        "When a target is not connected to a source you will get a gradient of `None`.\n"
+        "When a target is not connected to a source, gradient will return `None`.\n"
       ]
     },
     {

From 8780b9af2f414ae3668b8d1e8562546cd7b59e26 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 25 Jul 2022 06:39:51 -0700
Subject: [PATCH 234/872] Add backticks.

---
 site/en/guide/autodiff.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/autodiff.ipynb b/site/en/guide/autodiff.ipynb
index c063e9e0b5b..237a224569b 100644
--- a/site/en/guide/autodiff.ipynb
+++ b/site/en/guide/autodiff.ipynb
@@ -746,9 +746,9 @@
         "id": "egypBxISAHhx"
       },
       "source": [
-        "## Cases where gradient returns `None`\n",
+        "## Cases where `gradient` returns `None`\n",
         "\n",
-        "When a target is not connected to a source, gradient will return `None`.\n"
+        "When a target is not connected to a source, `gradient` will return `None`.\n"
       ]
     },
     {

From 26607a2ab94903fac9b409364c6f7c18c4e57e6c Mon Sep 17 00:00:00 2001
From: chunduriv <74177924+chunduriv@users.noreply.github.com>
Date: Mon, 25 Jul 2022 19:47:41 +0530
Subject: [PATCH 235/872] Update Broken References

Update Broken References:
1.  Change from `tf.experimental.dispatch_for_unary_elementwise_api` to `tf.experimental.dispatch_for_unary_elementwise_apis`
2.  Change from `tf.Keras` to `tf.Keras`
---
 site/en/guide/extension_type.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/extension_type.ipynb b/site/en/guide/extension_type.ipynb
index 542450945d1..0b56d904bb0 100644
--- a/site/en/guide/extension_type.ipynb
+++ b/site/en/guide/extension_type.ipynb
@@ -1056,7 +1056,7 @@
         "Extension types can be \"tensor-like\", in the sense that they specialize or extend the interface defined by the `tf.Tensor` type. Examples of tensor-like extension types include `RaggedTensor`, `SparseTensor`, and `MaskedTensor`. ***Dispatch decorators*** can be used to override the default behavior of TensorFlow operations when applied to tensor-like extension types. TensorFlow currently defines three dispatch decorators:\n",
         "\n",
         "* `@tf.experimental.dispatch_for_api(tf_api)`\n",
-        "* `@tf.experimental.dispatch_for_unary_elementwise_api(x_type)`\n",
+        "* `@tf.experimental.dispatch_for_unary_elementwise_apis(x_type)`\n",
         "* `@tf.experimental.dispatch_for_binary_elementwise_apis(x_type, y_type)`"
       ]
     },
@@ -1269,7 +1269,7 @@
         "An `ExtensionType` is *batchable* if a single instance can be used to represent a batch of values. Typically, this is accomplished by adding batch dimensions to all nested `Tensor`s. The following TensorFlow APIs require that any extension type inputs be batchable:\n",
         "\n",
         "* `tf.data.Dataset` (`batch`, `unbatch`, `from_tensor_slices`)\n",
-        "* `tf.Keras` (`fit`, `evaluate`, `predict`)\n",
+        "* `tf.keras` (`fit`, `evaluate`, `predict`)\n",
         "* `tf.map_fn`"
       ]
     },

From e6cc6e99b10357cb842fe577351fd86d631890ef Mon Sep 17 00:00:00 2001
From: chunduriv <74177924+chunduriv@users.noreply.github.com>
Date: Mon, 25 Jul 2022 19:57:36 +0530
Subject: [PATCH 236/872] Update Broken References

Change from `tf.Keras` to `tf.keras`
---
 site/en/guide/extension_type.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/extension_type.ipynb b/site/en/guide/extension_type.ipynb
index 0b56d904bb0..cf64a2b70f4 100644
--- a/site/en/guide/extension_type.ipynb
+++ b/site/en/guide/extension_type.ipynb
@@ -1317,7 +1317,7 @@
         "id": "PaOzUev6g3wT"
       },
       "source": [
-        "To make this type batchable, change the base type to `BatchableExtensionType`, and adjust the shape of each field to include optional batch dimensions. The following example also adds a `shape` field to keep track of the batch shape. This `shape` field is not required by `tf.data.Dataset` or `tf.map_fn`, but it *is* required by `tf.Keras`."
+        "To make this type batchable, change the base type to `BatchableExtensionType`, and adjust the shape of each field to include optional batch dimensions. The following example also adds a `shape` field to keep track of the batch shape. This `shape` field is not required by `tf.data.Dataset` or `tf.map_fn`, but it *is* required by `tf.keras`."
       ]
     },
     {

From 0e157483bc0ca3fb4c919085ac2615b4eac1308e Mon Sep 17 00:00:00 2001
From: SuperKenVery <39673849+SuperKenVery@users.noreply.github.com>
Date: Tue, 26 Jul 2022 13:14:22 +0800
Subject: [PATCH 237/872] tutorials/keras/text_classification.ipynb: change
 according to @MarkDaoust

Don't add softmax layer, but use the `tf.keras.losses.SparsedCategoricalCrossentropy(from_logits=True)`
---
 .../en/tutorials/keras/text_classification.ipynb | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/site/en/tutorials/keras/text_classification.ipynb b/site/en/tutorials/keras/text_classification.ipynb
index 759e59b500d..5929b66c3f3 100644
--- a/site/en/tutorials/keras/text_classification.ipynb
+++ b/site/en/tutorials/keras/text_classification.ipynb
@@ -947,9 +947,7 @@
         "\n",
         "1. Modify the last layer of your model to `Dense(4)`, as there are now four output classes.\n",
         "\n",
-        "1. Add a layer `tf.keras.layers.Softmax()` to the end of the list. \n",
-        "\n",
-        "1. When compiling the model, change the loss to `tf.keras.losses.SparseCategoricalCrossentropy()`. This is the correct loss function to use for a multi-class classification problem, when the labels for each class are integers (in this case, they can be 0, *1*, *2*, or *3*). In addition, change the metrics to `metrics=['accuracy']`, since this is a multi-class classification problem (`tf.metrics.BinaryAccuracy` is only used for binary classifiers).\n",
+        "1. When compiling the model, change the loss to `tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)`. This is the correct loss function to use for a multi-class classification problem, when the labels for each class are integers (in this case, they can be 0, *1*, *2*, or *3*). In addition, change the metrics to `metrics=['accuracy']`, since this is a multi-class classification problem (`tf.metrics.BinaryAccuracy` is only used for binary classifiers).\n",
         "\n",
         "1. When plotting accuracy over time, change `binary_accuracy` and `val_binary_accuracy` to `accuracy` and `val_accuracy`, respectively.\n",
         "\n",
@@ -976,8 +974,18 @@
       "toc_visible": true
     },
     "kernelspec": {
-      "display_name": "Python 3",
+      "display_name": "Python 3.9.13 ('base')",
+      "language": "python",
       "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.9.13"
+    },
+    "vscode": {
+      "interpreter": {
+        "hash": "01844806d85f8922708963be5fa80bf6a98a83d65554465eed6a4733f098979f"
+      }
     }
   },
   "nbformat": 4,

From 835c96b62cfda8d3e0ed965db1b746400169ddfa Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Tue, 26 Jul 2022 05:14:59 +0000
Subject: [PATCH 238/872] nbfmt

---
 site/en/tutorials/keras/text_classification.ipynb | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)

diff --git a/site/en/tutorials/keras/text_classification.ipynb b/site/en/tutorials/keras/text_classification.ipynb
index 5929b66c3f3..ff9a0c1d061 100644
--- a/site/en/tutorials/keras/text_classification.ipynb
+++ b/site/en/tutorials/keras/text_classification.ipynb
@@ -974,18 +974,8 @@
       "toc_visible": true
     },
     "kernelspec": {
-      "display_name": "Python 3.9.13 ('base')",
-      "language": "python",
+      "display_name": "Python 3",
       "name": "python3"
-    },
-    "language_info": {
-      "name": "python",
-      "version": "3.9.13"
-    },
-    "vscode": {
-      "interpreter": {
-        "hash": "01844806d85f8922708963be5fa80bf6a98a83d65554465eed6a4733f098979f"
-      }
     }
   },
   "nbformat": 4,

From e429c0232175eeae89cc92dca14fda790f922859 Mon Sep 17 00:00:00 2001
From: Joe Fernandez 
Date: Wed, 27 Jul 2022 09:30:36 -0700
Subject: [PATCH 239/872] Update TensorFlow tutorials page

PiperOrigin-RevId: 463606993
---
 site/en/tutorials/_index.yaml | 46 +++++++++++++----------------------
 1 file changed, 17 insertions(+), 29 deletions(-)

diff --git a/site/en/tutorials/_index.yaml b/site/en/tutorials/_index.yaml
index 718f187990e..c38a23ddb35 100644
--- a/site/en/tutorials/_index.yaml
+++ b/site/en/tutorials/_index.yaml
@@ -84,38 +84,16 @@ landing_page:
   - classname: devsite-landing-row-100
     items:
     - description: >
-        
Video and blog updates

-        Subscribe to the
-        TensorFlow blog,
-        YouTube channel,
-        and Twitter
-        for the latest updates.
+        
Video tutorials

+        Check out these videos for an introduction to machine learning with TensorFlow:
 
   - items:
-    - heading: "Intro to Machine Learning"
-      path: "https://www.youtube.com/watch?v=KNAWp2S3w94"
+    - heading: "TensorFlow ML Zero to Hero"
+      path: "https://www.youtube.com/watch?v=KNAWp2S3w94&list=PLQY2H8rRoyvwWuPiWnuTDBHe7I0fMSsfO"
       youtube_id: "KNAWp2S3w94?rel=0&show_info=0"
-    - heading: "TensorFlow 2.0 and Keras"
-      path: "https://www.youtube.com/watch?v=wGI_VtE9CJM"
-      youtube_id: "wGI_VtE9CJM?rel=0&show_info=0"
-
-  - classname: devsite-landing-row-cards
-    items:
-    - heading: "Looking Back at 2019"
-      path: https://blog.tensorflow.org/2019/12/looking-back-at-2019.html
-      buttons:
-      - label: "Read on the TensorFlow blog"
-        path: https://blog.tensorflow.org/2019/12/looking-back-at-2019.html
-    - heading: "TensorFlow 2 is now available"
-      path: https://blog.tensorflow.org/2019/09/tensorflow-20-is-now-available.html
-      buttons:
-      - label: "Read on the TensorFlow blog"
-        path: https://blog.tensorflow.org/2019/09/tensorflow-20-is-now-available.html
-    - heading: "Standardizing on Keras: Guidance on High-level APIs in TensorFlow 2"
-      path: https://blog.tensorflow.org/2018/12/standardizing-on-keras-guidance.html
-      buttons:
-      - label: "Read on the TensorFlow blog"
-        path: https://blog.tensorflow.org/2018/12/standardizing-on-keras-guidance.html
+    - heading: "Basic Computer Vision with ML"
+      path: "https://www.youtube.com/watch?v=bemDFpNooA8&list=PLQY2H8rRoyvwWuPiWnuTDBHe7I0fMSsfO"
+      youtube_id: "bemDFpNooA8?rel=0&show_info=0"
 
   - classname: devsite-landing-row-100
     items:
@@ -295,3 +273,13 @@ landing_page:
           icon_name: chevron_right
           foreground: theme
           background: grey
+
+  - classname: devsite-landing-row-100
+    items:
+    - description: >
+        
TensorFlow updates

+        Subscribe to the
+        TensorFlow blog,
+        YouTube channel,
+        and Twitter
+        for the latest updates.

From ebbc70617cb94fed3b1a9df13afe2cf42d65f50d Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 27 Jul 2022 10:37:36 -0700
Subject: [PATCH 240/872] Use tf>2.9 instead of nightly.

---
 site/en/guide/function.ipynb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index 896b7aae763..b8c9827a170 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -91,7 +91,8 @@
       },
       "outputs": [],
       "source": [
-        "!pip install -q tf_nightly\n",
+        "# Update tensorflow (this notebook requires tf>=2.9)\n",
+        "!pip install -q tensorflow>=2.9.0\n",
         "import tensorflow as tf"
       ]
     },

From 9e75aa508e267ea5f7d23158d2b5055c08ab44dc Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Wed, 27 Jul 2022 10:39:24 -0700
Subject: [PATCH 241/872] Add missing -U

---
 site/en/guide/function.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index b8c9827a170..2ef5363571e 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -92,7 +92,7 @@
       "outputs": [],
       "source": [
         "# Update tensorflow (this notebook requires tf>=2.9)\n",
-        "!pip install -q tensorflow>=2.9.0\n",
+        "!pip install -q -U tensorflow>=2.9.0\n",
         "import tensorflow as tf"
       ]
     },

From 1af7aa87ff6584de300ca9a69e86e4d01cf368fb Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Wed, 27 Jul 2022 11:39:03 -0700
Subject: [PATCH 242/872] Minor linting of tf.function guide to pass lint test

---
 site/en/guide/function.ipynb | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index 2ef5363571e..48235eb9568 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -566,10 +566,10 @@
         "get_mixed_flavor(Apple(), Mango()) # Traces a new concrete function\n",
         "get_mixed_flavor(Apple(), Mango()) # Traces a new concrete function again\n",
         "\n",
-        "# However, we, as the designers of the `Fruit` class, know that each subclass\n",
-        "# has a fixed flavor and we can reuse an existing traced concrete function if\n",
-        "# it was the same subclass. Avoiding such unnecessary tracing of concrete\n",
-        "# functions can have significant performance benefits.\n",
+        "# However, each subclass of the `Fruit` class has a fixed flavor, and you\n",
+        "# can reuse an existing traced concrete function if it was the same\n",
+        "# subclass. Avoiding such unnecessary tracing of concrete functions\n",
+        "# can have significant performance benefits.\n",
         "\n",
         "class FruitTraceType(tf.types.experimental.TraceType):\n",
         "  def __init__(self, fruit_type):\n",
@@ -599,7 +599,7 @@
         "class MangoWithTraceType(FruitWithTraceType):\n",
         "  flavor = tf.constant([3, 4])\n",
         "\n",
-        "# Now if we try calling it again:\n",
+        "# Now if you try calling it again:\n",
         "get_mixed_flavor(AppleWithTraceType(), MangoWithTraceType()) # Traces a new concrete function\n",
         "get_mixed_flavor(AppleWithTraceType(), MangoWithTraceType()) # Re-uses the traced concrete function"
       ]

From 35e3834fbbe3e55ec276b48aadc95508bc110fcf Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Wed, 27 Jul 2022 11:42:39 -0700
Subject: [PATCH 243/872] Update TF version comment in tf.function guide

---
 site/en/guide/function.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/guide/function.ipynb b/site/en/guide/function.ipynb
index 48235eb9568..280cec1fce5 100644
--- a/site/en/guide/function.ipynb
+++ b/site/en/guide/function.ipynb
@@ -91,7 +91,7 @@
       },
       "outputs": [],
       "source": [
-        "# Update tensorflow (this notebook requires tf>=2.9)\n",
+        "# Update TensorFlow, as this notebook requires version 2.9 or later\n",
         "!pip install -q -U tensorflow>=2.9.0\n",
         "import tensorflow as tf"
       ]

From 9ada61795328f9a295adfb8d72fa564bf9250414 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Thu, 28 Jul 2022 12:02:41 -0700
Subject: [PATCH 244/872] edited TF basics overview guide to maintain Core-API
 focus

PiperOrigin-RevId: 463892873
---
 site/en/guide/basics.ipynb | 176 ++++++++++++++++++++++++++++---------
 1 file changed, 136 insertions(+), 40 deletions(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index 0fe53c9920a..df8124cb343 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -647,7 +647,7 @@
         "id": "De5LldboSWcW"
       },
       "source": [
-        "Create a model:"
+        "Create a quadratic model with randomly initialized weights and a bias:"
       ]
     },
     {
@@ -658,21 +658,29 @@
       },
       "outputs": [],
       "source": [
-        "class Model(tf.keras.Model):\n",
-        "  def __init__(self, units):\n",
-        "    super().__init__()\n",
-        "    self.dense1 = tf.keras.layers.Dense(units=units,\n",
-        "                                        activation=tf.nn.relu,\n",
-        "                                        kernel_initializer=tf.random.normal,\n",
-        "                                        bias_initializer=tf.random.normal)\n",
-        "    self.dense2 = tf.keras.layers.Dense(1)\n",
+        "class Model(tf.Module):\n",
         "\n",
-        "  def call(self, x, training=True):\n",
-        "    # For Keras layers/models, implement `call` instead of `__call__`.\n",
-        "    x = x[:, tf.newaxis]\n",
-        "    x = self.dense1(x)\n",
-        "    x = self.dense2(x)\n",
-        "    return tf.squeeze(x, axis=1)"
+        "  def __init__(self):\n",
+        "    # Randomly generate weight and bias terms\n",
+        "    self.rand_W_q, self.rand_W_l, self.rand_b = tf.random.uniform(shape=[3], minval=0., maxval=5.)\n",
+        "    # Initialize model parameters\n",
+        "    self.W_q = tf.Variable(self.rand_W_q)\n",
+        "    self.W_l = tf.Variable(self.rand_W_l)\n",
+        "    self.b = tf.Variable(self.rand_b)\n",
+        "  \n",
+        "  @tf.function\n",
+        "  def __call__(self, x):\n",
+        "    # Quadratic Model : quadratic_weight * x^2 + linear_weight * x + bias\n",
+        "    return self.W_q * (x**2) + self.W_l * x + self.b"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "36o7VjaesScg"
+      },
+      "source": [
+        "First, observe your model's performance before training:"
       ]
     },
     {
@@ -683,7 +691,7 @@
       },
       "outputs": [],
       "source": [
-        "model = Model(64)"
+        "quad_model = Model()"
       ]
     },
     {
@@ -694,11 +702,48 @@
       },
       "outputs": [],
       "source": [
-        "plt.plot(x.numpy(), y.numpy(), '.', label='data')\n",
-        "plt.plot(x, f(x),  label='Ground truth')\n",
-        "plt.plot(x, model(x), label='Untrained predictions')\n",
-        "plt.title('Before training')\n",
-        "plt.legend();"
+        "def plot_preds(x, y, f, model, title):\n",
+        "  plt.plot(x, y, '.', label='data')\n",
+        "  plt.plot(x, f(x), label='Ground truth')\n",
+        "  plt.plot(x, model(x), label='Untrained predictions')\n",
+        "  plt.title(title)\n",
+        "  plt.legend()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Y0JtXQat-nlk"
+      },
+      "outputs": [],
+      "source": [
+        "plot_preds(x, y, f, quad_model, 'Before training')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hLzwD0-ascGf"
+      },
+      "source": [
+        "Now, define a loss for your model:\n",
+        "\n",
+        "Given that this model is intended to predict continuous values, the mean squared error (MSE) is a good choice for the loss function. The MSE is defined as the mean of the squared differences between the predicted values and the ground truth. \n",
+        "\n",
+        "$MSE = \\frac{1}{n}\\sum_{i=1}^{n}({y_{pred\\_i}}-y_i)^2$"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "eCtJ1uuCseZd"
+      },
+      "outputs": [],
+      "source": [
+        "def MSE(y_pred, y):\n",
+        "  return tf.reduce_mean(tf.square(y_pred - y))"
       ]
     },
     {
@@ -707,7 +752,7 @@
         "id": "ZebWva4vTBlC"
       },
       "source": [
-        "Write a basic training loop:"
+        "Write a basic training loop for the model. The loop will make use of the MSE loss function and its gradients with respect to the input in order to iteratively update the model's parameters."
       ]
     },
     {
@@ -718,20 +763,39 @@
       },
       "outputs": [],
       "source": [
-        "variables = model.variables\n",
-        "\n",
-        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)\n",
+        "# Set training parameters\n",
+        "steps = 1000\n",
+        "learning_rate = 0.01\n",
+        "losses = []\n",
         "\n",
-        "for step in range(1000):\n",
+        "# Format training loop\n",
+        "for step in range(steps):\n",
         "  with tf.GradientTape() as tape:\n",
-        "    prediction = model(x)\n",
-        "    error = (y-prediction)**2\n",
-        "    mean_error = tf.reduce_mean(error)\n",
-        "  gradient = tape.gradient(mean_error, variables)\n",
-        "  optimizer.apply_gradients(zip(gradient, variables))\n",
-        "\n",
+        "    loss = MSE(quad_model(x),y)\n",
+        "  losses.append(loss)\n",
         "  if step % 100 == 0:\n",
-        "    print(f'Mean squared error: {mean_error.numpy():0.3f}')"
+        "    print(f'Mean squared error for step {step}: {loss.numpy():0.3f}')\n",
+        "  # Update parameters with respect to the gradient calculations\n",
+        "  dy_dW_q, dy_dW_l, dy_db = tape.gradient(loss, [quad_model.W_q, quad_model.W_l, quad_model.b])\n",
+        "  quad_model.W_q.assign_sub(learning_rate * dy_dW_q)\n",
+        "  quad_model.W_l.assign_sub(learning_rate * dy_dW_l)\n",
+        "  quad_model.b.assign_sub(learning_rate * dy_db)\n",
+        "\n",
+        "# Plot model results\n",
+        "print(\"\\n\")\n",
+        "plt.plot(range(steps), losses)\n",
+        "plt.xlabel(\"Step\")\n",
+        "plt.ylabel(\"Mean Squared Error (MSE)\")\n",
+        "plt.title('MSE Loss vs Training Iterations');"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dW5B2TTRsvxE"
+      },
+      "source": [
+        "Now, observe your model's performance after training:"
       ]
     },
     {
@@ -742,11 +806,7 @@
       },
       "outputs": [],
       "source": [
-        "plt.plot(x.numpy(),y.numpy(), '.', label=\"data\")\n",
-        "plt.plot(x, f(x),  label='Ground truth')\n",
-        "plt.plot(x, model(x), label='Trained predictions')\n",
-        "plt.title('After training')\n",
-        "plt.legend();"
+        "plot_preds(x, y, f, quad_model, 'After training')"
       ]
     },
     {
@@ -758,6 +818,15 @@
         "That's working, but remember that implementations of common training utilities are available in the `tf.keras` module. So, consider using those before writing your own. To start with, the `Model.compile` and `Model.fit` methods implement a training loop for you:"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cjx23MiztFmT"
+      },
+      "source": [
+        "Begin by creating a Sequential Model in Keras using `tf.keras.Sequential`."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -766,7 +835,13 @@
       },
       "outputs": [],
       "source": [
-        "new_model = Model(64)"
+        "new_model = tf.keras.Sequential([\n",
+        "    tf.keras.layers.Dense(units=64,\n",
+        "                          input_shape=[1],\n",
+        "                          activation=tf.nn.relu,\n",
+        "                          kernel_initializer=tf.random.normal,\n",
+        "                          bias_initializer=tf.random.normal),\n",
+        "    tf.keras.layers.Dense(1)])"
       ]
     },
     {
@@ -786,7 +861,16 @@
         "                        batch_size=32,\n",
         "                        verbose=0)\n",
         "\n",
-        "model.save('./my_model')"
+        "new_model.save('./my_new_model')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "u3q5d1SzvzTq"
+      },
+      "source": [
+        "Observe your Keras model's performance after training:"
       ]
     },
     {
@@ -804,6 +888,17 @@
         "plt.title('Keras training progress');"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bB44a9YsvnfK"
+      },
+      "outputs": [],
+      "source": [
+        "plot_preds(x, y, f, new_model, 'After training: Keras')"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -818,6 +913,7 @@
     "colab": {
       "collapsed_sections": [],
       "name": "basics.ipynb",
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 15f80ebb94d8b3233f346d1fcd5784d4b664362d Mon Sep 17 00:00:00 2001
From: synandi <98147397+synandi@users.noreply.github.com>
Date: Fri, 29 Jul 2022 12:14:27 +0530
Subject: [PATCH 245/872] Fixed broken link

Fixed broken link for tf-idf. Changed "https://en.wikipedia.org/wiki/Tf%E2%80%93idf/" to "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"
---
 site/en/tutorials/structured_data/preprocessing_layers.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/structured_data/preprocessing_layers.ipynb b/site/en/tutorials/structured_data/preprocessing_layers.ipynb
index fd2a31151fe..40f1b73e003 100644
--- a/site/en/tutorials/structured_data/preprocessing_layers.ipynb
+++ b/site/en/tutorials/structured_data/preprocessing_layers.ipynb
@@ -363,7 +363,7 @@
         "In this tutorial, you will use the following four preprocessing layers to demonstrate how to perform preprocessing, structured data encoding, and feature engineering:\n",
         "\n",
         "- `tf.keras.layers.Normalization`: Performs feature-wise normalization of input features.\n",
-        "- `tf.keras.layers.CategoryEncoding`: Turns integer categorical features into one-hot, multi-hot, or tf-idf\n",
+        "- `tf.keras.layers.CategoryEncoding`: Turns integer categorical features into one-hot, multi-hot, or tf-idf\n",
         "dense representations.\n",
         "- `tf.keras.layers.StringLookup`: Turns string categorical values into integer indices.\n",
         "- `tf.keras.layers.IntegerLookup`: Turns integer categorical values into integer indices.\n",

From 9bfffb91247233025892f2d293aa4d206c0ccad9 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Fri, 29 Jul 2022 09:40:09 -0700
Subject: [PATCH 246/872] Mention the "convert_to_tensor" policy in the intro
 docs.

Also explain add_n and reduce_sum(..., axis=0).
Deprecate `accumulate_n`.

PiperOrigin-RevId: 464097005
---
 site/en/guide/basics.ipynb | 31 +++++++++++++++
 site/en/guide/tensor.ipynb | 78 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 107 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index df8124cb343..a3e360777b4 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -197,6 +197,37 @@
         "tf.reduce_sum(x)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TNHnIjOVLJfA"
+      },
+      "source": [
+        "Note: Typically, anywhere a TensorFlow function expects a `Tensor` as input, the function will also accept anything that can be converted to a `Tensor` using `tf.convert_to_tensor`. See below for an example."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "i_XKgjDsL4GE"
+      },
+      "outputs": [],
+      "source": [
+        "tf.convert_to_tensor([1,2,3])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "wTBt-JUqLJDJ"
+      },
+      "outputs": [],
+      "source": [
+        "tf.reduce_sum([1,2,3])"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
diff --git a/site/en/guide/tensor.ipynb b/site/en/guide/tensor.ipynb
index 5ac82475a9d..453dde97f7e 100644
--- a/site/en/guide/tensor.ipynb
+++ b/site/en/guide/tensor.ipynb
@@ -95,7 +95,7 @@
       "source": [
         "## Basics\n",
         "\n",
-        "Let's create some basic tensors."
+        "First, create some basic tensors."
       ]
     },
     {
@@ -352,7 +352,7 @@
         "id": "S3_vIAl2JPVc"
       },
       "source": [
-        "Tensors are used in all kinds of operations (ops)."
+        "Tensors are used in all kinds of operations (or \"Ops\")."
       ]
     },
     {
@@ -373,6 +373,48 @@
         "print(tf.nn.softmax(c))"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0MNM-q7-MZLz"
+      },
+      "source": [
+        "Note: Typically, anywhere a TensorFlow function expects a `Tensor` as input, the function will also accept anything that can be converted to a `Tensor` using `tf.convert_to_tensor`. See below for an example."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_wch0N8xNEt-"
+      },
+      "outputs": [],
+      "source": [
+        "tf.convert_to_tensor([1,2,3])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ngqIeWYeNJVI"
+      },
+      "outputs": [],
+      "source": [
+        "tf.reduce_max([1,2,3])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ThVMxqbVNOq3"
+      },
+      "outputs": [],
+      "source": [
+        "tf.reduce_max(np.array([1,2,3]))"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -461,6 +503,37 @@
         "print(\"Total number of elements (3*2*4*5): \", tf.size(rank_4_tensor).numpy())"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2ZGZp_JOOPOv"
+      },
+      "source": [
+        "But note that the `Tensor.ndim` and `Tensor.shape` attributes don't return `Tensor` objects. If you need a `Tensor` use the `tf.rank` or `tf.shape` function. This difference is subtle, but it can be important when building graphs (later)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Ptq0-y6APCpD"
+      },
+      "outputs": [],
+      "source": [
+        "tf.rank(rank_4_tensor)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "HslrDOEBPICN"
+      },
+      "outputs": [],
+      "source": [
+        "tf.shape(rank_4_tensor)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1491,6 +1564,7 @@
         "Tce3stUlHN0L"
       ],
       "name": "tensor.ipynb",
+      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From be563d905794ccc2e436b24ea519bc9e4d99b8ab Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Fri, 29 Jul 2022 15:41:16 -0700
Subject: [PATCH 247/872] edited quadratic model in TF basics overview guide

PiperOrigin-RevId: 464172900
---
 site/en/guide/basics.ipynb | 77 ++++++++++++++++++++++----------------
 1 file changed, 45 insertions(+), 32 deletions(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index a3e360777b4..7e8a556e0a7 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -693,16 +693,17 @@
         "\n",
         "  def __init__(self):\n",
         "    # Randomly generate weight and bias terms\n",
-        "    self.rand_W_q, self.rand_W_l, self.rand_b = tf.random.uniform(shape=[3], minval=0., maxval=5.)\n",
+        "    rand_init = tf.random.uniform(shape=[3], minval=0., maxval=5., seed=22)\n",
         "    # Initialize model parameters\n",
-        "    self.W_q = tf.Variable(self.rand_W_q)\n",
-        "    self.W_l = tf.Variable(self.rand_W_l)\n",
-        "    self.b = tf.Variable(self.rand_b)\n",
+        "    self.w_q = tf.Variable(rand_init[0])\n",
+        "    self.w_l = tf.Variable(rand_init[1])\n",
+        "    self.b = tf.Variable(rand_init[2])\n",
+        "    self.vars = [self.w_q, self.w_l, self.b]\n",
         "  \n",
         "  @tf.function\n",
         "  def __call__(self, x):\n",
         "    # Quadratic Model : quadratic_weight * x^2 + linear_weight * x + bias\n",
-        "    return self.W_q * (x**2) + self.W_l * x + self.b"
+        "    return self.w_q * (x**2) + self.w_l * x + self.b"
       ]
     },
     {
@@ -734,9 +735,10 @@
       "outputs": [],
       "source": [
         "def plot_preds(x, y, f, model, title):\n",
-        "  plt.plot(x, y, '.', label='data')\n",
+        "  plt.figure()\n",
+        "  plt.plot(x, y, '.', label='Data')\n",
         "  plt.plot(x, f(x), label='Ground truth')\n",
-        "  plt.plot(x, model(x), label='Untrained predictions')\n",
+        "  plt.plot(x, model(x), label='Predictions')\n",
         "  plt.title(title)\n",
         "  plt.legend()"
       ]
@@ -773,17 +775,30 @@
       },
       "outputs": [],
       "source": [
-        "def MSE(y_pred, y):\n",
+        "def mse_loss(y_pred, y):\n",
         "  return tf.reduce_mean(tf.square(y_pred - y))"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "ZebWva4vTBlC"
+        "id": "7EWyDu3zot2w"
       },
       "source": [
-        "Write a basic training loop for the model. The loop will make use of the MSE loss function and its gradients with respect to the input in order to iteratively update the model's parameters."
+        "Write a basic training loop for the model. The loop will make use of the MSE loss function and its gradients with respect to the input in order to iteratively update the model's parameters. Using mini-batches for training provides both memory efficienciy and faster convergence. The `tf.data.Dataset` API has useful functions for batching and shuffling."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8kX_-zily2Ia"
+      },
+      "outputs": [],
+      "source": [
+        "batch_size = 32\n",
+        "dataset = tf.data.Dataset.from_tensor_slices((x, y))\n",
+        "dataset = dataset.shuffle(buffer_size=x.shape[0]).batch(batch_size)"
       ]
     },
     {
@@ -795,29 +810,31 @@
       "outputs": [],
       "source": [
         "# Set training parameters\n",
-        "steps = 1000\n",
+        "epochs = 100\n",
         "learning_rate = 0.01\n",
         "losses = []\n",
         "\n",
         "# Format training loop\n",
-        "for step in range(steps):\n",
-        "  with tf.GradientTape() as tape:\n",
-        "    loss = MSE(quad_model(x),y)\n",
+        "for epoch in range(epochs):\n",
+        "  for x_batch, y_batch in dataset:\n",
+        "    with tf.GradientTape() as tape:\n",
+        "      batch_loss = mse_loss(quad_model(x_batch), y_batch)\n",
+        "    # Update parameters with respect to the gradient calculations\n",
+        "    grads = tape.gradient(batch_loss, quad_model.vars)\n",
+        "    for g,v in zip(grads, quad_model.vars):\n",
+        "        v.assign_sub(learning_rate*g)\n",
+        "  # Keep track of model loss per epoch\n",
+        "  loss = mse_loss(quad_model(x), y)\n",
         "  losses.append(loss)\n",
-        "  if step % 100 == 0:\n",
-        "    print(f'Mean squared error for step {step}: {loss.numpy():0.3f}')\n",
-        "  # Update parameters with respect to the gradient calculations\n",
-        "  dy_dW_q, dy_dW_l, dy_db = tape.gradient(loss, [quad_model.W_q, quad_model.W_l, quad_model.b])\n",
-        "  quad_model.W_q.assign_sub(learning_rate * dy_dW_q)\n",
-        "  quad_model.W_l.assign_sub(learning_rate * dy_dW_l)\n",
-        "  quad_model.b.assign_sub(learning_rate * dy_db)\n",
+        "  if epoch % 10 == 0:\n",
+        "    print(f'Mean squared error for step {epoch}: {loss.numpy():0.3f}')\n",
         "\n",
         "# Plot model results\n",
         "print(\"\\n\")\n",
-        "plt.plot(range(steps), losses)\n",
-        "plt.xlabel(\"Step\")\n",
+        "plt.plot(range(epochs), losses)\n",
+        "plt.xlabel(\"Epoch\")\n",
         "plt.ylabel(\"Mean Squared Error (MSE)\")\n",
-        "plt.title('MSE Loss vs Training Iterations');"
+        "plt.title('MSE loss vs training iterations');"
       ]
     },
     {
@@ -855,7 +872,7 @@
         "id": "cjx23MiztFmT"
       },
       "source": [
-        "Begin by creating a Sequential Model in Keras using `tf.keras.Sequential`."
+        "Begin by creating a Sequential Model in Keras using `tf.keras.Sequential`. One of the simplest Keras layers is the dense layer, which can be instantiated with `tf.keras.layers.Dense`. The dense layer is able to learn multidimensional linear relationships of the form $\\mathrm{Y} = \\mathrm{W}\\mathrm{X} +  \\vec{b}$. In order to learn a nonlinear equation of the form, $w_1x^2 + w_2x + b$, the dense layer's input should be a data matrix with $x^2$ and $x$ as features. The lambda layer, `tf.keras.layers.Lambda`, can be used to perform this stacking transformation."
       ]
     },
     {
@@ -867,12 +884,8 @@
       "outputs": [],
       "source": [
         "new_model = tf.keras.Sequential([\n",
-        "    tf.keras.layers.Dense(units=64,\n",
-        "                          input_shape=[1],\n",
-        "                          activation=tf.nn.relu,\n",
-        "                          kernel_initializer=tf.random.normal,\n",
-        "                          bias_initializer=tf.random.normal),\n",
-        "    tf.keras.layers.Dense(1)])"
+        "    tf.keras.layers.Lambda(lambda x: tf.stack([x, x**2], axis=1)),\n",
+        "    tf.keras.layers.Dense(units=1, kernel_initializer=tf.random.normal)])"
       ]
     },
     {
@@ -927,7 +940,7 @@
       },
       "outputs": [],
       "source": [
-        "plot_preds(x, y, f, new_model, 'After training: Keras')"
+        "plot_preds(x, y, f, new_model, 'After Training: Keras')"
       ]
     },
     {

From eaef2e3cfe61d3940846d00ba022b44db6c96996 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 1 Aug 2022 09:25:54 -0700
Subject: [PATCH 248/872] Make py_module_names a dict to capture the
 `{short_name: long_name}` mappings.

PiperOrigin-RevId: 464552675
---
 .../api_generator/generate_lib.py               |  2 +-
 .../api_generator/reference_resolver.py         | 17 +++++++++++------
 .../api_generator/reference_resolver_test.py    |  6 +++---
 3 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/tools/tensorflow_docs/api_generator/generate_lib.py b/tools/tensorflow_docs/api_generator/generate_lib.py
index 21bf617ceff..7eb441a0d13 100644
--- a/tools/tensorflow_docs/api_generator/generate_lib.py
+++ b/tools/tensorflow_docs/api_generator/generate_lib.py
@@ -384,7 +384,7 @@ def __init__(
 
   def make_reference_resolver(self, visitor):
     return reference_resolver_lib.ReferenceResolver.from_visitor(
-        visitor, py_module_names=[self._short_name])
+        visitor, py_module_names={self._short_name: self._py_module.__name__})
 
   def make_parser_config(self,
                          visitor: doc_generator_visitor.DocGeneratorVisitor):
diff --git a/tools/tensorflow_docs/api_generator/reference_resolver.py b/tools/tensorflow_docs/api_generator/reference_resolver.py
index 268d23c97f7..af5d2a03d76 100644
--- a/tools/tensorflow_docs/api_generator/reference_resolver.py
+++ b/tools/tensorflow_docs/api_generator/reference_resolver.py
@@ -22,7 +22,7 @@
 import posixpath
 import re
 
-from typing import Dict, List, Optional
+from typing import Optional, Union
 
 from tensorflow_docs.api_generator import parser
 
@@ -81,11 +81,11 @@ class ReferenceResolver:
   def __init__(
       self,
       *,
-      duplicate_of: Dict[str, str],
-      is_fragment: Dict[str, bool],
-      py_module_names: List[str],
+      duplicate_of: dict[str, str],
+      is_fragment: dict[str, bool],
+      py_module_names: Union[list[str], dict[str, str]],
       link_prefix: Optional[str] = None,
-      physical_path: Optional[Dict[str, str]] = None,
+      physical_path: Optional[dict[str, str]] = None,
   ):
     """Initializes a Reference Resolver.
 
@@ -95,7 +95,9 @@ def __init__(
       is_fragment: A map from full names to bool for each symbol. If True the
         object lives at a page fragment `tf.a.b.c` --> `tf/a/b#c`. If False
         object has a page to itself: `tf.a.b.c` --> `tf/a/b/c`.
-      py_module_names: A list of string names of Python modules.
+      py_module_names: A dict from short name to module name Like
+        `{'tf': 'tensorflow'}`. Or [deprecated] a list of short-names like
+        `['tf']`.
       link_prefix: The website to which these symbols should link to. A prefix
         is added before the links to enable cross-site linking if `link_prefix`
         is not None.
@@ -105,7 +107,10 @@ def __init__(
     self._duplicate_of = duplicate_of
     self._is_fragment = is_fragment
     self._physical_path = physical_path
+    if isinstance(py_module_names, list):
+      py_module_names = {short: short for short in py_module_names}
     self._py_module_names = py_module_names
+
     self._link_prefix = link_prefix
 
     self._all_names = set(is_fragment.keys())
diff --git a/tools/tensorflow_docs/api_generator/reference_resolver_test.py b/tools/tensorflow_docs/api_generator/reference_resolver_test.py
index fc31602eae0..b8184e88983 100644
--- a/tools/tensorflow_docs/api_generator/reference_resolver_test.py
+++ b/tools/tensorflow_docs/api_generator/reference_resolver_test.py
@@ -47,7 +47,7 @@ def testSaveReferenceResolver(self):
         'tf.AClass2': False,
         'tf.function': False
     }
-    py_module_names = ['tf', 'tfdbg']
+    py_module_names = {'tf': 'tensorflow'}
 
     resolver = reference_resolver_lib.ReferenceResolver(
         duplicate_of=duplicate_of,
@@ -79,7 +79,7 @@ def test_duplicate_fragment(self):
         'tf.Class2': False,
         'tf.sub.Class2': False
     }
-    py_module_names = ['tf']
+    py_module_names = {'tf': 'tensorflow'}
 
     reference_resolver = reference_resolver_lib.ReferenceResolver(
         duplicate_of=duplicate_of,
@@ -133,7 +133,7 @@ def test_partial_symbol_references(self, string, link):
         'tf.contrib.y.z': False,
     }
 
-    py_module_names = ['tf']
+    py_module_names = {'tf': 'tensorflow'}
 
     resolver = reference_resolver_lib.ReferenceResolver(
         duplicate_of=duplicate_of,

From 75971c26d34b6a820efa3d62e522e13aff373747 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Mon, 1 Aug 2022 13:52:56 -0700
Subject: [PATCH 249/872] updated TF basics overview guide to use tf.module
 functionality

PiperOrigin-RevId: 464615889
---
 site/en/guide/basics.ipynb | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index 7e8a556e0a7..8ee26c5703a 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -698,7 +698,6 @@
         "    self.w_q = tf.Variable(rand_init[0])\n",
         "    self.w_l = tf.Variable(rand_init[1])\n",
         "    self.b = tf.Variable(rand_init[2])\n",
-        "    self.vars = [self.w_q, self.w_l, self.b]\n",
         "  \n",
         "  @tf.function\n",
         "  def __call__(self, x):\n",
@@ -820,8 +819,8 @@
         "    with tf.GradientTape() as tape:\n",
         "      batch_loss = mse_loss(quad_model(x_batch), y_batch)\n",
         "    # Update parameters with respect to the gradient calculations\n",
-        "    grads = tape.gradient(batch_loss, quad_model.vars)\n",
-        "    for g,v in zip(grads, quad_model.vars):\n",
+        "    grads = tape.gradient(batch_loss, quad_model.variables)\n",
+        "    for g,v in zip(grads, quad_model.variables):\n",
         "        v.assign_sub(learning_rate*g)\n",
         "  # Keep track of model loss per epoch\n",
         "  loss = mse_loss(quad_model(x), y)\n",

From c7e6c58022db8248b0bf29771091dc83ba6a7225 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Tue, 2 Aug 2022 13:40:47 -0700
Subject: [PATCH 250/872] Use fewer epochs for CycleGan.

PiperOrigin-RevId: 464875766
---
 site/en/tutorials/generative/cyclegan.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/generative/cyclegan.ipynb b/site/en/tutorials/generative/cyclegan.ipynb
index 4dae5b77791..56ee1ae382c 100644
--- a/site/en/tutorials/generative/cyclegan.ipynb
+++ b/site/en/tutorials/generative/cyclegan.ipynb
@@ -634,7 +634,7 @@
       "source": [
         "## Training\n",
         "\n",
-        "Note: This example model is trained for fewer epochs (40) than the paper (200) to keep training time reasonable for this tutorial. Predictions may be less accurate. "
+        "Note: This example model is trained for fewer epochs (10) than the paper (200) to keep training time reasonable for this tutorial. The generated images will have much lower quality."
       ]
     },
     {
@@ -645,7 +645,7 @@
       },
       "outputs": [],
       "source": [
-        "EPOCHS = 40"
+        "EPOCHS = 10"
       ]
     },
     {

From 336b3db9731ecefa99c02b478fbe2639364d93ce Mon Sep 17 00:00:00 2001
From: Olzhas Akpambetov 
Date: Wed, 3 Aug 2022 13:41:18 -0700
Subject: [PATCH 251/872] Fix external link formatting, lint NMT with attention
 tutorial

PiperOrigin-RevId: 465137234
---
 site/en/install/gpu_plugins.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/install/gpu_plugins.md b/site/en/install/gpu_plugins.md
index 290cd48cf76..1083daa949a 100644
--- a/site/en/install/gpu_plugins.md
+++ b/site/en/install/gpu_plugins.md
@@ -4,7 +4,7 @@ Note: This page is for non-NVIDIA® GPU devices. For NVIDIA® GPU support, go to
 the [Install TensorFlow with pip](./pip.md) guide.
 
 TensorFlow's
-[pluggable device](https://github.com/tensorflow/community/blob/master/rfcs/20200624-pluggable-device-for-tensorflow.md){.external}
+[pluggable device](https://github.com/tensorflow/community/blob/master/rfcs/20200624-pluggable-device-for-tensorflow.md){:.external}
 architecture adds new device support as separate plug-in packages
 that are installed alongside the official TensorFlow package.
 
@@ -64,5 +64,5 @@ Metal `PluggableDevice` for macOS GPUs:
 DirectML `PluggableDevice` for Windows and WSL (preview):
 
 *   [PyPI wheel](https://pypi.org/project/tensorflow-directml-plugin/){:.external}.
-*   [GitHub repo](https://github.com/microsoft/tensorflow-directml-plugin){.external}.
+*   [GitHub repo](https://github.com/microsoft/tensorflow-directml-plugin){:.external}.
 *   For questions, feedback or to raise issues, please visit the [Issues page of `tensorflow-directml-plugin` on GitHub](https://github.com/microsoft/tensorflow-directml-plugin/issues){:.external}.

From 1a1909b87979d77988e5ec98edb55b1a2a19f768 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Thu, 4 Aug 2022 08:58:57 -0700
Subject: [PATCH 252/872] Update pip.md

---
 site/en/install/pip.md | 223 ++++++++++++++++++++++++-----------------
 1 file changed, 133 insertions(+), 90 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 6bc4a61e5c8..0e69a84c09f 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -3,10 +3,10 @@
 This guide is for the latest stable version of TensorFlow. For the
 preview build *(nightly)*, please use the pip package named
 `tf-nightly`. Refer to [these tables](./source#tested_build_configurations) for
-older TensorFlow version requirements. For TensorFlow 1.x users, please refer to
-the [migration guide](../guide/migrate) to upgrade to TensorFlow 2.
+older TensorFlow version requirements. For the CPU-only build use the pip
+package named `tensorflow-cpu`
 
-Here is a lookup table for the install commands. Scroll down for the
+Here are the quick versions of the install commands. Scroll down for the
 step-by-step instructions.
 
 * {Linux}
@@ -28,10 +28,37 @@ step-by-step instructions.
    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
    ```
 
-* {Windows}
+* {Windows Native}
 
+   Caution: The current TensorFlow version, `2.10`, is the **last** TensorFlow
+   release that will support GPU on native-Windows.
+   Starting with TensorFlow `2.11`, you will need to install
+   [TensorFlow in WSL2](https://tensorflow.org/install/pip#windows-wsl2),
+   or install `tensorflow_cpu` and, optionally, try the
+   [TensorFlow-DirectML-Plugin](https://github.com/microsoft/tensorflow-directml-plugin#tensorflow-directml-plugin-)
+
+   ```bash
+   conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+   python3 -m pip install tensorflow
+   # Verify install:
+   python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+   ```
+
+* {Windows WSL2}
+
+   Note: TensorFlow is supported on WSL2 on Windows 10 19044 or higher with GPU
+   access is now available. This corresponds to Windows 10 version
+   21H2, the November 2021 update. You can get the latest update from here:
+   [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
+   For instructions, please see 
+   [Install WSL2](https://docs.microsoft.com/en-us/windows/wsl/install){:.external}
+   and
+   [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
+   for CUDA in WSL.
+  
    ```bash
    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
    python3 -m pip install tensorflow
    # Verify install:
    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
@@ -105,120 +132,123 @@ The following NVIDIA® software are only required for GPU support.
 
 ## Step-by-step instructions
 
+{% setvar linux_instructions %}
+We only officially support Ubuntu. However, the following instructions may
+also work for other Linux distros.
 
-*   {Linux}
+We recommend using
+[Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
+create a separate environment to avoid changing any installed software in
+your system. This is also the easiest way to install the required software,
+especially for the GPU setup.
 
-    We only officially support Ubuntu. However, the following instructions may
-    also work for other Linux distros.
+### 1. Install Miniconda
 
-    We recommend using
-    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    create a separate environment to avoid changing any installed software in
-    your system. This is also the easiest way to install the required software,
-    especially for the GPU setup.
+You can use the following command to install Miniconda. During installation,
+you may need to press enter and type "yes".
 
-    ### 1. Install Miniconda
+```bash
+curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh
+```
 
-    You can use the following command to install Miniconda. During installation,
-    you may need to press enter and type "yes".
+You may need to restart your terminal or `source ~/.bashrc` to enable the
+`conda` command. Use `conda -V` to test if it is installed successfully.
 
-    ```bash
-    curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
-    bash Miniconda3-latest-Linux-x86_64.sh
-    ```
+### 2. Create a conda environment
 
-    You may need to restart your terminal or `source ~/.bashrc` to enable the
-    `conda` command. Use `conda -V` to test if it is installed successfully.
+    Create a new conda environment named `tf` with the following command.
 
-    ### 2. Create a conda environment
+```bash
+conda create --name tf python=3.9
+```
 
-    Create a new conda environment named `tf` with the following command.
+You can deactivate and activate it with the following commands.
 
-    ```bash
-    conda create --name tf python=3.9
-    ```
+```bash
+conda deactivate
+conda activate tf
+```
 
-    You can deactivate and activate it with the following commands.
+Please make sure it is activated for the rest of the installation.
 
-    ```bash
-    conda deactivate
-    conda activate tf
-    ```
+### 3. GPU setup
 
-    Please make sure it is activated for the rest of the installation.
+You can skip this section if you only run TensorFlow on CPU.
 
-    ### 3. GPU setup
+First, we need to install
+[NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
+if you have not. You can use the following command to verify it is
+installed.
 
-    You can skip this section if you only run TensorFlow on CPU.
+```bash
+nvidia-smi
+```
 
-    First, we need to install
-    [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
-    if you have not. You can use the following command to verify it is
-    installed.
+Then, we install the CUDA, cuDNN with conda.
 
-    ```bash
-    nvidia-smi
-    ```
+```bash
+conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+```
 
-    Then, we install the CUDA, cuDNN with conda.
+Configure the system paths. You can do it with following command everytime
+your start a new terminal after activating your conda environment.
 
-    ```bash
-    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
-    ```
+```bash
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
+```
 
-    Configure the system paths. You can do it with following command everytime
-    your start a new terminal after activating your conda environment.
+However, for your convenience, we recommend automating it with the following
+commands. The system paths will be automatically configured when you
+activate this conda environment.
 
-    ```bash
-    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
-    ```
+```bash
+mkdir -p $CONDA_PREFIX/etc/conda/activate.d
+echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
+```
 
-    However, for your convenience, we recommend automating it with the following
-    commands. The system paths will be automatically configured when you
-    activate this conda environment.
+### 4. Install TensorFlow
 
-    ```bash
-    mkdir -p $CONDA_PREFIX/etc/conda/activate.d
-    echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
-    ```
+TensorFlow requires a recent version of pip, so upgrade your pip
+installation to be sure you're running the latest version.
 
-    ### 4. Install TensorFlow
+```bash
+pip install --upgrade pip
+```
 
-    TensorFlow requires a recent version of pip, so upgrade your pip
-    installation to be sure you're running the latest version.
+Then, install TensorFlow with pip.
 
-    ```bash
-    pip install --upgrade pip
-    ```
+Note: Do not install with conda. It may not have the latest stable
+version. We recommend using pip since TensorFlow is only
+officially released to PyPI.
 
-    Then, install TensorFlow with pip.
+```bash
+pip install tensorflow
+```
 
-    Note: Do not install with conda. It may not have the latest stable
-    version. We recommend using pip since TensorFlow is only
-    officially released to PyPI.
+### 5. Verify install
 
-    ```bash
-    pip install tensorflow
-    ```
+Verify the CPU setup:
 
-    ### 5. Verify install
+```bash
+python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+```
 
-    Verify the CPU setup:
+If a tensor is returned, you've installed TensorFlow successfully.
 
-    ```bash
-    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
-    ```
+Verify the GPU setup:
 
-    If a tensor is returned, you've installed TensorFlow successfully.
+```bash
+python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+```
 
-    Verify the GPU setup:
+If a list of GPU devices is returned, you've installed TensorFlow
+successfully.
+{% endsetvar %}
 
-    ```bash
-    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
-    ```
+*   {Linux}
 
-    If a list of GPU devices is returned, you've installed TensorFlow
-    successfully.
+   {{ linux_instructions }}
 
 *   {MacOS}
 
@@ -309,15 +339,14 @@ The following NVIDIA® software are only required for GPU support.
 
     If a tensor is returned, you've installed TensorFlow successfully.
 
-*   {Windows}
+*   {Windows Native}
 
-    Note: Experimental support for WSL2 on Windows 10 19044 or higher with GPU
-    access is now available. This corresponds to Windows 10 version
-    21H2, the November 2021 update. You can get the latest update from here:
-    [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
-    For instructions, please see
-    [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
-    for CUDA in WSL.
+   Caution: The current TensorFlow version, `2.10`, is the **last** TensorFlow
+   release that will support GPU on native-Windows.
+   Starting with TensorFlow `2.11`, you will need to install
+   [TensorFlow in WSL2](https://tensorflow.org/install/pip#windows-[wsl2]),
+   or install `tensorflow_cpu` and, optionally, try the
+   [TensorFlow-DirectML-Plugin](https://github.com/microsoft/tensorflow-directml-plugin#tensorflow-directml-plugin-)
 
     ### 1. Install Microsoft Visual C++ Redistributable
 
@@ -418,6 +447,20 @@ The following NVIDIA® software are only required for GPU support.
     If a list of GPU devices is returned, you've installed TensorFlow
     successfully.
 
+*   {Windows WSL2}
+
+   Note: TensorFlow is supported for WSL2 on Windows 10 19044 or higher with GPU
+   access is now available. This corresponds to Windows 10 version
+   21H2, the November 2021 update. You can get the latest update from here:
+   [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
+   For instructions, please see 
+   [Install WSL2](https://docs.microsoft.com/en-us/windows/wsl/install){:.external}
+   and
+   [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
+   for CUDA in WSL.
+
+   {{ linux_instructions }}
+
 ## Package location
 
 A few installation mechanisms require the URL of the TensorFlow Python package.

From a30144caa1df397fe676ac871b3c168f3aa31269 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Thu, 4 Aug 2022 09:02:00 -0700
Subject: [PATCH 253/872] formatting

---
 site/en/install/pip.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 0e69a84c09f..1499a1b11e3 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -133,6 +133,7 @@ The following NVIDIA® software are only required for GPU support.
 ## Step-by-step instructions
 
 {% setvar linux_instructions %}
+
 We only officially support Ubuntu. However, the following instructions may
 also work for other Linux distros.
 
@@ -244,6 +245,7 @@ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'
 
 If a list of GPU devices is returned, you've installed TensorFlow
 successfully.
+
 {% endsetvar %}
 
 *   {Linux}

From 6ca36ba030f34532ffba61419b49a682db82acf6 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 4 Aug 2022 15:07:16 -0700
Subject: [PATCH 254/872] Lint Install TensorFlow with pip guide

---
 site/en/install/pip.md | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 1499a1b11e3..7a7dc847629 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -134,10 +134,10 @@ The following NVIDIA® software are only required for GPU support.
 
 {% setvar linux_instructions %}
 
-We only officially support Ubuntu. However, the following instructions may
-also work for other Linux distros.
+TensorFlow only officially supports Ubuntu. However, the following instructions may
+also work for other Linux distributions.
 
-We recommend using
+It's recommended to use
 [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
 create a separate environment to avoid changing any installed software in
 your system. This is also the easiest way to install the required software,
@@ -175,9 +175,9 @@ Please make sure it is activated for the rest of the installation.
 
 ### 3. GPU setup
 
-You can skip this section if you only run TensorFlow on CPU.
+You can skip this section if you only run TensorFlow on the CPU.
 
-First, we need to install
+First, you need to install the
 [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
 if you have not. You can use the following command to verify it is
 installed.
@@ -192,14 +192,14 @@ Then, we install the CUDA, cuDNN with conda.
 conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
 ```
 
-Configure the system paths. You can do it with following command everytime
+Configure the system paths. You can do it using the following command every time
 your start a new terminal after activating your conda environment.
 
 ```bash
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
 ```
 
-However, for your convenience, we recommend automating it with the following
+However, for your convenience, it's recommended to automate it with the following
 commands. The system paths will be automatically configured when you
 activate this conda environment.
 
@@ -219,8 +219,8 @@ pip install --upgrade pip
 
 Then, install TensorFlow with pip.
 
-Note: Do not install with conda. It may not have the latest stable
-version. We recommend using pip since TensorFlow is only
+Note: Do not install with conda because it may not have the latest stable
+version. It's recommended to use pip because TensorFlow is only
 officially released to PyPI.
 
 ```bash

From 5a55ff570908dc6b08ead380b4db99ce692d3d57 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 4 Aug 2022 15:10:12 -0700
Subject: [PATCH 255/872] Lint Install TensorFlow with pip guide

---
 site/en/install/pip.md | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 7a7dc847629..5ba07f321aa 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -22,7 +22,7 @@ step-by-step instructions.
 * {MacOS}
 
    ```bash
-   # Currently, we do not have official GPU support for MacOS.
+   # Currently, there is no official GPU support for MacOS.
    python3 -m pip install tensorflow
    # Verify install:
    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
@@ -186,7 +186,7 @@ installed.
 nvidia-smi
 ```
 
-Then, we install the CUDA, cuDNN with conda.
+Then, install the CUDA, cuDNN with conda.
 
 ```bash
 conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
@@ -263,8 +263,8 @@ successfully.
     you need those libraries, you will have to use TensorFlow with x86 emulation
     and Rosetta.
 
-    Currently, we do not have official GPU support for running TensorFlow on
-    MacOS. The following is instructions are for running on CPU.
+    Currently, there is no official GPU support for running TensorFlow on
+    MacOS. The following is instructions are for running on the CPU.
 
     ### 1. Check Python version
 
@@ -278,7 +278,7 @@ successfully.
     ```
 
     If you have the correct version of Python and pip, you may skip the next two
-    steps and go to "4. Install TensorFlow". However, we still recommend not
+    steps and go to "4. Install TensorFlow". However, it's still recommended not
     skipping the steps. Use
     [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
     install Python and pip. It create a separate environment to avoid
@@ -326,7 +326,7 @@ successfully.
     Then, install TensorFlow with pip.
 
     Note: Do not install with conda. It may not have the latest stable
-    version. We recommend using pip since TensorFlow is only
+    version. It's recommended to use pip since TensorFlow is only
     officially released to PyPI.
 
     ```bash
@@ -370,7 +370,7 @@ successfully.
 
     ### 2. Install Miniconda
 
-    We recommend using
+    It's recommended to use
     [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
     create a separate environment to avoid changing any installed software in
     your system. This is also the easiest way to install the required software,
@@ -401,11 +401,11 @@ successfully.
 
     You can skip this section if you only run TensorFlow on CPU.
 
-    First, we need to install
+    First, you need to install
     [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
     if you have not.
 
-    Then, we install the CUDA, cuDNN with conda.
+    Then, install the CUDA, cuDNN with conda.
 
     ```bash
     conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
@@ -423,7 +423,7 @@ successfully.
     Then, install TensorFlow with pip.
 
     Note: Do not install with conda. It may not have the latest stable
-    version. We recommend using pip since TensorFlow is only
+    version. It's recommended to use pip since TensorFlow is only
     officially released to PyPI.
 
     ```bash

From 7e3850c919523bd9080eab45d27d93795129f953 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Thu, 4 Aug 2022 15:15:03 -0700
Subject: [PATCH 256/872] Fix minor typos/grammar in Install TensorFlow with
 pip guide

---
 site/en/install/pip.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 5ba07f321aa..8b2ccc152fd 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -193,7 +193,7 @@ conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
 ```
 
 Configure the system paths. You can do it using the following command every time
-your start a new terminal after activating your conda environment.
+you start a new terminal after activating your conda environment.
 
 ```bash
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
@@ -264,7 +264,7 @@ successfully.
     and Rosetta.
 
     Currently, there is no official GPU support for running TensorFlow on
-    MacOS. The following is instructions are for running on the CPU.
+    MacOS. The following instructions are for running on the CPU.
 
     ### 1. Check Python version
 
@@ -281,7 +281,7 @@ successfully.
     steps and go to "4. Install TensorFlow". However, it's still recommended not
     skipping the steps. Use
     [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    install Python and pip. It create a separate environment to avoid
+    install Python and pip. It creates a separate environment to avoid
     changing any installed software in your system.
 
     ### 2. Install Miniconda
@@ -399,7 +399,7 @@ successfully.
 
     ### 4. GPU setup
 
-    You can skip this section if you only run TensorFlow on CPU.
+    You can skip this section if you only run TensorFlow on the CPU.
 
     First, you need to install
     [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}

From df2261fb31b07753e63d70a1c27c92464c7c787b Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Fri, 5 Aug 2022 09:25:46 -0700
Subject: [PATCH 257/872] updated TF basics overview guide to use cleaner math
 notation

PiperOrigin-RevId: 465580297
---
 site/en/guide/basics.ipynb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index 8ee26c5703a..dce5bdfa440 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -761,9 +761,9 @@
       "source": [
         "Now, define a loss for your model:\n",
         "\n",
-        "Given that this model is intended to predict continuous values, the mean squared error (MSE) is a good choice for the loss function. The MSE is defined as the mean of the squared differences between the predicted values and the ground truth. \n",
+        "Given that this model is intended to predict continuous values, the mean squared error (MSE) is a good choice for the loss function. Given a vector of predictions, $\\hat{y}$, and a vector of true targets, $y$, the MSE is defined as the mean of the squared differences between the predicted values and the ground truth.\n",
         "\n",
-        "$MSE = \\frac{1}{n}\\sum_{i=1}^{n}({y_{pred\\_i}}-y_i)^2$"
+        "$MSE = \\frac{1}{m}\\sum_{i=1}^{m}(\\hat{y}_i -y_i)^2$"
       ]
     },
     {

From ccbc0e50126ad431028b064d52c306014aa1c251 Mon Sep 17 00:00:00 2001
From: William Muir 
Date: Sat, 6 Aug 2022 06:09:18 -0500
Subject: [PATCH 258/872] Bump libtensorflow to v2.9.1

---
 site/en/install/lang_c.ipynb | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/site/en/install/lang_c.ipynb b/site/en/install/lang_c.ipynb
index da4bbf03fc6..b5d308551f4 100644
--- a/site/en/install/lang_c.ipynb
+++ b/site/en/install/lang_c.ipynb
@@ -130,25 +130,25 @@
         "  Linux\n",
         "  \n",
         "    Linux CPU only\n",
-        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-2.8.0.tar.gz\n",
+        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-2.9.1.tar.gz\n",
         "  \n",
         "  \n",
         "    Linux GPU support\n",
-        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-2.8.0.tar.gz\n",
+        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-2.9.1.tar.gz\n",
         "  \n",
         "  macOS\n",
         "  \n",
         "    macOS CPU only\n",
-        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-2.8.0.tar.gz\n",
+        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-2.9.1.tar.gz\n",
         "  \n",
         "  Windows\n",
         "  \n",
         "    Windows CPU only\n",
-        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-2.8.0.zip\n",
+        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-2.9.1.zip\n",
         "  \n",
         "  \n",
         "    Windows GPU only\n",
-        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-windows-x86_64-2.8.0.zip\n",
+        "    https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-windows-x86_64-2.9.1.zip\n",
         "  \n",
         ""
       ]
@@ -174,7 +174,7 @@
       "outputs": [],
       "source": [
         "%%bash\n",
-        "FILENAME=libtensorflow-cpu-linux-x86_64-2.8.0.tar.gz\n",
+        "FILENAME=libtensorflow-cpu-linux-x86_64-2.9.1.tar.gz\n",
         "wget -q --no-check-certificate https://storage.googleapis.com/tensorflow/libtensorflow/${FILENAME}\n",
         "sudo tar -C /usr/local -xzf ${FILENAME}"
       ]

From 84adf2c04d03b7a2ff5a94608a19120761be4f6f Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 8 Aug 2022 17:11:17 -0700
Subject: [PATCH 259/872] Resolve all comments.

---
 site/en/install/pip.md | 174 +++++++++++++++++++++++------------------
 1 file changed, 97 insertions(+), 77 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 8b2ccc152fd..aee909a2d4e 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -22,7 +22,7 @@ step-by-step instructions.
 * {MacOS}
 
    ```bash
-   # Currently, there is no official GPU support for MacOS.
+   # There is currently no official GPU support for MacOS.
    python3 -m pip install tensorflow
    # Verify install:
    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
@@ -46,12 +46,12 @@ step-by-step instructions.
 
 * {Windows WSL2}
 
-   Note: TensorFlow is supported on WSL2 on Windows 10 19044 or higher with GPU
-   access is now available. This corresponds to Windows 10 version
-   21H2, the November 2021 update. You can get the latest update from here:
-   [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
+   Note: TensorFlow with GPU access is supported for WSL2 on Windows 10 19044 or
+   higher. This corresponds to Windows 10 version 21H2, the November 2021
+   update. You can get the latest update from here:
+   [Download Windows 10](https://www.microsoft.com/software-download/windows10){:.external}.
    For instructions, please see 
-   [Install WSL2](https://docs.microsoft.com/en-us/windows/wsl/install){:.external}
+   [Install WSL2](https://docs.microsoft.com/windows/wsl/install){:.external}
    and
    [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
    for CUDA in WSL.
@@ -108,7 +108,8 @@ enable compute capabilities by [building TensorFlow from source](./source.md).
 
 *   Ubuntu 16.04 or higher (64-bit)
 *   macOS 10.12.6 (Sierra) or higher (64-bit) *(no GPU support)*
-*   Windows 7 or higher (64-bit)
+*   Windows Native - Windows 7 or higher (64-bit)
+*   Windows WSL2 - Windows 10 19044 or higher (64-bit)
 
 Note: GPU support is available for Ubuntu and Windows with CUDA®-enabled cards.
 
@@ -116,10 +117,11 @@ Note: GPU support is available for Ubuntu and Windows with CUDA®-enabled cards.
 
 *   Python 3.7–3.10
 *   pip version 19.0 or higher for Linux (requires `manylinux2010` support) and
-    Windows, version 20.3 or higher for macOS
-*   Windows Requires
+    Windows. pip version 20.3 or higher for macOS.
+*   Windows Native Requires
     [Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads){:.external}
 
+
 The following NVIDIA® software are only required for GPU support.
 
 *   [NVIDIA® GPU drivers](https://www.nvidia.com/drivers){:.external}
@@ -134,16 +136,13 @@ The following NVIDIA® software are only required for GPU support.
 
 {% setvar linux_instructions %}
 
-TensorFlow only officially supports Ubuntu. However, the following instructions may
-also work for other Linux distributions.
-
-It's recommended to use
-[Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-create a separate environment to avoid changing any installed software in
-your system. This is also the easiest way to install the required software,
-especially for the GPU setup.
+### 2. Install Miniconda
 
-### 1. Install Miniconda
+[Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} is the
+recommended approach for installing TensorFlow with GPU support.
+It creates a separate environment to avoid changing any installed
+software in your system. This is also the easiest way to install the required
+software especially for the GPU setup.
 
 You can use the following command to install Miniconda. During installation,
 you may need to press enter and type "yes".
@@ -156,7 +155,7 @@ bash Miniconda3-latest-Linux-x86_64.sh
 You may need to restart your terminal or `source ~/.bashrc` to enable the
 `conda` command. Use `conda -V` to test if it is installed successfully.
 
-### 2. Create a conda environment
+### 3. Create a conda environment
 
     Create a new conda environment named `tf` with the following command.
 
@@ -173,11 +172,11 @@ conda activate tf
 
 Please make sure it is activated for the rest of the installation.
 
-### 3. GPU setup
+### 4. GPU setup
 
-You can skip this section if you only run TensorFlow on the CPU.
+You can skip this section if you only run TensorFlow on CPU.
 
-First, you need to install the
+First install the
 [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
 if you have not. You can use the following command to verify it is
 installed.
@@ -186,20 +185,20 @@ installed.
 nvidia-smi
 ```
 
-Then, install the CUDA, cuDNN with conda.
+Then install CUDA and cuDNN with conda.
 
 ```bash
 conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
 ```
 
-Configure the system paths. You can do it using the following command every time
-you start a new terminal after activating your conda environment.
+Configure the system paths. You can do it with following command everytime
+your start a new terminal after activating your conda environment.
 
 ```bash
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
 ```
 
-However, for your convenience, it's recommended to automate it with the following
+For your convenience it is recommended that you automate it with the following
 commands. The system paths will be automatically configured when you
 activate this conda environment.
 
@@ -208,7 +207,7 @@ mkdir -p $CONDA_PREFIX/etc/conda/activate.d
 echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
 ```
 
-### 4. Install TensorFlow
+### 5. Install TensorFlow
 
 TensorFlow requires a recent version of pip, so upgrade your pip
 installation to be sure you're running the latest version.
@@ -219,15 +218,15 @@ pip install --upgrade pip
 
 Then, install TensorFlow with pip.
 
-Note: Do not install with conda because it may not have the latest stable
-version. It's recommended to use pip because TensorFlow is only
-officially released to PyPI.
+Note: Do not install TensorFlow with conda. It may not have the latest stable
+version. pip is recommended since TensorFlow is only officially released to
+PyPI.
 
 ```bash
 pip install tensorflow
 ```
 
-### 5. Verify install
+### 6. Verify install
 
 Verify the CPU setup:
 
@@ -245,15 +244,31 @@ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'
 
 If a list of GPU devices is returned, you've installed TensorFlow
 successfully.
-
 {% endsetvar %}
 
 *   {Linux}
 
+   ### 1. System requirements
+
+   *  Ubuntu 16.04 or higher (64-bit)
+
+   TensorFlow only officially support Ubuntu. However, the following
+   instructions may also work for other Linux distros.
+
+   Note: Linux Aarch64 TensorFlow builds are build and tested by a third party.
+   TensorFlow will endeavor to maintain availability and integrity of this
+   binary on a best-effort basis.
+
+
+
    {{ linux_instructions }}
 
 *   {MacOS}
 
+   ### 1. System requirements
+
+    *   macOS 10.12.6 (Sierra) or higher (64-bit)
+
     Note: For users of Apple M1 computers, to get native performance, you'll
     want to follow the instructions found
     [here](https://developer.apple.com/metal/tensorflow-plugin/){:.external}.
@@ -263,10 +278,10 @@ successfully.
     you need those libraries, you will have to use TensorFlow with x86 emulation
     and Rosetta.
 
-    Currently, there is no official GPU support for running TensorFlow on
-    MacOS. The following instructions are for running on the CPU.
+    Currently there is no official GPU support for running TensorFlow on
+    MacOS. The following is instructions are for running on CPU.
 
-    ### 1. Check Python version
+    ### 2. Check Python version
 
     Check if your Python environment is already configured:
 
@@ -277,17 +292,13 @@ successfully.
     python3 -m pip --version
     ```
 
-    If you have the correct version of Python and pip, you may skip the next two
-    steps and go to "4. Install TensorFlow". However, it's still recommended not
-    skipping the steps. Use
-    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    install Python and pip. It creates a separate environment to avoid
-    changing any installed software in your system.
-
-    ### 2. Install Miniconda
+   ### 2. Install Miniconda
 
-    You can use the following command to install Miniconda. During installation,
-    you may need to press enter and type "yes".
+   [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external}
+   is the recommended approach for installing TensorFlow with GPU support.
+   It creates a separate environment to avoid changing any installed
+   software in your system. This is also the easiest way to install the required
+   software especially for the GPU setup.
 
     ```bash
     curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o Miniconda3-latest-MacOSX-x86_64.sh
@@ -297,7 +308,7 @@ successfully.
     You may need to restart your terminal or `source ~/.bashrc` to enable the
     `conda` command. Use `conda -V` to test if it is installed successfully.
 
-    ### 3. Create a conda environment
+    ### 4. Create a conda environment
 
     Create a new conda environment named `tf` with the following command.
 
@@ -314,7 +325,7 @@ successfully.
 
     Please make sure it is activated for the rest of the installation.
 
-    ### 4. Install TensorFlow
+    ### 5. Install TensorFlow
 
     TensorFlow requires a recent version of pip, so upgrade your pip
     installation to be sure you're running the latest version.
@@ -325,15 +336,15 @@ successfully.
 
     Then, install TensorFlow with pip.
 
-    Note: Do not install with conda. It may not have the latest stable
-    version. It's recommended to use pip since TensorFlow is only
-    officially released to PyPI.
+    Note: Do not install TensorFlow with conda. It may not have the latest stable
+    version. pip is recommended since TensorFlow is only officially released to
+    PyPI.
 
     ```bash
     pip install tensorflow
     ```
 
-    ### 5. Verify install
+    ### 6. Verify install
 
     ```bash
     python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
@@ -350,7 +361,15 @@ successfully.
    or install `tensorflow_cpu` and, optionally, try the
    [TensorFlow-DirectML-Plugin](https://github.com/microsoft/tensorflow-directml-plugin#tensorflow-directml-plugin-)
 
-    ### 1. Install Microsoft Visual C++ Redistributable
+   ## 1. System requirements
+
+   *   Windows 7 or higher (64-bit)
+   
+   Note: Windows CPU TensorFlow builds are built and tested by a third party.
+   TensorFlow will endeavor to maintain availability and integrity of this
+   binary on a best-effort basis.
+
+    ### 2. Install Microsoft Visual C++ Redistributable
 
     Install the *Microsoft Visual C++ Redistributable for Visual Studio 2015,
     2017, and 2019*. Starting with the TensorFlow 2.1.0 version, the
@@ -368,19 +387,19 @@ successfully.
     [long paths are enabled](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing){:.external}
     on Windows.
 
-    ### 2. Install Miniconda
+    ### 3. Install Miniconda
 
-    It's recommended to use
-    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    create a separate environment to avoid changing any installed software in
-    your system. This is also the easiest way to install the required software,
-    especially for the GPU setup.
+    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external}
+    is the recommended approach for installing TensorFlow with GPU support.
+    It creates a separate environment to avoid changing any installed
+    software in your system. This is also the easiest way to install the
+    required software especially for the GPU setup.
 
     Download the
     [Miniconda Windows Installer](https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe){:.external}.
     Double-click the downloaded file and follow the instructions on the screen.
 
-    ### 3. Create a conda environment
+    ### 4. Create a conda environment
 
     Create a new conda environment named `tf` with the following command.
 
@@ -397,21 +416,21 @@ successfully.
 
     Please make sure it is activated for the rest of the installation.
 
-    ### 4. GPU setup
+    ### 5. GPU setup
 
-    You can skip this section if you only run TensorFlow on the CPU.
+    You can skip this section if you only run TensorFlow on CPU.
 
-    First, you need to install
+    First install
     [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
     if you have not.
 
-    Then, install the CUDA, cuDNN with conda.
+    Then install the CUDA, cuDNN with conda.
 
     ```bash
     conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
     ```
 
-    ### 5. Install TensorFlow
+    ### 6. Install TensorFlow
 
     TensorFlow requires a recent version of pip, so upgrade your pip
     installation to be sure you're running the latest version.
@@ -422,15 +441,15 @@ successfully.
 
     Then, install TensorFlow with pip.
 
-    Note: Do not install with conda. It may not have the latest stable
-    version. It's recommended to use pip since TensorFlow is only
-    officially released to PyPI.
+    Note: Do not install TensorFlow with conda. It may not have the latest stable
+    version. pip is recommended since TensorFlow is only officially released to
+    PyPI.
 
     ```bash
     pip install tensorflow
     ```
 
-    ### 6. Verify install
+    ### 7. Verify install
 
     Verify the CPU setup:
 
@@ -451,15 +470,16 @@ successfully.
 
 *   {Windows WSL2}
 
-   Note: TensorFlow is supported for WSL2 on Windows 10 19044 or higher with GPU
-   access is now available. This corresponds to Windows 10 version
-   21H2, the November 2021 update. You can get the latest update from here:
-   [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
-   For instructions, please see 
-   [Install WSL2](https://docs.microsoft.com/en-us/windows/wsl/install){:.external}
-   and
-   [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
-   for CUDA in WSL.
+   ### 1. System requirements
+
+   *   Windows 10 19044 or higher (64-bit). This corresponds to Windows 10
+       version 21H2, the November 2021 update.
+   
+   See the following documents to:
+   
+   * [Download the latest Windows 10 update](https://www.microsoft.com/software-download/windows10){:.external}.
+   * [Install WSL2](https://docs.microsoft.com/windows/wsl/install){:.external}
+   * [Setup NVIDIA® GPU support in WSL2](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
 
    {{ linux_instructions }}
 

From a08cac3899626fe742fd42c7cffd1cf05af7d31e Mon Sep 17 00:00:00 2001
From: tfdocsbot 
Date: Tue, 9 Aug 2022 00:12:41 +0000
Subject: [PATCH 260/872] nbfmt

---
 site/en/guide/basics.ipynb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/site/en/guide/basics.ipynb b/site/en/guide/basics.ipynb
index 8ee26c5703a..e63a1398af3 100644
--- a/site/en/guide/basics.ipynb
+++ b/site/en/guide/basics.ipynb
@@ -956,7 +956,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "basics.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {

From 4525d20756d1dfcc228e88d5ea35e8a44c6e85c3 Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Mon, 8 Aug 2022 17:16:54 -0700
Subject: [PATCH 261/872] Clear trailing white space.

---
 site/en/install/pip.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index aee909a2d4e..b4826465243 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -50,12 +50,12 @@ step-by-step instructions.
    higher. This corresponds to Windows 10 version 21H2, the November 2021
    update. You can get the latest update from here:
    [Download Windows 10](https://www.microsoft.com/software-download/windows10){:.external}.
-   For instructions, please see 
+   For instructions, please see
    [Install WSL2](https://docs.microsoft.com/windows/wsl/install){:.external}
    and
    [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
    for CUDA in WSL.
-  
+
    ```bash
    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
@@ -364,7 +364,7 @@ successfully.
    ## 1. System requirements
 
    *   Windows 7 or higher (64-bit)
-   
+
    Note: Windows CPU TensorFlow builds are built and tested by a third party.
    TensorFlow will endeavor to maintain availability and integrity of this
    binary on a best-effort basis.

From 1529486a44d867cc742d79a904aa994f695db17a Mon Sep 17 00:00:00 2001
From: Mark Daoust 
Date: Thu, 11 Aug 2022 10:58:59 -0700
Subject: [PATCH 262/872] Update install for changes to windows builds.

Split "Windows Native" and "Windows WSL2" into two separate tabs.
Add a caution that "TensorFlow version, 2.10, is the last TensorFlow release that will support GPU on native-Windows". Copy the linux instructions into the Windows WSL2 tab.

PiperOrigin-RevId: 466998549
---
 site/en/install/pip.md | 364 ++++++++++++++++++++++-------------------
 1 file changed, 198 insertions(+), 166 deletions(-)

diff --git a/site/en/install/pip.md b/site/en/install/pip.md
index 6bc4a61e5c8..f11c94b4175 100644
--- a/site/en/install/pip.md
+++ b/site/en/install/pip.md
@@ -1,57 +1,66 @@
 # Install TensorFlow with pip
 
-This guide is for the latest stable version of TensorFlow. For the
-preview build *(nightly)*, please use the pip package named
-`tf-nightly`. Refer to [these tables](./source#tested_build_configurations) for
-older TensorFlow version requirements. For TensorFlow 1.x users, please refer to
-the [migration guide](../guide/migrate) to upgrade to TensorFlow 2.
+This guide is for the latest stable version of TensorFlow. For the preview build
+*(nightly)*, please use the pip package named `tf-nightly`. Refer to
+[these tables](./source#tested_build_configurations) for older TensorFlow
+version requirements. For the CPU-only build use the pip package named
+`tensorflow-cpu`
 
-Here is a lookup table for the install commands. Scroll down for the
+Here are the quick versions of the install commands. Scroll down for the
 step-by-step instructions.
 
-* {Linux}
+*   ~~~bash conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0 export
+    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/ python3 -m pip install
+    tensorflow # Verify install: python3 -c "import tensorflow as tf;
+    print(tf.config.list_physical_devices('GPU'))" ``` {Linux} ~~~
 
-   ```bash
-   conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
-   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
-   python3 -m pip install tensorflow
-   # Verify install:
-   python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
-   ```
+*   ~~~bash # There is currently no official GPU support for MacOS. python3 -m
+    pip install tensorflow # Verify install: python3 -c "import tensorflow as
+    tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))" ``` {MacOS} ~~~
 
-* {MacOS}
+*   Caution: The current TensorFlow version, `2.10`, is the **last** TensorFlow
+    release that will support GPU on native-Windows. Starting with TensorFlow
+    `2.11`, you will need to install
+    [TensorFlow in WSL2](https://tensorflow.org/install/pip#windows-wsl2), or
+    install `tensorflow_cpu` and, optionally, try the
+    [TensorFlow-DirectML-Plugin](https://github.com/microsoft/tensorflow-directml-plugin#tensorflow-directml-plugin-)
+    {Windows Native}
 
-   ```bash
-   # Currently, we do not have official GPU support for MacOS.
-   python3 -m pip install tensorflow
-   # Verify install:
-   python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
-   ```
-
-* {Windows}
-
-   ```bash
-   conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
-   python3 -m pip install tensorflow
-   # Verify install:
-   python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
-   ```
+    ```bash
+    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+    python3 -m pip install tensorflow
+    # Verify install:
+    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+    ```
 
-* {CPU}
+*   Note: TensorFlow with GPU access is supported for WSL2 on Windows 10 19044
+    or higher. This corresponds to Windows 10 version 21H2, the November 2021
+    update. You can get the latest update from here:
+    [Download Windows 10](https://www.microsoft.com/software-download/windows10){:.external}.
+    For instructions, please see
+    [Install WSL2](https://docs.microsoft.com/windows/wsl/install){:.external}
+    and
+    [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
+    for CUDA in WSL. {Windows WSL2}
 
-   ```bash
-   python3 -m pip install tensorflow
-   # Verify install:
-   python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
-   ```
+    ```bash
+    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
+    python3 -m pip install tensorflow
+    # Verify install:
+    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+    ```
 
-* {Nightly}
+*   ~~~bash
+    python3 -m pip install tensorflow
+    # Verify install:
+    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+    ``` {CPU}
+    ~~~
 
-   ```bash
-   python3 -m pip install tf-nightly
-   # Verify install:
-   python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
-   ```
+*   ~~~bash python3 -m pip install tf-nightly # Verify install: python3 -c
+    "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000,
+    1000])))" ``` {Nightly} ~~~
 
 ## Hardware requirements
 
@@ -81,7 +90,8 @@ enable compute capabilities by [building TensorFlow from source](./source.md).
 
 *   Ubuntu 16.04 or higher (64-bit)
 *   macOS 10.12.6 (Sierra) or higher (64-bit) *(no GPU support)*
-*   Windows 7 or higher (64-bit)
+*   Windows Native - Windows 7 or higher (64-bit)
+*   Windows WSL2 - Windows 10 19044 or higher (64-bit)
 
 Note: GPU support is available for Ubuntu and Windows with CUDA®-enabled cards.
 
@@ -89,9 +99,10 @@ Note: GPU support is available for Ubuntu and Windows with CUDA®-enabled cards.
 
 *   Python 3.7–3.10
 *   pip version 19.0 or higher for Linux (requires `manylinux2010` support) and
-    Windows, version 20.3 or higher for macOS
-*   Windows Requires
-    [Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads){:.external}
+    Windows. pip version 20.3 or higher for macOS.
+*   Windows Native Requires
+    [Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads)
+    {:.external}
 
 The following NVIDIA® software are only required for GPU support.
 
@@ -105,122 +116,130 @@ The following NVIDIA® software are only required for GPU support.
 
 ## Step-by-step instructions
 
+{% setvar linux_instructions %}
 
-*   {Linux}
+### 2. Install Miniconda
 
-    We only officially support Ubuntu. However, the following instructions may
-    also work for other Linux distros.
+[Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} is the
+recommended approach for installing TensorFlow with GPU support. It creates a
+separate environment to avoid changing any installed software in your system.
+This is also the easiest way to install the required software especially for the
+GPU setup.
 
-    We recommend using
-    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    create a separate environment to avoid changing any installed software in
-    your system. This is also the easiest way to install the required software,
-    especially for the GPU setup.
+You can use the following command to install Miniconda. During installation, you
+may need to press enter and type "yes".
 
-    ### 1. Install Miniconda
+```bash
+curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh
+```
 
-    You can use the following command to install Miniconda. During installation,
-    you may need to press enter and type "yes".
+You may need to restart your terminal or `source ~/.bashrc` to enable the
+`conda` command. Use `conda -V` to test if it is installed successfully.
 
-    ```bash
-    curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
-    bash Miniconda3-latest-Linux-x86_64.sh
-    ```
+### 3. Create a conda environment
 
-    You may need to restart your terminal or `source ~/.bashrc` to enable the
-    `conda` command. Use `conda -V` to test if it is installed successfully.
+```
+Create a new conda environment named `tf` with the following command.
+```
 
-    ### 2. Create a conda environment
+```bash
+conda create --name tf python=3.9
+```
 
-    Create a new conda environment named `tf` with the following command.
+You can deactivate and activate it with the following commands.
 
-    ```bash
-    conda create --name tf python=3.9
-    ```
+```bash
+conda deactivate
+conda activate tf
+```
 
-    You can deactivate and activate it with the following commands.
+Please make sure it is activated for the rest of the installation.
 
-    ```bash
-    conda deactivate
-    conda activate tf
-    ```
+### 4. GPU setup
 
-    Please make sure it is activated for the rest of the installation.
+You can skip this section if you only run TensorFlow on CPU.
 
-    ### 3. GPU setup
+First install the
+[NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external} if
+you have not. You can use the following command to verify it is installed.
 
-    You can skip this section if you only run TensorFlow on CPU.
+```bash
+nvidia-smi
+```
 
-    First, we need to install
-    [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
-    if you have not. You can use the following command to verify it is
-    installed.
+Then install CUDA and cuDNN with conda.
 
-    ```bash
-    nvidia-smi
-    ```
+```bash
+conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
+```
 
-    Then, we install the CUDA, cuDNN with conda.
+Configure the system paths. You can do it with following command everytime your
+start a new terminal after activating your conda environment.
 
-    ```bash
-    conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
-    ```
+```bash
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
+```
 
-    Configure the system paths. You can do it with following command everytime
-    your start a new terminal after activating your conda environment.
+For your convenience it is recommended that you automate it with the following
+commands. The system paths will be automatically configured when you activate
+this conda environment.
 
-    ```bash
-    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
-    ```
+```bash
+mkdir -p $CONDA_PREFIX/etc/conda/activate.d
+echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
+```
 
-    However, for your convenience, we recommend automating it with the following
-    commands. The system paths will be automatically configured when you
-    activate this conda environment.
+### 5. Install TensorFlow
 
-    ```bash
-    mkdir -p $CONDA_PREFIX/etc/conda/activate.d
-    echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
-    ```
+TensorFlow requires a recent version of pip, so upgrade your pip installation to
+be sure you're running the latest version.
 
-    ### 4. Install TensorFlow
+```bash
+pip install --upgrade pip
+```
 
-    TensorFlow requires a recent version of pip, so upgrade your pip
-    installation to be sure you're running the latest version.
+Then, install TensorFlow with pip.
 
-    ```bash
-    pip install --upgrade pip
-    ```
+Note: Do not install TensorFlow with conda. It may not have the latest stable
+version. pip is recommended since TensorFlow is only officially released to
+PyPI.
 
-    Then, install TensorFlow with pip.
+```bash
+pip install tensorflow
+```
 
-    Note: Do not install with conda. It may not have the latest stable
-    version. We recommend using pip since TensorFlow is only
-    officially released to PyPI.
+### 6. Verify install
 
-    ```bash
-    pip install tensorflow
-    ```
+Verify the CPU setup:
 
-    ### 5. Verify install
+```bash
+python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
+```
 
-    Verify the CPU setup:
+If a tensor is returned, you've installed TensorFlow successfully.
 
-    ```bash
-    python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
-    ```
+Verify the GPU setup:
 
-    If a tensor is returned, you've installed TensorFlow successfully.
+```bash
+python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+```
 
-    Verify the GPU setup:
+If a list of GPU devices is returned, you've installed TensorFlow successfully.
+{% endsetvar %}
 
-    ```bash
-    python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
-    ```
+*   ### 1. System requirements {Linux}
 
-    If a list of GPU devices is returned, you've installed TensorFlow
-    successfully.
+    *   Ubuntu 16.04 or higher (64-bit)
+
+    TensorFlow only officially support Ubuntu. However, the following
+    instructions may also work for other Linux distros.
+
+    {{ linux_instructions }}
 
-*   {MacOS}
+*   ### 1. System requirements {MacOS}
+
+    *   macOS 10.12.6 (Sierra) or higher (64-bit)
 
     Note: For users of Apple M1 computers, to get native performance, you'll
     want to follow the instructions found
@@ -231,10 +250,10 @@ The following NVIDIA® software are only required for GPU support.
     you need those libraries, you will have to use TensorFlow with x86 emulation
     and Rosetta.
 
-    Currently, we do not have official GPU support for running TensorFlow on
-    MacOS. The following is instructions are for running on CPU.
+    Currently there is no official GPU support for running TensorFlow on MacOS.
+    The following is instructions are for running on CPU.
 
-    ### 1. Check Python version
+    ### 2. Check Python version
 
     Check if your Python environment is already configured:
 
@@ -245,17 +264,13 @@ The following NVIDIA® software are only required for GPU support.
     python3 -m pip --version
     ```
 
-    If you have the correct version of Python and pip, you may skip the next two
-    steps and go to "4. Install TensorFlow". However, we still recommend not
-    skipping the steps. Use
-    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    install Python and pip. It create a separate environment to avoid
-    changing any installed software in your system.
-
     ### 2. Install Miniconda
 
-    You can use the following command to install Miniconda. During installation,
-    you may need to press enter and type "yes".
+    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} is
+    the recommended approach for installing TensorFlow with GPU support. It
+    creates a separate environment to avoid changing any installed software in
+    your system. This is also the easiest way to install the required software
+    especially for the GPU setup.
 
     ```bash
     curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o Miniconda3-latest-MacOSX-x86_64.sh
@@ -265,7 +280,7 @@ The following NVIDIA® software are only required for GPU support.
     You may need to restart your terminal or `source ~/.bashrc` to enable the
     `conda` command. Use `conda -V` to test if it is installed successfully.
 
-    ### 3. Create a conda environment
+    ### 4. Create a conda environment
 
     Create a new conda environment named `tf` with the following command.
 
@@ -282,7 +297,7 @@ The following NVIDIA® software are only required for GPU support.
 
     Please make sure it is activated for the rest of the installation.
 
-    ### 4. Install TensorFlow
+    ### 5. Install TensorFlow
 
     TensorFlow requires a recent version of pip, so upgrade your pip
     installation to be sure you're running the latest version.
@@ -293,15 +308,15 @@ The following NVIDIA® software are only required for GPU support.
 
     Then, install TensorFlow with pip.
 
-    Note: Do not install with conda. It may not have the latest stable
-    version. We recommend using pip since TensorFlow is only
-    officially released to PyPI.
+    Note: Do not install TensorFlow with conda. It may not have the latest
+    stable version. pip is recommended since TensorFlow is only officially
+    released to PyPI.
 
     ```bash
     pip install tensorflow
     ```
 
-    ### 5. Verify install
+    ### 6. Verify install
 
     ```bash
     python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
@@ -309,17 +324,19 @@ The following NVIDIA® software are only required for GPU support.
 
     If a tensor is returned, you've installed TensorFlow successfully.
 
-*   {Windows}
+*   Caution: The current TensorFlow version, `2.10`, is the **last** TensorFlow
+    release that will support GPU on native-Windows. Starting with TensorFlow
+    `2.11`, you will need to install
+    [TensorFlow in WSL2](https://tensorflow.org/install/pip#windows-[wsl2]), or
+    install `tensorflow_cpu` and, optionally, try the
+    [TensorFlow-DirectML-Plugin](https://github.com/microsoft/tensorflow-directml-plugin#tensorflow-directml-plugin-)
+    {Windows Native}
 
-    Note: Experimental support for WSL2 on Windows 10 19044 or higher with GPU
-    access is now available. This corresponds to Windows 10 version
-    21H2, the November 2021 update. You can get the latest update from here:
-    [Download Windows 10](https://www.microsoft.com/en-us/software-download/windows10){:.external}.
-    For instructions, please see
-    [NVIDIA’s setup docs](https://docs.nvidia.com/cuda/wsl-user-guide/index.html){:.external}
-    for CUDA in WSL.
+    ## 1. System requirements
 
-    ### 1. Install Microsoft Visual C++ Redistributable
+    *   Windows 7 or higher (64-bit)
+
+    ### 2. Install Microsoft Visual C++ Redistributable
 
     Install the *Microsoft Visual C++ Redistributable for Visual Studio 2015,
     2017, and 2019*. Starting with the TensorFlow 2.1.0 version, the
@@ -337,19 +354,19 @@ The following NVIDIA® software are only required for GPU support.
     [long paths are enabled](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing){:.external}
     on Windows.
 
-    ### 2. Install Miniconda
+    ### 3. Install Miniconda
 
-    We recommend using
-    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} to
-    create a separate environment to avoid changing any installed software in
-    your system. This is also the easiest way to install the required software,
+    [Miniconda](https://docs.conda.io/en/latest/miniconda.html){:.external} is
+    the recommended approach for installing TensorFlow with GPU support. It
+    creates a separate environment to avoid changing any installed software in
+    your system. This is also the easiest way to install the required software
     especially for the GPU setup.
 
     Download the
     [Miniconda Windows Installer](https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe){:.external}.
     Double-click the downloaded file and follow the instructions on the screen.
 
-    ### 3. Create a conda environment
+    ### 4. Create a conda environment
 
     Create a new conda environment named `tf` with the following command.
 
@@ -366,21 +383,21 @@ The following NVIDIA® software are only required for GPU support.
 
     Please make sure it is activated for the rest of the installation.
 
-    ### 4. GPU setup
+    ### 5. GPU setup
 
     You can skip this section if you only run TensorFlow on CPU.
 
-    First, we need to install
+    First install
     [NVIDIA GPU driver](https://www.nvidia.com/Download/index.aspx){:.external}
     if you have not.
 
-    Then, we install the CUDA, cuDNN with conda.
+    Then install the CUDA, cuDNN with conda.
 
     ```bash
     conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
     ```
 
-    ### 5. Install TensorFlow
+    ### 6. Install TensorFlow
 
     TensorFlow requires a recent version of pip, so upgrade your pip
     installation to be sure you're running the latest version.
@@ -391,15 +408,15 @@ The following NVIDIA® software are only required for GPU support.
 
     Then, install TensorFlow with pip.
 
-    Note: Do not install with conda. It may not have the latest stable
-    version. We recommend using pip since TensorFlow is only
-    officially released to PyPI.
+    Note: Do not install TensorFlow with conda. It may not have the latest
+    stable version. pip is recommended since TensorFlow is only officially
+    released to PyPI.
 
     ```bash
     pip install tensorflow
     ```
 
-    ### 6. Verify install
+    ### 7. Verify install
 
     Verify the CPU setup:
 
@@ -418,6 +435,21 @@ The following NVIDIA® software are only required for GPU support.
     If a list of GPU devices is returned, you've installed TensorFlow
     successfully.
 
+*   ### 1. System requirements {Windows WSL2}
+
+    *   Windows 10 19044 or higher (64-bit). This corresponds to Windows 10
+        version 21H2, the November 2021 update.
+
+    See the following documents to:
+
+    *   [Download the latest Windows 10 update](https://www.microsoft.com/software-download/windows10){:.external}.
+    *   [Install WSL2](https://docs.microsoft.com/windows/wsl/install)
+        {:.external}
+    *   [Setup NVIDIA® GPU support in WSL2](https://docs.nvidia.com/cuda/wsl-user-guide/index.html)
+        {:.external}
+
+    {{ linux_instructions }}
+
 ## Package location
 
 A few installation mechanisms require the URL of the TensorFlow Python package.

From 344da5b2cb9c6a831d7719f78aff91a27e630bf3 Mon Sep 17 00:00:00 2001
From: Olzhas Akpambetov 
Date: Thu, 11 Aug 2022 13:07:01 -0700
Subject: [PATCH 263/872] Add TF Lite step to Image classification

PiperOrigin-RevId: 467029865
---
 site/en/tutorials/images/classification.ipynb | 240 ++++++++++++++++--
 1 file changed, 213 insertions(+), 27 deletions(-)

diff --git a/site/en/tutorials/images/classification.ipynb b/site/en/tutorials/images/classification.ipynb
index 80f950b1c0c..e027826de73 100644
--- a/site/en/tutorials/images/classification.ipynb
+++ b/site/en/tutorials/images/classification.ipynb
@@ -68,7 +68,8 @@
         "id": "gN7G9GFmVrVY"
       },
       "source": [
-        "This tutorial shows how to classify images of flowers. It creates an image classifier using a `tf.keras.Sequential` model, and loads data using `tf.keras.utils.image_dataset_from_directory`. You will gain practical experience with the following concepts:\n",
+        "This tutorial shows how to classify images of flowers using a `tf.keras.Sequential` model and load data using `tf.keras.utils.image_dataset_from_directory`. It demonstrates the following concepts:\n",
+        "\n",
         "\n",
         "* Efficiently loading a dataset off disk.\n",
         "* Identifying overfitting and applying techniques to mitigate it, including data augmentation and dropout.\n",
@@ -80,7 +81,9 @@
         "3. Build the model\n",
         "4. Train the model\n",
         "5. Test the model\n",
-        "6. Improve the model and repeat the process"
+        "6. Improve the model and repeat the process\n",
+        "\n",
+        "In addition, the notebook demonstrates how to convert a [saved model](../../../guide/saved_model.ipynb) to a [TensorFlow Lite](https://www.tensorflow.org/lite/) model for on-device machine learning on mobile, embedded, and IoT devices."
       ]
     },
     {
@@ -89,7 +92,9 @@
         "id": "zF9uvbXNVrVY"
       },
       "source": [
-        "## Import TensorFlow and other libraries"
+        "## Setup\n",
+        "\n",
+        "Import TensorFlow and other necessary libraries:"
       ]
     },
     {
@@ -102,7 +107,6 @@
       "source": [
         "import matplotlib.pyplot as plt\n",
         "import numpy as np\n",
-        "import os\n",
         "import PIL\n",
         "import tensorflow as tf\n",
         "\n",
@@ -243,9 +247,9 @@
         "id": "gIjgz7_JIo_m"
       },
       "source": [
-        "# Load data using a Keras utility\n",
+        "## Load data using a Keras utility\n",
         "\n",
-        "Let's load these images off disk using the helpful `tf.keras.utils.image_dataset_from_directory` utility. This will take you from a directory of images on disk to a `tf.data.Dataset` in just a couple lines of code. If you like, you can also write your own data loading code from scratch by visiting the [Load and preprocess images](../load_data/images.ipynb) tutorial."
+        "Next, load these images off disk using the helpful `tf.keras.utils.image_dataset_from_directory` utility. This will take you from a directory of images on disk to a `tf.data.Dataset` in just a couple lines of code. If you like, you can also write your own data loading code from scratch by visiting the [Load and preprocess images](../load_data/images.ipynb) tutorial."
       ]
     },
     {
@@ -254,7 +258,7 @@
         "id": "xyDNn9MbIzfT"
       },
       "source": [
-        "## Create a dataset"
+        "### Create a dataset"
       ]
     },
     {
@@ -285,7 +289,7 @@
         "id": "pFBhRrrEI49z"
       },
       "source": [
-        "It's good practice to use a validation split when developing your model. Let's use 80% of the images for training, and 20% for validation."
+        "It's good practice to use a validation split when developing your model. Use 80% of the images for training and 20% for validation."
       ]
     },
     {
@@ -379,7 +383,7 @@
         "id": "5M6BXtXFJdW0"
       },
       "source": [
-        "You will train a model using these datasets by passing them to `Model.fit` in a moment. If you like, you can also manually iterate over the dataset and retrieve batches of images:"
+        "You will pass these datasets to the Keras `Model.fit` method for training later in this tutorial. If you like, you can also manually iterate over the dataset and retrieve batches of images:"
       ]
     },
     {
@@ -415,7 +419,7 @@
       "source": [
         "## Configure the dataset for performance\n",
         "\n",
-        "Let's make sure to use buffered prefetching so you can yield data from disk without having I/O become blocking. These are two important methods you should use when loading data:\n",
+        "Make sure to use buffered prefetching, so you can yield data from disk without having I/O become blocking. These are two important methods you should use when loading data:\n",
         "\n",
         "- `Dataset.cache` keeps the images in memory after they're loaded off disk during the first epoch. This will ensure the dataset does not become a bottleneck while training your model. If your dataset is too large to fit into memory, you can also use this method to create a performant on-disk cache.\n",
         "- `Dataset.prefetch` overlaps data preprocessing and model execution while training.\n",
@@ -489,7 +493,7 @@
         "image_batch, labels_batch = next(iter(normalized_ds))\n",
         "first_image = image_batch[0]\n",
         "# Notice the pixel values are now in `[0,1]`.\n",
-        "print(np.min(first_image), np.max(first_image)) "
+        "print(np.min(first_image), np.max(first_image))"
       ]
     },
     {
@@ -498,7 +502,7 @@
         "id": "XWEOmRSBJ9J8"
       },
       "source": [
-        "Or, you can include the layer inside your model definition, which can simplify deployment. Let's use the second approach here."
+        "Or, you can include the layer inside your model definition, which can simplify deployment. Use the second approach here."
       ]
     },
     {
@@ -516,9 +520,11 @@
         "id": "WcUTyDOPKucd"
       },
       "source": [
-        "# Create the model\n",
+        "## A basic Keras model\n",
         "\n",
-        "The [Sequential](https://www.tensorflow.org/guide/keras/sequential_model) model consists of three convolution blocks (`tf.keras.layers.Conv2D`) with a max pooling layer (`tf.keras.layers.MaxPooling2D`) in each of them. There's a fully-connected layer (`tf.keras.layers.Dense`) with 128 units on top of it that is activated by a ReLU activation function (`'relu'`). This model has not been tuned for high accuracy—the goal of this tutorial is to show a standard approach."
+        "### Create the model\n",
+        "\n",
+        "The Keras [Sequential](https://www.tensorflow.org/guide/keras/sequential_model) model consists of three convolution blocks (`tf.keras.layers.Conv2D`) with a max pooling layer (`tf.keras.layers.MaxPooling2D`) in each of them. There's a fully-connected layer (`tf.keras.layers.Dense`) with 128 units on top of it that is activated by a ReLU activation function (`'relu'`). This model has not been tuned for high accuracy; the goal of this tutorial is to show a standard approach."
       ]
     },
     {
@@ -551,7 +557,7 @@
         "id": "EaKFzz72Lqpg"
       },
       "source": [
-        "## Compile the model\n",
+        "### Compile the model\n",
         "\n",
         "For this tutorial, choose the `tf.keras.optimizers.Adam` optimizer and `tf.keras.losses.SparseCategoricalCrossentropy` loss function. To view training and validation accuracy for each training epoch, pass the `metrics` argument to `Model.compile`."
       ]
@@ -575,9 +581,9 @@
         "id": "aMJ4DnuJL55A"
       },
       "source": [
-        "## Model summary\n",
+        "### Model summary\n",
         "\n",
-        "View all the layers of the network using the model's `Model.summary` method:"
+        "View all the layers of the network using the Keras `Model.summary` method:"
       ]
     },
     {
@@ -597,7 +603,16 @@
         "id": "NiYHcbvaL9H-"
       },
       "source": [
-        "## Train the model"
+        "### Train the model"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "j30F69T4sIVN"
+      },
+      "source": [
+        "Train the model for 10 epochs with the Keras `Model.fit` method:"
       ]
     },
     {
@@ -631,7 +646,7 @@
         "id": "dFvOvmAmMK9w"
       },
       "source": [
-        "Create plots of loss and accuracy on the training and validation sets:"
+        "Create plots of the loss and accuracy on the training and validation sets:"
       ]
     },
     {
@@ -673,7 +688,7 @@
       "source": [
         "The plots show that training accuracy and validation accuracy are off by large margins, and the model has achieved only around 60% accuracy on the validation set.\n",
         "\n",
-        "Let's inspect what went wrong and try to increase the overall performance of the model."
+        "The following tutorial sections show how to inspect what went wrong and try to increase the overall performance of the model."
       ]
     },
     {
@@ -695,7 +710,7 @@
         "\n",
         "When there are a small number of training examples, the model sometimes learns from noises or unwanted details from training examples—to an extent that it negatively impacts the performance of the model on new examples. This phenomenon is known as overfitting. It means that the model will have a difficult time generalizing on a new dataset.\n",
         "\n",
-        "There are multiple ways to fight overfitting in the training process. In this tutorial, you'll use *data augmentation* and add *Dropout* to your model."
+        "There are multiple ways to fight overfitting in the training process. In this tutorial, you'll use *data augmentation* and add *dropout* to your model."
       ]
     },
     {
@@ -744,7 +759,7 @@
         "id": "PN4k1dK3S6eV"
       },
       "source": [
-        "Let's visualize what a few augmented examples look like by applying data augmentation to the same image several times:"
+        "Visualize a few augmented examples by applying data augmentation to the same image several times:"
       ]
     },
     {
@@ -770,7 +785,7 @@
         "id": "tsjXCBLYYNs5"
       },
       "source": [
-        "You will use data augmentation to train a model in a moment."
+        "You will add data augmentation to your model before training in the next step."
       ]
     },
     {
@@ -781,11 +796,11 @@
       "source": [
         "## Dropout\n",
         "\n",
-        "Another technique to reduce overfitting is to introduce [dropout](https://developers.google.com/machine-learning/glossary#dropout_regularization) regularization to the network.\n",
+        "Another technique to reduce overfitting is to introduce [dropout](https://developers.google.com/machine-learning/glossary#dropout_regularization){:.external} regularization to the network.\n",
         "\n",
         "When you apply dropout to a layer, it randomly drops out (by setting the activation to zero) a number of output units from the layer during the training process. Dropout takes a fractional number as its input value, in the form such as 0.1, 0.2, 0.4, etc. This means dropping out 10%, 20% or 40% of the output units randomly from the applied layer.\n",
         "\n",
-        "Let's create a new neural network with `tf.keras.layers.Dropout` before training it using the augmented images:"
+        "Create a new neural network with `tf.keras.layers.Dropout` before training it using the augmented images:"
       ]
     },
     {
@@ -808,7 +823,7 @@
         "  layers.Dropout(0.2),\n",
         "  layers.Flatten(),\n",
         "  layers.Dense(128, activation='relu'),\n",
-        "  layers.Dense(num_classes)\n",
+        "  layers.Dense(num_classes, name=\"outputs\")\n",
         "])"
       ]
     },
@@ -918,7 +933,7 @@
         "id": "10buWpJbcCQz"
       },
       "source": [
-        "Finally, let's use our model to classify an image that wasn't included in the training or validation sets."
+        "Use your model to classify an image that wasn't included in the training or validation sets."
       ]
     },
     {
@@ -955,6 +970,177 @@
         "    .format(class_names[np.argmax(score)], 100 * np.max(score))\n",
         ")"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aOc3PZ2N2r18"
+      },
+      "source": [
+        "## Use TensorFlow Lite\n",
+        "\n",
+        "TensorFlow Lite is a set of tools that enables on-device machine learning by helping developers run their models on mobile, embedded, and edge devices."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cThu25rh4LPP"
+      },
+      "source": [
+        "### Convert the Keras Sequential model to a TensorFlow Lite model\n",
+        "\n",
+        "To use the trained model with on-device applications, first [convert it](https://www.tensorflow.org/lite/models/convert) to a smaller and more efficient model format called a [TensorFlow Lite](https://www.tensorflow.org/lite/) model.\n",
+        "\n",
+        "In this example, take the trained Keras Sequential model and use `tf.lite.TFLiteConverter.from_keras_model` to generate a [TensorFlow Lite](https://www.tensorflow.org/lite/) model:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mXo6ftuL2ufx"
+      },
+      "outputs": [],
+      "source": [
+        "# Convert the model.\n",
+        "converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
+        "tflite_model = converter.convert()\n",
+        "\n",
+        "# Save the model.\n",
+        "with open('model.tflite', 'wb') as f:\n",
+        "  f.write(tflite_model)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4R26OU4gGKhh"
+      },
+      "source": [
+        "The TensorFlow Lite model you saved in the previous step can contain several function signatures. The Keras model converter API uses the default signature automatically. Learn more about [TensorFlow Lite signatures](https://www.tensorflow.org/lite/guide/signatures)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7fjQfXaV2l-5"
+      },
+      "source": [
+        "### Run the TensorFlow Lite model\n",
+        "\n",
+        "You can access the TensorFlow Lite saved model signatures in Python via the `tf.lite.Interpreter` class.\n",
+        "\n",
+        "Load the model with the `Interpreter`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cHYcip_FOaHq"
+      },
+      "outputs": [],
+      "source": [
+        "TF_MODEL_FILE_PATH = 'model.tflite' # The default path to the saved TensorFlow Lite model\n",
+        "\n",
+        "interpreter = tf.lite.Interpreter(model_path=TF_MODEL_FILE_PATH)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nPUXY6BdHDHo"
+      },
+      "source": [
+        "Print the signatures from the converted model to obtain the names of the inputs (and outputs):\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZdDl00E2OaHq"
+      },
+      "outputs": [],
+      "source": [
+        "interpreter.get_signature_list()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4eVFqT0je3YG"
+      },
+      "source": [
+        "In this example, you have one default signature called `serving_default`. In addition, the name of the `'inputs'` is `'sequential_1_input'`, while the `'outputs'` are called `'outputs'`. You can look up these first and last Keras layer names when running `Model.summary`, as demonstrated earlier in this tutorial.\n",
+        "\n",
+        "Now you can test the loaded TensorFlow Model by performing inference on a sample image with `tf.lite.Interpreter.get_signature_runner` by passing the signature name as follows:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "yFoT_7W_OaHq"
+      },
+      "outputs": [],
+      "source": [
+        "classify_lite = interpreter.get_signature_runner('serving_default')\n",
+        "classify_lite"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b1mfRcBOnEx0"
+      },
+      "source": [
+        "Similar to what you did earlier in the tutorial, you can use the TensorFlow Lite model to classify images that weren't included in the training or validation sets.\n",
+        "\n",
+        "You have already tensorized that image and saved it as `img_array`. Now, pass it to the first argument (the name of the `'inputs'`) of the loaded TensorFlow Lite model (`predictions_lite`), compute softmax activations, and then print the prediction for the class with the highest computed probability."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sEqR27YcnFvc"
+      },
+      "outputs": [],
+      "source": [
+        "predictions_lite = classify_lite(sequential_1_input=img_array)['outputs']\n",
+        "score_lite = tf.nn.softmax(predictions_lite)\n",
+        "\n",
+        "assert np.allclose(predictions, predictions_lite)\n",
+        "\n",
+        "print(\n",
+        "    \"This image most likely belongs to {} with a {:.2f} percent confidence.\"\n",
+        "    .format(class_names[np.argmax(score_lite)], 100 * np.max(score_lite))\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5hJzY8XijM7N"
+      },
+      "source": [
+        "Of the five classes—`'daisy'`, `'dandelion'`, `'roses'`, `'sunflowers'`, and `'tulips'`—the model should predict the image belongs to sunflowers, which is the same result as before the TensorFlow Lite conversion.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1RlfCY9v2_ir"
+      },
+      "source": [
+        "## Next steps\n",
+        "\n",
+        "This tutorial showed how to train a model for image classification, test it, convert it to the TensorFlow Lite format for on-device applications (such as an image classification app), and perform inference with the TensorFlow Lite model with the Python API.\n",
+        "\n",
+        "You can learn more about TensorFlow Lite through [tutorials](https://www.tensorflow.org/lite/tutorials) and [guides](https://www.tensorflow.org/lite/guide)."
+      ]
     }
   ],
   "metadata": {

From 74eb0587a8933433f8de591116098337f82f0c41 Mon Sep 17 00:00:00 2001
From: artemboiko1 <108079274+artemboiko1@users.noreply.github.com>
Date: Sat, 13 Aug 2022 21:29:54 -0400
Subject: [PATCH 264/872] Update imbalanced_data.ipynb

remove unneeded symbol
---
 site/en/tutorials/structured_data/imbalanced_data.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/structured_data/imbalanced_data.ipynb b/site/en/tutorials/structured_data/imbalanced_data.ipynb
index 4d7e0f22cab..e51e0fee5b4 100644
--- a/site/en/tutorials/structured_data/imbalanced_data.ipynb
+++ b/site/en/tutorials/structured_data/imbalanced_data.ipynb
@@ -814,7 +814,7 @@
         "    else:\n",
         "      plt.ylim([0,1])\n",
         "\n",
-        "    plt.legend();"
+        "    plt.legend()"
       ]
     },
     {

From 19aaa6932b44643db28549ef50b8b813fddb42a6 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Sun, 14 Aug 2022 22:21:15 -0700
Subject: [PATCH 265/872] Helper for copying cells between notebooks.

Basic tool for copying notebook code cells from one book to another. It copies code cells using matching cell IDs, but does not yet support inserting, deleting or reordering cells - these will need to be done by a human.

It will also run `nbfmt` to make sure the notebook has the correct whitespace, etc. Users can pass `--oss` if operating directly on github files.

Tested with [en/guide/distributed_training.ipynb](https://github.com/tensorflow/docs/blob/master/site/en/guide/distributed_training.ipynb) and [zh_cn/guide/distributed_training.ipynb](https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/guide/distributed_training.ipynb).

PiperOrigin-RevId: 467593397
---
 tools/nbcp/__init__.py                        |  14 ++
 tools/nbcp/__main__.py                        |  92 ++++++++++
 .../tools/nbfmt/notebook_utils.py             |  65 ++++++-
 .../tools/nbfmt/notebook_utils_test.py        | 169 ++++++++++++++++++
 4 files changed, 338 insertions(+), 2 deletions(-)
 create mode 100644 tools/nbcp/__init__.py
 create mode 100644 tools/nbcp/__main__.py
 create mode 100644 tools/tensorflow_docs/tools/nbfmt/notebook_utils_test.py

diff --git a/tools/nbcp/__init__.py b/tools/nbcp/__init__.py
new file mode 100644
index 00000000000..78cb171abba
--- /dev/null
+++ b/tools/nbcp/__init__.py
@@ -0,0 +1,14 @@
+# Copyright 2022 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
diff --git a/tools/nbcp/__main__.py b/tools/nbcp/__main__.py
new file mode 100644
index 00000000000..c85b22129d1
--- /dev/null
+++ b/tools/nbcp/__main__.py
@@ -0,0 +1,92 @@
+# Copyright 2022 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Utility for copying cells between notebooks."""
+import pathlib
+import sys
+import textwrap
+
+from absl import app
+import nbformat
+
+from tensorflow_docs.tools.nbfmt import __main__ as nbfmt
+from tensorflow_docs.tools.nbfmt import notebook_utils
+
+
+def process_stats(stats: notebook_utils.CellCopyStats) -> bool:
+  """Displays summary stats to the user. Returns True if any warnings."""
+  print(
+      textwrap.dedent(f"""
+      Notebook copy complete.
+        - Total code cells processed: {stats.processed_cells}
+        - Cells updated: {stats.updated_cells}
+      """))
+
+  has_warnings = any((
+      stats.unmatched_target_cells,
+      stats.unmatched_source_cells,
+      stats.out_of_order_target_cells,
+  ))
+  if has_warnings:
+    print('Warnings:')
+
+    if stats.unmatched_target_cells:
+      notebook_utils.warn(
+          '- Cells in source notebook that are not in the destination: '
+          f'{" ".join(stats.unmatched_target_cells)}')
+
+    if stats.unmatched_source_cells:
+      notebook_utils.warn(
+          '- Cells in destination notebook that are not in the source: '
+          f'{" ".join(stats.unmatched_source_cells)}')
+
+    if stats.out_of_order_target_cells:
+      notebook_utils.warn(
+          '- Cells found earlier in destination notebook than source: '
+          f'{" ".join(stats.out_of_order_target_cells)}')
+
+    print()
+
+  return has_warnings
+
+
+def main(args: list[str]) -> int:
+  if len(args) != 3:
+    notebook_utils.warn('nbcp requires 2 notebooks as arguments:')
+    notebook_utils.warn('  $ ...nbcp src_notebook.ipynb dest_notebook.ipynb'
+                        ' [--nbfmt --args --supported]')
+    sys.exit(1)
+
+  src = pathlib.Path(args[1])
+  dest = pathlib.Path(args[2])
+
+  # Open files and copy cells.
+  with src.open('rt') as src_fh, dest.open('rt') as dest_fh:
+    dest_nb = nbformat.read(dest_fh, nbformat.NO_CONVERT)
+    stats = notebook_utils.copy_code_cells(
+        nbformat.read(src_fh, nbformat.NO_CONVERT), dest_nb)
+
+  # Write over destination file.
+  with dest.open('wt') as dest_fh:
+    nbformat.write(dest_nb, dest_fh)
+
+  warnings = process_stats(stats)
+
+  # Format the notebook.
+  nbfmt.main(['', str(dest)])
+
+  return int(warnings)
+
+
+if __name__ == '__main__':
+  app.run(main)
diff --git a/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py b/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py
index 577f2e1e18e..6e5e8a36553 100644
--- a/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py
+++ b/tools/tensorflow_docs/tools/nbfmt/notebook_utils.py
@@ -12,15 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-r"""A collection of utilties for working with notebook files."""
-import json
+"""A collection of utilities for working with notebook files."""
+import dataclasses
 import hashlib
+import json
+import logging
 import pathlib
 import sys
 import textwrap
 
 from typing import Any, Dict, List, Optional, Tuple, Union
 
+from nbformat import notebooknode
+
 
 def collect_notebook_paths(
     filepaths: List[Union[str, pathlib.Path]]
@@ -109,3 +113,60 @@ def del_entries_except(data: Dict[str, Any], keep: List[str]) -> None:
   to_delete = set(data.keys()) - frozenset(keep)
   for key in to_delete:
     del data[key]
+
+
+@dataclasses.dataclass
+class CellCopyStats:
+  processed_cells: int = 0
+  updated_cells: int = 0
+  unmatched_target_cells: list[str] = dataclasses.field(default_factory=list)
+  unmatched_source_cells: list[str] = dataclasses.field(default_factory=list)
+  out_of_order_target_cells: list[str] = dataclasses.field(default_factory=list)
+
+
+def copy_code_cells(source: notebooknode.NotebookNode,
+                    target: notebooknode.NotebookNode) -> CellCopyStats:
+  """Copies code cell source and outputs from source to target."""
+  stats = CellCopyStats()
+  if len(source.cells) != len(target.cells):
+    logging.warning('Source and target notebook have unequal cell counts.')
+
+  target_indices = {c['metadata']['id']: i for i, c in enumerate(target.cells)}
+
+  last_target_idx = -1
+  for cell in source.cells:
+    cell_id = cell['metadata']['id']
+
+    if cell.get('cell_type') != 'code':
+      target_indices.pop(cell_id, None)
+      continue
+
+    if cell_id not in target_indices:
+      logging.warning('Cell %s is not present in the target notebook.', cell_id)
+      stats.unmatched_target_cells.append(cell_id)
+      continue
+
+    stats.processed_cells += 1
+
+    if last_target_idx > (target_idx := target_indices.pop(cell_id)):
+      logging.warning(
+          'Cell %s has been moved earlier in the notebook than expected.',
+          cell_id)
+      stats.out_of_order_target_cells.append(cell_id)
+
+    target_cell = target.cells[target_idx]
+    modified = False
+    for field in 'source', 'outputs':
+      new_value = cell.get(field)
+      if target_cell.get(field) != new_value:
+        target_cell[field] = new_value
+        modified = True
+
+    stats.updated_cells += modified
+    last_target_idx = target_idx
+
+  stats.unmatched_source_cells = [
+      c for c, i in target_indices.items()
+      if target.cells[i].get('cell_type') == 'code'
+  ]
+  return stats
diff --git a/tools/tensorflow_docs/tools/nbfmt/notebook_utils_test.py b/tools/tensorflow_docs/tools/nbfmt/notebook_utils_test.py
new file mode 100644
index 00000000000..4866dcb21b8
--- /dev/null
+++ b/tools/tensorflow_docs/tools/nbfmt/notebook_utils_test.py
@@ -0,0 +1,169 @@
+# Copyright 2022 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Unit tests for notebook_utils."""
+
+from absl.testing import absltest
+from nbformat import notebooknode
+
+from tensorflow_docs.tools.nbfmt import notebook_utils
+
+
+class NotebookUtilsTest(absltest.TestCase):
+
+  def test_copy_code_cells(self):
+    source_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "markdown",
+            "metadata": {
+                "id": "id1"
+            },
+            "source": ["## Some text"]
+        }, {
+            "cell_type": "code",
+            "metadata": {
+                "id": "id2"
+            },
+            "source": ["# some python\n", "print('hi')"]
+        }]
+    })
+    target_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "markdown",
+            "metadata": {
+                "id": "id1"
+            },
+            "source": ["## Different text"]
+        }, {
+            "cell_type": "code",
+            "metadata": {
+                "id": "id2"
+            },
+            "source": ["# some old python\n", "print 'hi'"]
+        }]
+    })
+
+    stat = notebook_utils.copy_code_cells(source_notebook, target_notebook)
+
+    # Ensure we have the expected contents (markdown untouched, code copied)
+    self.assertIn("## Different text", target_notebook.cells[0]["source"])
+    self.assertIn("print('hi')", target_notebook.cells[1]["source"])
+
+    # Ensure only the code cell was updated
+    self.assertEqual(1, stat.updated_cells)
+    self.assertEqual(1, stat.processed_cells)
+
+  def test_missing_target_cell(self):
+    source_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell1"
+            },
+            "source": ["# some python\n", "print('hi')"]
+        }, {
+            "cell_type": "markdown",
+            "metadata": {
+                "id": "md1"
+            },
+            "source": ["## text"]
+        }]
+    })
+    target_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell2"
+            },
+            "source": ["# some old python\n", "print 'hi'"]
+        }]
+    })
+
+    stat = notebook_utils.copy_code_cells(source_notebook, target_notebook)
+
+    self.assertEqual(0, stat.updated_cells)
+    self.assertEqual(0, stat.processed_cells)
+    self.assertEqual(["cell1"], stat.unmatched_target_cells)
+
+  def test_missing_source_cell(self):
+    source_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell1"
+            },
+            "source": ["# some python\n", "print('hi')"]
+        }]
+    })
+    target_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell2"
+            },
+            "source": ["# some old python\n", "print 'hi'"]
+        }, {
+            "cell_type": "markdown",
+            "metadata": {
+                "id": "text1"
+            },
+            "source": ["## texty texty"]
+        }]
+    })
+
+    stat = notebook_utils.copy_code_cells(source_notebook, target_notebook)
+
+    self.assertEqual(0, stat.updated_cells)
+    self.assertEqual(0, stat.processed_cells)
+    self.assertEqual(["cell2"], stat.unmatched_source_cells)
+
+  def test_cell_ordering(self):
+    source_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell1"
+            },
+            "source": ["# first code\n"]
+        }, {
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell2"
+            },
+            "source": ["# second code\n"]
+        }]
+    })
+    target_notebook = notebooknode.NotebookNode({
+        "cells": [{
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell2"
+            },
+            "source": ["# update me\n"]
+        }, {
+            "cell_type": "code",
+            "metadata": {
+                "id": "cell1"
+            },
+            "source": ["# update me\n"]
+        }]
+    })
+
+    stat = notebook_utils.copy_code_cells(source_notebook, target_notebook)
+
+    self.assertEqual(2, stat.updated_cells)
+    self.assertIn("cell2", stat.out_of_order_target_cells)
+
+
+if __name__ == "__main__":
+  absltest.main()

From e513e01d0f202f697b9e27209f6078e888417e0a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Johannes=20Ball=C3=A9?= 
Date: Mon, 15 Aug 2022 09:26:34 -0700
Subject: [PATCH 266/872] Adds model compression tutorial.

PiperOrigin-RevId: 467687576
---
 site/en/tutorials/_toc.yaml                   |   12 +
 .../tutorials/optimization/compression.ipynb  | 1183 +++++++++++++++++
 2 files changed, 1195 insertions(+)
 create mode 100644 site/en/tutorials/optimization/compression.ipynb

diff --git a/site/en/tutorials/_toc.yaml b/site/en/tutorials/_toc.yaml
index 7c629d69dea..188a0975aa0 100644
--- a/site/en/tutorials/_toc.yaml
+++ b/site/en/tutorials/_toc.yaml
@@ -135,6 +135,7 @@ toc:
     status: external
   - title: "Image captioning"
     path: /tutorials/text/image_captioning
+
 - title: "Audio"
   style: accordion
   section:
@@ -186,6 +187,16 @@ toc:
     path: /tutorials/generative/data_compression
     status: new
 
+- title: "Model optimization"
+  style: accordion
+  section:
+  - title: "Scalable model compression with EPR"
+    path: /tutorials/optimization/compression
+    status: new
+  - title: "TensorFlow model optimization"
+    status: external
+    path: /model_optimization
+
 - title: "Model Understanding"
   style: accordion
   section:
@@ -196,6 +207,7 @@ toc:
   - title: "Probabilistic regression"
     path: /probability/examples/Probabilistic_Layers_Regression
     status: external
+
 - title: "Reinforcement learning"
   style: accordion
   section:
diff --git a/site/en/tutorials/optimization/compression.ipynb b/site/en/tutorials/optimization/compression.ipynb
new file mode 100644
index 00000000000..75b5517aa57
--- /dev/null
+++ b/site/en/tutorials/optimization/compression.ipynb
@@ -0,0 +1,1183 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Tce3stUlHN0L"
+      },
+      "source": [
+        "##### Copyright 2022 The TensorFlow Compression Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "tuOe1ymfHZPu"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qFdPvlXBOdUN"
+      },
+      "source": [
+        "# Scalable model compression"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MfBg1C5NB3X0"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    \n",
+        "    \n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    \n",
+        "    \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    \n",
+        "    \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "xHxb-dlhMIzW"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This notebook shows how to compress a model using [TensorFlow Compression](https://github.com/tensorflow/compression).\n",
+        "\n",
+        "In the example below, we compress the weights of an MNIST classifier to a much smaller size than their floating point representation, while retaining classification accuracy. This is done by a two step process, based on the paper [Scalable Model Compression by Entropy Penalized Reparameterization](https://arxiv.org/abs/1906.06624):\n",
+        "\n",
+        "- Training a \"compressible\" model with an explicit **entropy penalty** during training, which encourages compressibility of the model parameters. The weight on this penalty, $\\lambda$, enables continuously controlling the trade-off between the compressed model size and its accuracy.\n",
+        "\n",
+        "- Encoding the compressible model into a compressed model using a coding scheme that is matched with the penalty, meaning that the penalty is a good predictor for model size. This ensures that the method doesn't require multiple iterations of training, compressing, and re-training the model for fine-tuning.\n",
+        "\n",
+        "This method is strictly concerned with compressed model size, not with computational complexity. It can be combined with a technique like model pruning to reduce size and complexity.\n",
+        "\n",
+        "Example compression results on various models:\n",
+        "\n",
+        "Model (dataset)         | Model size | Comp. ratio | Top-1 error comp. (uncomp.)\n",
+        "------------------------|------------|-------------|------------\n",
+        "LeNet300-100 (MNIST)    | 8.56 KB    | 124x        |  1.9%  (1.6%)\n",
+        "LeNet5-Caffe (MNIST)    | 2.84 KB    | 606x        |  1.0%  (0.7%)\n",
+        "VGG-16 (CIFAR-10)       | 101 KB     | 590x        | 10.0%  (6.6%)\n",
+        "ResNet-20-4 (CIFAR-10)  | 128 KB     | 134x        |  8.8%  (5.0%)\n",
+        "ResNet-18 (ImageNet)    | 1.97 MB    |  24x        | 30.0% (30.0%)\n",
+        "ResNet-50 (ImageNet)    | 5.49 MB    |  19x        | 26.0% (25.0%)\n",
+        "\n",
+        "Applications include:\n",
+        "- Deploying/broadcasting models to edge devices on a large scale, saving bandwidth in transit.\n",
+        "- Communicating global model state to clients in federated learning. The model architecture (number of hidden units, etc.) is unchanged from the initial model, and clients can continue learning on the decompressed model.\n",
+        "- Performing inference on extremely memory limited clients. During inference, the weights of each layer can be sequentially decompressed, and discarded right after the activations are computed."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MUXex9ctTuDB"
+      },
+      "source": [
+        "## Setup\n",
+        "\n",
+        "Install Tensorflow Compression via `pip`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K489KsEgxuLI"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install tensorflow-compression~=$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d\\.\\d).*/\\1.0/sg')\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WfVAmHCVxpTS"
+      },
+      "source": [
+        "Import library dependencies."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IqR2PQG4ZaZ0"
+      },
+      "outputs": [],
+      "source": [
+        "import matplotlib.pyplot as plt\n",
+        "import tensorflow as tf\n",
+        "import tensorflow_compression as tfc\n",
+        "import tensorflow_datasets as tfds\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "wsncKT2iymgQ"
+      },
+      "source": [
+        "## Define and train a basic MNIST classifier\n",
+        "\n",
+        "In order to effectively compress dense and convolutional layers, we need to define custom layer classes. These are analogous to the layers under `tf.keras.layers`, but we will subclass them later to effectively implement Entropy Penalized Reparameterization (EPR). For this purpose, we also add a copy constructor.\n",
+        "\n",
+        "First, we define a standard dense layer:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "n_7ZRqiaO1WQ"
+      },
+      "outputs": [],
+      "source": [
+        "class CustomDense(tf.keras.layers.Layer):\n",
+        "\n",
+        "  def __init__(self, filters, name=\"dense\"):\n",
+        "    super().__init__(name=name)\n",
+        "    self.filters = filters\n",
+        "\n",
+        "  @classmethod\n",
+        "  def copy(cls, other, **kwargs):\n",
+        "    \"\"\"Returns an instantiated and built layer, initialized from `other`.\"\"\"\n",
+        "    self = cls(filters=other.filters, name=other.name, **kwargs)\n",
+        "    self.build(None, other=other)\n",
+        "    return self\n",
+        "\n",
+        "  def build(self, input_shape, other=None):\n",
+        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
+        "    if other is None:\n",
+        "      kernel_shape = (input_shape[-1], self.filters)\n",
+        "      kernel = tf.keras.initializers.GlorotUniform()(shape=kernel_shape)\n",
+        "      bias = tf.keras.initializers.Zeros()(shape=(self.filters,))\n",
+        "    else:\n",
+        "      kernel, bias = other.kernel, other.bias\n",
+        "    self.kernel = tf.Variable(\n",
+        "        tf.cast(kernel, self.variable_dtype), name=\"kernel\")\n",
+        "    self.bias = tf.Variable(\n",
+        "        tf.cast(bias, self.variable_dtype), name=\"bias\")\n",
+        "    self.built = True\n",
+        "\n",
+        "  def call(self, inputs):\n",
+        "    outputs = tf.linalg.matvec(self.kernel, inputs, transpose_a=True)\n",
+        "    outputs = tf.nn.bias_add(outputs, self.bias)\n",
+        "    return tf.nn.leaky_relu(outputs)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "RUZkcXegc0yR"
+      },
+      "source": [
+        "And similarly, a 2D convolutional layer:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "RDibtb8EWCSj"
+      },
+      "outputs": [],
+      "source": [
+        "class CustomConv2D(tf.keras.layers.Layer):\n",
+        "\n",
+        "  def __init__(self, filters, kernel_size,\n",
+        "               strides=1, padding=\"SAME\", name=\"conv2d\"):\n",
+        "    super().__init__(name=name)\n",
+        "    self.filters = filters\n",
+        "    self.kernel_size = kernel_size\n",
+        "    self.strides = strides\n",
+        "    self.padding = padding\n",
+        "\n",
+        "  @classmethod\n",
+        "  def copy(cls, other, **kwargs):\n",
+        "    \"\"\"Returns an instantiated and built layer, initialized from `other`.\"\"\"\n",
+        "    self = cls(filters=other.filters, kernel_size=other.kernel_size,\n",
+        "               strides=other.strides, padding=other.padding, name=other.name,\n",
+        "               **kwargs)\n",
+        "    self.build(None, other=other)\n",
+        "    return self\n",
+        "\n",
+        "  def build(self, input_shape, other=None):\n",
+        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
+        "    if other is None:\n",
+        "      kernel_shape = 2 * (self.kernel_size,) + (input_shape[-1], self.filters)\n",
+        "      kernel = tf.keras.initializers.GlorotUniform()(shape=kernel_shape)\n",
+        "      bias = tf.keras.initializers.Zeros()(shape=(self.filters,))\n",
+        "    else:\n",
+        "      kernel, bias = other.kernel, other.bias\n",
+        "    self.kernel = tf.Variable(\n",
+        "        tf.cast(kernel, self.variable_dtype), name=\"kernel\")\n",
+        "    self.bias = tf.Variable(\n",
+        "        tf.cast(bias, self.variable_dtype), name=\"bias\")\n",
+        "    self.built = True\n",
+        "\n",
+        "  def call(self, inputs):\n",
+        "    outputs = tf.nn.convolution(\n",
+        "        inputs, self.kernel, strides=self.strides, padding=self.padding)\n",
+        "    outputs = tf.nn.bias_add(outputs, self.bias)\n",
+        "    return tf.nn.leaky_relu(outputs)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6xWa1hHMdCpG"
+      },
+      "source": [
+        "Before we continue with model compression, let's check that we can successfully train a regular classifier.\n",
+        "\n",
+        "Define the model architecture:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8yZESLgW-vp1"
+      },
+      "outputs": [],
+      "source": [
+        "classifier = tf.keras.Sequential([\n",
+        "    CustomConv2D(20, 5, strides=2, name=\"conv_1\"),\n",
+        "    CustomConv2D(50, 5, strides=2, name=\"conv_2\"),\n",
+        "    tf.keras.layers.Flatten(),\n",
+        "    CustomDense(500, name=\"fc_1\"),\n",
+        "    CustomDense(10, name=\"fc_2\"),\n",
+        "], name=\"classifier\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9iRSvt_CdUuY"
+      },
+      "source": [
+        "Load the training data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "L4bsA3HFF2k0"
+      },
+      "outputs": [],
+      "source": [
+        "def normalize_img(image, label):\n",
+        "  \"\"\"Normalizes images: `uint8` -> `float32`.\"\"\"\n",
+        "  return tf.cast(image, tf.float32) / 255., label\n",
+        "\n",
+        "training_dataset, validation_dataset = tfds.load(\n",
+        "    \"mnist\",\n",
+        "    split=[\"train\", \"test\"],\n",
+        "    shuffle_files=True,\n",
+        "    as_supervised=True,\n",
+        "    with_info=False,\n",
+        ")\n",
+        "training_dataset = training_dataset.map(normalize_img)\n",
+        "validation_dataset = validation_dataset.map(normalize_img)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rR9WYjt_daRG"
+      },
+      "source": [
+        "Finally, train the model:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ROn2DbzsBirI"
+      },
+      "outputs": [],
+      "source": [
+        "def train_model(model, training_data, validation_data, **kwargs):\n",
+        "  model.compile(\n",
+        "      optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),\n",
+        "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
+        "      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],\n",
+        "      # Uncomment this to ease debugging:\n",
+        "      # run_eagerly=True,\n",
+        "  )\n",
+        "  kwargs.setdefault(\"epochs\", 5)\n",
+        "  kwargs.setdefault(\"verbose\", 1)\n",
+        "  log = model.fit(\n",
+        "      training_data.batch(128).prefetch(8),\n",
+        "      validation_data=validation_data.batch(128).cache(),\n",
+        "      validation_freq=1,\n",
+        "      **kwargs,\n",
+        "  )\n",
+        "  return log.history[\"val_sparse_categorical_accuracy\"][-1]\n",
+        "\n",
+        "classifier_accuracy = train_model(\n",
+        "    classifier, training_dataset, validation_dataset)\n",
+        "\n",
+        "print(f\"Accuracy: {classifier_accuracy:0.4f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "QupWKZ91di-y"
+      },
+      "source": [
+        "Success! The model trained fine, and reached an accuracy of over 98% on the validation set within 5 epochs."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yRqZFwb5dqQm"
+      },
+      "source": [
+        "## Train a compressible classifier\n",
+        "\n",
+        "Entropy Penalized Reparameterization (EPR) has two main ingredients:\n",
+        "\n",
+        "- Applying a **penalty** to the model weights during training which corresponds to their entropy under a probabilistic model, which is matched with the encoding scheme of the weights. Below, we define a Keras `Regularizer` which implements this penalty.\n",
+        "\n",
+        "- **Reparameterizing** the weights, i.e. bringing them into a latent representation which is more compressible (yields a better trade-off between compressibility and model performance). For convolutional kernels, [it has been shown](https://arxiv.org/abs/1906.06624) that the Fourier domain is a good representation. For other parameters, the below example simply uses scalar quantization (rounding) with a varying quantization step size."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e4jmnqEmO6eB"
+      },
+      "source": [
+        "First, define the penalty.\n",
+        "\n",
+        "The example below uses a code/probabilistic model implemented in the `tfc.PowerLawEntropyModel` class, inspired by the paper [Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory](https://arxiv.org/abs/2201.02664). The penalty is defined as:\n",
+        "$$ \\log \\Bigl(\\frac {|x| + \\alpha} \\alpha\\Bigr), $$\n",
+        "where $x$ is one element of the model parameter or its latent representation, and $\\alpha$ is a small constant for numerical stability around values of 0."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hh57nxjuwocc"
+      },
+      "outputs": [],
+      "source": [
+        "_ = tf.linspace(-5., 5., 501)\n",
+        "plt.plot(_, tfc.PowerLawEntropyModel(0).penalty(_));\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Gr3-6vLrwo-H"
+      },
+      "source": [
+        "The penalty is effectively a regularization loss (sometimes called \"weight loss\"). The fact that it is concave with a cusp at zero encourages weight sparsity. The coding scheme applied for compressing the weights, an [Elias gamma code](https://en.wikipedia.org/wiki/Elias_gamma_coding), produces codes of length $ 1 + \\lfloor \\log_2 |x| \\rfloor $ bits for the magnitude of the element. That is, it is matched to the penalty, and applying the penalty thus minimizes the expected code length."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "H1Yt6e1ub6pU"
+      },
+      "outputs": [],
+      "source": [
+        "class PowerLawRegularizer(tf.keras.regularizers.Regularizer):\n",
+        "\n",
+        "  def __init__(self, lmbda):\n",
+        "    super().__init__()\n",
+        "    self.lmbda = lmbda\n",
+        "\n",
+        "  def __call__(self, variable):\n",
+        "    em = tfc.PowerLawEntropyModel(coding_rank=variable.shape.rank)\n",
+        "    return self.lmbda * em.penalty(variable)\n",
+        "\n",
+        "# Normalizing the weight of the penalty by the number of model parameters is a\n",
+        "# good rule of thumb to produce comparable results across models.\n",
+        "regularizer = PowerLawRegularizer(lmbda=2./classifier.count_params())\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "kyQc35QTf8Aq"
+      },
+      "source": [
+        "Second, define subclasses of `CustomDense` and `CustomConv2D` which have the following additional functionality:\n",
+        "\n",
+        "- They take an instance of the above regularizer and apply it to the kernels and biases during training.\n",
+        "- They define kernel and bias as a `@property`, which perform quantization with straight-through gradients whenever the variables are accessed. This accurately reflects the computation that is carried out later in the compressed model.\n",
+        "- They define additional `log_step` variables, which represent the logarithm of the quantization step size. The coarser the quantization, the smaller the model size, but the lower the accuracy. The quantization step sizes are trainable for each model parameter, so that performing optimization on the penalized loss function will determine what quantization step size is best.\n",
+        "\n",
+        "The quantization step is defined as follows:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "60fMt3avgSFw"
+      },
+      "outputs": [],
+      "source": [
+        "def quantize(latent, log_step):\n",
+        "  step = tf.exp(log_step)\n",
+        "  return tfc.round_st(latent / step) * step\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "stKrchp7mB0b"
+      },
+      "source": [
+        "With that, we can define the dense layer:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Ciz1F1WsXre_"
+      },
+      "outputs": [],
+      "source": [
+        "class CompressibleDense(CustomDense):\n",
+        "\n",
+        "  def __init__(self, regularizer, *args, **kwargs):\n",
+        "    super().__init__(*args, **kwargs)\n",
+        "    self.regularizer = regularizer\n",
+        "\n",
+        "  def build(self, input_shape, other=None):\n",
+        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
+        "    super().build(input_shape, other=other)\n",
+        "    if other is not None and hasattr(other, \"kernel_log_step\"):\n",
+        "      kernel_log_step = other.kernel_log_step\n",
+        "      bias_log_step = other.bias_log_step\n",
+        "    else:\n",
+        "      kernel_log_step = bias_log_step = -4.\n",
+        "    self.kernel_log_step = tf.Variable(\n",
+        "        tf.cast(kernel_log_step, self.variable_dtype), name=\"kernel_log_step\")\n",
+        "    self.bias_log_step = tf.Variable(\n",
+        "        tf.cast(bias_log_step, self.variable_dtype), name=\"bias_log_step\")\n",
+        "    self.add_loss(lambda: self.regularizer(\n",
+        "        self.kernel_latent / tf.exp(self.kernel_log_step)))\n",
+        "    self.add_loss(lambda: self.regularizer(\n",
+        "        self.bias_latent / tf.exp(self.bias_log_step)))\n",
+        "\n",
+        "  @property\n",
+        "  def kernel(self):\n",
+        "    return quantize(self.kernel_latent, self.kernel_log_step)\n",
+        "\n",
+        "  @kernel.setter\n",
+        "  def kernel(self, kernel):\n",
+        "    self.kernel_latent = tf.Variable(kernel, name=\"kernel_latent\")\n",
+        "\n",
+        "  @property\n",
+        "  def bias(self):\n",
+        "    return quantize(self.bias_latent, self.bias_log_step)\n",
+        "\n",
+        "  @bias.setter\n",
+        "  def bias(self, bias):\n",
+        "    self.bias_latent = tf.Variable(bias, name=\"bias_latent\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CsykbQO0hxzW"
+      },
+      "source": [
+        "The convolutional layer is analogous. In addition, the convolution kernel is stored as its real-valued discrete Fourier transform (RDFT) whenever the kernel is set, and the transform is inverted whenever the kernel is used. Since the different frequency components of the kernel tend to be more or less compressible, each of them gets its own quantization step size assigned.\n",
+        "\n",
+        "Define the Fourier transform and its inverse as follows:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "rUFMKGHDguJS"
+      },
+      "outputs": [],
+      "source": [
+        "def to_rdft(kernel, kernel_size):\n",
+        "  # The kernel has shape (H, W, I, O) -> transpose to take DFT over last two\n",
+        "  # dimensions.\n",
+        "  kernel = tf.transpose(kernel, (2, 3, 0, 1))\n",
+        "  # The RDFT has type complex64 and shape (I, O, FH, FW).\n",
+        "  kernel_rdft = tf.signal.rfft2d(kernel)\n",
+        "  # Map real and imaginary parts into regular floats. The result is float32\n",
+        "  # and has shape (I, O, FH, FW, 2).\n",
+        "  kernel_rdft = tf.stack(\n",
+        "      [tf.math.real(kernel_rdft), tf.math.imag(kernel_rdft)], axis=-1)\n",
+        "  # Divide by kernel size to make the DFT orthonormal (length-preserving).\n",
+        "  return kernel_rdft / kernel_size\n",
+        "\n",
+        "def from_rdft(kernel_rdft, kernel_size):\n",
+        "  # Undoes the transformations in to_rdft.\n",
+        "  kernel_rdft *= kernel_size\n",
+        "  kernel_rdft = tf.dtypes.complex(*tf.unstack(kernel_rdft, axis=-1))\n",
+        "  kernel = tf.signal.irfft2d(kernel_rdft, fft_length=2 * (kernel_size,))\n",
+        "  return tf.transpose(kernel, (2, 3, 0, 1))\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "esZZrJ5ImVDY"
+      },
+      "source": [
+        "With that, define the convolutional layer as:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YKzXBNCO7bjB"
+      },
+      "outputs": [],
+      "source": [
+        "class CompressibleConv2D(CustomConv2D):\n",
+        "\n",
+        "  def __init__(self, regularizer, *args, **kwargs):\n",
+        "    super().__init__(*args, **kwargs)\n",
+        "    self.regularizer = regularizer\n",
+        "\n",
+        "  def build(self, input_shape, other=None):\n",
+        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
+        "    super().build(input_shape, other=other)\n",
+        "    if other is not None and hasattr(other, \"kernel_log_step\"):\n",
+        "      kernel_log_step = other.kernel_log_step\n",
+        "      bias_log_step = other.bias_log_step\n",
+        "    else:\n",
+        "      kernel_log_step = tf.fill(self.kernel_latent.shape[2:], -4.)\n",
+        "      bias_log_step = -4.\n",
+        "    self.kernel_log_step = tf.Variable(\n",
+        "        tf.cast(kernel_log_step, self.variable_dtype), name=\"kernel_log_step\")\n",
+        "    self.bias_log_step = tf.Variable(\n",
+        "        tf.cast(bias_log_step, self.variable_dtype), name=\"bias_log_step\")\n",
+        "    self.add_loss(lambda: self.regularizer(\n",
+        "        self.kernel_latent / tf.exp(self.kernel_log_step)))\n",
+        "    self.add_loss(lambda: self.regularizer(\n",
+        "        self.bias_latent / tf.exp(self.bias_log_step)))\n",
+        "\n",
+        "  @property\n",
+        "  def kernel(self):\n",
+        "    kernel_rdft = quantize(self.kernel_latent, self.kernel_log_step)\n",
+        "    return from_rdft(kernel_rdft, self.kernel_size)\n",
+        "\n",
+        "  @kernel.setter\n",
+        "  def kernel(self, kernel):\n",
+        "    kernel_rdft = to_rdft(kernel, self.kernel_size)\n",
+        "    self.kernel_latent = tf.Variable(kernel_rdft, name=\"kernel_latent\")\n",
+        "\n",
+        "  @property\n",
+        "  def bias(self):\n",
+        "    return quantize(self.bias_latent, self.bias_log_step)\n",
+        "\n",
+        "  @bias.setter\n",
+        "  def bias(self, bias):\n",
+        "    self.bias_latent = tf.Variable(bias, name=\"bias_latent\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1-ekDDQ9jidI"
+      },
+      "source": [
+        "Define a classifier model with the same architecture as above, but using these modified layers:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TQgp84L7qalw"
+      },
+      "outputs": [],
+      "source": [
+        "def make_mnist_classifier(regularizer):\n",
+        "  return tf.keras.Sequential([\n",
+        "      CompressibleConv2D(regularizer, 20, 5, strides=2, name=\"conv_1\"),\n",
+        "      CompressibleConv2D(regularizer, 50, 5, strides=2, name=\"conv_2\"),\n",
+        "      tf.keras.layers.Flatten(),\n",
+        "      CompressibleDense(regularizer, 500, name=\"fc_1\"),\n",
+        "      CompressibleDense(regularizer, 10, name=\"fc_2\"),\n",
+        "  ], name=\"classifier\")\n",
+        "\n",
+        "compressible_classifier = make_mnist_classifier(regularizer)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hJ-TMHE1kNFc"
+      },
+      "source": [
+        "And train the model:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6L5ZJAX4EiXW"
+      },
+      "outputs": [],
+      "source": [
+        "penalized_accuracy = train_model(\n",
+        "    compressible_classifier, training_dataset, validation_dataset)\n",
+        "\n",
+        "print(f\"Accuracy: {penalized_accuracy:0.4f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ZuE4NeY_kTDz"
+      },
+      "source": [
+        "The compressible model has reached a similar accuracy as the plain classifier.\n",
+        "\n",
+        "However, the model is not actually compressed yet. To do this, we define another set of subclasses which store the kernels and biases in their compressed form – as a sequence of bits."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AZhj8A2gnBkD"
+      },
+      "source": [
+        "## Compress the classifier\n",
+        "\n",
+        "The subclasses of `CustomDense` and `CustomConv2D` defined below convert the weights of a compressible dense layer into binary strings. In addition, they store the logarithm of the quantization step size at half precision to save space. Whenever the kernel or bias is accessed through the `@property`, they are decompressed from their string representation and dequantized.\n",
+        "\n",
+        "First, define functions to compress and decompress a model parameter:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xS19FhDajeto"
+      },
+      "outputs": [],
+      "source": [
+        "def compress_latent(latent, log_step, name):\n",
+        "  em = tfc.PowerLawEntropyModel(latent.shape.rank)\n",
+        "  compressed = em.compress(latent / tf.exp(log_step))\n",
+        "  compressed = tf.Variable(compressed, name=f\"{name}_compressed\")\n",
+        "  log_step = tf.cast(log_step, tf.float16)\n",
+        "  log_step = tf.Variable(log_step, name=f\"{name}_log_step\")\n",
+        "  return compressed, log_step\n",
+        "\n",
+        "def decompress_latent(compressed, shape, log_step):\n",
+        "  latent = tfc.PowerLawEntropyModel(len(shape)).decompress(compressed, shape)\n",
+        "  step = tf.exp(tf.cast(log_step, latent.dtype))\n",
+        "  return latent * step\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bPPABE9fjqHJ"
+      },
+      "source": [
+        "With these, we can define `CompressedDense`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CnaiNzhgaZ7s"
+      },
+      "outputs": [],
+      "source": [
+        "class CompressedDense(CustomDense):\n",
+        "\n",
+        "  def build(self, input_shape, other=None):\n",
+        "    assert isinstance(other, CompressibleDense)\n",
+        "    self.input_channels = other.kernel.shape[0]\n",
+        "    self.kernel_compressed, self.kernel_log_step = compress_latent(\n",
+        "        other.kernel_latent, other.kernel_log_step, \"kernel\")\n",
+        "    self.bias_compressed, self.bias_log_step = compress_latent(\n",
+        "        other.bias_latent, other.bias_log_step, \"bias\")\n",
+        "    self.built = True\n",
+        "\n",
+        "  @property\n",
+        "  def kernel(self):\n",
+        "    kernel_shape = (self.input_channels, self.filters)\n",
+        "    return decompress_latent(\n",
+        "        self.kernel_compressed, kernel_shape, self.kernel_log_step)\n",
+        "\n",
+        "  @property\n",
+        "  def bias(self):\n",
+        "    bias_shape = (self.filters,)\n",
+        "    return decompress_latent(\n",
+        "        self.bias_compressed, bias_shape, self.bias_log_step)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tzvMCM0El2iW"
+      },
+      "source": [
+        "The convolutional layer class is analogous to the above."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hS-2ADA6iWeQ"
+      },
+      "outputs": [],
+      "source": [
+        "class CompressedConv2D(CustomConv2D):\n",
+        "\n",
+        "  def build(self, input_shape, other=None):\n",
+        "    assert isinstance(other, CompressibleConv2D)\n",
+        "    self.input_channels = other.kernel.shape[2]\n",
+        "    self.kernel_compressed, self.kernel_log_step = compress_latent(\n",
+        "        other.kernel_latent, other.kernel_log_step, \"kernel\")\n",
+        "    self.bias_compressed, self.bias_log_step = compress_latent(\n",
+        "        other.bias_latent, other.bias_log_step, \"bias\")\n",
+        "    self.built = True\n",
+        "\n",
+        "  @property\n",
+        "  def kernel(self):\n",
+        "    rdft_shape = (self.input_channels, self.filters,\n",
+        "                  self.kernel_size, self.kernel_size // 2 + 1, 2)\n",
+        "    kernel_rdft = decompress_latent(\n",
+        "        self.kernel_compressed, rdft_shape, self.kernel_log_step)\n",
+        "    return from_rdft(kernel_rdft, self.kernel_size)\n",
+        "\n",
+        "  @property\n",
+        "  def bias(self):\n",
+        "    bias_shape = (self.filters,)\n",
+        "    return decompress_latent(\n",
+        "        self.bias_compressed, bias_shape, self.bias_log_step)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cJLCPoe3l8jG"
+      },
+      "source": [
+        "To turn the compressible model into a compressed one, we can conveniently use the `clone_model` function. `compress_layer` converts any compressible layer into a compressed one, and simply passes through any other types of layers (such as `Flatten`, etc.).\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WEHroUyhG56m"
+      },
+      "outputs": [],
+      "source": [
+        "def compress_layer(layer):\n",
+        "  if isinstance(layer, CompressibleDense):\n",
+        "    return CompressedDense.copy(layer)\n",
+        "  if isinstance(layer, CompressibleConv2D):\n",
+        "    return CompressedConv2D.copy(layer)\n",
+        "  return type(layer).from_config(layer.get_config())\n",
+        "\n",
+        "compressed_classifier = tf.keras.models.clone_model(\n",
+        "    compressible_classifier, clone_function=compress_layer)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b3wbN1XQmkDg"
+      },
+      "source": [
+        "Now, let's validate that the compressed model still performs as expected:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "R95kuURITpa9"
+      },
+      "outputs": [],
+      "source": [
+        "compressed_classifier.compile(metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n",
+        "_, compressed_accuracy = compressed_classifier.evaluate(validation_dataset.batch(128))\n",
+        "\n",
+        "print(f\"Accuracy of the compressible classifier: {penalized_accuracy:0.4f}\")\n",
+        "print(f\"Accuracy of the compressed classifier: {compressed_accuracy:0.4f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KtFhpXh6uaIY"
+      },
+      "source": [
+        "The classification accuracy of the compressed model is identical to the one achieved during training!\n",
+        "\n",
+        "In addition, the size of the compressed model weights is much smaller than the original model size:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Qp-ecfuYufbs"
+      },
+      "outputs": [],
+      "source": [
+        "def get_weight_size_in_bytes(weight):\n",
+        "  if weight.dtype == tf.string:\n",
+        "    return tf.reduce_sum(tf.strings.length(weight, unit=\"BYTE\"))\n",
+        "  else:\n",
+        "    return tf.size(weight) * weight.dtype.size\n",
+        "\n",
+        "original_size = sum(map(get_weight_size_in_bytes, classifier.weights))\n",
+        "compressed_size = sum(map(get_weight_size_in_bytes, compressed_classifier.weights))\n",
+        "\n",
+        "print(f\"Size of original model weights: {original_size} bytes\")\n",
+        "print(f\"Size of compressed model weights: {compressed_size} bytes\")\n",
+        "print(f\"Compression ratio: {(original_size/compressed_size):0.0f}x\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "K8A8v0df6TR2"
+      },
+      "source": [
+        "Storing the models on disk requires some overhead for storing the model architecture, function graphs, etc.\n",
+        "\n",
+        "Lossless compression methods such as ZIP are good at compressing this type of data, but not the weights themselves. That is why there is still a significant benefit of EPR when counting model size inclusive of that overhead, after also applying ZIP compression:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4hunDYxH1zqb"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import shutil\n",
+        "\n",
+        "def get_disk_size(model, path):\n",
+        "  model.save(path)\n",
+        "  zip_path = shutil.make_archive(path, \"zip\", path)\n",
+        "  return os.path.getsize(zip_path)\n",
+        "\n",
+        "original_zip_size = get_disk_size(classifier, \"/tmp/classifier\")\n",
+        "compressed_zip_size = get_disk_size(\n",
+        "    compressed_classifier, \"/tmp/compressed_classifier\")\n",
+        "\n",
+        "print(f\"Original on-disk size (ZIP compressed): {original_zip_size} bytes\")\n",
+        "print(f\"Compressed on-disk size (ZIP compressed): {compressed_zip_size} bytes\")\n",
+        "print(f\"Compression ratio: {(original_zip_size/compressed_zip_size):0.0f}x\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FSITvJrlAhZs"
+      },
+      "source": [
+        "## Regularization effect and size–accuracy trade-off\n",
+        "\n",
+        "Above, the $\\lambda$ hyperparameter was set to 2 (normalized by the number of parameters in the model). As we increase $\\lambda$, the model weights are more and more heavily penalized for compressibility.\n",
+        "\n",
+        "For low values, the penalty can act like a weight regularizer. It actually has a beneficial effect on the generalization performance of the classifier, and can lead to a slightly higher accuracy on the validation dataset:\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "4rhmKu98FdPJ"
+      },
+      "outputs": [],
+      "source": [
+        "#@title\n",
+        "\n",
+        "print(f\"Accuracy of the vanilla classifier: {classifier_accuracy:0.4f}\")\n",
+        "print(f\"Accuracy of the penalized classifier: {penalized_accuracy:0.4f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9UCfC4LQFdjL"
+      },
+      "source": [
+        "For higher values, we see a smaller and smaller model size, but also a gradually diminishing accuracy. To see this, let's train a few models and plot their size vs. accuracy:\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "diApPKHbAIqa"
+      },
+      "outputs": [],
+      "source": [
+        "def compress_and_evaluate_model(lmbda):\n",
+        "  print(f\"lambda={lmbda:0.0f}: training...\", flush=True)\n",
+        "  regularizer = PowerLawRegularizer(lmbda=lmbda/classifier.count_params())\n",
+        "  compressible_classifier = make_mnist_classifier(regularizer)\n",
+        "  train_model(\n",
+        "      compressible_classifier, training_dataset, validation_dataset, verbose=0)\n",
+        "  print(\"compressing...\", flush=True)\n",
+        "  compressed_classifier = tf.keras.models.clone_model(\n",
+        "      compressible_classifier, clone_function=compress_layer)\n",
+        "  compressed_size = sum(map(\n",
+        "      get_weight_size_in_bytes, compressed_classifier.weights))\n",
+        "  compressed_zip_size = float(get_disk_size(\n",
+        "      compressed_classifier, \"/tmp/compressed_classifier\"))\n",
+        "  print(\"evaluating...\", flush=True)\n",
+        "  compressed_classifier = tf.keras.models.load_model(\n",
+        "      \"/tmp/compressed_classifier\")\n",
+        "  compressed_classifier.compile(\n",
+        "      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n",
+        "  _, compressed_accuracy = compressed_classifier.evaluate(\n",
+        "      validation_dataset.batch(128), verbose=0)\n",
+        "  print()\n",
+        "  return compressed_size, compressed_zip_size, compressed_accuracy\n",
+        "\n",
+        "lambdas = (2., 5., 10., 20., 50.)\n",
+        "metrics = [compress_and_evaluate_model(l) for l in lambdas]\n",
+        "metrics = tf.convert_to_tensor(metrics, tf.float32)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "bhAi85KzGqTz"
+      },
+      "outputs": [],
+      "source": [
+        "#@title\n",
+        "\n",
+        "def plot_broken_xaxis(ax, compressed_sizes, original_size, original_accuracy):\n",
+        "  xticks = list(range(\n",
+        "      int(tf.math.floor(min(compressed_sizes) / 5) * 5),\n",
+        "      int(tf.math.ceil(max(compressed_sizes) / 5) * 5) + 1,\n",
+        "      5))\n",
+        "  xticks.append(xticks[-1] + 10)\n",
+        "  ax.set_xlim(xticks[0], xticks[-1] + 2)\n",
+        "  ax.set_xticks(xticks[1:])\n",
+        "  ax.set_xticklabels(xticks[1:-1] + [f\"{original_size:0.2f}\"])\n",
+        "  ax.plot(xticks[-1], original_accuracy, \"o\", label=\"float32\")\n",
+        "\n",
+        "sizes, zip_sizes, accuracies = tf.transpose(metrics)\n",
+        "sizes /= 1024\n",
+        "zip_sizes /= 1024\n",
+        "\n",
+        "fig, (axl, axr) = plt.subplots(1, 2, sharey=True, figsize=(10, 4))\n",
+        "axl.plot(sizes, accuracies, \"o-\", label=\"EPR compressed\")\n",
+        "axr.plot(zip_sizes, accuracies, \"o-\", label=\"EPR compressed\")\n",
+        "plot_broken_xaxis(axl, sizes, original_size/1024, classifier_accuracy)\n",
+        "plot_broken_xaxis(axr, zip_sizes, original_zip_size/1024, classifier_accuracy)\n",
+        "\n",
+        "axl.set_xlabel(\"size of model weights [kbytes]\")\n",
+        "axr.set_xlabel(\"ZIP compressed on-disk model size [kbytes]\")\n",
+        "axl.set_ylabel(\"accuracy\")\n",
+        "axl.legend(loc=\"lower right\")\n",
+        "axr.legend(loc=\"lower right\")\n",
+        "axl.grid()\n",
+        "axr.grid()\n",
+        "for i in range(len(lambdas)):\n",
+        "  axl.annotate(f\"$\\lambda = {lambdas[i]:0.0f}$\", (sizes[i], accuracies[i]),\n",
+        "               xytext=(10, -5), xycoords=\"data\", textcoords=\"offset points\")\n",
+        "  axr.annotate(f\"$\\lambda = {lambdas[i]:0.0f}$\", (zip_sizes[i], accuracies[i]),\n",
+        "               xytext=(10, -5), xycoords=\"data\", textcoords=\"offset points\")\n",
+        "plt.tight_layout()\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ajrHaFTAaLd2"
+      },
+      "source": [
+        "The plot should ideally show an elbow-shaped size–accuracy trade-off, but it is normal for accuracy metrics to be somewhat noisy. Depending on initialization, the curve can exhibit some kinks.\n",
+        "\n",
+        "Due to the regularization effect, the EPR compressed model is more accurate on the test set than the original model for small values of $\\lambda$. The EPR compressed model is also many times smaller, even if we compare the sizes after additional ZIP compression."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-RBhdXZTzoWw"
+      },
+      "source": [
+        "## Decompress the classifier\n",
+        "\n",
+        "`CompressedDense` and `CompressedConv2D` decompress their weights on every forward pass. This makes them ideal for memory-limited devices, but the decompression can be computationally expensive, especially for small batch sizes.\n",
+        "\n",
+        "To decompress the model once, and use it for further training or inference, we can convert it back into a model using regular or compressible layers. This can be useful in model deployment or federated learning scenarios.\n",
+        "\n",
+        "First, converting back into a plain model, we can do inference, and/or continue regular training without a compression penalty:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QBB2-X5XzvwB"
+      },
+      "outputs": [],
+      "source": [
+        "def decompress_layer(layer):\n",
+        "  if isinstance(layer, CompressedDense):\n",
+        "    return CustomDense.copy(layer)\n",
+        "  if isinstance(layer, CompressedConv2D):\n",
+        "    return CustomConv2D.copy(layer)\n",
+        "  return type(layer).from_config(layer.get_config())\n",
+        "\n",
+        "decompressed_classifier = tf.keras.models.clone_model(\n",
+        "    compressed_classifier, clone_function=decompress_layer)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ehE2ov8U0p0G"
+      },
+      "outputs": [],
+      "source": [
+        "decompressed_accuracy = train_model(\n",
+        "    decompressed_classifier, training_dataset, validation_dataset, epochs=1)\n",
+        "\n",
+        "print(f\"Accuracy of the compressed classifier: {compressed_accuracy:0.4f}\")\n",
+        "print(f\"Accuracy of the decompressed classifier after one more epoch of training: {decompressed_accuracy:0.4f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "jiSCvemQ04o8"
+      },
+      "source": [
+        "Note that the validation accuracy drops after training for an additional epoch, since the training is done without regularization.\n",
+        "\n",
+        "Alternatively, we can convert the model back into a \"compressible\" one, for inference and/or further training with a compression penalty:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "JDppVUdx1BvY"
+      },
+      "outputs": [],
+      "source": [
+        "def decompress_layer_with_penalty(layer):\n",
+        "  if isinstance(layer, CompressedDense):\n",
+        "    return CompressibleDense.copy(layer, regularizer=regularizer)\n",
+        "  if isinstance(layer, CompressedConv2D):\n",
+        "    return CompressibleConv2D.copy(layer, regularizer=regularizer)\n",
+        "  return type(layer).from_config(layer.get_config())\n",
+        "\n",
+        "decompressed_classifier = tf.keras.models.clone_model(\n",
+        "    compressed_classifier, clone_function=decompress_layer_with_penalty)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AJcnyOFW2IcK"
+      },
+      "outputs": [],
+      "source": [
+        "decompressed_accuracy = train_model(\n",
+        "    decompressed_classifier, training_dataset, validation_dataset, epochs=1)\n",
+        "\n",
+        "print(f\"Accuracy of the compressed classifier: {compressed_accuracy:0.4f}\")\n",
+        "print(f\"Accuracy of the decompressed classifier after one more epoch of training: {decompressed_accuracy:0.4f}\")\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Ciol315T_TwQ"
+      },
+      "source": [
+        "Here, the accuracy improves after training for an additional epoch."
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [
+        "Tce3stUlHN0L",
+        "xHxb-dlhMIzW"
+      ],
+      "name": "compression.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From 2e756845962c4ad735744ceb543a3b28b1729c5a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Johannes=20Ball=C3=A9?= 
Date: Mon, 15 Aug 2022 09:26:50 -0700
Subject: [PATCH 267/872] Shows how to sample from a compression model by
 feeding random bits.

PiperOrigin-RevId: 467687646
---
 .../generative/data_compression.ipynb         | 54 +++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
index f8a5fe3d201..b8edf946ae3 100644
--- a/site/en/tutorials/generative/data_compression.ipynb
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -825,6 +825,60 @@
         "\n",
         "This demonstrates that this model is agnostic to human perceptions of error, it just measures the absolute deviation in terms of pixel values. To achieve a better perceived image quality, we would need to replace the pixel loss with a perceptual loss."
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "v9cWHtH0LP_r"
+      },
+      "source": [
+        "## Use the decoder as a generative model.\n",
+        "\n",
+        "If we feed the decoder random bits, this will effectively sample from the distribution that the model learned to represent digits.\n",
+        "\n",
+        "First, re-instantiate the compressor/decompressor without a sanity check that would detect if the input string isn't completely decoded."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qnic8YsM0_ke"
+      },
+      "outputs": [],
+      "source": [
+        "compressor, decompressor = make_mnist_codec(trainer, decode_sanity_check=False)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "86uc9_Is1eeo"
+      },
+      "source": [
+        "Now, feed long enough random strings into the decompressor so that it can decode/sample digits from them."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "o4fP7BkqKCHY"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "strings = tf.constant([os.urandom(8) for _ in range(16)])\n",
+        "samples = decompressor(strings)\n",
+        "\n",
+        "fig, axes = plt.subplots(4, 4, sharex=True, sharey=True, figsize=(5, 5))\n",
+        "axes = axes.ravel()\n",
+        "for i in range(len(axes)):\n",
+        "  axes[i].imshow(tf.squeeze(samples[i]))\n",
+        "  axes[i].axis(\"off\")\n",
+        "plt.subplots_adjust(wspace=0, hspace=0, left=0, right=1, bottom=0, top=1)\n"
+      ]
     }
   ],
   "metadata": {

From 3715a3b51162ab5ef15689414a5134937e92b308 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Mon, 15 Aug 2022 15:57:25 -0700
Subject: [PATCH 268/872] Move nbcp.

PiperOrigin-RevId: 467782554
---
 tools/{ => tensorflow_docs/tools}/nbcp/__init__.py | 0
 tools/{ => tensorflow_docs/tools}/nbcp/__main__.py | 0
 2 files changed, 0 insertions(+), 0 deletions(-)
 rename tools/{ => tensorflow_docs/tools}/nbcp/__init__.py (100%)
 rename tools/{ => tensorflow_docs/tools}/nbcp/__main__.py (100%)

diff --git a/tools/nbcp/__init__.py b/tools/tensorflow_docs/tools/nbcp/__init__.py
similarity index 100%
rename from tools/nbcp/__init__.py
rename to tools/tensorflow_docs/tools/nbcp/__init__.py
diff --git a/tools/nbcp/__main__.py b/tools/tensorflow_docs/tools/nbcp/__main__.py
similarity index 100%
rename from tools/nbcp/__main__.py
rename to tools/tensorflow_docs/tools/nbcp/__main__.py

From 64a1a97afb999cefcfd4c464448272d0ac15b102 Mon Sep 17 00:00:00 2001
From: Mark McDonald 
Date: Mon, 15 Aug 2022 17:16:45 -0700
Subject: [PATCH 269/872] Add `nbformat` dependency to tensorflow docs package.

It's only required for type checking in `nbfmt`, but it's a runtime dep for `nbcp`, so install it as a dep.

PiperOrigin-RevId: 467798529
---
 setup.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/setup.py b/setup.py
index 3a726be637b..ea4a4c24c79 100644
--- a/setup.py
+++ b/setup.py
@@ -29,6 +29,7 @@
     'astor',
     'absl-py',
     'jinja2',
+    'nbformat',
     # TODO(b/182876485): Protobuf 3.20 results in linker errors on Windows
     # Protobuf 4.0 is binary incompatible with what C++ TF uses.
     # We need ~1 quarter to update properly.

From 1eadb62fe32913a6bb17a6bc5cd696d3af99a033 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Tue, 16 Aug 2022 14:01:24 -0700
Subject: [PATCH 270/872] added core quickstart tutorial

PiperOrigin-RevId: 468019110
---
 site/en/guide/_toc.yaml                  |   7 +
 site/en/guide/core/index.md              | 113 +++++
 site/en/guide/core/quickstart_core.ipynb | 581 +++++++++++++++++++++++
 3 files changed, 701 insertions(+)
 create mode 100644 site/en/guide/core/index.md
 create mode 100644 site/en/guide/core/quickstart_core.ipynb

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index 1f98745faee..58c5d810cd6 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -21,6 +21,13 @@ toc:
 - heading: "Keras"
 - include: /guide/keras/_toc.yaml
 
+- heading: "Build with Core"
+  status: new
+- title: "Overview"
+  path: /guide/core/index
+- title: "Quickstart for Core"
+  path: /guide/core/quickstart_core
+
 - heading: "TensorFlow in depth"
 - title: "Tensor slicing"
   path: /guide/tensor_slicing
diff --git a/site/en/guide/core/index.md b/site/en/guide/core/index.md
new file mode 100644
index 00000000000..cfe3b19ad27
--- /dev/null
+++ b/site/en/guide/core/index.md
@@ -0,0 +1,113 @@
+# TensorFlow Core APIs overview
+
+The TensorFlow Core APIs provide a set of low-level APIs for high-performance
+(distributed & accelerated) computation, primarily aimed at building machine
+learning (ML) models as well as authoring ML workflow tools and frameworks
+within the TensorFlow platform. These APIs provide a foundation for creating
+highly configurable models with fine-grain control and new frameworks from the
+ground up.
+
+The Core APIs can be used as an alternative to high-level machine learning APIs
+such as Keras. These high-level APIs are best suited for general machine
+learning needs. They offer a variety of modules that abstract away the
+complexities of ML while also offering functionalities for customization through
+subclassing. If you are looking for an overview of TensorFlow using Keras, see
+the Quickstarts and Keras sections in the
+[tutorials](https://www.tensorflow.org/tutorials)
+
+## Core API developer audience
+
+The TensorFlow Core low-level APIs are designed with the following ML Developers
+in mind:
+
+*   Researchers building complex models with high levels of configurability
+*   Developers interested in using TensorFlow as a high-performance scientific
+    computing platform
+*   Framework authors building tools on top of the TensorFlow platform
+*   High-level API users interested in:
+    *   Adding additional functionalities to their machine learning workflows
+        such as custom layers, losses, models, and optimizers
+    *   Learning more about the inner workings of their models
+
+## Core API applications
+
+The TensorFlow Core APIs provide access to low level functionality within the
+TensorFlow ecosystem. This API provides more flexibility and control for
+building ML models, applications, and tools, compared to high-level APIs, such
+as Keras.
+
+### Build models and workflows
+
+The Core APIs are most commonly used to build highly customizable and optimized
+machine learning models and workflows. Here are some of the ways that the
+TensorFlow Core APIs can improve your machine learning models and workflow
+development::
+
+TensorFlow
+
+*   Building non-traditional models or layers that do not fully fit the
+    structures supported by high-level APIs
+*   Building custom layers, losses, models, and optimizers within Keras
+*   Implementing new optimization techniques to expedite convergence during
+    training
+*   Creating custom metrics for performance evaluation
+*   Designing highly-configurable training loops with support for features like
+    batching, cross-validation, and distribution strategies
+
+### Build frameworks and tools
+
+The TensorFlow Core APIs can also serve as the building blocks for new
+high-level frameworks. Here are some examples of tools and frameworks that are
+created with the low-level APIs:
+TensorFlow
+
+*   [Keras](https://keras.io): deep learning for humans
+*   [TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization):
+    a suite of tools to optimize ML models for deployment and execution
+*   [TensorFlow Graphics](https://www.tensorflow.org/graphics): a library for
+    making useful graphics functions widely accessible
+
+### Build for scientific computing
+
+The TensorFlow Core APIs can also be applied outside the realm of machine
+learning. Here are a few general-purpose use cases of TensorFlow for scientific
+computing:
+TensorFlow
+
+*   Physics simulations for solid mechanics and
+    [fluid dynamics](https://arxiv.org/abs/2108.11076) problems
+*   Graphics rendering applications like
+    [ray tracing](https://github.com/BachiLi/redner)
+*   Solving
+    [constrained optimization problems](https://github.com/google-research/tensorflow_constrained_optimization/blob/master/README.md)
+
+## Core API components
+
+Here are some of the fundamental components that comprise TensorFlow Core’s low-
+level APIs. Note that this is not an all-encompassing list:
+
+TensorFlow
+
+*   Data structures : `tf.Tensor`, `tf.Variable`, `tf.TensorArray`
+*   Primitive APIs: `tf.shape`,
+    [slicing](https://www.tensorflow.org/guide/tensor_slicing), `tf.concat`,
+    `tf.bitwise`
+*   Numerical: `tf.math`, `tf.linalg`, `tf.random`
+*   Functional components: `tf.function`, `tf.GradientTape`
+*   Distribution: [DTensor](https://www.tensorflow.org/guide/dtensor_overview)
+*   Export: `tf.saved_model`
+
+## Next steps
+
+The *Build with Core* documentation provides tutorials of basic machine learning
+concepts from scratch. These tutorials in this section help you get comfortable
+with writing low-level code with Core APIs that you can then apply to more
+complex use cases of your own.
+
+Note: You should not use the Core APIs to simply re-implement high-level APIs,
+and it is possible to use high-level APIs, such as Keras, with the Core APIs.
+
+To get started using and learning more about the Core APIs, check out the
+[Quickstart for TensorFlow Core](https://www.tensorflow.org/guide/core/quickstart_core).
diff --git a/site/en/guide/core/quickstart_core.ipynb b/site/en/guide/core/quickstart_core.ipynb
new file mode 100644
index 00000000000..aad634aea4c
--- /dev/null
+++ b/site/en/guide/core/quickstart_core.ipynb
@@ -0,0 +1,581 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rX8mhOLljYeM"
+      },
+      "source": [
+        "##### Copyright 2022 The TensorFlow Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "BZSlp3DAjdYf"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3wF5wszaj97Y"
+      },
+      "source": [
+        "# Quickstart for the TensorFlow Core APIs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DUNzJc4jTj6G"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "04QgGZc9bF5D"
+      },
+      "source": [
+        "This short introduction uses the [TensorFlow Core low-level APIs](https://www.tensorflow.org/guide/core) to walk through the following stages of a machine learning example:\n",
+        "\n",
+        "1. Load a prebuilt dataset.\n",
+        "1. Build a multiple linear regression model that predicts fuel efficiency.\n",
+        "2. Train this multiple linear regression model.\n",
+        "3. Evaluate the performance of the model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hiH7AC-NTniF"
+      },
+      "source": [
+        "This tutorial is a [Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb) notebook. Python programs are run directly in the browser—a great way to learn and use TensorFlow. To follow this tutorial, run the notebook in Google Colab by clicking the button at the top of this page.\n",
+        "\n",
+        "1. In Colab, connect to a Python runtime: At the top-right of the menu bar, select *CONNECT*.\n",
+        "2. Run all the notebook code cells: Select *Runtime* > *Run all*."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nnrWf3PCEzXL"
+      },
+      "source": [
+        "## Set up TensorFlow\n",
+        "\n",
+        "Import TensorFlow and pandas into your program to get started:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0trJmd6DjqBZ"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "import pandas as pd\n",
+        "print(\"TensorFlow version:\", tf.__version__)\n",
+        "# Set a random seed for reproducible results \n",
+        "tf.random.set_seed(22)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7NAbSZiaoJ4z"
+      },
+      "source": [
+        "If you are following along in your own development environment, rather than [Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/core/quickstart_core.ipynb), see the [install guide](https://www.tensorflow.org/install) for setting up TensorFlow for development.\n",
+        "\n",
+        "Note: Make sure you have upgraded to the latest `pip` to install the TensorFlow 2 package if you are using your own development environment. See the [install guide](https://www.tensorflow.org/install) for details.\n",
+        "\n",
+        "## Load a dataset\n",
+        "\n",
+        "Load and prepare the [Auto MPG dataset](https://archive.ics.uci.edu/ml/datasets/auto+mpg) from the UCI Machine Learning Repository. This dataset uses a variety of quantitative and categorical features to predict the fuel efficiencies of automobiles in the late-1970s and early 1980s. Make sure to drop any missing values in the dataset and convert it to a tf.float32 tensor with the `tf.convert_to_tensor` and `tf.cast` functions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "HglhDsUfrJ98"
+      },
+      "outputs": [],
+      "source": [
+        "url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data'\n",
+        "column_names = ['MPG', 'Cylinders', 'Displacement', 'Horsepower', 'Weight',\n",
+        "                'Acceleration', 'Model Year', 'Origin']\n",
+        "\n",
+        "dataset = pd.read_csv(url, names=column_names, na_values='?', comment='\\t',\n",
+        "                          sep=' ', skipinitialspace=True)\n",
+        "\n",
+        "dataset = dataset.dropna()\n",
+        "dataset_tf = tf.convert_to_tensor(dataset, dtype=tf.float32)\n",
+        "dataset.tail()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0vgoDL3hYesB"
+      },
+      "source": [
+        "Next split the dataset into [training and testing groups](https://developers.google.com/machine-learning/crash-course/training-and-test-sets/splitting-data). Make sure to shuffle the dataset with `tf.random.shuffle` to avoid biased splits."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0mJU4kt6YiAp"
+      },
+      "outputs": [],
+      "source": [
+        "dataset_shuffled = tf.random.shuffle(dataset_tf, seed=22)\n",
+        "train_data, test_data = dataset_shuffled[100:], dataset_shuffled[:100]\n",
+        "x_train, y_train = train_data[:, 1:], train_data[:, 0]\n",
+        "x_test, y_test = test_data[:, 1:], test_data[:, 0]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Bscb2Vsbi3TE"
+      },
+      "source": [
+        "Perform basic [feature engineering](https://developers.google.com/machine-learning/crash-course/representation/feature-engineering) by one-hot-encoding the `\"Origin\"` feature. The `tf.one_hot` function is useful for transforming this categorical column into 3 separate binary columns. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_B8N9IV1i6IV"
+      },
+      "outputs": [],
+      "source": [
+        "def onehot_origin(x):\n",
+        "  origin = tf.cast(x[:, -1], tf.int32)\n",
+        "  # Use origin - 1 to account for 1-indexed feature\n",
+        "  origin_oh = tf.one_hot(origin - 1, 3)\n",
+        "  x_ohe = tf.concat([x[:, :-1], origin_oh], axis = 1)\n",
+        "  return x_ohe\n",
+        "\n",
+        "x_train_ohe, x_test_ohe = onehot_origin(x_train), onehot_origin(x_test)\n",
+        "x_train_ohe.numpy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qnoCDzzedite"
+      },
+      "source": [
+        "This example shows a multiple regression problem with predictors or features on vastly different scales. Therefore, it is beneficial to standardize the data so that each feature has zero mean and unit variance. Use the `tf.reduce_mean` and `tf.math.reduce_std` functions for standardization. The regression model's prediction can then be unstandardized to obtain its value in terms of the original units."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dJJFdvqydhyp"
+      },
+      "outputs": [],
+      "source": [
+        "class Normalize(tf.Module):\n",
+        "  def __init__(self, x):\n",
+        "    # Initialize the mean and standard deviation for normalization\n",
+        "    self.mean = tf.math.reduce_mean(x, axis=0)\n",
+        "    self.std = tf.math.reduce_std(x, axis=0)\n",
+        "\n",
+        "  def norm(self, x):\n",
+        "    # Normalize the input\n",
+        "    return (x - self.mean)/self.std\n",
+        "\n",
+        "  def unnorm(self, x):\n",
+        "    # Unnormalize the input\n",
+        "    return (x * self.std) + self.mean"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5BONV6fYYwZb"
+      },
+      "outputs": [],
+      "source": [
+        "norm_x = Normalize(x_train_ohe)\n",
+        "norm_y = Normalize(y_train)\n",
+        "x_train_norm, y_train_norm = norm_x.norm(x_train_ohe), norm_y.norm(y_train)\n",
+        "x_test_norm, y_test_norm = norm_x.norm(x_test_ohe), norm_y.norm(y_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BPZ68wASog_I"
+      },
+      "source": [
+        "## Build a machine learning model\n",
+        "\n",
+        "Build a linear regression model with the TensorFlow Core APIs. The equation for multiple linear regression is as follows:\n",
+        "\n",
+        "$${\\mathrm{Y}} = {\\mathrm{X}}w + b$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\underset{m\\times 1}{\\mathrm{Y}}$: target vector\n",
+        "* $\\underset{m\\times n}{\\mathrm{X}}$: feature matrix\n",
+        "* $\\underset{n\\times 1}w$: weight vector\n",
+        "* $b$: bias\n",
+        "\n",
+        "By using the `@tf.function` decorator, the corresponding Python code is traced to generate a callable TensorFlow graph. This approach is beneficial for saving and loading the model after training. It can also provide a performance boost for models with many layers and complex operations. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "h3IKyzTCDNGo"
+      },
+      "outputs": [],
+      "source": [
+        "class LinearRegression(tf.Module):\n",
+        "\n",
+        "  def __init__(self):\n",
+        "    self.built = False\n",
+        "\n",
+        "  @tf.function\n",
+        "  def __call__(self, x):\n",
+        "    # Initialize the model parameters on the first call\n",
+        "    if not self.built:\n",
+        "      # Randomly generate the weight vector and bias term\n",
+        "      rand_w = tf.random.uniform(shape=[x.shape[-1], 1])\n",
+        "      rand_b = tf.random.uniform(shape=[])\n",
+        "      self.w = tf.Variable(rand_w)\n",
+        "      self.b = tf.Variable(rand_b)\n",
+        "      self.built = True\n",
+        "    y = tf.add(tf.matmul(x, self.w), self.b)\n",
+        "    return tf.squeeze(y, axis=1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "l2hiez2eIUz8"
+      },
+      "source": [
+        "For each example, the model returns a prediction for the input automobile's MPG by computing the weighted sum of its features plus a bias term. This prediction can then be unstandardized to obtain its value in terms of the original units."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OeOrNdnkEEcR"
+      },
+      "outputs": [],
+      "source": [
+        "lin_reg = LinearRegression()\n",
+        "prediction = lin_reg(x_train_norm[:1])\n",
+        "prediction_unnorm = norm_y.unnorm(prediction)\n",
+        "prediction_unnorm.numpy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FIHANxNSvWr9"
+      },
+      "source": [
+        "## Defining a loss function\n",
+        "\n",
+        "Now define a loss function to evaluate the model's performance during the training process.\n",
+        "\n",
+        "Since regression problems deal with continuous outputs, the mean squared error (MSE) is an ideal choice for the loss function. The MSE is defined by the following equation:\n",
+        "\n",
+        "$$MSE = \\frac{1}{m}\\sum_{i=1}^{m}(\\hat{y}_i -y_i)^2$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\hat{y}$: vector of predictions\n",
+        "* $y$: vector of true targets\n",
+        "\n",
+        "The goal of this regression problem is to find the optimal weight vector, $w$, and bias, $b$, that minimizes the MSE loss function. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8tYNVUkmw35s"
+      },
+      "outputs": [],
+      "source": [
+        "def mse_loss(y_pred, y):\n",
+        "  return tf.reduce_mean(tf.square(y_pred - y))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ix4mEL65on-w"
+      },
+      "source": [
+        "## Train and evaluate your model\n",
+        "\n",
+        "Write a training loop to iteratively update your model's parameters by making use of the MSE loss function and its gradients with respect to the input parameters.\n",
+        "\n",
+        "This iterative method is referred to as gradient descent. At each iteration, the model's parameters are updated by taking a step in the opposite direction of their computed gradients. The size of this step is determined by the learning rate, which is a configurable hyperparameter. Recall that the gradient of a function indicates the direction of its steepest ascent; therefore, taking a step in the opposite direction indicates the direction of steepest descent, which ultimately helps to minimize the MSE loss function.\n",
+        "\n",
+        "Using mini-batches for training provides both memory efficiency and faster convergence. The `tf.data.Dataset` API has useful functions for batching and shuffling."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xS9hB17NY3sf"
+      },
+      "outputs": [],
+      "source": [
+        "batch_size = 64\n",
+        "dataset = tf.data.Dataset.from_tensor_slices((x_train_norm, y_train_norm))\n",
+        "dataset = dataset.shuffle(buffer_size=x_train.shape[0]).batch(batch_size)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "y7suUbJXVLqP"
+      },
+      "outputs": [],
+      "source": [
+        "# Set training parameters\n",
+        "epochs = 100\n",
+        "learning_rate = 0.01\n",
+        "train_losses, test_losses = [], []\n",
+        "\n",
+        "# Format training loop\n",
+        "for epoch in range(epochs):\n",
+        "  for x_batch, y_batch in dataset:\n",
+        "    batch_losses = []\n",
+        "    with tf.GradientTape() as tape:\n",
+        "      y_pred_batch = lin_reg(x_batch)\n",
+        "      batch_loss = mse_loss(y_pred_batch, y_batch)\n",
+        "    # Update parameters with respect to the gradient calculations\n",
+        "    grads = tape.gradient(batch_loss, lin_reg.variables)\n",
+        "    for g,v in zip(grads, lin_reg.variables):\n",
+        "      v.assign_sub(learning_rate * g)\n",
+        "    # Keep track of batch-level model performance \n",
+        "    batch_losses.append(batch_loss)\n",
+        "  # Keep track of epoch-level model performance\n",
+        "  train_loss = tf.reduce_mean(batch_losses)\n",
+        "  test_loss = mse_loss(lin_reg(x_test_norm), y_test_norm)\n",
+        "  train_losses.append(train_loss)\n",
+        "  test_losses.append(test_loss)\n",
+        "  if epoch % 10 == 0:\n",
+        "    print(f'Mean squared error for step {epoch}: {train_loss.numpy():0.3f}')\n",
+        "\n",
+        "# Output final losses\n",
+        "print(f\"\\nFinal train loss: {train_loss:0.3f}\")\n",
+        "print(f\"Final test loss: {test_loss:0.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4mDAAPFqVVgn"
+      },
+      "source": [
+        "Plot the changes in MSE loss over time. Calculating performance metrics on a designated [Validation-set](https://developers.google.com/machine-learning/glossary#validation-set) or [Test-set](https://developers.google.com/machine-learning/glossary#test-set) ensures the model does not overfit to the training dataset and can generalize well to unseen data."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "F7dTAzgHDUh7"
+      },
+      "outputs": [],
+      "source": [
+        "import matplotlib\n",
+        "from matplotlib import pyplot as plt\n",
+        "matplotlib.rcParams['figure.figsize'] = [9, 6]\n",
+        "\n",
+        "plt.plot(range(epochs), train_losses, label = \"Training loss\")\n",
+        "plt.plot(range(epochs), test_losses, label = \"Testing loss\")\n",
+        "plt.xlabel(\"Epoch\")\n",
+        "plt.ylabel(\"Mean squared error loss\")\n",
+        "plt.legend()\n",
+        "plt.title(\"MSE loss vs training iterations\");"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Aj8NrlzlJqDG"
+      },
+      "source": [
+        "It seems like the model does a good job of fitting the training data while also generalizing well to the unseen test data."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AUNIPubuPYDR"
+      },
+      "source": [
+        "## Saving your model\n",
+        "\n",
+        "Start by making an export module that takes in raw data and performs the following operations:\n",
+        "- Feature extraction \n",
+        "- Normalization \n",
+        "- Prediction\n",
+        "- Unnormalization"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "g-uOrGa9ZehG"
+      },
+      "outputs": [],
+      "source": [
+        "class ExportModule(tf.Module):\n",
+        "  def __init__(self, model, extract_features, norm_x, norm_y):\n",
+        "    # Initialize pre and postprocessing functions\n",
+        "    self.model = model\n",
+        "    self.extract_features = extract_features\n",
+        "    self.norm_x = norm_x\n",
+        "    self.norm_y = norm_y\n",
+        "\n",
+        "  @tf.function(input_signature=[tf.TensorSpec(shape=[None, None], dtype=tf.float32)]) \n",
+        "  def __call__(self, x):\n",
+        "    # Run the ExportModule for new data points\n",
+        "    x = self.extract_features(x)\n",
+        "    x = self.norm_x.norm(x)\n",
+        "    y = self.model(x)\n",
+        "    y = self.norm_y.unnorm(y)\n",
+        "    return y "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YPYYLQ8EZiU8"
+      },
+      "outputs": [],
+      "source": [
+        "lin_reg_export = ExportModule(model=lin_reg,\n",
+        "                              extract_features=onehot_origin,\n",
+        "                              norm_x=norm_x,\n",
+        "                              norm_y=norm_y)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6v8xi06XZWiC"
+      },
+      "source": [
+        "If you want to save the model at its current state, you can do so with the `tf.saved_model.save` function. To load a saved model and make predictions, use the `tf.saved_model.load` function."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K1IvMoHbptht"
+      },
+      "outputs": [],
+      "source": [
+        "import tempfile\n",
+        "import os\n",
+        "\n",
+        "models = tempfile.mkdtemp()\n",
+        "save_path = os.path.join(models, 'lin_reg_export')\n",
+        "tf.saved_model.save(lin_reg_export, save_path)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "rYb6DrEH0GMv"
+      },
+      "outputs": [],
+      "source": [
+        "lin_reg_loaded = tf.saved_model.load(save_path)\n",
+        "test_preds = lin_reg_loaded(x_test)\n",
+        "test_preds[:10].numpy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-47O6_GLdRuT"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "Congratulations! You have trained a machine learning model with a prebuilt dataset using the [TensorFlow Core APIs](https://www.tensorflow.org/guide/core). If you want learn more about loading and preparing data, see the tutorials on [image data loading](https://www.tensorflow.org/tutorials/load_data/images) or [CSV data loading](https://www.tensorflow.org/tutorials/load_data/csv).\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [
+        "rX8mhOLljYeM"
+      ],
+      "name": "quickstart_core.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From 9233d91ae135b495ab347b5c3c92c8d524273d33 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Tue, 16 Aug 2022 14:09:29 -0700
Subject: [PATCH 271/872] added core logistic regression tutorial

PiperOrigin-RevId: 468021272
---
 site/en/guide/_toc.yaml                       |   2 +
 .../guide/core/logistic_regression_core.ipynb | 872 ++++++++++++++++++
 2 files changed, 874 insertions(+)
 create mode 100644 site/en/guide/core/logistic_regression_core.ipynb

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index 58c5d810cd6..bb583ac1338 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -27,6 +27,8 @@ toc:
   path: /guide/core/index
 - title: "Quickstart for Core"
   path: /guide/core/quickstart_core
+- title: "Logistic regression"
+  path: /guide/core/logistic_regression_core
 
 - heading: "TensorFlow in depth"
 - title: "Tensor slicing"
diff --git a/site/en/guide/core/logistic_regression_core.ipynb b/site/en/guide/core/logistic_regression_core.ipynb
new file mode 100644
index 00000000000..23a82f10264
--- /dev/null
+++ b/site/en/guide/core/logistic_regression_core.ipynb
@@ -0,0 +1,872 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FhGuhbZ6M5tl"
+      },
+      "source": [
+        "##### Copyright 2022 The TensorFlow Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AwOEIRJC6Une"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EIdT9iu_Z4Rb"
+      },
+      "source": [
+        "# Logistic regression for binary classification with Core APIs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bBIlTPscrIT9"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DauaqJ7WhIhO"
+      },
+      "source": [
+        "## Introduction\n",
+        "\n",
+        "This notebook uses the [TensorFlow Core low-level APIs](https://www.tensorflow.org/guide/core) to build an end-to-end machine learning workflow from scratch. Visit the [Core APIs overview](https://www.tensorflow.org/guide/core) to learn more about TensorFlow Core and its intended use cases."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AHp3M9ZmrIxj"
+      },
+      "source": [
+        "[Logistic regression](https://developers.google.com/machine-learning/crash-course/logistic-regression/) is one of the most popular algorithms for binary classification. Given a set of examples with features, the goal of logistic regression is to output values between 0 and 1, which can be interpreted as the probabilities of each example belonging to a particular  class. "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nchsZfwEVtVs"
+      },
+      "source": [
+        "## Setup"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5lZoUK6AVTos"
+      },
+      "outputs": [],
+      "source": [
+        "# Use seaborn for pairplot.\n",
+        "!pip install -q seaborn"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1rRo8oNqZ-Rj"
+      },
+      "outputs": [],
+      "source": [
+        "import pandas as pd\n",
+        "import matplotlib\n",
+        "from matplotlib import pyplot as plt\n",
+        "import seaborn as sns\n",
+        "import tempfile\n",
+        "import os\n",
+        "# Preset Matplotlib figure sizes.\n",
+        "matplotlib.rcParams['figure.figsize'] = [9, 6]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9xQKvCJ85kCQ"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "print(tf.__version__)\n",
+        "# Set random seed for reproducible results \n",
+        "tf.random.set_seed(22)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "F_72b0LCNbjx"
+      },
+      "source": [
+        "## The Wisconsin Breast Cancer Dataset\n",
+        "\n",
+        "This tutorial uses the [Wisconsin Breast Cancer Dataset](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)), and demonstrates how to build a logistic regression model that can classify tumors as benign or malignant given features such as a tumor's radius, texture, and concavity.\n",
+        "The dataset is available from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/).\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gFh9ne3FZ-On"
+      },
+      "source": [
+        "### Get the data\n",
+        "First download and import the dataset using pandas."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CiX2FI4gZtTt"
+      },
+      "outputs": [],
+      "source": [
+        "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data'\n",
+        "\n",
+        "features = ['radius', 'texture', 'perimieter','area', 'smoothness', 'compactness', \n",
+        "            'concavity', 'concave_poinits','symmetry', 'fractal_dimension']   \n",
+        "column_names = ['id', 'diagnosis']\n",
+        "\n",
+        "for attr in ['mean', 'ste', 'largest']:\n",
+        "  for feature in features:\n",
+        "    column_names.append(feature + \"_\" + attr)\n",
+        "\n",
+        "dataset = pd.read_csv(url, names=column_names)\n",
+        "dataset.head()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "s4-Wn2jzVC1W"
+      },
+      "source": [
+        "### Split the data into training and test sets\n",
+        "\n",
+        "Now, split the dataset into a training set and a test set. Make sure to split the features from the target labels. The test set is used to evaluate your model's generalizability to unseen data."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "m2O60B-IVG9Q"
+      },
+      "outputs": [],
+      "source": [
+        "train_dataset = dataset.sample(frac=0.75)\n",
+        "test_dataset = dataset.drop(train_dataset.index)\n",
+        "# The `id` column can be dropped since each row is unique\n",
+        "x_train, y_train = train_dataset.iloc[:, 2:], train_dataset.iloc[:, 1]\n",
+        "x_test, y_test = test_dataset.iloc[:, 2:], test_dataset.iloc[:, 1]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3MWuJTKEDM-f"
+      },
+      "source": [
+        "### Data preprocessing\n",
+        "\n",
+        "This dataset contains the mean, standard error, and largest values for each of the 10 tumor measurements collected per example.  The `\"diagnosis\"` target column is categorical variable with `'M'` indicating a malignant tumor and `'B'` indicating a benign tumor diagnosis. This column needs to be converted into a numerical binary format for model training. The dataset should also be converted to a tensor with the `tf.convert_to_tensor` function after the preprocessing is complete."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "JEJHhN65a2VV"
+      },
+      "outputs": [],
+      "source": [
+        "y_train, y_test = y_train.map({'B': 0, 'M': 1}), y_test.map({'B': 0, 'M': 1})\n",
+        "x_train, y_train = tf.convert_to_tensor(x_train, dtype=tf.float32), tf.convert_to_tensor(y_train, dtype=tf.float32)\n",
+        "x_test, y_test = tf.convert_to_tensor(x_test, dtype=tf.float32), tf.convert_to_tensor(y_test, dtype=tf.float32)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "J4ubs136WLNp"
+      },
+      "source": [
+        "### Inspect the data\n",
+        "\n",
+        "Review the joint distribution a few pairs of mean-based features from the training set and observe how they relate to the target."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oRKO_x8gWKv-"
+      },
+      "outputs": [],
+      "source": [
+        "sns.pairplot(train_dataset.iloc[:, 1:6], hue = 'diagnosis', diag_kind='kde');"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5YOG5iKYKW_3"
+      },
+      "source": [
+        "This pairplot demonstrates that certain features such as radius, perimeter and area are highly correlated. This is expected since the tumor radius is directly involved in the computation of both perimeter and area. Additionally, note that malignant diagnoses seem to be more right-skewed for many of the features."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gavKO_6DWRMP"
+      },
+      "source": [
+        "Make sure to also check the overall statistics. Note how each feature covers a vastly different range of values."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "yi2FzC3T21jR"
+      },
+      "outputs": [],
+      "source": [
+        "train_dataset.describe().transpose()[:10]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "_8pDCIFjMla8"
+      },
+      "source": [
+        "### Standardize the data\n",
+        "\n",
+        "Given the inconsistent ranges, it is beneficial to standardize the data so that each feature has zero mean and unit variance. The target variable is binary; therefore, it does not require any scaling. Build a normalizer class to handle this standardization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FrzKNFNjLQDl"
+      },
+      "outputs": [],
+      "source": [
+        "class Normalize(tf.Module):\n",
+        "  def __init__(self, x):\n",
+        "    # Initialize the mean and standard deviation for normalization\n",
+        "    self.mean = tf.Variable(tf.math.reduce_mean(x, axis=0))\n",
+        "    self.std = tf.Variable(tf.math.reduce_std(x, axis=0))\n",
+        "\n",
+        "  def norm(self, x):\n",
+        "    # Normalize the input\n",
+        "    return (x - self.mean)/self.std\n",
+        "\n",
+        "  def unnorm(self, x):\n",
+        "    # Unnormalize the input\n",
+        "    return (x * self.std) + self.mean\n",
+        "\n",
+        "norm_x = Normalize(x_train)\n",
+        "x_train_norm, x_test_norm = norm_x.norm(x_train), norm_x.norm(x_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6o3CrycBXA2s"
+      },
+      "source": [
+        "## Logistic regression\n",
+        "\n",
+        "Before building a logistic regression model, it is crucial to understand the method's differences compared to traditional linear regression."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "lFby9n0tnHkw"
+      },
+      "source": [
+        "### Logistic regression fundamentals review\n",
+        "\n",
+        "Linear regression takes a linear combination of its inputs and outputs continuous values in the range, $(-∞, ∞)$. Recall that logistic regression is intended to output probabilities in the range, $(0, 1)$, for a binary classification problem. \n",
+        "\n",
+        "Logistic regression maps the continuous outputs of traditional linear regression, $(-∞, ∞)$, to probabilies, $(0, 1)$. This transformation is also symmetric so that flipping the sign of the output results in the inverse of the original probability. \n",
+        "Let $Y$ denote the probability of being in class 1 (malignant diagnosis). The desired mapping can be achieved by interpreting the linear regression output as the log odds ratio of being in class 1 as opposed to class 0: \n",
+        "\n",
+        "$$\\ln(\\frac{Y}{1-Y}) = wX + b$$\n",
+        "\n",
+        "By setting $wX + b = z$, this equation can then be solved for $Y$:\n",
+        "\n",
+        "$$Y = \\frac{e^{z}}{1 + e^{z}} = \\frac{1}{1 + e^{-z}}$$ \n",
+        "\n",
+        "The expression, $\\frac{1}{1 + e^{-z}}$, is known as the sigmoid function, $\\sigma(z)$. Hence, the equation for logistic regression can be written as $Y = \\sigma(wX + b)$.\n",
+        "\n",
+        "This dataset deals with a high-dimensional feature matrix; therefore, the above equation must be rewritten in matrix vector form:\n",
+        "\n",
+        "$${\\mathrm{Y}} = \\sigma({\\mathrm{X}}w + b)$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\underset{m\\times 1}{\\mathrm{Y}}$: target vector\n",
+        "* $\\underset{m\\times n}{\\mathrm{X}}$: feature matrix\n",
+        "* $\\underset{n\\times 1}w$: weight vector\n",
+        "* $b$: bias\n",
+        "* $\\sigma$: sigmoid function applied to each element of the output vector\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "n3N3pMC0uesW"
+      },
+      "source": [
+        "Start by visualizing the sigmoid function. It transforms the outputs of traditional linear regression, $(-∞, ∞)$, to fall between 0 and 1. The sigmoid function is available in `tf.math.sigmoid`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ThHaV_RmucZl"
+      },
+      "outputs": [],
+      "source": [
+        "x = tf.linspace(-10, 10, 500)\n",
+        "x = tf.cast(x, tf.float32)\n",
+        "f = lambda x : (1/20)*x + 0.6\n",
+        "plt.plot(x, tf.math.sigmoid(x))\n",
+        "plt.ylim((-0.1,1.1))\n",
+        "plt.title(\"Sigmoid Function\");"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "VMXEhrZuKECV"
+      },
+      "source": [
+        "### The log loss function\n",
+        "\n",
+        "The log loss or binary cross-entropy loss is the ideal loss function for a binary classification problem with logistic regression. For each example, the log loss quantifies the similarity between a predicted probability and the example's true value. It is determined by the following equation:\n",
+        "\n",
+        "$$L = -\\frac{1}{m}\\sum_{i=1}^{m}y_i\\cdot\\log(\\hat{y}_i) + (1- y_i)\\cdot\\log(1 - \\hat{y}_i)$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\hat{y}$: vector of predicted probabilities\n",
+        "* $y$: vector of true targets\n",
+        "\n",
+        "The `tf.nn.sigmoid_cross_entropy_with_logits` function can be used to compute the log loss. This function automatically applies the sigmoid activation to the regression output."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "JVBInnSqS36W"
+      },
+      "outputs": [],
+      "source": [
+        "def log_loss(y_pred, y):\n",
+        "  # Compute the log loss\n",
+        "  ce = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=y_pred)\n",
+        "  return tf.reduce_mean(ce)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Q_mutLj0KNUb"
+      },
+      "source": [
+        "###The gradient descent update rule\n",
+        "\n",
+        "The TensorFlow Core APIs support automatic differentiation with `tf.GradientTape`. If you are curious about the math behind the logistic regression gradient updates, here is the derivation:\n",
+        "\n",
+        "In the above equation for the log loss, recall that each $\\hat{y}_i$ can be rewritten in terms of the inputs as $\\sigma({\\mathrm{X_i}}w + b)$. \n",
+        "\n",
+        "The goal is to find a $w^*$ and $b^*$ that minimize the log loss: \n",
+        "\n",
+        "$$L = -\\frac{1}{m}\\sum_{i=1}^{m}y_i\\cdot\\log(\\sigma({\\mathrm{X_i}}w + b)) + (1- y_i)\\cdot\\log(1 - \\sigma({\\mathrm{X_i}}w + b))$$ \n",
+        "\n",
+        "By taking the gradient $L$ with respect to $w$, you get the following: \n",
+        "\n",
+        "$$\\frac{\\partial L}{\\partial w} = \\frac{1}{m}(\\sigma({\\mathrm{X}}w + b) - y)X$$\n",
+        "\n",
+        "By taking the gradient $L$ with respect to $b$, you get the following: \n",
+        "\n",
+        "$$\\frac{\\partial L}{\\partial b} = \\frac{1}{m}\\sum_{i=1}^{m}\\sigma({\\mathrm{X_i}}w + b) - y_i$$"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "uTCndUecKZho"
+      },
+      "source": [
+        "Now, build the logistic regression model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c0sXM7qLlKfZ"
+      },
+      "outputs": [],
+      "source": [
+        "class LogisticRegression(tf.Module):\n",
+        "\n",
+        "  def __init__(self):\n",
+        "    self.built = False\n",
+        "    \n",
+        "  def __call__(self, x, train=True):\n",
+        "    # Initialize the model parameters on the first call\n",
+        "    if not self.built:\n",
+        "      # Randomly generate the weight vector and bias term\n",
+        "      rand_w = tf.random.uniform(shape=[x.shape[-1], 1], seed=22)\n",
+        "      rand_b = tf.random.uniform(shape=[], seed=22)\n",
+        "      self.w = tf.Variable(rand_w)\n",
+        "      self.b = tf.Variable(rand_b)\n",
+        "      self.built = True\n",
+        "    # Compute model output\n",
+        "    z = tf.add(tf.matmul(x, self.w), self.b)\n",
+        "    z = tf.squeeze(z, axis=1)\n",
+        "    if train:\n",
+        "      return z\n",
+        "    return tf.sigmoid(z)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "eObQu9fDnXGL"
+      },
+      "source": [
+        "As a sanity check, make sure the untrained model outputs values in the range of (0,1) for a small subset of the training data."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5bIovC0Z4QHJ"
+      },
+      "outputs": [],
+      "source": [
+        "log_reg = LogisticRegression()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QJ2ievISyf0p"
+      },
+      "outputs": [],
+      "source": [
+        "y_pred = log_reg(x_train_norm[:5], train=False)\n",
+        "y_pred.numpy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "PribnwDHUksC"
+      },
+      "source": [
+        "Next, write an accuracy function to calculate the proportion of correct classifications during training. In order to retrieve the classifications from the predicted probabilities, set a threshold for which all probabilities higher than the threshold belong to class 1. This is a configurable hyperparameter that can be set to 0.5 as a default."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ssnVcKg7oMe6"
+      },
+      "outputs": [],
+      "source": [
+        "def predict_class(y_pred, thresh=0.5):\n",
+        "  # Return a tensor with : 1. if y_pred_i > 0.5 and 0. ow/\n",
+        "  return tf.cast(y_pred > thresh, tf.float32)\n",
+        "\n",
+        "def accuracy(y_pred, y):\n",
+        "  # Return the proportion of matches between y_pred and y\n",
+        "  y_pred = tf.math.sigmoid(y_pred)\n",
+        "  y_pred_class = predict_class(y_pred)\n",
+        "  check_equal = tf.cast(y_pred_class == y,tf.float32)\n",
+        "  acc_val = tf.reduce_mean(check_equal)\n",
+        "  return acc_val"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "J_0KHQ25_2dF"
+      },
+      "source": [
+        "### Model training\n",
+        "\n",
+        "Now write a training loop for the logistic regression model. The loop utilizes the log loss function and its gradients with respect to the input in order to iteratively update the model's parameters. Using mini-batches for training provides both memory efficiency and faster convergence. The `tf.data.Dataset` API has useful functions for batching and shuffling.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "vJD7-4U0etqa"
+      },
+      "outputs": [],
+      "source": [
+        "batch_size = 64\n",
+        "dataset = tf.data.Dataset.from_tensor_slices((x_train_norm, y_train))\n",
+        "dataset = dataset.shuffle(buffer_size=x_train.shape[0]).batch(batch_size)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jNC3D1DGsGgK"
+      },
+      "outputs": [],
+      "source": [
+        "# Set training parameters\n",
+        "epochs = 200\n",
+        "learning_rate = 0.01\n",
+        "train_losses, test_losses = [], []\n",
+        "train_accs, test_accs = [], []\n",
+        "\n",
+        "# Format training loop\n",
+        "for epoch in range(epochs):\n",
+        "  batch_losses, batch_accs = [], []\n",
+        "  for x_batch, y_batch in dataset:\n",
+        "    with tf.GradientTape() as tape:\n",
+        "      y_pred_batch = log_reg(x_batch)\n",
+        "      batch_loss = log_loss(y_pred_batch, y_batch)\n",
+        "    batch_acc = accuracy(y_pred_batch, y_batch)\n",
+        "    # Update parameters with respect to the gradient calculations\n",
+        "    grads = tape.gradient(batch_loss, log_reg.variables)\n",
+        "    for g,v in zip(grads, log_reg.variables):\n",
+        "      v.assign_sub(learning_rate * g)\n",
+        "    # Keep track of batch-level model performance \n",
+        "    batch_losses.append(batch_loss)\n",
+        "    batch_accs.append(batch_acc)\n",
+        "  # Keep track of epoch-level model performance\n",
+        "  train_loss, train_acc = tf.reduce_mean(batch_losses), tf.reduce_mean(batch_accs)\n",
+        "  y_pred_test = log_reg(x_test_norm)\n",
+        "  test_loss, test_acc = log_loss(y_pred_test, y_test), accuracy(y_pred_test, y_test)\n",
+        "  train_losses.append(train_loss)\n",
+        "  train_accs.append(train_acc)\n",
+        "  test_losses.append(test_loss)\n",
+        "  test_accs.append(test_acc)\n",
+        "  if epoch % 20 == 0:\n",
+        "    print(f\"Epoch: {epoch}, Training log loss: {train_loss:.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "NoLiAg7fYft7"
+      },
+      "source": [
+        "### Performance evaluation\n",
+        "\n",
+        "Observe the changes in your model's loss and accuracy over time. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mv3oCQPvWhr0"
+      },
+      "outputs": [],
+      "source": [
+        "plt.plot(range(epochs), train_losses, label = \"Training loss\")\n",
+        "plt.plot(range(epochs), test_losses, label = \"Testing loss\")\n",
+        "plt.xlabel(\"Epoch\")\n",
+        "plt.ylabel(\"Log loss\")\n",
+        "plt.legend()\n",
+        "plt.title(\"Log loss vs training iterations\");"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "D2HDVGLPODIE"
+      },
+      "outputs": [],
+      "source": [
+        "plt.plot(range(epochs), train_accs, label = \"Training accuracy\")\n",
+        "plt.plot(range(epochs), test_accs, label = \"Testing accuracy\")\n",
+        "plt.xlabel(\"Epoch\")\n",
+        "plt.ylabel(\"Accuracy (%)\")\n",
+        "plt.legend()\n",
+        "plt.title(\"Accuracy vs training iterations\");"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jonKhUzuPyfa"
+      },
+      "outputs": [],
+      "source": [
+        "print(f\"Final training log loss: {train_losses[-1]:.3f}\")\n",
+        "print(f\"Final Testing Log Loss: {test_losses[-1]:.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "d3DF4qyrPyke"
+      },
+      "outputs": [],
+      "source": [
+        "print(f\"Final training accuracy: {train_accs[-1]:.3f}\")\n",
+        "print(f\"Final testing accuracy: {test_accs[-1]:.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yrj1TbOJasjA"
+      },
+      "source": [
+        "The model does a great job of classifying tumors in the training dataset and also generalizes well to unseen data. To go one step further, you can explore error rates that give more insight beyond the overall accuracy score. The two most popular error rates for binary classification problems are the false positive rate (FPR) and the false negative rate (FNR). \n",
+        "\n",
+        "For this problem, the FPR is the proportion of malignant tumor predictions amongst tumors that are actually benign. Conversely, the FNR is the proportion of benign tumor predictions among tumors that are actually malignant. These metrics can be visualized with a confusion matrix. Scikit-learn has some useful tools for visualizing confusion matrices."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OJO7YkA8ZDMU"
+      },
+      "outputs": [],
+      "source": [
+        "import sklearn.metrics as sk_metrics\n",
+        "\n",
+        "def show_confusion_matrix(y, y_classes, typ):\n",
+        "  # Compute confusion matrix and normalize\n",
+        "  plt.figure(figsize=(10,10))\n",
+        "  confusion = sk_metrics.confusion_matrix(y.numpy(), y_classes.numpy())\n",
+        "  confusion_normalized = confusion / confusion.sum(axis=1)\n",
+        "  axis_labels = range(2)\n",
+        "  ax = sns.heatmap(\n",
+        "      confusion_normalized, xticklabels=axis_labels, yticklabels=axis_labels,\n",
+        "      cmap='Blues', annot=True, fmt='.4f', square=True)\n",
+        "  plt.title(f\"Confusion matrix: {typ}\")\n",
+        "  plt.ylabel(\"True label\")\n",
+        "  plt.xlabel(\"Predicted label\")\n",
+        "\n",
+        "y_pred_train, y_pred_test = log_reg(x_train_norm, train=False), log_reg(x_test_norm, train=False)\n",
+        "train_classes, test_classes = predict_class(y_pred_train), predict_class(y_pred_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OQ5DFcleiDFm"
+      },
+      "outputs": [],
+      "source": [
+        "show_confusion_matrix(y_train, train_classes, 'train')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gtfcsAp_iCNR"
+      },
+      "outputs": [],
+      "source": [
+        "show_confusion_matrix(y_test, test_classes, 'test')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DlivxaDmTnGq"
+      },
+      "source": [
+        "Observe the error rate measurements and interpret their significance in the context of this example. In many medical testing studies such as cancer detection, having a high false positive rate to ensure a low false negative rate is perfectly acceptable and in fact encouraged since the risk of missing a malignant tumor diagnosis (false negative) is a lot worse than misclassifying a benign tumor as malignant (false positive).\n",
+        "\n",
+        "In order to control for the FPR and FNR, try changing the threshold hyperparameter before classifying the probability predictions. A lower threshold increases the model's overall chances of making a malignant tumor classification. This inevitably increases the number of false positives and the FPR but it also helps to decrease the number of false negatives and the FNR."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7ADEN2rb4Nhj"
+      },
+      "source": [
+        "## Saving your model\n",
+        "\n",
+        "Start by making an export module that takes in raw data and performs the following operations:\n",
+        "- Normalization \n",
+        "- Probability prediction\n",
+        "- Class prediction\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6KPRHCzg4ZxH"
+      },
+      "outputs": [],
+      "source": [
+        "class ExportModule(tf.Module):\n",
+        "  def __init__(self, model, norm_x, class_pred):\n",
+        "    # Initialize pre and postprocessing functions\n",
+        "    self.model = model\n",
+        "    self.norm_x = norm_x\n",
+        "    self.class_pred = class_pred\n",
+        "\n",
+        "  @tf.function(input_signature=[tf.TensorSpec(shape=[None, None], dtype=tf.float32)]) \n",
+        "  def __call__(self, x):\n",
+        "    # Run the ExportModule for new data points\n",
+        "    x = self.norm_x.norm(x)\n",
+        "    y = self.model(x, train=False)\n",
+        "    y = self.class_pred(y)\n",
+        "    return y "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2YzRclo5-yjO"
+      },
+      "outputs": [],
+      "source": [
+        "log_reg_export = ExportModule(model=log_reg,\n",
+        "                              norm_x=norm_x,\n",
+        "                              class_pred=predict_class)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gtofGIBN_qFd"
+      },
+      "source": [
+        "If you want to save the model at its current state, you can do so with the `tf.saved_model.save` function. To load a saved model and make predictions, use the `tf.saved_model.load` function."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "a4Qum1Ts_pmF"
+      },
+      "outputs": [],
+      "source": [
+        "models = tempfile.mkdtemp()\n",
+        "save_path = os.path.join(models, 'log_reg_export')\n",
+        "tf.saved_model.save(log_reg_export, save_path)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3KPILr1i_M_c"
+      },
+      "outputs": [],
+      "source": [
+        "log_reg_loaded = tf.saved_model.load(save_path)\n",
+        "test_preds = log_reg_loaded(x_test)\n",
+        "test_preds[:10].numpy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vgGQuV-yqYZH"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "This notebook introduced a few techniques to handle a logistic regression problem. Here are a few more tips that may help:\n",
+        "\n",
+        "- The [TensorFlow Core APIs](https://www.tensorflow.org/guide/core) can be used to build machine learning workflows with high levels of configurability\n",
+        "- Analyzing error rates is a great way to gain more insight about a classification model's performance beyond its overall accuracy score. For more information on classification error rates, visit the following [crash course](https://developers.google.com/machine-learning/crash-course/classification/true-false-positive-negative).\n",
+        "- Overfitting is another common problem for logistic regression models, though it wasn't a problem for this tutorial. Visit the [Overfit and underfit](overfit_and_underfit.ipynb) tutorial for more help with this.\n",
+        "\n",
+        "For more examples of using the TensorFlow Core APIs, check out the [guide](https://www.tensorflow.org/guide/core). If you want learn more about loading and preparing data, see the tutorials on [image data loading](https://www.tensorflow.org/tutorials/load_data/images) or [CSV data loading](https://www.tensorflow.org/tutorials/load_data/csv)."
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [
+        "FhGuhbZ6M5tl"
+      ],
+      "name": "logistic_regression_core.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From 859ccf4e853edbf09eb78b3044fb4e5ac9846824 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Tue, 16 Aug 2022 14:21:09 -0700
Subject: [PATCH 272/872] added core multilayer perceptrons tutorial

PiperOrigin-RevId: 468024089
---
 site/en/guide/_toc.yaml           |    2 +
 site/en/guide/core/mlp_core.ipynb | 1002 +++++++++++++++++++++++++++++
 2 files changed, 1004 insertions(+)
 create mode 100644 site/en/guide/core/mlp_core.ipynb

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index bb583ac1338..fc22705f95b 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -29,6 +29,8 @@ toc:
   path: /guide/core/quickstart_core
 - title: "Logistic regression"
   path: /guide/core/logistic_regression_core
+- title: "Multilayer perceptrons"
+  path: /guide/core/mlp_core
 
 - heading: "TensorFlow in depth"
 - title: "Tensor slicing"
diff --git a/site/en/guide/core/mlp_core.ipynb b/site/en/guide/core/mlp_core.ipynb
new file mode 100644
index 00000000000..dcd2997a723
--- /dev/null
+++ b/site/en/guide/core/mlp_core.ipynb
@@ -0,0 +1,1002 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FhGuhbZ6M5tl"
+      },
+      "source": [
+        "##### Copyright 2022 The TensorFlow Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AwOEIRJC6Une"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EIdT9iu_Z4Rb"
+      },
+      "source": [
+        "# Multilayer perceptrons for digit recognition with Core APIs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bBIlTPscrIT9"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SjAxxRpBzVYg"
+      },
+      "source": [
+        "## Introduction\n",
+        "\n",
+        "This notebook uses the [TensorFlow Core low-level APIs](https://www.tensorflow.org/guide/core) to build an end-to-end machine learning workflow from scratch. Visit the [Core APIs overview](https://www.tensorflow.org/guide/core) to learn more about TensorFlow Core and its intended use cases."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AHp3M9ZmrIxj"
+      },
+      "source": [
+        "The Multilayer Perceptron (MLP) is a special type of feedforward neural network used to approach [multiclass classification](https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/video-lecture) problems. \n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "GHVMVIFHSzl1"
+      },
+      "source": [
+        "## Multilayer perceptron (MLP) overview\n",
+        "\n",
+        "Before building an MLP, it is crucial to understand the concepts of perceptrons, layers, and activation functions.\n",
+        "\n",
+        "Multilayer Perceptrons are made up of functional units called perceptrons. The equation of a  perceptron is as follows:\n",
+        "\n",
+        "$$Z = \\vec{w}⋅\\mathrm{X} + b$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $Z$: perceptron output\n",
+        "* $\\mathrm{X}$: feature matrix\n",
+        "* $\\vec{w}$: weight vector\n",
+        "* $b$: bias\n",
+        "\n",
+        "When these perceptrons are stacked, they form structures called dense layers which can then be connected to build a neural network. A dense layer's equation is similar to that of a perceptron's but uses a weight matrix and a bias vector instead: \n",
+        "\n",
+        "$$Y = \\mathrm{W}⋅\\mathrm{X} + \\vec{b}$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $Z$: dense layer output\n",
+        "* $\\mathrm{X}$: feature matrix\n",
+        "* $\\mathrm{W}$: weight matrix\n",
+        "* $\\vec{b}$: bias vector\n",
+        "\n",
+        "\n",
+        "In an MLP, multiple dense layers are connected in such a way that the outputs of one layer are fully connected to the inputs of the next layer. Adding non-linear activation functions to the outputs of dense layers can help the MLP classifier learn complex decision boundaries and generalize well to unseen data."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nchsZfwEVtVs"
+      },
+      "source": [
+        "## Setup"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mSfgqmwBagw_"
+      },
+      "outputs": [],
+      "source": [
+        "# Use seaborn for countplot.\n",
+        "!pip install -q seaborn"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1rRo8oNqZ-Rj"
+      },
+      "outputs": [],
+      "source": [
+        "import pandas as pd\n",
+        "import matplotlib\n",
+        "from matplotlib import pyplot as plt\n",
+        "import seaborn as sns\n",
+        "import tempfile\n",
+        "import os\n",
+        "# Preset Matplotlib figure sizes.\n",
+        "matplotlib.rcParams['figure.figsize'] = [9, 6]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9xQKvCJ85kCQ"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "import tensorflow_datasets as tfds\n",
+        "print(tf.__version__)\n",
+        "# Set random seed for reproducible results \n",
+        "tf.random.set_seed(22)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "F_72b0LCNbjx"
+      },
+      "source": [
+        "## The MNIST Dataset\n",
+        "\n",
+        "This tutorial uses the [MNIST Dataset](http://yann.lecun.com/exdb/mnist), and demonstrates how to build an MLP model that can classify handwritten digits. The dataset is available from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/mnist).\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gFh9ne3FZ-On"
+      },
+      "source": [
+        "### Get the data\n",
+        "\n",
+        "Split the MNIST dataset into training, validation, and testing sets. The validation set can be used to gauge the model's generalizability during training so that the test set can serve as a final unbiased estimator for the model's performance."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Uiuh0B098_3p"
+      },
+      "outputs": [],
+      "source": [
+        "train_data, val_data, test_data = tfds.load(\"mnist\", \n",
+        "                                            split=['train[10000:]', 'train[0:10000]', 'test'],\n",
+        "                                            as_supervised=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "U3kn85nzHhQN"
+      },
+      "source": [
+        "### Visualizing the data\n",
+        "\n",
+        "The MNIST dataset consists of handwritten digits and their corresponding true labels. Visualize a couple of examples below. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7b3uzLGIHlNc"
+      },
+      "outputs": [],
+      "source": [
+        "train_viz = train_data.map(lambda x, y: (tf.reshape(x, shape=[28, 28]), y))\n",
+        "for i, (x, y) in enumerate(train_viz):\n",
+        "  plt.subplot(3,3,1+i)\n",
+        "  plt.axis('off')\n",
+        "  plt.imshow(x, cmap='gray')\n",
+        "  plt.title(f\"True Label: {y}\")\n",
+        "  plt.subplots_adjust(hspace=.5)\n",
+        "  if i == 8:\n",
+        "    break\n",
+        "plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9ljx31IWJCwx"
+      },
+      "source": [
+        "Also review the distribution of digits in the training data to verify that each class is well represented in the dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YLnpnaLIJDh7"
+      },
+      "outputs": [],
+      "source": [
+        "x_train, y_train = [data for data in train_data.batch(len(train_data))][0]\n",
+        "sns.countplot(y_train.numpy());\n",
+        "plt.xlabel('Digits')\n",
+        "plt.title(\"MNIST Digit Distribution\");"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Z-JYDIjx-gg5"
+      },
+      "source": [
+        "### Pre-Processing the data\n",
+        "\n",
+        "First, reshape the feature matrices to be 2-dimensional by flattening the images. Next, rescale the data so that the pixel values of [0,255] fit into the range of [0,1]. This step ensures that the input pixels have similar distributions and helps with training convergence."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2PBHcHtR-_Jn"
+      },
+      "outputs": [],
+      "source": [
+        "def preprocess(x, y):\n",
+        "  # Reshaping the data\n",
+        "  x = tf.reshape(x, shape=[-1, 784])\n",
+        "  # Remove dimension of size 1\n",
+        "  x = tf.squeeze(x, axis=0)\n",
+        "  # Rescaling the data\n",
+        "  x = x/255\n",
+        "  return x, y\n",
+        "\n",
+        "train_data, val_data = train_data.map(preprocess), val_data.map(preprocess)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6o3CrycBXA2s"
+      },
+      "source": [
+        "## Build the MLP \n",
+        "\n",
+        "Start by visualizing the [ReLU](https://developers.google.com/machine-learning/glossary#ReLU) and [Softmax](https://developers.google.com/machine-learning/glossary#softmax) activation functions. Both functions are available in `tf.nn.relu` and `tf.nn.softmax` respectively. The ReLU is a non-linear activation function that outputs the input if it is positive and 0 otherwise: \n",
+        "\n",
+        "$$\\text{ReLU}(X) = max(0, X)$$"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hYunzt3UyT9G"
+      },
+      "outputs": [],
+      "source": [
+        "x = tf.linspace(-2, 2, 201)\n",
+        "x = tf.cast(x, tf.float32)\n",
+        "plt.plot(x, tf.nn.relu(x));\n",
+        "plt.xlabel('x')\n",
+        "plt.ylabel('ReLU(x)')\n",
+        "plt.title('ReLU activation function');"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fuGrM9jMwsRM"
+      },
+      "source": [
+        "The softmax activation function is a normalized exponential function that converts $m$ real numbers into a probability distribution with $m$ outcomes/classes. This is useful for predicting class probabilities from a neural network's output:\n",
+        "\n",
+        "$$\\text{Softmax}(X) = \\frac{e^{X}}{\\sum_{i=1}^{m}e^{X_i}}$$"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fVM8pvhWwuwI"
+      },
+      "outputs": [],
+      "source": [
+        "x = tf.linspace(-4, 4, 201)\n",
+        "x = tf.cast(x, tf.float32)\n",
+        "plt.plot(x, tf.nn.softmax(x, axis=0));\n",
+        "plt.xlabel('x')\n",
+        "plt.ylabel('Softmax(x)')\n",
+        "plt.title('Softmax activation function');"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "OHW6Yvg2yS6H"
+      },
+      "source": [
+        "### The dense layer\n",
+        "\n",
+        "Create a class for the dense layer. By definition, the outputs of one layer are fully connected to the inputs of the next layer in an MLP. Therefore, the input dimension for a dense layer can be inferred based on the output dimension of its previous layer and does not need to be specified upfront during its initialization. The weights should also be initialized properly to prevent activation outputs from becoming too large or small. One of the most popular weight initialization methods is the Xavier scheme, where each element of the weight matrix is sampled in the following manner:\n",
+        "\n",
+        "$$W_{ij} \\sim \\text{Uniform}(-\\frac{\\sqrt{6}}{\\sqrt{n + m}},\\frac{\\sqrt{6}}{\\sqrt{n + m}})$$\n",
+        "\n",
+        "The bias vector can be initialized to zeros."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "re1SSFyBdMrS"
+      },
+      "outputs": [],
+      "source": [
+        "def xavier_init(shape):\n",
+        "  # Computes the xavier initialization values for a weight matrix\n",
+        "  in_dim, out_dim = shape\n",
+        "  xavier_lim = tf.sqrt(6.)/tf.sqrt(tf.cast(in_dim + out_dim, tf.float32))\n",
+        "  weight_vals = tf.random.uniform(shape=(in_dim, out_dim), \n",
+        "                                  minval=-xavier_lim, maxval=xavier_lim, seed=22)\n",
+        "  return weight_vals"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "otDFX4u6e6ml"
+      },
+      "source": [
+        "The Xavier initialization method can also be implemented with `tf.keras.initializers.GlorotUniform`"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IM0yJos25FG5"
+      },
+      "outputs": [],
+      "source": [
+        "class DenseLayer(tf.Module):\n",
+        "\n",
+        "  def __init__(self, out_dim, weight_init=xavier_init, activation=tf.identity):\n",
+        "    # Initialize the dimensions and activation functions\n",
+        "    self.out_dim = out_dim\n",
+        "    self.weight_init = weight_init\n",
+        "    self.activation = activation\n",
+        "    self.built = False\n",
+        "\n",
+        "  def __call__(self, x):\n",
+        "    if not self.built:\n",
+        "      # Infer the input dimension based on first call\n",
+        "      self.in_dim = x.shape[1]\n",
+        "      # Initialize the weights and biases using Xavier scheme\n",
+        "      self.w = tf.Variable(xavier_init(shape=(self.in_dim, self.out_dim)))\n",
+        "      self.b = tf.Variable(tf.zeros(shape=(self.out_dim,)))\n",
+        "      self.built = True\n",
+        "    # Compute the forward pass\n",
+        "    z = tf.add(tf.matmul(x, self.w), self.b)\n",
+        "    return self.activation(z)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "X-7MzpjgyHg6"
+      },
+      "source": [
+        "Next, build a class for the MLP model that executes layers sequentially.\n",
+        "Remember that the model variables are only available after the first sequence of dense layer calls due to dimension inference."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6XisRWiCyHAb"
+      },
+      "outputs": [],
+      "source": [
+        "class MLP(tf.Module):\n",
+        "\n",
+        "  def __init__(self, layers):\n",
+        "    self.layers = layers\n",
+        "   \n",
+        "  @tf.function\n",
+        "  def __call__(self, x, preds=False): \n",
+        "    # Execute the model's layers sequentially\n",
+        "    for layer in self.layers:\n",
+        "      x = layer(x)\n",
+        "    return x"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "luXKup-43nd7"
+      },
+      "source": [
+        "### The MLP architecture\n",
+        "\n",
+        "Build an MLP model with the following architecture:\n",
+        "\n",
+        "Input Size = Image Size = 784
\n",
+        "Hidden Layer 1 Size = 700
\n",
+        "Hidden Layer 2 Size = 500
\n",
+        "Output Size = Number of Classes = 10\n",
+        "\n",
+        "Forward Pass: ReLU(784 x 700) x ReLU(700 x 500) x Softmax(500 x 10)\n",
+        "\n",
+        "The softmax activation function does not need to be applied by the MLP. It is computed separately in the loss and prediction functions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VmlACuki3oPi"
+      },
+      "outputs": [],
+      "source": [
+        "mlp_model = MLP([\n",
+        "    DenseLayer(out_dim=700, activation=tf.nn.relu),\n",
+        "    DenseLayer(out_dim=500, activation=tf.nn.relu),\n",
+        "    DenseLayer(out_dim=10)])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tyBATDoRmDkg"
+      },
+      "source": [
+        "### Defining a loss function\n",
+        "\n",
+        "The cross-entropy loss function is a great choice for multiclass classification problems since it measures the negative-log-likelihood of the data according to the model's probability predictions. The higher the probability assigned to the true class, the lower the loss. The equation for the cross-entropy loss is as follows:\n",
+        "\n",
+        "$$L = -\\frac{1}{n}\\sum_{i=1}^{n}\\sum_{i=j}^{n} {y_j}^{[i]}⋅\\log(\\hat{{y_j}}^{[i]})$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\underset{n\\times m}{\\hat{y}}$: predicted class distributions\n",
+        "* $\\underset{n\\times m}{y}$: one hot encoded matrix of true classes\n",
+        "\n",
+        "The `tf.nn.sparse_softmax_cross_entropy_with_logits` function can be used to compute the cross-entropy loss. This function does not require the model's last layer to apply the softmax activation function nor does it require the class labels to be one hot encoded"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "rskOYA7FVCwg"
+      },
+      "outputs": [],
+      "source": [
+        "def cross_entropy_loss(y_pred, y):\n",
+        "  # Compute cross entropy loss with a sparse operation\n",
+        "  sparse_ce = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=y_pred)\n",
+        "  return tf.reduce_mean(sparse_ce)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BvWxED1km8jh"
+      },
+      "source": [
+        "Write a basic accuracy function that calculates the proportion of correct classifications during training. In order to generate class predictions from softmax outputs, return the index that corresponds to the largest class probability. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jPJMWx2UgiBm"
+      },
+      "outputs": [],
+      "source": [
+        "def accuracy(y_pred, y):\n",
+        "  # Compute accuracy after extracting class predictions\n",
+        "  class_preds = tf.argmax(tf.nn.softmax(y_pred), axis=1)\n",
+        "  is_equal = tf.equal(y, class_preds)\n",
+        "  return tf.reduce_mean(tf.cast(is_equal, tf.float32))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JSiNRhTOnKZr"
+      },
+      "source": [
+        "### Model training\n",
+        "\n",
+        "Using an optimizer can result in significantly faster convergence compared to standard gradient descent. The Adam optimizer is implemented below. Visit the [Optimizers](https://www.tensorflow.org/guide/core/optimizers_core) guide to learn more about designing custom optimizers with TensorFlow Core."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "iGIBDk3cAv6a"
+      },
+      "outputs": [],
+      "source": [
+        "class Adam:\n",
+        "\n",
+        "    def __init__(self, learning_rate=1e-3, beta_1=0.9, beta_2=0.999, ep=1e-7):\n",
+        "      # Initialize optimizer parameters and variable slots\n",
+        "      self.beta_1 = beta_1\n",
+        "      self.beta_2 = beta_2\n",
+        "      self.learning_rate = learning_rate\n",
+        "      self.ep = ep\n",
+        "      self.t = 1.\n",
+        "      self.v_dvar, self.s_dvar = [], []\n",
+        "      self.built = False\n",
+        "\n",
+        "    def reset(self):\n",
+        "      # Reset variables after each epoch\n",
+        "      for v in self.v_dvar:\n",
+        "        v.assign(tf.zeros(shape=v.shape))\n",
+        "      for s in self.s_dvar:\n",
+        "        s.assign(tf.zeros(shape=s.shape))\n",
+        "      self.t = 1.\n",
+        "      return\n",
+        "      \n",
+        "    def apply_gradients(self, grads, vars):\n",
+        "      # Initialize variables on the first call\n",
+        "      if not self.built:\n",
+        "        for var in vars:\n",
+        "          v = tf.Variable(tf.zeros(shape=var.shape))\n",
+        "          s = tf.Variable(tf.zeros(shape=var.shape))\n",
+        "          self.v_dvar.append(v)\n",
+        "          self.s_dvar.append(s)\n",
+        "        self.built = True\n",
+        "      # Update the model variables given their gradients\n",
+        "      for i, (d_var, var) in enumerate(zip(grads, vars)):\n",
+        "        self.v_dvar[i].assign(self.beta_1*self.v_dvar[i] + (1-self.beta_1)*d_var)\n",
+        "        self.s_dvar[i].assign(self.beta_2*self.s_dvar[i] + (1-self.beta_2)*tf.square(d_var))\n",
+        "        v_dvar_bc = self.v_dvar[i]/(1-(self.beta_1**self.t))\n",
+        "        s_dvar_bc = self.s_dvar[i]/(1-(self.beta_2**self.t))\n",
+        "        var.assign_sub(self.learning_rate*(v_dvar_bc/(tf.sqrt(s_dvar_bc) + self.ep)))\n",
+        "      self.t += 1.\n",
+        "      return "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "osEK3rqpYfKd"
+      },
+      "source": [
+        "Now, write a custom training loop that updates the MLP parameters with mini-batch gradient descent. Using mini-batches for training provides both memory efficiency and faster convergence."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CJLeY2ao1aw6"
+      },
+      "outputs": [],
+      "source": [
+        "def train_step(x_batch, y_batch, loss, acc, model, optimizer):\n",
+        "  # Update the model state given a batch of data\n",
+        "  with tf.GradientTape() as tape:\n",
+        "    y_pred = model(x_batch)\n",
+        "    batch_loss = loss(y_pred, y_batch)\n",
+        "  batch_acc = acc(y_pred, y_batch)\n",
+        "  grads = tape.gradient(batch_loss, model.variables)\n",
+        "  optimizer.apply_gradients(grads, model.variables)\n",
+        "  return batch_loss, batch_acc"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oC85kuZgmh3q"
+      },
+      "outputs": [],
+      "source": [
+        "def train_model(mlp, train_data, val_data, loss, acc, optimizer, epochs):\n",
+        "  # Initialize data structures\n",
+        "  train_losses, val_losses = [], []\n",
+        "  train_accs, val_accs = [], []\n",
+        "  x_val, y_val = [data for data in val_data][0]\n",
+        "\n",
+        "  # Format training loop\n",
+        "  for epoch in range(epochs):\n",
+        "    optimizer.reset()\n",
+        "    batch_losses, batch_accs = [], []\n",
+        "    for x_batch, y_batch in train_data:\n",
+        "      # Compute gradients and update the model's parameters\n",
+        "      batch_loss, batch_acc = train_step(x_batch, y_batch, loss, acc, mlp, optimizer)\n",
+        "      # Keep track of batch-level model performance\n",
+        "      batch_losses.append(batch_loss)\n",
+        "      batch_accs.append(batch_acc)\n",
+        "    # Keep track of epoch-level model performance\n",
+        "    train_loss, train_acc = tf.reduce_mean(batch_losses), tf.reduce_mean(batch_accs)\n",
+        "    val_pred = mlp(x_val)\n",
+        "    val_loss, val_acc = loss(val_pred, y_val), acc(val_pred, y_val)\n",
+        "    train_losses.append(train_loss)\n",
+        "    train_accs.append(train_acc)\n",
+        "    val_losses.append(val_loss)\n",
+        "    val_accs.append(val_acc)\n",
+        "    print(f\"Epoch: {epoch}\")\n",
+        "    print(f\"Training loss: {train_loss:.3f}, Training accuracy: {train_acc:.3f}\")\n",
+        "    print(f\"Validation loss: {val_loss:.3f}, Validation accuracy: {val_acc:.3f}\")\n",
+        "  return train_losses, train_accs, val_losses, val_accs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FvbfXlN5lwwB"
+      },
+      "source": [
+        "Train the MLP model for 10 epochs with batch size of 128. Hardware accelerators like GPUs or TPUs can also help speed up training time. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Ijx0jkVmO47S"
+      },
+      "outputs": [],
+      "source": [
+        "batch_size = 128\n",
+        "train_data = train_data.shuffle(len(train_data)).batch(batch_size)\n",
+        "val_data = val_data.batch(len(val_data))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zPlT8QfxptYl"
+      },
+      "outputs": [],
+      "source": [
+        "train_losses, train_accs, val_losses, val_accs = train_model(mlp_model, train_data, val_data, \n",
+        "                                                             loss=cross_entropy_loss, acc=accuracy,\n",
+        "                                                             optimizer=Adam(), epochs=10)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "j_RVmt43G12R"
+      },
+      "source": [
+        "### Performance evaluation\n",
+        "\n",
+        "Start by writing a plotting function to visualize the model's loss and accuracy during training. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VXTCYVtNDjAM"
+      },
+      "outputs": [],
+      "source": [
+        "def plot_metrics(train_metric, val_metric, metric_type):\n",
+        "  # Visualize metrics vs training Epochs\n",
+        "  plt.figure()\n",
+        "  plt.plot(range(len(train_metric)), train_metric, label = f\"Training {metric_type}\")\n",
+        "  plt.plot(range(len(val_metric)), val_metric, label = f\"Validation {metric_type}\")\n",
+        "  plt.xlabel(\"Epochs\")\n",
+        "  plt.ylabel(metric_type)\n",
+        "  plt.legend()\n",
+        "  plt.title(f\"{metric_type} vs Training epochs\");"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DC-qIvZbHo0G"
+      },
+      "outputs": [],
+      "source": [
+        "plot_metrics(train_losses, val_losses, \"cross entropy loss\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "P-w2xk2PIDve"
+      },
+      "outputs": [],
+      "source": [
+        "plot_metrics(train_accs, val_accs, \"accuracy\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tbrJJaFrD_XR"
+      },
+      "source": [
+        "## Saving your model\n",
+        "\n",
+        "Start by making an export module that takes in raw data and performs the following operations:\n",
+        "- Data preprocessing \n",
+        "- Probability prediction\n",
+        "- Class prediction"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1sszfWuJJZoo"
+      },
+      "outputs": [],
+      "source": [
+        "class ExportModule(tf.Module):\n",
+        "  def __init__(self, model, preprocess, class_pred):\n",
+        "    # Initialize pre and postprocessing functions\n",
+        "    self.model = model\n",
+        "    self.preprocess = preprocess\n",
+        "    self.class_pred = class_pred\n",
+        "\n",
+        "  @tf.function(input_signature=[tf.TensorSpec(shape=[None, None, None, None], dtype=tf.uint8)]) \n",
+        "  def __call__(self, x):\n",
+        "    # Run the ExportModule for new data points\n",
+        "    x = self.preprocess(x)\n",
+        "    y = self.model(x)\n",
+        "    y = self.class_pred(y)\n",
+        "    return y "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "p8x6gjTDVi5d"
+      },
+      "outputs": [],
+      "source": [
+        "def preprocess_test(x):\n",
+        "  # The export module takes in unprocessed and unlabeled data\n",
+        "  x = tf.reshape(x, shape=[-1, 784])\n",
+        "  x = x/255\n",
+        "  return x\n",
+        "\n",
+        "def class_pred_test(y):\n",
+        "  # Generate class predictions from MLP output\n",
+        "  return tf.argmax(tf.nn.softmax(y), axis=1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vu9H5STrJzdo"
+      },
+      "source": [
+        "This export module can now be saved with the `tf.saved_model.save` function. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fN9pPBQTKTe3"
+      },
+      "outputs": [],
+      "source": [
+        "mlp_model_export = ExportModule(model=mlp_model,\n",
+        "                                preprocess=preprocess_test,\n",
+        "                                class_pred=class_pred_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "idS7rQKbKwRS"
+      },
+      "outputs": [],
+      "source": [
+        "models = tempfile.mkdtemp()\n",
+        "save_path = os.path.join(models, 'mlp_model_export')\n",
+        "tf.saved_model.save(mlp_model_export, save_path)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "_zZxO8iqBGZ-"
+      },
+      "source": [
+        "\n",
+        "Load the saved model with `tf.saved_model.load` and examine its performance on the unseen test data."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "W5cwBTUqxldW"
+      },
+      "outputs": [],
+      "source": [
+        "mlp_loaded = tf.saved_model.load(save_path)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bmv0u6j_b5OC"
+      },
+      "outputs": [],
+      "source": [
+        "def accuracy_score(y_pred, y):\n",
+        "  # Generic accuracy function\n",
+        "  is_equal = tf.equal(y_pred, y)\n",
+        "  return tf.reduce_mean(tf.cast(is_equal, tf.float32))\n",
+        "\n",
+        "x_test, y_test = [data for data in test_data.batch(len(test_data))][0]\n",
+        "test_classes = mlp_loaded(x_test)\n",
+        "test_acc = accuracy_score(test_classes, y_test)\n",
+        "print(f\"Test Accuracy: {test_acc:.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "j5t9vgv_ciQ_"
+      },
+      "source": [
+        "The model does a great job of classifying handwritten digits in the training dataset and also generalizes well to unseen data. Now, examine the model's class-wise accuracy to ensure good performance for each digit. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "UD8YiC1Vfeyp"
+      },
+      "outputs": [],
+      "source": [
+        "print(\"Accuracy breakdown by digit:\")\n",
+        "print(\"---------------------------\")\n",
+        "label_accs = {}\n",
+        "for label in range(10):\n",
+        "  label_ind = (y_test == label)\n",
+        "  # extract predictions for specific true label\n",
+        "  pred_label = test_classes[label_ind]\n",
+        "  label_filled = tf.cast(tf.fill(pred_label.shape[0], label), tf.int64)\n",
+        "  # compute class-wise accuracy\n",
+        "  label_accs[accuracy_score(pred_label, label_filled).numpy()] = label\n",
+        "for key in sorted(label_accs):\n",
+        "  print(f\"Digit {label_accs[key]}: {key:.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rcykuJFhdGb0"
+      },
+      "source": [
+        "It looks like the model struggles with some digits a little more than others which is quite common in many multiclass classification problems. As a final exercise, plot a confusion matrix of the model's predictions and its corresponding true labels to gather more class-level insights. Sklearn and seaborn have functions for generating and visualizing confusion matrices. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "JqCaqPwwh1tN"
+      },
+      "outputs": [],
+      "source": [
+        "import sklearn.metrics as sk_metrics\n",
+        "\n",
+        "def show_confusion_matrix(test_labels, test_classes):\n",
+        "  # Compute confusion matrix and normalize\n",
+        "  plt.figure(figsize=(10,10))\n",
+        "  confusion = sk_metrics.confusion_matrix(test_labels.numpy(), \n",
+        "                                          test_classes.numpy())\n",
+        "  confusion_normalized = confusion / confusion.sum(axis=1)\n",
+        "  axis_labels = range(10)\n",
+        "  ax = sns.heatmap(\n",
+        "      confusion_normalized, xticklabels=axis_labels, yticklabels=axis_labels,\n",
+        "      cmap='Blues', annot=True, fmt='.4f', square=True)\n",
+        "  plt.title(\"Confusion matrix\")\n",
+        "  plt.ylabel(\"True label\")\n",
+        "  plt.xlabel(\"Predicted label\")\n",
+        "\n",
+        "show_confusion_matrix(y_test, test_classes)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JT-WA7GVda6d"
+      },
+      "source": [
+        "Class-level insights can help identify reasons for misclassifications and improve model performance in future training cycles."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "VFLfEH4ManbW"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "This notebook introduced a few techniques to handle a multiclass classification problem with an [MLP](https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax). Here are a few more tips that may help:\n",
+        "\n",
+        "- The [TensorFlow Core APIs](https://www.tensorflow.org/guide/core) can be used to build machine learning workflows with high levels of configurability\n",
+        "- Initialization schemes can help prevent model parameters from vanishing or exploding during training.\n",
+        "- Overfitting is another common problem for neural networks, though it wasn't a problem for this tutorial. Visit the [Overfit and underfit](overfit_and_underfit.ipynb) tutorial for more help with this.\n",
+        "\n",
+        "For more examples of using the TensorFlow Core APIs, check out the [guide](https://www.tensorflow.org/guide/core). If you want learn more about loading and preparing data, see the tutorials on [image data loading](https://www.tensorflow.org/tutorials/load_data/images) or [CSV data loading](https://www.tensorflow.org/tutorials/load_data/csv)."
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "collapsed_sections": [
+        "FhGuhbZ6M5tl"
+      ],
+      "name": "mlp_core.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "gpuClass": "standard",
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From 5f5641f084771d582622afe40523420e6b524b92 Mon Sep 17 00:00:00 2001
From: "A. Unique TensorFlower" 
Date: Tue, 16 Aug 2022 14:32:24 -0700
Subject: [PATCH 273/872] added core image compression tutorial

PiperOrigin-RevId: 468026822
---
 site/en/guide/_toc.yaml              |   2 +
 site/en/guide/core/matrix_core.ipynb | 731 +++++++++++++++++++++++++++
 2 files changed, 733 insertions(+)
 create mode 100644 site/en/guide/core/matrix_core.ipynb

diff --git a/site/en/guide/_toc.yaml b/site/en/guide/_toc.yaml
index fc22705f95b..c48689f554b 100644
--- a/site/en/guide/_toc.yaml
+++ b/site/en/guide/_toc.yaml
@@ -31,6 +31,8 @@ toc:
   path: /guide/core/logistic_regression_core
 - title: "Multilayer perceptrons"
   path: /guide/core/mlp_core
+- title: "Matrix approximation"
+  path: /guide/core/matrix_core
 
 - heading: "TensorFlow in depth"
 - title: "Tensor slicing"
diff --git a/site/en/guide/core/matrix_core.ipynb b/site/en/guide/core/matrix_core.ipynb
new file mode 100644
index 00000000000..491f373cf89
--- /dev/null
+++ b/site/en/guide/core/matrix_core.ipynb
@@ -0,0 +1,731 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FhGuhbZ6M5tl"
+      },
+      "source": [
+        "##### Copyright 2022 The TensorFlow Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AwOEIRJC6Une"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EIdT9iu_Z4Rb"
+      },
+      "source": [
+        "# Matrix approximation with Core APIs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bBIlTPscrIT9"
+      },
+      "source": [
+        "\n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "  \n",
+        "
\n",
+        "    View on TensorFlow.org\n",
+        "  \n",
+        "    Run in Google Colab\n",
+        "  \n",
+        "    View source on GitHub\n",
+        "  \n",
+        "    Download notebook\n",
+        "  
"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qGw8TF2vtzru"
+      },
+      "source": [
+        "## Introduction \n",
+        "\n",
+        "This notebook uses the [TensorFlow Core low-level APIs](https://www.tensorflow.org/guide/core) to showcase TensorFlow's capabilities as a high-performance scientific computing platform. Visit the [Core APIs overview](https://www.tensorflow.org/guide/core) to learn more about TensorFlow Core and its intended use cases.\n",
+        "\n",
+        "This tutorial explores the technique of [singular value decomposition](https://developers.google.com/machine-learning/recommendation/collaborative/matrix) (SVD) and its applications for low-rank approximation problems. The SVD is used to factorize real or complex matrices and has a variety of use cases in data science such as image compression. The images for this tutorial come from Google Brain's [Imagen](https://imagen.research.google/) project. "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5_FdwaovEkCC"
+      },
+      "source": [
+        ">![svd_intro](http://tensorflow.org/images/core/svd_intro.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nchsZfwEVtVs"
+      },
+      "source": [
+        "## Setup"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1rRo8oNqZ-Rj"
+      },
+      "outputs": [],
+      "source": [
+        "import matplotlib\n",
+        "from matplotlib.image import imread\n",
+        "from matplotlib import pyplot as plt\n",
+        "import requests\n",
+        "# Preset Matplotlib figure sizes.\n",
+        "matplotlib.rcParams['figure.figsize'] = [16, 9]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9xQKvCJ85kCQ"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "print(tf.__version__)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "so_ewq3gAoEI"
+      },
+      "source": [
+        "## SVD fundamentals\n",
+        "\n",
+        "The singular value decomposition of a matrix, ${\\mathrm{A}}$, is determined by the following factorization:\n",
+        "\n",
+        "$${\\mathrm{A}} = {\\mathrm{U}} \\Sigma {\\mathrm{V}}^T$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\underset{m \\times n}{\\mathrm{A}}$: input matrix where $m \\geq n$\n",
+        "* $\\underset{m \\times n}{\\mathrm{U}}$: orthogonal matrix, ${\\mathrm{U}}^T{\\mathrm{U}} = {\\mathrm{I}}$, with each column, $u_i$, denoting a left singular vector of ${\\mathrm{A}}$\n",
+        "* $\\underset{n \\times n}{\\Sigma}$: diagonal matrix with each diagonal entry, $\\sigma_i$, denoting a singular value of ${\\mathrm{A}}$\n",
+        "* $\\underset{n \\times n}{{\\mathrm{V}}^T}$: orthogonal matrix, ${\\mathrm{V}}^T{\\mathrm{V}} = {\\mathrm{I}}$, with each row, $v_i$, denoting a right singular vector of ${\\mathrm{A}}$\n",
+        "\n",
+        "When $m < n$, ${\\mathrm{U}}$ and $\\Sigma$ both have dimension $(m \\times m)$, and ${\\mathrm{V}}^T$ has dimension $(m \\times n)$."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "enGGGXCQKNv8"
+      },
+      "source": [
+        ">![svd_full](http://tensorflow.org/images/core/svd_full.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "NlP-cBdSKLtc"
+      },
+      "source": [
+        "TensorFlow's linear algebra package has a function, `tf.linalg.svd`, which can be used to compute the singular value decomposition of one or more matrices. Start by defining a simple matrix and computing its SVD factorization.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "C3QAcgyoeIpv"
+      },
+      "outputs": [],
+      "source": [
+        "A = tf.random.uniform(shape=[40,30])\n",
+        "# Compute the SVD factorization\n",
+        "s, U, V = tf.linalg.svd(A)\n",
+        "# Define Sigma and V Transpose\n",
+        "S = tf.linalg.diag(s)\n",
+        "V_T = tf.transpose(V)\n",
+        "# Reconstruct the original matrix\n",
+        "A_svd = U@S@V_T\n",
+        "# Visualize \n",
+        "plt.bar(range(len(s)), s);\n",
+        "plt.xlabel(\"Singular value rank\")\n",
+        "plt.ylabel(\"Singular value\")\n",
+        "plt.title(\"Bar graph of singular values\");"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6H_C9WhFACm4"
+      },
+      "source": [
+        "The `tf.einsum` function can be used to directly compute the matrix reconstruction from the outputs of `tf.linalg.svd`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TPE6QeMtADUn"
+      },
+      "outputs": [],
+      "source": [
+        "A_svd = tf.einsum('s,us,vs -> uv',s,U,V)\n",
+        "print('\\nReconstructed Matrix, A_svd', A_svd)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "x1m6JIsM9DLP"
+      },
+      "source": [
+        "## Low rank approximation with the SVD\n",
+        "\n",
+        "The rank of a  matrix, ${\\mathrm{A}}$, is determined by the dimension of the vector space spanned by its columns. \n",
+        "The SVD can be used to approximate a matrix with a lower rank, which ultimately decreases the dimensionality of data required to store the information represented by the matrix.\n",
+        "\n",
+        "The rank-r approximation of ${\\mathrm{A}}$ in terms of the SVD is defined by the formula:\n",
+        "\n",
+        "$${\\mathrm{A_r}} = {\\mathrm{U_r}} \\Sigma_r {\\mathrm{V_r}}^T$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $\\underset{m \\times r}{\\mathrm{U_r}}$: matrix consisting of the first $r$ columns of ${\\mathrm{U}}$\n",
+        "* $\\underset{r \\times r}{\\Sigma_r}$:  diagonal matrix consisting of the first  $r$ singular values in $\\Sigma$\n",
+        "* $\\underset{r \\times n}{\\mathrm{V_r}}^T$: matrix consisting of the first $r$ rows of ${\\mathrm{V}}^T$"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nJWMJu36QyUV"
+      },
+      "source": [
+        ">![svd_approx](http://tensorflow.org/images/core/svd_approx.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TkiVUxeaQybq"
+      },
+      "source": [
+        "Start by writing a function to compute the rank-r approximation of a given matrix. This low-rank approximation procedure is used for image compression; therefore, it is also helpful to compute the physical data sizes for each approximation. For simplicity, assume that data size for an rank-r approximated matrix is equal to the total number of elements required to compute the approximation. Next, write a function to visualize the original matrix, $\\mathrm{A}$ its rank-r approximation, $\\mathrm{A}_r$ and the error matrix, $|\\mathrm{A} - \\mathrm{A}_r|$."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2oY3pMPagJrO"
+      },
+      "outputs": [],
+      "source": [
+        "def rank_r_approx(s, U, V, r, verbose=False):\n",
+        "  # Compute the matrices necessary for a rank-r approximation\n",
+        "  s_r, U_r, V_r = s[..., :r], U[..., :, :r], V[..., :, :r] # ... implies any number of extra batch axes\n",
+        "  # Compute the low-rank approximation and its size\n",
+        "  A_r = tf.einsum('...s,...us,...vs->...uv',s_r,U_r,V_r)\n",
+        "  A_r_size = tf.size(U_r) + tf.size(s_r) + tf.size(V_r)\n",
+        "  if verbose:\n",
+        "    print(f\"Approximation Size: {A_r_size}\")\n",
+        "  return A_r, A_r_size\n",
+        "\n",
+        "def viz_approx(A, A_r):\n",
+        "  # Plot A, A_r, and A - A_r\n",
+        "  vmin, vmax = 0, tf.reduce_max(A)\n",
+        "  fig, ax = plt.subplots(1,3)\n",
+        "  mats = [A, A_r, abs(A - A_r)]\n",
+        "  titles = ['Original A', 'Approximated A_r', 'Error |A - A_r|']\n",
+        "  for i, (mat, title) in enumerate(zip(mats, titles)):\n",
+        "    ax[i].pcolormesh(mat, vmin=vmin, vmax=vmax)\n",
+        "    ax[i].set_title(title)\n",
+        "    ax[i].axis('off')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "O3ZRkYCkX2FQ"
+      },
+      "outputs": [],
+      "source": [
+        "print(f\"Original Size of A: {tf.size(A)}\")\n",
+        "s, U, V = tf.linalg.svd(A)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "S1DR83VMX4cM"
+      },
+      "outputs": [],
+      "source": [
+        "# Rank-15 approximation\n",
+        "A_15, A_15_size = rank_r_approx(s, U, V, 15, verbose = True)\n",
+        "viz_approx(A, A_15)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "KgFT70XFX57E"
+      },
+      "outputs": [],
+      "source": [
+        "# Rank-3 approximation\n",
+        "A_3, A_3_size = rank_r_approx(s, U, V, 3, verbose = True)\n",
+        "viz_approx(A, A_3)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DS4XoSlTJgX0"
+      },
+      "source": [
+        "As expected, using lower ranks results in less-accurate approximations. However, the quality of these low-rank approximations are often good enough in real world scenarios. Also note that the main goal of low-rank approximation with SVD \n",
+        "is to reduce the dimensionality of the data but not to reduce the disk space of the data itself. However, as the input matrices become higher-dimensional, many low-rank approximations also end up benefiting from reduced data size. This reduction benefit is why the process is applicable for image compression problems."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "IhsaiOnnZs6M"
+      },
+      "source": [
+        "## Image loading\n",
+        "\n",
+        "The following image is available on the [Imagen](https://imagen.research.google/) home page. Imagen is a text-to-image diffusion model developed by Google Research's Brain team. An AI created this image based on the prompt: \"A photo of a Corgi dog riding a bike in Times Square. It is wearing sunglasses and a beach hat.\" How cool is that! You can also change the url below to any .jpg link to load in a custom image of choice. \n",
+        "\n",
+        "Start by reading in and visualizing the image. After reading a JPEG file, Matplotlib outputs a matrix, ${\\mathrm{I}}$, of shape $(m \\times n \\times 3)$ which represents a 2-dimensional image with 3 color channels for red, green and blue respectively."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OVsZOQUAZ2C7"
+      },
+      "outputs": [],
+      "source": [
+        "img_link = \"https://imagen.research.google/main_gallery_images/a-photo-of-a-corgi-dog-riding-a-bike-in-times-square.jpg\"\n",
+        "img_path = requests.get(img_link, stream=True).raw\n",
+        "I = imread(img_path, 0)\n",
+        "print(\"Input Image Shape:\", I.shape)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Qvs7uftcZ54x"
+      },
+      "outputs": [],
+      "source": [
+        "def show_img(I):\n",
+        "  # Display the image in matplotlib\n",
+        "  img = plt.imshow(I)\n",
+        "  plt.axis('off')\n",
+        "  return"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZbesXO3HZ6Qs"
+      },
+      "outputs": [],
+      "source": [
+        "show_img(I)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tdnUBVg_JoOa"
+      },
+      "source": [
+        "## The image compression algorithm\n",
+        "\n",
+        "Now, use the SVD to compute low-rank approximations of the sample image. Recall that the image is of shape $(1024 \\times 1024 \\times 3)$ and that the theory SVD only applies for 2-dimensional matrices. This means that the sample image has to be batched into 3 equal-size matrices that correspond to each of the 3 color channels. This can be done so by transposing the matrix to be of shape $(3 \\times 1024 \\times 1024)$. In order to clearly visualize the approximation error, rescale the RGB values of the image from $[0,255]$ to $[0,1]$. Remember to clip the approximated values to fall within this interval before visualizing them. The `tf.clip_by_value` function is useful for this."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "i7DDp0h7oSIk"
+      },
+      "outputs": [],
+      "source": [
+        "def compress_image(I, r, verbose=False):\n",
+        "  # Compress an image with the SVD given a rank \n",
+        "  I_size = tf.size(I)\n",
+        "  print(f\"Original size of image: {I_size}\")\n",
+        "  # Compute SVD of image\n",
+        "  I = tf.convert_to_tensor(I)/255\n",
+        "  I_batched = tf.transpose(I, [2, 0, 1]) # einops.rearrange(I, 'h w c -> c h w')\n",
+        "  s, U, V = tf.linalg.svd(I_batched)\n",
+        "  # Compute low-rank approximation of image across each RGB channel\n",
+        "  I_r, I_r_size = rank_r_approx(s, U, V, r)\n",
+        "  I_r = tf.transpose(I_r, [1, 2, 0]) # einops.rearrange(I_r, 'c h w -> h w c')\n",
+        "  I_r_prop = (I_r_size / I_size)\n",
+        "  if verbose:\n",
+        "    # Display compressed image and attributes\n",
+        "    print(f\"Number of singular values used in compression: {r}\")\n",
+        "    print(f\"Compressed image size: {I_r_size}\")\n",
+        "    print(f\"Proportion of original size: {I_r_prop:.3f}\")\n",
+        "    ax_1 = plt.subplot(1,2,1)\n",
+        "    show_img(tf.clip_by_value(I_r,0.,1.))\n",
+        "    ax_1.set_title(\"Approximated image\")\n",
+        "    ax_2 = plt.subplot(1,2,2)\n",
+        "    show_img(tf.clip_by_value(0.5+abs(I-I_r),0.,1.))\n",
+        "    ax_2.set_title(\"Error\")\n",
+        "  return I_r, I_r_prop"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "RGQ_rTyKDX9F"
+      },
+      "source": [
+        "Now, compute rank-r approximations for the following ranks : 100, 50, 10"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7GlKkVLGDjre"
+      },
+      "outputs": [],
+      "source": [
+        "I_100, I_100_prop = compress_image(I, 100, verbose=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "XdvUkF5_E75D"
+      },
+      "outputs": [],
+      "source": [
+        "I_50, I_50_prop = compress_image(I, 50, verbose=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "MsCNZ8416Sbk"
+      },
+      "outputs": [],
+      "source": [
+        "I_10, I_10_prop = compress_image(I, 10, verbose=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "RfYYBhcuNkvH"
+      },
+      "source": [
+        "## Evaluating approximations\n",
+        "\n",
+        "There are a variety of interesting methods to measure the effectiveness and have more control over matrix approximations."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "D2Lotde9Zg7v"
+      },
+      "source": [
+        "### Compression factor vs rank\n",
+        "\n",
+        "For each of the above approximations, observe how the data sizes change with the rank."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "O1ariNQe6Wbl"
+      },
+      "outputs": [],
+      "source": [
+        "plt.figure(figsize=(11,6))\n",
+        "plt.plot([100, 50, 10], [I_100_prop, I_50_prop, I_10_prop])\n",
+        "plt.xlabel(\"Rank\")\n",
+        "plt.ylabel(\"Proportion of original image size\")\n",
+        "plt.title(\"Compression factor vs rank\");"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dvHcLRj2QoDg"
+      },
+      "source": [
+        "Based on this plot, there is a linear relationship between an approximated image's compression factor and its rank. To explore this further, recall that the data size of an approximated matrix, ${\\mathrm{A}}_r$, is defined as the total number of elements required for its computation. The following equations can be used to find the relationship between compression factor and rank:\n",
+        "\n",
+        "$$x = (m \\times r) + r + (r \\times n) = r \\times (m + n + 1)$$\n",
+        "\n",
+        "$$c = \\large \\frac{x}{y} = \\frac{r \\times (m + n + 1)}{m \\times n}$$\n",
+        "\n",
+        "where\n",
+        "\n",
+        "* $x$: size of ${\\mathrm{A_r}}$\n",
+        "* $y$: size of ${\\mathrm{A}}$\n",
+        "* $c = \\frac{x}{y}$: compression factor\n",
+        "* $r$: rank of the approximation\n",
+        "* $m$ and $n$: row and column dimensions of ${\\mathrm{A}}$\n",
+        "\n",
+        "In order to find the rank, $r$, that is necessary to compress an image to a desired factor, $c$, the above equation can be rearranged to solve for $r$:\n",
+        "\n",
+        "$$r = ⌊{\\large\\frac{c \\times m \\times n}{m + n + 1}}⌋$$\n",
+        "\n",
+        "Note that this formula is independent of the color channel dimension since each of the RGB approximations do not affect each other. Now, write a function to compress an input image given a desired compression factor."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "viVO-I60QynI"
+      },
+      "outputs": [],
+      "source": [
+        "def compress_image_with_factor(I, compression_factor, verbose=False):\n",
+        "  # Returns a compressed image based on a desired compression factor\n",
+        "  m,n,o = I.shape\n",
+        "  r = int((compression_factor * m * n)/(m + n + 1))\n",
+        "  I_r, I_r_prop = compress_image(I, r, verbose=verbose)\n",
+        "  return I_r"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gWSv58J6LSRQ"
+      },
+      "source": [
+        "Compress an image to 15% of its original size."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "HVeeloIwQ1b6"
+      },
+      "outputs": [],
+      "source": [
+        "compression_factor = 0.15\n",
+        "I_r_img = compress_image_with_factor(I, compression_factor, verbose=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "LkeRyms7jZMd"
+      },
+      "source": [
+        "### Cumulative sum of singular values\n",
+        "\n",
+        "The cumulative sum of singular values can be a useful indicator for the amount of energy captured by a rank-r approximation. Visualize the RGB-averaged cumulative proportion of singular values in the sample image. The `tf.cumsum` function can be useful for this."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CteJ6VbKlndu"
+      },
+      "outputs": [],
+      "source": [
+        "def viz_energy(I):\n",
+        "  # Visualize the energy captured based on rank\n",
+        "  # Computing SVD\n",
+        "  I = tf.convert_to_tensor(I)/255\n",
+        "  I_batched = tf.transpose(I, [2, 0, 1]) \n",
+        "  s, U, V = tf.linalg.svd(I_batched)\n",
+        "  # Plotting average proportion across RGB channels \n",
+        "  props_rgb = tf.map_fn(lambda x: tf.cumsum(x)/tf.reduce_sum(x), s)\n",
+        "  props_rgb_mean = tf.reduce_mean(props_rgb, axis=0)\n",
+        "  plt.figure(figsize=(11,6))\n",
+        "  plt.plot(range(len(I)), props_rgb_mean, color='k')\n",
+        "  plt.xlabel(\"Rank / singular value number\")\n",
+        "  plt.ylabel(\"Cumulative proportion of singular values\")\n",
+        "  plt.title(\"RGB-averaged proportion of energy captured by the first 'r' singular values\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Vl9PKow-GgCp"
+      },
+      "outputs": [],
+      "source": [
+        "viz_energy(I)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vQtwimKuQP19"
+      },
+      "source": [
+        "It looks like over 90% of the energy in this image is captured within the first 200-250 singular values. Now, write a function to compress an input image given a desired energy retention factor."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fum5Cvm7R5vH"
+      },
+      "outputs": [],
+      "source": [
+        "def compress_image_with_energy(I, energy_factor, verbose=False):\n",
+        "  # Returns a compressed image based on a desired energy factor\n",
+        "  # Computing SVD\n",
+        "  I_rescaled = tf.convert_to_tensor(I)/255\n",
+        "  I_batched = tf.transpose(I_rescaled, [2, 0, 1]) \n",
+        "  s, U, V = tf.linalg.svd(I_batched)\n",
+        "  # Extracting singular values\n",
+        "  props_rgb = tf.map_fn(lambda x: tf.cumsum(x)/tf.reduce_sum(x), s)\n",
+        "  props_rgb_mean = tf.reduce_mean(props_rgb, axis=0)\n",
+        "  # Find closest r that corresponds to the energy factor\n",
+        "  r = tf.argmin(tf.abs(props_rgb_mean - energy_factor)) + 1\n",
+        "  actual_ef = props_rgb_mean[r]\n",
+        "  I_r, I_r_prop = compress_image(I, r, verbose=verbose)\n",
+        "  print(f\"Proportion of energy captured by the first {r} singular values: {actual_ef:.3f}\")\n",
+        "  return I_r"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Y_rChG0OLby1"
+      },
+      "source": [
+        "Compress an image to retain 75% of its energy."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xDXBaZQ4c5jF"
+      },
+      "outputs": [],
+      "source": [
+        "energy_factor = 0.75\n",
+        "I_r_img = compress_image_with_energy(I, energy_factor, verbose=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2tmqTW0CYX-v"
+      },
+      "source": [
+        "### Error and singular values\n",
+        "\n",
+        "There is also an interesting relationship between the approximation error and the singular values. It turns out that the squared Frobenius norm of the approximation is equal to the sum of the squares of its singular values that were left out:\n",
+        "\n",
+        "$${||A - A_r||}^2 = \\sum_{i=r+1}^{R}σ_i^2$$\n",
+        "\n",
+        "Test out this relationship with a rank-10 approximation of the example matrix in the beginning of this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hctOvN8BckiS"
+      },
+      "outputs": [],
+      "source": [
+        "s, U, V = tf.linalg.svd(A)\n",
+        "A_10, A_10_size = rank_r_approx(s, U, V, 10)\n",
+        "squared_norm = tf.norm(A - A_10)**2\n",
+        "s_squared_sum = tf.reduce_sum(s[10:]**2)\n",
+        "print(f\"Squared Frobenius norm: {squared_norm:.3f}\")\n",
+        "print(f\"Sum of squared singular values left out: {s_squared_sum:.3f}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vgGQuV-yqYZH"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "This notebook introduced the process of implementing the singular value decomposition with TensorFlow and applying it to write an image compression algorithm. Here are a few more tips that may help:\n",
+        "\n",
+        "*   The [TensorFlow Core APIs](https://www.tensorflow.org/guide/core) can be utilized for a variety of high-performance scientific computing use cases.\n",
+        "*   To learn more about TensorFlow's linear algebra functionalities, visit the docs for the [linalg module](https://www.tensorflow.org/api_docs/python/tf/linalg).\n",
+        "*   The SVD can also be applied to build [recommendation systems](https://developers.google.com/machine-learning/recommendation/labs/movie-rec-programming-exercise).\n",
+        "\n",
+        "\n",
+        "For more examples of using the TensorFlow Core APIs, check out the [guide](https://www.tensorflow.org/guide/core). If you want learn more about loading and preparing data, see the tutorials on [image data loading](https://www.tensorflow.org/tutorials/load_data/images) or [CSV data loading](https://www.tensorflow.org/tutorials/load_data/csv)."
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "matrix_core.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

From 8e7b0c3d29502b1aba25cf94dffd6656c29ecba4 Mon Sep 17 00:00:00 2001
From: synandi <98147397+synandi@users.noreply.github.com>
Date: Wed, 17 Aug 2022 09:36:13 +0530
Subject: [PATCH 274/872] Updated a link in pix2pix.ipynb

Changed the link in the line 81
---
 site/en/tutorials/generative/pix2pix.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/en/tutorials/generative/pix2pix.ipynb b/site/en/tutorials/generative/pix2pix.ipynb
index 5c97053c50b..5b96a011c54 100644
--- a/site/en/tutorials/generative/pix2pix.ipynb
+++ b/site/en/tutorials/generative/pix2pix.ipynb
@@ -78,7 +78,7 @@
         "\n",
         "The architecture of your network will contain:\n",
         "\n",
-        "- A generator with a [U-Net]([U-Net](https://arxiv.org/abs/1505.04597))-based architecture.\n",
+        "- A generator with a [U-Net](https://arxiv.org/abs/1505.04597)-based architecture.\n",
         "- A discriminator represented by a convolutional PatchGAN classifier (proposed in the [pix2pix paper](https://arxiv.org/abs/1611.07004)).\n",
         "\n",
         "Note that each epoch can take around 15 seconds on a single V100 GPU.\n",

From 875db999374298afcc8d7f429a33eea51d43c9f3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Johannes=20Ball=C3=A9?= 
Date: Wed, 17 Aug 2022 08:00:34 -0700
Subject: [PATCH 275/872] Fixes one-line scripts to install TFC.

PiperOrigin-RevId: 468198363
---
 site/en/tutorials/generative/data_compression.ipynb | 5 ++++-
 site/en/tutorials/optimization/compression.ipynb    | 6 +++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/site/en/tutorials/generative/data_compression.ipynb b/site/en/tutorials/generative/data_compression.ipynb
index b8edf946ae3..b6c043c0598 100644
--- a/site/en/tutorials/generative/data_compression.ipynb
+++ b/site/en/tutorials/generative/data_compression.ipynb
@@ -104,8 +104,11 @@
       },
       "outputs": [],
       "source": [
+        "%%bash\n",
         "# Installs the latest version of TFC compatible with the installed TF version.\n",
-        "!pip install tensorflow-compression~=$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d\\.\\d).*/\\1.0/sg')\n"
+        "\n",
+        "read MAJOR MINOR <<< \"$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d+)\\.(\\d+).*/\\1 \\2/sg')\"\n",
+        "pip install \"tensorflow-compression<$MAJOR.$(($MINOR+1))\"\n"
       ]
     },
     {
diff --git a/site/en/tutorials/optimization/compression.ipynb b/site/en/tutorials/optimization/compression.ipynb
index 75b5517aa57..b94ecaf6476 100644
--- a/site/en/tutorials/optimization/compression.ipynb
+++ b/site/en/tutorials/optimization/compression.ipynb
@@ -122,7 +122,11 @@
       },
       "outputs": [],
       "source": [
-        "!pip install tensorflow-compression~=$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d\\.\\d).*/\\1.0/sg')\n"
+        "%%bash\n",
+        "# Installs the latest version of TFC compatible with the installed TF version.\n",
+        "\n",
+        "read MAJOR MINOR <<< \"$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d+)\\.(\\d+).*/\\1 \\2/sg')\"\n",
+        "pip install \"tensorflow-compression<$MAJOR.$(($MINOR+1))\"\n"
       ]
     },
     {

From 59912d4f2f1be3851638b632b42d3422e7285464 Mon Sep 17 00:00:00 2001
From: 8bitmp3 <19637339+8bitmp3@users.noreply.github.com>
Date: Wed, 17 Aug 2022 10:18:58 -0700
Subject: [PATCH 276/872] Lint links in conditional GAN pix2pix tutorial

---
 site/en/tutorials/generative/pix2pix.ipynb | 25 +++++++++++-----------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/site/en/tutorials/generative/pix2pix.ipynb b/site/en/tutorials/generative/pix2pix.ipynb
index 5b96a011c54..3026ae5500f 100644
--- a/site/en/tutorials/generative/pix2pix.ipynb
+++ b/site/en/tutorials/generative/pix2pix.ipynb
@@ -70,16 +70,16 @@
         "id": "ITZuApL56Mny"
       },
       "source": [
-        "This tutorial demonstrates how to build and train a conditional generative adversarial network (cGAN) called pix2pix that learns a mapping from input images to output images, as described in [Image-to-image translation with conditional adversarial networks](https://arxiv.org/abs/1611.07004) by Isola et al. (2017). pix2pix is not application specific—it can be applied to a wide range of tasks, including synthesizing photos from label maps, generating colorized photos from black and white images, turning Google Maps photos into aerial images, and even transforming sketches into photos.\n",
+        "This tutorial demonstrates how to build and train a conditional generative adversarial network (cGAN) called pix2pix that learns a mapping from input images to output images, as described in [Image-to-image translation with conditional adversarial networks](https://arxiv.org/abs/1611.07004){:.external} by Isola et al. (2017). pix2pix is not application specific—it can be applied to a wide range of tasks, including synthesizing photos from label maps, generating colorized photos from black and white images, turning Google Maps photos into aerial images, and even transforming sketches into photos.\n",
         "\n",
-        "In this example, your network will generate images of building facades using the [CMP Facade Database](http://cmp.felk.cvut.cz/~tylecr1/facade/) provided by the [Center for Machine Perception](http://cmp.felk.cvut.cz/) at the [Czech Technical University in Prague](https://www.cvut.cz/). To keep it short, you will use a [preprocessed copy]((https://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/datasets/)) of this dataset created by the pix2pix authors.\n",
+        "In this example, your network will generate images of building facades using the [CMP Facade Database](http://cmp.felk.cvut.cz/~tylecr1/facade/) provided by the [Center for Machine Perception](http://cmp.felk.cvut.cz/){:.external} at the [Czech Technical University in Prague](https://www.cvut.cz/){:.external}. To keep it short, you will use a [preprocessed copy](https://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/datasets/){:.external} of this dataset created by the pix2pix authors.\n",
         "\n",
         "In the pix2pix cGAN, you condition on input images and generate corresponding output images. cGANs were first proposed in [Conditional Generative Adversarial Nets](https://arxiv.org/abs/1411.1784) (Mirza and Osindero, 2014)\n",
         "\n",
         "The architecture of your network will contain:\n",
         "\n",
-        "- A generator with a [U-Net](https://arxiv.org/abs/1505.04597)-based architecture.\n",
-        "- A discriminator represented by a convolutional PatchGAN classifier (proposed in the [pix2pix paper](https://arxiv.org/abs/1611.07004)).\n",
+        "- A generator with a [U-Net](https://arxiv.org/abs/1505.04597){:.external}-based architecture.\n",
+        "- A discriminator represented by a convolutional PatchGAN classifier (proposed in the [pix2pix paper](https://arxiv.org/abs/1611.07004){:.external}).\n",
         "\n",
         "Note that each epoch can take around 15 seconds on a single V100 GPU.\n",
         "\n",
@@ -125,7 +125,7 @@
       "source": [
         "## Load the dataset\n",
         "\n",
-        "Download the CMP Facade Database data (30MB). Additional datasets are available in the same format [here](http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/). In Colab you can select other datasets from the drop-down menu. Note that some of the other datasets are significantly larger (`edges2handbags` is 8GB). "
+        "Download the CMP Facade Database data (30MB). Additional datasets are available in the same format [here](http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/){:.external}. In Colab you can select other datasets from the drop-down menu. Note that some of the other datasets are significantly larger (`edges2handbags` is 8GB in size). "
       ]
     },
     {
@@ -274,7 +274,7 @@
         "id": "PVuZQTfI_c-s"
       },
       "source": [
-        "As described in the [pix2pix paper](https://arxiv.org/abs/1611.07004), you need to apply random jittering and mirroring to preprocess the training set.\n",
+        "As described in the [pix2pix paper](https://arxiv.org/abs/1611.07004){:.external}, you need to apply random jittering and mirroring to preprocess the training set.\n",
         "\n",
         "Define several functions that:\n",
         "\n",
@@ -490,7 +490,7 @@
       "source": [
         "## Build the generator\n",
         "\n",
-        "The generator of your pix2pix cGAN is a _modified_ [U-Net](https://arxiv.org/abs/1505.04597). A U-Net consists of an encoder (downsampler) and decoder (upsampler). (You can find out more about it in the [Image segmentation](https://www.tensorflow.org/tutorials/images/segmentation) tutorial and on the [U-Net project website](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/).)\n",
+        "The generator of your pix2pix cGAN is a _modified_ [U-Net](https://arxiv.org/abs/1505.04597){:.external}. A U-Net consists of an encoder (downsampler) and decoder (upsampler). (You can find out more about it in the [Image segmentation](../images/segmentation.ipynb) tutorial and on the [U-Net project website](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/){:.external}.)\n",
         "\n",
         "- Each block in the encoder is: Convolution -> Batch normalization -> Leaky ReLU\n",
         "- Each block in the decoder is: Transposed convolution -> Batch normalization -> Dropout (applied to the first 3 blocks) -> ReLU\n",
@@ -722,7 +722,7 @@
       "source": [
         "### Define the generator loss\n",
         "\n",
-        "GANs learn a loss that adapts to the data, while cGANs learn a structured loss that penalizes a possible structure that differs from the network output and the target image, as described in the [pix2pix paper](https://arxiv.org/abs/1611.07004).\n",
+        "GANs learn a loss that adapts to the data, while cGANs learn a structured loss that penalizes a possible structure that differs from the network output and the target image, as described in the [pix2pix paper](https://arxiv.org/abs/1611.07004){:.external}.\n",
         "\n",
         "- The generator loss is a sigmoid cross-entropy loss of the generated images and an **array of ones**.\n",
         "- The pix2pix paper also mentions the L1 loss, which is a MAE (mean absolute error) between the generated image and the target image.\n",
@@ -797,7 +797,7 @@
       "source": [
         "## Build the discriminator\n",
         "\n",
-        "The discriminator in the pix2pix cGAN is a convolutional PatchGAN classifier—it tries to classify if each image _patch_ is real or not real, as described in the [pix2pix paper](https://arxiv.org/abs/1611.07004).\n",
+        "The discriminator in the pix2pix cGAN is a convolutional PatchGAN classifier—it tries to classify if each image _patch_ is real or not real, as described in the [pix2pix paper](https://arxiv.org/abs/1611.07004){:.external}.\n",
         "\n",
         "- Each block in the discriminator is: Convolution -> Batch normalization -> Leaky ReLU.\n",
         "- The shape of the output after the last layer is `(batch_size, 30, 30, 1)`.\n",
@@ -937,7 +937,7 @@
       "source": [
         "The training procedure for the discriminator is shown below.\n",
         "\n",
-        "To learn more about the architecture and the hyperparameters you can refer to the [pix2pix paper](https://arxiv.org/abs/1611.07004)."
+        "To learn more about the architecture and the hyperparameters you can refer to the [pix2pix paper](https://arxiv.org/abs/1611.07004){:.external}."
       ]
     },
     {
@@ -1007,8 +1007,7 @@
         "id": "Rb0QQFHF-JfS"
       },
       "source": [
-        "Note: The `training=True` is intentional here since\n",
-        "you want the batch statistics, while running the model on the test dataset. If you use `training=False`, you get the accumulated statistics learned from the training dataset (which you don't want)."
+        "Note: The `training=True` is intentional here since you want the batch statistics, while running the model on the test dataset. If you use `training=False`, you get the accumulated statistics learned from the training dataset (which you don't want)."
       ]
     },
     {
@@ -1248,7 +1247,7 @@
       "source": [
         "You can view the [results of a previous run](https://tensorboard.dev/experiment/lZ0C6FONROaUMfjYkVyJqw) of this notebook on [TensorBoard.dev](https://tensorboard.dev/).\n",
         "\n",
-        "TensorBoard.dev is a managed experience for hosting, tracking, and sharing ML experiments with everyone.\n",
+        "[TensorBoard.dev](https://tensorboard.dev){:.external} is a managed experience for hosting, tracking, and sharing ML experiments with everyone.\n",
         "\n",
         "It can also included inline using an `